Sometimes the option to hide the GUI button in the upper right corner of the screen does not work. I have to enable and then disable it again in the settings for the setting to take effect.
Is "sometimes" when you start a new game? Because the current version doesn't check the setting when creating the button, that will be fixed in the next update (probably sunday).