Thanks, I see now what you mean! If the character selector is disabled or shortcuts are used instead of buttons, I remove the toggle button and the button flow containing it as well as the button flow's parent element. There actually is a check that should make sure button flow + parent are only deleted if the toggle button was the only button in that flow, but it seems I messed up its logic. :-D
The fix should be replacing lines 696/697 in scripts/gui_stuff/gui_char_selector.lua with this:
if not minime.character_selector or
(not have_button and must_remove(buttons)) then
with this:
if (not minime.character_selector or not have_button) and
must_remove(buttons) then
Can you confirm that this works?