As of now I'm not sure what triggers it, but I have had the hide big recipes option not work in mod settings. so far it only happens during gameplay. I'll open my crafting menu and all the tabs will be doubled up with all the big recipes in one set and the normal recipes in another set. when i go to check the mod setting, it is still applied, but i can fix it temporarily by toggling it off, hitting confirm, going back to the mod settings page and then toggling it back and hitting confirm.

Are you in multiplayer? I have to turn it on/off for the whole team (only way to do it), so you'll get that kind of thing if your teammate has one of the WSF dialogs open.

I finally figured it out and apparently when I went to go report it, it has already been fixed. From the 0.17.9 changelog:

Fixed that mod settings could get scrambled/reset when adding removing or changing mods.

That was the issue. The recipe-hiding was getting turned off when adding/removing/changing mods. Thanks again for your report!