Disabling the changes to attack and movement speed in settings after the mod has already run on a save results in this:

The mod Biter Nest Quality! (1.1.0) caused a non-recoverable error.
Please report this error to the mod author.

Error while running event BiterNestQuality::on_entity_spawned (ID 146)
Unknown entity name: rare-behemoth-spitter
stack traceback:
[C]: in function 'create_entity'
_ BiterNestQuality _ /control.lua:70: in function < _ BiterNestQuality _ /control.lua:62>

The initial warning popup does mention removing a single rare-behemoth-spitter among plenty of other things. Trying a previous autosave produces a similar result but with a rare-big instead.
Maybe it was in the process of being created or something?...

Upd: The temporary solution seems to be completely removing the mod, loading in naturally to let the game remove all the new entities properly, waiting for at least a couple minutes(yes this was actually necessary), making a new save, adding the mod back in and loading into the new save with changed settings.
Not exactly the most convenient way to change settings and it likely wiped a bunch of enemies that didn't need to disappear at all, but at least its something.

...aand an hour later it threw out the exact same error out of nowhere anyway.

Okay after spending a while trying to understand how this modding thing works, inserting this at line 64 in control.lua does the trick.

if settings.startup["more_biter_quality_scaling"].value == false then goto continue end

The mod doesn't actually create fake version of enemies if there is no need to change their stats, but it still tries to replace stuff with those fake versions even if they don't exist.
Edit: Yep can now load the savefile that originally started this issue.

I updated it the way you said it, thanks a lot and sorry you had to sort through this yourself. <3