I seem to be losing biter evolution when I destroy a spawner, I have not tested it with hand weapons, but with the Robotarmy mod I lose right now ~0.2 evolution everytime and the time / polution evolution factor goes bonkers. I have tried doing it manually and it also does the same thing, so I guess it might not be the robot army mod

Yeah, I have the same problem. 7h 30min into the game and the evolution factor is sitting at 0.0-0.1% most of the time, because I have been liberating ore patches from spawners.