I feel there are two main factors here:
1/ Firstly it is about the number of available spawnable (alien) unit types. The game seems to consume more processing power with the number of unit types. I am not sure about the exact cause, but it could be related to the unit spawn calculations. The spawn probability (or weight) of each unit type depends on the current evolution factor, which is calculated each frame (if the game does not use cunning tricks to optimize that). So installing a lot of biter mods will add such burden.
You can count how many unit types in your game! Vanilla game has 8 types. Using this mod up to tier 10 makes this number to 20 types. Now you can continue the count with other mods you are using...
2/ Secondly it about the unit size. Large unit has larger size in both VRAM usage and UPS/FPS. (As you mentioned the UPS drop in a new world, this factor is less relevant as no rendering is involved yet.)
So yeah, use only what you and your computer can handle. The max setting (first being 10, next 20, currently 15) is there does not mean it is suitable for everyone.
The default 6 already provides 50% more than vanilla, which should be challenging enough for vanilla players. In my personal solo game I am using 10, where I feel comfortable with the help of my own weapons mod series.