I wanted to add rso-mod to my headless server but when I started new map starting area was swarming with biters including behemoth worms. This does not happen when I start singleplayer map.

Here is the test I've done:
1st you can see screenshot of my headless map using default settings without rso-mod loaded. You can see biters are far and resources are 'vanilla'.
Then following screenshots show new map with rso-mod loaded (no other changes). Biters spawn next to me and behemoth worm is present.

https://imgur.com/a/IQ1A1VA

Game version: 0.17.38
Mod version: rso-mod_5.0.5

Can you check one thing - when you use F5 debug with detailed info there should be x,y coordinates of cursor location there.
Please post them here. It seems that for some reason game spawns you on a position that is quite far from 0,0. I'm not sure how headless differs from normal single player spawn. Do you have any other mods or scenario?

For the purpose of testing I removed all other mods.
It says 999, 1999 with rso-mod loaded. New save - same exact coordinates.

I have a copy of that 'behemoth' save if that helps in any way.

I'm sorry but I don't know how to help with that.
Game is spawning you pretty far from starting area and you land next to the enemy base. Why it would do that I don't know. RSO doesn't interact with player spawning in any way.
Do I understand correctly that simple fact of having RSO changes where the game will spawn your character?
Please post the save file I'll see if I can find something out from it.

I changed starting coordinates in server settings to 0x0 and seems that "fixes" the part of being spawned in deadly area. Thing I don't understand is why with rso loaded even given odd coordinates there's a developed base literally next to me while without it's all good.
File save: https://ufile.io/1d6hol6l

Hmm so server has starting coordinates as a setting. I'm not sure how that works but it seems that RSO doesn't support that option properly. Base game might move starting area appropriately based on that option. Or it's just different generation.

I think I figured it out. You had an island preset selected there - it seems that for some reason game moves starting area to different place when on island. I'll make RSO support it properly in next version.

I can test without 0_17-island property if you wish to double check?

I was wrong above - doing restart on your save would keep the offset but a new game with island preset wouldn't.
So it's strictly related to the offset setting I think and it will work properly in next version.