Some further testing showed, that it only happens for planets which already have stuff on when adding mid game. Will try to remember to do further testing when colonizing the next planet to see if the performance gets also bad, when it is enabled from the beginning.
What exactly does the mod do on player movements? I can't really think of anything needed for player movement.