At first I thought it was the infinite worker carrying capacity mod, but it was actually this one causing the desyncs.

Mod List:
Factorio Library
Bottleneck Lite
More Roboport Charge Pads with Quality

I tried removing the charge pads mod and the desyncs haven't happened since then. I have had other issues, but those aren't the same thing. I'm running super huge megabases on 3 planets, so I'm definitely pushing limits. Also it's multiplayer with a dedicated server.

Desyncs are present in the current factorio releases, nothing to do with this mod. Check https://forums.factorio.com/viewtopic.php?f=30&t=121499 and other threads on the forum. This mod doesn't even add anything. It just changes 1 in-game value of the roboport entity. Desyncs can be caused by mods which have the control.lua file.

I've reported this multiple times already across many game versions, I hope they fix whatever is causing this in the next release. It happens on vanilla as well.