There is no "size of the factory", you can build stuff anywhere, so the mod has the check the entire revealed map.
Now that I think about it, I guess it isn't necessarily needed when a new mod is added (since the new entities from that mod can't exist on the map yet), but it might still be needed on removal. Depends on if Factorio tells my mod that an entity was removed (I doubt it does).
I'll take a look at what can be done, in the meantime you can change the size of the recalculate batches to a large number, this will cause a single large freeze in the game instead of a drawn out lag.