That's what I noticed when you create a house, put it up, a feeding universe is created. The downside is that this house is put forever on the map. It doesn't matter if it's deleted or destroyed!
These endless houses are always in the game, all content. What affects performance.

It seems to me that a function should be added to remove unused houses from the map so that they do not load the game and cause errors.

The unused buildings don't affect performance. It's the same as having some land in unloaded chunks.
In this mod, factory buildings are never destroyed. Even if they are killed via biters/nuke they will just drop on the ground as an item.