All the buildings can be destroyed.
What are the invisible buildings? The only ones are the electric poles, which you can see persist with their copper wire connections. A single grenade will destroy them, you do collide with them as well and you can run over them with vehicles if needed.
I'd need some more information about what these buildings are? If you place something over them, it will say what entity is in the way. Does that happen?