Hrm, you're the second person to mention this issue, but I cannot reproduce it. My main map has 5,300 entities being tracked, and I can toggle on/off with the shortcut or the GUI toggle, and they disappear appropriately when destroyed.
What other mods are you using? Do you have a save file I can snag somewhere?