Thank you for your answer. I can confirm that the lag scales with the inventory size.
SE has robot attrition. When a robot network size is above (50?), robots will randomly crash and the character with active personal roboport will pick up the stuff the robot carried. So my inventory was filled up with random stuff and I had to clean up regularly. Which is either a manual process or a lot of trash configurations (since there is a lot items).
To circumvent this, I just installed "Quality of Life Research", set the inventory bonus to 5000 to never bother with that again. Not that I needed 5000, but the problem was solved for good :D I never changed that setting in my SE+K2 setup - altough I use "robot attrition - return to sender" to prevent the random pickup of the content of crashed robots in recent games. When I reduced the bonus to 250, the lag went down from ~10s to ~1.5s.
To me, this still feels too unresponsive to use.
From your post I take that nav view is considered a character switch technically. Is it possible to set the presumably hidden character used during nav view to the same amount of inventory slots as the play character to prevent the need for a backup? Or would it be possible to have an option to disable the character backup alltogether? Then we could work around the lag problem by setting the backup to enabled if we actually want to switch real chars and just play with the option disabled for 99% of time (nav view). I personally chose a character at the start and stick to it in one game.
I uploaded my current savegame. The factory won't work properly as I unloaded my personal unpublished mod, which adds certain inserters and adjusts things. 3000 entities were removed, but the game loads without it and the nav view still takes ~1.5s with 250 bonus slots.
https://file.io/chARo8rTAOAA