I'm not sure if it should be possible through remote view (especially if you aren't even on the same surface as the chest), since the functionality is supposed to mimic the player physically moving the items from one inventory to the other.
Hmm, I can probably add a check to ensure the player is at least on the same surface as the items being moved (reliant on a setting, of course), but apart from that, I don't see any reason for it not to work that way. It's a manual process that should be automated by the player, so it's up to the player's prerogative if they want to abuse a quality of life feature in that way.
EDIT: Ahhh, I was wondering why this was weird in my testing. Pressing the hotkey again over a non-chest entity is what clears the selected chest, and allows picking from a new chest. That's quite non-obvious... and for some reason, there was a specific section of code that prevents overwriting the first copy. I don't really see any a reason why it was set up this way, removing that bit of code and it seems to work exactly as expected.