They connect on all sides, as if you had placed a regular piece of pipe on the same tile as the inserter.
I think it should be theoretically possible to make it perpendicular only (or straight across only), but it would require a bit more work (and I need it to be unrestricted for my own purposes, so this would likely be a setting that is disabled by default.)
Tbh, the ideal situation would be to just make pipes and inserters not collide with each other at all, allowing any of them to be placed in the same position (so you could just pick whatever orientation you want from Flow Control or something), but I have no idea how to even begin doing that sort of thing.
