Hm, while the current implementation (leaving all unsupported inserters behind in the original subgroup) isn't entirely wrong, it would probably be better to make some kind of attempt to sort the ones that aren't fully supported.
The sorting logic is actually pretty annoying to deal with for what it is; it's basically entirely hard-coded, with the only major differences depending on the number of modded inserters in the game.
Under mostly vanilla conditions (less than 10 total inserters), then they can all fit in one row, which are sorted by base, long (in the order of burner, yellow, blue, bulk, stack). This excludes Heat, Nutrient, and Pipe, which I believe get sorted into different rows, separate from the vanilla ones.
If there are less than 20 total inserters, then they can all fit into two rows, which are separated by base and long variants (in the order of burner, yellow, blue, bulk, stack). Modded inserters generally fall next to the most appropriate inserters, like steam next to burner, faster-non-bulk after blue, faster bulk after bulk, faster stack after stack, etc. Oh, and Pelagos gets its own rows as well, for their diesel powered inserters (since there are multiple and use different fuel type)
And then the most common scenario, more than 20 total inserters or complex burner setting enabled; row 1 gets burner, steam, Pelagos diesel; row 2 gets base electric; row 3 gets long variants; and then heat/nutrient/pipe inserters fall into rows 4-6 (yes, we seriously need that many rows)
And then of course, anything that's not in the list gets left behind.
I suppose if I wanted to do this automatically, I'd probably keep the simplest variation as it is (under 10 inserters, use manually coded sorting), otherwise group by fuel type (burner/steam/diesel), then electric arm length 1, electric arm length 2, electric arm length.. more than 2, and then my heat/nutrient/pipe inserters after that. Hmmm... I'll work on that after my other work is completed (currently focused on space things)