This is more of a general question vs a specific RSO question - though you could treat it as a feature request if you prefer.

I was curious if Factorio ore gen could be modded to do more interesting shapes than just the kind of 'random circular-ish with resources slightly less dense toward the edges'

Like strings that look more vein-like. Or bullseye patches with the resources densest at the edges. Small spots within a larger distribution area. Mixed ores but only for certain ore type(s). Ores that specifically spawn in certain biomes or near water. Etc.

Is that outside the purview of modding or just something noones put together, or maybe my googling has failed me?

Thats a lot of questions at once. Factorio base system can actually tie ores to biomes etc, RSO can't due to different generation. RSO uses something called Meta Balls thats why most patches are some variant of circular blob. There is an option to enable donuts in RSO that will make the patches more varied. I didn't try to look into actual vein like resources.
I also don't know how much base game system can do since it's quite complex and I didn't play with it a lot.

Interesting! Sorry to roll so many questions together. Also wow that was a much faster reply than I expected.

Metaballs like a distance field the specifies resource density from a number of points you generate per ore patch?

Kk. Sounds like if you've entirely rolled your own generation algorithm there's untapped potential out there.

There certainly is - game ore patch generation was upgraded I think at least twice from the time RSO was created.
It's certainly possible now to get quite spread out ores similar to RSO in base game (at least with not to many ores present, AFAIK it gets tricky if mod adds like 5 more ores).

Yeah I was thinking about this for Py that ends up just looking like a rainbow of polka dots due to how many ores end up getting spawned. If there was a way to make that more interesting than just adjusting frequency, which of course can cause issues to due to random dispersion allowing for too low of a local population of something near start.