Not the author, but I was really looking into compatibility before starting a new save so I might be able to help here. Author please correct me or add to this if you want to though, as I haven't fully played the mod to understand, but this is what I understand so far. (But don't worry, I have pictures, its not just "Source: trust my assumptions)
Huh, this is super weird, I guess I just don't even understand what this mod is supposed to do but I can kind of answer your question.
First off, I don't think having the solar system edge is a problem. I tried a new save with only this mod, and it was still there, so that's all fine, PlanetsLib is just showing up because that's what it does. (proof shown in image later, "Solar System Edge" appears as an effect of the "Promethium Science Pack" research that this mod still includes)
But secondly, I don't think there's any direct incompatibilities. Or at least any more than vanilla (only Nexus) would have.
The thing is even without Maraxsis or Tenebris, you're still getting double promethium science researches. Here is a super scuffed ms paint compilation of the tech trees of vanilla Nexus and modded Nexus. Theyre basically identical.
Nexus itself never removes the default "Promethium science pack" technology, but still adds its own "Promethium research" technology that gives the exact same science pack as a reward, except the Nexus version also includes some Lab. But the issue is still that science pack you mentioned. Why are we getting it twice? I thought the second one would be a more efficient recipe maybe, but its just the same. Either way, my point is it isn't an issue with compatibility between Maraxsis and Tenebris, so you don't have to worry about that. Its still all there with Nexus standalone.
You're right in the difference about "no gatekeep promethium", as those planets adding their science packs to the vanilla "Promethium research pack" though, but the thing is since none of the problem is on Tenebris/Maraxsis end its still all self-contained and fine from them. There's no issues, those mods are optional as they already describe
If you don't install those "xyz no gatekeep promethium", you'll still be able to play perfectly fine. Maraxsis/Tenebris are meant to be gone to after Aquilo, and for that reason by default also "gatekeep" promethium behind their science, and I think that is probably a good intended play order but it doesn't have to be that strict.
I assume Nexus is a Promethium based mod because it wants to do end-game changes, and for that reason in my brain I think it makes sense to keep it entirely after Aquilo -> Maraxsis -> Tenebris -> Nexus, so its in the intended progression order. Of course you can install the no gatekeep mods if you want to just head straight to Nexus and not be bothered with Tenebris/Maraxsis, but I'd assume the whole point of downloading them is to play them, so keeping the intended play order is probably the best way to do it.
Bonus: today was the day I realized that this mod uses the exact same planet icon as Tenebris. It isn't credited in the description or anything though, I've forgotten pieces in my mod credits sometimes so I get it but I just wanted to point this out as well.
tldr;
Taking a look at the science trees alone, there is no reason that Maraxsis/Tenebris should be incompatible. And the solar system edge is still supposed to be there. (see image linked earlier, it is a reward in the base mod w/o additions)
The mod author could come in and correct me on half of this, so take it with a grain of salt, but this is what I've found / thought at the least. (Too busy worrying about incompatibilities to actually ever play lmfao </3) but as of right now it looks like this should still all be fine. The double prom research is definitely weird though and something I'm questioning about this mod by itself, but except for this mod wanting to give you Prom science as a research in one of its own technologies (which it knows you already need to unlock earlier to even achieve this second prom technology?), there isn't anything "incompatible", just this mod being WIP.