I've been trying this, but I might be going about it wrong. Is it possible to support modded planners (selection-tool prototype) like tree feller?

Apparently it is not possible directly.
I see 2 options: the mod provides an API to its function, or I copy the desired function into my mod.

That's unfortunate, there's a lot of useful alternative planners provided by mods. The interface certainly sounds like the cleaner approach.

I thought about this idea a bit.
Firstly, the selection tools cannot be placed in a book. Because of this, I will have to make a separate command for each tool. (not difficult, but I'll have to do it by request)
Secondly, some tools, frankly speaking, are cheating. Therefore, the functions will be disabled without the appropriate mods installed.

That's fair - I wouldn't expect surprise "features" without the relevant mods installed. To be honest, it seems like a big thing to take on but if you're interested more power to you.