I faced the same problem as you did so I looked at the code of this mod to get at least general idea how it works and found interesting thing. It actually does not matter which area to expand you select, only matters the top left corner plus there is some kind of validity check in a code so probably it stops working when that corner is occupied already or you try to make a new group for expansion when old group is still on its way.
What works so far for me:
When I need to expand:
1) I take an area selection tool and select only 1 tile with no trees, resources and nests on it
2) Then I take a biter selection tool and carefully select only one biter.
3) Then I wait carefully when that biter goes and settles a new nest somewhere near that area.
4) Once I need new nest I repeat the process once again from #1
So far I managed to create around 25 nests in wide area around my base
