Is there a particular ordering to what items appear when there are more than 1000 ghosts? (Other than modules which I see were deprioritized.)

For each Logistics Cell (Roboport) in a Logistics Network requests are generated in the order:
Cliff Explosives > Upgrade Requests > Ghosts (including modules) > Request Proxies (modules without ghosts) > Tile ghost

So for example cell 1 can add tiles while cell 2 runs into scan limit during ghosts.

Is there any ordering within those categories? For example, do belt ghosts come before assembler or train stop ghosts?

Is it feasible to do each request on all cells rather than do all requests for each cell? That would avoid fetching aesthetic elements (e.g. tiles) in one cell when belts, assemblers, etc. are still pending in a connected cell.

Here's a example patch that should basically do that. In order to make the diff a minimum, I didn't do any indent changes. The "for method in 0..4" is certainly the wrong syntax, as may be other things. I've never coded in LUA before.

https://drive.google.com/file/d/12mXYIswJOp2wp53VRWUDZ2edzVIdcHfo/view?usp=sharing

BTW, I noticed an extra "ceil = math.ceil" in the original at line 343.

Wrapping an already terribly slow method like find_entities_filtered in another loop ruins any hope of acceptable performance.
Bots pick a "random" ghost to build anyway, so what does it matter if we scan all types per cell for ever so slightly better performance?

What should be possible is wrapping each find_entities_filtered in its own for _,cell in pairs(logsiticNetwork.cells) do .

PS thanks for noticing the unused localisation.

I'm thinking about it not in terms of what the bots build but in what I fetch for them to build with. I'd prefer to have my trains fetch the most important items first; the bots will be able to build only with those meaning important items get built first.

Regarding performance, I don't think there will be any difference. Instead of X * Y operations (stopping at the max) it's Y * X operations (stopping at the max). Though there is an extra outside loop of 5 iterations, the inside logic does only 1 of the 5 queries on each of those iterations.

I believe we're both suggesting the same thing just with different implementation... Though I think your version is less "hacky".

What do you think of "progressive scanning" in order to reduce jank? Instead of scanning the entire network every update, keep a cache of every cell's ghost count plus a total for the network and do something like this on every update:

• if the list of pending cells is empty, generate a new one and return (enough work for one update)
• otherwise, pop the next cell from the list
• subtract from the total the previous cached count for that cell
• do the scans on that cell and cache the new counts
• add the newly cached counts to the total
• export total

The limit setting would then be per-cell or could maybe be removed completely.

I think this would be a big improvement with regard to jank especially for those who need to know all ghosts (i.e. no limits). What I don't know is if it's feasible memory-wise.

What do you think?

Progressive scanning is only masking the underlying issues.
The only way this mod will ever get decent performance is to switch the entire thing from scanning everything each time to event based updates. However key events are not raised by the API for this:

https://forums.factorio.com/viewtopic.php?f=221&t=74375
https://forums.factorio.com/viewtopic.php?f=28&t=72240

Edit:
Added these threads to FAQ so I can find them easier as well.

1.6.1 scans each type over all cells before proceeding and doesn't utterly tank performance, it's still worse than 1.6.0.

That's largely what I'm proposing, though: Don't scan everything each time. Instead, scan only one cell each time.

It doesn't make for less computation, true, but it at least breaks it up so there shouldn't be noticable jank if somebody (e.g. me) needs a complete list of ghosts (limit=0). The scan for all cells on a network still has to be done in one piece but it doesn't have to be done with every piecemeal update of the ghosts.

Since I have independent networks per city block, the scan for cells is not a problem -- it's the scan for ghosts that causes the jank and that can be limited to one cell per update.