I've some thought and ideas I'd like to share, maybe you'll find some of it useful enough to implement.

1. Crafting recipe ingredients
   Provide a proper crafting recipe with some ingredients. While this won't make much difference in terms of balacing (it shouldn't in a cheaty-as-hell mod lol), it would at least provide some material demands, it also would allow other mods like Reverse Factory to disassemble the surplus items back into ingredients, rather than throwing the stuff into a wooden box and shooting it up with a shotgun.

1. Chest sizes
   Different network chest sizes, i.e. 2x2, 4x4, 6x6 etc. with corresponding slot capacities like in https://mods.factorio.com/mod/aai-containers would be pretty nice. This would allow to utilize a smaller amount of "bigger" chests for the same number of producing facilitites. Right now the tightest bottleneck in heavily modded endgame factories is the rather slow network chest processing speed. And you've mentioned it elsewhere that you don't wanna speed that up since it would completely bog down the entire game, which I fully agreee with. However with 50k+ "small" network chests you also start noticing a periodical hiccup in the UPS. So less network chests could "fix that". At least that's my understanding, correct me if you have different experience on that matter.
   Also how about the option (checkbox in the chest's GUI or so) to auto-merge neighboring chests into 2x1, 3x1, 4x1 with corresponding amount of slots (48, 96, 144...). Check out the Merging Chests mod for further info.

1. Loader speeds
   Speaking of bottlenecks, how about providing the network loader with a GUI option to adjust item speed. There are mods for up to 360 items per second. I don't know if manipulating the speed of belt/loader/splitter entities by the player is possible or whether those values are locked per recipe, but this certainly would be a nice idea to make your network loader seamlessly integrate with the various belt mods out there. Or alternatively allow your network loader to automatically adjust own item speed according to the belt it is connected to.

That's all basically. Some of it might be out of scope for this mod or the modding API in general (idk really), just let me know what you think about it.

Thanks for the thoughtful feature ideas. I've had many similar thoughts so figured I'd share how my current plans align with this.

1. I've also been thinking of changing the recipe to require some ingredients. If I do add a recipe it will still be cheaty (think 1 iron plate), but I know others have tweaked their version of the mod to require end-game ingredients. My main concern is that adding a recipe is a "breaking change" and could hurt the experience of current players, especially if they are using a mod that disables or changes iron plates.

2. Supporting different chest sizes is definitely feasible but I want to separate that from performance issues. When updating chests, the mod currently spends most of the time just resetting filters on every slot so there are big opportunities to make it faster. See this issue for more thoughts: https://github.com/year6b7a/item-network-factorio-mod/issues/27. What I've found from my usage of this mod is I'm rarely restricted by chest size and 4 loaders are almost always enough for transportation needs for parts of my factory. It sounds like you've built a massive map (50K chests!!!) and I'd love to see screenshots or take a look around if possible so I can see how other people use the mod.

3. Loader speed can't be changed while the game is running but I can just increase it. I set it to the speed of the fastest belt in Exotic Industries so it's arbitrary and can be changed, hopefully without messing up factorio's super efficient belt code.

I was wondering whether utilizing the LinkedContainer principle would solve the Request & Provide processing queue issue as well as the contents problem described here.

The current mod concept indicates, that each placed down item network container entity is unique with unique contents. It also means it has to be processed as a unique entity and with the growing amount of containers, the processing queue also grows, while the prototype mentioned above seemingly allows to have a shared container with instantly available conents everywhere at the same time.

It seems to me, the "shared container" principle, (if I'm not completely wrong here) would allow to skip the entity check process. The problem with huge amounts of item network containers is not even the UPS drag as I initially assumed, but the fact, that some containers are randomly placed so far back in the queue and have to wait for so long to be processed, that I have to set the Request order to the maximum possible amount of given resource to avoid bottlenecking production, oftentimes one dedicated network chest is not even enough for one machine. The situation with network tanks is similar, especially when modded machines from mods like SE, KS2 etc.

The obvious drawback that comes to mind is, that it's a shared storage after all, so if there's no [item X] on the network, there's none in the shared chest either, in case you wanted to prioritize a particular assembly line for resource supply before others.
The second issue is, there must be some option to limit the input into the network as you probably don't want 10 billion inserters in your item network and prefer to use the crafting ingredients elsewhere.

Maybe this can be solved by allowing to reserve n amount of items like with the requester chest, but then again I'm clueless whether a bunch of same-id containers can have different logistic attributes.

That's really cool that your factory is large enough to process so many resources. It's great to hear about the challenges you run into as I plan out next steps of the mod.

You are correct that each network chest has a unique entity id and unique buffer contents.

Linked containers are great for transporting items between 2 fixed chests but don't have the load balancing advantages of network chests where there can be N producers and M consumers for an item. If it were possible to link more then 2 chests then linked chests would be a better solution over Network Chests.

I've been considering a ton of potential solutions including but not limited to
- Make the chest update code faster.
- Write some clever algorithm that detects when Network Chests have more throughput and give them a higher priority in the queue.
- Add priorities to chests, similar to OS process scheduling
- Make a "high volume network chest" that has a separate update queue from normal network chests. This also has the opportunity to design HVNCs to be aggressively optimized.
- Use huge (like 100x100) linked chests on another surface as load balancers to avoid updating chests directly.

Unfortunately the best advise I can give for now is to reduce the number of network chests in your factory. This is a tricky problem but definitely something I'm interested in solving.

Make the chest update code faster.

How about dynamically adjusting the interval of processed containers relative to their total amount across all surfaces?
The lower the amount of containers, the faster they are processed, as the amount of containers grows, the interval returns to its current default value.
This probably won't solve the mega factory issue, but will satisfy early-mid game or compact factories in general. At least this could be a good first temporary optimization attempt, until the mod evolves further.

Write some clever algorithm that detects when Network Chests have more throughput and give them a higher priority in the queue.

I guess this would require some sort of container traffic schedule with priority categories, let's say [1-5] where [1] has the highest processing priority. Sounds like the most promising but also the most coplex option, since the algorithm has to be tested and fine-tuned thoroughly.

Add priorities to chests, similar to OS process scheduling

Similar to the above.

Make a "high volume network chest" that has a separate update queue from normal network chests. This also has the opportunity to design HVNCs to be aggressively optimized.

As an additional entitiy, if understand you correctly. Sounds actually useful.

Use huge (like 100x100) linked chests on another surface as load balancers to avoid updating chests directly.

Never thought of this one but wouldn't that suffer from the issue you described above, that linked chests aren't well optimized (or not intended at all) as buffers? Probably worth trying and finding out.

Anyways here's a new batch of utterly random thoughts as per the usual principle "hmm-this-might-be-actually-nice-to-have".

• GUI Button: Guy in the other thread asked for an inventory window, not knowing there already is (a hidden) one, that you open via keyboard hotkey. Might be worth adding a GUI button to the left top of the corner like many other mods do, incl. the option to hide it. Also check out the GUI Unifier mod which was created to somewhat unify that chaos.

• Logic network connectivity: Allow connecting, reading and possibly even configuring container entities via combinator logic. Especially the ability to mass-confige container entities would be an absolute blessing. You could create your own type of an item network command interface which offers granular mod-specific controls, better than the current combinators.

• Inventory QOL: Speaking of the inventory, here's some QOL ideas.

  • Items separated into categories for easier navigation. Here's a visual example of what I had in mind. This is the original train window overview, that I photoshopped a bit by adding item network elements.
  • Tooltip when hovering pointer over an item tile: Exact item name, exact amount (if possible with decimals for fluids). With growing amount of mods, it's hard to memorize and tell apart which slightly-color-edited plate was which type of metal, and the 10m is too abstract due to the fact that it's something between 10,000,000 and 10,099,999.
  • Same as above but a small popup window when clicking an item tile.
  • Show amount of network container entities, per type, per-surface and in total. Right now you either use console command to get that info or something like Factory Search.

That's all I have for now. Let me know what you think.

Agree that most of features would be nice to have. I've been thinking about a few of these already and here's a list of small issues that capture a good bit of this functionality:

• https://github.com/year6b7a/item-network-factorio-mod/issues/34
• https://github.com/year6b7a/item-network-factorio-mod/issues/31
• https://github.com/year6b7a/item-network-factorio-mod/issues/35
• https://github.com/year6b7a/item-network-factorio-mod/issues/36

While all these are nice-to-have, they don't seem to prevent anyone from using the mod so they fall after chest-update-logic-performance work in priority. These would be great issues if anyone else wants to contribute to the mod since I probably won't get to them for a month or two.

original train window overview

Which mod did you get the base picture from? Is it from Cybercyn?

Which mod did you get the base picture from? Is it from Cybercyn?

It's the default train overview window. You find the button above the minimap or use hotkey "O".

• Container Capacity: What do you think about adding a startup option for chest and tank capacity? Let's say the chest can range from 48 to 1000 slots and the tank from 5k to 100k, for example. This might help overcome the bottleneck issue by using fewer containers in high productivity setups.

• Network Tank HP buff: Is the tank supposed to have 10HP value as opposed to the 200HP chest and 300HP loader? Seems like a typo to me.

• I've looked into implementing dynamic container capacity but I don't think it's possible since that value is set in the prototype stage rather than when loading a save. If that is possible then it makes sense.
• The network tank HP is definitely a bug. https://github.com/year6b7a/item-network-factorio-mod/issues/40 for tracking progress.

It's definitely possible to set those values in startup settings, as seen in some other mods.
Anyways I've been experimenting with M.NUM_INVENTORY_SLOTS = 1000 and M.TANK_AREA = 1000 which works fine but for whatever reason uses more CPU time. Any idea why?

EDIT:
Check out my benchmark save where I compare 6k vs 6k between "Item Network" and "Quantum Network Distribution" mods. I modified my src/constants.lua with the values mentioned above. Toggle [x] show-lua-object-statistics via debug settings (F4) and pay attention to the item-network CPU time.
[ IN x QRD ] - 6k vs 6k Benchmark.zip

It's definitely possible to set those values in startup settings, as seen in some other mods.

Oh cool I'll take a look into that.

Thanks for sending that save. It's cool to see the comparison between the two mods. With the default IN mod I'm seeing 1.5ms/tick for IN and 2.2ms/tick for QND. It's nice to see that even with fancy features like buffers and limits, IN is still competitive with this other mod.

Quick question, is it intentional that the Network Chests in that map are only buffered to 500 items? Even with the default 48 slots you should be able to raise the buffer a lot more to take advantage of the space. I don't think raising M.NUM_INVENTORY_SLOTS = 1000 will improve throughput if you don't also raise the buffer in each chest.

And for:

M.NUM_INVENTORY_SLOTS = 1000 for whatever reason uses more CPU time. Any idea why?

The most expensive part of the chest updating code is where resets slot filters for every slot in the inventory. This means that when you raise the chest to have 1000 slots, it has to loop through 1000 expensive operations instead of 48 expensive operations.

Also I highly recommend taking a look at the Factorio Modding Tool Kit if you want to see which lines of code take up the most time.

Quick question, is it intentional that the Network Chests in that map are only buffered to 500 items?

Yep, the purpose is to keep the mod permanently running with all containers involved, in order to test how both mods behave in extreme conditions with practically zero idle time. This simulation is very close to my SE multi-surface factory, which I sadly can't upload due to it being ~2.2GB in size.

The most expensive part of the chest updating code is where resets slot filters for every slot in the inventory.

I've come to realize that, after modifying the stack size of items to 50x via Noxys StackSizeMultiplier, now the containers use 1-2 slots instead of 500-1000 and the mod runs super smooth again.
Is it actually possible to have independent stack sizes for entities? Let's say your mod comes with a 100x stack for the network chests while the rest of the entities and player inventory remains vanilla? To avoid updating large amount of slot filters in an equally large amount of network chests.

As a matter of fact I've figured that I can somewhat overcome the throughput bottleneck issue by using the stack size mod and go back to 48 slots chests. And to reduce the amount of chest entities on the ground, I'm using Ultimate Belts and Deadlock's Stacking Beltboxes & Compact Loaders which provide belts and loader with up to 450 items/s. I guess the last thing left to add here would be a proper recipe scaling mod.

It seems none of the Factorio devs expected people would shoot waaaaay higher than 60 items/s crafting and the logistical challenges involved in brainmelting gigafactory layouts, and neither have you! :P

Also thanks for the modding toolking link.

Is it actually possible to have independent stack sizes for entities? Let's say your mod comes with a 100x stack for the network chests while the rest of the entities and player inventory remains vanilla?

If it's possible I would definitely uptake it but I haven't found anything in the LuaInventory API :(.

You're right, your factory is way larger than anything I've tested. But it sounds like a cool test case. How many chests do you have in your 2.2Gb factory? What's your belt speed? About how many belts per chest?

It seems that stack_size in LuaItemPrototype is read only and cannot be manipulated on run time.

About the base, I'm currently trying to heavily scale down the amount of chests, so each one updates more often and compesate it with the higher stack value as mentioned above. Before I had about 46k network chests with default 48 slots where basically each assembler would use 2 network chests (in/out), which allowed me to avoid belts alltogether. Then I scaled down to about 2k chests experimenting with 1000 slots and classic belt lines but that caused an even bigger resource hog as mentioned above. Now im aiming for max. 1-2 provider and 1 collector type of network chests (far left input >->-> belts >->-> far right output) with 4 attached loaders to each chest, and at least 2 belt lanes (450 items/s).

A network chest with bigger bounding box, like 1x2, 2x2, 4x4 would come in handy, since the 1x1 chest has only 4 sides to attach loaders to it.