Having built a similar mod that attempts to do something like this recently, I'm intrigued by the slightly different approach you took. Admittedly I'm not sure which of our two techniques is (generally) better; it probably very much depends on the use case.

Anyway, having slaved over my solution for this, I know how much of a pain attempting to accomplish this can be. So nice work and glad to see someone else trying to solve this problem. Long live HUGE logistics networks!

Aaron

P.S. Search "Tactical Construction" if you want to take a look at what I did. PRs welcome too but ofc no pressure :-)

Ya know, I wonder if a hybrid approach between our two methods could solve a lot of problems. The main one with mine is the annoyance of the player living on the wrong game 'force' while the feature is toggled on, which ultimately can be pretty annoying. But yours I imagine would have trouble with the 'niceties' of the placed templates (detailed attributes of the templates), since you are deleting them and re-creating them.

What if instead of deleting and re-creating the ghost entity templates, you instead used my trick of moving them between forces. But instead of moving them over to the alternate force along with the player, you just moved them to the alt force and then back on the next tick? Would that trigger the robots to re-evaluate the construction orders?

Edit: of course the downside would be the creation of the alternate force in the game, which is definitely dirty. So you'd lose that benefit of your approach.

Hello Aaron,

thanks for the praise, it's greatly appreciated! I wasn't aware of your mod. I'll definitely have a look at it.

Just changing forces of entites sounds interesting on the one hand, but on the other hand, as you've already mentionned there's the downside of creating a separate force only for this mod.

I also thought of PvP games (who plays this? surely people do...) as well. That's the reason I've included the "force" parameter in all the function calls, even if it's just optional. So you can't run around and mess with your opponents base using this mod. The last player isn't rigurously preserved, I think. But I believe that's alright for the moment.

Again, I have to look at your mod; great name, by the way!

Chris

Edit: Wow, you've been featured by Xterminator! Nice!

Edit2: Haha! I love your approach! (Gonna read the code now) I think fiddling with force would be rather cumbersome with my selection tool, but the "roboport area mode" is functionally similar to your mod. (Mine needs to be activated by the player and runs once)

I skimmed your code and I have to say... MAN, OH MAN! I can only imagine the history of all those functions. I'd have pulled out all my hair by now.

"Let's just switch player forces!"
...
- Damn! I don't see the base anymore.
- Where are the researched technologies?!
- Where have my logistics requests gone?
- Why are the bots not working? What!? Each bot has its own force assignment?
- ...

:-)

I mean, the approach is great and it has definitely an advantage over my very simple method. (i.e. running around like a madman and prioritizing all the construction orders)
The downside is all the auxilliary code, that is neccessary to make it work without significantly altering the game state/interfering in the game. And not being able to remove the mod without repercussions (losing parts of your base, because it's still part of the wrong force) unfortunately is a no-go for me. I've always liked, that you can try out mods, disable them again if you don't like them and continue your game as if nothing had happened.

What I've learned by reading your code is, that I definitely don't want to mess around with the forces, LOL! Also, I might "steal" bits of your code, because I was looking for a method to continuously give the personal bots priority (one user already asked about this).

What I didn't understand yet is what you meant by "the 'niceties' of the placed templates"? What is a "placed template"? Do you mean entity-ghosts? If so, which 'niceties' am I losing? I haven't noticed that the mod messes things up, does it?
I just clone the original ghost directly on top of itself, delete the original and keep the copy. I'm not aware that I'm losing information. (I got the idea from a bug in @seancheey's BeltRouter. He was unintentionally placing ghosts on top of each other.)

Also,
Long live HUGE logistics networks!

Chris

Hi Chris!

Thanks for your response & various thoughts! I've had a lot of fun building this mod (also a lot of frustration too) so it's nice to be able to chat with someone else who appreciates the complexities of this type of feature and can also offer advise.

Re: "What I didn't understand yet is what you meant by "the 'niceties' of the placed templates"? What is a "placed template"? Do you mean entity-ghosts?"

Yeah, entity ghosts, sorry. I use my own made up verbiage for everything in this damn game :-)

Re: "If so, which 'niceties' am I losing? I haven't noticed that the mod messes things up, does it?"

So, I only stumbled on your mode late yesterday so I haven't had a chance to try it out. But I would have expected that deleting and then re-creating an entity-ghost would clear any of its advanced attributes such as circuit connections, logistic requests slots, etc. But from the sound of it, this actually isn't a problem given your approach due to cloning first, and then deleting? If so, that's pretty cool (I wouldn't have expected that was possible)!

Re: complexity of the code, definitely each function is representative of the painful process. It doesn't help that I didn't know any Lua when I started.

One of the most annoying things was the fact that switching the player force would 'detach' all of the player's robots for some reason. I still consider this a bug, but I found a workaround (which was manually re-attaching each follower robot back to the player after moving the player's force).

Re: "I also thought of PvP games"

You and I were thinking alike. In the case of my mod, I think PVP games should theoretically work because I make a point to create an alternate force per 'primary' force. And if the 'primary' force goes away then I garbage collect the alternative force corresponding to it at some point. And I try to avoid caching any force information about any player. But admittedly I've not tested it so there could definitely be bugs. Haha.

Re: being able to disable the mod without corrupting state; agree with your analysis 100% here. One idea I had was a console command which would tell the mod to clean up its mess and go into a zombie state. Granted, not NEARLY as good as the stated 'target' goal of being able to turn it off in the menu without repercussion (since you'd have to know to run this command first) but at least it would be a way to clean up your file.

The name "tactical construction" is partly an inside reference. When I was in college I worked on a game with some friends called Tactical Destruction.

Best Regards
Aaron

Hey Aaron,

Re: "to be able to chat with someone else who appreciates the complexities of this type of feature and can also offer advise"

Yes, I'm glad that you chatted me up, because simmering over such issues alone can be quite draining. Especially in times where we tend to be more alone than ever (with all the social distancing and stuff).

Ah, those "niceties"! LOL, I haven't tested so many different cases! I assume it'll retain all the connections because of the cloning. I was so glad to have found this function (a while back Klonan (i think) even denied the request for such a function, because there was no use for it), because I started to deal with all this stuff (circuit connections, iten requests, receipes, etc.). It was a pain!

I know, that currently a "module upgrade" doesn't work. I have yet to deal with those item_requests and I also try to make the feature togglable, so you can run around and reassign orders left and right.

What I don't know yet, maybe you have some insight you can share is if you can find out which bot has been assigned which work order?! This would help me immensely, because if I make the feature "permanently active" I have to distinguish between entities the personal robots do and those that base bots do to avoid reassigning the same order multiple times.

If your curious I created a branch in which I'm working on the feature being togglable https://github.com/ChrislyBear-GH/Factorio-BotPrioritizer/tree/Construction_Brush
The line where I'm stuck at the moment is here: https://github.com/ChrislyBear-GH/Factorio-BotPrioritizer/blob/596b4b650a31ac2feaf2f328754cb393b33312bd/control.lua#L198

All the best,
Chris

Hi Chris!

Re: "What I don't know yet, maybe you have some insight you can share is if you can find out which bot has been assigned which work order?!"

So, I (embarrassingly) decompiled part of the Factorio native code to try to figure out this very thing because I was initially desperate for ideas for how to accomplish the bot prioritization behavior. While my understanding of how it works is still very limited (and unfortunately, my memory is also fading a bit here since it's been about 6 months since I did that, and therefore came up with the approach I used based on that), I unfortunately don't believe there's any mapping from entity -> assigned bot. Thus, what you want to do is currently not feasible.

The bot tasking algorithm works (logically) as follows:

On every so many ticks:
--For each force:
----jobs = find_construction_jobs_that_need_a_bot(); // searches all entities in the game filtering based on ghost entities, upgrade requests, deconstruction requests, etc
----jobs = assign_any_available_player_bots(jobs);
----assign_base_construction_bots(jobs);

Notably, only the robots themselves know what job they're assigned to. I think the way a robot gets freed due to a canceled job is that it's job target disappears, and the game separately reviews each robot every so often. Therefore, if the robot's target job disappears, it eventually knows to mark itself free and return to port.

As a result of this unidirectional map of bot -> entity, I think conceptually the ONLY way you're gonna be able to find which bot is mapped to a particular entity is to iterate over every robot in the game and check if its job target is the entity you're interested in. The game just doesn't otherwise have the reverse mapping information available, even to its native code. (Of course I doubt this approach is feasible from a UPS standpoint; furthermore I don't recall there being any current Lua API to read this information out of a construction robot.)

Given this, you can see why I opted for the multiple-force approach in my mod; it is purely to trick the scheduling algorithm into "partitioning" the construction robot force so that the player's robots handle one part of the map (the nearby build orders, which are temporarily moved to the alternate force along with the player), while the base handles the remaining set. Your mod also accomplishes a similar thing by cloning/deleting the construction request. Since every time a new request appears it always gets player bots FIRST, by re-creating you force the old job to get canceled and the new jobs to be rescheduled (possibly with the player robots instead).

Hope this description helps. I honestly am having trouble thinking of a good way to work around this but I will keep chewing on it.

Best regards,
Aaron

Thanks for your in-depth answer! I scoured the forums and bugged people to come to the same conclusion. There's a bot to entity mapping but it's not available through the API. Too bad. I think I'll drop this for the moment.

Regarding the "niceties": You were right! The circuit connections do get messed up! I'll have to fix this.

All the best,
Chris

Re: circuit connections, actually, now that I think of it my approach might have similar issues if one of the 'ends' of the connection is moved to the alt force without the other. Will have to try that to see.

Regarding your problem, I wonder if what you're trying to do could be accomplished using a slightly different algorithm. For instance, what about something like:

• Assume you will do work (let's call this 'tick') every X seconds to look for jobs to be cloned/deleted (let's call this 'prioritized').
• Assume that most jobs - if successfully prioritized - will be accomplished in Y seconds at longest (where Y = n*X), else the base network is assigned to them and they need to be re-prioritized.
• Maintain a state Y/X lists of entities. Each of these lists will cache the 'refreshes' that took place in the corresponding 'tick'.

Every tick, do:

1. Discard oldest 'tick' list of refreshes. Move all tick lists one slot older in the list of lists.

2. Create a new empty 'refresh' list for this 'tick'.

3. Find ghost entities, etc in range of player roboport needing work done.

4. For each found job:
   ----- If this job is not in any of the list of lists of refreshes from all cached ticks, perform a refresh of this entity now and record it to the current tick's 'refresh' list.

Something like that. Suppose X = 2.5 sec and Y = 10 sec.

In this way, once a 'refresh' happens to a ghost entity nearby the player, it won't be refreshed again for 10 seconds (assuming the player stays in range). This would give the player's bots some time to fulfill the request. If the player's bots didn't get assigned, the job won't be done 10 seconds later and another refresh will happen.

Just a random idea. Definitely fuzzy logic :-)

Regards,
Aaron

Hey Aaron,

Thanks for the idea! I thought of a "timed" approach, but I discarded it, because I thought it would be just a crutch. Your approach though seems like a good solution! X and Y could even be configurable in the mod settings!

Currently I tried to do work every 6th tick (1/10 of a second) every second, and even every tick. BUT I have subscribed to the on_player_changed_position event for maximum compute time saving! I didn't try larger time settings though, because if you have lots of exoskeletons you'll miss entities while running around. But I think this could be just a drawback that I would have to accept and inform the users about.

Also, riffing off on your idea: I could just keep those lists of entites indexed by the actual tick as the key (AFAIK you get the tick with every event). This way I could do without the Y list. Just throw away all entries in the table with keys less than "current tick - Y*60".

This all seems totally doable and probably won't even bog down the game too much! I'll have to try it out!

And regarding those pesky circuit connections: After fixing the mod to also copy over those connections, I'd have probably missed the case you described! Idk how circuit connections work yet, but maybe we even have to exclude entites from reassignment when the circuit target/source isn't in range!? Hmmm...

Also: Have you dealt with item request slots yet? Like if you use the upgrade planner to change modules, trains requesting fuel, etc.(is there anything else?). Those aren't working yet as well.

All the best,
Chris

P.S.: Regarding response time: In which time zone are you located? I'm in CET and we currently have GMT+2.

Hi Chris,

Yeah, I mean, this 'hacky' approach might work. I agree with your comments regarding the algorithm implementation.

Ofc it's up to you whether you're gonna spend the time to try it :-)

Re: circuit connections just on the edge of the range

Yeah, not 100% sure how to handle that. I think before I did anything I would have to see what it does right now with both of our mods. I'll test it sometime and see....

Re: item request slots

My mod currently doesn't handle that unless the entity is originally a ghost entity when you toggle the local construction feature 'on'. Have you been able to detect entities that have an outstanding sub-item slot? I would be interested in how you did that.

I'm in Pacific Time (currently UTC-7).

Regards,
Aaron

Hi Aaron,

Re: Circuit connections

I'm currently using this method https://github.com/ChrislyBear-GH/Factorio-BotPrioritizer/blob/4bf9f41fb779fef62fe7b57d39d4ea8a2214ab21/circuit_manager.lua#L10 to successfully keep the circuit connections.
My trick is, to follow the connections one step to the entities that are connected to the entitiy I'm currently looking at. I can then execute the connect_neighbour() function to connect back to the entity I'm looking at (conveniently, the connection definition already exists over there, I just have to swap out the target_entity). This way I can connect up new entities and if I look at the next entity in my list, the new one is already connected. Repeat.

Re: Item Request slots

I'm currently exploring this myself. I don't have any solution for it, yet. My mod also just handles item request slots for ghosts (since I just clone them).

Best regards,
Chris

Edit: Item Request slots are rater easy to handle:

if entity.name == "item-request-proxy" then
                refreshed_entity = hlp.tbl_deep_copy(surface.create_entity({
                                                    name=entity.name,
                                                    target=entity.proxy_target,
                                                    position=entity.position,
                                                    force=force,
                                                    modules=entity.item_requests
                                                }))
                entity.destroy()

Only drawback is, that if you don't have the materials in your inventory and the request gets recreated, the base bots will never arrive... I may have to access the player's inventory to check if a fullfillment is possible... idk. But then I'll probably have to check the other cases too... hmmm...