YARM - Resource Monitor

by Narc

This mod helps you to keep track of your mining sites, with useful data such as the percent mined, and estimated time to depletion.

Utilities
7 months ago
0.14 - 1.1
124K
Mining

b Loading lag

4 years ago

I cant explain it but only whem I have this mod my loading time of saves jumps to something of 1 Minute. Is that normal?

4 years ago

It's normal for loading times to get a little bit longer, but to go that far you'd have to have an awful lot of ore tiles being monitored (or a slow disk, maybe). Can you do me a favor? Load your game and run /c remote.call("YARM", "how_many_entities_tracked", game.player.name), and let me know the number of entities tracked.

Note: you'll have to run the command twice if you haven't used console commands in this save (the game will tell you if that's the case). Also, if you don't want to have achievements disabled, don't save that game again -- just exit to main menu after you get the numbers.

4 years ago
(updated 4 years ago)

Its an private server which needs restart a bit more than normal. Its a relative big save.
I use RSO mod.

Answer to command : Tracking 53175 entitys

4 years ago

That's a fair few entities, but it shouldn't be affecting load times by that much -- my test map had over 62,000 ore entities being tracked and it delayed loading by only about 20 seconds, certainly not a whole minute. That said, maybe my disk happens to be a lot faster (or maybe it's RAM-bound? It's really Factorio internals we're running into here). So, without knowing much more about your setup, I think it could be a normal amount of slow-down.

The good news is you're only paying that cost at load time, and it's making it possible to maintain all 53,175 entities without dropping your UPS -- which the old implementation definitely would've done. Less good is that load time includes whenever a player joins (they have to load the map that gets sent by the server on join), but it's still a better trade-off than reducing UPS and making the whole in-game experience noticeably worse.

The bad news is there's only one way to prune the tracked entities at the moment, and it's an awful hack: you'd have to disable YARM, load and resave the game without it, then re-enable YARM and only add sites you really want to keep (deleting a site doesn't stop its ore from being tracked -- it keeps tracking until the entity runs out).

I intend to change things in the future so that YARM automatically stops tracking ores that become unimportant, but that's going to be a while -- I'd really like to only track ores that are in range of a miner, but with various kinds of modded miners this means making sure I'm not missing any. Probably in the short run I'll set up a way to force a manual prune, or even a periodic automated cleanup, but even that is still going to be a while.

4 years ago
(updated 4 years ago)

CPU: AMD Ryzen 5 1600 Six-core Prozessor
Ram:16Gb
Festplatte: St1000LX015-1U7172
GPU: Raedon(TM) RX580

4 years ago
(updated 4 years ago)

I know this is a little late but I noticed this started when you did a rewrite of the ore_tracker. (commit dc3323546331fd478e0ea6a68ad6b5e16e23ad3a). Large patches with a lot of entities started causing slow saves and loads. It becomes real noticable due to the autosaves. I stopped using it due to this. I tried it again recently hoping it was fixed but it still has the same problem.
I was investigating and observed that you are saving a reference to the entity itself in the global.ore_tracker.entities[].entity property. I believe this is causing the entire factorio entity into your mod save file thus resulting in a very large save and load time. I don't know the feasibility but you likely need to remove these references before save and then rebuild the references on load. I could be wrong on this as I did not spend a lot of time investigating. I really enjoyed this mod and I am missing it's use.
Also, by way of example to demonstrate it is the size of the mod save causing save/load performance issues. On a save file before I use YARM my script.dat file in the save file zip is only 746kb. After I enable YARM and add 4 large RSO patches the script.dat file grows to 4MB.

4 years ago
(updated 4 years ago)

@waranator: What I previously stated remains valid -- many tracked entities lead to some performance issues regardless of tracking method -- but there's a possible fix that would require changing the way the mod works somewhat.

Before we get to that, though: there is no method for implementing on-save or on-load behaviour, otherwise I would like to do exactly what you said: retain only coordinates of the ore entities and link to the entities as needed -- not even at load time, but on first query, and retain the references thereafter. That would give the best performance envelope, with simple lists of numbers (coordinates) being very fast to load out of script.dat, and world.find_entities_filtered being restricted to a few calls that then don't need to be repeated until the next time the map is loaded.

There are basically two ways to do ore tracking, and a third method that would work even better, but will be more difficult to implement. In order:

  • you can keep track of entity coordinates (very fast saving/loading, they're just two lists of numbers) and do find_entities_filtered to acquire an entity reference when you need to ask it how much ore it has left (slow runtime, you have to do a lot of find_entities_filtered and it adds up). You can't make a transitory cache (one not persisted in global) because that would lead to (rare, but guaranteed) desyncs.
  • you can keep track of actual entity references (slow saving/loading, Factorio itself has to save a permanent identifier and then reconnect the entity reference on load), and just ask the entity how much ore it has whenever you're interested (very fast runtime, the entities are already right there). There's no opportunity to speed things up here aside from a periodic pruning of tracked entities, which would occasionally slow down the runtime but would allow removing unnecessary tracked entities from the savegame.
  • as a hybrid, you can place some (YARM defined) mining drill entities and set them to count connected ore (one entity per contiguous ore patch); you'd still save the entity references, but there would be orders of magnitude fewer of them (tens instead of tens of thousands), and Factorio itself does the ore counting very quickly (in C++) so the runtime would be more than acceptably fast. The disadvantage is that there must be these entities that get placed in the world and that must be managed (and which must never actually mine any ore).

Actually, on that note: we could have YARM tag ordinary mining drills directly -- then there's no need for us to define a mining drill entity, nor to stop it from doing its intended job. This would remove a lot of complexity from the code, but would force players to manually update their tracked sites.

I'd like to actually implement one of the hybrid approaches, but either of them is non-trivial effort and most of my time has been taken by IRL stuff -- until I get the opportunity to work on YARM again, you'll have to live with the save/load performance being scuttled in favor of runtime performance. Of course, you're welcome to fork the code and maybe pull request the changes -- it's open source precisely so that someone else can take over when I can't.

New response