Initial scan


This mod does a large radar scan around every new player

Utilities
3 years ago
0.17 - 1.1
671

i UPS friendly ? memory friendly ?

5 years ago
(updated 5 years ago)

as nice as the idea is (i always was a bit annoyed to see a large area in the map preview and then ingame have to remember where all the ore patches, oil spots and lakes were that i had seen in the preview), as annoying it became when using this mod, although it is only one small detail missing to make it much nicer and really useful: a configuration option for the radius (at least, i couildn't find one in the options, and i don't want to edit some files).

when i started a new game, my FPS/UPS immediately dropped from 60 to 50 where it stayed for a while (at least for 2 hours realtime until the first scan is completed, including the generation of all the chunks). after almost two hours now (the first scan didn't complete yet), my FPS/UPS is now down to even 30-40 only :-(
you wrote "UPS friendly - scans one chunk every 5 ticks." but 12 chunks per second seems to still be a bit too fast (heavy on UPS, especially when the chunks need to be generated first) while i understand that it is a compromise to finish a full scan in less than 2 hours realtime.

save and autosave also became quite slow pretty soon, and currently it takes around 30 seconds each time. this is caused by the excessive radius of the scan and having revealed too many chunks. factorio now uses 30% CPU time and 5GB memory on my "not too bad" computer, still increasing. and the savegame size is 250-300 MB, increasing by almost 10MB every 3 minutes when more chunks are revealed.

nearing the radius, i also see that the corners of the scanned area are cut off, probably continueing the incremental scan of the area going along the edges of a square (like radar does), but not displaying (and still computing) what is outside the circle with 300 chunks radius. even though it might look nice to have a round area, this method is quite slow at the end when most chunks are not needed and it takes a while until the scan gets back into the displayed area again.

solution to all of the above:
- please add a mod option to specify the radius so that it can be adjusted to a usefully large/small area,
- scan a square (like normal radar or the preview) by default, maybe with an additional option to scan a circle
- have at most the size of the vanilla map preview as default(!) for the radius/edgelength
- and finally have one more setting for the number of ticks between scanning/generating two chunks so that people can select themselves how much UPS they want to sacrifice to get the full scan completed faster

ps: all of the above applies to singleplayer. i have no idea how much it would cost on a server where several people spawn at different locations and several such scans are done at the same time.

pps: just to show how insanely huge 300 chunks are: 302x32 is more than 10k, thus a train would take at least 5, maybe 10+ minutes to travel from one side to the other, 20 kilometers away ...
ppps: is the radius 300 or the diameter (edge length of the scanned radar square) ? if it is really 300 radius, the entire scan would probably be 280k to 300k chunks, taking 500 minutes or 8+ hours to scan at the current speed of 5 ticks per chunk (double that time for 30 UPS, triple for 20 UPS :-) but if it is only the diameter, this would match my timing of 2+ hours (a quarter of these numbers "only") for the scan ...

5 years ago

WOW. I AM STUNNED :D

I uploaded this mod to the portal just to simplify sharing it with few my friends who I occasionally play with (or intend to play with). I did not really intended it to be used by others. But it looks like I am in the modding business too now :D . I certainly will try to make the mod better for others as well as for me.

Lets have a look at your complaints and suggestions specifically:
- Game save size: this is unavoidable no matter what method one uses to reveal the map and is out of scope of this mod.
- Game save time: apparently factorio is terrible at saving a large table of numbers (I am learning), I will change this to compute the chunks that should be revealed to implicit expression rather than storing the coordinates queue.
- "i also see that the corners of the scanned area are cut off" this is weird. The mod scans actual square. Seeing the corners cut of might have something to do with the game prioritizing generating chunks by distance to players?
- Scanning time - I have 3 GHz cpu and running this mod takes about one third of cpu time (generating the chunks). However, I just realized that I was trying it with disabled pollution and biters. This might make a big difference.
- Radius: yes, it is radius, not diameter. The mod, as it is now, will reveal 600*600 chunks, which is, indeed, insane :D But I prefer to play railworlds :D I thought it good :D
- Configurable radius and frequency totally makes sense.

I was also considering an alternative, where the mod would be scanning indefinitely until the player presses a button.

To summarize, the mod is in its infancy and will be improved :D

5 years ago

I did not really intended it to be used by others. But it looks like I am in the modding business too now :D

welcome to the big community :-)

  • Game save size: this is unavoidable no matter what method one uses to reveal the map and is out of scope of this mod.

yes, there is nothing you can do about the size of savegames (and related to that also the time it takes to (auto)save a map), except for not generating/uncovering more chunks than needed. but for most playthroughs, i see no need to automatically and always reveal 300 chunks in radius. therefore i suggested to make it an option with a default of eg the size of the vanilla preview area, and to be only an option to extend it to 100, 200 or 300+ chunks of radius.

  • "i also see that the corners of the scanned area are cut off" this is weird. The mod scans actual square. Seeing the corners cut of might have something to do with the game prioritizing generating chunks by distance to players?

no, you show an almost perfect circle (as perfect as it can be with square chunks) after 3+ hours. but the scanning is done just like the radar scans, going around the edges of a square, and when that scan hits the border of the circle not showing those chunks outside of the circle. when i wrote the above ("cut off corners") the mod had just reached the bounding circle after two hours, and the "cut off corners" were those spots on the circumference that are nearest to the player (ie on the diagonals).
this is perfectly fine if you want to reveal a round area, but if you let the algorithm run as usual and then only not display those chunks outside of the circle, it costs lots of time in which seemingly nothing is done. and that "problem" (spending more time the more distant you scan, but showing less new chunks the more are outside the circle) gets more severe during the last third of the time your mod is busy.

  • Radius: yes, it is radius, not diameter. The mod, as it is now, will reveal 600*600 chunks, which is, indeed, insane :D But I prefer to play railworlds :D I thought it good :D

i think that even for a railworld such a large scan would be too much and half the fun is in exploring and adjusting what has been built. and i mostly would want it to do a scan to show me what i already have seen in the map preview.
for some later scanning, i use the ScanningRadar mod which can give end results similar to what your mod shows, but with the appropriate energy costs for scanning, etc.
anyway, everybody has his own playstyle and thus imho the most excessive radius (that then causes the problems with ups, savesize, savetime, etc) shouldn't be the default, and/or at least be adjustable with some mod option.

I was also considering an alternative, where the mod would be scanning indefinitely until the player presses a button.

have a look at the existing ScanningRadar mod. that mod already allows to set starting and ending angle of a sweeping scan by network circuitry, as well es inner and outer radius of the scan area, and energy consumption when scanning at different speeds. but as all those options are (at least for me) not needed at the beginning and also intended for the middle or even lategame, your mod would be a very nice addition to have one early scan that shows eg the same that the map preview already has shown (and thus i consider this "no cheating" :-)

To summarize, the mod is in its infancy and will be improved :D

which part of factorio and our factories is not always improved all the time ? :-)
good luck with your mod !

5 years ago

welcome to the big community :-)

Thanks :D


  • "i also see that the corners of the scanned area are cut off" this is weird. The mod scans actual square. Seeing the corners cut of might have something to do with the game prioritizing generating chunks by distance to players?

no, you show an almost perfect circle (as perfect as it can be with square chunks) after 3+ hours. but the scanning is done just like the radar scans, going around the edges of a square, and when that scan hits the border of the circle not showing those chunks outside of the circle. when i wrote the above ("cut off corners") the mod had just reached the bounding circle after two hours, and the "cut off corners" were those spots on the circumference that are nearest to the player (ie on the diagonals).
this is perfectly fine if you want to reveal a round area, but if you let the algorithm run as usual and then only not display those chunks outside of the circle, it costs lots of time in which seemingly nothing is done. and that "problem" (spending more time the more distant you scan, but showing less new chunks the more are outside the circle) gets more severe during the last third of the time your mod is busy.

I do not. There is NO code in the mod that has anything to do with circles. All it does is to reveal square centered at the players initial position with side equal to double the "radius". It does NOT hide any chunks after it generates them. However, since you observe such behavior, it is likely that the game does it on its own?


I do agree with your suggestion to make it configurable. I should have made that more obvious in my previous comment. :D

To be specific:
- I will make the mod configurable (radius, frequency)
- I will set more reasonable (smaller, slower) defaults
- I will fix what the mod needs to save - this will speed up saving/loading by some margin
- I will, in time, improve the behavior for games with many players.

I am at work right now, hopefully, I can get to it in the evening.


Finally, let me apologize that you had to experience this initial version of the mod. I myself did not test it this much yet.
But I really appreciate both your interest and your comments :D

5 years ago
(updated 5 years ago)

i just did some more tests about the circle, using the console:
/c for xx=0,6144,512 do for yy=0,6144,512 do if (xx+yy)>5500 then game.player.force.chart(game.player.surface, {{x = xx-16, y = yy-16}, {x = xx+16, y = yy+16}}) end end end

it seems to be some conflict with another mod which causes all scans outside of a radius of something like 5500 tiles to be suppressed, really strange, and i keep searching which of the many mods causes that, and most of all how that happens since it's really strange to cut off scans at a circular boundary :-(
when i used no mods, everything is ok, and also when using the (as far as i know) only mod that has circular radars. now I'll restart factorio with different subsets of mods until the problem happens again ...

edit: problem narrowed down: it happens on an absolutely new game, with only the space exploration mod which consists of 4 mods: the mod itself, graphics, hr graphics, postprocess, and two required dependencies (alien biomes and robot attrition). but still no idea why scanning is restricted to the inside of a circle with a bit more than 5500 tiles radius.
these mods also seem to cause some of the bad UPS since that command (see above) with pure vanilla is clearly faster (less ups loss).

I'll ask on that mod, but i already have one theory: thas modpack creates a small universe with several planets. maybe they have reduced nauvis from an endless plane to a "flat earth nauvis" with a radius of 5500+ tiles ?

the planned mod options (radius, frequency, better/smaller default) are much appreciated anyway.

5 years ago

And it is done :D
Version 0.7.0 is configurable with smaller defaults and has much faster saves and reloads :D


Interference with another mod has crossed my mind too, but only later on. I am glad you narrowed it down.

5 years ago

I enjoyed reading this thread and great work! I'll give this a go :)

5 years ago

thanks for the new options.
they look great although i didn't start a new map yet to see the effect of all your changes.
the new default for the range is 70 ... what is the range of the vanilla map preview ? I'll have to test ...

btw: there is a small bug that your mod shares with many (most?) other mods: the labels on mod options and/or the popup help when hovering them shows up as "unknown key: ..."

about the interference with another mod: i saw that mod being used by some youtubers and thus simply installed it without having read its mod page on the portal to "just experience the changed gameplay" and explore everything only ingame. thus i had no idea why some radar range might be restricted (and it probably really isn't :-). but seeing lots of planets and "spaceships", it appears quite obvious now that the radar can scan chunks that really are not there (not generated) outside of that circle :-)

5 years ago

I am beginning to wonder how the Space Exploration mod restricts the surface to a circle. I might try to look it up someday.

Thanks for the notice about the "unknown key". I was glad that I made all the other changes work that I forgot to add localization strings. I will fix that, eventually, too.

Let me know if you find the range of the map preview. It would , indeed, make a good default.

5 years ago
(updated 5 years ago)

because of some other problem (factorio can't update mods under certain conditions: "api/mods error 414", see https://forums.factorio.com/viewtopic.php?f=49&t=70726), i looked around for default directories and config files, and in a file config/config.ini i found several nice settings, many of them available in the ingame settings, but several only editable directly in that config file.
eg two lines that say "autosave-interval=3" and "; autosave-slots=3", the first of these available ingame to set the interval for autosave, and the second only settable in the config by removing the leading ";" to determine how many autosaves are done before the first one is overwritten again, default 3, but i use 9 now with an interval of 3 minutes to be able to skip back almost half an hour in small increments :-)
btw: i believe that this setting was available ingame in some older version like 0.12 or 0.13 :-)

a few lines further down, another line reads "; max-map-preview-chunk-side=64" and thus i would assume (not having tested it yet) that the default preview size is radius 32, and that it can be changed, which i probably will do since on my system the preview is slightly bigger than the window and certainly bigger than the selected special area around my base. thus it should also be good enough (and less cheaty!) to see only a smaller radius (maybe 25) in MY preview.

about the "unknown key": this effect shows up in several variants on several locations, eg as label in mod options, as well as on an (optional) popup help on mod options, and such an additional help text can also be added as popup help when setting the keyboard shortcuts (only very few mods -including factorio itself- do this, and only for a few of their keys) causing a small blue "?" circle to show up when such help is available.

5 years ago

Hi Anson_AKB,
great finding about the max-map-preview-chunk-side. I will look at this the next time I get to update the mod.
Thanks for letting me know.

5 years ago
(updated 5 years ago)

just found in the newest patchnotes of 0.17.48:

Scripting:
- Added a sanity check to LuaForce::chart() that errors on large sizes instead of trying to use 71,000 gigabytes of RAM.

i don't know whether or not that has influence on scanning round areas with a radar, but i found it interesting since we just talked about the range of scanning which uses this chart() function ...

5 years ago

70 TB. :D
sounds like an integer overflow to me :D that is likely much much further than what we were talking :D
but interesting finding anyway :D

On a second thought, "large SIZES" could mean scanning a lot of chunks at once. But Initial Scan does not do that either :D

5 years ago

to me it looks not related but at least similar to the last paragraph "Crashing on dereferencing null? Add a null check" in FFF#296 (https://factorio.com/blog/post/fff-296 :-)

for testing the old version of this mod (see somewhere above) i added some lines to scan quicker by calling the function 20 times per tick in a small loop, and already after a few seconds i got the debug message that the scan was finished, while in reality only a queue was generated and the scanning continued for another few hours. it's not always obvious what happens, and i would guess that factorio did something similar, trying to generate a long (up to 70+TB) list of "chunks to scan" relatively fast, and then slowly (throttled to N chunks per tick) working on that list. usually that wouldn't be bad and too distant chunks might even have been skipped, but only when they were handled "sometime later", temporarily causing lots of ram to be used up to hold the queue.
I absolutely love those reports about bugfixing in the FFFs 😍🤭😇

5 years ago

The localisation issues with mod settings should be fixed now (english only).

Sorry for taking so long, I had a break from factorio... I had other factories to grow.. :D

New response