/tmp out of memory when loading large(?) mods

[ilonachan]

I was trying to install the Girly Animation Pack mod, which has a lot of option variants. During the initial load process UKMM kept aborting claiming there's "no space on device", which initially confused me because neither the usage of my main drive nor the big hard drive I put my games and mods on was going up at all! Turns out my Linux /tmp partition was limited to 8GB, and remounting it slightly larger did allow the import to complete (though it did break large parts of my system and forced me to restart).

This is a huge problem for Linux users specifically (where I've been told /tmp is normally allocated to be half of the user's physical RAM), but potentially for anyone whose OS temp directory is on a drive with little space left. I understand that it's not really UKMM's responsibility to allocate its own temp directories, but we might want to consider it in this case: BotW is a big game, mods can get pretty big, and if they require multiple GBs of drive space to even be loaded in this cannot be reliably done in the OS temp folder. On Linux, forcing the user to remount a tmpfs (or even causing issues while doing it automatically) is an extremely bad idea for usability, and on Windows I can't even think of a solution (perhaps symlinking the temp folder might work).

One proposal to fix this in general would be to have a UKMM-specific tmp folder under its storage directory, or give the user a settings option pointing to a really large fallback drive. This doesn't even need to be the default (since having the system handle temp files where possible is probably good), but if we receive an OSError because there's not enough space then UKMM could now fall back to this path where there IS space. Even if this problem only ever comes up on Linux with low RAM specs for big mods, having a fallback like this would be broadly useful.

Another potential solution could be, for mods with many different options, to let the user select their desired options upfront and only even load those. Though this would be incompatible with the ability to select different options later, it might be useful as an emergency option for low-end devices.

It's also possible that this is indicative of a deeper issue with how (BNP) mods are processed; because both the input BNP and the output ZIP file end up being just 60MB, it must be the diffing(?) step of the conversion that balloons the temp directory, and it doesn't seem like this is cleaned up when moving on to the next option. Maybe there's a way this step could be made more efficient, eliminating the need for fallbacks and workarounds altogether?

[NiceneNerd]

This is weird, as even with the Second Wind BNP, a truly massive mod, I've never seen this issue before. Any chance you could track down exactly what folder(s) is getting so full?

[ilonachan]

I've used Filelight to check it for sure now, and my suspicion was correct: it's the TitleBG.pack, which is first inflated into a full patched version for each option, and then later each of these is probably re-deflated into UKMM's patch format. Problem is, all those inflated files accumulate and overall take up as much space as there are options. The mod I tried has 75 options, a bunch of different permutations for different feature toggles, and that results in ~100mb x 75 ~= 7.5GB usage.

I suspect this is why Second Wind doesn't have this issue, because it doesn't have options: it's just one big main mod that gets patched once, and it's gonna be pretty big, but it's pretty big once and doesn't accumulate in the same way.

And that's why I think this problem could be solved by doing the deflation right away for each option before moving on to the next one, rather than first inflating everything and accumulating drive usage, then having to fail midway through. Unless there's some requirement for one option to know something about the inflated version of the others (which seems unlikely to me) or at least the base mod (which would mean each file potentially takes up space twice, but that's already a constant and much better).

Even if this specific inefficiency is resolved, I'd still suggest having some kind of fallback just in case an even bigger mod comes around. Probably low priority though.

[NiceneNerd]

Unfortunately, there's no way without some major structural changes to fully process the options one at a time; the implementation is simply to turn the BNP into proper game source files and then run the ordinary UKMM mod packaging process, completely separate steps which know nothing about each other.

What I might be able to do―though I'll need to go back through the option processing implementation to verify if this will work―is minimally inflate packs which exist in the base of the mod, and exclude them from file-deletion detection.

For wider protection against running out of space in general, I'll have to give it further thought.

[ilonachan]

Then moving the temp directory to a larger location in an emergency would probably be easier than restructuring the entire import logic, for now... It seems like the logic is kinda modular enough that eventually the "one option at a time" model should be easily possible, but I also see that it'd take some massive refactoring. I might try my luck on it if I'm bored, but it's probably too big a task for me.