kentonv changed the topic of #sandstorm to: Welcome to #sandstorm: home of all things sandstorm.io. Say hi! | Have a question but no one is here? Try asking in the discussion group: https://groups.google.com/group/sandstorm-dev
TimMc has quit [Quit: leaving]
_whitelogger has joined #sandstorm
xet7 has quit [Remote host closed the connection]
xet7 has joined #sandstorm
<ill_logic>
Ian Denhardt: Just got a Sandstorm update. 500 when I try to do a backup.
<ill_logic>
Gonna look at logs in a second.
<JacobWeisz[m]>
Same on Alpha.
<JacobWeisz[m]>
@ill_logic I kicked open an issue on GitHub with what was in my Sandstorm log.
<ill_logic>
What triggers releases?
<JacobWeisz[m]>
What do you mean triggers?
<JacobWeisz[m]>
The Sandstorm instal script has historically been configured to fail if the latest release is more than 30 days old, presumably to avoid downgrade attacks.
<JacobWeisz[m]>
install*
<JacobWeisz[m]>
So roughly every fourth weekend, Kenton does a release to avoid getting a bunch of random "the installer doesn't work" reports.
<ill_logic>
Okay, so in the sense I meant my question, the answer is Kenton
<JacobWeisz[m]>
Yes
<JacobWeisz[m]>
In the scenario we have now where a release introduced a major regression, Kenton can push a release sooner, with the best case scenario being that isd can come up with a good fix before the weekend is over, and the worst case being Kenton reverts that one PR for now.
<JacobWeisz[m]>
Presumably you, I, and Alpha all run some case which Ian didn't test for. I think my server is Ubuntu 18.04 or something similar.
TimMc has joined #sandstorm
xet7 has quit [Quit: Leaving]
<isd>
Can you guys ssh into your boxes and poke around in /sys/fs/cgroup for me?
<isd>
you know what, I can just spin up a VM
<JacobWeisz[m]>
What would I be looking for?
<isd>
don't worry about it, figured it out.
<JacobWeisz[m]>
Oh. freezer
<isd>
This means we probably can't rely on even newer systems necessarily having this functionality. So we may need to think about backup strategies to handle that case.
<JacobWeisz[m]>
?
<isd>
I really don't like the idea of just giving up on atomicity and going ahead; if we're doing backups frequently that seems like it's going to bite us.
<isd>
I hate computers.
<JacobWeisz[m]>
Is freezer and freeze different things then, I presume?
<isd>
freezer?
<isd>
Don't know off-hand what that directory is
<isd>
sys/fs/cgroup/unified is the root of the cgroups2 hierarchy; each directory in there has a bunch of cgroup.* files
<isd>
Apparently cgroup.freeze doesn't exist on your system.
<isd>
Anyway, I have a patch. Making sure it doesn't break my system, and then I'll send a pr and ask you to verify that it fixes yours...
<isd>
pr created
<kentonv>
so some kernels just don't have this functionality? But why? It works on my local machine but not on Alpha? But they are both running Debian Buster...
<isd>
kentonv: seems to be a kernel config thing
<isd>
There's an option CONFIG_CGROUP_FREEZER, which I was assuming controlled that, though I haven't checked beyond "oh, that sounds relevant"
<isd>
Annoyingly /proc/config.gz doesn't seem to exist on ubuntu 18.04...
<isd>
Maybe buster has this compiled as a module, and it's not loading for some reason on alpha?
<kentonv>
weirdly, /boot/config-$(uname -r) on Alpha shows CONFIG_CGROUP_FREEZER=y
<kentonv>
yet it isn't working there?
<isd>
nvm, looks like it's not something you can compile as a module
<isd>
But ocdtrekkie says he has a /sys/fs/cgroup/freezer, so maybe that's not actually the right option.
<kentonv>
Alpha has that too
<kentonv>
On my local machine there is also CONFIG_SUSPEND_FREEZER=y, which doesn't appear on Alpha
<kentonv>
that seems unrelated, though
* isd
goes to look at kernel soruce
<JacobWeisz[m]>
Ever spend too much time trying to look up how to do something, before realizing you did it already, and then wonder why you didn't document it, and then check your documentation and realize that you did document it?
xet7 has joined #sandstorm
<JacobWeisz[m]>
Did the research reveal anything compelling other than skipping out on this for servers that don't have it?
<JacobWeisz[m]>
There's not any way to easily add it into a server, is there?
<isd>
Not as far as I can tell.
<isd>
I'm still not sure why it's not there; perusing the kernel source it doesn't look like it's configurable.
<isd>
...maybe ubutnu patched it out or something? I can't imagine why they'd do that though.
xet7 has quit [Quit: Leaving]
xet7 has joined #sandstorm
<JacobWeisz[m]>
I saw tasks inside /sys/fs/cgroup/freezer, and freezer-subsystem.txt says something about echoing a PID into freezer tasks to freeze it... Is it possible that the system in question is there, but that you need to use a different command to do it on Ubuntu?
<JacobWeisz[m]>
I'm reading cgroup.freeze was added for cgroups2 in the Linux 5.2 kernel. I'm on Ubuntu 18.04.3, which is 5.0 based, I believe.