04:21 <trn> Hello - I'm getting some extremely bizarre behavior on LibreELEC on an Rpi3, it runs for about 24 hours and then locks up - happens even with no services or kodi running, just copying data over SSH/SFTP ... it takes 5-10 seconds to respond to echo back typing when this happens, before it dies. Tunning anything triggers the OOM killer but almost no processes are running.

04:22 <trn> I managed to grab the dmesg when this happened. I had a little swap on a hard disk and tried to use swapoff and it was OOM killed.

04:22 <trn> https://i.imgur.com/RNoQf5L.jpg and https://i.imgur.com/ZkeMds3.jpg

04:23 <trn> After rebooting: https://i.imgur.com/Fktq8Op.jpg

04:24 <trn> Doesn't make a lot of sense to me as only 9 processes are running. This is 9.0.1. Before I try a new Pi and SD card to rule out the hardware, any idea what might possibly do that? Seems crazy to me.

04:26 <trn> This is reproducible - copy a large amount of data via SFTP, it locks up in ~18-30 hours.

04:27 <trn> Milhouse ? maybe ^

04:27 <trn> Anyway thanks for all your hard work on LibreELEC, it's great.

04:49 <trn> Interestingly, searching, my problem seems nearly identical to https://serverfault.com/questions/645831/server-overloaded-acts-like-out-of-memory-but-thats-not-true ... I'll do something like:

04:49 <trn> while (true); do echo 1 > /proc/sys/vm/drop_caches ; echo 2 > /proc/sys/vm/drop_caches ; echo 3 > /proc/sys/vm/drop_caches ; echo 1 > /proc/sys/vm/compact_memory ; clear ; uptime ; echo ; free -m ; echo ; iostat -m ; cat /proc/buddyinfo ; sleep 5; done

04:50 <trn> for now, while it copies over the next couple days, see if it locks up again tomorrow.

04:50 <trn> Perhaps the Pi has bad memory, maybe - unlikely - been in use for years without issue.

04:53 <trn> No swap enabled now, at least this gives me a nice display ... https://i.imgur.com/82AcZcW.jpg

13:36 <Milhouse> @trn: Maybe it's this issue https://github.com/raspberrypi/linux/issues/2829 - can you reproduce with one of my test builds with kernel 5.0.3? Although LibreELEC isn't using the same kernel config settings as Raspbian so it could be a completely different issue.

13:43 <trn> Milhouse: Good morning! I'll give it a try later and let you know.

13:50 <trn> Milhouse: It's been going fine for awhile now, but I'm still running the script I pasted above. It's not easy for me to visually parse slabinfo, but so far looking at slabinfo it appears saneZ

13:51 <trn> *sane. It's hard to catch the problem because it happens only after 18-36 hrs, and the only way I know it's about to fall apart is to see the transfer speed start going down

13:52 <trn> From ~12M to 2 to 1 ... I have about 5 minutes once it starts slowing while it's still usable to get some debugging info before it's fully hung, makes debugging it hit or miss.

13:54 <trn> It doesn't actually ever die completely, sometimes when it seems hung you can wait ~30m and another 1K comes through, ends up taking >45m to echo a key back over SSH (or to even just login).

14:01 <trn> I still think it's more related to fragmentation than caching - but my drop_caches loop would mitigate both.

14:07 <trn> Also, on another Pi I'm running 4.19.30 without an issue but didn't notice the issue when I was running older versions, though.

14:14 <trn> For LibreELEC use it isn't a problem to run with all the caches essentially disabled. But, that other issue you pointed at about the ext4_inode_cache is potentially relevant as well - I am using ext4 for the local disks.

14:15 <trn> Thanks Milhouse.

18:36 <fysa> looking for testers for https://github.com/digitalsanity/libreelec-in-docker/ .. grabs/runs latest nightly libreelec front-end as a docker service. Update config to point to the build URL and run install.sh from any debian/ubuntu environment -- e.g. lets you run libreelec on an OMV box or while other host services are running and easy upgrades without reflashing

18:43 <trn> fysa cool project, saw you mention it yesterday

18:44 <fysa> also have recalbox working :)

18:44 <fysa> would be possible to make a recalbox container installable via libreelec docker add-on

18:48 <fysa> would be nice to have a couple odroid testers :)

19:23 <TheTrash> I'm fiddling around with the Chrome addon on x86 to tweak the UI settings for TV use a bit, but I'm having a hard time figuring out where to start changing stuff like the UI font, window decoration settings, etc. -- would I set it through xorg, or through gtk, or pango, or something entirely else? And what would the config file paths be, running from the addon dir?

19:25 <TheTrash> I don't entirely understand what it shares with the running xorg session and not. E.g. if it's the same session Kodi was running in, or a new one.

21:08 <MaDMaLKaV> bad news is I only got one person to test my alpha build for a5x box. good news is the guy is testing a lot of stuff I couldn't

21:08 <MaDMaLKaV> so, not bad