16:04 <busu> hm-hm. the new (as in, mirage 3.7 based) mirage-firewall builds ... stop working in a weird way.

16:04 <hannes> what is a "weird way"? busy loop? allocating memory?

16:04 <busu> oh. trying to connect a new vm gives some kind of "out of memory" error.

16:05 <busu> weird way ... symptoms intially were "why has my backup not synced out in days".

16:06 <busu> existing client interfaces keep working, just new ones run (reliably) into the OOM error (which seems to result in the interface being nonfunctional/ignored)

16:06 <hannes> in respect to out of memory we need to investigate what causes this, but i recently changed the behaviour of the mirage-net-xen interface (unreleased) https://github.com/mirage/mirage-net-xen/pull/97 to _not_ hide out of memory errors

16:07 <busu> meminfo writer reports 16MB / 40% at the same time, that is "about usual". it seems to be a little more hectic about reporting mempressure.

16:07 <hannes> which may be related, and you may get a better picture with that merged PR (opam pin add mirage-net-xen --dev ; opam pin add netchannel --dev should do the trick)

16:08 <busu> ok. will try if i get more details this way.

16:09 <hannes> I experienced a web server hiding out of memory (in the network interface layer) resulting it not sending back any replies, but still being alive -> now it crashes on out of memory, and my orchestration restarts it -- agreed that i need to find the memleaks ;)

16:14 <busu> yes, if it had crashed properly automation would have hidden the problem from me for at least a while...

16:23 <hannes> (s/i/we/)

16:41 <busu> deployed a build with the two --dev pins, lets see how that goes...

16:59 <busu> not sure what i am looking for, the error looks the same as before.

16:59 <busu> 2020-01-27 16:58:48 -00:00: WRN [client_net] Error with client {domid=4248;device_id=0}: Out of memory

17:15 <hannes> busu: can you get the console of that vm, there should be a backtrace..

17:16 <busu> that is from the console log, no backtrace at all.

17:16 <hannes> otoh, it is unlikely that the allocation site is responsible for the out of memory :/

17:19 <busu> feels more like a mem-fragmentation than actual leak issue.