14:25 <hannes> and, any luck with a firewall that does compactions every 10 seconds?

19:12 <verg> hannes: didnt try that, but the one doing a compaction every time before printing ram stats seems to prevent "long term leakage".

19:12 <verg> has been runnning 20h now, and besides two dips (68 and 64mb) it has been hovering around 72MB the whole time.

19:15 <verg> (oh, actualy the dips look like i was mssing around manually at the time)

19:20 <verg> added an update to the ticket

19:22 <verg> restarting the mfwt this client is behind, see you on the other side.

19:36 <hannes> and earlier it was ~40 MB, now 72 MB?

19:36 <mezu> it used to be 42MB total with ~ half that free. for the last years.

19:37 <mezu> now it seems every client vm takes 4MB+ mem, with GC being ... lazy.

19:37 <mezu> the 72MB is the "free" space for the single-vm-but-restarting-hourly usecase ("backup fw vm")

19:37 <hannes> mmmhhh

19:37 <hannes> so it is more 28?

19:38 <mezu> i gave the backupvm 100MB for now, and the main one (lots of clients, fewer restarts) 300MB.

19:38 <hannes> and earlier 21?

19:38 <hannes> i am mainly interested in the used memory, not the assigned, neither the free ;p

19:38 <mezu> well, "used" is a complicated topic if it reports "65MB free" and "out of memory" at the same time. :P

19:39 <hannes> i agree

19:39 <hannes> the next question would be whom do you ask for bytes free/allocated?

19:39 <mezu> i mainly cranked up the total so far to see if that makes a difference ... which supports the "fragmentation" theory.

19:40 <mezu> https://github.com/mirage/qubes-mirage-firewall/blob/master/memory_pressure.ml#L15

19:41 <mezu> so i guess OS.MM.Heap_pages.used and OS.MM.Heap_pages.total

19:41 <mezu> (my Gc.compact() call is in line 13 there)

19:41 <hannes> yes, ok -- and that's implemented https://github.com/mirage/mirage-xen/blob/master/lib/mM.ml as calls to c functions in https://github.com/mirage/mirage-xen/blob/master/bindings/mm_stubs.c

19:42 <hannes> so it is what mini-os thinks about memory, which is fine (good); there's as well the OCaml gc whom you could ask (will report less used memory since the io-page/cstruct is not part of the ocaml gc-managed heap)

19:43 <mezu> hm. which one is Gc.compact() hitting? or both?

19:43 <hannes> tl;dr: the Gc.compact calls help you and we should figure out how to trigger this implicitly :)

19:44 <hannes> Gc.compact is mainly concerned about the gc heap, but it as well takes care to free those non-gc-managed pieces by running finalisers..

19:44 <mezu> yes, that would be one of the two issues i see currently. "call Gc.compact more frequently than daily". :P

19:44 <mezu> (the other is the "4mb per client".)

19:45 <hannes> was this "4mb per client" there before? (or did you not look, ...)?

19:46 <mezu> i never really looked, but i would have hit a "4mb per client" limit _very_ hard a lot of times.

19:46 <hannes> ok

19:46 <mezu> my sys-mfwt has 17 running downstream vms right now, that is an "average" situation for it.

19:47 <mezu> (and it used to run on 42MB and i would have to do log digging to say how much used to be reported as free)

19:47 <hannes> i checked for tooling in respect to the new statmemprof -- unfortunately the tooling is not yet there (see https://github.com/ocaml/ocaml/pull/9230#issuecomment-580730423)

19:48 <mezu> you mentioned Gc triggers were removed in some place(s)?

19:48 <hannes> 17 * 4 > 42 in any case, so there seems to be a memory regression

19:48 <hannes> i mentioned that the call to Gc.compact (triggered by Io_page.get) are happening less often

19:51 <mezu> another unverfied impression: it seems to "leak harder" after memory got actualy used, like "more likely to not free up memory if the client vm shutting down did a bunch of serious traffic"

19:51 <mezu> (which again fits with fragmentation)

20:11 <hannes> and to add to the pile of things to investigate is https://github.com/ocaml/ocaml/pull/8809 -- a best-fit allocator which should help against fragmentation. NB this code is only in OCaml 4.10.0+ (which is not released, but in beta2)