19:41 <aion> this is sooooo a fragmentation issue ... now reporting "out of memory" and "65MB+ free" at the same time.

19:41 <aion> (well, fragmentation or alignment needs...)

19:49 <tg> oi, any mirage related things at fosdem this time?

22:47 <hannes> kuya: i read you... i am investigating a similar oom issue in another application (has ~1GB, once 380MB allocated is passed, the probability is very high an out_of_memory is raised)

22:47 <kuya> hannes: filed an issue + pinged you by now.

22:48 <hannes> now, I already added (as workarounds / see whether it changes anything) cyclic calls to the garbage collector to run a full major collection and compaction...

22:48 <kuya> hannes: can not tell about any specific limits. this soooo smells like fragmentation/alignment...

22:48 <hannes> yes

22:49 <hannes> i can think of two ways forward: attempt to just get rid of any bigarray allocations, or b rebase and restart the statistical memory profiler

22:49 <kuya> i can run whatever tests needed. (or spin up the guest vms again)

22:49 <hannes> more info at https://jhjourdan.mketjh.fr/pdf/jourdan2016statistically.pdf -- this line of work has now been merged into ocaml-master (there are branches for 4.07 etc. around)

22:50 <hannes> i got it to work with mirage (by exporting the stats on request via tcp)

22:50 <kuya> i was hoping the "4mb" part might be hint though, because even without any leak/frag issues, 4MB per client VM would be _some_ change.

22:50 <hannes> likely will need to adapt some code (+compiler code) to get that up and running, not sure whether i'll make it before marrakesh

22:51 <hannes> i don't know of any hardcoded "give me 4MB" in the dependency cone, sorry :/

22:51 <kuya> my guess would be "default vchan buf size"

22:52 <kuya> (because thats the only thing in the situation where i see the oom that probably cares about alignment)

22:52 <hannes> yes, maybe.. i don't know too much about the xen backend, for me its an issue on solo5-hvt / kvm / freebsd bhyve..

22:53 <kuya> oh. so not backend specific?

22:53 <hannes> i sometimes get some reasonable backtraces from the out of memory exception (usually in sexp_conv :/)

22:53 <hannes> yes, well, not entirely sure we have the same issue

22:54 <kuya> not seen any stacktraces, the oom always happens in the same place, and the "sometimes i recovers some memory after some hours" part was rather unexpected.

22:54 <hannes> but there's this code around https://github.com/mirage/io-page/blob/master/lib/io_page.ml#L36-L45 -- which tries to allocate n pages, and on failure does a gc compaction

22:54 <kuya> where can i find your "GC on steroids" version?

22:55 <hannes> https://github.com/hannesm/statmemprof-mirage

22:56 <hannes> meh, i should rebase/update that stuff (esp now that in 4.11 we'll get the statmemprof into ocaml-baseline)

22:56 <hannes> i don't know whether it still compiles and works (it is likely if it compiles as documented that it'll work)

22:57 <hannes> the UI is an emacs "GUI" (where i already forgot what's in the user interface)

22:58 <kuya> *frowns* ... considering to just add some forced GC.compact calls to the main loop...

22:59 <hannes> the code I linked to above, Io_page.get -- is something getting less call sites in newer mirage versions (for various reasons, usually not needed); this also means less gc compactions (which may or may not be related..)

22:59 <hannes> there were also changes in the ocaml runtime system (when a collection is triggered), but i don't remember which version that was introduced..

23:00 <hannes> but now i just drink beer and think i'll do something else with my life ;p

23:00 <hannes> you could try a Gc.compact every 10s to see if it solves anything for you

23:00 <kuya> how would i do a "every 10s" kind of thing? (as in, whats the keyword to google for?)

23:02 <hannes> Lwt.async (fun () -> let rec loop () = Gc.compact () ; Time.sleep_ns (Duration.of_sec 10) >>= fun () -> loop () in loop ());

23:03 <kuya> hm. i could just compact from the mempressure reporting code ...

23:03 <hannes> sure, that as well

23:03 <hannes> that's likely easier :)

23:04 <hannes> Otherwise you've to hunt for a Time implementation (or use OS.Time.sleep_ns)

23:10 <kuya> added in front of mem reporting, built+deployed, lets see how that goes.

23:12 <hannes> cool

23:16 <hannes> in your report, would you mind to specify which OCaml version you are using?

23:22 <kuya> added note on versions to the issue post. basicly 4.08.1 with netchannel and mirage-net-xen pinned to --dev