<kuya>
boots and basic functionality works, cant say anything about mem-behavior yet.
<kuya>
deployed in backup ("one client, hourly restarts") role.
Haudegen has joined #mirage
<hannes>
good, thanks
<kuya>
will report in some hours how that worked
<kuya>
this doesnt have forced-gc, and a pre-mirage-3.7 memory limit of 42mb
<hannes>
..so the comparison is slightly unfair ;)
<kuya>
hm?
<hannes>
well, the forced-gc is something which disappeared i suspect between mirage 3.5 and 3.7 ;)
<hannes>
but let's see what your memory stats say
<kuya>
oh, by forced-gc i meant my compact() call before every mem stats print. :)
<hannes>
it'd be good if the "Gc.compact ()" is not needed
<kuya>
very much so, thats why i didnt add it back just now.
<kuya>
some "dry" starts with rsyncs that dont do much anything besides comparing trees work, there will be new backups to sync in 10 minutes.
<kuya>
(with the LRU.M and no-forced-gc, pushing some serious bytes through the FW seemed to make the problem much worse or more visible. which makes sense.)
adhux0x0f0x3f has quit [Remote host closed the connection]
adhux0x0f0x3f has joined #mirage
<kuya>
its not printing any mem stats, even under load. (thats a good sign, it prints only when it is thinking hard about memory...)
<hannes>
hmm, can you look in dom0 about the allocated (not reserved) memory for that VM?
<kuya>
doesnt work that way, sadly.
<hannes>
true...
<kuya>
not using dynamic vm mem allocation
<hannes>
ah
<hannes>
out of curiousity, is there any monitoring service shipped and used with qubes? i'm thinking of influx/grafana or similar (mrtg? munin?)
<kuya>
not shipped with, i used grafana in the past. (installed in a vm, feeding it data through qubes-rpc / xen-vchan pipes)
<hannes>
i use for my server-side unikernels a separate network interface where a syslog sink and an influxd are running to get metrics every 10s
<kuya>
that looks much better. 12-14MB in use is what i would expect there.
<hannes>
and https://berlin.ccc.de/~hannes/gc-compaction.png is showing what a GC compact does (roughly in the middle ~6:15) -- from 222 to 64MB -- the compaction was only done after it ran for ~1 month
<hannes>
\o/
<kuya>
before todays patch. older paste because i wanted one with same total mem:
<kuya>
not sure why it shows 1-2MB more free now. :D
<hannes>
i do ;p
<hannes>
before today, the nat table (i.e. source ip, dst ip, source port, dst port) -- as well a LRU -- used Lru.M.t, now it uses Lru.F.t as well (again) ;)
<kuya>
so +1 from me for that triad of patches/PRs, it certainly works better than the builds from the last two weeks.
<hannes>
kuya: if you could comment on your qubes-firewall issue about memory pressure your experimental evaluation, that'd be very nice
<kuya>
ok.
<kuya>
(ah. the 1-2MB seem to be something _very_ basic because they are already present in that one mem-stats print right after startup before anything load related happened.)
<hannes>
the 1-2MB, if it is the nat table, would be very basic -- allocated at startup
<hannes>
so thanks for your reports, and sorry for the bad intermediate/current state -- glad it's improving and we managed to spot the culprit
<kuya>
no worries about breakage. you know how long i have been in the open sores circus. this doesnt disturb me much at all. :)
<kuya>
thanks for investigating + fixing and for the whole mirage madness in general. :)
<hannes>
you're welcome
Haudegen has quit [Ping timeout: 260 seconds]
Haudegen has joined #mirage
jnavila has joined #mirage
<kuya>
seems the memory is "fully fragmented" now, its printing a lot of mem-stats under load, still reporting 25.3 free (thats about 500k less than earlier)
<hannes>
..which means a Gc.compact before printing the mem stats would be suitable -- or indeed due to reading of http://caml.inria.fr/pub/docs/manual-ocaml/libref/Gc.html#TYPEcontrol (a) set allocation_policy to 1 (next-fit), (b) lower max_overhead (so that compactions are triggered more often)
<hannes>
it looks compaction is by default only done if the estimated wasted space is 500% more than the live gc heap
mahmudov has quit [Ping timeout: 240 seconds]
mahmudov has joined #mirage
Haudegen has quit [Quit: Bin weg.]
<hannes>
but afaict for such gc tuning it'd be interesting to graph the different gc counters
<kuya>
well, it is working, so tbh i dont care too much if the memory-management code is hyperventilating ...
<hannes>
;)
<kuya>
i will just keep it running for now, perhaps it will break after some more rounds.
<kuya>
but yes, i really need to metric up on this kind of thing. i cant tell how the cpu usage developed over the last three weeks. :(
mato has quit [Quit: WeeChat 2.6]
Haudegen has joined #mirage
<reynir>
hannes: Hey I noticed I missed the registration deadline for the retreat - is it too late?
<hannes>
reynir: you're already signed up
<reynir>
Ah right, thanks
jnavila has quit [Remote host closed the connection]