<avsm>
But first, who is here? Wave for the Irmin logging camera.
<dbuenzli>
\o/
<engil>
~o~
<thomasga>
/o/
* yomimono
waves
<rudenoise>
\o/
* djs55
waves
<avsm>
We've had quite a few releases in the last couple of weeks: Mirage 2.8.0, followed swiftly by Mirage 2.9.0 to deal with some regressions. thomasga and talex5, would you like to tell us more?
<talex5>
\o/
<dinosaure>
hi
<yallop>
hello
<mort___>
.
kayceesrk_ has joined #mirage
<thomasga>
2.9.0 doesn't fix regressions, it adds support for logging
kayceesrk_ has quit [Client Quit]
noddy has joined #mirage
amirmc has joined #mirage
<avsm>
my bad, 2.8.0 was by yomimono to fix up various tcp/ip interface issues
<talex5>
Yes. We can now start adding logs support to other libraries and users will be able to see it by default.
<thomasga>
so basically if you use Logs in your libraries, you can use the mirage tool to configure a simple log reporter
djwillia has quit [Ping timeout: 250 seconds]
<thomasga>
either at configuration time or runtime
<talex5>
I've added support to tcpip and mirage-net-xen so far.
kayceesrk_ has joined #mirage
<thomasga>
and you use simple patterns to turn on/off some log sources
<avsm>
Quite a few libraries have now started to shift to the logging interface I see. So if you see "printf" or "Console.log", then consider porting that to use Logs instead
<thomasga>
also, I've fixed the argv parsing issue
<talex5>
Confirmed: I can boot my Qubes firewall without hacks now :-)
<avsm>
thomasga: awesome!
<djs55>
@thomasga I was just wondering about that. What was the fix?
<thomasga>
I've added a way to change the default parser for command-line argument
<thomasga>
and there a new `no_argv` parser which just doesn't care
<djs55>
ah yes sounds handy
<avsm>
One quirk of the newer releases it that you may need to pass "--network 0" to mirage configure. Previously we were very permissive about how network interfaces were named, but now they need to be distinct.
<thomasga>
so it's a configuration-only parameter
<avsm>
yomimono: thomasga: talex5: is there a short paragraph about these new features we should put somewhere?
<thomasga>
not yet, I will try to write something before the next meeting
<avsm>
"how to use logging in a library", "the networking naming situation"
<avsm>
suggestions for where to put this welcome -- I was thinking expanding CHANGES would be easiest
<avsm>
to explain the import of new features
<dbuenzli>
In the documentation !
<avsm>
yes indeed...
<talex5>
How to use logging in a library is basically: read the Logs docs - nothing Mirage specific.
<avsm>
one motivation to do more documentation PRs: the more often we do so, the more often the website is redeployed and we catch bugs in the master branch :-)
<yomimono>
I think the "how to use" is less compelling than an example of how much nicer it is to deal with problems now that we have logs
<avsm>
yomimono: yeah, that's true
<djwillia>
is "--network 0" the only change to the configure step with the new release?
<avsm>
debugging no longer requires grep over console logs
<yomimono>
for example, one or two of the use cases talex5 was working on at the hackathon
<thomasga>
yes, my plan is just to write a small wiki page with an example
<avsm>
Fantastic. Closing out mirage release topic and moving on shortly...
<dbuenzli>
Just curious is someone using tags ?
<avsm>
tags?
<dbuenzli>
Logs tags
<yomimono>
I think the answer is no
<talex5>
mirage-logs will print them as key-value pairs I think.
<avsm>
should we be?
kayceesrk has quit [Quit: Page closed]
<dbuenzli>
No not necessarily
<thomasga>
not yet
<avsm>
oh I see, they can be used to attach (for example) a session tag so that we can trace from Ethif->IP->TCP->HTTP->TLS->App
insitu has joined #mirage
<dbuenzli>
Basically any kind of typed value can be attached to a logs message.
<avsm>
this seems really useful to filter out sessions... could also filter by NAT state on UDP packets to find one session in DNS djs55 yomimono
<talex5>
yominono: the examples were mainly error reporting rather than logging. We'd better get that stuff in soon, or people will start using logging instead to return errors :-(
<thomasga>
yes, lots of unexplored use-cases
kayceesrk_ is now known as kayceesrk
<yomimono>
extremely good point
<mort___>
(cool— i can see "magpie for mirage" — phoenix? — becoming an MPhil project soon :))
<hannes>
(soory, late... needed to punt)
<avsm>
Ok, next topic: good news and bad news
<avsm>
Bad news: our first security advisory! Good news: Hannes did a great job of driving and root diagnosing it and pulling in everyone needed.
<avsm>
Semi-good-news: it was actually fixed back in January
<hannes>
bad news: it was actually our second flaw (remember mirage-fs-unix 2 years ago with a directory traversal) ;)
<avsm>
So we definitely need to speed up our disclosure process somewhat
<avsm>
Aww hannes, I was zero-indexing :-)
<avsm>
This one was quite serious, but only affected the Xen backend
<avsm>
But there is precious little documentation about how the Xen device drivers should work
<avsm>
so the Mirage libraries are increasingly also becoming oracles that explain it
<avsm>
So anyone motivated to learn more about the low levels are encouraged to dive into mirage-net-xen, mirage-block-xen and friends and send doc PRs with questions/clarifications
<dbuenzli>
Is there a well defined perimeter were these can happen. So that e.g. would could focus on these for auditing.
<djs55>
they're quite hard to unit test in their current state (except vchan which is functorised and runs over any source of pages, events etc)
<hannes>
dbuenzli: unfortunately it seems those "protocols" are "documented" in the linux source code...
<avsm>
dbuenzli: it's anything down at the Xen layer. The interfaces are generally defined in code and not specs.
<talex5>
djs55: I think you had a branch doing something similar for mirage-net-xen...
<djs55>
talex5: yes that's true — it wasn't complete… however how that we have a netback it would be easier to resurrect I think. I wanted to write client/server tests which was tricky as we had no server code at the time
<dbuenzli>
I see. Do these layer then leak higher-up ?
<hannes>
(I got btw good feedback on the disclosure and the detailed analysis of how I cornered the issue)
companion_cube has joined #mirage
<avsm>
One very nice PPX extension that we can now use thanks to the the 4.02 minimum version is ppx_inline_test https://github.com/janestreet/ppx_inline_test -- figuring out how to embed more documentation that can also act as tests over a functorised version of the driver (that doesnt require xen) would be interesting
<talex5>
djs55: yes, I extracted the netback support from that branch. It'll need rebasing on trunk now.
<hannes>
dbuenzli: at least mirage-net-xen not, for TX it has its own page pool; for RX it copies the incoming data into a freshly allocated buffer... thus, the memory used for xen <-> mirage is very well contained
<avsm>
dbuenzli: ideally, we would pass zero-copy pages up through the application stack, but we don't have sufficient support for this yet at the library level. It would require a blocking interface over Bigarray fragments, so parsers would need to restart in an IO loop and so on
<avsm>
Ok, thank you again to hannes for his efforts in the last week to isolate this bug and push out the releases.
<avsm>
Onto 4.03.
<thomasga>
(thx hannes!)
<raboof>
(yes great read!)
<yomimono>
(VERY BIG thanks indeed!)
<avsm>
I have a PR that adds OCaml 4.03 support! We have fixed a lot of libraries to now work with 4.03! 4.03 has been released!
<hannes>
imho for the future: it would be nice if those low-level libraries get PRs approved by at least 4 eyes (please, let us stop to merge them individually)
<thomasga>
yay, everything broke with 4.03
<avsm>
hannes: well its not the number of eyes that concerns me, but the accuracy with which they see
<thomasga>
hannes: maybe we can start by 2 eyes
<thomasga>
I mean 2 pair of eyes
<avsm>
very few people understand those low level interfaces
kayceesrk has quit [Ping timeout: 250 seconds]
<yomimono>
...maybe we can define this in terms of brains instead
<thomasga>
indeed
<dbuenzli>
Also the problem is that diffs are not such a good review process.
<hannes>
(well, actually I'd like to establish this code review principle for mirage libraries... just let a PR sit and invite people @foo to review (with whom you feel confident they'll understand something))
<djs55>
If we refactor the code a bit we could start adding unit tests, perhaps that would catch some things
<thomasga>
we should start to add test first indeed
<avsm>
One suggestion I had for a future hacking evening would be for everyone to pick a driver and document/understand/add-tests to it
<thomasga>
I always trust better tests than eyes
<yomimono>
I think it's a good idea to do *all* that stuff
<yomimono>
tests can also be bad code
<dbuenzli>
The problem in case was a little bit hard to test...
<yomimono>
and it's very useful to have >1 brain think about them
<avsm>
dbuenzli: not really :-( we could have had a unit test that just sat in a loop granting and ungranting pages
<dbuenzli>
ok
* hannes
tries all the time to find another brain on ebay... so far not much success
<thomasga>
yes, that's a problem with the library design then. we should design testable components :p anyway, more brain/eyes/tests is good, we should do all of that.
<avsm>
i must say that oasis infuriates me with respect to test cases, since there's a flag and something else that invisibly runs only some tests
<talex5>
@hannes: I don't think we have a big problem with things becoming broken right now. Changes at the moment usually make things better.
<mort___>
agree with all that. if i understand correctly, not self-merging PRs would be a simple start though
<talex5>
We have a lot of very research-quality code we need to upgrade.
<hannes>
talex5: I'd be really careful with those "core" parts...
<avsm>
yeah, it's just that there's a lot of code written in the early days of Mirage that needs a critical look at
<djs55>
I think not self-merging PRs is good, although we do need to promise to review things quickly and not let them stagnate
<avsm>
please ignore authorship of said code while reviewing it :P
<thomasga>
yes please
<hannes>
talex5: and in order to upgrade there should maybe a design document!? (specifying things handled and not handled, lifecycles of pieces of memory, ...)!?!?
<noddy>
time for a Full Rewrite (TM)? :)
<dbuenzli>
No we'll whip
<thomasga>
(and delete as much as my code as possible)
<avsm>
hannes: in the ocamldoc signatures!
<avsm>
more good news about ocaml 4.03 : a unified documentation tool is on its way
<hannes>
I mean you can hack as much on broken code as you want, but it won't turn into high quality code unless you really think about it (sth I learned from noddy and buenzli ;)
<mort___>
@hannes: such documentation has been discussed many times but not been written. perhaps avsm suggestion of a doc/comment/query focused hackathon is the best way to start it...
<avsm>
the issue with doing it piecemeal is that there's lot of random knowledge about those layers that is buried in various developers' minds
<avsm>
so much like the ocaml compiler hacking sessions, you can get a surprising amount done in an evening over wine and code
<avsm>
i bet the device drivers are in that class -- it's quite easy to write tests
<dbuenzli>
I can drink a surprising amount of wine.
<avsm>
one area where we will *have* to do this is with the forthcoming Solo5 merge with djwillia
<hannes>
so, code reading weekends? one of the things I've planned for marrakesh2017 is to print out all the libraries in order to be able to use pen and paper..
<avsm>
for the first time we will have two driver paths for the low-levels, so abstraction will be required
<thomasga>
(yes we need more wine to improve MirageOS pliz)
<gemma>
Noted
<dbuenzli>
I encourage to print code (don't think too much about the trees) I do this for most of my libraries at a certain point.
<talex5>
Agree we should set aside time for code reviews (existing and PRs). Maybe not weekends though...
<dbuenzli>
It makes it for a very different kind of review.
<djwillia>
will need a lot of wine for the Solo5 merge
<hannes>
dbuenzli: also better readable than screens on the roof terrace
<avsm>
Ok, let's move on: I'll take an action to figure out when to have a local hacking session (perhaps in June when dbuenzli is in town) to do code review explicitly.
<dbuenzli>
Arriving the very 1st of june.
<djs55>
I also print code — I used that technique to understand domain migration in Xen while I was in a hospital waiting room
<avsm>
you picked a bad time to have a head injury djs55
<dbuenzli>
Yeah also reading code on the toilets is quite nice.
<avsm>
Onto Ocaml 4.03: is anyone blocked on a library that depends on 4.02?
<thomasga>
js_of_ocaml
<avsm>
The most notable one is js_of_ocaml, which is fixed in trunk
<hannes>
avsm: mirage-platform needs a release
<avsm>
yeah need to pin that for now
<avsm>
hannes: yes, I'm just bringing up mirage-www on it before releasing it, to get more testing. It also adds strtod/l so that float parsing works
<thomasga>
I've released new version of irmin and git which are working fine on 4.03
<avsm>
anything else? oasis released, all the Jane Street libraries are good, Irmin/Git are going great. Things are generally in decent shape.
<hannes>
(one last thing about PRs: I still don't understand where pp_hex for cstruct.t should life... or what is the point of hex if it doesn't provide it, but cstruct does it instead)
<avsm>
One thing: test with the flambda variation as well, since it is stricter with respect to .cmx files due to the inliner.
rgrinberg has joined #mirage
<dbuenzli>
If bigarrays were in the stdlib I'd put it in fmt...
<avsm>
hannes: was working on the basis that this is a specific printer for Cstruct to output in hex format, not a general hex printer
<avsm>
i.e. its more of a debug interface for cstruct
<dbuenzli>
So it should live in cstruct then.
<avsm>
yeah, and will be in cstruct.2.1.0 released today hopefully
<hannes>
in the end, I'd like to have one single place of code which converts a byte buffer into hex... and this might be functorized over all the buffers we use!?
<dbuenzli>
Functors... The hammer.
<thomasga>
the code in hex should be very fast (@yallop spent some time on it)
<hannes>
(anyways, I'l shut up now)
<avsm>
Before closing out OCaml 4.03: has anyone tried -flambda with Mirage libraries?
<thomasga>
would be better to rely on that if possible
<avsm>
it should now work, including in the xen mode, but I havent looked at the build magic
<thomasga>
not tried flamba yet
<avsm>
i'm hoping desperately that flambda is only a link time thing and not for every object file compilation
<avsm>
if it requires changing every object file, I'll be crying in the corner
<avsm>
or waiting for dbuenzli's topkg release
<avsm>
or both
<dbuenzli>
ducks
<talex5>
I tried 4.03 a few times, but had problems. And every time I do opam switch, it kills my shell configuration :-( So staying with 4.02.3 for now...
<dsheets>
Packages need to install cmx files at least and many do not right now, afaik.
<avsm>
alright, I'll take a look at that... a bug should be filed :-)
<avsm>
dsheets: ah ok. So we're making progress with 4.03 support, but 4.03-flambda should be considered as the next step (we wont get it entirely for free)
<dbuenzli>
Yes they changed the compilation model without really warning the world about it.
<dbuenzli>
A lot of my packages say they have no releases. But they do have.
<avsm>
dbuenzli: i guess this works from the github api, and not git tags?
<rudenoise>
that's right
<dbuenzli>
I see.
<djs55>
I think we need a zoomed-out view, perhaps one pixel per repo, just to cope with the number of repos we have :)
<avsm>
dbuenzli: any suggestions on how to get a changelog from a git tag? we just need a convention
<avsm>
rudenoise: actually a summary table would be really helpful, hrefing down to the detailed boxes
<dbuenzli>
Checkout the repo and extract it from the change log file.
<rudenoise>
sure, I was thinking about making them all colapsable
<rudenoise>
cool, currently using Github.Release.for_repo
<engil>
rudenoise: If you are open to pull requests about CSS I might give a try at changing a few things, if you don't have any plans already on this side :)
<avsm>
rudenoise: dbuenzli: i wonder if we can abstract this in an `opam-changes` plugin that figures it out from the package
<avsm>
some packages have CHANGES.md, others have different formats...
<rudenoise>
that'd be great, I bunged bootstrap in to get it going but I'm not particularly attached to it
<hannes>
using opam information would be great, also for tags (in order to group projects)
<talex5>
How does it work out the contributors?
dsheets has quit [Remote host closed the connection]
<rudenoise>
it loops any events and grabs the user from them
<talex5>
So... just recent contribs then?
<dbuenzli>
In topkg if your change log has a well defined format, I automatically extract the lateset release notes.
<thomasga>
no the Github API exposes all the contributors
<avsm>
alright, thanks for the continued work on this rudenoise! Onto Canopy... aside from finding serious security holes in MirageOS/Xen, how's it going engil?
<dbuenzli>
Ah no in fact it's in the man page of topkg-log sorry.
<engil>
well, hard to tell
<engil>
a few bugs that we need to fix
<engil>
and hmm
<avsm>
The one deployed on canopy.mirage.io is working well
<hannes>
thomasga: thx (I deploy on 128MB and fail on crash)
<avsm>
Infromation retrieval!
<dbuenzli>
Le serpent de mer.
<hannes>
thomasga: but those I don't really understand / cannot zoom out enough / doesn't show me its uptime
<avsm>
the OCaml Labs website has been wallowing in unmaintained mode for some time, so gemma has been centralising and updating information on ocaml.io -- a simpler mediawiki than the older site
<avsm>
This is intended just to capture OCL activity around multicore/etc, but it's proving to be quite a useful place to keep notes on stuff.
<hannes>
yay :) ocaml.io :D
<gemma>
Yes
<avsm>
So for example https://ocaml.io/w/PPX is where I noted down blog posts etc to find useful tips on moving to PPX
<gemma>
Feel free to add to/create new pages as per the PPX and HardCaml pages
<hannes>
(my only criticism is that it is not a mirageos unikernel, but some piece of php ;)
<avsm>
This wiki isn't really "announced" yet, but if you find some useful OCaml tips and need somewhere to put them, the ocamllabs wiki will continue to be around.
dsheets has quit [Remote host closed the connection]
<avsm>
hannes: yes indeed. but it's running TLStunnel for HTTPS (soon to be a unikernel), and also I've got a cohttp/fastcgi branch so that the webserver can also be a mirage unikernel soon ;-)
<thomasga>
hannes: I think there is tlstunnel between the crap php and you
<avsm>
also there is a http2https unikernel that redirects 80->443
brson has joined #mirage
dsheets has joined #mirage
<avsm>
so the unikernels are bravely defending the weak PHP centre
<mort___>
microservices ftw!
<avsm>
As dbuenzli often notes, we do want to refresh the Mirage documentation more easily, and so the next step is to port mirage-www to not crunch the data into itself but to git pull its data from a remote source
<avsm>
most of this is "almost done" (and Canopy is helping)
<avsm>
but ocaml.io is there for the OCaml-specific features. Anyone can register on the site (it's a normal wikimedia setup), so go ahead. Anonymous posts are discouraged
<avsm>
...but I haven't locked out anonymous posts yet, until abuse actually occurs. Please dont...
<avsm>
Finally: blog posts! A number of people have volunteered to do blog posts on the new features. Thank you!
<avsm>
yomimono: how's the hackathon round up post going?
<yomimono>
If anyone has last-minute contributions, that's the place to say so :)
<avsm>
There was also a lot of interest in AFL fuzzing, so if anyone wants to pick that up see what yomimono did in morocco...
<hannes>
a friend was not satisfied with the mirage tutorial stuff, and thus started to bug me and wrote things down (at https://github.com/cfcs/mirage-examples -- imho more easy to get into than some of the content of mirage.io)
<avsm>
mirage-examples is awesome!
<gemma>
That's great!
<thomasga>
hannes: that's great!
<thomasga>
folding that to main website would be awesome
<thomasga>
(replace folding by replacing or whatever is simpler)
<avsm>
I'll add a link to it shortly if my website ever builds again on the 4.03 branch
<avsm>
Any other blog posts people would like to do?
<avsm>
Also my apologies: I missed out the very exciting Solo5 release that djwillia did. How did the release go djwillia ?
<avsm>
I notice that mato leapt in to test it on dodgy old x86 hardware without the latest greatest 2MB page support :)
<djwillia>
well, we're still in the process of making everything work the right way with opam rather than the old build it had
<djwillia>
mato has been a great help
<avsm>
yes, moving the various libraries to an OPAM base is really useful
<mato>
avsm: 1G page support actually. not dodgy, just not there on mobile CPUs
<mato>
avsm: mkt segmentation by intel :(
<avsm>
mato: ah, didnt even realise x86 had 1G page support these days
<djwillia>
yeah, we used 1G pages by default in ukvm, which wasn't the most compatible choice :)