<thomasga>
avsm: can you change the topic of the chan?
<avsm>
thomasga: nope, not authenticated here on this client
<avsm>
Who's here? Hands up
<thomasga>
\o/
<engil>
o/
mcclurmc has quit [Ping timeout: 264 seconds]
<talex5>
\o
<djwillia>
o/
<reynir>
oh!
<avsm>
wiredsister: wake upppp
<avsm>
Alright, first up we have quality and test
<dinosaure>
λ
<seangrove>
o/
<hannes>
hello
<amirmc>
yo
<avsm>
The Docker containers up at `docker pull ocaml/opam` have worked out pretty well. I started a thread on opam-devel about removing the Git opam-repository from there, but it looks like they are already quite heavily used, so I'm leaving things the way they are.
Algebr` has joined #mirage
mort___ has joined #mirage
<mato>
\o/
<avsm>
A number of people have started using them for their OCaml CI efforts, hence the difficulty in changing the ones that are there. The changelog for those will include Alpine 3.4 (released yesterday) and OPAM2 tracking so we can test it out when the first alpha is released by Algebr`
ricarkol has joined #mirage
<avsm>
thomasga: do you want to say anything about DataKit? Just open sourced recently at https://github.com/docker/datakit and should be useful for Mirage CI
<hannes>
s/Algebr`/AltGr/
<avsm>
hannes: woops, tab complete fail
<Algebr`>
lol, I was like what
<yallop>
./
<thomasga>
datakit will be used to track version-controlled data-flow, based on PRs updates
<Drup>
o/
<thomasga>
not totally ready yet, but first prototype should be ready shortly
<thomasga>
I'm keen to beta-test on MirageOS infra when available :-)
<avsm>
It has GitHub integration which is very handy -- so you can create the "green/yellow/red" tests on the fly for each PR
<thomasga>
don't spoil
<avsm>
okok, If anyone wants to try out some 9P filesystem magic, get in touch with thomasga then :-)
<thomasga>
:p
<mort___>
.
<avsm>
In order to keep track of the releases more easily, I've also created a This Week in OPAM alpha repo
<avsm>
It uses the opamfu library from opam2web, so predicates can be specified to filter on tags for project-specific reports (not tried this yet).
<seangrove>
Very nice
<avsm>
The ultimate idea is to incorporate structured changelogs into it as well.
<avsm>
Working on updating http://github.com/dsheets/ocaml-changes with an `opam-changes` plugin, so that we can have a CLI for maintaining ChangeLogs in our repositories more easily.
<thomasga>
avsm: do you have an example of a report?
<avsm>
it's super basic right now, but with ocaml-changes we can inline in changes for packages that are compliant. The out/ directory includes the raw CHANGES file for convenience (not linked in the gist above)
<thomasga>
would be great if we can easily link to blog posts or other media
<GemmaG>
Looks great :)
<hannes>
this is nice... any plans to integrate into dashboard?
<avsm>
yeah its easy to customise, just a single file. I got slightly distracted porting ocaml-changes to use carcass/topkg, hence the delay :-)
<Drup>
I guess the goal is to plug it in canopy and have regular posts that way
<avsm>
will mail the list when done then.
<Drup>
?
<avsm>
Yeah, commit to Canopy via DataKit, but also do manual updates
<avsm>
GemmaG has offered to add it to the OCaml Weekly News if we help her with editing information on the packages
<GemmaG>
Yes!
<avsm>
If anyone has thoughts on how to automatically get some interesting anecdotes automatically, that'd be useful. For instance, opamfu might be able to extract the GitHub PR that it came from, or we could tag something in the commit message
<avsm>
it'll take a while to port packages to use the structured CHANGES format(s), but might as well start sometime...
<avsm>
thanks for helping out with this GemmaG, and dsheets for starting off ocaml-changes and opamfu!
<avsm>
Anything else on quality and test?
<avsm>
Alright! Moving onto Mirage 3 planning
<thomasga>
if anyone has issues compiling on windows, fdopen is very reactive to help
<thomasga>
(about quality)
<avsm>
ah yes -- before we move on. Windows support is important
<avsm>
djs55 (currently on vacation) and yomimono have also made significant inroads into several core Mirage packages, and djs55 commented before he left that a native `mirage` on Windows is now quite easy
<thomasga>
most of the MirageOS libraries work on Windows now thanks to fdopen
<avsm>
we'll soon hopefully have CI for this thanks to DataKit, but this is a big milestone. There appear to be several OPAM Windows remotes but we're using fdopen's as he's been so helpful (especially with ctypes)
<avsm>
If anyone has strong opinions about merging those repositories, now's a good time to volunteer :-)
<avsm>
Ok, moving onto Mirage 3 APIs...
<avsm>
There are quite a few changes queued up, and this will involve a lot of package motion in the database.
<avsm>
So rather than go into the details on this IRC meeting, I want to ensure that we have the basic groundwork for package constraints sorted before we make the move.
<avsm>
hannes notes (quite rightly) that module names should not include versioning information, so this means that the responsibility falls to OPAM
<avsm>
So the first thing to do is probably have a mirage minor release that places upper bounds on packages during the OPAM install, and then work on breaking up the mirage-types into the new revisions.
<avsm>
We'll probably create an issue and work over there -- release manager TBD.
<avsm>
But this is also a good time to put a stake in the ground with any interface issues that are particularly annoying you.
<hannes>
(my argument holds until someone wants to use a V2 and V4 network device in a single unikernel... imho too complex and unneeded)
<thomasga>
I think the right path is still a bit unclear
<avsm>
hannes: it does have precedent -- for instance, Xen device drivers can negotiate different versions for offload purposes.
<thomasga>
I remember when I had to update all the FS implementation at the same time. It was painful.
<avsm>
Cstruct 2 was fun as well, had to edit hundreds of packages
<avsm>
I think that we're at the point when mechanically rewriting the OPAM repository to add bounds is the only practical way
<thomasga>
I dislike putting version in module names, but we need an easy path to evolve the interfaces.
<hannes>
thomasga: but you can just conflict all of them with <X.Y, and don't need any actual code changes...
<talex5>
(note: I think modules from a package shouldn't have the package's version in their name. There's no reason why they shouldn't include some other version if it's relevant)
<thomasga>
in that case you want have a `mirage configure` working
<avsm>
we need to think about breaking up every single module type into its own OPAM package so that we can do fine grained versioning on them, and only depend on the ones that a unikernel needs.
<avsm>
talex5: yes good point, that's the case currently as well
<thomasga>
avsm hannes: version + conflict won't solve the problem. The mirage tool wants to use all the implementation of the interfaces he knows about.
<thomasga>
if you don't upgrade then in lockstep, you can't use the mirage tool.
<mort___>
it seems that interfaces evolve relatively slowly in any case. if there are situations where it's particularly useful to have concurrent different interfaces for a type, then presumably modules with explicit versions in names can be created at that point. but it should be the default (imo)
<mort___>
*shouldn't
<avsm>
thomasga: yeah, I see what you mean. We need to be very explicit about dependencies
<thomasga>
out of curiosity: does anyone already tried to change an interface in mirage-types?
<talex5>
mort: The situation where it's useful is when you're migrating to the new version.
<thomasga>
(I did, it was painful)
<avsm>
thomasga: yeah I did and even in the early days it was very laborious
<avsm>
and we still havent eradicated the fairly broken RANDOM
<avsm>
but at least our tooling is vastly better now
<talex5>
I'm in the process (multi-year) of changing some.
<avsm>
we're just not taking advantage of the tooling :-)
<thomasga>
the problem is not being explicit about dependencies, it still the same issue about the mirage tool needing a global consistent state of the world
<avsm>
but it has this through the repository and the local package pin, right?
<avsm>
if we generated an `opam` file for the unikernel
<thomasga>
so it needs to know which implementations are available for a given interface
<thomasga>
e.g. you want to have some meta-information about dependencies in the tool
<avsm>
yeah so we want to say "this package can be applied against the functor in this other package"
<thomasga>
yes
<avsm>
which actually comes up a LOT with cohttp upgrades
dsheets has quit [Remote host closed the connection]
<avsm>
every release now, there are a few packages that break with a signature tweak, but not all of them
<avsm>
rgrinberg and I see this quite a bit
<thomasga>
it's just a hard problem I think. I welcome simpler solution, so maybe `<INTERFACE-NAME>.V<X>` is not a bad approximation after all.
<thomasga>
even if it's ugly
<avsm>
ok, let's take this one to an issue -- it could be that `<INTERFACE-NAME>.V<X>` but with each one split into a separate OPAM package so it's easier to rev independently is the way to go
<Algebr`>
that's pretty ugly.
<wiredsister>
avsm: Hi!
<avsm>
wiredsister: greetings!
<hannes>
I still don't understand why you are eager to introduce version numbers into mirage... yes, updating interfaces is complex. maybe we should document what needs to be done? but discovering in a few months that we need a sat solver in mirage does not sound like a good move to me
<thomasga>
hannes: I am not eager at all.
<avsm>
hannes: the versions aren't really used in any form other than structurally
<thomasga>
I just don't know how to do that properly with that yet.
<avsm>
and I do agree that anything outside of OPAM (i.e. an external sat solver) would be a terrible UI and unlikely to pass muster
dsheets has joined #mirage
<avsm>
Ok, lets move on... lots to cover. Will resume this on the mailing list.
<thomasga>
ok, I can try to describe why what you are proposing will be a pain
<talex5>
I think you can get a long way with "pick the highest version both modules support".
<avsm>
Onto Solo5, which djwillia and mato have been making lots of progress with integrating as the first non-Xen unikernel backend!
<avsm>
Care to describe progress djwillia mato?
<mato>
Sure.
<mato>
What I've been doing over the past couple weeks is essentially cleaning up the mirage-solo5 package (i.e. the "platform bindings") and it's dependencies.
<mato>
('sec, phone call)
<djwillia>
we were looking at the package structure and trying to simplify out some of the -xen related packages
rudenoise has left #mirage [#mirage]
<djwillia>
like mirage-posix-xen and mirage-ocaml-xen, and especially minios-xen
<avsm>
This is most useful -- it forces us to be more portable wrt Xen C library compilation
<djwillia>
(which were all being required in some way from the mirage-solo5 because of quick and dirty hacks)
<avsm>
djwillia: what are the next steps? would you like local testing from anyone for an OPAM remote?
Algebr` has quit [Ping timeout: 250 seconds]
<talex5>
Yes. We build "xen" binaries of many libraries but they're really just "kernel-mode" binaries.
<djwillia>
now, mato has extracted the ocaml with the posix pieces it needs and that's called "ocaml-freestanding"
<avsm>
talex5: or more generally, just a variant of the same library built with different CFLAGS/LDFLAGS
<djwillia>
the idea is that we build out some of these "freestanding" packages that could be then linked to whatever backend we like
<talex5>
avsm: Yes, true.
<avsm>
djwillia: but those backends need to share CFLAGS/
<avsm>
?
<mato>
(back)
<djwillia>
in the process, mato also removed all POSIX or libc looking functions from solo5, so that it's clear that solo5 isn't pretending to be posix
nullcat has joined #mirage
<avsm>
djwillia: excellent!
<djwillia>
each backend can put its own CFLAGS which is being fetched by PKGCONFIG i believe
<avsm>
djwillia: makes sense, this is looking more and more like BSD ports every day :-)
<avsm>
(except for pkgconfig)
AltGr has left #mirage [#mirage]
<mato>
i would really like to get rid of pkgconfig and replace it opam with per-package variables, need to sync up on what's happening there
<avsm>
AltGr just left, but OPAM2 has good support for this
<mato>
yes, i know. so will need to keep track of that.
<avsm>
There's an active thread on opam-devel about sorting out container support (and opam-repository migration scripts), so i'll mail when its ready
<thomasga>
opam1 support per-package variables already :-)
<avsm>
am keen to start using opam2 day-to-day as well to dig out bugs
<thomasga>
(kind of)
<avsm>
thomasga: in 1.2.2 only right?
<talex5>
So what happens with libraries like cstruct? Do we get cstruct.xen and cstruct.kvm now?
<thomasga>
1.* I think
<mato>
thomasga: kind of, but no interfaces to set them unless you do it yourself by sedding the .config
<thomasga>
yes, if your package generate a .config file that's fine
<thomasga>
but yes, no option to set it, which is fixed in opam2
<avsm>
it's still not clear to me who builds cstruct.kvm -- the cstruct package, or the solo5 package?
<mato>
right now most of the C stubs are shoved in mirage-solo5 for expediency
<avsm>
ideally the solo5 package would depend on the cstruct package and generate it, but this is probably complicated by some ocamlfind nonsense
<mato>
my idea is to eventually have -freestanding (or "kernel mode" as talex5 calls it) versions of packages
<mato>
then (final step) would be to migrate -xen versions of packages and mirage-xen on top of ocaml-freestanding
<avsm>
Yeah, that would be nice if the kernel mode packages can agree on CFLAGS and an ABI
<avsm>
like no red zone on x86_64
mort___ has quit [Ping timeout: 276 seconds]
<djwillia>
in general I'm not clear yet on how far up the stack is affected by changing backends (e.g., how far up kernel versions of packages go)
<mato>
obviously Xen-*specific* packages (i.e. those using mini-os/Xen interfaces directly) are a different story (and would still need to depend on / get CFLAGS from mini-os directly)
<avsm>
right, this is the good thing about the solo5 upstreaming -- it forces us to be very clear on where to cut off Xen-specific dependencies
<mato>
indeed, that's my aim with this work. i'd also like to document this (as a blog post) on "porting mirage to a new platform"
<avsm>
mato: yeah but any deviations have to be looked at very carefully since there are still raw function calls going between ABIs!
mort___ has joined #mirage
<avsm>
mato: +1 to the post!
<mato>
avsm: still some way away.
<avsm>
alright, thanks for the update. Lets move onto the next topic
<mato>
djwillia: i think we should co-author the post, if you're happy with that?
<avsm>
Flambda testing!
<djwillia>
mato: sure, although i'm a novice compared to you mato :)
<thomasga>
but that memory profiler looks very nice, I'd definitely will try it on more examples
<avsm>
thomasga: quick mail to the list with your notes? I'd like to try it too
<avsm>
or is it all documented on the PR somewhere?
<thomasga>
(it can profile the C heap too)
<thomasga>
not really
jermar has quit [Ping timeout: 244 seconds]
<thomasga>
I can try to document the steps
<avsm>
that would be really useful
<thomasga>
ok
<avsm>
I can containerise the switch to make it more convenient if you want it in the regular 4.03 variant tags
<avsm>
Ok, onto the next topic! Outreachy!
<wiredsister>
If anyone ever needs documentation help with anything, ping me.
<wiredsister>
I am a happy lackey.
<avsm>
wiredsister: perfect timing!
<avsm>
how goes your lacke^H^H^HOutreachy experience so far?
<wiredsister>
haha, so far pretty good. Per mentors' advice, I am formulating roadmap of what things need to be done with syslog and turning that into a blog post.
<avsm>
fantastic. mort___ and i were chatting about some devices that would be helpful to run solo5 unikernels in production, and a syslog terminator came up pretty high...
<djwillia>
what's a "syslog terminator"?
<wiredsister>
Great. My understanding at the moment from the great people on #syslog is that we need to make it into "a real boy" and target Xen.
<avsm>
djwillia: something that listens in the host for unikernel logging events and stashes them somewhere
<djwillia>
avsm: thanks, not as scary as it sounded :)
<avsm>
wiredsister: not even clear to me yet which of the syslog protocol variants are appropriate (reliable syslog?)
<avsm>
djwillia: syslog does need termination, but not yet ;-)
<hannes>
avsm: depends on your usecase and setup
<mort___>
one alternative would be simplest variant first :)
<mato>
avsm: is there a plan to have mirage log to syslog via UDP?
<avsm>
wiredsister: likewise, please do be ruthless about filing doc issues on libraries you are running into that are poorly linked
<wiredsister>
I will be noisey.
<avsm>
mato: that should work today, but UDP is going to be messy. I was wondering about the TCP variants but havent looked
<mato>
messy, why?
<avsm>
Is Theo here?
<avsm>
mato: losing messages due to full buffers will be frustrating. need to add a realibility layer on top.
<wiredsister>
avsm: Can you explain how "terminator" would be different than line 18 of this PR?
<avsm>
wiredsister: you are logging to Irmin? Epic
<wiredsister>
Yes.
<avsm>
djwillia: there you go. Syslog terminated by Git :-P
<mato>
cool
<avsm>
Alright, we have five minutes left, so just wanted to wrap up with the last item in the agenda (skipping library updates)
<djwillia>
:)
<avsm>
MirageOS hackathon! Mid June! Cambridge!
<avsm>
Dates not been finalised at all. Is this being organised by GemmaG?
<thomasga>
can we have some sun?
<wiredsister>
Oh man, what are the dates?
<avsm>
If someone really wants to come, now's the time to shout out date constraints
<thomasga>
(and tajine?)
<seangrove>
Ah, bummer, didn't realize that
<avsm>
I suspect that July will be more viable if people want to visit
<GemmaG>
hahaha was originally just looking at Cambridge, but can look elsewhere if there is consensus....
<GemmaG>
Yes - not sure of Mindy's UK dates at present
<seangrove>
July is certainly easier for me...
<wiredsister>
Yeah, I vote for July
<avsm>
Let's have one in Cambridge please... lots of people wanted to go to Marrakech but couldn't
<GemmaG>
There *might* be more sun in July also
<djwillia>
will there be another one in the fall?
<avsm>
and I solemnly promise thomasga there will be sun in Cambridge in July *crosses fingers*
<wiredsister>
There is sun in Cambridge? Who knew.
<GemmaG>
eeeeks not atm
<GemmaG>
ok cool - looking at venues
<GemmaG>
Any preferences?
<avsm>
I'd suggest taking this to the list to ensure that everyone who wants to go hears about it (not just this meeting) and narrow down the date range
<mato>
djwillia: Mirage hackathon sounds like a plausible excuse for you to come visit Cambridge :)
<GemmaG>
Looking at colleges currently as easiest with the links we already have, but open to other places
<GemmaG>
Yes list is good :)
<GemmaG>
Let's narrow dates and places
<djwillia>
mato: yes, i really want to visit
<avsm>
djwillia: there can certainly be one in the fall as well, but lets find a date you can make it in July!
<djwillia>
ok, will look into july dates
<GemmaG>
:D
<mato>
First half of July works for me
<avsm>
AOB: in the closing minutes, anyone got any cool hacks they've done? Libraries that are works in progress?
<avsm>
I'm trying out dinosaure's awesome email MIME parser on 3 million of my past emails atm
<wiredsister>
Request: I need help understanding the steps required to make a project target Xen. If someone feels like helping me, please do.
<seangrove>
I'm working on Jitsu to get a drag-and-drop http-based demo working
<seangrove>
Drag a unikernel artifact onto a page, get a working instance in a few seconds
<mort___>
wiredsister: i guess you mean more than simply `mirage config —xen` ?
<wiredsister>
Yes.
<hannes>
I just slacked for a week to make 15kg espresso on the countryside
<mort___>
…because that plus the OPAM magic is supposed to just work, right? :)
<talex5>
wiredsister: what kind of things?
<wiredsister>
right, it doesn't for syslog lol
<wiredsister>
well, let's put it this way, I'm going to try to make things work tonight and tomorrow
<mort___>
ah— Irmin has only in memory backend for xen still (i believe)
<avsm>
wiredsister: this is definitely something that needs some illumination, because the "OPAM magic" (as mort puts it :) will probably disappear with the solo5 merge and need to be generalised
<wiredsister>
and when it doesn't, who can I bother is my question
<avsm>
seangrove: that sounds epic (the jitsu demo).
<hannes>
wiredsister: the mailing list :)
<mort___>
avsm: what thing are you anticipating disappearing exactly?
<wiredsister>
word up
<avsm>
mort___: the hardcoded --xen flag, replaced with a functoria variable for backend
<avsm>
mort___: it was only there initially for the "linking hack"
<mort___>
oh right — so the CLI interface changes? but the basic process (specify backend, get the right packages filled in by OPAM) will stay, right?
dsheets has quit [Remote host closed the connection]
<djwillia>
(bye all, i have to run, will let you know some possible July dates for a Cambridge visit!)
<avsm>
alright, meetings done! for the purposes of logging anyway :)
<mato>
avsm: isn't that already done via --target ?
<avsm>
thanks djwillia, hope we can get you down here!
<avsm>
mato: sort of, it's not first class in functoria
<thomasga>
I'm not sure how this will work :-)
djwillia has quit [Quit: Page closed]
GemmaG has left #mirage [#mirage]
<thomasga>
but happy to discuss/help when there is some concrete stuff to discuss :-)
<thomasga>
anyway, thanks all, leaving now
<engil>
unpurecamelbot: commit done
<unpurecamelbot>
done
<engil>
unpurecamelbot: bye
unpurecamelbot has quit [Quit: unpurecamelbot]
thomasga has quit [Quit: Leaving.]
dsheets has joined #mirage
talex5 has quit [Quit: Leaving]
qli has quit [Quit: Page closed]
avsm has quit [Quit: Page closed]
mort___ has left #mirage [#mirage]
dsheets has quit [Remote host closed the connection]
brson has joined #mirage
yallop has quit [Ping timeout: 250 seconds]
yallop has joined #mirage
seangrove has quit [Ping timeout: 276 seconds]
ricarkol has quit [Quit: Page closed]
dsheets has joined #mirage
nullcat has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
wiredsister has quit [Ping timeout: 260 seconds]
seangrove has joined #mirage
jermar has joined #mirage
copy` has quit [Quit: Connection closed for inactivity]
rgrinberg has quit [Ping timeout: 276 seconds]
insitu has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
dsheets has quit [Remote host closed the connection]
rgrinberg has joined #mirage
dsheets has joined #mirage
insitu has joined #mirage
mort___ has joined #mirage
mort___ has quit [Client Quit]
dsheets has quit [Ping timeout: 250 seconds]
wiredsister has joined #mirage
mort___ has joined #mirage
dsheets has joined #mirage
rgrinberg has quit [Quit: WeeChat 1.5]
rgrinberg has joined #mirage
dsheets has quit [Ping timeout: 250 seconds]
brson has quit [Quit: leaving]
brson has joined #mirage
wiredsister has left #mirage ["ERC (IRC client for Emacs 24.5.1)"]
insitu has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
insitu has joined #mirage
dsheets has joined #mirage
rgrinberg has quit [Ping timeout: 264 seconds]
rgrinberg has joined #mirage
Bluerise_ has joined #mirage
Bluerise has quit [Ping timeout: 250 seconds]
dsheets_ has joined #mirage
dsheets has quit [Ping timeout: 250 seconds]
srenatus has quit [Quit: Connection closed for inactivity]
dsheets_ has quit [Remote host closed the connection]
mort___ has left #mirage [#mirage]
smkz has quit [Quit: x]
smkz has joined #mirage
AltGr has joined #mirage
AltGr has left #mirage [#mirage]
insitu has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
mort___ has joined #mirage
copy` has joined #mirage
<apache2>
speaking of camps, registration just opened for a Danish hacker camp, https://bornhack.dk - I will be going, and I think 2-3-4 other ocaml/mirage enthusiast will join
rgrinberg has quit [Ping timeout: 244 seconds]
<Leonidas>
apache2: is that a successor to HAR2009?
<ahf>
no, the successor to HAR2009 was called OHM and was in 2013 - the next one will be SHA and will be in 2017 :-)
<ahf>
this is a new thing in denmark, but based upon OHM and the CCC camps :-)
<Leonidas>
i guess we need a thing in belgium, then it will be one hacker con yearly :-)
Bluerise_ is now known as Bluerise
mort___ has quit [Quit: Leaving.]
mcclurmc has joined #mirage
<apache2>
Leonidas: Danish != Dutch ;) HAR/OHM is in the Netherlands (Holland); it's Dutch
<Leonidas>
oh, you're right. I blame it on the late time at night for mixing it up!
<apache2>
this will be slightly smaller, but hopefully similar in all other ways :) I think it will be a great venue for mirage-hacking in the spirit of the Marrakech hackathon if enough ocaml-people decide to go :)