<avsm>
we'll definitely need this moving forward -- is anyone available to work on this one? Samoht is sadly on vacation in exotic climes
<avsm>
I was interested in looking into `opam` file generation -- I guess this is part of that
<yomimono>
i've intended to work on it repeatedly but it's time for me to face that I don't have time for it myself
<avsm>
That's fair -- its a big release this time! I'll put it on my list by default unless someone else steps up
<avsm>
Does anyone have any objections to generating an `opam` file per unikernel?
<avsm>
This way you could pin a project and all the other nice opam things
<yomimono>
no objections, but I seem to recall some doubts about whether that would work in earlier discussion
<mort___>
no — this would be part of the configure step presumably?
<yomimono>
I would love to have it though
<mato>
What would "installing" the generated package do?
<avsm>
mort___: yes part of configure
<mort___>
does opam / will opam2 have some way to consume an opam file and generate a switch thereby?
<hannes>
avsm: a highly customized opam for the concrete configuration? or a general one with lots of disjunctions?
<avsm>
mato: no installation phase, it would just be for the purpose of assembling dependencies and building
<Drup>
I'm not convinced it's the right way to solve the problem, but I don't have any great idea either
<mort___>
(not a requirement but might be nice in this case)
<avsm>
hannes: good question; i'd prefer a highly customised one because we know at `mirage configure` time which concrete backend is being built for
<avsm>
Drup: I was also in two minds about it, but the way I'm using unikernels is quite compatible with a package per application...
<Drup>
avsm: generating an opam file doesn't solve the pinning aspect of the question
<avsm>
mort___: there is support for 'local switches' in opam2 which will work like npm/node by grabbing opam state from the local directory
ptrf has joined #mirage
<avsm>
Drup: no but once you have an opam file you can generate a remote fairly easily that contains overrides
<mort___>
cool. then an opam file seems a reasonable way to go, particularly if `mirage configure` could default to *not* overwriting an existing one when run
<Drup>
avsm: I'm not sure I see how that would work
<avsm>
Drup: generate a packages/ directory with the various development versions of packages. We have some shell scripts to do this, but something with opam-lib would be better...
<avsm>
mort___: do not delete all your files. check.
<hannes>
avsm: clearly the opam file name will then be unikernel_name-configN-backendY!?
<hannes>
(and I fail to see how generating an opam file will solve 499)
<avsm>
hannes: yes, we could generate `opam.name-solo5` or similar
<mort___>
avsm: not quite :) specifically— main.ml and the Makefile can be obliterated every time (has anyone ever edited them?) but one might imagine that the opam file will get edited to specify constraints and obliterating it would be unhelpful
<yomimono>
I agree that it should be differentiated by backend to avoid making the "building a unikernel for multiple backends from the same directory" problem any worse
<avsm>
it doesnt solve it, but it gives us a vehicle to output the constraints
<Drup>
but we can already output the constraints
djs55 has joined #mirage
<avsm>
running `opam install` with a ton of constraints on the CLI is not pretty...
<Drup>
we don't need an opam file for that
<yomimono>
mort___: a human has definitely ever edited a main.ml. we also have obnoxious overwriting behavior for config in the xen case
<Drup>
ah, ok, I'm not sure why it's not pretty (the user doesn't input the constraint himself) but ok
<avsm>
no but an opam file is not side effecting -- I would prefer if all the CLI operations were saved as late as possible, rather than running opam install at configure time
<yomimono>
+1
<avsm>
makes scripting and CI significantly easier
<mort___>
yomimono: oh right. in that case it would be nice if the default behaviour of `mirage configure` would detect that and not do the obnoxious thing. or is it just easier to keep it the case that mirage configure obliterates everything en masse?
<Drup>
right, so the real goal is to be able to separate the opam part from the `mirage config` part
<Drup>
then I agree it's a good solution
<hannes>
I'd appreciate to have `--no-opam` as default in mirage-3
<yomimono>
mort___: I think we're derailing a bit here, we should probably take it to an issue or something
<avsm>
yeah, `mirage configure` running opam is pretty obnoxious, but more so when it's resetting a bunch of constraints and probably recompiling things
<avsm>
anyone any questions on a specific issue there
<yomimono>
curious to hear about pp_error and result types
<Drup>
avsm: before we wrap up: this should be a new ticket. 499 is only loosely related
<avsm>
Drup: true... we can add the constraints and then opam generation
<avsm>
Moving the C libraries out of mirage-platform would help quite a bit
<yomimono>
#568 (aka "prediquery") is merged, fwiw
<mato>
I was just looking at that issue (mirage/mirage-platform#124)
<avsm>
any thoughts as relates to solo5's platform needs?
<mato>
The biggest problem I see there is it's unclear if that can be done without teaching all the packages the C stubs would be moved into about the different backends
<avsm>
yeah, the cross compilation debate again :(
<mato>
yes :(
<avsm>
i think it's best to revisit it that one separately on the ticket or in a small group, it's too involved to go into again
<mato>
Stepping back from it, what do we expect to gain from moving the C stubs out of the platform package(s)?
<avsm>
lack of repetition -- they frequently get out of sync
<avsm>
upon the dependent package being updated
<avsm>
but we can survive in the current state if things dont change too fast, and it's a strong encouragement to not depend on C stubs :-)
<mato>
Yeah. I can see pros and cons for both approaches.
<avsm>
Ok, lets take this one into the issue
<mato>
Ack
<avsm>
TROVE: Lars pinged me for updated stats
<avsm>
would everyone be able to glance at mirage/mirage-www and push any new libraries of theirs to TROVE? I'll do a pass later this week as well
<yomimono>
solo5 is a particularly glaring omission
<avsm>
hah!
<mato>
what's TROVE? :)
<yomimono>
in the mirage/mirage-www repository there is a file called TROVE
<hannes>
(in the past, there were multiple breakages due to adjusting cstruct C stubs, but not platform ones, I suspect this will repeat and we should solve this issue, although it is painful)
<avsm>
and there's a tool in xenproject that clones it and generates feel-good graphs
<djwillia>
yeah all of the solo5 ones need to be added
<Drup>
avsm: oh, can I see the graphs ? :D
<mato>
ok, i'll do that...
<djwillia>
cool thanks mato
<avsm>
Drup: sure, will mail mirageos-devel when Lars generates them
<avsm>
samoht also has something fancy based on vossibility when he's back
<avsm>
i tried to deploy it but got banned from GitHub API due to the number of repos :P
<avsm>
darn rate limits
<avsm>
Runtime information passing
<Drup>
nice. pretty graphs are cool.
<avsm>
Magnus asked about this earlier -- do we have an issue for this yomimono ?
<hannes>
well, there are more libraries in dashboard somewhere
<yomimono>
avsm: sorry, "this" is what specifically?
<avsm>
There generally seem to be requests for various things to pass CLI options around to the mirage tool
<avsm>
"runtime information passing"
<avsm>
I believe it refers to the Env module
<avsm>
but I'm not sure who added this to the agenda :)
<yomimono>
it was me, suggesting that some folks continue a private conversation about bootvars, argv, and env in a place where there might be more opinions
<mato>
yomimono did based on our Slack discussion this afternoon
<yomimono>
we now have duplicated code for bootvar parsing; perhaps passing runtime information could be done more intelligently and in a more structured manner for many unikernel backends
<yomimono>
that may or may not be an accurate summary
<mato>
more or less. Even if we don't pass runtime information in a more structured way, there's still the question of "who does the parsing" (as in what library / package)
<hannes>
(well, the overall question is rather: how do I write a backend-agnostic mirageos unikernel which should receive runtime information)
<mato>
Yes. The current interface to "runtime information" from the Mirage point of view is "an argv-style array of strings"
<mato>
Which then gets parsed by Functoria (using cmdliner?)
<Drup>
One of the issue is that the whole key thing is based on cmdliner, and cmdliner really want an argv-style string array
<mato>
Right. Except that on the Xen/Solo5 side, a bunch of back and forth is happening, since both of these targets pass the "command line" as a simple string.
<avsm>
so the issue is string vs string list?
<mato>
So, the bootvar code for both has to reparse *that* string, then pass it to Functoria, which parses it with cmdliner.
<avsm>
ah so bootvar splits it up by spaces to get the string list
<avsm>
but has to deal with quoting issues, presumably
<mato>
Yeah
<avsm>
It's hard to get away from the string list, it's the standard way args are passed
<avsm>
I'd suggest fixing solo5 :)
<mato>
It's not a Solo5 problem
gemma_ has joined #mirage
<yomimono>
also cmdliner doesn't like it when you try to pass stuff it doesn't know about, but we don't always have control over the incoming string (e.g. an `extra` passed by EC2)
<mato>
Xen does the same thing (pass a string)
<yomimono>
so there's potentially more extra stuff needed to shim some runtime problems
GemmaG has quit [Quit: Leaving.]
<avsm>
Xen can be made to pass a string list though
<avsm>
we control both sides
<yomimono>
avsm: ? not always, unless I misunderstand what the sides are
<Drup>
yomimono: dbunzli is not very favorable to an "any" combinator that would just be a huge catchall for all the things not recognized by the rest
<avsm>
for the Env parsing on Xen we use a xenstore subtree to pass the args, if i remember right
<avsm>
that's written by our tools, or is it written by xl ?
<yomimono>
it's written by whoever invokes the code to start the VM, which isn't always the user
<yomimono>
it might be amazon
<avsm>
ah no, it comes in as `cmdline`
<avsm>
ho hum...
<avsm>
guess we need a string -> string list parser :)
<mato>
which is currently hiding in parse_argv.ml in bootvar-xen (and duplicated in solo5)
<avsm>
right, so that can be pulled out into a library
<Drup>
last time I remember, parse_argv's implementation was not very reliable
<avsm>
it's been improved by jonludlam, but needs some more u se
<avsm>
lets move on... doesnt sound like anything particularly controversial here. Just need to make sure we track it
<avsm>
anything else burning about argv? sounds like string should be the input, and we need to have a reliable converter to string list
<avsm>
Ok, is everyone sitting down? CI is now available and being rolled out step by step! https://ci.ocaml.io:8443 (temporary URL) !!!
<Drup>
yomimono's point still stand. qubes treats the cmdliner as an env, so it adds lot's of stuff
<mato>
IMO parsing a string into string list will never be terribly reliable due to quoting issues (also thinking about passing *in* quotes on the whatever-launches-the-vm side)
<Drup>
we treat it as a command line, hence refuse things that we don't know about
<avsm>
dont all hit it at once, the web interface is painfully slow and ugly, but is at least relatively easy to fix
<avsm>
thing is, we cant fix it the other way since kernel launchers give us cmdline as a baked in interface
<avsm>
so we have to fix it, lets just do it
<Drup>
avsm: I'm just not sure how :)
<avsm>
painful parsing, very painful parsing :-)
<Drup>
avsm: no, that's another issue
<avsm>
oh you mean extending the env
<Drup>
Well, we'll discuss it when yomimono makes the PR for qubes's argv
<avsm>
Yeah
<yomimono>
it's probably going to be a PR for argv_xen more generally
<yomimono>
the problem is bigger than qubes; we'll see it anywhere we don't control the full cmdline input
<yomimono>
see issues for booting on ec2
<Drup>
exactly
<avsm>
it would be easier with a registry-style model somewhat
<avsm>
but not for mirage3
<mato>
One option would be to switch to a different "channel" than the "kernel cmdline". Like the pass-data-in-multiboot-module trick I did for rumprun.
<avsm>
yeah mato -- only thing is that cloud providers make setting the PV cmdline easy
<Drup>
mirage="...." ?
<avsm>
however if more people are doing HVM boot then this can happen in grub and could do multiboot module
<avsm>
and ec2 certainly does pvgrub, not direct pv
<avsm>
Lets move on, 15 minutes left
<avsm>
Quickly covering CI
<avsm>
CI is now available and being rolled out step by step! https://ci.ocaml.io:8443 (temporary URL) !!!
<avsm>
It tests multiple distros (Alpine, Debian, Ubuntu, CentOS, Fedora, OpenSUSE), multiple compiler versions (4.02.3,4.03.0,4.03.0+flambda,4.04.0) and reverse dependencies. I'm adding multiple remote support (so we can add mirage/mirage-dev).
<avsm>
A nice feature is that all the logs are stored in git (temporarily at https://github.com/avsm/mirage-ci-logs) so it's quite easy to clone a particular log run. Also, all of the git remotes (including opam-repository) are tracked exactly, so it rebuilds reliably if any of the dependent repositories are pushed to.
<avsm>
This should satisfy mato's CI requests quite soon with multi-remote tracking, and then I'll roll it out to mirage/cohttp and mirage/tcpip
<avsm>
and to more repos as it gets more stable
<talex5>
Why do some of them say "(Allowed failure)"?
<avsm>
because revdeps hardly ever succeed and i didnt want it to fail the overall test
<avsm>
So in the above PR, the revdeps were: Ok (mirage-fs,mirage-seal,nbd) Err (imaplet-lwt,tftp)
<avsm>
that can be changed pretty easily
<mato>
how does it decide what to build? i.e. what is configured in the repo?
<avsm>
it does an opam pin and then installs mirage, and then revdeps
<avsm>
talex5: ah compiler variants other than 4.03.0 are allow_fail atm since 4.04 fails; I'll change that to make only development compilers allow_fail
<Drup>
I don't have much to add compared to my last email :p
<mato>
hannes: quite possibly. also, i need to switch the code to dynamically alloc memory based on the actual virtqueue size otherwise we waste tons of memory on the rx/tx buffers
<avsm>
how much?
<avsm>
hyperv just forces guests to allocate a 16MB contig physpage
<hannes>
as mentioned in the agenda, there are tags in use, and a Canopy issue to get a nicer index
<avsm>
So I'm proposing to use Canopy for the ICFP liveblogging, so it'll get more users
<yomimono>
canopy: the canopy move is great, but the abstracts for each project now seem rather cryptic compared to the view we got when they were on the wiki
<avsm>
yeah -- and editing is slightly harder
<avsm>
can we move hannesm/canopy-data to mirage/canopy-data btw?
<hannes>
yomimono: the nice thing is everybody can extend them now *hint* :)
<avsm>
hannes: indeed ;-)
<hannes>
avsm: sure, well, I can't. you can't. but a simple permission problem
<avsm>
ok sent you/me a reminder mail to sort this out later
<yomimono>
hannes: an argument could be made that the paragraph summaries from the wiki should be the abstract, and each project should have *even more* information behind the link I guess
<yomimono>
but I can't do the second part of that for any project but my own
<mato>
avsm: several megabytes worth, i forget the exact number. in any case we'll need dynamic allocation to support >1 net/block interface.
<hannes>
yomimono: I'd appreciate having a single sentence as abstract to not convolute the index page too much
<avsm>
well, the better curated your projects are, the more people you'll get helping me
<avsm>
cohttp is awful right now so i need to improve mine...
<yomimono>
hannes: I don't know much about this information architecture thing everyone keeps talking about when we mention websites so I'll take your word for it :P
* hannes
no web
<avsm>
engil: any objections to a canopy deployment for ICFP liveblogging? I'll set that up
<Drup>
I'm not convinced the index is much better than the big list of before
<hannes>
but the full descriptions were way too long (tried it and failed)
<engil>
no objection
<hannes>
Drup: look and add to Canopy#60
<seangrove>
avsm: I was thinking that some tickets to flesh out cohttp (multi-part POST support, etc.) could be a great starter project
<avsm>
seangrove: yeah; a lot of this is helped/solved by ocaml-webmachine on the server side, but not the client side
<avsm>
i was poking around to see if a client webmachine is practical, but its not really
<seangrove>
Yeah, sorry, I meant purely for the client right now
<avsm>
yep, agreed
<seangrove>
But that's perhaps a topic for the next mirage meeting....
<gemma_>
Speaking of icfp - along with a general mirage meet, we could do a BoF session too for anyone who has mirage related questions
<gemma_>
If we find a time that works on the Saturday...
<avsm>
gemma_: yes! I'm in japan for the last half of ICFP (ocaml workshop through to cufp)
<avsm>
there's also an ocaml tutorial which we could partially hijack
<avsm>
if you want a mirage clump
<gemma_>
Ah that could work
<avsm>
who else is in japan for icfp?
<yomimono>
from my experience teaching the ocaml tutorial last year, the audiences would be quite dissimilar
<gemma_>
Ideas for how we could add to canopy for live blogging/capturing the event also welcome!
<yomimono>
but I'm very down with the idea of a BoF session
<avsm>
yes audiences would be, but usually a bit of spare room to hang out?
<yomimono>
that I can't speak to, not knowing the venue
<Drup>
hannes: hum, I actually don't have much to add to that. It really needs categories/sorting
<avsm>
gemma_: any chance of a doodle on mirageos-devel?
<yomimono>
I'm in japan for ICFP though and would be very stoked to have a mirage meetup
<yomimono>
STOKED and HYPED
<avsm>
yeah! and seangrove is speaking at cufp!
<seangrove>
I'll be in japan speaking at CUFP (won't be for ICFP), was thinking about doing a Mirage/Reason hacking session. Would like to see some Reason unikernels.
<avsm>
seangrove: i'm up for that
<gemma_>
Sounds great!
<avsm>
alright </meeting>
<avsm>
let the great git bot in the sky commit!
<avsm>
all hail the great git bot in the sky
<djwillia>
thanks all!
<avsm>
just a few more weeks to mirage3, lets hack hack
* yomimono
puts on big headphones
<mato>
hackity hack
* yomimono
cues up uplink soundtrack
<mato>
thanks all indeed
* yomimono
starts projector to put scrolling text on her face
<engil>
unpurecamelbot: commit done
<unpurecamelbot>
done
<seangrove>
Awesome!
<engil>
unpurecamelbot: bye bye bye bye
unpurecamelbot has quit [Quit: unpurecamelbot]
talex5 has quit [Quit: Leaving]
avsm has quit [Quit: Page closed]
balduin has joined #mirage
seangrove has quit [Ping timeout: 244 seconds]
yomimono has quit [Ping timeout: 250 seconds]
djwillia has quit [Quit: Page closed]
seangrove has joined #mirage
yomimono has joined #mirage
mort___ has quit [Quit: Leaving.]
seangrove has quit [Ping timeout: 250 seconds]
seangrove has joined #mirage
seangrove has quit [Ping timeout: 265 seconds]
balduin has quit [Ping timeout: 244 seconds]
gemma_ has quit [Quit: Connection closed for inactivity]
mort___ has joined #mirage
rgrinberg has quit [Ping timeout: 244 seconds]
balduin has joined #mirage
mort___ has quit [Quit: Leaving.]
seangrove has joined #mirage
seangrove has quit [Ping timeout: 265 seconds]
rgrinberg has joined #mirage
<yomimono>
balduin has quit [Ping timeout: 264 seconds]
jermar has joined #mirage
seangrove has joined #mirage
yomimono has quit [Quit: Leaving]
jermar has quit [Ping timeout: 244 seconds]
seangrove has quit [Ping timeout: 252 seconds]
GemmaG has joined #mirage
seangrove has joined #mirage
seangrove has quit [Ping timeout: 255 seconds]
Bluerise_ is now known as Bluerise
balduin has joined #mirage
seangrove has joined #mirage
agarwal1975 has quit [Quit: agarwal1975]
GemmaG has quit [Quit: Leaving.]
balduin has quit [Remote host closed the connection]