10:42 <ehirdoy1> Hello, I got "Fatal error: exception Stack overflow" with "mirage configure -t hvt --kv_ro=direct" but not with "mirage configure -t unix --kv_ro=direct". Any hint to debug futher would be really appreciated.

10:45 <hannes> ehirdoy1: according to https://github.com/mirage/mirage/blob/v3.5.0/lib/mirage_impl_kv.ml#L49 on unix it uses mirage-kv-unix, whereas on hvt/.. it uses crunch

10:46 <hannes> how large (in byte size, number of files, depth) is your data directory? would you mind to open an issue on the mirage/mirage repository including these information?

10:50 <ehirdoy1> hannes: sure, I'll after narrowing down a bit more....thanks!

10:52 <hannes> ehirdoy1: there were various changes in crunch 3.0.0 which may have introduced such a regression -- it may as well be that you end up in the same stack overflow with earlier crunch. (crunch is supposed to be executed at build time, not at configure time)

10:52 <hannes> do you see the stack overflow at configure or build time?

10:59 <ehirdoy1> hannes: yes, it's at make, exactly where ocamldep is caled.

11:00 <hannes> how big is your data? there's https://github.com/mirage/mirage/issues/396 ... may as well be that your data directory contains symlinks and is cyclic?

11:01 <ehirdoy1> 213MB...is there any max limit?

11:04 <Drup> I'm not sure crunching that kind of size is reasonable.

11:04 <hannes> well, it's all marshalled into OCaml code, I'd not expect that it works for more than 1MB.

11:05 <hannes> alternatives (though I haven't tested them properly): use a block device with either FAT or TAR data.

11:07 <ehirdoy1> aha, ok. I'll use a block device, then. Thanks!

11:10 <hannes> I'm actually not sure whether the block device works with mirage 3.5.0 properly... but I think that mirage-www uses a fat filesystem... (sorry, I don't have much experience with block devices and mirageos)

11:20 <ehirdoy1> I would be happy if pointing to some example code of using disk && fat;) mirage-skelton/block doesn't use FAT but bare sectors. Here's some gap to me....:(

11:21 <hannes> ehirdoy1: ah, yes -- https://github.com/Solo5/solo5/issues/325 reminded me that going the block-way on solo5 may be a dead end right now.

12:15 <pie_> hey hannes, I had some questions left from bobkonf so I hope you dont mind me dumping them on you :p ,

12:16 <pie_> 1) how is it that i managed to run a solo5 hvt binary without it asking me for root access or something, I thought this stuff uses virtualization? (the workings of which I admit to not being intimate with)

12:16 <pie_> 2) have you looked at halvm?

12:17 <pie_> 3) unikernels decrease the attack surface, but now you have to go through a complex networking stack for IPC, isn't this worse than microkernels? (though I suppose there's nothing stopping someone from using unikernels and microkernels together)

12:17 <hannes> pie_: (1) on some unix operating systems, such as GNU/Linux and OpenBSD IIRC, there's no need to be root to create virtual machines, but write permissions to /dev/kvm are sufficient (and/or being member of a privileged group that has these)

12:18 <pie_> 4) how does the unikernel being a single unmanaged address space influence security and bugs? (obviously using something like ocaml over c helps)

12:19 <hannes> pie_: (2) yes. but I'm really not interested in Haskell, neither in a project that requires quite some work to get going.

12:19 <pie_> 5)how does a unikernel with virtualization compare to microkernels with ?processes? (is the question even reasonable)

12:19 <pie_> and thats all I've got right now

12:20 <hannes> pie_: (3) I don't understand. "complex networking stack" is worse than "shared mutable memory"? for me, security is also a question of the amount and usage of communication channels: if you've only one, you need to review only one. if you have two, you have to review two (and decide what to use when)

12:21 <hannes> pie_: (4) we rely on the isolation guaranteed by the OCaml type system, indeed. for our C code, we're working towards best practises (R^X, ALSR, canaries, ...)

12:22 <hannes> pie_: (5) did you see https://github.com/Solo5/solo5/pull/310 ? this deploys as linux process with a strict seccomp ruleset

12:23 <pie_> thanks ill think about your answers

12:24 <pie_> 5) I haven't thought about how using capabilities will change the relationship of a monolithic kernel to the other things (disclaimer: I'm very new to this stuff)

12:27 <pie_> hannes, and if you accept some criticism;

12:27 <pie_> 1) I felt it was kind of hard to ask questions, but maybe there was something wrong with my question asking strategy :p ,

12:28 <pie_> 2) it would have been nice if you listed virtualization as something to have working for the demo, but as above it turned out not to be a problem for me , I did get scared for a moment that since I hadn't explicitly set stuff up I wouldn't be able to run all the cool binaries

12:28 <hannes> pie_: sorry for that. i was also a bit short on time.

12:29 <pie_> which is to say, ehaustive prereqs would be nice i guess :p

12:29 <hannes> pie_: "You can follow the tutorial on a Linux, FreeBSD, OpenBSD, or macOS box. You won’t need a hypervisor to follow the tutorial, but will be able to utilize Linux KVM during the tutorial." from https://bobkonf.de/2019/mehnert.html was not sufficient?

12:30 <pie_> ah ok, my bad then

12:30 <pie_> I think I missed that

12:31 <pie_> 3) related to 1) I suppose; at least for me personally, the delve into the code with the time reading stuff or what was it, wasn't very illuminating, and might have been better spent, but I did miss the first 15 minutes so maybe I was just missing all the background

12:31 <hannes> pie_: thanks for your feedback, and yes indeed -- next time I'll have a much better idea what to prereq, and how to organise this session better to allow for questions

12:32 <pie_> I think the tutorials would have been better with 4 hours instead of two, but you only have so much time :)

12:32 <pie_> with the fpga hardware tutorial we just barely got to some demos at the end

12:32 <pie_> and with another two hours we could have probably been running some stuff ourselves

12:33 <hannes> pie_: I introduced the high-level constructs at the beginning, yes. afterwards, as planned, I went down into examples, answering questions such as "where's the main loop" by showing the code.

12:35 <pie_> yeah I got a lot of that. I suppose another unhelpful thing is I've not really written any ocaml (I did peek at a quick guide beforehand), but I think it was mostly followable.

12:36 <pie_>

12:38 <hannes> pie_: sorry about that, but tbh, in 90 minutes I will never be able to teach OCaml and MirageOS concepts. in a talk, i obviously try to bring that across, but in a tutorial i really want to dive deep into the code.

12:39 <reynir> Was it recorded?

12:39 <hannes> reynir: no

12:40 <pie_> hannes, yeah, I want to clarify that I did not expect an ocaml tutorial

12:41 <pie_> I did not mean your quick intro of ocaml was unhelpful

14:18 <apache2> ehirdoy1: crunch is a bit interesting, with the new API you have to first chunk your files before compile, then read the full pieces from your process, where before you could do streaming reads

14:19 <apache2> I found that larger than ~4 MB caused the unikernels to hang using 100% cpu

14:19 <apache2> not sure if that bug is still there

14:21 <apache2> pie_: re: attack surface from unikernels using complex network stacks being worse than using limited IPC interface from a microkernel -> yes, it's a lot worse.

14:23 <apache2> with solo5 you use a `tap` interface (so only ethernet), which is much less insane than the IP stack, but there's still a decent amount of surface. if you take a bit of care you can turn off a lot of the funkier parts (arp, etc), but that indeed takes care.

14:24 <pie_> sidenote, iirc solus is in C? (or i think hannes said ada/spark, but the code looked like c?)

14:24 <pie_> *solo

14:25 <apache2> if you want to limit attack surface you probably also want to limit access to physical network card drivers (ie try to avoid directly attaching it to an interface associated with a network card), and be aware that if you're using a single network bridge for multiple unikernels that they may be able to access packets meant for each other

14:25 <apache2> bridges have a tendency to go from switch mode to hub mode for instance when you attach `tcpdump` which reconfigures the interface to promiscuous mode by default

14:26 <apache2> I'm hoping we will get a more IPC-like interface in solo5 at some point, but for now this is what we have.

14:26 <apache2> pie_: yes, solo5 is C

14:27 <apache2> Muen-SK ("Muen Separation Kernel") is a microkernel in Ada/SPARK that iirc also has some mirage support, maybe that's what you're referring to?

14:28 <pie_> oh oops

14:28 <pie_> probably

14:28 <apache2> solo5 is very clean C code and not that much of it altogether though, so as far as C code goes I think it's pretty sane

14:29 <apache2> and the hypercall interface is also very limited

14:29 <apache2> I'd be more worried about hardware bugs than solo5 bugs tbh

14:29 <pie_> sounds reasonable

14:29 <pie_> but what can you do about that

14:29 <pie_> one kernel per core?

14:30 <apache2> ie your unikernels can't speak directly to KVM or whatever, they have to go through the "tender" using a pretty limited API (I believe the name "Solo5" comes from having only 5 hypercalls, nowadays there's been a few added, but a lot of thought went into producing a pretty clean interface imho)

14:31 <apache2> so a solo5 unikernel running under KVM is by no means comparable to a Linux subject running under KVM with full access to the KVM hypercall interface

14:31 <apache2> pie_: one kernel per machine ;)

14:32 <apache2> but yeah, if you're worried about stuff like the recent SPECTRE-like attacks, one unikernel per core sounds like a good trade-off

14:32 <pie_> i spent like two weeks looking at that stuff and i still feel like i dont *really* get it :/

14:32 <pie_> but maybe thats because I havent written POCs yet

14:32 <pie_> I get the fundamentals but I feel like I'm still missing somethign

14:33 <hannes> apache2: you may want to report your "crunch" issues to the issue tracker, so others can think about it and fix it, no?

14:33 <pie_> like, I wouldnt have figured out "use separate cores" is a mitigation

14:35 <apache2> hannes: maybe, it's not a big issue for me to run `split` on the binary though, and given the response time on that repo I don't want to overburden the maintainer(s)

14:36 <apache2> ie if someone finds energy to think about issues there I'd prefer to get some response to my two year old post on the issue tracker before I pile up further workloads for them

14:38 <apache2> pie_: it's hard to say for unfound issues where they're going to be found, but the trend seems to be that the less workloads you schedule on the same hardware, the better. ie turning off hyperthreading a.k.a. SMT (where processor threads share a core) seems to mitigate some of the recent vulnerabilities

14:38 <pie_> who knows what else is hiding in the weeds

14:38 <pie_> cue pokemon battle music

14:39 <apache2> I haven't followed the recent cascade of Intel bugs, but this is something that hits Intel hw across the board and not really specific to unikernels

14:39 <pie_> not really intel specific either

14:39 <pie_> or at least, not in general

14:39 <pie_> idk about recent stuff

14:39 <pie_> last i heard was uhh

14:39 <pie_> "SPOILER"

14:39 <apache2> a lot of the benefit of a mirage unikernel is that it is harder to get to the point where you have these memory corruption bugs in the first place by having a strong type system

14:40 <pie_> right. except i wouldnt have used "these" because its not memory corruption

14:40 <pie_> iin the case of microarch sidechannels i mean, i may have misread your flow

14:41 <apache2> exploiting the sidechannel attacks usually require access to specific instructions in very specific patterns that are pretty unlikely to be exposed to normal users of the programs. in most cases the research assumes control flow control in the first place

14:42 <apache2> either through program compromise our through execution of arbitrary JIT-compiled code

14:42 <apache2> (e.g. javascript). sure, you can be unlucky, but you have to be pretty unlucky

14:44 <apache2> if you want to get your feet wet with mirageos, I would wholeheartedly recommend the solo5 seccomp target, I'm a big fan of that

14:44 <apache2> the requirements are pretty low (a linux shell), and it is easy to manage and get a feel for it without having to deal too much with hypervisors and hardware support.

14:45 <apache2> mirage also has a unix target that can be used pretty much like a normal unix program

14:56 <pie_> fair enough

14:56 <pie_> ty \o/