asheesh changed the topic of #sandstorm to: Welcome to #sandstorm: home of all things sandstorm.io. Say hi! | Channel glossary: "i,i" means "I have no point, I just want to say". b == thumbs up. | Public logs at https://botbot.me/freenode/sandstorm/ & http://logbot.g0v.tw/channel/sandstorm/today
isd has quit [Quit: Leaving.]
<mokomull> hrm, I'm getting the EIO from spk dev as well. CONFIG_FUSE_FS=y on Ubuntu 16.04, so shouldn't be a modules issue...
<mokomull> $ spk dev
<mokomull> App is now available from Sandstorm server. Ctrl+C to disconnect.
<mokomull> *** Uncaught exception ***
<mokomull> sandstorm/fuse.c++:378: failed: read(/dev/fuse): Input/output error
<mokomull> stack: 0x56740b 0x564ecc 0x5dd9a3 0x5dc65d 0x48a79f 0x48a2ba
<mokomull> I can't get any symbols out of that stack trace.
esmiurium has joined #sandstorm
<dwrensha> mokomull: what code are you running under spk dev?
<dwrensha> like, are you able to run a simple "hello world" without hitting the EIO?
<mokomull> I've got a Rust binary that answers HTTP requests by just dumping the headers in the response - was hoping to poke around at the platform a little bit.
<mokomull> This is my hello-world equivalent, so no :)
<dwrensha> :/
<dwrensha> does `sudo modprobe fuse` do anything for you?
<mokomull> exits cleanly. Like I said, $ grep -i FUSE /boot/config-$(uname -r)
<mokomull> CONFIG_FUSE_FS=y
<dwrensha> Is your Rust server actually serving HTTP or are you using the raw Cap'n Proto API, as with https://github.com/dwrensha/sandstorm-rawapi-example-rust?
<mokomull> It actually serves HTTP, and I can run it locally.
<mokomull> https://gist.github.com/mokomull/8b1ed207e38fcd54a9918dc280119bfd is the strace from spk dev -- as far as I can tell, read(4) is what returns EIO, and fd 4 comes from socketpair() - not sure why the error message is being attributed to /dev/fuse.
<dwrensha> mokomull: would you be willing to share your code with me so I can try if it works on my dev setup?
<dwrensha> My plan would be: 1. eliminate the possibility that this is a sandstorm bug
<dwrensha> then 2. try to figure out what configuration on your system is making this not work
<mokomull> sure, trying to work out a rube-goldberg contraption to get the code into the same machine as my Github keys
<dwrensha> hm. it does look like the crash is happening before your app even gets invoked
<dwrensha> when you do `spk dev`, does the crash happen immediately?
<mokomull> very quickly, yeah.
<dwrensha> by the way, I recently learned about strace's "-y" flag
<dwrensha> which makes it easier to understand what file descriptors refer to what
<mokomull> oooh.
<mokomull> aha, ok, that makes more sense now - the fd to /dev/fuse was received over an SCM_RIGHTS so I missed it when grepping.
sydney_untangle has joined #sandstorm
sydney_u1tangle has quit [Ping timeout: 252 seconds]
<mokomull> ok, finally got it where I can upload it: https://github.com/mokomull/reflect_header
<dwrensha> wow, and I thought my Rust programs took a long time to compile
<mokomull> yeeeeah, matroska doll of depedencies there
<mokomull> I'm probably going to take that out back and light it on fire when I'm done :P
<mokomull> Pencil pulled in gems such as "winapi" >_< And that's supposed to be the "lightweight" alternative.
<dwrensha> turns out to be difficult to not include winapi
<dwrensha> because the `time` crate unconditionally depends on it
<dwrensha> OK, I got your app to work on my machine
<dwrensha> at first it failed with "No such file or director; argvp[0] = reflect_header"
<dwrensha> but then I edited sandstorm-pkgdef so that the binary could be found, and it worked
<mokomull> aye, I'd cp'ed the binary to somewhere it could also be found.
<dwrensha> so I wonder what's peculiar about your system that makes read(/dev/fuse) fail with EIO
<dwrensha> what kernel are you running? (`uname -a`)
<mokomull> Not just mine, mnutt indicated the same earlier. I'd like to avoid Vagrant since this is already a purpose-built VM.
<mokomull> Just upgraded to 4.4.0-31-generic an hour or so ago, happened on -24-generic too
* dwrensha looks at history
<dwrensha> I wonder if mnutt is also using Ubuntu 16.04
<mokomull> Let me try sshfs and make sure *anything* FUSE is working...
<mokomull> ok, that does work.
<dwrensha> weird
<dwrensha> could there maybe be some odd configuration of selinux or such that is overzealously killing these calls?
<dwrensha> (just a wild speculation)
<mokomull> I did "service apparmor stop" on that suspicion, but I'd expect that to be EPERM too.
* dwrensha attempts to set up a xenial vagrant box
<dwrensha> "The machine is in the 'gurumeditation' state. Please verify everything is configured properly and try again."
<dwrensha> wat
<dwrensha> now trying `do-release-upgrade` on a Ubuntu 15.10 box I had lying around
ocdtrekkie has quit [Remote host closed the connection]
<mokomull> heh, never seen that one before
<dwrensha> OK, updated to 16.04
<dwrensha> and I get the same FUSE error!
<mokomull> Iiiinteresting. Wonder if it's some Canonical specialsauce.
<mokomull> Bingo! Used a mainline 4.4.14 from http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.4.14-xenial/, works *fine*.
<dwrensha> so it's a kernel problem?
<mokomull> I believe so. After that, had to do nothing (except fix the argv[0] for the program) to get it to work as expected.
<dwrensha> will Xenial's default kernel eventually get updated?
<digitalcircuit> dwrensha: Xenial will probably follow the same policy as past releases, e.g. https://wiki.ubuntu.com/Kernel/LTSEnablementStack As to the default in the near future, or the 4.4 series, I don't know.
<mokomull> hm, I didn't try anything out of xenial-proposed. Lemme install the 4.4.0-32 from there and see what happens.
<mokomull> My wild-ass-guess is that the breakage is a Canonical patch on top of mainline, and if so, it'll get carried to future releases anyway :)
<mokomull> Same error on -32, so I'm probably going to bisect the Canonical patches when I get back.
<dwrensha> it'd be awesome if we could pinpoint the problem
xet7__ has joined #sandstorm
xet7 has quit [Read error: Connection reset by peer]
wolcen has quit [Ping timeout: 250 seconds]
jemc has joined #sandstorm
isd has joined #sandstorm
xet7__ has quit [Quit: Leaving]
xet7 has joined #sandstorm
jemc has quit [Ping timeout: 264 seconds]
isd has quit [Quit: Leaving.]
<asheesh> dwrensha / mokomull / digitalcircuit / mnutt: Fascinating re: 16.04 kernel vs. FUSE thing
<mokomull> asheesh: so I started bisecting the kernel, and I've still got another 11 steps or so, but this looks super suspicious: http://kernel.ubuntu.com/git/ubuntu/ubuntu-xenial.git/commit/?id=9b869708bde3dd34f0e1c59c7c2abe1e433e5b35
<asheesh> Fascinating.
<asheesh> Also you are clearly willing to dig into things which is super fun to see.
<asheesh> Arguably we need a FUSE RPC proxy to work around that I guess.
digitalcircuit has quit [Remote host closed the connection]
<asheesh> Or we need to talk with the Ubuntu kernel maintainers, more likely.
<mokomull> Starting to put two and two together - guessing the SCM_RIGHTS message is to pipe FUSE out of the container.
<mokomull> If that commit is it, then you're probably already getting gibberish for pids over FUSE, but I'll wager nothing's actually conditional on pids.
digitalcircuit has joined #sandstorm
<asheesh> What does "pids over FUSE" mean?
<asheesh> Does FUSE have a way to talk about PIDs? Honest question; I'm more of a n00b than dwrensha at this stuff.
<mokomull> asheesh: looking at the diff, the pid is in the fuse_req struct, and had previously been shoved in there untranslated.
<asheesh> Ah hah, I see that now, thanks!
<mokomull> As for the actual consequences of that - I've got no idea.
<asheesh> Yeah; it's fascinating but probably not that important for us. The "return -EIO;" is more troublesome.
<asheesh> I guess I should email them.
<asheesh> Or at least write this up in a Sandstorm bug first.
<asheesh> Or you can, if you prefer to get credit for it! Also feel free to keep bisecting, but this does look thoroughly relevant.
<asheesh> I wonder if you can test with this one patch reverted, even.
<asheesh> Are you doing kernel builds as part of your bisect?
<mokomull> I will try that after the last step. I kicked off a 'make' before I read through the logs for fs/fuse.
<mokomull> Yeah, and I'm being lazy and building all the modules that Ubuntu ships with, so it's not exactly what we call "fast". But at least I have a lot of CPU in this box.
<asheesh> Well that's good at least (-:
<asheesh> https://github.com/sforshee/lxcfs is interesting; FUSE filesystem for fake /proc for containers.
<asheesh> Time for me to fall asleep I think.
<asheesh> Talk to you tomorrow or so!
<mokomull> asheesh: more than fake /proc - fakes /sys/fs/cgroup too, IIRC, since cgroup namespaces are new (and require unified hierarchy). I've pointed zarvox at lxcfs before :)
<mokomull> And this kernel testing exploded XFS on my test VM. I think that's a sign I should stop for the day.
dagelf_ is now known as dagelf
Telesight has joined #sandstorm
Isla_de_Muerte has joined #sandstorm
NwS has quit [Ping timeout: 244 seconds]
xet7_ has quit [Quit: AndroIRC - Android IRC Client ( http://www.androirc.com )]
nwf has quit [Ping timeout: 244 seconds]
nwf has joined #sandstorm
mnutt has joined #sandstorm
sydney_untangle has quit [Ping timeout: 240 seconds]
Isla_de_Muerte is now known as NwS
mnutt has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
<dwrensha> mokomull: when you said "specialsauce", I didn't realize that Ubuntu's modifications to the kernel are actually labeled "SAUCE" :)
jemc has joined #sandstorm
jemc has quit [Ping timeout: 265 seconds]
jemc has joined #sandstorm
mnutt has joined #sandstorm
jemc has quit [Quit: WeeChat 1.4]
jemc has joined #sandstorm
mnutt has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
mnutt has joined #sandstorm
mnutt has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
bony has quit [Ping timeout: 240 seconds]
bony has joined #sandstorm
isd has joined #sandstorm
jemc has quit [Ping timeout: 250 seconds]
dwrensha has quit [Quit: ChatZilla 0.9.92 [Firefox 47.0/20160604131506]]
frigginglorious has joined #sandstorm
<mokomull> asheesh: reverting that patch [and fixing merge conflicts to keep the user-namespaces support they also added] seems to have done the trick.
frigginglorious has quit [Quit: frigginglorious]
<asheesh> And I guess what if you remove just one or two of the "return -EIO"?
<asheesh> I guess presumably we know the answer. I guess I'm wondering exactly what to do now! Let me think about it more in a little bit.
frigginglorious has joined #sandstorm
bony has quit [Ping timeout: 240 seconds]
dwrensha has joined #sandstorm
<kentonv> hmm, so who should tell Canonical that they broke Sandstorm development?
frigginglorious has quit [Quit: frigginglorious]
<mokomull> asheesh: I mean ... that's basically what the revert-and-fixup ended up being: https://gist.github.com/mokomull/a6dbb46e44bd0d55092ee565321ae40f
frigginglorious has joined #sandstorm
frigginglorious has quit [Quit: frigginglorious]
xet7_ has joined #sandstorm
Isla_de_Muerte has joined #sandstorm
NwS has quit [Ping timeout: 260 seconds]
frigginglorious has joined #sandstorm
Telesight has quit [Quit: Leaving.]
lukexj has joined #sandstorm
frigginglorious has quit [Quit: frigginglorious]
mnutt has joined #sandstorm