harish_ has quit [Read error: Connection reset by peer]
harish has quit [Ping timeout: 265 seconds]
harish has joined #sandstorm
shachaf has joined #sandstorm
shachaf has quit [Changing host]
shachaf has joined #sandstorm
<ocdtrekkie>
sy: Your MongoDB is sad. I do not know why it is sad or how to fix it.
<ocdtrekkie>
But I do know your MongoDB is sad.
<ocdtrekkie>
Is this a new install? Or have there been any recent changes to your server?
ocdtrekkie_ has joined #sandstorm
ocdtrekkie has quit [Ping timeout: 264 seconds]
<ccx>
Wild guess: it exceeds memory limit, fails allocation, and crashes
<sy>
ocdtrekkie_: only relevant thing i can think of is ^^ and that I'm using grsec
<sy>
it's got 4G though so should be plenty
<sy>
unless it hits a limit
<ccx>
Mongo is known both for inefficient data format and for aggresively mmaping it into memory. Check for any limits you might have there especially on vmem.
<ccx>
But as I said it's wild guess, for something more concrete you'd want debug symbols in there.
<sy>
alrighty
<sy>
seems like a really cool software from what I've seen in oasis
<sy>
So I'm now getting ** Front-end died immediately after starting.
<sy>
** Sleeping for a bit before trying again...
<sy>
and it's just a gateway timeout
<sy>
alright well i guess i can't use grsec
<sy>
because of security
<sy>
which sounds ironic, but i can't use grsec because it disables unprivileged user unshare()
<isd>
sy: There is an option for a priviledged sandbox, which was added for systems that don't have user namespaces enabled.
<sy>
and run it as root?
<sy>
i'm guessing it does drop
<sy>
how do I do that?
<isd>
sy: it the standard install should set it up automatically if it detects that your kernel doesn't support user namespaces. Maybe they're enabled but blocked by grsec, so it's getting confused?
<isd>
There are also trade-offs here, regardless of just getting it working; sandstorm itself will be more secure if it's able to use the unprivileged sandbox, so the privileged sandbox might make sense if there's other stuff on the box that you don't want to have access to user namespaces, but I wouldn't recommend it for e.g. a dedicated sandstorm box.
<sy>
alright
<sy>
well this is a seedbox so there's absolutely nothing of value to steal
<sy>
nor anything major running
<sy>
i have nextcloud but that's getting rm'd if i like this :)
<sy>
because php is really *really* bad
<isd>
:P
<sy>
really nice to see you package ghost
<CcxWrk>
Grsec disables unprivileged user namespaces? Good thing I'm not the only one who considers them a security hazard with no actual useful use, besides workarounding badly-written software. :]
<sy>
that was always really painful for me
<sy>
hey nice
<sy>
the panel doesn't, like, suck, on mobile
pie_ has joined #sandstorm
<sy>
oh my god
<isd>
?
<CcxWrk>
isd: I can't figure out how is running it non-root is supposed to work. Looking at supervisor.c++ it does pivot_root and umounting original root, which to my knowledge requires CAP_SYS_ADMIN outside of the userns.
<isd>
CcxWrk: if you're using the unprivileged sandbox, it will have already moved into its own user namespace, inside of which it has CAP_SYS_ADMIN. but not for the parent namespace.
<isd>
(user namespaces do give you "root," but only inside of that namespace)
<CcxWrk>
I know, that's why I said the given operations require admin *outside* the userns.
<isd>
(which is why they're terrifying -- 20+ years of APIs written by people who thought they were only accessible by root, suddenly unprivileged.
<CcxWrk>
If I could umount filesystems from userns I'd consider that a valid design, but according to what I've gathered that's not supposed to be possible.
<CcxWrk>
(Sandstorm still can seccomp the offending APIs out, but it's problematic in general)
<isd>
The whole pivot root trick is really confusing and troubling to me, actually. I forget the details, but last I was picking through it, both the docs and my own attempt at trying to read the Linux source code for that sysall suggest that it should fail. I couldn't figure out why it wasn't just getting EINVAL
<CcxWrk>
I was getting failure with pivot_root executable. Same for umount.
<isd>
CcxWrk: I think it is legal, but maybe you have to also be in your own mount namespace (as well as user)? that seems like it would be a requirement for it to be sane.
<CcxWrk>
Is perhaps the sandstorm executable marked with CAP_SYS_ADMIN flag? Making it kinda sorta setuid?
<isd>
It is not.
<isd>
I think you're just mistaken about what the rules are.
<CcxWrk>
Anyway, it's dropping the uid either way so I'm not sure how should the userns version be safer.
<CcxWrk>
(specifically for dedicated sandstorm box)
<isd>
The binary retains the ability to setuid(0). You get some benefit from having a non-root effective uid in that the system won't just blindly let you do stuff, but ultimately it can still get root if it wants it.
<isd>
While on the topic of the whole privot root thing, kentonv, I don't understand why https://github.com/sandstorm-io/sandstorm/blob/master/src/sandstorm/supervisor.c%2B%2B#L1029 doesn't just throw EINVAL; pivot_root(2) says "put_old must be underneath new_root, that is, adding a nonzero number of /.. to the string pointed to by put_old must yield the same directory as new_root."
<isd>
Way back I even stared at the implementation of pivot_root(2) for a while and it seemed like it was checking for that. But it *does* work, so I'm very confused.
<isd>
Also, the whole thing really kinda sketches me out. A friend of mine is one of the authors of switch_root(8), and when I tried to pick his brain on it he told me "I stopped using pivot_root(2) when Al Viro told me he didn't understand it."
<CcxWrk>
That's what I'm getting. At least with /sbin/pivot_root
<CcxWrk>
Running as my regular user outside the userns and 0:0 inside.
<CcxWrk>
/sbin/capsh --print confirms having cap_sys_admin in the userns
<mokomull>
EINVAL is not EPERM.
pie_ has quit [Ping timeout: 256 seconds]
<mokomull>
The "new_root" of pivot_root must itself be a different mount than / currently is; if it's just an arbitrary subdirectory, you need to "mount --bind stuff /path/to/pwd" to make it one. But it's tough to chase down exactly why the kernel thinks something is EINVAL :(
<CcxWrk>
It is a separate mountpoint.
<mokomull>
Good luck then :) It usually takes me an hour to craft a pivot_root that actually works :P
<mokomull>
It usually involves a kernel debugger to figure out where the return -EINVAL comes from
<CcxWrk>
This is *so* messy.
<CcxWrk>
But I got it working somehow.
<mokomull>
:thumbsup:
<mokomull>
to both statements :)
<CcxWrk>
I wonder if grsec does something sensible there, like having a whitelist on who can use userns.
<sy>
isd: sy | oh my god -> this is really good
<sy>
i'm moving out of nextcloud
<CcxWrk>
I still can't umount the old filesystems after pivot_root. If that CWD magic from sandstorm works, this is way more evil code than I expected.
<CcxWrk>
Ah, it works when lazy. Okay then.
<sy>
can i combine grains?
<sy>
say i have a document in davros i want to edit in something else
<isd>
sy: there are mechanisms that apps can use to communicate in theory, but not many apps take advantage of this yet (and davros doesn't).
<isd>
This is what the powerbox is for, but not many apps make use of it
digitalcircuit has quit [Quit: Signing off from Quassel - see ya!]
<isd>
it was finally implemented basically right before sandstorm-the-company went out of business, and while I've written some demos that use it, I think some work needs to be done before it will be a super practical option.
digitalcircuit has joined #sandstorm
<sy>
alright
<isd>
The main thing is it's difficult to talk to while using the sandstorm-http-bridge compatibilty shim, and not doing the latter requires talking directly to sandstorm's web-session api, which (a) requires your language to have a capnproto rpc implemnetation, and (b) means either eschewing your languages standard pluggable web handler interface (Python's wsgi, ruby's rack, go's http.Handler, haskell's WAI, etc), and everything that buil
<isd>
s on that, or writing your own compatibilty bridge. or
<isd>
I've got a partial compatibility layer for Go's http.Handler. For a while it was one of those projects where I implemented the bits I needed for my apps as I went, eventually I decided I needed to sit down and build a version that was reasonably robust and complete. There's some work on that, but my focus lately has been on a capnproto library for Haskell, which has been coming along.
<isd>
I hope to get back to the Go project at some point (maybe I'll take a break from the hs thing for a bit soon), and once that wrapper lib is ready I may build a daemon on top of it that other apps can embed instead of sandstorm-http-bridge, so they can not deal with the web-session api for the most part, but still easily get at the powerbox stuff if they need to.
<isd>
There's a demo app that shows the kind of thing you can do.
<ocdtrekkie>
isd builds all the cool toys I wish I could use. :P
<isd>
We really need better dev tools. The APIs are conceptually reallly neat, but they're not terribly accessible.
<isd>
I need to get better at *finishing* the cool toys.
<ocdtrekkie>
Yes. And then we need to make them easy to add to apps, which is the harder part. :P
<ocdtrekkie>
Presumably, for example, with the file system bit, we could have some sort of vagrant-spk command to help someone mark a certain folder of their grain as being exposable, and then have it add the necessary package definition and dependent code to make that happen?
<isd>
I think it would make sense to build higher-level libraries for certain common protocols, so if you wanted to share a filesystem you wouldn't have to do much.
<isd>
I think the go http.Handler wrapper & daemon so it can be used by other languages is probably the most important first step it getting people able to use the powerbox
<isd>
(I looked at trying to tweak sandstorm-http-bridge to do what was needed a while back, but it turned out to be more complex than I thought, and if I'm doing the work this approach seems more fruitful)
<CcxWrk>
You could probably turn it into sandstorm-fastcgi-bridge or something like that, while passing the capnproto socket around as needed.
<CcxWrk>
But it wouldn't be very efficient.
<isd>
Given available Go libs, fastcgi support would be pretty trivial.
<CcxWrk>
I'm afraid you'd have to degrade it basically to just CGI to work with existing tools the way they are written.
<CcxWrk>
Alternative would be some unix socket passing magic that would make it easier to modify existing libraries. But too much magic for my tastes.
<isd>
My original thinking was to just do roughly what sandstorm-http-bridge does today: it speaks web-session and then basically acts as a reverse-proxy; the app itself listens on a port with an http server, which handles the bulk of the web stuff. We also listen on a socket with a capnproto connection and set a header in the requests that the app can use to get the context object for the current request.
<isd>
So if the app wants to speak some capnp api covering something other than http, it can connect to that socket, but it doesn't have to worry about web-session for the most part.
<isd>
the problem with sandstorm-http-bridge is that it doesn't expose the powerbox stuff in a usable way.
* CcxWrk
ponders
<isd>
The design of this is really not very difficult.
<CcxWrk>
I assume the issue is passing specific caps to specific HTTP request handlers. Or isn't?
<CcxWrk>
Hmm, you already have the header for that.
<CcxWrk>
What is the problem with proxying that in sandstorm-http-bridge?
<isd>
Basically the idea is that the app would connect to the capnp socket offered by the daemon, and there would that it could call, passing the id in the http header, to get the SessionData for that request.
<isd>
It could totally be done.
<isd>
The issue is just that (1) I'm not really a C++ programmer (2) there's a bunch of goop in the way that is used to request certain (http) apis, and when I looked at it reworking that was more than I was willing to take on.
<isd>
I had been working on the Go lib anyway, and once the lib is done, the daemon is trivial
<isd>
but it's largely a function of, if *I* am doing the work, this is going to be more tractable. I think if kentonv or someone else more comfy with the sandstorm-http-bridge code were to tackle it, it would be very tractable to add it to the existing bridge.
<CcxWrk>
I'm not really all that familiar with capnp, I guess the proxying is more complicated than in foolscap/E/monte where you'd just pass the reference around.
<isd>
CcxWrk: proxying capnp isn't complicated.
<isd>
but sandstorm-http-bridge already has a bunch of code that actually responds to the APIs that I'd want to get at, and figuring out how to shove that code aside was something I wasn't up to.
<isd>
It provides some limited capacity to talk to the power box to get http connections to things (but nothing else). And it wasn't immediately clear to me the best way to allow that to be bypassed.
<CcxWrk>
How does the UI for getting access from one grain to (part of) other looks? Same as the "add grain to collection"?
<isd>
Yeah. The collection grain uses the powerbox.
<CcxWrk>
How does publishing files (eg. in Hugo) work?
<CcxWrk>
For stuff that's supposed to be accessible without cookie, on static subdomain.
<isd>
There's a specialized static publishing mechanism; it's in the docs.
<isd>
It could in theory be just a powerbox api, but it pre-dates the powerbox by a fair bit.
pie_ has joined #sandstorm
<isd>
Hm, the lib is actually farther along than I remember. Major limitations: it doesn't yet support streaming request bodies or websockets, or the webdav methods. Maybe I'll tie up the other loose ends, write the daemon and release that, and then work on streaming as a next step
<isd>
I think I'd been focused on getting streaming post/put working with the "new" version because it was a requirement for some of the apps I wanted to write, but it would be sensible I think without that.
<ccx>
So I poked around some more and what I think would really help with sharing content across grains is to add arguments to the creation. Like etherpad(file) or roundcube(imap_credentials, smtp_credentials).
<ccx>
That way you wouldn't need to really edit the webapp but instead you could just map stuff into the grain container.
<isd>
I could see a mechnaism where apps could respond to powerbox requests (rather than just grains); doing so could automatically create a new grain.
<isd>
Not sure I totally follow what you're saying.
<ccx>
Hmm, so you mean like wiki edit button opening up powerbox where you'd select etherpad to edit collaboratively?
<isd>
Yeah.
<ccx>
That could work for things that'd need just one cap. Not sure if there are many cases where it'd be more.
<isd>
I suspect it would cover a lot of cases.
<isd>
Probably for multiple-cap cases you should probably just have the grain do powerbox requests for what it needs, maybe as part of a setup wizard that's part of the app.
<ccx>
Though I'd like to see a way to pass in things external to Sandstorm, such as IMAP accounts or Tahoe-LAFS hosted file or dir.
<isd>
You could do wrapper apps for that sort of thing. e.g. you write an app that requests a network connection to the Tahoe-LAFS server, and then fulfills powerbox requests for the filesystem protocol, using the tahoe server to back them.
<isd>
Apps can already request raw network access via the powerbox, though it's admin-only for obvious reasons. The way the API is designed allows requesting e.g. just a TLS connection to some location, though I don't know how finely grained sandstorm itself implements request handlers for that. But you could again do wrapper apps.
<ccx>
How would special capabilities for accessing outside stuff like that be handled? Some special admin UI?
<ccx>
I see.
<isd>
It's just a powerbox request, but the platform itself provides an option to fullfil it.
pie_ has quit [Read error: Connection reset by peer]
<ccx>
Well, what I think would be handy for not having port applications that much is sharing filesystem path or socket between the grains. Without the grain needing to always use capnp api to access it.
<isd>
Yeah. sandstorm-http-bridge provides some stuff for requesting http endpoints, so apps can do powerbox requests for http based apis without doing stuff with capnp. I think it can act as a newtork proxy as well?
<isd>
The filesystem is harder, and probably can't be implemented purely in an in-app compatibility daemon.
<isd>
Because it can't easily just spoof the local filesystem.
<ccx>
It would need to be handled by the supervisor code.
<ccx>
To add suitable bind-mounts.
<isd>
It might be interesting to investigate the security track record of the kernel's 9p implementation; I could see having some API call that the app (compat daemon) made that provides a Directory (https://github.com/zenhack/sandstorm-filesystem/blob/master/filesystem/filesystem.capnp#L58) interface, and then have the supervisor use that to back a 9p mount that's inside the grain.
<isd>
I guess you could do it via fuse, actually
<isd>
which for good reasons is blocked to the app itself, but you could enter the sandbox with a mountpoint for a supervisor-backed fuse filesystem.
<isd>
Again, would want to stare at the security track record for fuse.
<ccx>
That or run file synchronizers on a socket in both grains.
<isd>
Yeah, that would cover some use cases.
<ccx>
What is wrong with bind mounts between grains? That is the simplest solution.
<ccx>
Except for clustered blackrock or how is it called.
<isd>
That would require having some process that had access to both grain's storage, and could mount in them. So it seems like it would be expanding the privileges needed by the sandbox significantly.
<isd>
That *is* simpler.
<isd>
But it's also a bit more limited; my suggestion provides a general way for apps to do stuff with capnp filesystems without having to actually speak capnp.
<isd>
I'm not finding fuse CVEs that are exploitable without the ability to actually manage fuse filesystems (fusermount or /dev/fuse). So having a pre-mounted fuse filesystem fulfilled by the supervisor might actually be safe.
<ccx>
Unix file api, unfortunately, sucks heavily for any synthetic or non-local filesystem. I think at the point you are accessing something like that you really want to use something with pipelining that does not turn your processes into unkillable zombies.
<isd>
Like capnp? :P
<ccx>
Or 9p. Yeah.
<isd>
9p actually has pretty terrible latency problems.
<ccx>
plan9port stuff in particular would be quite interestin
<isd>
Unfortunately it doesn't do pipelining either, and while the plan9 filesystem API is a bit cleaner, it's still got the problem that it's fundamentally synchronous.
<ccx>
9p has latency problems when mounted as unix fs
<isd>
The protocol allows mutliple requests in flight, but not pipelining of requests with dependencies between them.
<isd>
I think there was a later system (osprey?) that tried to address that.
<ccx>
It does pipelining. You just won't use it by using Linux clients.
<ccx>
fids work for such pipelining, just read the 9p2000 spec
<ccx>
you can do walk/open/read in a single roundtrip
<isd>
Ah, you're right there.
<ccx>
It's considered one of the major features. Sadly due to the way threading libraries were used in plan9 it wasn't made really utilized much.
<isd>
There's still problems for basic use cases though; you can pipeline on fids, but not other info, so even just copying data from one file to another without latency problems is fiddly.
<isd>
If you want to know how much data you have to copy, you have to wait for the result of a stat.
<ccx>
Yeah, it's not perfect.
<isd>
You could I suppose just start issuing a bunch of read/write requests until you hit eof.
<isd>
But that's a bit kludgy
<isd>
and the data still has to go through you, so you're potentially needlessly copying data back and forth when you just want to copy a file within the same remote server.
<ccx>
Sure, sure, I know enough ocap systems to know of the potential there. :]
<isd>
:P
<isd>
The file system api I designed avoids both of these problems; copying a full file is just read(startAt = 0, amount = -1, write(startAt = 0))
<isd>
Assuming the proper level of protocol support, if the receivers of both read and write are remote capabilities, the data won't go through the caller.
<isd>
write returns a byte sink, rather than taking the data. read accepts a byte sink, rather than returning the data. So you just feed the latter into the former.
<isd>
The -1 passed to amount is a sentinel value that means "everything"
<isd>
But the whole point of the fuse thing is as a compatibility layer. You can't really talk about redesigining an API in that use case; the whole point is to provide an existing API.
<isd>
Well, I need to get going.
<ccx>
For apps that already operate on files I think it's best to avoid IPC altogether.
<ccx>
I think a file synchronizer based on the capnp api might also be worthwhile.
<isd>
Yeah, I would agree with that. Much simpler, covers a good number of use cases, and doesn't require any changes in sandstorm istelf.
<isd>
*itself.
<isd>
Anyhow. 'night all.
isd has quit [Quit: Leaving.]
<ccx>
Alternatively you could bundle subversion (ugh), which has arbitrary path ACLs and checkouts so it could work as one such synchronizer.