kentonv changed the topic of #sandstorm to: Welcome to #sandstorm: home of all things sandstorm.io. Say hi! | Have a question but no one is here? Try asking in the discussion group: https://groups.google.com/group/sandstorm-dev
xet7 has joined #sandstorm
wings has joined #sandstorm
wings has quit [Client Quit]
<abliss>
pty update: I spent a couple days staring at the gvisor pty code, trying to understand how it handles job control (e.g. translating ^C into SIGINT, ^Z to SIGSTP). Stared hard at the code, right where i thought it should be implemented, couldn't find it. Grepped all over. Turns out: it's not implemented at all yet. So, popping way back to the earlier discussion, where kentonv speculated that GCP might be using gvisor for
<abliss>
its web shell: it must not be (at least not the public version of gvisor), because those webshells (i'm pretty sure) do provide job control.
<abliss>
google left inside the container a socket pointing to the docker instance that runs on the external host. This dude stumbled across it, used it to break out of the container, and earned $100k
<abliss>
that is some pretty low-hanging fruit, which makes me think that there are probably other privilege-escalation bugs in Cloud SHell waiting to be found...
<isd>
So I'm starting to think about cgroups again.
<isd>
One snag with using them is that we can't really rely on their availability on an arbitrary linux system without asking the user to tweak some system config. I guess we could try to do this in the install script, but it would involve some complexity and I'm not sure how portably we could write that. And there would be the matter of upgrading existing installations.
<isd>
I suppose we could have it be the sort of thing where we use it if available, like the way we treat user namespaces.
xet7 has quit [Quit: Leaving]
xet7 has joined #sandstorm
<abliss>
sandstorm works without user namespaces? does it just require the sandstorm daemon to be run as root?
<kentonv>
abliss, yes, it supports two modes, one where it runs an root and one where it uses user namespaces
<kentonv>
we did that because a few distros disallowed unprivileged user namespaces for a long time (and maybe still do?)
<kentonv>
also wanted to support some ancient CentOS version that used kernel 3.10 where user namespaces didn't fully work yet
<JacobWeisz[m]>
I sent a typo PR this morning for node-capnp. Unrelated to the issue someone opened this morning, but found while browsing because of it.
<JacobWeisz[m]>
github.com/kenton is not kentonv :P
<kentonv>
funny that no one ever noticed that in 6 years of existence
<abliss>
yeah. (we don't know for sure it was there the whole time, but I bet it was). Reminiscent of the first android device, the Tmobile G1, that shipped with a root console listening to the physical keyboard.
<kentonv>
slightly less severe than that... :)
<abliss>
i dunno, that required physical access to exploit. running amok with root access on a google container host could attack sideways at other users in the cloud, right?
<JacobWeisz[m]>
It's the sort of thing you'd only hit if you explicitly copy pasted that line. If you were looking at the repo name or whatever, you'd not hit it. And since it moved to the capnproto org, most people probably would glance at that line and go "oh, that's the old location".
<JacobWeisz[m]>
But it's an invalid old location.
<JacobWeisz[m]>
He means this typo is slightly less severe than that. Especially since it just led to a 404.
<kentonv>
yeah I was talking about the readme typo
<kentonv>
which I guess Mr. Kenton Newby could have exploited maybe, but he didn't.
<abliss>
oh haha, yeah, sorry. i was still thinking of the cloud shell docker bug.
<kentonv>
ahh
<kentonv>
that bug I'd say is pretty embarrassing considering how many times the docker socket being exposed has led to security bugs in other systems... people should be looking for that
<kentonv>
also I guess this implies that they aren't running the shell inside a VM which is very surprising to me
<kentonv>
and yes, it's more severe than the G1 bug
frigginglorious has joined #sandstorm
frigginglorious1 has joined #sandstorm
frigginglorious has quit [Ping timeout: 246 seconds]
frigginglorious1 is now known as frigginglorious
<JacobWeisz[m]>
kentonv: I think you didn't sync 0.267
frigginglorious has quit [Read error: Connection reset by peer]
<kentonv>
fixed
<JacobWeisz[m]>
Woo. Probably unlikely to cause conflicts but likely nice to have updated. ;)
frigginglorious has joined #sandstorm
<abliss>
random thought of the day: would it be possible to implement the matrix protocol (for a sparsely-used server) atop google cloud functions / aws lambda / cloudflare workers? (is there any difference between them?) -- the one big problem I see right a way is that clients will do long-polling with hanging GET requests, and i will be uneconomical (or downright forbidden) to keep the backend VM alive the whole time.
<abliss>
Which makes me wonder, again, whether there could be some kind of middleware that can detect when a process is stuck waiting for IO (accept/read/poll/select), and freeze the whole thing to disk, and then later, thaw it all out (including restoring TCP connections) when IO becomes possible.
<kentonv>
definitely should not be pushing releases containing non-public code (unless maybe if it's a critical security fix and we want everyone to update before revealing)
<kentonv>
sorry about that
<kentonv>
abliss, CF Workers doesn't mind long polling (as long as you're OK with the occasional random cancellation)
<abliss>
do i get charged a per-hour for how long the worker stays resident?
<kentonv>
nope, workers is billed on request count only
<abliss>
oh, nice, looks like CF wokers are "0.50 per million requests, with a $5 monthly minimum"
<abliss>
so as long as i can keep the hanging get alive for at least 259 msec on average, i can be resident 24/7 for that $5 minimum
<abliss>
AWS lambda seems like GCF: charges for duration, but with a free tier. i wonder why CFW pricing is so different, and whether it'll eventually converge? I guess the backend implementation model is quite different to goog/amazon, and maybe CF itself is a fundamentally different kind of company?
<kentonv>
our implementation is massively more efficient than Lambda and GCF
<kentonv>
we don't allocate a VM to run your function. We allocate a V8 isolate. Overhead is very low.
<kentonv>
Lambda basically reserves a minimum of 128MB of RAM on one of their machines for each concurrent request. We don't reserve anything, we just allocate a V8 isolate wherever the request landed and run it. A typical isolate takes a couple megs of RAM.
<kentonv>
and we let one isolate handle multiple concurrent requests
<abliss>
and storage/ingress/egress/cpu/ram is all free/unmetered?
<abliss>
that $5/mo seems like it might be competitive with matrix's own pay-for-hosting, Modular ($1.50 / active-user / month)
<abliss>
if you have around 4-50 users. which describes most chat backends, i'd guess
<isd>
I mean, you can get a vps for around that.
<isd>
...and not have to write a custom matrix server to run on it...
<kentonv>
abliss, keep in mind CPU time per request is limited to 50ms, and RAM per instance is limited to 128MB
<isd>
Unless your workload is very bursty or having to manage the OS is a big burden I think a VPS is probably a better fit.
<kentonv>
FWIW I don't think you could write a chat app entirely on Workers today... there's no way to coordinate state between requests
<kentonv>
(but I'm working on it...)
<abliss>
there's no r/w persistent state at all? i guess i could use CF workers as the frontend to hold open the hanging gets, and do all the actual logic in google/amazon
<kentonv>
there's an eventually-consistent KV store, but since it's eventually-consistent you can't use it for coordination...
<abliss>
is there really no limit /metering on how much i store in the KV or how often i access it? what's a realistic propagation time for a write?
<kentonv>
you are charged for KV operations and KV storage
<kentonv>
propagation time can be several seconds. More importantly, there's no way to present concurrent writes from clobbering each other.
<abliss>
i could manually shard keys to prevent concurrent writes ... or either fetch out to a lockserver somewhere or figure out how to elect a leader internally. if one worker can poll the KV every few hundred MS and block until it detects a change, then another worker could write to that key to trigger an event. several-second latency might be acceptable for a chat app...
<abliss>
why the $5 minimum, i wonder? is there a forum somewhere where 5 people who each have $1/month of workload can meet up and split costs?
<JacobWeisz[m]>
There's probably some point where it's silly to charge a credit card for something. Arguably every account has a support cost burden of "this guy might have questions" too.
<kentonv>
We don't like billing people less than $5 at a time. The payment fees become excessive.
<JacobWeisz[m]>
I wonder if you could get around that with a rechargeable account system like my toll pass...
<kentonv>
Trying to synchronize on KV will lead to tears and hair loss, trust me.
<abliss>
why not offer a no-support $5/yr prepaid level?
<kentonv>
we want to work on the product, not on the billing. :)
<JacobWeisz[m]>
I still find it hilarious that AWS bills me like 79 cents a month or something. They probably get nothing.
<abliss>
makes sense :)
<kentonv>
that said there may be some changes in the works...
sknebel has quit [Ping timeout: 240 seconds]
<abliss>
i don't suppose there's some secret header i can send to force my request to be routed to a particular worker?
sknebel has joined #sandstorm
<abliss>
if I can manage to force two requests to hit the same ipv4 address, is it guaranteed (or at least likely) that they'll be routed to the same workers (or workers that are close-by, as measured in 99pctile KV write-propagation time)?
<kentonv>
nope, not in the slightest
<kentonv>
all our colos advertise the same addresses.
<kentonv>
so a request to some IP could land on absolutely any machine in our fleet
<abliss>
i see 3 different ipv4 addys when resolving my playground worker... it's the same 3 for all of them?
<kentonv>
we have a lot of addresses, but your hostname will resolve to the same addresses no matter where you are in the world, and those same addresses are advertised by all of our locations
<abliss>
gotcha, that's what i figured
<kentonv>
it's intentional that you can't send a request to a specific location since we want DDoSes to be spread out. :)
<abliss>
if a worker tries to fetch its own url, will it be routed back to itself?
<kentonv>
no, it will be routed to your back-end.
<kentonv>
the thing you're trying to do here is exactly the thing that I'm building a whole system to do so I'm pretty sure you can't do it yet. :)
<abliss>
aw, no free tier for the KV. guess i'll pony up the $5 just to play around with it for a month
<abliss>
i don't doubt you at all. but it sounds like a (doomed!) attempt to prove you wrong would be fun and educational for me :)
<kentonv>
yeah unfortunately we kind of have to charge for KV because we use third-party storage backends that charge us...
<abliss>
(and my alternative is to go try to contribute job-control to gvisor's pty impl, which sounds kinda dull and i'd rather wait a bit to see if they're about to implement it themselves :)
<abliss>
(alternative in the sense of "my other current for-fun project", not that the functionality is at all related)
<kentonv>
just don't burn yourself out on Workers right before I go and ship the thing that makes your project super-easy... :)
<abliss>
oh, the playground doesn't let one bind a KV namespace anyway, so now i'll have to take the plunge and actually install the tooling...
<kentonv>
it does actually.
<kentonv>
assuming you made an account and you're on dash.cloudflare.com (not cloudflareworkers.com)
<abliss>
oh! yeah i guess i had to do a hard refresh after buying the plan.
<kentonv>
you can back out of the editor and then look at your worker, and there's a place where you can add bindings
<kentonv>
ohhh, I see
<kentonv>
I, uh, have a little bit of a debate with my team, where they really want to just delete the online editor and tell people to use the tooling because writing code online is Bad Software Engineering and we shouldn't let people do it, or something
<abliss>
thanks for chatting about it. i'm gonna go have a think about what i could do with this architecture. this ends my hijacking of the sandstorm channel. :)
<abliss>
(the online editor is crap for Software Engineering, but awesome for fast-onramp fooling around!)
<kentonv>
yeah I keep trying to explain that...
<kentonv>
also for trivial things like my e-mail catch-all implementation from last friday
<kentonv>
sometimes you just need a really simple HTTP endpoint
kentonv has quit [Ping timeout: 260 seconds]
kentonv has joined #sandstorm
frigginglorious1 has joined #sandstorm
frigginglorious has quit [Ping timeout: 246 seconds]