<afuentes>
haha, i did google the problem, but only skimmed the faq there... i admit it :(
<asheesh>
On storm.debian.net I see this: "
<asheesh>
Error: Requested session that no longer exists, and timed out waiting for client to restore it. This can happen if you have opened an app's content in a new window and then closed it in the UI. If you see this error *inside* the Sandstorm UI, please report a bug and describe the circumstances of the error. [504] at server/proxy.js:582:24"
<asheesh>
in the logs
<asheesh>
Plus there's a grain or two that seems to never really start.
<dwrensha>
afuentes: if you generate a link through the "Share Access" button or through the admin "/Admin/Users/Invite" interface, then the link will carry authority and you wouldn't want to share it here
<asheesh>
On the server side, it does seem from 'ps auxwwww' that 'node //node_modules/ep_etherpad-lite/node/server.js' is running.
<asheesh>
The fact that I see a session-related error here is pretty odd.
<dwrensha>
asheesh: I'm seeing a spinner
<asheesh>
Yeah, exactly :|
<asheesh>
That's the symptoms I mean when I say "seems to never really start"
<asheesh>
After 60 sec nginx eventually says 504 Gateway Timeout, I believe. Note that _sandstorm-init seems to load fine; it's requests to the grain that are failing.
<asheesh>
well hanging forever
<asheesh>
I guess I can try to figure out if the grain is truly running by finding out what the grain ID is, one sec.
<asheesh>
... looks like that grain has been online since June 20.
<asheesh>
based on its supervisor being created back then.
<asheesh>
I could go kill the supervisor, which would presumably fix the problem....
<dwrensha>
is this a grain you own?
<asheesh>
No, alas
<asheesh>
I am the sysadmin though!
<dwrensha>
I suppose that doesn't matter if you have shell access
<dwrensha>
I'm curious if there are any errors in the grain log
<asheesh>
I can go read its log, yeah! THat's what just occurred to me .one sec. Pardon typos
jemc has quit [Quit: WeeChat 1.4]
jemc has joined #sandstorm
<asheesh>
Lots of "Grain still in use, staying up for now" but also this:
<asheesh>
ESC[32m[2016-06-21 14:27:46.840] [INFO] access - ESC[39m[LEAVE] Pad "main": Author "a.iolhVQBjwC1dEBY0" on client tKy9H6KGFiv2yHf-AAAA with IP "127.0.0.1" left the pad
<asheesh>
Is it possible that requests aren't getting routed to the grain, and it's some kind of Sandstorm problem where if you keep a grain alive over N Sandstorm restarts or across specific versions, then the grain becomes unreachable?
<dwrensha>
asheesh: maybe try attaching strace to the supervisor?
<asheesh>
Similarly, worst-case, it seems that Sandstorm should detect that the HTTP response is taking >5 minutes for the grain to respond-to, and kill it.
<asheesh>
Smart; I'll do that
<dwrensha>
yeah, I'm curious how far these requests are getting
<asheesh>
Now that's weird, I think, right? Why is inotify getting involved?
<asheesh>
I'll update the texteditor anyway.
<asheesh>
afuentes: Yes!
<afuentes>
i,i im impressed with sandstorm tbh... it used to look good, but now its amazing!
<asheesh>
It's a weird generic term because it's a user-facing term as well, and we want something generic enough to mean "Document" or "Chat app" etc.
<asheesh>
I must AFK but will be back in a little bit.
<dwrensha>
asheesh: the supervisor uses inotify to track disk usage
<asheesh>
BTW it's hilarious that the supervisor does a write() to the log, then inotify wakes it up to say the log has changed (-:
<asheesh>
afuentes: Great! Glad you are excited about it!
jacksingleton has joined #sandstorm
<dwrensha>
asheesh: I assume you attempted to make some requests to the grain during this strace capture?
<afuentes>
okay... now i have a problem, because this need wildcard ssl certificate and certbot does not provide that kind of certs :(
<afuentes>
its a shame, because I have a reverse proxy... meaning its making all the conections with sandstorm locally
<afuentes>
but i guess the websockets try to access it directly or something
<dwrensha>
Yeah, it's too bad that Let's Encrypt does not support wildcard certificates yet
<afuentes>
thinking about it, i dont think i can use some local non https redirection
<afuentes>
because the websocket is trying to access some https arbitrary domain and my apache needs to be able to offer that cert :/
<afuentes>
is there a list of these arbitrary subdomains it tries to access?
jacksingleton has quit [Ping timeout: 240 seconds]
<afuentes>
the domains doesnt even show on the browser console, do they?
<afuentes>
it just show error accessing wss://
<afuentes>
doh "...but actually creates a new origin for every session"
<afuentes>
yay for security i guess! :)
<afuentes>
this doesnt look its going to have an easy work around... im off to take a shower... where my most brilliant ideas usually happen :)
<dwrensha>
the only solution I know of are: 1. buy yourself a wildcard cert, or 2. use sandcats
mnutt has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
<afuentes>
during my shower I had a couple of "genius" ideas. 1. make my apache offer the root domain cert for arbitrary domains or 2. offer a self-signed certificated
<asheesh>
Yeah dwrensha I did
<afuentes>
lets see how firefox handle wrong domain certificates websockets connections ^^
<afuentes>
Maybe If I can click acept this certificate once and be done with
<asheesh>
Also: '''Web browsers do not show a "OK to continue?" prompt for IFRAMEs, and Sandstorm embeds IFRAMEs to subdomains of its main domain, so there is no warning that users can click through'''
<asheesh>
Exception in setTimeout callback: TypeError: Cannot call method 'stop' of undefined at server/proxy.js:581:26 at [object Object]._.extend.withValue (packages/meteor/dynamics_nodejs.js:56:1) at packages/meteor/timers.js:6:1 at runWithEnvironment (packages/meteor/dynamics_nodejs.js:110:1)
<asheesh>
Also, the first "error: Backend hasn't responded in 30 seconds! Reconnecting." is before the "capnp.js: ..." printouts, but the second one ("error: Backend ping threw error! Error: remote exception: Peer disconnected.") is after them.
<asheesh>
I will share the actual log file, one sec.
<asheesh>
Man I wish we used syslog.
mnutt has joined #sandstorm
<afuentes>
where should i put my self signed certificates? /etc/ngingx seems to be inside every app under var/sandstorm/apps/<id>
<afuentes>
you are right... how dumb i am... I have to serve it myself on apache on my reverse proxy
<asheesh>
np you are clearly pretty smart!
<afuentes>
you can tell, coz I im using sandstorm :D
<asheesh>
dwrensha: I figure it makes sense to file a bug for easier debugging.
Lionel_Debroux_ has joined #sandstorm
<dwrensha>
heh, log timestamps would be nice for a situation like this
afuentes has quit [Ping timeout: 258 seconds]
<dwrensha>
"SandstormPermissions.grainPermissions() took 33503 milliseconds to complete for the vertex ..." is troubling
jacksingleton has joined #sandstorm
Telesight has joined #sandstorm
<dwrensha>
asheesh: what happened to the strace grain? it looks empty now
saneki_ has quit [Ping timeout: 276 seconds]
saneki has joined #sandstorm
xet7 has quit [Quit: Leaving]
c-mart_ has quit [Ping timeout: 240 seconds]
<zarvox>
all the log noise is from Capnp.enableVerboseDebugLogging(true) getting set, which happens when the ping to the sandstorm backend doesn't respond for 30 seconds
<zarvox>
it's possible that an overly-long blocking permissions computation prevented us from processing the ping reply in time to think the backend was alive
jadewang has joined #sandstorm
keturn has quit [Ping timeout: 240 seconds]
Daemoen has quit [Ping timeout: 252 seconds]
Daemoen has joined #sandstorm
<dwrensha>
I'm thinking the long permissions computation was a symptom, but not the root problem
<dwrensha>
gwah, cloudflare 502 again for oasis
keturn has joined #sandstorm
c-mart_ has joined #sandstorm
jadewang has quit [Remote host closed the connection]
<zarvox>
dwrensha: asheesh: It's worth noting most of that logfile is quite old; you can tell by the "Migrations already applied" when upgrades were applied
jadewang has joined #sandstorm
jacksingleton has quit [Ping timeout: 244 seconds]
isd has joined #sandstorm
frankier has quit [Ping timeout: 240 seconds]
mnutt has quit [*.net *.split]
jemc has quit [*.net *.split]
papa has quit [*.net *.split]
gelnior54 has quit [*.net *.split]
[d__d] has quit [*.net *.split]
tg has quit [*.net *.split]
joeyh has quit [*.net *.split]
dograt has quit [*.net *.split]
CaptainCalliope has quit [*.net *.split]
KevinMGranger has quit [*.net *.split]
_iwc has quit [*.net *.split]
ckocagil has quit [*.net *.split]
KevinMGranger has joined #sandstorm
papa has joined #sandstorm
jemc has joined #sandstorm
tg has joined #sandstorm
ckocagil has joined #sandstorm
joeyh has joined #sandstorm
dograt has joined #sandstorm
fkautz has quit [Ping timeout: 258 seconds]
CaptainCalliope has joined #sandstorm
[d__d] has joined #sandstorm
_iwc has joined #sandstorm
esmiurium has joined #sandstorm
_iwc has quit [K-Lined]
fkautz has joined #sandstorm
digitalcircuit has quit [Ping timeout: 250 seconds]
digitalcircuit has joined #sandstorm
Telesight has quit [Quit: Leaving.]
jacksingleton has joined #sandstorm
c-mart_ has quit [Ping timeout: 258 seconds]
c-mart has joined #sandstorm
c-mart is now known as c-mart_
frigginglorious has quit [Quit: frigginglorious]
wolcen has quit [Ping timeout: 250 seconds]
xet7 has joined #sandstorm
frigginglorious has joined #sandstorm
wolcen has joined #sandstorm
isd has quit [Ping timeout: 250 seconds]
afuentes has joined #sandstorm
c-mart_ has quit [Quit: c-mart_]
mnutt has joined #sandstorm
mnutt has quit [Quit: My Mac has gone to sleep. ZZZzzz…]