<anthropy>
hey all, I have a weird issue with my sandstorm install.. the sandcats SSL certificate expired randomly and I can't figure out what broke to cause it stopping renewals
funwhilelost has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
harish has quit [Ping timeout: 276 seconds]
wolcen has joined #sandstorm
<kentonv>
anthropy, what's your sandcats hostname? (Feel free to PM me if you don't want to say it publicly.)
<anthropy>
I /msg'd it to you but I'm using pidgin and I'm not sure it arrived :P
<kentonv>
yep, got it
<kentonv>
it looks to me like your server did not request a new cert when expected. Other servers are still requesting and receiving just fine, so this is odd.
<kentonv>
is there anything in the system log? See /admin/status
<anthropy>
yea, I'm having some idea about why though
<anthropy>
for some reason the virtual harddisk got corrupted the other day, I shut the machine down, fixed it with qemu-img, and then let btrfs run a scrub which fixed most of the errors, but not everything
<anthropy>
heh yep, basically this message repeated:
<anthropy>
I'm thinking of just redoing the machine from scratch with another OS though.. (open to suggestions). Is there an easy way to migrate sandstorm to a new server?
<kentonv>
yeah that'd do it. Although that specific error is not from certificate renewal, it indicates that DNS isn't working.
<kentonv>
(there's supposed to be a symlink in /usr/local/bin but I guess not for you)
neynah has joined #sandstorm
<anthropy>
ohey that seemed to fix the certificate
<kentonv>
heh
<kentonv>
so I think what happened is
<kentonv>
Sandstorm copies your DNS config at startup, in order to make it available inside the self-container
<kentonv>
however, if your DNS config changed after Sandstorm started, then Sandstorm would be running with a broken config
<kentonv>
so a restart fixed it and then it was able to get a new cert
<anthropy>
perhaps the system fixed itself after bootup, there /was/ a corrupt dns config and now there isn't anymore
<anthropy>
or something
<anthropy>
anyway neat
<anthropy>
"WARNING: This server seems to be unable to create a WebSocket. " hmm
<kentonv>
are you using an HTTP proxy?
<kentonv>
(like is your browser configured to go through a proxy)
<anthropy>
kinda maybe yes though it's a SOCKS5 proxy (SSH)
<kentonv>
usually WebSocket problems aren't actually the server's fault but something about your browser or your network
<anthropy>
I'll try some things
<anthropy>
is there a way to test it without logging in?
<kentonv>
a socks proxy ought to work, though... it's HTTP proxies that would typically cause trouble.
<kentonv>
unless your socks proxy is doing deep-packet inspection somewhere
<anthropy>
nah
<anthropy>
I do have a HTTPS proxy running on my webserver to route it to the VM based on hostname, but that doesn't touch the traffic either, it uses the TLS SNI tag and is completely transparent otherwise
<kentonv>
hmm, when I open your sandcats domain I do not see Sandstorm.
<kentonv>
I just see "More interesting stuff up here soon."
<anthropy>
be sure to use https, http points to the nginx of the frontend webserver right now which doesn't redirect yet l3;
<kentonv>
oh right
<kentonv>
so you can check this in the JS console: Meteor.connection._stream.socket.protocol
<kentonv>
if websockets are working, it should be the string "websocket"
<kentonv>
seems to be working for me, on your server
<kentonv>
so I think it's your browser or network.
<kentonv>
oh, one possibility
<kentonv>
sometimes the browser will decide, after failing to connect WebSockets once or twice, that the host simply doesn't support them at all. And then it will stop allowing new ones.
<kentonv>
maybe it decided this while the certificate was broken
<kentonv>
I usually find that restarting the browser (full exit, then restart, not just closing the tab) solves this
<kentonv>
FWIW most things work fine without WebSocket, though a few apps will break
aldeka_limechat has quit [Remote host closed the connection]
<anthropy>
it seems to be working now
<anthropy>
neat, I hope the server keeps working, it'd be nice not having to setup a new server for no real reason
<asheesh>
+1
<asheesh>
I just showed up and read scrollback and yeah, +1 to understanding and fixing problems. I'm glad that approach worked!
<anthropy>
is fedora a decent OS to run sandstorm on?
<anthropy>
(and BTRFS a decent filesystem for sandstorm? or are there known issues?)
aldeka_limechat has joined #sandstorm
<asheesh>
Totally reasonable; no known issues to me, at least.
<anthropy>
yay! then I hope nothing will fall over despite that corruption issue at one point
<asheesh>
That's the spirit with btrfs, I feel/fear.
<anthropy>
I'm making nightly snapshots with ZFS now so it should be easier to fix next time
<asheesh>
I had a sad time with it in 2011 but it's not 2011 anymore.
<asheesh>
Great.
<anthropy>
still no idea what caused it in the first place though, hardware is full ECC (xeon + ddr4 ecc), using ZFS RAID1+0 with weekly scrubs (was daily).. guess sometimes applications just feel like being a pain in the dick
<strugee>
heya
<strugee>
so I have a Sandstorm install - strugee.sandcats.io - which (according to the admin panel) has a misconfigured WILDCARD_HOST
<strugee>
I don't see anything useful other than that in the console (which may be because I'm on Firefox Nightly and have a newer/different console) but the Network pane shows all 200 responses from the main domain along with a bunch of requests to "static.strugee.sandcats.io"... ah
<strugee>
^^^ I proposed Sandstorm's inclusion in prism-break.org
<strugee>
kentonv: ping
afuentes has quit [Ping timeout: 248 seconds]
aldeka_limechat has joined #sandstorm
ill_logic has joined #sandstorm
aldeka_limechat has quit [Ping timeout: 240 seconds]
ill_logic has quit [Ping timeout: 276 seconds]
demonimin has quit [Remote host closed the connection]
demonimin has joined #sandstorm
intelligence has joined #sandstorm
intelligence has quit [Client Quit]
<anthropy>
Is there a way to redownload and install an app in sandstorm? the host suffered some corruption and the Davros app is giving vague errors
IQFred has joined #sandstorm
<IQFred>
hi!
<IQFred>
I'm evaluating Sandstorm for a project that my company is working on and I wonder if anyone here has experience with Sandstorm deploying on mobile devices?
<anthropy>
I've tried most apps on mobile devices and they seem to perform well
<anthropy>
Some apps don't work with the native android/iOS apps though, such as Rocketchat doesn't appear to be working with the Android Rocketchat app
<anthropy>
but the web parts all seem to work well
<anthropy>
even the little MMORPG thingy :P
<IQFred>
I guess the reason I ask is that I created two demo accounts on Oasis and I'm testing them on iOS and Android and the web interface is really clunky
<IQFred>
The biggest issue I noticed is that the viewport doesn't scroll in the main menu.
<IQFred>
So if you have a bunch of grains "open" and the list scrolls past the end of the viewport,... it doesn't scroll at all. There's no way to reach it
<IQFred>
(btw, didn't they just fix the rocket chat client on Android? I thought I saw that ticket closed)
<anthropy>
yea it has some.. usability needs, though it worked well for me whenever I have to use an app on the go. I can work around the bug by going back to selecting an app from the grains list
<anthropy>
really? I'll check that out right now then
<IQFred>
Oh, absolutely. It's technically usable.... just not really good UX
<IQFred>
Are you part of the development team or a user?
<anthropy>
I'm a user, I have a few sandstorm setups running
<anthropy>
and hey you're right, the app works now! Sweet
<jemc>
I'm having trouble with developing a vagrant-spk packaged meteor grain
<IQFred>
Right :) I saw it in the development thread over on GitHub
<jemc>
I'm getting the infinite loading spinner when I try to view the grain
<neynah>
Yeah the current state of our ux on mobile is pretty unspectacular... If you'd like to file an issue for the scrolling issue we can make an effort to fix it. :)
<jemc>
is there any way to see stdout/stderr from the `launcher.sh` that is being run?
<jemc>
I know I can use `vagrant-spk enter-grain` to execute commands in the grain, but I'd like to see the output of the command that sandstorm itself is executing
<IQFred>
Hey Neynah, I'll do that... might take me a few days because I'll need to document it properly so you can see the use cases where the issue is present.
<jemc>
is there a log where this would be written to, or a way to attach to the stdout/stderr?
<IQFred>
Thanks all :)
aldeka_limechat has joined #sandstorm
aldeka_limechat has quit [Ping timeout: 248 seconds]
<anthropy>
hmm, I tried restarting sandstorm and then reinstalling the app after removing all the above mentioned entries, but now Davros just shows up as '<Unknown>' after installation. Installation from the market seems to work, but after that that happens
<kentonv>
let me try doing the same steps here and see what happens
<anthropy>
Davros used to work before the corruption event, just afterwards I never got it working again
<kentonv>
huh, that File::open() error appears for me too, guess I never noticed
<anthropy>
I intend to use it to save my scanned bills and mail so I have it together with all my other stuff inside sandstorm, would be neat to have davros working :3
<kentonv>
ok I've reproduced the same error here. Interesting, apparently I'm forgetting some place that is caching some metadata or something. Lemme see here...
<kentonv>
oh
<kentonv>
you'll need to do a "sandstorm restart"
<kentonv>
because it caches some package info in-memory
<kentonv>
after that you should be able to reinstall davros
<anthropy>
I did that actually
<anthropy>
but I'll try it again
<kentonv>
maybe repeat all the steps again
<kentonv>
including clicking "uninstall" in the UI, which I forgot before, but I think you figured out
<anthropy>
oh, it works now!
<anthropy>
neat
<kentonv>
great!
<anthropy>
now to see if it starts without errors
<anthropy>
yep! it's all working now :)
<anthropy>
thanks for the help! I'll definitely add those two purging commands to my personal knowledge base
<kentonv>
we should probably add a "reinstall app" feature for admins
<anthropy>
yea it would be great for recovery scenarios
<strugee>
I guess Travis doesn't work for some usecase Sandstorm has?
<strugee>
couldn't you use e.g. Vagrant to build PRs in a virtualized environment?
* strugee
is just curious
<strugee>
like to be clear "it's too much work" would be a very valid answer IMHO :P
<maurer>
strugee: Sandstorm requires certain kernel features to properly test. These are certainly not present in travis's containerized builds, and I am unsure about their VM builds
<strugee>
ohhhhhh right
<strugee>
the newest you can get is Ubuntu Precise IIRC
<strugee>
which I'm guessing is probably too old
<maurer>
I would expect sandstorm works on precise, but I cannot say for sure (I'm not an official sandstorm person0
<maurer>
My point is basically "They do stuff with new namespace features, it makes sense that they would want their own testing environment"
<kentonv>
Travis almost certainly blocks some of the syscalls we need to test Sandstorm
<kentonv>
if they don't then they have a security problem. :)
<maurer>
kentonv: I mean, it's possible their non-container version wouldn't block the syscalls?
<kentonv>
oh, I suppose yeah
<kentonv>
if it's VM-based
<maurer>
Yeah, they have 3 stacks
<maurer>
default is container based, which will certainly not work
<maurer>
they have a VM based stack, whose kernel may or may not be too old
<maurer>
and they have an OSX stack, which is not relevant to sandstorm
<kentonv>
Precise is too old though, if that's really the newest they support.
<maurer>
Newest is 14.04
<maurer>
which is quite old
<kentonv>
14.04 is good enough
<maurer>
they really want people to switch to container-based, it's much cheaper fo rthem to run
<kentonv>
yeah I would imagine so
<maurer>
Yeah, if you feel like playing with it in your copious free time, the magic words are "dist: trusty" and "sudo: required"
<maurer>
to get onto a trusty vm
<kentonv>
yeah, our Jenkins VM seems to be working fine, so... nah. :)
<kentonv>
someday Sandstorm itself needs a VM API, in order to be able to put our tests on our own infrastructure...
<maurer>
Yeah, there are definite plusses to using jenkins
samba_ has joined #sandstorm
<jemc>
just a quick bump for my earlier question - is there a way to see the stdout/stderr of `launcher.sh` for a `vagrant-spk dev` grain that is failing to start (infinite loading spinner)?
<kentonv>
jemc, click the debug log button in the top bar in the UI (icon looks like a monitor)
samba_ has quit [Client Quit]
<kentonv>
just had an idea: we should automatically display the debug log in a bottom panel when an app is in dev mode
<jemc>
ah, that sounds like it's a lot simpler than I was making it...