kentonv changed the topic of #sandstorm to: Welcome to #sandstorm: home of all things sandstorm.io. Say hi! | Have a question but no one is here? Try asking in the discussion group: https://groups.google.com/group/sandstorm-dev
<isd>
Re: mastodon, I'm told Pleroma is much more resource friendly, so it may make sense to investigate that first.
<isd>
I'm thinking now a bit about composing stacks. With docker-spk, it would be easy enough to compose them linearly, but it gets clumsy if you want to pull in multple "stack dependencies" -- i.e. maybe both postgres and redis or such (botbot used both...).
<isd>
I think nix might be a particularly good platform for trying to build composable templates like that.
<isd>
Also, ill_logic, I posted a thing about postgres in the wekan thread. tl;dr: way back I tried to port botbot.me to sandstorm, and as part of that set up postgres. buyer beware, I don't remember how far I got. repo: https://github.com/zenhack/botbot-sandstorm
<isd>
I think wekan cards might not be a great place for discussion, esp. since the notification support isn't really there.
<abliss>
agreed, no discussion in wekan cards
<ocdtrekkie>
Composable stacks would be really nice. But it'd be a major redesign of how they're built.
<ocdtrekkie>
You'd presumably need like pieces of each setup.sh, launcher.sh, etc. for the database software, environment, etc. and then have it build them? In theory, like, if you assume you should setup the database and run it first, I think it'd mostly usually work?
<ocdtrekkie>
But it's not like we have a huge number of stacks, we can afford to just build some partially redundant ones. But if we get Postgres working in Sandstorm, we should have it clearly denoted a way that works.
<ocdtrekkie>
And vagrant-spk stacks are a handy way to do that.
<isd>
Yeah, I'm more thinking long term. I agree in the short term doing the simple thing with vagrant or docker is a good approach
<isd>
I guess what I had in mind would probably work with other distros as well, but we could build wrapper packages that integrate a sandstorm-friendly setup of these components, so you'd just start with the base vm/container , install the "sandstorm postgres" package or the like, and we'd have the fiddly bits of setting things up wrapped in some script. Probably with nix you could just add 'postgres' to a list variable we define
<isd>
somewhere and be done with it.
<ill_logic>
isd abliss: Do you not like comments at all? Like I was thinking of maybe putting progress details in there. I sort of have already.
<abliss>
oh, sure, progress comments are fine (one-way publishing data) but not discussion (asking questions)
<ill_logic>
isd thanks for the project! I'll take a look.
<ill_logic>
project link*
<isd>
Yeah, my gripe was just that it's easy to lose track of conversations if there aren't notifications.
<isd>
you're welcome
<ill_logic>
Short of going to Nix, which I like the idea of (not that I know it, but it would be a great excuse to learn it),
<ill_logic>
What if the stacks were sort of split up into commands like "setup_mysql.sh", "setup_nginx.sh", "launch_nginx.sh" or what have you.
<ill_logic>
And then if you want a simple stack, setup.sh would just call those two scripts.
<isd>
Yeah, that was kinda what I was thinking.
<ill_logic>
But if you have overlapping concerns, you could call whichever ones you want.
<ill_logic>
Er, if you want something less generic. (not sure why I said "overlapping concerns")
<isd>
I think that's a pretty reasonable approach.
<ill_logic>
That's assuming the various setups and launchers can be totally independent like that.
<isd>
Yeah. I think a lot of them probably can be though.
<ocdtr_web>
(Though as an aside, it's possible SandMD might perform differently, revisiting why there is not a clear upgrade path, SandMD used SQLite, while CodiMD uses MySQL, as is recommended/preferred upstream.)
<ocdtr_web>
This might factor into the overall discussion about how we should be able to mark apps as deprecated or discouraged from use in the market, without requiring a resubmission. We want SandMD to be available for people restoring or mass transferring or whatever, but it should pretty much be buried as an app market option.
frigginglorious1 has joined #sandstorm
frigginglorious has quit [Ping timeout: 268 seconds]
frigginglorious1 is now known as frigginglorious
_whitelogger has joined #sandstorm
ill_logic has joined #sandstorm
<isd>
I think we only went for SandMD because Lyre suggested it; I was unaware of the newer thing.
<ocdtr_web>
The unfortunate bit is that SandMD appears higher on the default view of the apps list.
* isd
takes CodiMD for a spin
<isd>
unpacking the app is taking a very long time...
<isd>
...and a stupidly long time to start up, at least on first start.
<ocdtr_web>
That's probably the MySQL effect.
<ocdtr_web>
SandMD new grain is 92 KB, CodiMD new grain is 48 MB.
<isd>
These apps are basically identical. I'm not sure we actually should push CodiMD first.
<ocdtr_web>
The argued claim is for security fixes largely (they're both from PDIS), and the switch to MySQL is because CodiMD basically says they're not going to support any database upgrade/migration from anything but MySQL or Postgres.
<isd>
can codi still use sqlite?
<isd>
Ah.
<ocdtr_web>
So yeah, it could still use SQLite but it may impact future upgradability to do so.
<isd>
(I suppose that's fair; SQLite's migration support is a bit lacking)
<isd>
postgres has some really nice features. I wish there were more options for lightweight databases; there's stuff like LMDB, but if you want any kind of non-trivial query support you've basically got sqlite and that's it. sqlite is pretty nice in a lot of ways, but...
<isd>
I hate electron so much.
<ocdtr_web>
"Hi, we installed an extra web browser for each website you go to"
<isd>
I mean, at least with distro packages you've only got one copy of election itself.
<isd>
But they couldn't have just used a lightweight webkit? They really needed to include all of chrome?
<ocdtr_web>
I wish the world would just call Chrome adware, mark it for removal in security software, and deal with the fallout.
<isd>
It wouldn't be *that* hard to build a lighter weight version with webkit; it's pretty easy to use it as a library.
<isd>
There was an article that hit hacker news a few weeks back about how some engineers at YouTube had a big hand in killing ie6, through means that had totally not been approved by management.
<ocdtr_web>
I recall that story, lol
<ocdtr_web>
Unfortunately I suspect a notice about deprecating support for Chrome in YouTube would not survive management scrutiny. :P
<ocdtr_web>
Meanwhile, my Sandstorm server is fine.
ecloud has quit [Read error: Connection reset by peer]
ecloud has joined #sandstorm
<isd>
Unfortunately basically nothing but riot supports matrix's end to end encryption well right now, which we've got turned on in the channel my housemates use.
<isd>
In fairness to electron, my immediate inspiration for complaining is probably riot's bug. Which I should just report instead of wighning to you all.
<ocdtr_web>
It's not like we were discussing anything of importance in here today.
<isd>
(but I'm not seeing the bug on mobile, so...)
<CaptainCalliope>
Just catching up from like, last tuesday. And I think I fixed my nick. Someone who's not accessing via matrix, who do I appear to be?
<CaptainCalliope>
(This is lyre, btw. In case I'm still fracking this up.)
<ocdtr_web>
You are who you appear to be!
<ocdtr_web>
\o/
<CaptainCalliope>
<ocdtr_web "You are who you appear to be!"> Well, that's slightly different question. ;)
<ocdtr_web>
LOL, but you do appear as your actual nick, not a random guest.
<CaptainCalliope>
I'm going to comment on/respond to a few things from the past week.
<ocdtr_web>
By all means, feel free, including from the Office Hours.
<CaptainCalliope>
<isd "I think we only went for SandMD "> My suggestion for SandMD vs CodiMD is because of the size difference for each grain. 100k vs 50 meg seems unreasonable to me. If CodiMD is the one being updated, I think this needs to be addressed. Perhaps I should file a bug somewhere..
<isd>
ocdtr_web, do you have a link for the discussion about the change?
xet7 has quit [Remote host closed the connection]
<CaptainCalliope>
I'm having difficulty accessing the meeting notes from yesterday's meeting. The grain just keeps spinning. Anyone else having this problem?
xet7 has joined #sandstorm
<abliss>
spinner of doom for me also
<CaptainCalliope>
Well, looks like I can download the grain and some of the sqlite file is readable.
<CaptainCalliope>
I'm going to guess the text at the bottom of the file is the most up to date version.
<abliss>
maybe you can reupload to your own server to read the notes
<CaptainCalliope>
Oh, huh. Let me try that.
<CaptainCalliope>
How do I do that?
<abliss>
whose instance is constellation.sandcats.io ? maybe they can debug it
<isd>
That's Lyre Calliope 's
<CaptainCalliope>
Oh, yeah. That's mine.
<CaptainCalliope>
Restoring the backup as a new grain worked!
<CaptainCalliope>
Oh, restarting the app works too! I should have tried that.
<CaptainCalliope>
Original url should work now.
xet7 has joined #sandstorm
<CaptainCalliope>
What was the WeKan board for?
<CaptainCalliope>
Oh, maybe this would be a great place to track communications and documentation work.
<CaptainCalliope>
Random thought re: federation. Within wekan, there's the feature of being able to link to boards and cards outside of the board/card you're viewing. Sandstorm basically breaks this feature. It'd be amazing for features like this to be made possible between grains on the same server as well as grains that our on other servers seamlessly.
<ill_logic>
abliss: Sorry I just noticed your messages. Yes, thank you! I didn't see anything non-daemon for Apache before when I searched. I also figured Docker users might have found a way for similar reasons to ours.
<isd>
The thinking with using wekan was as a way to organize priorities, rather than the details of specific issues, which the issue tracker is good for.
<ill_logic>
And secondly - yes, it did cross my mind to ask if we could find some sort of alternative to init that we could use within a grain. But I wasn't sure if it was a dumb question, clearly there's a reason you're doing it this way.
<isd>
ill_logic: yeah, an init daemon occured to me too, though I think in the short term just sequential scripts for launching things is probably fine.
<ocdtr_web>
isd: SandMD vs. CodiMD discussion is an email thread when I did app review.
<abliss>
<CaptainCalliope "What was the WeKan board for?"> I just threw the wekan together on a whim, because I was having a hard time tracking what everyone was up to and what the next natural things to work on were.
<isd>
ocdtr_web: link?
<CaptainCalliope>
<abliss "I just threw the wekan together "> I think this is a good idea if we're clear on what should and shouldn't be documented there vs the github repo issues.
<abliss>
(has anyone figured out how to use wekan in landscape orientation on a mobile device? all the cards drag instead of scrolling the lane. in portrait it seems to do the right thing.)
<abliss>
Lyre Calliope: yeah, fragmenting the discussion is bad, and as was pointed out above, notifications don't seem to work, so it's not a good place for back-and-forth discussion.
<abliss>
but i was finding the github issue list a bit overwhwelming to digest
<isd>
I think most things on the board should probably just be links to issues, but it's nice for organizing them.
<isd>
GitHub has "projects" feature that would proabably be a good tool as well, though I like trying to dogfood our stuff.
<abliss>
i think if we tried hard enough (and all coordinated well) we could get everything we need out of the github issues. but the wekan seems to be better at low-effort information synch
<ocdtr_web>
isd: I can't link the email thread. :P
<abliss>
<abliss "(has anyone figured out how to u"> (aha: the answer is click avatar pic -> settings -> "show desktop drag handles")
<CaptainCalliope>
Oh, huh. Maybe that should be on by default.
<ocdtr_web>
abliss: Perhaps Wekan will be good for the process of organizing our thoughts and plans, and then they can be cemented down into GitHub milestones in the future.
<ocdtr_web>
Though as I am the only one of the community group who can edit GitHub milestones and tags, I hesitant to lock us to the issue system for that.
<ocdtr_web>
isd: You have email.
<abliss>
ugh, does anyone happen to know how to get virtualenv to actually work on stretch?
<isd>
What's the problem you're having?
<abliss>
```File "/usr/lib/python3.5/contextlib.py", line 5, in <module>
<abliss>
from functools import wraps
<abliss>
ImportError: cannot import name 'wraps'```
<ill_logic>
At what point do you see this? Creating the venv? enabling it?
<abliss>
creating the venv
<ill_logic>
That is frickin weird. Is this a fresh install of the OS? Anything odd about your Python environment?
<abliss>
this is in a fresh vagrant install of stretch, yeah.
<ocdtr_web>
uswsgi stack?
<isd>
ocdtr_web: thanks for forwarding it. Probably addressing the issues means working with the package maintainer.
<isd>
The data codi stores is really trivial -- it seems a shame to pull in such a huge db for it.
<abliss>
<ocdtr_web "uswsgi stack?"> diy
<ocdtr_web>
Ah alright
<ocdtr_web>
Because I know the uswsgi stack installs, and I think, uses venv, and I think I even tested it recently.
<ocdtr_web>
But I don't know much about venv.
<abliss>
did debian just pick an entirely broken version of py3.5 to package as python3?
<ocdtr_web>
isd: I mean, one could report an issue, though if it's "I prefer we don't use MySQL", \o/
<isd>
If you just pop open a py35 shell in the vm, and do `from functools import wraps`, what do you get?
<ocdtr_web>
One thing I'd maybe suggest, is they removed the "publish" button for CodiMD, but that's really a button that burns to be remapped for static publishing or something.
<abliss>
ian: it works
<isd>
Where's the package you're working on?
<abliss>
hangon i'll push it somewhere. thanks for being willing to look.
<ill_logic>
Dumb question: Are you specifying Python 3 in the virtualenv creation command? (Or do you need to nowadays?)
<abliss>
(i'm using the virtualenv installed by `pip3 install virtualenv` because for some reason `apt-get install python3-virtualenv` doesn't install one)
<isd>
I tried doing vagrant-spk vm up && vagrant-spk dev and get a command not found for venv.
<abliss>
getting kinda sick of installing installers for my installers frankly :/
<isd>
:P
<isd>
tbh I don't think debian is a great choice for building apps. packages are often way behind what apps are going to be using, which makes things painful
<abliss>
what vagrant box do you suggest instead?
<ocdtr_web>
We are also not exactly using the latest debian either.
<ocdtr_web>
I would perhaps try going up to a more modern debian/contrib-____64 before trying a whole different distro, but I guess really anything that doesn't break horrifically on global-setup.sh will probably work okay.
<ocdtr_web>
That has vboxsf support.
<isd>
Yeah, as is often the case I am pontificating about long term direction.
<ocdtr_web>
Also, vagrant-spk pins a specific version of the contrib-stretch64 box, and I'm not sure there's even a good reason for that except that we haven't tested it against a newer one yet.
<ocdtr_web>
I think that's a holdover from that jessie64 box that definitely broke if you used the current version.
<isd>
We also never do apt-get upgrade, which makes the problem worse.
<ocdtr_web>
I mean, I think some people do in their scripts.
<isd>
One consideration I have in mind wrt tooling direction: the big challenge with arm support, which is something a lot of people have expressed interest in, is tooling for multiarch builds, and generally making it easy for developers to test their stuff on all supported archs
<isd>
I'd really like to move towards solutions that make reproducability easier, that reason being one.
<isd>
The fact that spk dev just slurps random binaries off the host system is kinda horrible.
<ocdtr_web>
I definitely think vagrant-spk is the key to multiple people from multiple machines being able to hack on a given Sandstorm app.
<abliss>
why upgrade? doesn't that harm reproducibility? doesn't the box come with a working version of everything?
<ocdtr_web>
I couldn't get ill_logic's uploader to work on my box using spk.
<ocdtr_web>
So I was thinking of backtracking to his vagrant-spk version of Kiwix to see what it did.
<isd>
abliss: the staleness of versions is often a pain point for doing development. But yes, it does worsen reproducability.
<ocdtr_web>
It's pretty rare that someone needs to rebuild the exact same version of a Sandstorm package though. Generally you're trying to get it working for the purpose of updating it.
<ocdtr_web>
And presumably upgrading gets you bugfixes in various dependencies, along with the fact that your app you're porting probably slides it's minimum requirements up from time to time.
<isd>
I'd really love to see a nix-spk, and maybe a hydra setup that devs can use for CI, which can do the multiarch builds too...
<isd>
I'd also like to be able to introspect on packages to see what versions of things they're running. Would be useful for vuln scanning.
<ocdtr_web>
Right now with our lack of push for apps to apt-get upgrade/etc. internally, I think we're just relying on that Sandstorm is supposed to be secure. :P
<isd>
Yeah. And that's ok for now, but as we work on features that make apps that have net access, and esp. "public facing" apps, I think it becomes more of a concern.
<isd>
Sandstorm can do damage control, but you still don't want vulns.
<ocdtr_web>
Maybe at least for vagrant-spk apps we can start to nudge things forwards. Unpinning the box, maybe having apt-get upgrade run during global-setup. So that ideally from version a to version b, an app is picking up those updates.
<ocdtr_web>
And then encourage people to use the upgradevm command I want to draft to update that baseline the app is built on to whatever vagrant-spk currently prefers.
<isd>
+1
<ocdtr_web>
While it may occasionally introduce new breakage when updating/rebuilding apps, honestly, it might save people effort when trying to port new apps from having more modern things out of the gate.
<isd>
Yeah, that's my feeling.
<isd>
Maybe we could add a feature that dumps a list of debian packages that are installed into the spk, so you can inspect it after the fact to figure out what changed.
<ocdtr_web>
I also want to try to make the stacks less box-dependent, so we don't have the same difficulty we had going from jessie->stretch happening each time we pick a new base box. In the case of the new Node stacks, I think that's done, and the install scripts are fundamentally written by NodeSource now, and detect what it's installing on.
<ocdtr_web>
But some of our stacks specifically reference installing a package for stretch or what have you, and we should either find a way to install them that doesn't require we specify that, or we should detect the box and handle that more dynamically.
<ocdtr_web>
(This is all good post-1.0 stuff, I will try to start putting together issues and a milestone for it.)
<isd>
Maybe we could actually build debian packages for the logic in the stacks, and have global-setup add the appropriate repo.
<isd>
(But yes, this is longer term brainstorming)
<ocdtr_web>
I do like the simplicity of the design of the stacks, tbh. It's fairly easy to figure out how to modify/gut them to your needs.
frigginglorious has quit [Read error: Connection reset by peer]
<ocdtr_web>
Re: feature that dumps a list of debian packages, is there a good command that does this? I imagine the cost of making a text file with this info in .sandstorm is low.
<isd>
I'm sure there is, don't know what it is off hand.
<ocdtr_web>
As it is, we have the "stack" file in .sandstorm which is literally a one-word note of which stack was used to create the package, even though it's fundamentally irrelevant/ignored after the scripts are pasted in.
<ocdtr_web>
It is a handy reference though. And I imagine we could do the same for the package info.
ocdtr_web has quit [Remote host closed the connection]
<ill_logic>
isd: Nix can do deterministic builds right? So in theory we could have multiple signers on packages too?
<isd>
Yeah, it would be a huge boon for reproducibility
<isd>
I'm investigating where musl support is with nix right now. long term I'd like to deprecate the file access detection stuff that spk dev does, but with a glibc based system that ends up with huge packages. With docker-spk I found packages built with alpine to be reasonable.
<isd>
It looks like there is some support.
<abliss>
i may have missed it, but did any of those great long-term ideas include a clue for getting a working python inside vagrant-spk today?
<isd>
:P Sadly no, I got sidetracked. Let me stare at it a bit more.
<abliss>
(I tried an "apt-get upgrade" inside the vm ssh, but it seems to have crashed the box)
<isd>
crashed?
<isd>
Did vagrant-spk up get removed at some point? there's a vm up, but the docs mention regular `up` and I get an error about an invalid command.
frigginglorious has joined #sandstorm
<abliss>
ssh session hangs, and after aborting, the box is not up
<isd>
Yeah, looks like vagrant-spk up was removed way back
<isd>
Submitting a pr for the docs
frigginglorious has quit [Ping timeout: 265 seconds]
frigginglorious has joined #sandstorm
<isd>
abliss: the first error you talked about, you got when setting up the venv?
<abliss>
(oh, i guess the reason `apt-get upgrade` crashed the box was that i ran out of disk space. d'oh.)
<abliss>
anyone know where vagrant stashes all the stuff it downloads?
<isd>
I didn't hit the exact issue you did, probably that was partially an artifactof changes you made while experimenting.
<isd>
The first thing I hit was "virtualenv: command not found,", hence doing python3 -m virtualenv, and then went from there. A couple of other bumps but none of them were too bad