kentonv changed the topic of #sandstorm to: Welcome to #sandstorm: home of all things sandstorm.io. Say hi! | Have a question but no one is here? Try asking in the discussion group: https://groups.google.com/group/sandstorm-dev
<isd>
Reminder: office hours in a little under an hour
<isd>
Reminder: office hours now
frigginglorious has quit [Ping timeout: 265 seconds]
<abliss>
oh snap... let me guess, the fuse in spk-dev doesn't capture a file if it only get stat()ed and never opened
<abliss>
so if your app is written in python, when you run 'spk dev' it will stat e.g. `lib/python3.6/os.py` but then notice it is older than `/usr/lib/python3.6/__pycache__/os.cpython-36.pyc`, so it will only open the latter, so only the latter ends up in the .spk pack file
<abliss>
but, if you then try to run that pack file, it'll fail to stat the original os.py, and then decline to use the os.pyc file, and you're screwed
<abliss>
and then it crashes with `Could not find platform independent libraries <prefix>`
<abliss>
surely i'm not the first person to hit this? unless literally nobody has ever tried to use `[vagrant-]spk dev` on an app with literally any python inside?
<isd>
I thought it created empty files in that case? not sure. If you spk unpack, are the files there?
<isd>
I've definitely done spk dev on a python project before.
pie_[bnc] has quit [Ping timeout: 260 seconds]
pie_[bnc] has joined #sandstorm
frigginglorious has joined #sandstorm
frigginglorious has quit [Ping timeout: 244 seconds]
CcxWrk has joined #sandstorm
<abliss>
nope, os.py doesn't show up if i spk unpack (and doesn't show up in /var/sandstorm/apps/$hex when i install)
<abliss>
my vagrant vm seems to have spk from sandstorm v262 fwiw
<isd>
I think that's current.
<isd>
You could try putting stuff in alwaysInclude to dodge the issue. That's weird though.
<abliss>
looks like only zero-size files get added in response to stat.
<abliss>
Maybe this is only a problem if you use python in a venv? maybe if you use the system python, it will eventually read() the system os.py... maybe only a venv is self-contained enough to skip straight to the pyc
<abliss>
... nope, strace seems to indicate that even outside a venv, python never actually calls open() on `/usr/lib/python3.6/os.py` . so i'm confused how anyone's ever successfully used `spk dev` with python.
<abliss>
maybe really old versions of python didn't have this behavior?
<isd>
So am I.
<isd>
What version are you on?
<abliss>
py3.5 and py3.6 both seem to have this behavior
<isd>
Oh wait, 3.6 duh
<isd>
My approach here would probably be to just use alwaysInclude. But yeah, I'm now really confused about what changed.
<abliss>
well i can't really alwaysinclude beacuse i don't have a good list of which .py files are needed... even if i did, it'd be a list thousands long that i wouldn't want to maintain in my pkgdef
<abliss>
i guess i could write a script that crawls the sandstorm-files.list and converts each .pyc back to a .py by guessing where it's probably located
<isd>
You can give it a directory.
<abliss>
it's still dozens of directories when you include all of my transitive python dependency packages... and lots of them i don't even want
<abliss>
at this point i'm kind of inclined to add a command-line flag to spk dev to include all stats
<isd>
I feel like an option in th sourceMap might make more sense.
<abliss>
ooh, good idea
<abliss>
it's a bit of a mixture of concerns...
<abliss>
like in theory you could use `sourceMap` and `spk pack` and never use `spk dev` (assuming you hand-write your `sandstorm-files.list`
<abliss>
but there's already a comment about `spk dev` inside the default `sourceMap` so the concerns are already slightly mixed
<abliss>
this would be a good question for kentonv to weigh in on. (executive summary: `spk dev` doesn't include files that are only stat()ed but never open()ed, unless they are zero-byte. This doesn't work for python since it will stat() os.py but then only ever open os.pyc. I'm suggesting an optional mode to make it include all stat()ed files, not just the zero-byte ones. Should it be a command-line flag to `spk dev` or a flag on
<abliss>
the `sourceMap` in the pkgdef?)
<isd>
Maybe it put it in manifest.metadata instead, next to the fileList field?
<isd>
That way it's also grouped near alwaysInclude, which is nice in that it keeps all of the autodetection-related things together.
<isd>
In theory you could make the package smaller by just including empty .py files and making sure their timestamps were older than the corresponding .pyc files
<abliss>
yeah , i was thinking about that... haven't come up with a good way to do it that isn't finicky
<abliss>
and given that the package already has 25MB of stuff, i'm not super concerned about the weight of the extra .py files. we'll have to see if the comment is really accurate that "it is normal to stat() every file in a directory when listing that directory" , to see how much other bloat is added
<abliss>
what would be awesome is a flag to python that says "if you can't find the .py file, assume the .pyc file is current" but I couldn't find anything like that
<abliss>
i found a lot of stackoverflow questions from people saying "how do i distribute an app with only .pyc files and not .py files" and the answer is always "you don't"
<isd>
From what I'm seeing the reasons are things like incompatibility between python versions (not an issue because we're bundling it with the python interpreter anyway) and usually the person asking the question wants to avoid shipping .py files because they're shipping proprietary software and they don't want to provide source, but .pyc files are really easy to decompile, so this doesn't really help them anyway.
<isd>
So I'm not sure those answers apply to us
<abliss>
right, i do think it would work if python would support it, but i doubt there's any easy way to make the python executable do what we want.
<isd>
It probably compresses well anyway
<isd>
It might be nice to have some python-specific tooling that just understands python dependencies so we don't have to rely on the fuse nonsense in the first place
<abliss>
oh, i wouldn't touch the python dependency ecosystem with a ten foot pole
<isd>
tbh I really kinda don't like the fuse thing.
<isd>
I guess I'm just thinking of something that does a pip freeze and includes those packages, rather than trying to muck with file detection.
<simpson>
abliss: Python in general is only defined on .py files. .pyc files are a CPython implementation detail. Indeed they are not portable.
<abliss>
sure, but i'm bundling a cpython interpreter inside a .spk file. i don't care about portability at all here.
<abliss>
ooh, i know: since i'm only using .py files from within my venv and never from the real /usr/lib , maybe i can just touch every single .py file to freshen it before starting spk dev,
<isd>
Stupid hacky idea that we should never ever implement: We could set the permissions on files that were only stat()'d to 0 (not even readable by owner) and just include them as sparse files in the package even if they're zero size. This way the metadata can still be read, but they don't take up space. Obviously the permissions will be off, but we don't fully preserve that as-is.
<isd>
I mostly mention this because it amuses me, not because it's actually a resonable idea.
<abliss>
yeah, i was thinking about the sparse file too. permissions shouldn't matter right? assuming it never got opened (or rm'd or whatever), swapping out the contenst for zeroes shouldn't disturb anything
<isd>
The danger is that if the app does try to open() it later, even though it didn't in spk dev, then if the contents are wrong it could result in bugs that are hard to understand, vs. a simple "permission denied"
<abliss>
true... i mean, in general, if an app tries to open() something at runtime that it didn't open() during spk dev, there are going to be hard-to-understand bugs
<abliss>
that seems kind of inseparable from the basic idea of spk dev :)
<isd>
Like I said, I really don't like the fuse thing.
<isd>
But "can't open the file" is much easier to understand than "the file unexpectedly contains a bunch of zeros wtf"
<abliss>
it is awfully convenient though.
<abliss>
(and even if you did it with an LD_PRELOAD or a ptrace or other hack besides fuse, the basic underlying problem would remain. it's inherent in "automatic dependency discovery". the alternative is completely specifying every file you depend on, a la bazel)
<isd>
I mean, you could do it at a higher level of abstraction than bazel. But you'd need more knowledge of the tools involved
<kentonv>
abliss, the problem with including stat()ed files is that it's pretty common for various frameworks to stat() everything in a directory when listing it even if they don't intend to use the contents. But maybe adding zero-size files (or even sparse files, to fake the file size) could make sense?
<kentonv>
(NOTE: I haven't read most of the conversation)
<isd>
I guess what I mean is being able to just say "packages referred to by requirements.txt" and not have to fuss with what that menas.
<isd>
means
<abliss>
yeah, that's why we were thinking of making it optional.
<abliss>
sparse files to make the file-size correct is what we're currently discussing
<kentonv>
making things optional is kind of sad because then the developer has to think about it
<simpson>
abliss: For example, Nix uses a higher level of abstraction; the *build* of the build tools is sandboxed, and extensionality is used to infer which parts of the filesystem are actually relevant. But you must be okay with /nix and abandoning FHS.
<isd>
zero-size files wouldn't help here, since the files in question are not zero size.
<abliss>
true. i'm gonna try my current idea of force-refreshing every .py file during my "build.sh" , that may obviate the entire issue for this .py/.pyc problem thanks for showing up kentonv
<abliss>
i think he meant "if a file is only stat()ed but never opened, make a new zero-size file with most of the other attributes the same"
<isd>
I'm generally in favor of not trying to make the flaky heuristic thing more complicated. I think the right answer here is "if spk dev isn't doing what you want, you have to specify stuff explicitly, sorry."
<isd>
I'd really like to see a nix-spk or such.
<isd>
Also: with docker-spk I found that if you use alpine as the base image then even without spk-dev's pruning the package sizes are still not ridiculous usually
<isd>
A lot of the bloat that spk dev gets rid of is from base-system stuff that's too big
<abliss>
dang, doing a global freshen at the end of my `build.sh` didn't work
<isd>
You'd need to do it in launch.sh
<abliss>
no, the idea was to do it after all my python runs (pip install, etc) , but just before spk-dev starts up
<abliss>
the idea is not to force-include every .py file, but to force-freshen every .py file, so that the ones that actually get imported (and only those) will be opened
<isd>
Oh, you mean to make sure the app actually open()s the files.
<abliss>
i'm guessing python outsmarted me, and touching os.py isn't enough to invalidate os.pyc
<isd>
Maybe it's doing something special for system directories?
<isd>
Honestly I'd just add /usr/lib/python3.6 to alwaysInclude and be done with it.
<abliss>
that's 94MB
<isd>
Ok I guess that is a bit much. I remember it being a lot smaller when using the alpine images.
<abliss>
ah, i think i missed os.py itself in my find-and-touch touch
<abliss>
build.sh doesn't have root access to touch /usr/lib/python3.6 stuff. i'll have to do it in setup.sh
<isd>
I think you could just do sudo
<isd>
Pretty sure the vagrant user can do that without a password.
<isd>
setup.sh will probably not do what you want, since it runs before the build and therefore might not actual result in the file it touches being as new as needed.
<abliss>
well currently my build.sh is empty, so the end of setup.sh should be fine right?
<isd>
Yeah I guess so
<abliss>
actually, i guess i need to touch all the /usr/lib stuff in setup.sh , and touch all the ./venv/ stuff in build.sh
<isd>
It might actually hurt startup times to force it to read the .py files.
<abliss>
i'm just forcing it to read them in the spk dev phase, not in the actual app
<isd>
Sure, but if they're newer than the pyc files when it runs, it's going to go through the same logic
<abliss>
does spk pack preserve mtimes?
<abliss>
if so, it should detect that the pyc files are newer
<isd>
I don't think the app can update the pyc files in the system directories, because it's a read only file system
<abliss>
wahoo, freshening in setup.sh works, i have os.py in my sandstorm-files.list
<isd>
It might not be that bad in comparison to everything else it's doing on startup though
<abliss>
wow, the spk went from 23Mb to 27Mb, that's quite surprising. I have a hard time believing there's 4Mb compressed of .py files in there.
<isd>
I don't :P
<isd>
What's the diff in your sandstorm-files.list? is there anything other than .py files added?
<isd>
I really wish nix could handle dynamically gathering dependency info better.
<abliss>
mostly. looks like a couple new icons snuck in there
<isd>
I'd like some tool to discover the dependency graph and just put it in a .lock file or such
<isd>
Hm, I wonder if we can get video calls to work with this.
<isd>
Requesting browser permissions doesn't seem to work through the iframe
<isd>
Definitely some rough edges.
<abliss>
oh tons. i'm just chuffed to have any edges at all :)
<isd>
Still, it's exciting.
frigginglorious has joined #sandstorm
<abliss>
desktop notifications also get denied without prompting due (i think) to the iframe. Is there a general way to make chrome allow iframes to do cool stuff?
<abliss>
"More generally, permission requests from iframes cause significant confusion for users. It is difficult to distinguish between the main frame origin and the origin requesting the permission. The scope of the permission decision (within or outside an iframe) is also unclear, making it difficult to reason about when a granted or denied permission takes effect."
<abliss>
Seems like they might entertain a carve-out when the iframe is a subdomain of the main frame?
<abliss>
"Instead of using an iframe, affected websites can open a new window (i.e. a new main frame) to request notification permission. "
<abliss>
Ugh, that's an ugly invasive change to every single app that wants any permissions. i wonder if we should add to the sandstorm shell a "pop ui frame out into its own window" button, and teach users to use that when requesting perms?
<abliss>
or maybe we continue with the idea of using a js proxy object to "fix" localStorage. a similar proxy could perhaps intercept all notification requests (and camera/microphone and everything else that needs permission) and route them through to the root iframe?
<JacobWeisz[m]>
There would be issues with asking people to pop out the iframe. Like the fact that they'd be granting permission to a ethereal subdomain and not the site they're using.
<JacobWeisz[m]>
In the case of notifications, at least, I thought the genera consensus was that apps should use Sandstorm's notifications and then Sandstorm should have options for how to email or display them to the user.
frigginglorious1 has joined #sandstorm
frigginglorious has quit [Ping timeout: 256 seconds]
frigginglorious1 is now known as frigginglorious
<abliss>
oh right, good point, i forgot about that!
<JacobWeisz[m]>
I am not sure how well that would work with granting access to the camera or microphone though.
<JacobWeisz[m]>
Does foo.com asking for access also grant it to bar.foo.com?
<kentonv>
I think most permissions are keyed by full hostname
<kentonv>
it'd be neat if there were a way to pass the permission from the parent frame to the child in the form of a capability... but I seriously doubt that's possible
<isd>
Yeah, I'd be very surprised.
<isd>
So many things are harder tthan they have to be because web.
<kentonv>
but pretty much they build the things Google wants, not people like us
<isd>
yeah, I think to have any real pull at all we'd probably have to actually being hacking on a browser or something.
<JacobWeisz[m]>
If the server on Sandstorm works with apps/clients we avoid a lot of the challenges of working with browser/web issues.
<JacobWeisz[m]>
I think the in-Sandstorm interface only needs to have the basics if the apps can work.
<abliss>
yeah, high on my list is to patch the android riot.im port (and/or possibyl the new rewrite riotX) to use sandstorm webkey urls
<kentonv>
ehhh personally I'm a big fan of doing everything in the browser, TBH
<kentonv>
(just speaking for myself as a user)
<JacobWeisz[m]>
I hate browsers doing audio video and notifications.
<JacobWeisz[m]>
Personally I think notifications in particular should be deprecated entirely from browsers. I solely see them in the real world in use by malicious actors.
<isd>
+1 for O(1) copies of chrome on my laptop.
<JacobWeisz[m]>
Every old lady who thinks their computer is hacked just has 62 sites authorized to send popup ads via the notifications API.
<isd>
The legitimate use cases are for things like chat apps. Which I would much prefer to be running in a browser over having another electron process going.
<JacobWeisz[m]>
Well, chat apps shouldn't be written in Electron, IMHO. :P
<isd>
Also, it's a lot easier to onboard some rando with an app I want to use if they don't have to install anything first.
<isd>
I mean, I agree. I fire up riot only when I can't figure out how to do something in weechat nowadays
<isd>
Basically entirely because of battery life concerns.
<abliss>
how is weechat's matrix support on android?
<isd>
No idea, I'm using it on my laptop
<abliss>
oh
<isd>
I've been using RiotX on android.
<abliss>
is it stable? i'm still on the old riot.im app
<isd>
It's missing some features, but it seems to work fine. I jumped on captaincalliope's reccommendation
<JacobWeisz[m]>
Notifications in the browser though is something I've never seen someone want outside the tech bubble. And seen a ton a spam with. Iunno, I don't even want my web apps doing popup notifs.
<abliss>
(riot.im through chrome on android is kinda usable, but the native experience is definitely better.)