17:38 <MrWizard> hi folks .. got a noob question .. Trying to build a simple fileshare with sandstorm.io + davros but I'm hitting some kind of filesize limit around 22mb .. I've been all over the documentation looking for the obvious clues for a tunable but can't find it .. hopefully someone can help a brother out with a pointer in the right direction

17:51 <JacobWeisz[m]> AFAIK, the only limitation is that connection interruptions can cause issues with uploads and downloads to Sandstorm servers.

17:52 <JacobWeisz[m]> https://github.com/sandstorm-io/sandstorm/issues/2464

17:52 <MrWizard> Thats really strange then, its constantly stopping at 22mb on the upload but nothing in the log to indicate a limit is being hit

17:53 <JacobWeisz[m]> Is there anything on your box with regards to memory or swap or something that could be running out?

17:54 <MrWizard> Not that I know of, its a fresh linode install -- literally I fired up the instance and installed sandstorm on it

17:55 <MrWizard> dmesg and the usual log files don't show anything odd .. checked that before trying to dig the documents

17:55 <abliss> where are you uploading from? any chance a proxy in between is dropping the cxn?

17:56 <abliss> anything in your browser debug console when the limit gets hit? does the connection just drop, or does the server send a response code?

17:57 <MrWizard> The browser is strangely silent .. It seems to think the entire file gets uploaded .. I've tried different browsers (firefox/chrome) on two different macbooks one here and one overseas so we really diversified on this test

17:57 <MrWizard> let me send up another on with the debug console open and see what the response code is

18:11 <MrWizard> really nothing in the console except lots of white on yellow and one 409 message from webdav at the beginning ..

18:12 <isd> Another thing to try: Does a Filedrop grain behave any differently?

18:12 <isd> Also, how much memory does the VM have?

18:12 <MrWizard> its a 4g VM

18:12 <isd> Ok, that should be fine.

18:12 <MrWizard> let me try a filedrop grain .. didn't try that one

18:14 <MrWizard> I can throw a share link out there if anyone wants to take a crack at uploadng .. this is fresh so there's nothing that can't be seen or proprietary or anything

18:14 <abliss> i'll try

18:14 <isd> Sure, might tell us if it's an issue on the client end.

18:15 <MrWizard> https://glanzig.sandcats.io/shared/qcBlVXkGoJmwQCin_vrhhRf5JwAUw2aoOaf4ZapwbN_

18:17 <MrWizard> I doubt its client .. did the browser, computer, internet provider dance already

18:18 <MrWizard> installing filedrop now

18:18 <abliss> i uploaded a 23MB random file fine

18:18 <MrWizard> try something > 23mb like 1g as it cuts off at 22

18:20 <MrWizard> file drop cut off too

18:20 <TimMc> I just uploaded a 40 MB Debian package and downloaded it again, and was able to verify it was identical.

18:20 <isd> Just succesfully uploaded a ~60MiB file

18:20 <abliss> 50Mb went through fine

18:20 <TimMc> Firefox 80 from a good residential connection in the US.

18:20 <MrWizard> ok .. i see them ..

18:21 <abliss> i used the latest chrome beta on android

18:21 <isd> Same.

18:21 <MrWizard> I wonder if theres something about the file .. let me try the same thing

18:22 <TimMc> The request timing says it took 20 seconds to transfer and then 1.5 seconds for the server to send back the 201 response.

18:22 <TimMc> I can reduce my connection bandwidth and try again if you'd like.

18:23 <MrWizard> haha I'm droppng a debian iso in there now to see what happens

18:24 <TimMc> 150 MB so far on that one, hmm.

18:25 <MrWizard> ok now this has gotten strange for sure .. the ISO file uploaded .. but the CR2 files from the cannon camera cut off ..

18:26 <isd> I wonder if davros is doing something weird with the files after upload?

18:26 <MrWizard> Now -- I'm not a fan of cannon cameras .. but thats kind of judgmental for an app

18:26 <TimMc> :-)

18:26 <isd> Does uploading into Filedrop work?

18:26 <isd> (I know davros does previews and such, maybe it's triggering a bug or something...?)

18:28 <MrWizard> filedrop seems to do the same thing ..

18:29 <TimMc> Next question: Are you uploading from an SD card that could be corrupted?

18:29 <TimMc> Maybe copying the files *anywhere* will break.

18:29 <MrWizard> No ..

18:30 <MrWizard> I've done this from two different computers and both had the files in the filesystem

18:30 <MrWizard> just like the .ISO that worked

18:30 <MrWizard> I'm going to try with a different extension to see if we are tripping a preview bug

18:32 <MrWizard> same thing -- cut it off on upload

18:32 <mnutt> perhaps try downloading the grain backup to see if the file was saved to disk?

18:33 <MrWizard> I think davros saves into the filesystem .. let me look there

18:33 <mnutt> it does, yes

18:34 <TimMc> If you send me one, I can try uploading it. (Aw... send.firefox.com is either down or gone. Something else, then.)

18:34 <mnutt> it got shut down a few days ago I think :(

18:34 <TimMc> oof

18:34 <TimMc> https://transfer.sh/ is still up for another month :-X

18:34 <MrWizard> I think I can solve the work problem by cluster zipping them and uploading that as zip files seem to come up intact

18:35 <MrWizard> renaming didn't work but zipping them together does

18:35 <mnutt> I wonder if the content-type header is causing problems

18:35 <mnutt> I can't think of why, though

18:36 <MrWizard> confirmed .. zipping works

18:36 <MrWizard> I'll go open an official issue with the github repository and put in some sample files I can share and how to duplicate

18:37 <abliss> it'll be funny if the sample files fail to upload to github too :)

18:37 <MrWizard> at least I don't feel like I can't RTFM /sigh

18:37 <mnutt> Thanks, that'd be helpful. If I can reproduce I can throw some debugging info into davros to figure out what the issue is

18:37 <abliss> the file doesn't have a string like `X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H` at the 22mb offset, does it?

18:38 <MrWizard> I never looked .. This is a bunch of raw camera files one person in my company wants to send to another and I got pulled into this as they were emailing 1.5gb files and it was failing .. so I thought Oh I read about this sandstorm / davros thing and thought "This would be an excellent time to play with this" and down the rabbit hole I went

18:39 <MrWizard> Give me a day or two to package it @mnutt and I'll have you some kind of reproduction method

18:40 <mnutt> thanks. and sorry your very first attempt at uploading a file failed

18:41 <MrWizard> no worries .. at least it sorted to a method that works enough .. and feel slightly vindicated that i'm not the IT idiot these people think i am .. though the day is still young

18:44 <isd> Bear in mind there is an issue we've seen with large files on shaky connections, since Sandstorm doesn't handle range requests -- but afaik it should work fine as long as the connection doesn't drop.

19:56 <mnutt> I think I may have asked this a while back, but it has been a few years. Is anybody doing automated spk builds in CI?

19:57 <isd> Not as far as I know

19:58 <isd> I think you could probably rig it up by feeding it the app signing key via an environment variable or such.

19:58 <isd> Probably doing it in one of the off-the-shelf CI systems means not using vagrant-spk.

19:59 <isd> ...I'm half tempted to try setting this up for SandCal, but I have a non-specific aversion to uploading the signing key to GitHub.

20:01 <isd> It would be neat to rig something up so that I could supply the signature and CI would try to re-build the package; if the build is reproducible, it could then just attach that signature rather than needing to have the signing key itself. And it provides extra assurance that the built package is relly what's in the source repo.

20:02 <mnutt> yeah I've found that I often delay releasing minor changes because there's a fairly tedious process for getting it built and released

20:02 <isd> It would definitely be nice if tagging a commit would result in CI doing spk publish for me.

20:02 <mnutt> even having CI produce self-signed packages that I could then pull down and re-sign would be a big step up

20:03 <isd> I forget, does davros use vagrant-spk?

20:03 <mnutt> yes

20:03 <mnutt> but I'm not opposed to reworking it

20:04 <isd> I think that's probably the biggest hurdle. Can't really launch VMs in most CI solutions afaik.

20:04 <isd> Building on docker-spk or just nix or something might work better.

20:05 <mnutt> whattt I somehow totally missed docker-spk

20:06 <mnutt> that's huge. I have issues on MacOS with what I think is a bad interaction between virtualbox and docker causing sporadic crashes so I try to run virtualbox as little as possible

20:10 <JacobWeisz[m]> Ian's side hobby is writing new ways to write spks. :P

20:10 <JacobWeisz[m]> He started nix-spk too I think.

20:10 <mnutt> haha

20:19 <isd> Not in earnest; I did some design work and moved some of the code from docker-spk out into a library, but didn't actually break ground on the nix stuff.

20:20 <simpson> How much different would it be from what nixpkgs' dockerTools already does?

20:20 <simpson> (Why not add a new spkTools to nixpkgs?)

20:23 <isd> Not familiar. Will have to look into it.

20:26 <simpson> Basically, it's going to be easier to build stuff from within Nix and have it be packed into whatever format you like, rather than to try to subprocess Nix and extract stuff from it.

20:29 <isd> Yeah, my intention was that the tool would basically invoke nix-build and then bundle the result. But it would also make sense to have a tool that would bundle a pre-existing build in the store, which could be invoked by further nix builds...

20:31 <simpson> Yep, Nix the ultimate.

21:17 <isd> Some talk on the capnproto mailing list about adding Level 3 RPC to the Go implementation.