<stevenaleach>
IPFS will having amazing potential for machine learning and enable Public Computing as a new form of engagement and collaboration - this is terrific. I've finally learned the importance of backups the hard way, and am starting the new year with a regrettably clean digital slate ;-( But now my work (Jazz-Standards dataset w. personnel/instrumentation, year, standard ID from public-domain/publicly available 78rpm and etc.
<stevenaleach>
recordings) is being constructed around IPFS and my work pinned on a VPS with a 100GB IPFS partition. Soon, I hope to be able to simply mount the dataset from EC2 GPU instances - worker nodes which will in turn publish updates to their models as they train which the VPS will pin. Thanks Yall!
dignifiedquire has quit [Quit: Connection closed for inactivity]
infinity0 has quit [Remote host closed the connection]
patcon has quit [Ping timeout: 258 seconds]
stevenaleach has quit [Quit: Leaving]
aerobil[m] has joined #ipfs
wallacoloo____ has joined #ipfs
<pguth>
Do you pre-encrypt sensitive files? Or is that of no concern?
<pguth>
That's actually a general question of mine. How to handle that?
<vtomole>
pre-encrypt. IPFS is not mature enough, especially if you are using a mounted file-system.
tilgovi has joined #ipfs
cemerick has joined #ipfs
seagreen has quit [Quit: WeeChat 1.6]
dPow has quit [Ping timeout: 246 seconds]
dPow has joined #ipfs
Earnie has quit [Ping timeout: 252 seconds]
tg has quit [Ping timeout: 255 seconds]
plddr has quit [Ping timeout: 258 seconds]
Charley has joined #ipfs
xming has quit [Ping timeout: 240 seconds]
tmg has quit [Ping timeout: 248 seconds]
ygrek has quit [Ping timeout: 258 seconds]
mguentner has quit [Quit: WeeChat 1.6]
hqip has joined #ipfs
mguentner has joined #ipfs
jedahan has joined #ipfs
wrouesnel has quit [Remote host closed the connection]
cyanobacteria has quit [Ping timeout: 240 seconds]
s_kunk has joined #ipfs
jkilpatr has joined #ipfs
mildred3 has joined #ipfs
robattila256 has quit [Ping timeout: 260 seconds]
cwahlers_ has quit [Read error: Connection reset by peer]
cwahlers has joined #ipfs
Qwazerty has quit [Ping timeout: 258 seconds]
tmg has quit [Ping timeout: 260 seconds]
bastianilso____ has joined #ipfs
ebarch has quit [Ping timeout: 255 seconds]
Neurolit[m] has joined #ipfs
Qwazerty has joined #ipfs
mildred1 has joined #ipfs
lemmi has quit [Ping timeout: 248 seconds]
lemmi has joined #ipfs
tabrath has quit [Remote host closed the connection]
tabrath has joined #ipfs
DiCE1904 has quit [Read error: Connection reset by peer]
Mizzu has joined #ipfs
pfrazee has joined #ipfs
pfrazee has quit [Ping timeout: 240 seconds]
arpu has quit [Ping timeout: 240 seconds]
<richardlitt>
lgierth: You around?
<richardlitt>
or victorbjelkholm?
<victorbjelkholm>
yeah
maxlath has quit [Quit: maxlath]
<richardlitt>
victorbjelkholm: Any idea of the best way to cache github results for project-repos?
<richardlitt>
I don't know how https://project-repos.ipfs.io/ is hosted - wondering if there is a server-side way to store the GitHub data so that it only gets it every hour or so, instead of depending on the client
DiCE1904 has joined #ipfs
<richardlitt>
If we're going to have people using it, I think that storing the data somewhere and not depending on the client is probably the best way to go.
<richardlitt>
But I don't know how it is hosted or updated.
Boomerang has joined #ipfs
<victorbjelkholm>
richardlitt, you could save it in localStorage in the browser, so we avoid having a backend
<victorbjelkholm>
save it together with the date it being updated last time, so you can refresh if it's older than say one hour or something
<victorbjelkholm>
if we really want a backend, I'm not sure how project-repos is deployed, probably with dokku? Then it would be trivial to add some backend with hapijs for example
tabrath has quit [Ping timeout: 245 seconds]
kenshyx has joined #ipfs
bastianilso_____ has joined #ipfs
bastianilso____ has quit [Ping timeout: 240 seconds]
<Mateon1>
I thought project-repos was hosted on IPFS
<Mateon1>
$ ipfs name resolve /ipns/project-repos.ipfs.io # /ipfs/QmcoqjP5SJEwKWJyLPeCe38syWioLfFsFTN3FqVuMv3vTT
ralphtheninja has quit [Ping timeout: 245 seconds]
tmg has joined #ipfs
Boomerang has quit [Quit: leaving]
ckwaldon has quit [Remote host closed the connection]
infinity0 has joined #ipfs
Jackneilll has quit [Read error: Connection reset by peer]
pfrazee has joined #ipfs
Jackneilll has joined #ipfs
kenshyx has quit [Quit: Leaving]
DiCE1904 has quit [Read error: Connection reset by peer]
pfrazee has quit [Remote host closed the connection]
ashark has joined #ipfs
pfrazee has joined #ipfs
DiCE1904 has joined #ipfs
ralphtheninja has joined #ipfs
maxlath has joined #ipfs
<richardlitt>
I'm not sure how it gets to project-repos.ipfs.io
<richardlitt>
All I know is I have to open a ticket in ops-requests
<Kubuxu>
project-repos is hosted on ipfs
<Kubuxu>
it doesn't have real backend
<Kubuxu>
it is static site
<Kubuxu>
what we could do is have API that would crawl those repos
<Kubuxu>
and if it is not available fallback to making requests ourselves
<richardlitt>
That would be nice. So, just a bot on Heroku that makes those calls and then provides them as needed?
anewuser has joined #ipfs
jadedctrl has joined #ipfs
ivo_ has joined #ipfs
Boomerang has joined #ipfs
<victorbjelkholm>
richardlitt, yeah, a simple API that would just return cached responses if possible, otherwise fetch again
<victorbjelkholm>
could be deployed to Heroku, or our own Dokku instance we have running as well
<jadedctrl>
I'm getting an invalid multiaddr when using swarm connect on Windows in a LAN network.
<lgierth>
jadedctrl: what does the addr look like?
<jadedctrl>
The address I'm using is "/ip4/10.145.24.153/tcp/4001/ipfs/", which, as far as I can tell, is valid
<jadedctrl>
*pretend the peerid is after ipfs/
<lgierth>
mh yeah that looks good
<lgierth>
mind filing an issue? we still get the occasional weird windows issue every once in a while :)
<Kubuxu>
in most cases it won't be 300TB or even 100TB in one server
arkimedes has quit [Quit: Leaving]
<Kubuxu>
we can aim fore 300TB but if tests are done of 50TB or even 20TB (which is much easier to achieve) it will be enough imo
<Kubuxu>
we need to know scaling
<flyingzumwalt>
these are all great notes to add to that GH issue.
<Kubuxu>
yeah, doing just that
arkimedes has joined #ipfs
<flyingzumwalt>
It's very likely that the first work we dive into today will be setting up the test suite, so the more you can do to lay the groundwork for that, the better off we will be.
<Kubuxu>
me and why have started something during teamweek but it needs work
<flyingzumwalt>
👍
<Kubuxu>
also it would be great if we could get even a TiB of sample data from them (different data different optimizations)
<Kubuxu>
for quite a long time we have thought that most blocks would be close to max size
<Kubuxu>
but it turns out that most datasets do not consists of files MiBs in size
<Kubuxu>
flyingzumwalt: or if they have cloned the data (even small amount of it), if the could run small script over them that would give us more info
<lgierth>
also note we never properly tried retrieving the existing 3.8TiB dataset on another host
<lgierth>
it failed miserably due to a bitswap/wantlists bug
<Kubuxu>
that too
<victorbjelkholm>
completely unrelated question, could we add opt-in analytics in the cli of go-ipfs? I would like to report my usage somehow, like the average size of file and number of files per add etc
<lgierth>
on that note, while updating grafana i noticed it has opt-out analytics
inaltoasinistra[ has left #ipfs ["User left"]
<Kubuxu>
have you opt-outed?
<lgierth>
yes
<lgierth>
now that i noticed
<Kubuxu>
we for sure can, is it useful? I have no idea.
<Kubuxu>
the sample size and possible skews might render the data not suitable
<victorbjelkholm>
Kubuxu, I think it could be, at least to see some average of things and how people usually use ipfs, even though it's just a small subset of the users
<lgierth>
it might be useful if you combine it with concrete performance metrics
<lgierth>
ipfs analytics start ; ipfs do some stuff ; ipfs analytics submit
<victorbjelkholm>
lgierth, something like that, yeah!
s_kunk has quit [Ping timeout: 248 seconds]
ttk2[m] has joined #ipfs
<Kubuxu>
flyingzumwalt: so it is a bit similar problem we had with npm on ipfs
bastianilso_____ has quit [Quit: bastianilso_____]
<Kubuxu>
my question then was always: do we need all of this on IPFS?
<Kubuxu>
if what you really use is small percentage and there is normal secondary endpoint, you can have fallback system, if data of interest is in IPFS (index) then use IPFS if not, fetch it from fallbakc
<Kubuxu>
if the data that isn't on IPFS gets popular, add it to IPFS, add it to index
<lgierth>
this would have deserved a proper cam ;)
<lgierth>
got to see any elfs?
<whyrusleeping>
There werent any elves out, I think the harsh winter weather kept them in hiding
<lgierth>
i heard they're also pretty antiamerican
<whyrusleeping>
oh?
<lgierth>
probably just a rumour
<flyingzumwalt>
kubuxu this idea of rotating things into and out of the index based on demand is new to me. It sounds like it overlaps with the discussions about ipfs-pack and filestore. Example: you could use ipfs-pack manifests to track the stuff that's not currently in the index. One entry point for all the info about ipfs-pack:
<daviddias>
Kubuxu: if you can set up a new livestream url
<daviddias>
then folks and drop the current one
<richardlitt>
Wasn't my loop, sorry about that!
<Kubuxu>
daviddias: not really
<Kubuxu>
I am recording it on my notebook
<Kubuxu>
I don't have OBS to YT there
<daviddias>
got it
<H3ndr1k[m]>
Hey, I set up an ipfs gateway for me and whoever is interested at https://ipfs.h3ndr1k.de/ - Would it make sense to apply some content security policy?
<kevina>
Kubuxu lgierth thanks, I will have to watch it later. Not really available now.
ygrek has joined #ipfs
<dignifiedquire>
daviddias: lgierth is it fixed now?
<daviddias>
dignifiedquire: perfect !:D
<daviddias>
thank you :)
<dignifiedquire>
sorry about that
<daviddias>
dignifiedquire: <3 it's ok
koalalorenzo has quit [Quit: Sto andando via]
cemerick has quit [Ping timeout: 255 seconds]
googlfvKI6B[m] has joined #ipfs
cemerick has joined #ipfs
<googlfvKI6B[m]>
I've finally found this goo.gl/fvKI6B
jadedctrl has quit [Remote host closed the connection]
<A124>
Is there any outlook on how chunking will be done in future? For example if you have gzip file and it has multiple members, it could be really handy splitting it also on member bounadry, so those could be used as separate files. Will the fixed chunker stay or will there be a smart way to decide how to chunk in general?
<lgierth>
googlfvKI6B[m]: we don't like spam here, soo please stop it.
sprint-helper has quit [Remote host closed the connection]
sprint-helper has joined #ipfs
<whyrusleeping>
ohai FrankPetrilli
sprint-helper1 has joined #ipfs
sprint-helper has quit [Write error: Broken pipe]
Remramm has quit [Changing host]
Remramm has joined #ipfs
Remramm has quit [Changing host]
Remramm has joined #ipfs
cemerick has quit [Quit: Leaving]
jkilpatr has joined #ipfs
palkeo has joined #ipfs
palkeo has joined #ipfs
palkeo has quit [Changing host]
espadrine has quit [Ping timeout: 240 seconds]
<A124>
Well, the chunking I mentioned is connected to those issues. WARCs and MEGAWARCs are multimember gzip files. Just not sure if the meta is a separate member, or only each record. Having meta separate would not invalidate the standard if it was not the case, but old WARCs could not be chunked as much.
<A124>
The idea is to be able to have alternate / made up tree structure using the same datasets, which in original form would be the (mega)warcs.
<A124>
Archivebot is what Archive Team does use to archive sites that are not humongous in size (hundreds GB max). Grabsite is a variant for other users. They are not the most efficient, but with some patches and using pypy (JIT) it speeds that up vastly.
<A124>
Not sure what to make out of all this, I will probably file the two chunker proposals.
<lgierth>
Kubuxu: cool :) consider by scepticism proven wrong :)
<lgierth>
whyrusleeping Kubuxu: i'm about to start coding on the relay thing for real, so if there's any substantial feedback feel welcome to voice it https://github.com/libp2p/go-libp2p/issues/82
maxlath has joined #ipfs
<lgierth>
that of course doesn't imply we can't change it later
<haad>
whyrusleeping: I did. That one I'm working on will be an example and tutorial for go-ipfs<->js-ipfs interop, with orbit-db for the dynamic stuff.
<A124>
Sweet.
<haad>
..and also an example how to use js-ipfs and bundle it with webpack
<Kubuxu>
deps
cemerick has quit [Ping timeout: 255 seconds]
cemerick has joined #ipfs
robattila256 has joined #ipfs
bastianilso_____ has joined #ipfs
bastianilso_____ has quit [Client Quit]
bastianilso_____ has joined #ipfs
bastianilso_____ is now known as bastianilso
cyanobacteria has quit [Ping timeout: 240 seconds]
structuralist has quit [Remote host closed the connection]
structuralist has joined #ipfs
structuralist has quit [Ping timeout: 248 seconds]
nunofmn has joined #ipfs
anewuser has quit [Quit: anewuser]
weetest has quit [Remote host closed the connection]
Encrypt has quit [Quit: Quit]
jkilpatr has quit [Ping timeout: 256 seconds]
structuralist has joined #ipfs
<Kubuxu>
whyrusleeping: any idea how to recover gx package from gosrc?
ckwaldon has joined #ipfs
mildred has quit [Quit: WeeChat 1.6]
<lgierth>
"recover"?
mildred has joined #ipfs
<Kubuxu>
I have package in my gosrc not in ipfs
<lgierth>
ipfs add -r doesn't help?
<Kubuxu>
nope
<lgierth>
different hash? hrm
<lgierth>
it should have the exact same hash...
<Kubuxu>
not really
<lgierth>
maybe it's a matter of gx-go rewrite?
<Kubuxu>
doesn't help either
<Kubuxu>
but doesn't matter
<lgierth>
did you only add the package dir itself, or the wrapper too?
<Kubuxu>
wrapper too
<Kubuxu>
gx-go rewrite formats imports
<Kubuxu>
maybe they were bad
<lgierth>
or unordered
<lgierth>
or un-gofmt'd
<lgierth>
mh. what about .gx/
<Kubuxu>
doesn't matter
<Kubuxu>
I will just create new pkg
pcre has joined #ipfs
jkilpatr has joined #ipfs
<kevina>
whyrusleeping: you around? I want to clear how I envision the conversion going now with the flatfs
Guest21966 has quit [Ping timeout: 260 seconds]
tmg has joined #ipfs
<whyrusleeping>
kevina: i'm here!
<whyrusleeping>
Kubuxu: gx-go uw and publish -f maybe?
<Kubuxu>
nah, recovered it from src already
<Kubuxu>
I thought it was /x/crypto
<Kubuxu>
it wasn't
<whyrusleeping>
oooo
wrouesnel has quit [Remote host closed the connection]
cemerick has quit [Ping timeout: 255 seconds]
<kevina>
whyrusleeping: so here is how I envision the conversion going...
<kevina>
1) You first "upgrade" the old repo by adding the sharding file
<kevina>
2) You then create the new datastore
<kevina>
3) you than "move" the contents from the old datastore to the new