<kyledrake>
jbenet Every time a site was updated it was triggering an update. Since the current job catches them all up, I can delete the other ones and it won't attempt them.
<jbenet>
ahh nice.
<kyledrake>
jbenet The comedy one is that someone dumped 25,000 files on a particular site (which is amazingly only about 50MB), which was some kind of randomly generated pile of stuff.
<kyledrake>
jbenet It's.. having some trouble with that one.
<jbenet>
oh man
<jbenet>
was each file generating more work?
<jbenet>
or is it just 25k files not so good?
<jbenet>
We should make a set of test cases like these and make sure ipfs does well on them.
<jbenet>
it's been on my long todo list
<kyledrake>
Yeah... I don't think people should really be uploading that much stuff in a free neocities site, so I'll probably just put a throttle on that
<kyledrake>
He generated a response page for every possible page.. so you enter a bunch of questions and then you get this http://criqwiz.neocities.org/InPaEnSo.htm
<kyledrake>
err for every possible set of answers to the questions. Never a dull moment :)
<kyledrake>
I will tell the archiver to skip sites with more than 1k files
antonio_l has left #ipfs [#ipfs]
<nell>
hi kyledrake
<nell>
and jbenet ^_^
<nell>
the metaverse project has been going very well
<nell>
alusion got x11 applications inside VR with in a very elegant solution
<kyledrake>
jbenet It's dropped 50% since I added these changes, I think it's going to be caught up by EOD tomorrow!
reit has joined #ipfs
<kyledrake>
actually probably by morning at this rate
Wallacoloo has joined #ipfs
Taek has quit [Quit: No Ping reply in 180 seconds.]
Taek has joined #ipfs
inconshreveable has quit [Ping timeout: 256 seconds]
chriscool has joined #ipfs
zabirauf_ has joined #ipfs
<ipfsbot>
[go-ipfs] chriscool pushed 1 new commit to use-go-sleep: http://git.io/vIny7
<ipfsbot>
go-ipfs/use-go-sleep cc276b5 Christian Couder: sharness: add missing bin/go-sleep...
zabirauf_ has quit [Ping timeout: 256 seconds]
domanic has joined #ipfs
<ipfsbot>
[go-ipfs] chriscool force-pushed use-go-sleep from cc276b5 to 14a3315: http://git.io/vIndU
<ipfsbot>
go-ipfs/use-go-sleep fd449af Christian Couder: test/Makefile: build go-sleep...
<ipfsbot>
go-ipfs/use-go-sleep 14a3315 Christian Couder: sharness/test-lib: use go-sleep to sleep...
<kyledrake>
jbenet whyrusleeping the import has caught up. An interesting stat: It's compressed 410,434 IPFS hashes into an impressive 65,429 blocks, which of course means it's also saving a lot of inodes too, and thus would also reduce network transfer metadata. This jogged my memory about Facebook's Haystack, which I think you might find an interesting read
<jbenet>
kyledrake hmm. since you just added a huge amount ofi'd love to _make sure_ the data is all there. wonder if you can take a subset and expand it out to diff it + make sure we handle it _exactly_ right.
<jbenet>
Encrypt o/
Wallacoloo has quit [Ping timeout: 265 seconds]
<jbenet>
nell that post is awesome
<Encrypt>
jbenet, I took the TOEFL yesterday
<Encrypt>
That was horrible :D
<jbenet>
Encrypt: congrats being done
<Encrypt>
Thanks
<Encrypt>
I hope I'll get more than 79/120
<Encrypt>
Otherwise, no Plymouth University next year... :S
<kyledrake>
jbenet it might be hard to compute a diff with it since it's a running system, but I could take one of the backups and try to repro from that on a different machine.
<kyledrake>
I'll take a look into that tomorrow
<jbenet>
maybe we can just make a test set that is similarly large
<jbenet>
i've no reason to think it would be any different
<kyledrake>
jbenet I think /dev/urandom into a couple million files between 5k and 300k would give you a similar story. Maybe throw in a couple comedy 1-2gb files for the full effect.
<kyledrake>
bonus points if your entropy pool reads you the riot act
<jbenet>
hahahahha
grncdr has quit [Quit: Leaving.]
<kyledrake>
IPFS would work pretty well for a CDN. You really just need an expire mechanism (time, remote API call or pubsub with local worker). You could run your own network if you didn't want to be a hippie and share.
<kyledrake>
Well, what is expiring, though? Wow. Okay I'm going to bed gnight
<ipfsbot>
[go-ipfs] jbenet deleted use-go-sleep at 14a3315: http://git.io/vIc4Q
<jbenet>
haha
<jbenet>
gnight kyledrake o/
<jbenet>
me too
* jbenet
suspends.
Encrypt has quit [Quit: Quitte]
cjdmax has joined #ipfs
anshukla_ has joined #ipfs
anshukla has quit [Ping timeout: 245 seconds]
anshukla_ has quit [Remote host closed the connection]
Encrypt has joined #ipfs
zabirauf_ has joined #ipfs
elima_ has joined #ipfs
chriscool has quit [Ping timeout: 264 seconds]
williamcotton has joined #ipfs
zabirauf_ has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
zabirauf_ has joined #ipfs
zabirauf_ has quit [Ping timeout: 256 seconds]
chriscool has joined #ipfs
williamcotton has quit [Ping timeout: 245 seconds]
grncdr has joined #ipfs
grncdr has quit [Client Quit]
grncdr has joined #ipfs
chriscool has quit [Ping timeout: 276 seconds]
bedeho2 has joined #ipfs
step21 is now known as step21_
bedeho has quit [Ping timeout: 255 seconds]
atrapado has joined #ipfs
pfraze has joined #ipfs
pfraze has quit [Remote host closed the connection]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
pfraze has joined #ipfs
chriscool has joined #ipfs
pfraze has quit [Remote host closed the connection]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
Encrypt has quit [Quit: Quitte]
chriscool has quit [Ping timeout: 276 seconds]
domanic has quit [Ping timeout: 272 seconds]
Bioblaze has quit [Remote host closed the connection]
silotis has quit [Remote host closed the connection]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
Bioblaze has joined #ipfs
silotis has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
www1 has quit [Ping timeout: 276 seconds]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
domanic has joined #ipfs
pfraze has joined #ipfs
www1 has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
rht__ has quit [Quit: Connection closed for inactivity]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
Encrypt has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<Encrypt>
Question: Do you have an example of a website published on IPFS?
<headbite>
yeah... give me a second I'll grab an updated link
notduncansmith has quit [Read error: Connection reset by peer]
inconshreveable has quit [Remote host closed the connection]
inconshreveable has joined #ipfs
* jbenet
misses berlin.
<jbenet>
i do have sun + coffee.
<kyledrake>
jbenet oh, it's pretty nice there in the summer, though. :)
<Encrypt>
jbenet, Where are you now? :)
<jbenet>
palo alto
<Encrypt>
Meh
<Encrypt>
Silicon Valley :x
<Encrypt>
I'm dreaming to go there
<Encrypt>
And it's surely beutiful
inconshreveable has quit [Remote host closed the connection]
<jbenet>
it's very nice. warm, sunny, friendly. lots of woods near by. but the best thing is the people + thinking
<Encrypt>
And Standford nearby \o/
<kyledrake>
Oh, I was referring to Berlin actually.
<jbenet>
kyledrake: yeah i was riffing off lgierth's "sun + beer" o/
<jbenet>
Encrypt: yep! make sure to visit Stanford. one of my favorite places.
<Encrypt>
jbenet, When I am rich then :P
<kyledrake>
Encrypt: Just go there. You don't need to pay to visit the campus.
<kyledrake>
jbenet I'm having quite the time getting make vendor to run. GOPATH is just any directory you want it to be, right?
<kyledrake>
godep: error while inspecting "/home/kyle/code/go-ipfs": directory "/home/kyle/code/go-ipfs" is outside source root "src"
<jbenet>
kyledrake: sort of-- go has this way of setting up everything where it puts _everything_ in the same path.
<jbenet>
it's sort of a good idea, but it's a huge hurdle cause they don't explain it well / clearly / help out.
<jbenet>
so, for example, my GOPATH is ~/go, so when i `go get github.com/ipfs/go-ipfs` it goes into `~/go/src/github.com/ipfs/go-ipfs`
<jbenet>
_then_ godeps is awful, so it will be very unhelpful when vendorig.
<jbenet>
need to get all the packages, which may be able to do by running (in go-ipfs/): `godep restore`
<Encrypt>
kyledrake, But to go to the USA, I have to buy a plane ticket
<Encrypt>
+ hotel
<jbenet>
so `godep restore && make vendor` may work.
<Encrypt>
And so on :P
<jbenet>
BUT we'll get rid of godeps soon.
<jbenet>
whyrusleeping is prototyping a vendor tool
<jbenet>
that stores everything in ipfs
<jbenet>
Encrypt: just get really good at hacking / programming and then people will give you money and/or fly you places
<Encrypt>
:D
<Encrypt>
I'm taking note of that :]
<kyledrake>
jbenet thanks for the explanation, it made more sense than the official one. I guess I just assumed there was a packaging system by now.
<jbenet>
Encrypt: also-- edx.org and coursera.org -- most classes from the best universities around the world are now offered for free. including Stanford, MIT, CMU, ETH Zurich, Caltech,
inconshreveable has joined #ipfs
<Encrypt>
Ok, I didn't know that!
<jbenet>
Encrypt: now more than every, you don't need a degree, just an internet connection + motivation + hard work.
<Encrypt>
Thanks! :)
inconshreveable has quit [Read error: Connection reset by peer]
inconshreveable has joined #ipfs
<kyledrake>
Encrypt: and just have fun. Otherwise it's the bad kind of work, and the bad kind of work sucks.
<Encrypt>
kyledrake, I'm already having fun ;)
<Encrypt>
I have lots of projects
<Encrypt>
Unfortunately, some are electronics projects which are really hard to achieve
<Encrypt>
Doing electronics is much harder than programming
<kyledrake>
There we go. Tests running.
<jbenet>
kyledrake: yay \o/ -- all this needs to get easier.
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<kyledrake>
For a fixed C from the original authors, I'll take it.
inconshreveable has quit [Ping timeout: 265 seconds]
grncdr has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<ipfsbot>
[go-ipfs] jbenet created dev from master (+0 new commits): http://git.io/vIW0F
<jbenet>
i do think we need something to merge into that doesn't push to the "releases"
<jbenet>
i am annoyed by the complication of master not being the thing people PR into
<jbenet>
it's hyper simple.
<jbenet>
deviation from that will add a lot of noise.
<tperson>
Ya
zabirauf_ has joined #ipfs
<tperson>
Would be cool if gobuilder was configurable and you could have it create builds for versions, and have a nightly off master.
<jbenet>
other solutions to consider are: (a) tag releases, link people to download builds only from tagged releases, (b) a separate repo for releases that pulls from go-ipfs, or something.
<tperson>
That way people understand that it's not 100% stable.
www1 has quit [Ping timeout: 265 seconds]
<tperson>
^ yep, I like the tagged releases.
<jbenet>
i like that it builds master on push, but i would like to mark it as not the _default_ thing people should download.
<jbenet>
then again, we control our links here: http://ipfs.io/docs/install so we can have a `stable` branch
<jbenet>
that we move up with every new tag
<jbenet>
and have gobuilder links to that.
<tperson>
Ah, it basically already does what I was thinking.
<tperson>
Ya, so instead of master we'd just point it to one of the v3.4 builds
<tperson>
though it would be neat to see a v3.x.y link.
<tperson>
So you didn't have to update the website for each new release and it simply pointed to the latest on the major release.
<jbenet>
right that's what the `latest` branch is for
<jbenet>
website links to `latest`
<jbenet>
latest is moved up with every tag.
<jbenet>
i think this is nicer than `dev into master`, because everyone's going to keep PRing into master.
<tperson>
daviddias: Have you made any headway on the multistreams stuff?
<ralphtheninja>
village with nodeschool, ruby and rust
<ralphtheninja>
sorry using ipfs channel for this, but I know some of you are going
<jbenet>
ralphtheninja no worries, it's welcome
<ralphtheninja>
cool :)
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<lgierth>
jbenet: if you set the github repo's default branch, PRs will go there
elima_ has joined #ipfs
<lgierth>
it depends on what role master should have
<lgierth>
the thing that people normally work with, or the stable thing
Wallacoloo has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
inconshreveable has joined #ipfs
www1 has quit [Ping timeout: 246 seconds]
elima_ has quit [Ping timeout: 265 seconds]
warner has quit [Read error: Connection reset by peer]
warner has joined #ipfs
Blame has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
Guest15 has joined #ipfs
Guest15 has quit [Client Quit]
<jbenet>
lgierth: yeah but it's not a full solution-- the default branch would then show up on the front page of the repo, so it will be just like substituting master + the confusion that people may send PRs with cli tools like `hub` assuming it's going to master.
zabirauf_ has joined #ipfs
domanic has quit [Ping timeout: 265 seconds]
aluchan has joined #ipfs
<aluchan>
thx for the mention jbenet ^_^
<aluchan>
new ghost in the shell is coming out soon :o
therealplato has quit [Read error: Connection reset by peer]
therealplato has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
ianopolous has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
grncdr has quit [Quit: Leaving.]
zabirauf_ has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
grncdr has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
Bioblaze has quit [Remote host closed the connection]
zabirauf_ has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
anshukla has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
zabirauf_ has quit [Quit: My Mac has gone to sleep. ZZZzzz…]