<edrex>
I'm putting my gateway behind nginx. Is it going to break some of the endpoint discovery stuff to split map /ipfs/ -> ipfs.mydomain/ and /ipns/ -> ipns.mydomain, or are there other endpoints besides /ipfs/, /ipns/?
<edrex>
s/split //
<richardlitt>
lgierth: sounds good to me!
<whyrusleeping>
edrex: if you do that, some relative links might not work
<whyrusleeping>
er, i think more likely absolute links wont work
Eudaimonstro has quit [Ping timeout: 240 seconds]
<edrex>
oh, makes sense
<whyrusleeping>
we tell people to link to '/ipfs/<hash>' whenever possible
<edrex>
Blame: wanting to add my IPFS gateway to providers.txt. I force https. It seems like for gateways https is useful, since browsers aren't doing any verification on their own.
<edrex>
but i guess if you don't trust the gateway's operator, it's moot.
<whyrusleeping>
voldial: interesting, so it uses essentially rabin fingerprinting and then does some fancy stuff to make those chunks smaller
<voldial>
whyrusleeping: looks like. I'm still chewing on it.
<whyrusleeping>
voldial: let me know what you think when you finish reading
<whyrusleeping>
i think we could benefit from it :)
<voldial>
whyrusleeping: Will do. Immediately I think of the tar importer, and hypothetical other archive importers. It seems obvious that running the chunker on the uncompressed data is important. I started looking at it today because I'm interested in ways to take an archive and store it efficently, but keep the ability (save the transform) to reconstruct the orig tar.
JohnClare has joined #ipfs
shea256 has quit [Remote host closed the connection]
JohnClare has quit [Ping timeout: 252 seconds]
<lgierth>
richardlitt: can't push
<whyrusleeping>
voldial: ah, i'm glad you like the tar importer :)
<whyrusleeping>
i used CDC (rabin) on the actual file data
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
pfraze has quit [Remote host closed the connection]
pfraze has joined #ipfs
therealplato has quit [Quit: Leaving.]
therealplato has joined #ipfs
shea256 has joined #ipfs
hellertime has quit [Ping timeout: 252 seconds]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<larsks>
Why doesn't /refs just return a JSON list, rather than a bunch of concatenated JSON objects?
<larsks>
whyrusleeping: Yes. I guess I was asking, "why is it a JSON stream"?
Quiark has joined #ipfs
<larsks>
Is it something that may be expensive or slow to compute, so results are returned as avaialble?
bengl has joined #ipfs
<whyrusleeping>
yeah, refs are either read from the disk, or fetched over the network
<whyrusleeping>
so it takes time to get them, and the list may also be larger than your memory
voxelot has joined #ipfs
shea256 has joined #ipfs
patcon has quit [Ping timeout: 256 seconds]
shea256 has quit [Ping timeout: 250 seconds]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<larsks>
Okay, makes sense.
hellertime has joined #ipfs
CarlWeathers has joined #ipfs
shea256 has joined #ipfs
HostFat has joined #ipfs
<HostFat>
maybe it can be useful if you check the hash of the image before sending it to your servers
<HostFat>
to avoid the upload, if there is already
shea256 has quit [Read error: Connection reset by peer]
shea256 has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
simonv3 has quit [Quit: Connection closed for inactivity]
<voldial>
whyrusleeping: I do! Finished reading, in a nutshell (my very rudimentary understanding) you get a ~1.5x improvement in dedup (with index overhead included) vs standard CDC. Worst case is same as CDC. CPU overhead is worse(not sure how much), but the authors discuss how to fix that. They appear to first apply CDC and feed the chunk candidates rabin fingerprints into a pre-filter that only selects some of them
<voldial>
(rabins mod 32 = 1 in this case) chunks that make it through are then further filtered by frequency counting using a set of n(3) bloom filters, each rabin added randomly to one bloom each time it makes it through the pre-filter. Once a rabin is in all of the bloom filters, it's then considered worth counting the frequency of. This freq estimation is then used to re-chunk some of the large first pass CDC chunks
<voldial>
said another way, they apply CDC and while sliding the window, they put the fingerprints through this pre-filter -> 3x bloom filter, those that make it out are then frequency counted. Then the CDC chunks that can be further chunked for profit are.
patcon has joined #ipfs
crossdiver has quit [Quit: Connection closed for inactivity]
notduncansmith has joined #ipfs
<voxelot>
using node i can recurs through 374 blocks to pull file type in half a second
notduncansmith has quit [Read error: Connection reset by peer]
<voxelot>
too bad the blocks only store mime type application/octet-stream, no way around pulling data down from cat
<voxelot>
for indexing purposes
JohnClare has joined #ipfs
HostFat has quit [Quit: Leaving]
JohnClare has quit [Ping timeout: 246 seconds]
hellertime has quit [Ping timeout: 252 seconds]
hellertime has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
captain_morgan has quit [Ping timeout: 246 seconds]
machrider has joined #ipfs
feross has joined #ipfs
rjeli has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
captain_morgan has joined #ipfs
<whyrusleeping>
zignig: heyo
<noffle>
Q: since peers only advertise the hashes of the blocks they want, is it implied that there's no way to learn what blocks a given peer owns (beyond explicitly asking for them)?
<whyrusleeping>
noffle: true
sseagull has quit [Quit: leaving]
machrider has quit [Ping timeout: 246 seconds]
captain_morgan has quit [Ping timeout: 246 seconds]
rjeli has quit [Ping timeout: 246 seconds]
Xe is now known as jneurlc
FunkyELF has quit [Ping timeout: 246 seconds]
jneurlc is now known as jneruld
nicolagreco has joined #ipfs
<zignig>
whyrusleeping: output from nostril : Qme56TpUKgNTv1JFYGGDLpN2565UG3skzn9L8B1BNZxATk
<noffle>
whyrusleeping: but doesn't the act of pinning adverise what you have to DHTs?
<whyrusleeping>
zignig: whats nostril?
<whyrusleeping>
noffle: yeah, you could listen to the dht to get an idea of what people have
<whyrusleeping>
but you cant just ask a node 'what blocks do you have?'
<zignig>
whyrusleeping: it's a node/ipns scanner collator.
<zignig>
scans current peers , see's if they have ipns entries and puts them into a database.
jneruld is now known as Xe
<whyrusleeping>
oooooooooh, thats cool!
<whyrusleeping>
i like :)
<zignig>
i'm working on republishing them into a ipns ref and get it to probe for other nostrils.
<whyrusleeping>
i'm sooooooo ready to work on ipns and make it really nice.....
<zignig>
whyrusleeping: pleeeeez !
<whyrusleeping>
i want to!!
<noffle>
whyrusleeping: got it! :) does sharing objects only happen for a node once it pins it? or does anything downloaded locally (via e.g. ipfs get) too?
<whyrusleeping>
if i could, i would just make a bunch of coffee and stay up all night to do it!
<whyrusleeping>
but i'm waiting on jbenet for a few things first
<zignig>
whyrusleeping: indeed, slowly and properly is better.
<zignig>
whyrusleeping: current dodgy halfbaked version : Qme2hZANUuz8PtFphXEKA6V7zU9umhFHNMtu6QSV1nN7zQ
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<jbenet>
hey nicolagreco! whyrusleeping and i are, well we've a couple things to merge before we can land some improvements
<jbenet>
(dependencies type of thing)
<nicolagreco>
nice to e-meet you
<nicolagreco>
I should have come here long time ago
<nicolagreco>
(feross introduced me to this more than a year ago)
<jbenet>
ah, cool :) yeah, welcome
<jbenet>
(and thanks for the intro feross!)
<jbenet>
is the ipfs.pics author in here?
<jbenet>
and, nicolagreco yeah let's talk sometime this week. up for a video call wed or thu afternoons.
<kyledrake>
nicolagreco welcome!
<nicolagreco>
so, I have basically brought your article kyledrake to the attention of many, and it is now a point of reference when I discuss this topic
<nicolagreco>
(sorry for being off-topic)
JohnClare has joined #ipfs
<nicolagreco>
I actually wanted to ask if you have seen the move of onename from namecoin to bitcoin and if there was something interesting for ipns
caseorga_ has joined #ipfs
caseorga_ has quit [Client Quit]
<kyledrake>
nicolagreco Ah, interesting. I hope it's useful for you.
<jbenet>
not off topic at all. this is a very organic place
ryepdx has quit [Ping timeout: 250 seconds]
<jbenet>
nicolagreco: yep, i know the onename folks well. i already planned on integrating onename as another namesys resolver, but for me the bitcoin/namecoin distinction wasn't that significant.
<nicolagreco>
ok, thing is that I am very ignorant on how you are planning ipns
<jbenet>
i think muneeb and ryan hang out here sometimes, so you may catch them if you idle
JohnClare has quit [Ping timeout: 252 seconds]
<jbenet>
so, at its core, ipns is a pki name system. therefore it circumvents the core problem of naming: consensus.
<nicolagreco>
but I am very interested to understand
<jbenet>
the ideas are all mazieres', see his SFS phd thesis. (i think he was working with frans k.)
ryepdx has joined #ipfs
<jbenet>
but we adapted it to put it on another idea from that pair: dhts. though really, not just a dht. the idea of ipns is that it (a) does PKI naming, and (b) distributes its name records over whatever "content routing system" you pick-- e.g. a dht, or mdns, or a regular ht, or WoT, etc.
<jbenet>
bbiab.
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<nicolagreco>
jbenet: what do you mean you "pick", can two "pick" two different content routing system?
jamescarlyle has joined #ipfs
Tv` has quit [Quit: Connection closed for inactivity]
pfraze has quit [Remote host closed the connection]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
pfraze has joined #ipfs
<ipfsbot>
[go-ipfs] jbenet pushed 1 new commit to master: http://git.io/vZ9jW
<ipfsbot>
go-ipfs/master df861af Juan Benet: Merge pull request #1672 from ipfs/feat/unwant...
<jbenet>
nicolagreco: yes, think of it like being on two intranets in that case. there are situations where users are on disjoint networks on the internet. but in general its more about overlapping on the same content, but accessing it with different protocols because of a difference in resource capabilities. (e.g. a phone is not a good dht node, but is an ok dht
<jbenet>
client)
<nicolagreco>
ok, I got just now the parallelism with the routers/ip
akhavr has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
JohnClare has joined #ipfs
pfraze has quit [Remote host closed the connection]
nessence has quit []
JohnClare has quit [Ping timeout: 264 seconds]
nicolagreco has quit [Quit: nicolagreco]
<davidar>
jbenet, but phones can make very good nodes while charging on WiFi, as someone pointed out recently
amstocker has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
pfraze has joined #ipfs
amstocker has quit [Quit: Leaving]
M-amstocker has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
captain_morgan has quit [Ping timeout: 240 seconds]
Pyrotherm has joined #ipfs
captain_morgan has joined #ipfs
Eudaimonstro has joined #ipfs
steve_ has quit [Ping timeout: 268 seconds]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
pfraze has quit [Remote host closed the connection]
JohnClare has joined #ipfs
JohnClare has quit [Ping timeout: 250 seconds]
chriscool has quit [Ping timeout: 272 seconds]
AtnNn has quit [Ping timeout: 255 seconds]
AtnNn has joined #ipfs
interfect has quit [Remote host closed the connection]
Hello_ has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
Eudaimonstro has quit [Remote host closed the connection]
Hello_ has quit [Quit: Page closed]
nomoremo_ has joined #ipfs
Guest73396 has joined #ipfs
voxelot has quit [Ping timeout: 256 seconds]
jamescarlyle has joined #ipfs
shea256 has quit [Read error: Connection reset by peer]
jamescarlyle has quit [Remote host closed the connection]
jamescarlyle has joined #ipfs
jamescarlyle has quit [Remote host closed the connection]
shea256 has joined #ipfs
jamescarlyle has joined #ipfs
jamescarlyle has quit [Remote host closed the connection]
mquandalle has quit [Quit: Connection closed for inactivity]
rendar has joined #ipfs
reit has quit [Quit: Leaving]
nomoremo_ has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
reit has joined #ipfs
<fleeky>
ooo ipfs.pics is cool
<borgtu>
Hello! I installed ipfs on my server a couple of days ago and now i've got an abuse message from my hosting provide complaining about portscans on RFC1918 ip ranges
<borgtu>
i'm not sure if you care but I just thought you should know that could be problematic in some enviroments :)
zen|merge has joined #ipfs
nomoremo_ has joined #ipfs
<reit>
i've had that happen too, i had to assure them ipfs wasn't a botnet
nomoremo_ has quit [Client Quit]
f[x] has joined #ipfs
<borgtu>
reit: okay
<borgtu>
I kinda understand why they are concerned... Is there some way to disable this kind of behaviour?
<davidar>
lgierth has had problems with that too, so it's a known issue
<davidar>
borgtu: enable filters
<borgtu>
davidar: okay thanks!
<zorun>
ipfs tries to contact other ipfs peers over their advertised RFC1918 addresses
<zorun>
which is good if you are behind the same NAT, but otherwise...
wasabiiii has quit [Quit: Leaving.]
<borgtu>
zorun: yeah, i thought that might be the reason...
<borgtu>
davidar: Is it possible to mention this explicitly in the installation notes? I'll be happy to help out if I can somehow.
<davidar>
borgtu, open an issue in ipfs/faq
<borgtu>
davidar: Will do! Thanks..
<borgtu>
great project btw! I've been ready about it all weekend :)
jamescarlyle has quit [Remote host closed the connection]
<daviddias>
jbenet richardlitt do you know what is the IRC + Github handle of Nathan (who joined us yesterday at the sprint meeting?)
slothbag has joined #ipfs
magneto1 has quit [Ping timeout: 250 seconds]
<ipfsbot>
[go-ipfs] klauspost opened pull request #1705: Fix build on Windows. (master...exclude-fuse-on-windows) http://git.io/vZQTb
Quiark has quit [Ping timeout: 250 seconds]
jamescarlyle has joined #ipfs
reit has quit [Quit: Bye]
reit has joined #ipfs
edrex has quit [*.net *.split]
silotis has quit [*.net *.split]
giodamelio has quit [*.net *.split]
jjanzic has quit [*.net *.split]
rschulman has quit [*.net *.split]
giodamelio has joined #ipfs
jjanzic has joined #ipfs
edrex has joined #ipfs
silotis has joined #ipfs
JasonWoof has quit [Ping timeout: 255 seconds]
ei-slackbot-ipfs has quit [Remote host closed the connection]
ei-slackbot-ipfs has joined #ipfs
bedeho has quit [Ping timeout: 268 seconds]
dignifiedquire has joined #ipfs
atomotic has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
jamescarlyle has quit [Remote host closed the connection]
jamescarlyle has joined #ipfs
magneto1 has joined #ipfs
kanzure has joined #ipfs
phrearch has joined #ipfs
<phrearch>
hi
<davidar>
hi phrearch
<jamescarlyle>
i've got a question for anyone building apps on top of ipfs: what level of the stack are you building on? i.e. using the file/directory api of ipfs.add and ipfs.cat, or the object api and setting "data" and "links" explicitly
<jamescarlyle>
hope this question makes sense
jamescarlyle has quit [Remote host closed the connection]
atomotic has joined #ipfs
hellertime has joined #ipfs
<daviddias>
whyrusleeping starting to think that a I'm getting addicted to a parallel of jbenet's multi* solution, the abstract-* everything :D
<jamescarlyle>
davidar: take ipfs.pics for example; looks like the service has a server connected to ipfs daemon through an api: I wondered if they were adding objects "manually" by constructing some "data" and "links" json and then using ipfs object put, or treating ipfs as a file store
<blame>
Morning folks
<jamescarlyle>
morning
<davidar>
jamescarlyle: i think they're just treating it like a file store
<davidar>
wouldn't surprise me if they're just manually calling out to ipfs add
<jamescarlyle>
thanks. I was thinking of prototyping a wiki on top - was wondering whether to construct the "data" and "links" myself
<jamescarlyle>
'cause as I understand it, ipfs could be used as a merkle dag persistence mechanism, without any reference to files and directories
qqueue has quit [Ping timeout: 252 seconds]
qqueue has joined #ipfs
<blame>
I'm going to be focusing on the python api this morning
pfraze has joined #ipfs
<cryptix>
jamescarlyle: it is - you dont have to use the unixfs semantics
<cryptix>
jamescarlyle: take a look at the 'ipfs object *' commands - you can also pipe json into them with --inputenc=json
<jamescarlyle>
cryptix: thanks, answered my qu perfectly. just sounding out how others are using it, don't want to go down a blind alley
<davidar>
jamescarlyle: i think federated wiki is already doing that
<jamescarlyle>
yes, i've been coercing stuff using ipfs object * and equiv
f[x] has joined #ipfs
<lgierth>
davidar: could you post your sprint update to the sprint issue?
<jamescarlyle>
davidar: do you have a contact / name for federated wiki?
<achin>
if there are no objections to the idea, i might try to teaching "ipfs ls" about a recursive flag
<whyrusleeping>
achin: hrm... we might have some squabbles on the naming, but i've no objection to the functionality
josephdelong has joined #ipfs
<reit>
whyrusleeping: saw that, pretty neat but i'm thinking more in terms of what that markdown viewer does, how it properly renders it out and so on
<reit>
as in, syntax highlighting
<whyrusleeping>
reit: ah, yeah. i dont know that anyone has done that yet
<cryptix>
hey whyrusleeping :)
<whyrusleeping>
cryptix: long time no see!
<cryptix>
yeesss way to long
<whyrusleeping>
hows it been?
<cryptix>
hard to describe
<cryptix>
hell-ish but without the good weather
<cryptix>
no - im fine just a lot to do
<whyrusleeping>
mmmm, come to america. we have good food and good weather
<whyrusleeping>
most of the time
<cryptix>
i made some tests for go-ipfs-api
<davidar>
cryptix, don't listen to him, Australia is better! ☺
<cryptix>
they use the example data that is added on init and should work on travis too with the docker image i use for the git-remote tests
<cryptix>
davidar: he lost me at 'good food' ;)
<davidar>
Current temperature is 16C here
<davidar>
Haha
<whyrusleeping>
we have this hawaiian korean barbeque tacos in a food truck
HostFat has joined #ipfs
<whyrusleeping>
its the best thing you can ever imaginge
<whyrusleeping>
ask daviddias
<davidar>
We've got Dagwood dogs, which are kind of disgusting really
<davidar>
Pavlova is good though ☺
<whyrusleeping>
isnt pavlova just whipped cream and strawberries?
<davidar>
It's like a giant soft meringue
<davidar>
You put the fruit and whipped cream on top
<josephdelong>
Hey does anyone know what the DAG in Merkel DAG stands for?
<davidar>
Directed acyclic graph
<achin>
the best kind of graph!
<whyrusleeping>
cryptix: i'll take a look at the go-ipfs-api tests today
<josephdelong>
Ahhhh! now it makes more sense
<whyrusleeping>
for the longest time i thought the 'A' stood for acryllic, as in like, porcelain.
<josephdelong>
Thank you davidar
<josephdelong>
lol
<davidar>
achin, you are technically correct, the best kind of correct
<achin>
:D
<josephdelong>
I have been looking at ZeroNet it seems like it is similar but is missing a lot of the bells and whistles of IPFS
<achin>
whyrusleeping: are you on the east coast or the west coast?
magneto1 has quit [Ping timeout: 260 seconds]
<larsks>
How do I make the `ipfs` command use an API other than localhost:5001?
<whyrusleeping>
achin: west coast best coast!
rekoil has joined #ipfs
<whyrusleeping>
(sorry rschulman_ )
<achin>
lies!
<rschulman_>
whatevs
<achin>
pretty much your entire coast is on fire
<whyrusleeping>
josephdelong: pretty sure zeronet doesnt use the awesome power of the merkledag eitehr
<whyrusleeping>
achin: its not as bad anymore...
<achin>
meanwhile, here in new england we are slipping into a very beautiful autumn
f[x] has quit [Ping timeout: 250 seconds]
<josephdelong>
This might be too complicated for an IRC conversation, but how do you traverse a merkeldag tree?
<achin>
parent to child!
<josephdelong>
what happens if the tree becomes lop sided? Does it just take more iterations?
<achin>
a DAG isn't like a balanced tree, so i'm not sure the concept of "lop-sided" applies
<achin>
think of it like a directory structure, except there is no "..". that is, there is no way to get into the previous directory
<josephdelong>
interesting, it sounds like I have more reading to do. Thanks achin
tsenior`` has quit [Ping timeout: 244 seconds]
shea256 has quit [Read error: Connection reset by peer]
<josephdelong>
It also looks like ZeroNet doesn’t have any type of caching scheme. Which means someone has to actively host a file.
shea256 has joined #ipfs
* whyrusleeping
has so many github notifications to get through
<achin>
the same is true with ipfs, as i understand it. if you want to be assured that your content is available, you have to host it yourself (by pinning the content)
<josephdelong>
I was under the impression that if someone accessed it the software cached it and began hosting it
<achin>
it would be cached, yes, but it's not guarenteed that it'll be cached forever
<josephdelong>
True
<josephdelong>
I think the caching is interesting for high traffic high availability websites
<josephdelong>
No more “hug of death"
<whyrusleeping>
josephdelong: yesss, no more hug of death :)
<josephdelong>
Dat ping tho. :)
magneto1 has joined #ipfs
shea256 has quit [Ping timeout: 268 seconds]
<cryptix>
okay.. who here runs <peer.ID YHfAsK>? i have 30 pages full of this in my logs: QmQDpWGxKPeCXvtSKhUSyeyLTiqWAW1Ui8BisLG1ER6JpM (cc whyrusleeping)
<whyrusleeping>
lol, thats interesting
<whyrusleeping>
i wonder whats happening there...
<solarisfire>
Not my node :-)
<solarisfire>
Trying to ipfs add a very large folder atm, filled with 4,000 images, and it's just hanging... :(
<solarisfire>
It's just a node that acts as a gateway between ipfs and http
<wh1ter0se>
imagine if google chrome/firefox etc had a plugin that let you utilise ipfs as a cdn
<necro666>
i would rather use it as proxy but you have your reasons for not to do it for sure :)
<necro666>
and how much space do i need to reserve on my hdd?
<achin>
the latest diag-net report shows 210 nodes: QmesTCvTdWSYfZe5qefBHaC2TPuSLwVyHR2yx6hPZATjKd
akhavr has quit [Ping timeout: 250 seconds]
<wh1ter0se>
id happily run a node, but im worried about people abusing it for illegal content or such, and id rather not have that
akhavr has joined #ipfs
<necro666>
yeah, i am not sure about anonymity on this network :D
<necro666>
sorry guys, i will stick to tor/freenet for now :D
notduncansmith has joined #ipfs
<achin>
i don't believe anonymous browsing is a usecase that ipfs is trying to support. am i wrong?
notduncansmith has quit [Read error: Connection reset by peer]
<necro666>
dunno, i was invited here just few minutes ago and someone was talking about freenet
<necro666>
so i assumie it may be a thing :D
<solarisfire>
Yeah, I don't think it's to support anonymity...
<solarisfire>
Either way, on TOR you don't run a tor exit node if you're scared of illegal content XD
wopi has quit [Read error: Connection reset by peer]
wopi has joined #ipfs
<necro666>
i am not sure if this is illegal. Yes, ok, there was childporn sent from my IP, ok
<wh1ter0se>
ipfs != tor/freenet... completely different use-case
<whyrusleeping>
wh1ter0se: the only content that will be on your nodes is the content you request
<necro666>
but is it said i did it?
<solarisfire>
Well, yeah...
<wh1ter0se>
whyrusleeping, ok, but for instance, ipfs.pics which went up today (which i ntoiced on /r/bitcoin on reddit).. allow uploading of images. Seems like a dangerous thing for them to do
<necro666>
no, someone was using my network
<necro666>
they can tell it may be me, but not for sure
<whyrusleeping>
(answers to a lot of these questions are there, if they need elaboration, feel free to comment on them)
shea256 has joined #ipfs
<wh1ter0se>
so if you run a get on your own ipfs node (or whatever command it is).. then you have a copy of it?
<achin>
that's a neat use of github issues for an FAQ. i like it
<achin>
yes, the "ipfs get" command is used to download copied of data
<wh1ter0se>
so, only if someone has "get" your hash, wll it survive if ur host goes down? or does it propigate in any other way?
<achin>
if other nodes have requested the file too, they might also have copied. but if you were the one to add the content to ipfs in the first place, it's possible that only you have the content, and if you node does down, it will become unavailable
<whyrusleeping>
cryptix: ping
besenwesen is now known as meskarune17b
<necro666>
well, so i copy content from nodes i am visiting?
<whyrusleeping>
wh1ter0se: yeah, ipfs.pics will have to filter and moderate the content that gets uploaded to them
<necro666>
how to tell if it was changes and I am not looking at old version
<whyrusleeping>
but normal nodes dont have that functionality
<whyrusleeping>
that just like you allowing people to ftp to your computer
<whyrusleeping>
you have to make sure the content people put on it is stuff youre okay with
sseagull has joined #ipfs
<achin>
your local node will cache content that you download (with 'ipfs get'). and you can also ask your local node make sure the data is not deleted by pinning it (with 'ipfs pin'). otherwise cached data can be deleted by a garbage collection step
<wh1ter0se>
achin, and will your node automatically redownload updated versions of it?
<reit>
what's the difference between example-viewer and markdown viewer?
<whyrusleeping>
reit: theyre probably the same thing, the example viewer is probably just a 'branded' version of the markdown viewer
<reit>
ah, yeah that makes sense now
<wh1ter0se>
so i assume gateway.ipfs.io and whatever else is similar, copy the hash to their own nodes when you enter the url?
meskarune17b is now known as besenwesen
<wh1ter0se>
achin, intereting that it talks about Chains. Freenet has a similar concept, but uses "versions" - so after the freenet key, is a version number
josephdelong has quit [Quit: josephdelong]
phrearch has quit [Quit: Ik ga weg]
Tv` has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<achin>
i'm sorry, i don't know about freenet, so i can't help compare the two
<wh1ter0se>
thats ok. was just thinking
<wh1ter0se>
so, is there a way to trace the origin node of a hash?
shea256 has quit [Read error: Connection timed out]
<achin>
i don't believe ipfs makes a distinction between a node that has a copy of a block and a node that has pinned a block
<whyrusleeping>
wh1ter0se: 'ipfs dht findprovs <hash>' will show who has that hash locally
<whyrusleeping>
you cant know who created an object
bedeho has joined #ipfs
<achin>
here's a test you can do with findprovs: create a file with some unique content and add it to ipfs. run findprovs on that hash and see that only you have a copy. then reqest that hash via the ipfs gateway. run findprovs again and you'll see that some new nodes have a copy
<solarisfire>
This kinda sucks...
<solarisfire>
ERRO[16:45:48:000] open /home/solarisfire/.ipfs/blocks/1220182e/put-391140217: too many open files module=commands/http
<whyrusleeping>
solarisfire: default limit on linux machines is 1024
<whyrusleeping>
this is a known problem that i'm working on
<whyrusleeping>
we didnt notice it until the network started growing larger
<wh1ter0se>
whyrusleeping, can you do a ulimit when u run the app?
<whyrusleeping>
wh1ter0se: you mean have the ipfs process make a ulimit syscall?
<wh1ter0se>
is that possible? maybe if it can start as root then ulimit then chuser
<solarisfire>
ulimit says unlimited....
<solarisfire>
still getting the error even after changing the conf file.
<whyrusleeping>
solarisfire: ulimit -n
<achin>
i would much perfer that ipfs stay away from root or automatically setting ulimits
<whyrusleeping>
wh1ter0se: its difficult to do cross platform
shea256_ has joined #ipfs
<whyrusleeping>
npm just tells their users to expand the ulimit themselves before running npm commands
<whyrusleeping>
ideally, we just fix our issue and use fewer file descriptors
shea256 has quit [Remote host closed the connection]
<whyrusleeping>
achin: agreed.
danslo has quit [Remote host closed the connection]
<nomoremoney>
daviddias, just to clarify, the difference between node-ipfs and node-ipfs-api is that node-ipfs is an implementation of ipfs while nods-ipfs-api is a client library?
<daviddias>
nomoremoney: exactly
<nomoremoney>
cool, thanks
<daviddias>
no problem :)
chiui has quit [Ping timeout: 244 seconds]
<solarisfire>
WOW
<solarisfire>
ERRO[16:58:38:000] open /home/solarisfire/.ipfs/blocks/122001bb/put-883531639: too many open files module=commands/http
<solarisfire>
solarisfire@ipfs:~/imgur$ ulimit -n
<solarisfire>
50000
<pjz>
whyrusleeping: is there an in-place filestore yet?
<pjz>
whyrusleeping: though I guess I should maybe ask 'will there be' as well
<solarisfire>
There's only 4411 files, is it opening them 11+ times each???
<pjz>
solarisfire: directories also take filehandles maybe?
<pjz>
solarisfire: sockets too
<achin>
for every peer in "ipfs swarm peers" there is at least 1 connection, up to 10 i think
<wh1ter0se>
maybe a virtual fs inside cached files is the answer?
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<whyrusleeping>
pjz: no and probably
<whyrusleeping>
yeah, the issue is that when we are dialing new peers, we dial up to ten addresses concurrently
<solarisfire>
Yeah it's all network connections...
<solarisfire>
Maybe... A project like this shouldn't expect people to alter a ton of OS configuration just for it to work though... It should just work...
wasabiiii1 has joined #ipfs
<wh1ter0se>
valid point
wasabiiii has quit [Ping timeout: 255 seconds]
hamster211 has joined #ipfs
Guest73396 has quit [Ping timeout: 250 seconds]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
Guest73396 has joined #ipfs
Guest73396 has quit [Ping timeout: 265 seconds]
<edrex>
solarisfire: guess it's early for adding that kind of ornamentation
<solarisfire>
Yeah it's still alpha remember :D
<edrex>
do I need to expose TCP:4001 (the swarm port) in order to serve blocks to other nodes?
<wh1ter0se>
how young is this project?
<solarisfire>
No, never expose 4001
<whyrusleeping>
you can expose 4001 if your router doesnt support upnp
j0hnsmith has quit [Ping timeout: 246 seconds]
<solarisfire>
Really? I got told not to for security reasons...
<whyrusleeping>
thats 5001
<whyrusleeping>
5001 is the priveleged api port
<solarisfire>
oh, lol.
<solarisfire>
cool.
j0hnsmith has joined #ipfs
<solarisfire>
But ipfs handles nat traversal itself?
<whyrusleeping>
yeap
<solarisfire>
cool.
wasabiiii1 has quit [Quit: Leaving.]
<edrex>
i have upnp turned off, prefer manually enabling. Although, that makes me think, would be cool if miniupnpd could queue up port forward requests for approval... maybe with a page in the openwrt gui..
<edrex>
but then, we're all going to be running ipv6 next year anyway right?
<solarisfire>
next year? I'm already running it...
<edrex>
well yeah me too, because my ISP supports it. but i have yet to find a coffee shop with it turned on
<edrex>
so tunnels
<whyrusleeping>
edrex: youll have to pry my ipv4 addrs from my cold dead fingers
<solarisfire>
whyrusleeping: Is the webui still under active development?
<whyrusleeping>
solarisfire: yes
<solarisfire>
Cool, the git repo for it seems a littie quiet...
<solarisfire>
*little
<edrex>
whyrusleeping: thanks. just curious, if I don't have 4001 exposed can I still serve blocks to nodes which do?
<whyrusleeping>
edrex: not having 4001 open means nodes cant dial to you
<whyrusleeping>
you can still dial out just fine
<edrex>
nat traversal is tricky. sad to see every p2p project implementing it by hand.
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<whyrusleeping>
edrex: we're hoping to rip out our p2p networking lib and make it so other people can use it easily
<whyrusleeping>
the node impl of ipfs is starting this way
<edrex>
whyrusleeping: as a C lib?
<edrex>
oh, you mean the node impl is starting with a JS impl of a p2p lib factored out
<jbenet>
no, from scratch. needs to run in a browser
<jbenet>
yep
simonv3 has joined #ipfs
j0hnsmith has quit [Quit: j0hnsmith]
<wh1ter0se>
whats the ulimit in javascript :p
<Vyl>
Bring on ipv6.
notduncansmith has joined #ipfs
akhavr has quit [Quit: akhavr]
notduncansmith has quit [Read error: Connection reset by peer]
<solarisfire>
wh1ter0se: in a browser, or in something like node.js?
<wh1ter0se>
i cant imagine javascript has a concept of ulimit really
<ipfsbot>
[go-ipfs] jbenet deleted feat/extract-logging at e5a2896: http://git.io/vZ5Hs
<solarisfire>
Well running on V8 inside chrome it can't even touch the filesystem...
<jbenet>
no, but the browser is a platform, like any other.
<jbenet>
it's an OS, really. it has a networking stack, a fs (local storage), and more.
<solarisfire>
node.js would still be running as a linux process... So would be limited by ulimit when running on a linux box.
<jbenet>
and yeah node.js is still on regular os. (distinction between running in the os and running on the browser(os) )
<wh1ter0se>
local storage?
<solarisfire>
Hmmm but most browsers have limitations too...
shea256_ has quit [Remote host closed the connection]
<Vyl>
What about the path bit?
nicolagreco has quit [Client Quit]
<jbenet>
edrex: we will, it's not there yet. right now it sort of does that because latencies in updating requests will just get to them faster, but not yet.
<jbenet>
Vyl: path bit?
<edrex>
this is the sort of fiddly, detailed stuff that every p2p app needs
<jbenet>
edrex: yeah exactly. we want to solve it really well, too and modularize it so people can use it elsewhere.
<josephdelong>
tree fiddly?
<jbenet>
have some good ideas using state of the art bandwidth + latency estimators
<edrex>
but at the same time, the requirements are slightly different for each app, and i think app devs are afraid of having to conform to an interface that doesn't perfectly fit
<jbenet>
edrex: yes, but i mean look at http and json. both are massive successes, and neither "perfectly fit" tons of the things they're used for.
<jbenet>
it has to be _great enough_ to be worth using for its strengths, and be able to mitigate/fix the weaknesses yourself.
nicolagreco has joined #ipfs
<edrex>
dig it
<Vyl>
I get the idea of putting a file in and getting it back by hash,
<Vyl>
But I don't understand the more sophisticated aspects yet.
<Vyl>
I did put QmdjMSVakZT3tDh62i1SH5incGaCznVS2XcMUFFfZTNu5z in, just to make sure I had that much right.
nicolagreco has quit [Client Quit]
notduncansmith has joined #ipfs
nicolagreco has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<Vyl>
So I'm only handling blocks, not objects?
shea256 has joined #ipfs
shea256 has quit [Remote host closed the connection]
nicolagreco_ has joined #ipfs
nicolagreco has quit [Ping timeout: 246 seconds]
nicolagreco_ is now known as nicolagreco
nicolagreco has quit [Read error: Connection reset by peer]
nicolagreco has joined #ipfs
nicolagreco has quit [Client Quit]
magneto1 has joined #ipfs
<Vyl>
What exactly is an object?
<jbenet>
Vyl: have you read the ipfs paper?
<Vyl>
I have, but I'm still getting my head around it.
<jbenet>
take a look at the latter sections around objects. that part is a bit ill-defined still in the paper, as we've discovered more, but it's still following in that model
wasabiiii has joined #ipfs
<Vyl>
As well as I can follow, objects don't actually contain very much data in themselves, but serve as indexes and stores of metadata for blocks? Where a block can actually be a list of pointers to smaller blocks that compose it.
<jbenet>
"block" is a nice word for talking about moving opaque chunks of data.
<jbenet>
object is one level up the abstraction ladder.
<Vyl>
Ok. I've been through this before when I designed an object storage filesystem...
<jbenet>
"block" is in the wire/disk. "object" is in the algorithms using them.
<Vyl>
Got to confused with blocks and files and objects, I ended up calling the objects 'meatballs' and the metadata structure 'sauce.' The index was 'spagetti.'
<jbenet>
yeah, sorry, lots of abstractions
<Vyl>
Ok... so, I have my thingie, QmdjMSVakZT3tDh62i1SH5incGaCznVS2XcMUFFfZTNu5z. You should be able to retrieve it, it's an ogg file.
therealplato1 has joined #ipfs
<jbenet>
Vyl: i looooove that sing
<jbenet>
song*
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<Vyl>
But if I wanted to insert others by that group, and have them accessible under, say, /ipfs/blahhashblahhashblahhash/The_Chorettes/variousfiles.ogg addresses?
<Vyl>
So they could be listed by ipfs ls, or linked in a webpage to insert too?
<Vyl>
How would that work?
therealplato has quit [Ping timeout: 244 seconds]
<Vyl>
Where I hope the blahhash might be a node ID, which would allow me (and only me) to update it should I get better quality files or more to add in future?
<Vyl>
I think I got that right.
<jbenet>
Vyl: i know it's old, but copyright terms in the US re like a million years. please don't put any copyrighted content on ipfs unencrypted. if you have the rights to move it, move it encrypted pls. (we'll eventually start getting DMCA complaints and so on). sorry, wish the world was different.
<jbenet>
Vyl: try "ipfs add -r <directory>
<Vyl>
It's public domain to me.
<Vyl>
Shorter term over here. But more importantly, it's going to happen.
<Vyl>
Free hosting? Yeah... you've basically invented the greatest tool for piracy since Sharereactor.
<jbenet>
"to me" is a matter of opinion, and won't hold in court. please read our code of conduct.
<Vyl>
Your code of conduct is going to be utterly enforceable if this takes off.
<Vyl>
*unenforceable
<jbenet>
not at all. our gateways will respect DMCA notices and will broadcast a list of blocks for other nodes to comply with. its up to you whether you follow it or not, but we're not going to be responsible for consequences if you chose not to follow them. i.e. this is the cleanest path between "freedom of speech" and "abiding by laws" that there is.
<spikebike>
sadly DMCA violations seemed to work reasonably for both sides. Seems largely replaced with ransom letters these days
hellertime has quit [Quit: Leaving.]
<rschulman>
Vyl: Also related (I am a lawyer, though I'm not jbenet's lawyer), Juan is the US, so the DMCA and US copyright law are absolutely the relevant question here.
josephdelong has quit [Quit: josephdelong]
wasabiiii1 has joined #ipfs
wasabiiii has quit [Ping timeout: 246 seconds]
shea256 has joined #ipfs
<Vyl>
This doesn't seem very scalable. No better or worse than regular HTTP in that regard. Whack-a-mole all over again.
<Vyl>
Needs more sites like neocities to win respectability. Except... well, sites people have actually heard of.
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<Vyl>
Because otherwise, once IPFS is well-known, the invasion of the pirates is inevitable. They are always among the first to adopt any new means of distributing information.
<Vyl>
The offending block isn't pinned anyway, it'll disappear eventually.
<rschulman>
Vyl: Of course, nobody is saying that pirates won't use IPFS
<rschulman>
pirates will use any technology available to them.
<rschulman>
but we also don't have to promote it as such in our IRC channel
pfraze has quit [Remote host closed the connection]
<ekaron>
would a mechanism to block a list of DMCA request involved hashes be a huge parsed list each time a hash is retrieved. Yuck, but seems necessary for an endpoint not to cache/pin locally illegal data...
<ekaron>
that seems unwieldy, but it's easy to see who is a peer holding that data and who to go after.
<ekaron>
being a us citizen subject to DMCA safe harbor, that seems important to me...
wscott_ has left #ipfs ["Leaving"]
<voxelot>
plus couldn't one byte of the file be changed and then the whole hash changes, seems hard to keep a list like that
<jbenet>
ekaron: a simpler strategy is to publish a list of DMCAed blocks, and nodes simply check if it's on that list. you can make this really fast using bloom filters (to keep majority of checks really fast), only resorting to comparing hashes when the probability is high.
<jbenet>
ekaron: also, please note that IPFS does not request objects you did not explicitly ask for.
<mmuller>
it's easy to conceive of a situation where illicit blocks get DMCA'ed as soon as they are published, with the same works being immediately republished under new hashes (slightly modified blocks) leading to an explosion of blacklisted blocks.
<mmuller>
jbenet: that's not the case with bitswap, right?
<M-Kodo>
Or we could just abolish copyright and be a sensible sort of humanity
<mmuller>
M-Kodo: that's not a technical problem :-)
* M-Kodo
goes back to his corner
<jbenet>
mmuller: those strategies are not employed yet, and when they are they will likley be opt-in AND respect these lists.
<M-Kodo>
The very notion of copyright implies that you can own a number which is just ridiculous </rant>
<mmuller>
jbenet: yes, understood. But important to clarify :-)
<jbenet>
indeed :)
<edrex>
was just thinking through the static site hosting use-case, and wondering how directory index pages work
<solarisfire>
You can always look at the ipfs.io website ;-)
<edrex>
can i publish an index.html as <root hash>/<some path>, while still supporting accessing other objects under <root hash>/<some path>/<some file>?
<edrex>
solarisfire: good point
ScoreUnder has joined #ipfs
<edrex>
just, that issue has always been a sticky point for static site hosting
<edrex>
but how do you go about serving a custom index page, while still preserving the merkle dag structure of the site?
<edrex>
it seems to work, just wondering how to replicate
<edrex>
i guess the object schema supports both content and links
ab14 has quit [Quit: Leaving.]
<rschulman>
I'm by no means a copyright defender, and I think that how its implemented in the US is atrocious, but its not facially ridiculous to imply that the output of a large number of hours of work should inherently not reserve some kind of property protection.
<edrex>
need to read the schema docs some more
<Vyl>
Well, that was fun.. half an hour of trying to explain to someone how to edit PDFs. And now I am going for a time. Bah. Users.
ab14 has joined #ipfs
<edrex>
i guess i'm wondering if the ipfs add command automatically adds index.html content as the bytes of the object, or if there's another tool purpose-built for adding sites.
<M-Kodo>
Interesting argument that large numbers are products of labour. I guess the whole bitcoin economy is predicated in that idea
<M-Kodo>
On that
rjeli has quit [Ping timeout: 246 seconds]
<rschulman>
they're not random large numbers
<rschulman>
(a novel, for example)
<rschulman>
they are carefully cultivated large numbers
<lgierth>
borgtu: tell your hoster about this gh issue: https://github.com/ipfs/go-ipfs/issues/1226 -- and bug them to fix their filters, we're *not* scanning, these are addresses of potential peers
<M-Kodo>
Right but still the hashing labour is called proof of work for a reason
<M-Kodo>
It takes real word resources
<M-Kodo>
World
<M-Kodo>
Bloody mobile
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<rschulman>
M-Kodo: Oh sure, in the bitcoin example, yes.
pfraze has joined #ipfs
rjeli has joined #ipfs
<edrex>
looking some more, I suppose the gateway handler must serve index.html entries if present (it's a mini apache.. where's my mod_rewrite?)
<lgierth>
edrex: yep it generates its dir-listing only if there's no index.html it could serve
shea256 has quit [Remote host closed the connection]
shea256 has joined #ipfs
Ferdz has joined #ipfs
<edrex>
ah, makes sense now. I guess I'm a little bit wary of magical get handlers (the s3 way relies on the putter to put the right object to each path, which makes deploying a static site harder, but is simple to reason about)
rendar has quit [Ping timeout: 265 seconds]
<edrex>
i've thought about establishing a standard (subset of what apache does) for how directories will be served as static sites, which different servers could implement.
<lgierth>
that gateway handler is a mess of its own
<lgierth>
we wanna move it to its own lib/repo
<edrex>
right now everybody does their own subset
pfraze has quit [Remote host closed the connection]
<edrex>
i don't know of any that actually look inside .htaccess, but I imagine that's just for lack of looking.
j0hnsmith has joined #ipfs
<edrex>
the s3 way feels a lot more future proof to me.
<edrex>
if you get the PUTs right, you can guarantee that they'll be served the same way forever
nessence has quit [Remote host closed the connection]
wasabiiii has joined #ipfs
patcon has joined #ipfs
wasabiiii1 has quit [Ping timeout: 265 seconds]
* blame
is going to take a nap, and then add bloom filter based search to cachewarmer
boxxa has joined #ipfs
<jbenet>
hey everyone, it would be awesome if a lot of questions can be asked on https://github.com/ipfs/faq/issues or if you want to propose + design something and need a scratch pad use https://github.com/ipfs/notes/issues -- we all want to chat a lot here and keep it active, but we also want to build a good archive for other people to search through and spend less
<jbenet>
time answering similar questions half-way here, when we could have great answers on github, etc.
simonv3 has quit [Quit: Connection closed for inactivity]
<edrex>
solarisfire: that's pretty cool
<voldial>
whyrusleeping: I got the updated FBC (IFBC) paper on loan. If you or anyone here wants a copy pm me. It's attached to my university acct so I cant just post it.
Score_Under has joined #ipfs
<solarisfire>
Yeah sure jbenet
<jbenet>
solarisfire: thank you!
captain_morgan has quit [Ping timeout: 246 seconds]
ScoreUnder has quit [Ping timeout: 244 seconds]
<edrex>
Seems like the gateway should send 301 responses for "deep" URLs to the canonical version of the resource (eg GET /ipfs/<root hash>/<some path>: 301 /ipfs/<hash of linked object>) rather than serving it as a 200. I can't think of a reason not to. Is there a ticket for that?
<jbenet>
edrex: there is a reason. websites an relative urls
<jbenet>
edrex: imagine an html file or a css file with a relative import
<edrex>
oh, dang
Score_Under has quit [Ping timeout: 252 seconds]
<edrex>
yeah..
ScoreUnder has joined #ipfs
<lgierth>
edrex: we give the canonical path in the X-Ipfs-Path response header though
<edrex>
that sucks though
<jbenet>
it's fine. :) it's only html cachinc that suffers
<jbenet>
caching*
<edrex>
yeah
<lgierth>
if that's not there everywhere, it's a bug imo
<jbenet>
nbd. ipfs nodes in your machine will fix that.
<jbenet>
lgierth: we should have both paths, the accessed path, and the root hash.
<jbenet>
"rooted" hash
<edrex>
yeah, but it takes a lot of the value out of using the gateway to serve static sites
dignifiedquire has joined #ipfs
<lgierth>
-- and adds value on the other end
notduncansmith has joined #ipfs
<lgierth>
ipfs.io and chat.ipfs.io for example are hosted right out of ipfs
notduncansmith has quit [Read error: Connection reset by peer]
* spikebike
checks chat.ipfs.io
<spikebike>
oh, that's what you mean
<edrex>
i guess it really just points out that HTTPs caching semantics aren't designed around content addressability.
<edrex>
Because, you know, it wasn't really a thing yet.
<lgierth>
jbenet: don't understand what you mean re: we should have both paths -- in the response headers?
Score_Under has joined #ipfs
<lgierth>
edrex: it's actually not so hard to cache stuff -- use the hash as the etag, done
<edrex>
it would be nice if there was a response that said: hey, get the resource from over there, but don't change consider the URL as having changed.
ScoreUnder has quit [Ping timeout: 250 seconds]
<edrex>
yeah, i'm more thinking of globally deduplicated caching.
<edrex>
(for JS libs etc)
<jbenet>
lgierth: yeah. could have X-IPFS-Access-Path and X-IPFS-Rooted-Path
<jbenet>
voldial awesome! look fwd to reading them.
<spikebike>
edrex: heh, content addressability predates http by quite a bit
<lgierth>
yeah like 60s or so? :P
<lgierth>
i keep forgetting that system's name
<spikebike>
plan-9 used and published it
<voldial>
jbenet: they do about 1.5x better (avg) total dedup (metadata included)
<spikebike>
not that I claim that's the first implementation.
<voldial>
some of the results are way better
<jbenet>
voldial: very nice. so actually, i started https://github.com/jbenet/go-chunk in the hopes we could make a standard "splitter" interface and that way try out a ton of algos on various datasets.
<jbenet>
would be _fantastic_ to have really large standard workloads for ipfs and other apps and try them on good impls of this stuff
<edrex>
spikebike: i meant "a thing" more in the sense of it being a comfortable tool in the toolbox of your average network service architect
<edrex>
which is relatively recent
ScoreUnder has joined #ipfs
Score_Under has quit [Ping timeout: 256 seconds]
<voldial>
jbenet: excellent. I was reading whyrusleeping's tar importer thinking the same thing. I'll dig in further. Would be great to have the test suite able to compare the various chunking alg's
<jbenet>
edrex spikebike: yeah, i've often wondered about that. i think http couldn't have worked because all the p2p global systems didnt exist yet. dns + ntp were the widest deployed online protocols and they weren't known to be super fast and reliable. i think stuff wasn't good enough for the global web yet. wish http had just had the foresight to bake it in more
<jbenet>
closely though, certainly the "not treating locations as the resource identity"
<edrex>
jbenet: oh neat!
<edrex>
(the chunker lib)
<jbenet>
i mean, it's like just an interface, but it's more importantly an idea.
<jbenet>
and we could standardize this across other languages too, get impls in js and so on.
<jbenet>
chunk all the things! \o/
<voldial>
the #bup project has the smae "problem"
<edrex>
(i might port the camlistore chunking algo just for fun)
twistedline has joined #ipfs
<twistedline>
just playing with IPFS, looks great and all seems to work...apart from the webui, which loads but shows no info, had blank Peer ID 0 connections etc.
<twistedline>
no files.
<spikebike>
how long has the daemon been running?
M-edrex has joined #ipfs
<M-edrex>
oh huh
<M-edrex>
didn't realize there was a bridge
<M-edrex>
but chirality is reversed when you go through the conduit
wasabiiii1 has joined #ipfs
<spikebike>
use two conduits ;-)
<jbenet>
twistedline: wait really? on :5001 ?
wasabiiii has quit [Ping timeout: 252 seconds]
<bengl>
i saw the npm on ipfs discussion being rebooted. anything i can/should help with?
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
josephdelong has joined #ipfs
wopi has quit [Read error: Connection reset by peer]
<jbenet>
bengl: yes! i think we still need to do the ipfs-blob-store -- daviddias and mappum and i could help with this. we may be able to knock it out in a couple hours
wopi has joined #ipfs
<jbenet>
bengl: it's way easier with some of the latest ipfs additions (and will get easier later)
dignifiedquire has quit [Quit: dignifiedquire]
<bengl>
jbenet: yeah as long as ipfs-blob-store can write to an ipns the same way it would with fs-blob-store, you're most of the way there.
<ipfsbot>
[node-ipfs] RichardLitt created feature/cleanup-roadmap-add-issue-links (+1 new commit): http://git.io/vZFn2
<ipfsbot>
node-ipfs/feature/cleanup-roadmap-add-issue-links edc5886 Richard Littauer: Cleaned readme, added discussion links
<bengl>
slap a slightly smarter gateway (or forked npm client) in front of it and you've got yourself a registry mirror
<jbenet>
bengl: yeah that'd be the final goal, but we can do it before-hand by keeping a root at a well known place.
<jbenet>
(cause ipns is still not 100% perfect)
<bengl>
jbenet: yeah that would be fine. you'd just have ipfs-blob-store updating the well known place, right?
<bengl>
can directory-listing be overridden?
<bengl>
for gateways
<bengl>
by the content
<jbenet>
bengl: it can with an index.html
domsch has joined #ipfs
<jbenet>
we could see about overriding it altogether, but it's not trivial.
<jbenet>
(or rather, it's straightforward but some amt of work)
devbug has joined #ipfs
<domsch>
Hey guys
Score_Under has joined #ipfs
ScoreUnder has quit [Ping timeout: 240 seconds]
devbug has quit [Ping timeout: 250 seconds]
<jbenet>
hey domsch
devbug has joined #ipfs
ScoreUnder has joined #ipfs
<Vyl>
The more I think about the copyright thing, the more problems I see. Not with copyright, but jurisdictional problems in general. A lot of content is illegal in various countries.
<solarisfire>
Added a pull request for that example jbenet
<domsch>
I'm from the College Crypto Network and we are right now trying to organize a "Borderless Innovation" hackathon. Since there can't be a borderless innovation hackathon without IPFS, I was wondering if we could get someone (or even a few people) form IPFS that can be there to help provide guidance, post some good documentation or even tutorials to help guide the devs to create amazing new d(Apps)
<jbenet>
Vyl, yes and you can have lists per jurisdiction
<Vyl>
People in America get DMCAed. People in Europe for hosting leaked personal information and 'revenge porn.' In Russia you can go to jail for material that encourages gay rights, China has criminalised material promoting certain religions, and half the middle east has laws against insulting Islam.
hellertime has joined #ipfs
<domsch>
The hackathon is about 2 months away, we will release specifics soon. Right now we're just organizing everything, getting sponsors on board and making sure we have all the right tech partners :)
<jbenet>
domsch: sounds great! yeah post up there to keep us all organized in one place.
<Vyl>
There's no way you'll keep up. There's a reason Freenet was designed for plausible deniability.
<Vyl>
Plus so you really want to be maintaining the 'pull this from the internet, China wants it banned' list?
<domsch>
P.S.: is someone from the community already working on a Wikipedia on top of IPFS?
<Vyl>
Pornography alone is a nightmare, with so many different national laws, often relying on very vague standards and lax enforcement.
<jbenet>
Vyl: i'm really done talking about this. everything you're saying we've already discussed before and we _can address_ with the designs i've described before, both in this channel and on github.
therealplato1 has quit [Ping timeout: 260 seconds]
hellertime has quit [Client Quit]
pfraze has joined #ipfs
ScoreUnder has quit [Ping timeout: 240 seconds]
<jbenet>
domsch: thanks! great to have it be global.
<achin>
hmm, wikipedia over IPFS sounds super neat
hellertime has joined #ipfs
<jbenet>
domsch: yeah i really really want this. we should start work on it soon. are you interested in helping us out?
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<domsch>
yeh we wanted to create a Wikipedia version with a "gist" mode (that summarizes everything), combined with Microtransactions (I run a Bitcoin/Fiat tipping platform btw) and with Primavera's concept of reputation (backfeed.cc)
<domsch>
yeh absolutely! would love to work on something like this.
<achin>
(compressed it seems to only be about 12 GB)
<jbenet>
Vyl: i dont mean to be a jerk, but please read more. and think deeply through it all. look into how DMCA takedowns are served right now for p2p networks. and look into how these (already very real lists) are maintained. nothing will prevent people from publishing their own lists and forcing people under their jurisdiction to adopt them. and on the flip side,
<jbenet>
nothing will prevent people from encrypting data end to end, either.
<jbenet>
domsch: awesome, what's your gh name? github.com/domsch ?
ELFrederich has quit [Read error: Connection reset by peer]
ELFrederich has joined #ipfs
nessence has joined #ipfs
<domsch>
I will make a post on the Google Group and see if people want to join the Wikipedia project ;)
Davididid has joined #ipfs
<lgierth>
domsch: better post to github.com/ipfs/notes
<lgierth>
the google group is a bit dead-ish
<domsch>
ahh ok for sure
<jbenet>
actually, let me post i'll tag you
<domsch>
yeh that sounds good
dignifiedquire has joined #ipfs
<achin>
given a hash, is there any way to figure out what it refers to? an object/block/peer?
<lgierth>
achin: you fetch it
<lgierth>
:)
<domsch>
btw Juan, (I'm not an expert in P2P systems, so take what I say with a grain of salt), but aren't semantic overlay networks a perfect usecase for IPFS as well? Unless you guys are already using a similar system?
<achin>
and then look at the header of the data?
phorse has joined #ipfs
<lgierth>
object is a subset of block, and peer is completely separate at the moment, will change once IPNS is complete
<lgierth>
achin: ipfs object get <hash> iirc
<achin>
that doesn't seem to work with peers
<cryptix>
domsch: there is _some_ work of using cjdns
<cryptix>
personally id like to also see i2p and gnunet
<cryptix>
in principal it should be fairly easy because of the muliaddrs
<cryptix>
but we def need relay (ip4<>ip6) to get all that off the ground
<cryptix>
id also like relay at some point (soonish) to get out of overblocking networks but thats just me :)
notduncansmith has quit [Read error: Connection reset by peer]
patcon has quit [Ping timeout: 240 seconds]
shea256_ has quit [Read error: Connection reset by peer]
shea256 has joined #ipfs
chriscool has joined #ipfs
<twistedline>
jbenet: sorry for the delay, had to take a call. yeah on :5001. however I am loading it from its local ip 10.* rather than 127.*
<twistedline>
because this is a headless server.
<lgierth>
twistedline: yeah that might be cors
<lgierth>
look in the browser console for errors
shea256 has quit [Read error: Connection reset by peer]
<lgierth>
403 usually means cors
<twistedline>
just one for Path Resolve error: no link named "fontawesome-webfont.woff2" under QmWjympWW8hpP5Jgfu66ZrqTMykUBjGbFQF2XgfChau5NZ
shea256 has joined #ipfs
<jbenet>
twistedline: yeah i think that's cors
<lgierth>
yep that's one's known -- no other errors?
<lgierth>
i.e. non 2xx/3xx responses?
JohnClare has joined #ipfs
<twistedline>
just in daemon output ?
<twistedline>
or are there some logs?
<twistedline>
ok did some port forwarding and it works fine on 127.0.0.1:5001
magneto1 has joined #ipfs
<twistedline>
but all in all very cool I like it a lot!
dignifiedquire has quit [Quit: dignifiedquire]
pfraze has quit [Remote host closed the connection]
pfraze has joined #ipfs
<spikebike>
ah, nice, 48 IPFS nodes are on IPv6 on 45 different prefixes
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<lgierth>
spikebike: thanks for these stats
<lgierth>
we don't have ipfs.io on ipv6 yet tbh :(
atrapado has quit [Quit: Leaving]
jamescarlyle has quit [Remote host closed the connection]
jamescarlyle has joined #ipfs
<spikebike>
lgierth: I didn't walk all nodes, just listed the peers from 2 IPFS nodes
<spikebike>
(and uniq'd them)
shea256_ has joined #ipfs
josephdelong has quit [Quit: josephdelong]
cblgh is now known as jnerule
shea256_ has quit [Remote host closed the connection]
pfraze has quit [Remote host closed the connection]
shea256_ has joined #ipfs
shea256_ has quit [Remote host closed the connection]
boxxa has quit [Quit: Connection closed for inactivity]
shea256 has quit [Read error: Connection reset by peer]
jamescarlyle has quit [Client Quit]
shea256 has joined #ipfs
<lgierth>
spikebike: still interesting :)
zorun has quit [Ping timeout: 268 seconds]
Ferdz has quit [Ping timeout: 246 seconds]
<ekaron>
I'm a network engineer and I can't even get ipv6 when I want it set up anywhere.
JimmyTheWhale has joined #ipfs
<ekaron>
telepacific, att fiber, don't issue ipv6 addresses with new orders. Time warner cable used to do a ipv4-6 gateway thing, but I played with my firewall this weekend and ipv6 tests said not working...
<ekaron>
so doesn't seem to be an industry priority now
<JimmyTheWhale>
hey qts o/
dignifiedquire has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<achin>
cox told me that they are in the middle of an ipv6 rollout to all consumers, scheduled to be completed by the middle of 2016. so some ISPs are working on it
Score_Under has joined #ipfs
jnerule is now known as cblgh
<JimmyTheWhale>
I checked the IRC logs and known issues and didn't see any mention, but I thought I might ask in here anyway: Has anyone reported any trouble with Comcast router/modem-combos & ipfs?
domsch has quit [Quit: Page closed]
hellertime has quit [Quit: Leaving.]
pfraze has joined #ipfs
jfis has quit [Quit: s]
nessence has joined #ipfs
voldial has quit [Quit: WeeChat 1.2-dev]
shea256 has quit [Remote host closed the connection]
chriscool has quit [Ping timeout: 268 seconds]
JohnClare has quit [Remote host closed the connection]
shea256 has joined #ipfs
<jbenet>
JimmyTheWhale: not yet. did you observe any?
hellertime has joined #ipfs
captain_morgan has joined #ipfs
<JimmyTheWhale>
Yeah. I'm going to spend the day plunking at it again, but a few min after i start it up all device connections to the modem are severed and it requires a power cycle to receive new connections again (though the wifi network never actually goes down)
hellertime has quit [Client Quit]
hellertime has joined #ipfs
<lgierth>
JimmyTheWhale: do you have a console on the router?
wasabiiii1 has joined #ipfs
<ipfsbot>
[go-ipfs] jbenet pushed 2 new commits to master: http://git.io/vZFFv
<ipfsbot>
go-ipfs/master c77a96f Juan Benet: Merge pull request #1703 from Luzifer/signals...
<ipfsbot>
go-ipfs/master 1cdafc7 Knut Ahlers: Use exec to start IPFS daemon...
wasabiiii has quit [Ping timeout: 250 seconds]
<JimmyTheWhale>
Igierth Yessir. AKA the admin panel?
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
magneto1 has quit [Ping timeout: 260 seconds]
therealplato has quit [Ping timeout: 260 seconds]
hellertime has quit [Ping timeout: 256 seconds]
wopi has quit [Read error: Connection reset by peer]
therealplato has joined #ipfs
wopi has joined #ipfs
hellertime has joined #ipfs
JohnClare has joined #ipfs
therealplato has quit [Ping timeout: 264 seconds]
<lgierth>
JimmyTheWhale: i mean a terminal console -- many of these routers run some kind of linux
<lgierth>
that'd make it a bit easier to debug
<JimmyTheWhale>
ahh doubtful.
hellertime has quit [Ping timeout: 272 seconds]
JohnClare has quit [Ping timeout: 264 seconds]
jfis has joined #ipfs
JimmyTheWhale has quit [Ping timeout: 264 seconds]
solarisfire has left #ipfs [#ipfs]
<jbenet>
hmm this is definitely weird.
Xe is now known as jnerule
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
ScoreUnder has joined #ipfs
<edrex>
ekaron: comcast has ipv6 rolled out everywhere afaict
Score_Under has quit [Ping timeout: 265 seconds]
JimmyTheWhale has joined #ipfs
Score_Under has joined #ipfs
ScoreUnder has quit [Ping timeout: 250 seconds]
G-Ray has quit [Quit: Konversation terminated!]
JimmyTheWhale has quit [Remote host closed the connection]
j0hnsmith has quit [Quit: j0hnsmith]
JimmyTheWhale has joined #ipfs
ScoreUnder has joined #ipfs
<JimmyTheWhale>
Just to be embarrassingly clear: if someone asks you that question again, tell them to open 4001. ;)
JimmyTheWhale has quit [Remote host closed the connection]
Score_Under has quit [Ping timeout: 250 seconds]
ScoreUnder has quit [Ping timeout: 246 seconds]
Score_Under has joined #ipfs
patcon has joined #ipfs
shea256 has quit [Remote host closed the connection]
<jbenet>
JimmyTheWhale ah bummer. guessing the router does not have upnp/pmp?
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
Score_Under has quit [Ping timeout: 244 seconds]
Score_Under has joined #ipfs
shea256 has joined #ipfs
dignifiedquire has quit [Quit: dignifiedquire]
rendar has quit []
ygrek has joined #ipfs
patcon has quit [Ping timeout: 268 seconds]
ScoreUnder has joined #ipfs
Score_Under has quit [Ping timeout: 244 seconds]
<ekaron>
I guess it begs the question, which goes widespread first, ipv6 or ipfs, not exclusive but which catches on faster?
jnerule is now known as Xe
ScoreUnder has quit [Ping timeout: 250 seconds]
jfis has quit [Quit: s]
<whyrusleeping>
jbenet: i need to update both peerstream and datastore, peerstream should be a clean update, but datastore might require some integration, want me to make separate PRs? or is one PR okay for both
<lgierth>
ekaron: for *.ipfs.io, it's definitely ipfs :(
<lgierth>
on the other hand, we do use ipv6 exclusively for all management traffic in the infrastructure
<lgierth>
within the private fc00::/8 prefix though
Score_Under has joined #ipfs
magneto1 has joined #ipfs
wasabiiii has joined #ipfs
wasabiiii1 has quit [Ping timeout: 264 seconds]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
magneto1 has quit [Ping timeout: 240 seconds]
<jbenet>
whyrusleeping: one pr with 2 commits is fine
<whyrusleeping>
okay
<jbenet>
whyrusleeping: or if they dont separate cleanly (dont build / fail tests) then 1 commit
<whyrusleeping>
they separate cleanly, 2 commits is totally okay
<ekaron>
I'm not too worried. I'm just excited for ipfs and kinda ignorant about ipv6.
<lgierth>
:)
<ansuz>
ipv6 is the one true way
<ansuz>
one of us
<ansuz>
one of us
<ansuz>
etc
* ansuz
lurks harder
ygrek has quit [Ping timeout: 240 seconds]
<ipfsbot>
[ipfs] jbenet closed pull request #93: Accidentally a word (master...patch-1) http://git.io/vZ9v5
shea256 has quit [Remote host closed the connection]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
ScoreUnder has joined #ipfs
devbug has joined #ipfs
shea256 has joined #ipfs
Score_Under has quit [Ping timeout: 265 seconds]
<jbenet>
ok i think i just got done with all the notifs _for today_.