pfraze has quit [Remote host closed the connection]
pfraze has joined #ipfs
domanic_ has joined #ipfs
clever has quit [Ping timeout: 256 seconds]
domanic has quit [Ping timeout: 246 seconds]
Eudaimonstro has joined #ipfs
amstocker has joined #ipfs
sseagull has joined #ipfs
clever has joined #ipfs
pfraze has quit [Remote host closed the connection]
slothbag has quit [Quit: Leaving.]
domanic_ has quit [Ping timeout: 256 seconds]
Guest73396 has joined #ipfs
Guest73396 has quit [Ping timeout: 255 seconds]
Luzifer has quit [Ping timeout: 240 seconds]
sickill has quit [Ping timeout: 240 seconds]
gwillen has quit [Ping timeout: 252 seconds]
gendale__ has quit [Ping timeout: 240 seconds]
ZioFork has quit [Ping timeout: 240 seconds]
gwillen has joined #ipfs
mappum has quit [Ping timeout: 240 seconds]
Luzifer has joined #ipfs
Guest73396 has joined #ipfs
gwillen is now known as Guest933
gendale_ has joined #ipfs
Guest933 has joined #ipfs
Guest933 has quit [Changing host]
Guest933 is now known as gwillen
sickill has joined #ipfs
mappum has joined #ipfs
ZioFork has joined #ipfs
<CaioAlonso>
how high would my publishing frequency need to be to start breaking IPNS?
<sonatagreen>
define breaking
<CaioAlonso>
I mean, if I were to have some dynamic content that changes every second, and that I would re-publish every second, would things go wrong?
<CaioAlonso>
like old versions still being considered fresh for a while
<sonatagreen>
(i don't actually know what i'm talking about, but) i imagine there's a lower threshold for 'people might see a several-seconds outdated version of your site' than for 'the network is brought to its knees by your denial-of-service'
<CaioAlonso>
but how should my expectations be about ipns freshness? would it be like dns that takes some hours
<sonatagreen>
i'd be happy to pull your site repeatedly
<davidar>
CaioAlonso (IRC): personally I'd treat it more like DNS updates than real-time
<davidar>
Real time dynamic stuff should be done client side
<CaioAlonso>
I've been theorizing about how a video streaming service would work, but I still haven't been able to figure out how to serve an index of all the videos
<CaioAlonso>
without resorting to a centralized service outside of IPFS or something else like ethereum
<davidar>
Although that still leaves open the question about getting real-time data into ipfs
<CaioAlonso>
davidar yeah
<CaioAlonso>
sonatagreen I'll make some tests later
<davidar>
CaioAlonso (IRC): there's also going to be a pubsub service on ipfs in the future, which might help
<CaioAlonso>
I'll look into it
<davidar>
whyrusleeping (IRC): any thoughts on real-time ipns?
<CaioAlonso>
I guess it comes down to the problem of how a user would discover new hashes
<CaioAlonso>
I thought about keeping an ipns record indexing all of the video hashes, and would update it frequently
<amstocker>
you can use the DHT to get peers corresponding to a certain hash, and then periodically poll those peers at a certain ipns path
<davidar>
CaioAlonso (IRC): also note that the 4h is how often your node automatically announces the current ipns to the network, not how long it takes to propagate
Guest73396 has quit [Ping timeout: 264 seconds]
<davidar>
I think it *should* be possible to do real-time ipns updates, but I wouldn't rely on it quite yet
<CaioAlonso>
davidar I understand
<whyrusleeping>
davidar: i have thoughts, can chat about it later
<davidar>
whyrusleeping (IRC): cool
<kandinski>
I'm reading through the codebase, and have a question about go-datastore
<kandinski>
how much of the abstraction provided is used by ipfs?
<CaioAlonso>
davidar I was just about to do this, thanks :D
amstocker_ has joined #ipfs
od1n1 has joined #ipfs
<CaioAlonso>
davidar I think there's no need to open an issue talking about this, because it is simply a specific usecase of the aggregation problem that you reported a few weeks ago https://github.com/ipfs/notes/issues/40
amstocker has quit [Ping timeout: 246 seconds]
<CaioAlonso>
its just finding a way to aggregate/search videos + a js player
<CaioAlonso>
although I'm afraid that solving this in a truly distributed manner is a task I'm not equipped to do
amstocker_ has quit [Ping timeout: 246 seconds]
od1n1 has quit [Quit: WeeChat 1.1.1]
sbruce has quit [Ping timeout: 240 seconds]
sbruce has joined #ipfs
od1n has joined #ipfs
* whyrusleeping
takes a breath
<ipfsbot>
[go-ipfs] whyrusleeping created feat/utp (+2 new commits): http://git.io/vciYy
<ipfsbot>
go-ipfs/feat/utp 24275a6 Jeromy: vendor in utp code...
<multivac>
[WIKIPEDIA] Peercasting | "Peercasting is a method of multicasting streams, usually audio and/or video, to the Internet via peer-to-peer technology. It can be used for commercial, independent, and amateur multicasts. Unlike traditional IP Multicast, peercasting can facilitate on-demand content delivery...."
<reit>
it intuitively seems like it should be super doable.. but it would likely need more to the network than just ipns
<davidar>
reit (IRC): in real-time?
<reit>
yeah
<sonatagreen>
beware of scope creep. ipxs is a tool for a specific purpose.
<reit>
well i mean ipns would still be necessary, but primarily as an anchor point
<reit>
the actual transfer under this scenario would be the peers organize themselves into a hierarchy
<reit>
based on their respective upload capacities
<davidar>
sonatagreen (IRC): yeah, that was my initial thought too, but peercasting does look kind of IPFSish
<sonatagreen>
the question i would ask is, if someone wanted to build a peercasting system, how would ipfs be helpful?
<reit>
primarily addressing
<davidar>
sonatagreen (IRC): the same way ipfs helps with static video distribution, but in real-time
<sonatagreen>
and... i dunno, i feel like the strengths of ipfs have to do with permanent storage and large latency
<reit>
also, imo the point of ipfs is to deprecate http
<sonatagreen>
neither of which applies here
<reit>
for that to happen it really should be able to assimilate all of the current functions of http
<davidar>
whyrusleeping (IRC): when you say you have thoughts about real-time for ipfs, are they positive or negative?
<sonatagreen>
http does lots of things that http was never supposed to do. ipfs excels at http's /theoretical/ purpose, but other things should replace its /actual/ usages.
<sonatagreen>
e.g. smtp vs. webmail.
<davidar>
sonatagreen (IRC): the stream can still be archived permanently
<sonatagreen>
it seems to me the main advantage of ~web sites~ as a thing is sandboxed code
<whyrusleeping>
davidar: +
<sonatagreen>
you can download and run arbitrary software from strange people without worrying too much about evil
<davidar>
I think the question is basically, can you watch the video as its being created?
<reit>
i think so, it'd be done by chopping the stream into say 256k blocks, hashing each and then pushing it down the peercasting hierarchy
<reit>
as the stream is playing
<reit>
a sensible application-level convention might be for each block to have a pointer to the preceding block in the sequence, so that you could wind the stream back at any time
<sonatagreen>
I think this generalizes as, instead of a normal request for 'please send me the data at /ipns/foo', a standing request for 'please send me the data at /ipns/foo whenever you hear that it's updated'. potentially also useful for following blogs?
<davidar>
sonatagreen (IRC): sounds like pubsub
<sonatagreen>
reit, if you do that then you end up with a linked list in the wrong direction for later playback
<davidar>
sonatagreen (IRC): yeah, but you can't do the other direction without a crystal ball ;)
<sonatagreen>
or without generating the entire video before uploading, as is the case in literally every circumstance except livestreaming.
<sonatagreen>
so you'd end up with two different file formats depending on use case
<sonatagreen>
which seems perverse
<davidar>
sonatagreen (IRC): well, yeah, but then you can just upload a file like we already do
<davidar>
You guys have already seen the ipfs video player, right?
<sonatagreen>
but then you end up uploading two almost-identical copies of the file with different hashes
<sonatagreen>
yeah, it doesn't perform very well on my machine, i always end up using mplayer from the command line
<reit>
sonatagreen: but, people want to be able to do real time streaming, i don't think people are going to accept 'oh, you just don't get to do that anymore'
<sonatagreen>
true
<sonatagreen>
/ugh/
<davidar>
sonatagreen (IRC): also, the same thing happens if you upload a file with different chunking or dag building strategies
<sonatagreen>
still, i question whether ipxs is the right tool for the job here.
<sonatagreen>
i feel like the system's main features are more liabilities than assets for this purpose.
<sonatagreen>
and using ipxs doesn't mean /never using anything else/
<sonatagreen>
i get the ~aesthetic~ connection, but from a technical perspective, it's just really not suited.
<davidar>
sonatagreen (IRC): yeah, I'm not sure tbh, I'm waiting for whyrusleeping to elaborate on his single character response :p
<sonatagreen>
how i imagine it is, you stream through a dedicated torrentstream tool, then upload the completed video to ipfs afterwards.
Eudaimonstro has quit [Ping timeout: 252 seconds]
<davidar>
sonatagreen (IRC): I don't see why ipfs is any less suited to the job than torrents
<whyrusleeping>
when I get time we're going to implement a steaming system on the dht which would allow data streams through ipfs using bitswap
<whyrusleeping>
it will basically be a super cool pub sub
<davidar>
sonatagreen (IRC): BTW, I agree with you that ipfs isn't a solution to every possible problem, but this usecase seems reasonable
<sonatagreen>
mh.
<davidar>
whyrusleeping (IRC): cool
<davidar>
People keep saying mh, what does it mean?
<sonatagreen>
i dunno about ipfs exactly as such, but i guess it's plausible that part of the underlying network could be useful.
<sonatagreen>
like, the resulting thing would be not ipfs in the same way that ipns is not ipfs.
<sonatagreen>
is my feeling.
<reit>
sonatagreen: i'm not convinced the reverse linked list idea is without merit
<reit>
the intuitive human approach is for video streams to be arranged via sequential frames, but in terms of loading the structure there's no reason you couldn't grab the last item in the stream (when it's finished) and rebuild the stream structure by traversing the list, then actually loading the content by going forwards from the first block
<sonatagreen>
so in summary, i guess i'm convinced
<davidar>
sonatagreen (IRC): ipfs + real-time data updates
<sonatagreen>
reit, sure, just make sure you don't have to load the entire chunk of data just to get the pointer to the previous item
<davidar>
You could build it up hierarchically like the current system, just in reverse order
<davidar>
To allow seeking without downloading everything
<reit>
sonatagreen: that's fine, it's just another dag entry
<davidar>
Actually, not even reverse order
<davidar>
Just literally how the day gets built now
<reit>
give it two links, the first to the actual content and the second to the previous encapsulating node
<davidar>
s/day/dag/
<multivac>
davidar meant to say: Just literally how the dag gets built now
pinbot has quit [Ping timeout: 244 seconds]
hpk has quit [Ping timeout: 244 seconds]
pinbot has joined #ipfs
hpk has joined #ipfs
<davidar>
As in, you could stream and build the archival copy simultaneously
<sonatagreen>
oh i see
<sonatagreen>
yeah, that sounds good
<sonatagreen>
i'm thinking leaf blocks are 5 seconds of video, and then binary tree above that?
Quiark has joined #ipfs
pfraze has joined #ipfs
<davidar>
sonatagreen (IRC): yeah, something like that (maybe more edges than binary though)
<sonatagreen>
i think binary has a nice combination of searchability and streamability
<sonatagreen>
the more edges you have, the less frequently you can make indexer nodes while streaming
<davidar>
sonatagreen (IRC): yeah, but you also have to request lots more nodes from the network to do anything, so there's a tradeoff
<sonatagreen>
isn't there a ~1kb limit on how much data can be stored in a single node? it seems like, in comparison to the amount of data involved in video, it's not that much
<davidar>
sonatagreen (IRC): 256kb
livegnik_ has quit [Quit: Aaaaand it's gone!]
<sonatagreen>
i mean, a binary tree is only 2*(leaves) non-leaf nodes
livegnik has joined #ipfs
jager has quit [Read error: Connection reset by peer]
jager has joined #ipfs
<sonatagreen>
and 256kb is ~1second of hd video
<sonatagreen>
hmm.
<davidar>
My concern is the extra network roundtrips during seeking
<davidar>
But whatever works well in practice, it's just an implementation detail ;)
<sonatagreen>
...
<sonatagreen>
you wouldn't /contact the network/ when seeking
<sonatagreen>
you'd already have the tree on your computer
<sonatagreen>
in your node's local storage thingy
<sonatagreen>
as part of downloading the video
<davidar>
sonatagreen (IRC): assuming you've already downloaded it
<sonatagreen>
or are you thinking of like a webplayer and a public gateway?
<davidar>
I though we were talking about seeking a video you haven't downloaded in full
<sonatagreen>
is it a video you're /going to/ download in full? or do you want to like excerpt a part of it?
<davidar>
"reit, sure, just make sure you don't have to load the entire chunk of data just to get the pointer to the previous item"
<davidar>
^ that's what I was replying to
* sonatagreen
rereads
<sonatagreen>
ok, i'm with you again now.
<sonatagreen>
i think my main question at this point is, do you want the seeking-tree to be available /while streaming/
<davidar>
Probably not
<davidar>
Well, maybe
<davidar>
I'm just talking about streaming with immediate archival afterwards
<sonatagreen>
because if an index-node has 30s worth of video as direct children, then the most recent up-to-30s of the stream won't be indexed yet
<sonatagreen>
whereas a binary tree allows you to start building indexer nodes as the data comes in
<sonatagreen>
so it's a tradeoff between streaming index and shallow tree
<davidar>
Yeah, but binary tree still has a delay as you build a big enough subtree to connect to the previous one
<davidar>
Only the first half of the stream will ever be complete, until the end
<davidar>
Index wise
<sonatagreen>
but you'll have subindexes for the newer parts
<davidar>
True
<sonatagreen>
and the client will only have to juggle at most log(n) orphan index nodes
<sonatagreen>
at a time
<davidar>
All I'm saying is that maintaining a real-time index introduces more complications :)
<sonatagreen>
true
Guest73396 has joined #ipfs
Guest73396_r has joined #ipfs
Guest73396 has quit [Ping timeout: 268 seconds]
Guest73396_r has quit [Ping timeout: 265 seconds]
domanic_ has joined #ipfs
pfraze has quit [Remote host closed the connection]
od1n has quit [Ping timeout: 264 seconds]
domanic_ has quit [Ping timeout: 246 seconds]
od1n has joined #ipfs
sseagull has quit [Quit: Lost terminal]
<ipfsbot>
[go-ipfs] whyrusleeping pushed 2 new commits to feat/utp: http://git.io/vci4R
<ipfsbot>
go-ipfs/feat/utp af3f628 Jeromy: refactor of swarm dialing, getting closer......
<ipfsbot>
go-ipfs/feat/utp 7db57cd Jeromy: if dialing a utp address, dont use a local address...
<ipfsbot>
[go-ipfs] whyrusleeping opened pull request #1789: Feat/utp (master...feat/utp) http://git.io/vci4z
<achin>
is UDT support planned at all?
<achin>
(i'm closing in ver 0.1 of my UDT bindings for rust)
<jbenet>
whyrusleeping: think we can merge utp tonight? if so, i can handle shipping 0.8.3 (know it's late for you)
<jbenet>
err 0.3.8*
<jbenet>
whyrusleeping: we can also merge it without all the bootstrap changes, and have that come in 0.3.9
<ion>
Weren’t you looking at UDT because of the state of µTP libraries for go? Are they better now?
<ion>
I bet go-ipfs 0.8.3 is going to be awesome.
rendar has joined #ipfs
<whyrusleeping>
jbenet, try the utp branch out. it worked for the things I was doing
<whyrusleeping>
only weird part was that there were duplicate connections...
<whyrusleeping>
not sure why
<jbenet>
ion: we were and we'll put it in soon too, but it's annoyingly hard as it requires static libs compiled, which requires we do all the builds as Luzifer's gobuilder only uses go get atm and we prob cannot wrangle it to work with the static c libs
<jbenet>
duplicate connections?
<whyrusleeping>
yeah, the peers in my test net had multiple connections to the same peer
NeoTeo has joined #ipfs
<whyrusleeping>
I'm not super familiar with the code for that
<jbenet>
whyrusleeping: duplicate utp conns? or one utp one tcp/
<jbenet>
(duplicate conns is fine, duplicate utp is odder)
<ipfsbot>
[go-ipfs] jbenet pushed 2 new commits to master: http://git.io/vci9k
<ipfsbot>
go-ipfs/master d9d9798 Juan Benet: Merge pull request #1697 from rht/fix/tar...
<ion>
whyrusleeping, jbenet: How about solving the problem of migrating users to new configuration (such as new protocols, new bootstrap peers) by only writing settins that were actually modified to .ipfs/config, and falling back to the built-in default values when reading a config? This is more or less the norm with a lot of software.
<jbenet>
ion: that makes sense. though also being able to look at the config can tell you what options there are to change. we may be able to show this some other way, too.
Encrypt has joined #ipfs
<jbenet>
ion: maybe write up a proposal on go-ipfs repo?
<ion>
jbenet: 0) “ipfs config show”, 1) .ipfs/config could have keys like "Swarm.default", "Bootstrap.default" etc. which are ignored when reading and overwritten with the defaults of the current version when writing. I will write an issue.
<ion>
Actually, perhaps _ instead of . given how JavaScript handles JSON objects.
<ipfsbot>
[go-ipfs] chriscool created fix-test-cmp-args (+4 new commits): http://git.io/vciQy
<ipfsbot>
go-ipfs/fix-test-cmp-args 93aa8d0 Christian Couder: t0220: check if 'ipfs bitswap' fails...
<ipfsbot>
go-ipfs/fix-test-cmp-args 7b81e4e Christian Couder: t0220: use test_must_be_empty()...
<ipfsbot>
go-ipfs/fix-test-cmp-args bf8adc6 Christian Couder: t0044: fail when 'ipfs add' fails...
simpbrain has quit [Quit: Leaving]
<ipfsbot>
[go-ipfs] chriscool opened pull request #1792: Fix test cmp args (master...fix-test-cmp-args) http://git.io/vci7v
<Luzifer>
jbenet: yeah as soon as you're using CGO gobuilder is out… :( I don't have build machines on all ${GOARCH}'s and from what I read until now cross-compiling CGO programs is a huge pain or even impossible… :(
<Luzifer>
(and might even not be able to learn because there are no windows-builder and mac-builder and ${GOARCH}-builder…)
<jbenet>
yeah, understood.
<jbenet>
we'll likely have to have VMs for all the archs when we go the cgo route. we're considering limiting the cgo creep by putting those pieces in other programs, like a separate binary.
<jbenet>
we'd have some copy overhead for transports, but it will make our building easier.
<Luzifer>
thats one of the reasons I have a strong bias against CGO and immediately throw away any library having C-files in the repo…
<yoshuawuyts>
not sure what to think of it, though I find it interesting
tsenart has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
Guest73396 has joined #ipfs
bedeho has quit [Ping timeout: 252 seconds]
chriscool has quit [Ping timeout: 246 seconds]
ilyaigpetrov has quit [Remote host closed the connection]
chriscool has joined #ipfs
amstocker has quit [Ping timeout: 265 seconds]
<jbenet>
we need to improve bitswap a lot. if we can get simple ways to produce lots of these perf graphs on bitswap, coupled with good tooling (maybe parting from bssim) it would help a ton. cc ion whyrusleeping rht
tsenart has joined #ipfs
Encrypt has joined #ipfs
<DavidBurela>
perf graphs like what? Global status of the IPFS network, and how much is being bitswapped? like the Ethereum dashboard? https://stats.ethdev.com/
nikor has joined #ipfs
NeoTeo has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
tsenart has quit [Ping timeout: 250 seconds]
<ipfsbot>
[webui] jbenet pushed 2 new commits to master: http://git.io/vcPca
<ipfsbot>
webui/master f696d90 Juan Benet: Merge pull request #86 from CaioAlonso/master...
<ipfsbot>
webui/master b547700 Caio Alonso: updates the 'back to console home' link in the 404
<jbenet>
DavidBurela: no, of test networks. like, spawn 10 nodes, run through a workload, gather data.
<DavidBurela>
gotcha
ygrek has quit [Ping timeout: 246 seconds]
NeoTeo has joined #ipfs
reit has quit [Quit: Leaving]
reit has joined #ipfs
DrPete_ has joined #ipfs
DrPete has quit [Ping timeout: 246 seconds]
chriscool has quit [Ping timeout: 240 seconds]
jhulten_ has joined #ipfs
jhulten_ has quit [Ping timeout: 246 seconds]
Encrypt has quit [Quit: Quitte]
_fil_ is now known as _fil_s_en_va_
_fil_ has joined #ipfs
slothbag has quit [Read error: Connection reset by peer]
infinity0 has quit [Ping timeout: 268 seconds]
infinity0 has joined #ipfs
infinity0 has quit [Remote host closed the connection]
infinity0 has joined #ipfs
NeoTeo has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
infinity0 has quit [Remote host closed the connection]
<ion>
jbenet: ack
infinity0 has joined #ipfs
alipico has quit [Quit: Page closed]
<ipfsbot>
[go-ipfs] jbenet created mocknet-fix (+1 new commit): http://git.io/vcPK3
<ipfsbot>
go-ipfs/mocknet-fix a7576c9 Juan Batiz-Benet: mocknet: conn close idempotent with process....
rschulman has joined #ipfs
NeoTeo has joined #ipfs
ehmry is now known as emery
emery is now known as ehmry
ehmry is now known as emery
emery is now known as ehmry
dysbulic has joined #ipfs
Guest73396 has quit [Ping timeout: 256 seconds]
Soft has joined #ipfs
<ipfsbot>
[go-ipfs] jbenet pushed 1 new commit to master: http://git.io/vcPSv
<ipfsbot>
go-ipfs/master e0f0417 Juan Benet: Merge pull request #1796 from ipfs/mocknet-fix...
<ipfsbot>
[go-ipfs] rht opened pull request #1797: Add context to coreunix.Cat argument (master...fix/cat) http://git.io/vcXvZ
chriscool has joined #ipfs
bedeho has joined #ipfs
Encrypt has quit [Ping timeout: 268 seconds]
Encrypt has joined #ipfs
AriUgwu_ has quit [Ping timeout: 252 seconds]
_fil_ is now known as _fil_znc
_fil_s_en_va_ is now known as _fil_
gamemanj has joined #ipfs
jhulten_ has joined #ipfs
<CaioAlonso>
how scared do you guys get when thinking about the incoming copyright-issues-avalanche coming towards IPFS?
<CaioAlonso>
not that I want this to happen, far from it, but this project is going so well and is delivering something so adequate to replace bittorrent that it gets scary
jhulten_ has quit [Ping timeout: 240 seconds]
<CaioAlonso>
just the other day I saw someone hosting the dark knight on ipfs (an 8ch/tech/ thread...) and it got me worried
<achin>
what in particular are you worried about?
<CaioAlonso>
that the owners of public gateways may have legal problems, and the owners of the bootstrap nodes too
<achin>
there is a proposal/plan to allow gateways to blacklist certain hashes, right? if that works as expected, it seems like gateway owners would be in the same bucket at http content hosters
<whyrusleeping>
CaioAlonso: we're going to have a system to prevent bad bits from coming through our gateways
<CaioAlonso>
yeah, I think we're going to see a lot of whitelist-only gateways
<whyrusleeping>
and also make it easy for others to do the same on their gateways
<whyrusleeping>
yeah, whitelists will be very common
<achin>
in fact, if i was a copyright owner, ipfs gives me a nice way to find out who is hosting my content ("ipfs dht findprovs")
pfraze has joined #ipfs
<achin>
whitelists will likely be common for non-copyright reasons (i.e. bandwidth/capacity reasons)
<achin>
if i run a gateway, it'll be for the purpose of providing a HTTP interface to my own content, so i'll only allow my own content to be available via my gateway
chriscool has quit [Ping timeout: 240 seconds]
<achin>
(in short, i do not think things will radically change when comparing copyright+http with copyright+ipfs. the content-addressible nature of ipfs might actually make things better for everyone)
<CaioAlonso>
achin good point
<rschulman>
whyrusleeping: Anyone talking to NCMEC yet about getting their hash list?
<rschulman>
(NCMEC = National Center for Missing and Exploited Children -> child porn list)
<CaioAlonso>
yeah, warez is a mild nuisance, exploited children and other kinds of disturbing content are a whole new level
<achin>
hmm, it just occurs to me that the same piece of data can actually be made avaialble under a fairly arbitrary number of hashes
<CaioAlonso>
blacklists are a cat and mouse game
<rschulman>
achin: If you change just one bit in the input, you mean?
<achin>
rschulman: no, if you change the number/content of the leaf merkle nodes
<whyrusleeping>
rschulman: we havent yet. we're still working on the infrastructure for the lists stuff
<rschulman>
whyrusleeping: Well, let me know when you do, I can make the outreach. Something I'm actually good at!
<achin>
big files are split up in to multiple blocks. you can split up a big file in a bunch of different ways, each way would yield the same content (once put back together via "ipfs get"), but have different hashes
<ion>
Man, imagine having to maintain *that* list.
<rschulman>
achin: Well, yes, but the hash of the final resulting file will be the same.
<rschulman>
After IPFS does its magic.
chriscool has joined #ipfs
<CaioAlonso>
ion Google employees taking care of this kind of work are known to have psychological problems after a few months
<rschulman>
ion: Its not cool. Lots of large companies give their content teams free therapy and shit like that.
<achin>
no, it wouldn't
<whyrusleeping>
if you do a straight shasum of the file after its put together, it would be the same
<rschulman>
achin: How not? The result of ipfs get will be the same, no?
<rschulman>
whyrusleeping: Yeha.
<whyrusleeping>
but chunked hashes would be different
atrapado has joined #ipfs
Animazing has quit [Ping timeout: 264 seconds]
<whyrusleeping>
ion achin one of you was making pretty bitswap transfer graphs, right?
<rschulman>
right, but any hash you get to check against by an outside party isn't going to know or care about IPFS internals.
<whyrusleeping>
coffee hasnt quite kicked in and my memory is failing me
<achin>
whyrusleeping: it was ion that made the pretty pictures
<ion>
whyrusleeping: I made the two graphs in the issue.
<rschulman>
its just going to be a sha of the final fial
<rschulman>
*file
<whyrusleeping>
ion: cool, do you remember what issue that was?
<whyrusleeping>
i'm going to implement those bitswap changes pretty soon
<achin>
rschulman: gateways are not going to want to have to fetch the data before figuring out if it should be blocked
<achin>
rschulman: let me try to put together an example (since i'm not sure this can be done using stock ipfs tools)
<rschulman>
achin: Yes, that's a good point, they would prefer to be allowed to rely on the IPFS hash
Animazing has joined #ipfs
Quiark has quit [Ping timeout: 240 seconds]
<ion>
whyrusleeping: Have you thought of smarter generation of the wantlist in the first place? Like, start adding chunks from the next top-level object before the wantlist becomes empty to avoid the sawtooth pattern in the first graph, and avoid adding 8000 blocks to the wantlist at once which seems to result in the blocks being received in a random order and almost none of the files being written until the very end
<ion>
in the second graph?
<whyrusleeping>
ion: yeah, i have a different fetching algorithm around
<whyrusleeping>
it uses a lot more memory
<whyrusleeping>
but i think i can write a compromise between the two
<ion>
Alright, sounds nice.
simpbrain has joined #ipfs
amstocker has joined #ipfs
mhhf has joined #ipfs
<mhhf>
is somebody here familiar with the node-ips-api?
<ion>
whyrusleeping: Some filesystems try to predict whether a given block will be needed soon or at all based on whether the reads seem to follow a linear pattern or not. Their algorithms must be fast for that to be an optimization in the first place. Perhaps there’s something useful in the literature.
pfraze has quit [Remote host closed the connection]
<ilyaigpetrov>
May I use IPFS for live video broadcasting today already?
<whyrusleeping>
ilyaigpetrov: not yet
chriscool has joined #ipfs
Tv` has joined #ipfs
thorax has joined #ipfs
jhulten_ has joined #ipfs
<achin>
phew, ok after some hacking i have these two hashes: Qmf412jQZiuVUtdgnB36FXFX7xg5V6KEbSJ4dpQuhkLyfD and Qmb4ftM1Dw6ChxesN4KLgMpvQV2NfSv4iCVdSkN43fxdTS
Encrypt has quit [Quit: Eating time!]
<achin>
if you fetch them with "ipfs get" they will have identical content (cc rschulman )
<rschulman>
gamemanj: Same content, but different chunking internally, so the merkle dag has a different hash.
<rschulman>
achin: So yes, that will be an interesting question.
<gamemanj>
ah, different chunking... though why would someone do that?
<achin>
to get around hash blacklists
ilyaigpetrov has quit [Remote host closed the connection]
<gamemanj>
Well, this is just plain weird... I was trying to work out why the webui's DAG reported 2 bytes less than ipfs object stat, and tried creating a new object... QmdfTbBqBPQ7VNxZEYEj14VmRuZBkqFbiwReogJgS1zR1n
<gamemanj>
Is that the IPFS equal to a null hash?
arrdem has joined #ipfs
simpbrain has quit [Ping timeout: 264 seconds]
nikor has quit [Quit: Lost terminal]
<ipfsbot>
[go-ipfs] whyrusleeping force-pushed feat/utp from 657809f to d3e9792: http://git.io/vcXyV
<gamemanj>
That doesn't seem like a normal part of a hash.
<achin>
oh, that's the first few bytes of the multihash. it identifies the hash type (SHA 256 in this case) and then hash length
<achin>
i probably should have stripped those off
<achin>
(it's why all ipfs hashes start with Qm)
<gamemanj>
So could non-Qm hashes be produced, or aren't they supported?
<achin>
in theory they should be supported, but i don't believe the current version of go-ipfs has any support for creating them
<achin>
i created one manually, but no peers were able to get it (but i'm not sure why)
amstocker has quit [Ping timeout: 255 seconds]
<achin>
for example, the empty object you listed above could ahve the hash 8VwwAr1TJYarvTxAhesR2dntLJjthkFwsRoDVyajBSgpZQiK9qQj5df9WdbDX8Y3yTJmdWBzsrKfDjrbWiWjCmjApd
<arrdem>
I should be able to do ipfs list /ipfs/<my-hash>, right?
<arrdem>
well ls not list
<achin>
in theory, yes
<achin>
but if you try it, it'll just hang there, because my ipfs node doesn't seem to know how to tell people that i ahve this hash
<arrdem>
Ok. That explains it. I'm able to ls and cat objects on other machines, but I can't ls or cat local stuff :|
<achin>
arg, sorry. i got confused about who i was talking to
<achin>
arrdem: i thought you were gamemanj ! sorry
<arrdem>
all good
<achin>
what was your question again? :P
<gamemanj>
"So could non-Qm hashes be produced, or aren't they supported?"
<ipfsbot>
[go-ipfs] whyrusleeping force-pushed feat/utp from d3e9792 to 128624a: http://git.io/vcXyV
<ipfsbot>
go-ipfs/feat/utp 13b11ee Jeromy: vendor in utp code...
<ipfsbot>
go-ipfs/feat/utp dd49d4e Jeromy: refactor of swarm dialing, getting closer......
<achin>
no i mean what was arrdem trying to ask
<arrdem>
According to `ipfs id` and /webui this machine is /ipfs/QmRvvpD1zTdztXwfXdvHG1pnrejCXaHarrBgfqCDZwTHvB but I can't `ipfs ls` or `ipfs cat` anything on this box. Objects on other machines seem visible, but objects on this box cause the ipfs program to hang.
<arrdem>
at least that's the observed behavior
<achin>
you can't use "ipfs ls" or "ipfs cat" with hashes that are peerIDs
<achin>
but if you try to "ls" or "cat" a hash that you don't have locally, your ipfs daemon will go find it and download it for you
<arrdem>
but /ipfs/<peerid>/<objid> should work, right?
<arrdem>
(for ls or cat)
<achin>
no, but maybe you are thinking of /ipns/<peerid>/<objid> ?
<achin>
(note the difference: ipfs versus ipns)
<jbenet>
wait, i just finally built with 1.5 -- how are people dealing with this api failing all the time crap
<achin>
with the IPNS system, you can publish a message that states "i am peer <hash> and i am publishing object <hash>", which allows other nodes to look up the objecthash based only on your peerID hash
<jbenet>
whyrusleeping this is like unusable
<achin>
but in general, it is not possible to list what a given peer (even your own) has available
<arrdem>
achin: gotcha
<achin>
(the expection for your own peer is that you can run "ipfs refs local" to see all blocks your node has, but this isn't a useful directdory listing or anything like that)
<gamemanj>
So that makes discovery of content... something close to impossible?
<achin>
you mean without using IPNS?
<jbenet>
oh that's bizarre-- nodes A, B, C. a has f1. B gets all of f1. then ipfs refs f1 on B stalls. ipfs refs f1 on C stalls. i kill B and C multiple times and try again. always stall on the same hash. (~10 down.
<gamemanj>
Even with IPNS you need to know a peer ID, which limits discovery to peers on the network that you're connected to,
<jbenet>
i repeated 10x times. alaways same result. stall, eventual timeout
<gamemanj>
unless there are links externally or internally (none of which someone new may know)
<jbenet>
THEN, despite B having the whole thing, i kill A. B resumes listing all refs, and then C makes forward progress
<jbenet>
wtf. cc whyrusleeping
<achin>
gamemanj: in my mind, it's not too different from http. given a domain name, how are you supposed to discover the name itself, and then how are you supposed to discover content available on that domain?
<gamemanj>
achin: That's true, but with HTTP there are search engines. I suppose it should be easier to make IPNS search (because someone could just scrape the peerlist and do IPNS lookups on everyone)...
deltab_ has joined #ipfs
dPow has joined #ipfs
revolve_ has joined #ipfs
edsu_ has joined #ipfs
afdudley0 has joined #ipfs
ralphthe1inja has joined #ipfs
<achin>
that's an interesting idea. i wonder how many peers in the network have published something via ipns... let's find out!
stick` has joined #ipfs
<achin>
arrdem: also note that you *can* get a directory listing if someone gives you a hash of a directory
hrjet has quit [Ping timeout: 260 seconds]
M-edrex has quit [Ping timeout: 260 seconds]
vijayee_ has quit [Ping timeout: 260 seconds]
M-jgrowl has quit [Ping timeout: 260 seconds]
deltab has quit [Ping timeout: 260 seconds]
edsu has quit [Ping timeout: 260 seconds]
cryptix has quit [Ping timeout: 260 seconds]
reit has quit [Ping timeout: 260 seconds]
Tristitia has quit [Ping timeout: 260 seconds]
Erkan_Yilmaz has quit [Ping timeout: 260 seconds]
wemeetagain has quit [Ping timeout: 260 seconds]
rabbitface2 has joined #ipfs
stick` has left #ipfs [#ipfs]
cryptix_ has joined #ipfs
Erkan_Yilmaz has joined #ipfs
hrjet_ has joined #ipfs
zml_ has joined #ipfs
Tristit1a has joined #ipfs
[tymat] has joined #ipfs
M-edrex has joined #ipfs
NeoTeo_ has joined #ipfs
M-whyrusleeping1 has joined #ipfs
M-jgrowl has joined #ipfs
M-whyrusleeping has quit [Ping timeout: 240 seconds]
dylanPowers has quit [Quit: No Ping reply in 180 seconds.]
bengl has quit [Ping timeout: 240 seconds]
ralphtheninja has quit [Ping timeout: 240 seconds]
revolve has quit [Ping timeout: 240 seconds]
afdudley has quit [Ping timeout: 240 seconds]
NeoTeo has quit [Ping timeout: 240 seconds]
tymat has quit [Ping timeout: 240 seconds]
zml has quit [Ping timeout: 240 seconds]
Keiya has quit [Ping timeout: 240 seconds]
ion has quit [Ping timeout: 240 seconds]
rabbitface1 has quit [Ping timeout: 240 seconds]
bengl has joined #ipfs
wemeetagain_ has joined #ipfs
[tymat] is now known as tymat
Keiya_ has joined #ipfs
ion- has joined #ipfs
ion- is now known as ion
bengl is now known as Guest80443
<gamemanj>
Aha
<gamemanj>
I found something with my automatic IPNS resolver
<gamemanj>
I think, anyway. Qmf2KsXpfQmV2o38PGaF2PvTVGBpmQiUCCbe9GkXUUSZuV
<achin>
that was just a test
<gamemanj>
Oh :(
<gamemanj>
well, at least it's actually finding things
<arrdem>
how muich network traffic is ipfs expcted to generate... right now if I run two nodes looks like I can consistently knock over my router or modem -> pole link whichever
<achin>
there are currently 215 nodes in the network... /me starts to query all of them for ipns info
<gamemanj>
I've found two so far, but the auto-resolver isn't done yet
<achin>
arrdem: i *think* the issue is not just about bandwidth, but also about the large number of connections ipfs makes
atrapado has quit [Quit: Leaving]
<arrdem>
achin: I've only had ~80 hosts connected, which shouldn't be a problem...
<achin>
several others have reported that ipfs does bad theirs to either the routers or modems
<achin>
i've never had the issue happen to me, so i've not been able to investige first-hand. so i don't have a lot of data about the problem
<gamemanj>
it should probably be connection-limited, if you think about it - connecting to > 10 peers is a little much, surely?
<arrdem>
jbenet: I hear ya
<jbenet>
gamemanj: not for dht-connected nodes. fd limited, yes (with utp/udt/quic we can drop to one fd for all net io), but connections to peers, no. you want a large fanout for a fast dht.
<achin>
and whyrusleeping is hard at work with the UTP stuff as we speak!
<multivac>
mappum: 2015-10-03 - 11:32:33 <davidar> ask mappum how Gateway.RootRedirect is supposed to work for _p4bl0
<ion>
whyrusleeping: Thanks
silotis has quit [Remote host closed the connection]
silotis has joined #ipfs
<m4gyk>
newbie question: if i create an ipfs node out of a directory on my machine, is it possible to add new files to that node at a later date?
chriscool has quit [Ping timeout: 265 seconds]
<clever>
all ipfs objects are read-only, they can never be changed
<clever>
but you can create a new object, which re-uses files from an old one
<m4gyk>
okay, thanks
sonatagreen has joined #ipfs
<mappum>
m4gyk: thanks to whyrusleeping 's very recent work (the mutable filesystem), you'll soon be able to do that with the `ipfs files` command
<mappum>
so you'll be able to add/remove files and get the new hash of the directory
<Bedlamb>
so, does that handle things automatically under the surface when pulling resources?
<Bedlamb>
like, can we tell that we have a new version of an old resource so we don't have to redownload everything, but merely the changes (like any version control)?
<ion>
Yes, all links to unchanged data point to the same objects.
<Bedlamb>
right, makes sense!
Bedlamb has quit [Quit: Leaving.]
Bedlamb has joined #ipfs
pfraze has joined #ipfs
<lgierth>
!botsnack
<pinbot>
om nom nom
<sonatagreen>
lolwut
<achin>
nom nom om
<lgierth>
just wanted to check it's alive
m4gyk has quit [Ping timeout: 268 seconds]
<achin>
plus snack
<M-matthew>
congrats on getting techcrunched ipfsfolks :)
Bedlamb has quit [Quit: Leaving.]
Bedlamb has joined #ipfs
hayden has quit [Quit: leaving]
rendar has quit []
<jbenet>
M-matthew thanks :)
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
NeoTeo_ has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
<rschulman>
woo, reading now!
<ion>
ipf{,olk}s
<ion>
H-hello M-matthew
Bedlamb has quit [Quit: Leaving.]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]