<wking>
Writing a generic BubbleChangesTo(children..., ancestors...) without holding sizeable chunks of DAG in memory seems difficult.
<Tv`>
that's why it's *allowed* to push them out on memory pressure
<Tv`>
they just become not-dirty
<wking>
what's "it"? Your background dirty-cleanup task?
<wking>
or does Node.BubbleChangesTo(ancestors...) just add dirty nodes to a queue, and only flush the queue when you hit a memory/queue-size/staleness threshold?
inconshreveable has quit [Remote host closed the connection]
<Tv`>
the mechanism you think fits in a single method of Node ;)
<wking>
my initial idea had no queue. Or backlinks ;). So clearly needs more thought
rschulman has quit [Quit: Leaving.]
Wallacoloo has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
rschulman has joined #ipfs
domanic has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
inconshreveable has joined #ipfs
inconshreveable has quit [Remote host closed the connection]
hellertime has quit [Ping timeout: 252 seconds]
hellertime has joined #ipfs
domanic has quit [Ping timeout: 264 seconds]
bengo has quit [Quit: Computer has gone to sleep.]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
bengo has joined #ipfs
tilgovi has quit [Read error: Connection reset by peer]
bengo has quit [Quit: Computer has gone to sleep.]
bengo has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
bengo has quit [Quit: Computer has gone to sleep.]
pfraze has quit [Remote host closed the connection]
anshukla has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<ipfsbot>
[go-ipfs] NodeGuy opened pull request #1215: Add CORS middleware handler to the API. (master...cors) http://git.io/vUqA7
lgierth_ has quit [Quit: Ex-Chat]
<whyrusleeping>
yeah... i remember why i *didnt* implement this previously
<whyrusleeping>
fitting things nicely in memory sucks hard
<whyrusleeping>
also something something premature optimization
rschulman has quit [Quit: Leaving.]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
rschulman has joined #ipfs
rschulman has quit [Client Quit]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
pfraze has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
rschulman has joined #ipfs
<whyrusleeping>
ugh, the bitswap testnet is a terrible approximation
<whyrusleeping>
its making this refactor really obnoxious
<whyrusleeping>
as soon as i start relying on the 'connected'/'disconnected' notifications, nothing works in testing
<ipfsbot>
go-ipfs/master 19823c6 W. Trevor King: path/resolver_test: Test recursive Link resolution...
<ipfsbot>
go-ipfs/master d6fc414 Juan Batiz-Benet: Merge pull request #1212 from wking/fix-recursive-path-resolution...
sharky has joined #ipfs
<whyrusleeping>
agreed on the tests
<whyrusleeping>
when i get to seattle and get settled down ill probably take a few entire days to just write more tests
<whyrusleeping>
wow. so bitswaps 'TestLargeSwarm'
<whyrusleeping>
is being slowed down massively by eventlog
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<whyrusleeping>
taking out one eventlog line changed the runtime of the test from 33 seconds to 10
inconshreveable has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
anshukla has quit [Remote host closed the connection]
anshukla has joined #ipfs
anshukla has quit [Remote host closed the connection]
<whyrusleeping>
jbenet: i gotta rip out some eventlogs... these are crazy expensive
anshukla has joined #ipfs
anshukla has quit [Read error: Connection reset by peer]
anshukla has joined #ipfs
inconshreveable has quit [Remote host closed the connection]
anshukla has quit [Remote host closed the connection]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<jbenet>
yeah agreed-- we should nuke them writing to disk.
anshukla has joined #ipfs
anshukla has quit [Remote host closed the connection]
pfraze has quit [Remote host closed the connection]
<whyrusleeping>
well, the writing to disk isnt the bad part
<whyrusleeping>
its the string formatting that they do
<whyrusleeping>
sooooo much wasted memory
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
therealplato has joined #ipfs
Tv` has quit [Quit: Connection closed for inactivity]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
chriscool has joined #ipfs
inconshreveable has joined #ipfs
inconshr_ has joined #ipfs
inconshreveable has quit [Ping timeout: 265 seconds]
inconshreveable has joined #ipfs
<ipfsbot>
[go-ipfs] whyrusleeping created refactor/bitswap (+1 new commit): http://git.io/vUmi0
<ipfsbot>
go-ipfs/refactor/bitswap 7dbb329 Jeromy: large refactor of bitswap, implement peermanager to manage sending of messages
inconshr_ has quit [Ping timeout: 265 seconds]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
domanic has quit [Read error: Connection reset by peer]
inconshr_ has joined #ipfs
<ipfsbot>
[go-ipfs] whyrusleeping opened pull request #1218: large refactor of bitswap, implement peermanager to manage messages (master...refactor/bitswap) http://git.io/vUm1O
crossdiver has joined #ipfs
inconshreveable has quit [Ping timeout: 265 seconds]
<whyrusleeping>
thats ugly, but could use some feedback as i'm working on it
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
inconshr_ has quit [Remote host closed the connection]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
domanic has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
domanic has quit [Ping timeout: 240 seconds]
<ipfsbot>
[go-ipfs] whyrusleeping pushed 1 new commit to refactor/bitswap: http://git.io/vUmN6
<ipfsbot>
go-ipfs/refactor/bitswap 9a08737 Jeromy: clean up, and buffer channels
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<Bat`O>
whyrusleeping: would you accept a patch that make /api/v0/refs return a json, as well as the size of each block returned ?
<Bat`O>
whyrusleeping: the same way as /api/v0/ls does
chriscool has quit [Ping timeout: 256 seconds]
<Bat`O>
jbenet: ^ ?
<jbenet>
Bat`O: yeah-- it isn't json now?
<Bat`O>
no, just a list of raw hash
<Bat`O>
one per line
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<jbenet>
Bat`O: oh the problem is that it's streamed
<jbenet>
Could stream it as an array, or as nd-json
<Bat`O>
jbenet: i'm looking at the code, it's not as easy as i hoped
<Bat`O>
<-- go novice
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
grncdr has joined #ipfs
<Bat`O>
jbenet: should i remove the ErrPassThroughReader stuff and return a struct instead ?
* Bat`O
tries stuff
<cryptix>
hey ppl
<cryptix>
jbenet: did i cause 1212 :x
<cryptix>
Bat`O: if you need help, let me know
grncdr has quit [Ping timeout: 255 seconds]
grncdr has joined #ipfs
<ipfsbot>
[go-ipfs] cryptix pushed 1 new commit to refactor/httpGateway: http://git.io/vUYLx
<cryptix>
jbenet: chose io because it already has the dirBuilder, seemd more consistent than having them spread across
<jbenet>
Bat`O no-- you still need a reader, becuase you still want to stream. the thing is marshaling it out could be streamed json. i dont think there is a streaming json thing out there (for returning part of an array each time), so maybe we can return nd-json, like `ipfs stats bw --poll` and `ipfs dht` works
* jbenet
cryptix: *shrug* sgtm not really my domain
<jbenet>
cryptix: lmk when you want CR, etc
<cryptix>
jbenet: we will (hopefully soon) get a streaming json de/encoder in stdlib
<cryptix>
jbenet: sure - just want to get that putHandler fixed first
<jbenet>
cryptix: ehh. arrays suck anyway. wish json had a "objects back to back = array" case
<jbenet>
sgtm
<Bat`O>
unrelated, how can i know if a block is available locally ?
<cryptix>
Bat`O: hmmm.. why wouldnt you just assume it is?
<Bat`O>
cryptix: let say i want to download some big files and track the progress
<Bat`O>
or a tree of files
<cryptix>
Bat`O: wanted that for some time, too - i think we need that facility closer to the core though
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<cryptix>
meaning: polling 'is x local yet?' might work but... not so nice
<cryptix>
search 'significant caveat' tldr: need to call runtime.SetBlockProfileRate()
notduncansmith has quit [Read error: Connection reset by peer]
<cryptix>
maybe want to do that in IPFS_DEBUG=t
<jbenet>
cryptix: sgtm.
<jbenet>
may want to ask whyrusleeping though. possible it could be its own env var.
<insanity54>
what's the multiaddress? Is that in here somewhere? -> /ip4/172.17.0.3/tcp/4001/ipfs/QmdGpfxT1zxi59PFkwrtD4Lr8aN3zr6dSXyP5zm8JaEGpe
<cryptix>
jbenet: we already have IPFS_PROF for the mem profiling, thats a good place i think
<insanity54>
bootstrap --help: [<peer>]... - A peer to add to the bootstrap list (in the format '<multiaddr>/<peerID>')
<cryptix>
insanity54: the hole thing is a multiaddr
<insanity54>
cryptix: I see. Any idea why `ipfs bootstrap add /ip4/25.196.147.100/tcp/4001/QmaMqSwWShsPg2RbredZtoneFjXhim7AQkqbLxib45Lx4S` says, "Error: no protocol with name QmaMqSwWShsPg2RbredZtoneFjXhim7AQkqbLxib45Lx4S" ?
<cryptix>
you are missing the /ipfs/ bafter the tcp port :)
<insanity54>
oh lawdy
<insanity54>
cryptix: thanks
<cryptix>
insanity54: de nada :)
<cryptix>
jbenet: added runtime.SetBlockProfileRate(1) to 'func profileIfEnabled', started adding the 370megs of gopath github.com profile (which takes longer than 15minutes) and took some profiles: /ipfs/QmeHfchizRtKD7RbkW2ZwuKqRT5MdUx8wuYQJZoEDweafP
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<cryptix>
the pprof tool is a bit eratic, sometimes overwriting the .svg's...
lgierth has quit [Quit: Ex-Chat]
neofreak has quit [Remote host closed the connection]
<cryptix>
hmmm - should propably run this with no peers for comparison
<cryptix>
sooo much todo...
nausea has joined #ipfs
nausea has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<jbenet>
lol
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
notduncansmith has quit [Read error: Connection reset by peer]
chriscool has quit [Ping timeout: 250 seconds]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
pfraze has joined #ipfs
<cow_2001>
ipfs pin ls shows nothing
<cow_2001>
but i've ipfs add a ton of files
* cow_2001
is worried
<insanity54>
I think if you `ipfs add` something, it doesn't show up in your pin list. (I think.)
<cow_2001>
ah
<insanity54>
pin list is if I share something that you like, you can pin it
<insanity54>
and it'll cache on your machine
<insanity54>
s/pin list/pinning
EricJ2190 has joined #ipfs
<cow_2001>
how do i get the list of files i've added?
<insanity54>
I don't actually know how you would see the files you've `ipfs add`ed. I know they go in ~/.ipfs but idk how to get the ipfs process to give a list
<insanity54>
k maybe they don't go in ~/.ipfs, seeing as I don't have that dir.
<insanity54>
cow_2001: I want to know how to list added files too, so I'll look for a way
<Vendan>
`ipfs pin ls`
<cow_2001>
nope
<cow_2001>
i've ipfs add a ton of files and they're not on ipfs pin ls
<insanity54>
Vendan: doesn't that not include added files?
<Vendan>
did you add them directly, or recursively through a directory?
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<insanity54>
in my case I think it would have been directly. `echo 'fart' > fart.txt; ipfs add fart.txt`
<Vendan>
if I do that, then do `ipfs pin ls --type=all` and grep for the hash spit out by ipfs add, that hash is in there
inconshreveable has joined #ipfs
<insanity54>
Vendan: oh yep. that's the problem, cow_2001
<insanity54>
pin types
<insanity54>
--type=all
<Vendan>
not too helpful, cause you don't get anything about what file is each pin, but better then nothing
inconshreveable has quit [Remote host closed the connection]
<cow_2001>
can i somehow convert my repository into a new id with twice as many bits?
<Vendan>
`ipfs init -b <bitcount> -f`?
<cow_2001>
won't it erase everything?
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<Vendan>
no idea
<insanity54>
lmao
<insanity54>
I can do it I've got unimportant filez
<insanity54>
`ipfs init -b 2046 -f` => Error: Could not convert empty value to type 'int' (for option '-b')
<Vendan>
maybe -b=2048
<insanity54>
ya that did the trick
<insanity54>
now to see if me files are still there
<insanity54>
I have a shorter list now when I do `ipfs pin ls --type=all`
<insanity54>
I'm assuming those are the default pins
<insanity54>
readmes, examples, etc.
<cow_2001>
where do i get those readmes?
<cow_2001>
ipfs cat …?
<cow_2001>
what's the path
<insanity54>
idk the path... do you want the hashes?
<cow_2001>
okay
<insanity54>
oh
<insanity54>
i see
<insanity54>
"go get started, enter:" ipfs cat /ipfs/Qmc8ZcdAUduQfC32oSXnLsUyDb7GNZ9rX4VjrtB4s2212v4/readme
<insanity54>
i entered that by hand (ipfs running on different computer) let me verify i typed it correctly
notduncansmith has quit [Read error: Connection reset by peer]
lgierth has joined #ipfs
headbite has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<jbenet>
cow_2001: I'd init a node in another directory and just copy over the config file
<jbenet>
IPFS_PATH=temp ipfs init
<jbenet>
cp temp/config ~/.ipfs/config
<jbenet>
(Or only the id section or whatever)
<cow_2001>
too late :D
<whyrusleeping>
Lol, we need an easy "ipfs reinit" command that keeps your I'd around
<whyrusleeping>
ID*
<cow_2001>
ipfs init --reinit?
<cow_2001>
but id is defined by the key, so i cannot really reuse it if i want to increase key's bits
tilgovi has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
patcon has joined #ipfs
grncdr has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
lgierth has quit [Quit: Ex-Chat]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
grncdr has quit [Quit: Leaving.]
<jbenet>
mafintosh: there's no pbs eof, right?
<whyrusleeping>
jbenet: afaik, no
<jbenet>
I think there's should be a well known eof message that pops out. Think sending some pbs and then switching to --idk, http. If haven't setup a serialized muxer ahead of time the connection is hosed.
ei-slackbot-ipfs has quit [Remote host closed the connection]
ei-slackbot-ipfs has joined #ipfs
<mafintosh>
jbenet: you can just add a eos message to your schema
<mafintosh>
eof
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<jbenet>
mafintosh: does protobuf say anything about using tag 0?
<mafintosh>
jbenet: no i think thats fine
<mafintosh>
jbenet: good idea
<Vendan>
"The smallest tag number you can specify is 1"
notduncansmith has quit [Read error: Connection reset by peer]
tilgovi has quit [Ping timeout: 240 seconds]
anshukla has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
anshukla has quit [Remote host closed the connection]
grncdr has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
Wallacoloo has joined #ipfs
anshukla has joined #ipfs
Wallacoloo has left #ipfs [#ipfs]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
Wallacoloo has joined #ipfs
Wallacoloo has left #ipfs [#ipfs]
crossdiver has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
grncdr has quit [Quit: Leaving.]
grncdr has joined #ipfs
tilgovi has joined #ipfs
domanic has joined #ipfs
grncdr has quit [Quit: Leaving.]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
anshukla has quit [Remote host closed the connection]
anshukla has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
ehmry has joined #ipfs
inconshreveable has joined #ipfs
inconshr_ has joined #ipfs
inconshr_ has quit [Read error: Connection reset by peer]
inconshr_ has joined #ipfs
inconshreveable has quit [Ping timeout: 256 seconds]
ehmry has quit [Quit: Leaving]
grncdr has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
chriscool has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
shazow has joined #ipfs
ehmry has joined #ipfs
<shazow>
is there any guidance/policy for using ipfs's DHT to bootstrap a DHT for an unrelated p2p project?
anshukla has quit [Remote host closed the connection]
anshukla has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
ehmry has quit [Ping timeout: 246 seconds]
<whyrusleeping>
shazow, just be nice to the dht
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
Tv` has joined #ipfs
inconshr_ has quit [Remote host closed the connection]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
chriscool has quit [Ping timeout: 245 seconds]
patcon has quit [Ping timeout: 264 seconds]
Wallacoloo has joined #ipfs
domanic has quit [Ping timeout: 252 seconds]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<wking>
jbenet: For #1208 to move forward, I'd like to open a new issue to flesh out the fs-root stuff currently sketched out in jbenet/random-ideas#28. Where should I open that issue?
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<wking>
I feel like the fs-root idea ("use paths to fetch content across assorted protocols") is pretty independent of IPFS ("use a Merkle DAG to store content-addressable content with a DHT for routing"), so it doesn't feel like a great fit for the main IPFS repositories
<wking>
and s/fetch/access/, since writing is important too ;)
anshukla_ has joined #ipfs
anshukla has quit [Ping timeout: 256 seconds]
anshukla_ has quit [Ping timeout: 244 seconds]
grncdr has quit [Quit: Leaving.]
anshukla has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
pfraze has quit [Remote host closed the connection]
patcon has joined #ipfs
patcon has quit [Ping timeout: 245 seconds]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
anshukla has quit [Remote host closed the connection]
zignig has quit [Ping timeout: 256 seconds]
zignig has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]