notduncansmith has quit [Read error: Connection reset by peer]
j-pb has quit [Remote host closed the connection]
inconshreveable has joined #ipfs
inconshreveable has quit [Remote host closed the connection]
<jbenet>
whyrusleeping: not yet. on running outside of docker, i no longer thing that's a good idea-- we run many procs, including nginx and docker so far has done really. is the only problem attaching?
<jbenet>
(actually, _many_ = 2, but it's still easier to manage than other things)
<jbenet>
but on the ipfs daemon instead of other binary-- yeah let's do that
lgierth_ has quit [Quit: Ex-Chat]
pfraze_ has joined #ipfs
neoteo has joined #ipfs
neoteo has quit [Ping timeout: 246 seconds]
<jbenet>
ipfs.io/blog or blog.ipfs.io
<jbenet>
?
Tv` has joined #ipfs
<krl>
we should concider supporting hardlinks or similar somewhow
<krl>
just showed ipfs to my sysadmin friend
<krl>
the argument that files change, only apply locally, since if you're asking another peer, they will tell you that you gave them bad data
<krl>
and this could trigger a re-hash of the changed file
orzo has joined #ipfs
mondkalbantrieb has joined #ipfs
<krl>
or you could have a hashing reader for your blocks, if you don't trust yourself not changing files
<krl>
you would need custom hardlinks though, since i don't think you can set offsets on unix hardlinks in general, anyone know!
<orzo>
is it okay to ask really basic questions here
<krl>
orzo: yes, ask away!
<krl>
there's plenty of people willing to help if they have time
<orzo>
wll i just installed ipfs, and i tried, without adding, "ipfs cat QmZ4tDuvesekSs4qM5ZBKpXiZGun7S2CYtEZRB3DYXkjGx" which based on the link jbenet just posted, i though *might* output "hello worlds"
<orzo>
it reported "Error: blockstore: block not found" instead
<orzo>
i'm thinking i'll be abl to incorporat ipfs heavily into that projct
<jbenet>
orzo: very cool-- stick around and lets collaborate wherever we can
<orzo>
we hav our own notion of peer identity that is similarly a hash of a public key, but it's actually a gpg public key and we have tools to completely integrate all a peer's crypto keys into a gpg identity
<krl>
orzo: looks interesting! i did not know debian was using gpg in this way, cool
<orzo>
well our code is ont top of debian, not integrated
crossdiver has joined #ipfs
<orzo>
incidentally, we have a whole dns system that will automatically resolve identities similar the way you suggested in one of the talks posted at ipfs.io
<orzo>
it's a social network where your friend links become resolvable vpn peers
<orzo>
i've also recently wrote code for peers to find each other through bittorrent's dht network
<orzo>
but it looks like i should scrap that and probably use yours instead
tilgovi has joined #ipfs
ei-slackbot-ipfs has quit [Remote host closed the connection]
ei-slackbot-ipfs has joined #ipfs
<whyrusleeping>
Tv` can you make our spring meeting tomorrow at noon?
<whyrusleeping>
sprint*
<Tv`>
whyrusleeping: this is the first i hear of it, really.. Hangout?
<whyrusleeping>
yeah, it's a hangout
<Tv`>
then yes
<whyrusleeping>
cool, 12:15 tomorrow
<Tv`>
assuming that time is US/Pacific?
<whyrusleeping>
yeah, Lol
<whyrusleeping>
19:00utc
<Tv`>
wilco
<Tv`>
latest fun discovery in sharness:
<Tv`>
sed -i="" "s/\(.*\)/\1 indirect/g" refsout &&
<Tv`>
that writes to a file named refsout=
<whyrusleeping>
yay sed!
<Tv`>
it gets better
<whyrusleeping>
orzo, yes use our dht :)
<Tv`>
i don't see a way to call gnu and osx seds both the right way, without giving an extension ;)
<whyrusleeping>
oh they're different? awesome.
<whyrusleeping>
can you find a sed written in go?
<Tv`>
what a horrible idea
<Tv`>
if you aren't bug-for-bug compatible, why call it sed; if you are, uhh you realize that's incredibly difficult to get right
EricJ2190 has quit [Ping timeout: 255 seconds]
<Tv`>
oh wow it seems there's no common sed -i syntax *even with the extension*
<Tv`>
linux: -i[SUFFIX], --in-place[=SUFFIX]
<Tv`>
osx: -i extension
<Tv`>
gnu sed parses "-i foo" as if foo was a sed command, not a suffix to -i
<Tv`>
oh osx sed does handle -ibak
<Tv`>
my savior
<orzo>
osx has handicapped bsd versions of unix tools
<orzo>
install gnu user tools
<orzo>
heh
<Tv`>
so now i just have indirect pins getting lost under sharness, but not when i'm just manually poking things
<Tv`>
combined with sharness failing due to what looks like umount races and what not
pfraze_ has quit [Remote host closed the connection]
Wallacoloo has joined #ipfs
patcon has joined #ipfs
<jbenet>
hey Tv` o/
<Tv`>
i jumped the gun and wrote a spike
shazow has left #ipfs [#ipfs]
<Tv`>
it seems to work, but some details in sharness are ruining my day
<Tv`>
e.g. some of the tests assume they'll know exactly what objects gc removes, but now pinning creates objects on the fly
<Tv`>
and i can't get good feelgood runs of all of sharness because it keeps crapping out on unmount of busy dirs etc that sound like races
<jbenet>
Tv` ahh nasty
<jbenet>
Tv` we can re-write those.
<jbenet>
to only check for objects existing or being removed,
<jbenet>
but not the whole thing
<Tv`>
yeah, what's bothering me is the process of trying to discover whether that's an overly picky test or a bug
<Tv`>
the branch looks like a bunch of fixes to pinning-related topics, and *then* swap out the implementation
<Tv`>
and the bunch of fixes keeps accumulating
anshukla has quit [Remote host closed the connection]
<Tv`>
and now sharness is crapping out because i left debug output around :(
<Tv`>
(it should have been to stderr! why is that a problem!)
<Tv`>
oh because this thing verifies what the error message is
<Tv`>
gah fiddlybits
<jbenet>
Yeah-- cause the error message is part of the UX
<jbenet>
we've had regressions where error messages got bad and confusing and we want to avoid that
<Tv`>
sure, it's just that i've tripped on this sort of accidental complexity way too much with this one
<Tv`>
both on spec level and implementation
<Tv`>
but anyway, in manual testing it works, i feel like these aren't actual bugs that sharness keeps crashing on
<Tv`>
i just need to clean up all of the things, to make it see that
anshukla has joined #ipfs
<Tv`>
uhh i just saw t0045-ls.sh fail with "not ok 13 - 'ipfs add' output looks good", actual_add was empty
<Tv`>
and then i re-run just it, and it worked
<Tv`>
that sort of stuff
<jbenet>
/:| that seems fishy-- add tests have been very reliable (there's one or two that flake in `ipfs get`, but we try to mark them broken if we find them.)
<Tv`>
well the unmount part craps out *all* the time for me
<Tv`>
so this is obviously environment dependent
inconshreveable has joined #ipfs
Wallacoloo has quit [Ping timeout: 265 seconds]
<jbenet>
cc whyrusleeping o/ re mount
<jbenet>
it craps our for me a lot too :( (though not in tests)
<jbenet>
are you in osx?
<Tv`>
linux
neoteo has joined #ipfs
nessence has quit [Read error: Connection reset by peer]
nessence has joined #ipfs
neoteo has quit [Ping timeout: 245 seconds]
<Tv`>
okay finally got a full successful run (sleep 2 before unmount helped ;)
<Tv`>
i had to temporarily disable a bunch of output comparisons, mostly gc output but one place where it looks at output of ipfs pin ls --type=indirect
<Tv`>
i don't understand why that would show up different
<Tv`>
but this is looking good already; will clean up and push a branch for review tomorrow
<whyrusleeping>
Tv`: youre the one who would know why mount stuff is weird
<whyrusleeping>
i just read the docs
Wallacoloo has joined #ipfs
anshukla has quit [Remote host closed the connection]
domanic has joined #ipfs
inconshreveable has quit [Remote host closed the connection]
Wallacoloo has quit [Read error: Connection reset by peer]
<Tv`>
whyrusleeping: yeah will happily look into it, but trying to get things done first
Wallacoloo has joined #ipfs
<whyrusleeping>
no worries
Wallacoloo has quit [Read error: Connection reset by peer]
Wallacoloo has joined #ipfs
Wallacoloo has quit [Read error: Connection reset by peer]
inconshreveable has joined #ipfs
Wallacoloo has joined #ipfs
pfraze has joined #ipfs
Wallacoloo has quit [Read error: Connection reset by peer]
Wallacoloo has joined #ipfs
tilgovi has quit [Ping timeout: 272 seconds]
tilgovi has joined #ipfs
Wallacoloo has quit [Read error: Connection reset by peer]
Wallacoloo has joined #ipfs
inconshr_ has joined #ipfs
inconshreveable has quit [Ping timeout: 265 seconds]
domanic has quit [Ping timeout: 264 seconds]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
tilgovi has quit [Ping timeout: 256 seconds]
anshukla has joined #ipfs
Wallacoloo has quit [Read error: Connection reset by peer]
Wallacoloo has joined #ipfs
<whyrusleeping>
weird... does 'ipfs get' work for anyone else?
Wallacoloo has quit [Read error: Connection reset by peer]
Wallacoloo has joined #ipfs
tilgovi has joined #ipfs
inconshr_ has quit [Read error: Connection reset by peer]
inconshreveable has joined #ipfs
neoteo has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
neoteo has quit [Ping timeout: 265 seconds]
anshukla has quit [Quit: Leaving...]
<whyrusleeping>
looks like add is slow because it allocates memory like its going out of style
inconshreveable has quit [Read error: Connection reset by peer]
<whyrusleeping>
i'm seeing it go from 200->400MB every ten or so seconds
<whyrusleeping>
before a GC triggers
sharky has quit [Ping timeout: 256 seconds]
Wallacoloo has quit [Read error: Connection reset by peer]
<whyrusleeping>
i suspect eventlogs
Wallacoloo has joined #ipfs
ei-slackbot-ipfs has quit [Remote host closed the connection]
ei-slackbot-ipfs has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
sharky has joined #ipfs
catsup has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
inconshreveable has joined #ipfs
domanic has joined #ipfs
anshukla has joined #ipfs
<whyrusleeping>
Tv`: your work is much anticipated
<whyrusleeping>
add is heavily bottlenecked by pinning
<Tv`>
Oh wow, did not know
<whyrusleeping>
yeah, just got a couple profiles down
<whyrusleeping>
99.9% of time is being spent in storeIndirPin
<Tv`>
This iteration is definitely not meant to speed up, but to make it more flexible
<whyrusleeping>
it will likely be faster than my shitty code
<whyrusleeping>
80% of that time is base58 encoding
<whyrusleeping>
GENIUS
notduncansmith has joined #ipfs
www1 has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
www has quit [Ping timeout: 244 seconds]
frankenmint has joined #ipfs
Wallacoloo has quit [Read error: Connection reset by peer]
patcon has quit [Ping timeout: 246 seconds]
pfraze has quit [Remote host closed the connection]
Wallacoloo has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
tilgovi has quit [Remote host closed the connection]
notduncansmith has joined #ipfs
Wallacoloo has quit [Read error: Connection reset by peer]
notduncansmith has quit [Read error: Connection reset by peer]
Wallacoloo has joined #ipfs
mildred has joined #ipfs
nessence has quit [Read error: Connection reset by peer]
nessence has joined #ipfs
neoteo has joined #ipfs
Wallacoloo has quit [Read error: Connection reset by peer]
Wallacoloo has joined #ipfs
neoteo has quit [Ping timeout: 256 seconds]
frankenmint has quit [Remote host closed the connection]
Wallacoloo has quit [Read error: Connection reset by peer]
Wallacoloo has joined #ipfs
inconshreveable has quit [Ping timeout: 256 seconds]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
frankenm_ has joined #ipfs
zen|merge has joined #ipfs
zen|merge has joined #ipfs
frankenm_ is now known as frankenmint
Wallacoloo has quit [Read error: Connection reset by peer]
Wallacoloo has joined #ipfs
domanic has quit [Ping timeout: 252 seconds]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
atomotic has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
neoteo has joined #ipfs
Wallacoloo has quit [Read error: Connection reset by peer]
Wallacoloo has joined #ipfs
octalberry has joined #ipfs
Wallacoloo has left #ipfs [#ipfs]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
torpor has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
m0ns00n has quit [Remote host closed the connection]
<jbenet>
whyrusleeping: the pinning being held up in base58 is not likely to be what's holding up #1216, cause his cpu usage is very low. There's io or scheduling problems. Something's rate limiting
inconshreveable has quit [Ping timeout: 256 seconds]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
chrisr_ has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<krl>
jbenet: cannot make it to the sprint today :(
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
marklock has joined #ipfs
pfraze has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
therealplato has quit [Ping timeout: 264 seconds]
pfraze has quit [Remote host closed the connection]
therealplato has joined #ipfs
Tv` has joined #ipfs
marklock has quit [Ping timeout: 246 seconds]
lgierth has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
mildred has quit [Ping timeout: 272 seconds]
tilgovi has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
chrisr_ has quit [Ping timeout: 252 seconds]
pfraze_ has joined #ipfs
pfraze_ has quit [Remote host closed the connection]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
patcon has joined #ipfs
MrKeon has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<bitemyapp>
jbenet: what's the current story on a spec/
j-pb has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<whyrusleeping>
jbenet: re: pinning, i know, but its what was holding me up while trying to get a repro on 1216
<jbenet>
whyrusleeping: do you want to make the next sprint issue? or want me to do it? (i dont care either way)
<Tv`>
some ipfs add -r timings, old vs new, two trivial benchmarks: few large files, lots of small files
<Tv`>
times are master -> wip-pin
<whyrusleeping>
ill do it
<whyrusleeping>
can i change our notation again?
<whyrusleeping>
im thinking roman numerals this time
<Tv`>
29s -> 30s, 1m19s -> 1m31s
<whyrusleeping>
or maybe greek letters
<Tv`>
cpu usage for the small file case switched from userspace to system, which sounds like the base58 load disappeared and it's hitting the next bottleneck
<jbenet>
whyrusleeping only if you change all of them.
<jbenet>
Tv` ah that's interesting. yeah the base58 thing will also disappear when event logs are removed
<Tv`>
tl;dr new pinning code ain't worse
<jbenet>
Tv` add is just nasty right now.
<Tv`>
so who wants to talk to me about the migration for it ;)
<Tv`>
i'm still semi-clueless about migrations and sort of happy with that....
<jbenet>
I think either whyrusleeping or I can write it
<jbenet>
we wrote the last ones
<whyrusleeping>
yeah, migrations are pretty easy now that we have the framework for them in place
<Tv`>
the keys for that look like /local/pins/{direct,recursive,indirect}/keys (parents of the above), contain json, and can be fed to the new pinning mechanism to generate the new state
<whyrusleeping>
jbenet: posted the sprint issue, edit it as you like, im not sure how to best format these
<Tv`>
does that make sense?
<Tv`>
the direct and recursive json are arrays, the indirect json is a dict {key: refcount}
<Tv`>
i'll go poke around manually in the generated pin objects to make sure they look sane, after that looks good i'll create a pull request
<jbenet>
yeah that makes sense
<jbenet>
whyrusleeping: ok
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<jbenet>
whyrusleeping: the issue numbers should be actual links
<jbenet>
(i'll do it)
lgierth has joined #ipfs
nessence has quit [Read error: Connection reset by peer]
<Tv`>
btw in case this is not a well known trick: ./ipfs object data QmfJFMonGMzqjJWGG2sx3GDmzN4Vf8NGhq8tjugpCk9Xt4/direct|dd skip=1 bs=1|protoc -I$GOPATH/src --decode=ipfs.pin.Set $GOPATH/src/github.com/ipfs/go-ipfs/pin/internal/pb/header.proto
<Tv`>
decode stdin as a protobuf object
<Tv`>
very useful
<jbenet>
whyrusleeping: links-- numbers arent enough across repos
<Tv`>
(skip 1 byte as it's uvarint length prefixed, and might contain other data at the end; this one doesn't)
<jbenet>
Tv` that's great
tilgovi has quit [Quit: No Ping reply in 180 seconds.]
torpor has quit [Quit: Leaving.]
notduncansmith has joined #ipfs
mildred has quit [Ping timeout: 252 seconds]
notduncansmith has quit [Read error: Connection reset by peer]
tilgovi has joined #ipfs
<Tv`>
alright confirmed, pinning seems to encode correctly into ipfs objects
<rschulman>
And yes, they would be downloaded in a distributed fashion, if multiple people had the blocks involved.
<rschulman>
Lots of other groups are already hosting things on ipfs
<rschulman>
eg: I believe npm is
<orzo>
multiple people would have the blocks if there are multiple downloaders, right? nothing particular the end user needs to do to have them cache blocks is there?
neoteo has quit [Quit: Be back later ...]
Wallacoloo has joined #ipfs
bengo has quit [Quit: Computer has gone to sleep.]
jbenet_ has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
bengo has joined #ipfs
jbenet_ has quit [Ping timeout: 276 seconds]
therealplato1 has joined #ipfs
bengo has quit [Quit: Computer has gone to sleep.]
therealplato has quit [Ping timeout: 245 seconds]
bengo has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
bengo has quit [Quit: Computer has gone to sleep.]
bengo has joined #ipfs
bengo has quit [Client Quit]
bengo has joined #ipfs
bengo has quit [Client Quit]
bengo has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
therealplato1 has quit [Ping timeout: 264 seconds]
therealplato has joined #ipfs
EricJ2190 has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
therealplato has quit [Ping timeout: 272 seconds]
therealplato has joined #ipfs
patcon has joined #ipfs
nessence has joined #ipfs
lgierth_ has joined #ipfs
therealplato has quit [Ping timeout: 272 seconds]
lgierth has quit [Ping timeout: 245 seconds]
therealplato has joined #ipfs
cow_2001 has quit [Quit: ASCII Muhammad - @o<-<]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
jbenet_ has joined #ipfs
pfraze_ has joined #ipfs
jbenet_ has quit [Ping timeout: 272 seconds]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
neoteo has joined #ipfs
lgierth_ has quit [Read error: Connection reset by peer]
lgierth_ has joined #ipfs
neoteo has quit [Ping timeout: 240 seconds]
therealplato has quit [Quit: Leaving.]
therealplato has joined #ipfs
tilgovi has quit [Quit: No Ping reply in 180 seconds.]
tilgovi has joined #ipfs
headbite has joined #ipfs
bengo has quit [Quit: Computer has gone to sleep.]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<whyrusleeping>
jbenet: i can try mars on that, youre not moving all the gateways to that are you?
<whyrusleeping>
orzo: that is correct
<whyrusleeping>
simply downloading the file makes you a rehoster until a gc runs
bengo has joined #ipfs
<whyrusleeping>
or you 'pin' the content to be a host of it permanently
<whyrusleeping>
cryptix: krl wking, any of you around?
<wking>
sortof, and only for five minutes or so ;)
fiatjaf has left #ipfs ["undefined"]
<orzo>
why is there two implementations of ipfs? It seems young for that
<orzo>
was there friction over language choice?
<whyrusleeping>
orzo: there is but a single implementation of ipfs
<orzo>
oh
<whyrusleeping>
other languages have 'started' with no real progress yet
<whyrusleeping>
we really would like to have a javascript implementation for running ipfs in the browser
<orzo>
i was under the impression that nodejs was the other
chriscool has quit [Ping timeout: 256 seconds]
<orzo>
i do a lot of coding in haskell
<orzo>
it can be compiled to js nowadays
<whyrusleeping>
it has a few small libraries, but other than that, nothing on the nodejs front works
<whyrusleeping>
orzo: someone had the intention of writing a haskell impl at one point
<whyrusleeping>
i cant imagine that being fun to write, lol
<whyrusleeping>
then again, ive never implemented anything beyond algorithms in haskell
<orzo>
the bittorrent dht code i spoke of was haskell