<domanic>
(was collecting stuff about crypto in sbot wiki, but i'll send you a pr)
EricJ2190 has quit [Ping timeout: 244 seconds]
nham has joined #ipfs
inconshr_ has joined #ipfs
inconshreveable has quit [Ping timeout: 256 seconds]
<whyrusleeping>
domanic: good stuff :)
therealplato has quit [Read error: Connection reset by peer]
therealplato has joined #ipfs
<ipfsbot>
[go-ipfs] dominictarr opened pull request #1198: make comments match the code in secure channel implementation (master...master) http://git.io/vJHnN
Daboloskov has joined #ipfs
Daboloskov has quit [Client Quit]
Daboloskov has joined #ipfs
Daboloskov has quit [Client Quit]
<domanic>
whyrusleeping, hey can you help me understand the go module system?
<whyrusleeping>
haha, was it an anonymous function?
Wallacoloo has joined #ipfs
<ipfsbot>
[go-ipfs] whyrusleeping created feat/bsrefactor (+1 new commit): http://git.io/vJHll
<ipfsbot>
go-ipfs/feat/bsrefactor 8bb660b Jeromy Johnson: mild refactor of bitswap
<ipfsbot>
[go-ipfs] whyrusleeping created fix/blocksize-calc (+1 new commit): http://git.io/vJHlV
<ipfsbot>
go-ipfs/fix/blocksize-calc 2a06433 Jeromy Johnson: correct the blocksize calculation for link blocks
<ipfsbot>
[go-ipfs] whyrusleeping opened pull request #1199: mild refactor of bitswap (master...feat/bsrefactor) http://git.io/vJHlH
<ipfsbot>
[go-ipfs] whyrusleeping opened pull request #1200: correct the blocksize calculation for link blocks (master...fix/blocksize-calc) http://git.io/vJHld
pfraze has quit [Remote host closed the connection]
tilgovi has quit [Ping timeout: 240 seconds]
tilgovi has joined #ipfs
gatesvp has joined #ipfs
<gatesvp>
so two weeks ago I saw substack on this channel, now Domanic Tarr is on the channel... I feel like I'm at NodeConf again :)
<gatesvp>
in a good way of course :)
doublec has joined #ipfs
sharky has quit [Ping timeout: 246 seconds]
sharky has joined #ipfs
Daboloskov has joined #ipfs
<jbenet>
domanic: secio is going to be rewritten and possibly just trashed. it's broken.
<domanic>
jbenet, can you link me to discussion?
<domanic>
whats wrong with it?
<jbenet>
we do our own hmac and im pretty sure it's got padding problems.
<jbenet>
that, and there's known plaintexts: the protocol is predictable not not pre-compressed/shuffled.
<whyrusleeping>
jbenet: im thinking of how to fix the bitswap issue
<whyrusleeping>
and ive got a couple ideas, but nothing super solid
<domanic>
right, but doesn't that come down to choosing a reasonable cipher?
<domanic>
the key stretching is a bit weird
<jbenet>
whyrusleeping what are the ideas?
<whyrusleeping>
so, the first idea is to just have multiple client workers and set a short timeout on the wantlist updates
<jbenet>
domanic: no, it doesn't -- no matter the cipher, if you've got known plaintexts OR padding problems you can leak out information and recover a plaintext. once you do from then on the cipher is useless.
<whyrusleeping>
the other is to keep a heap thing of peers in bitswap and keep them sorted by last contact
<whyrusleeping>
and only send wantlist updates to peers that meet a certain last contact minimum
<whyrusleeping>
which gets a little weird
<domanic>
jbenet, so I notice that it starts out by sending the nonce (which is plaintext)
<domanic>
but surely if you have a decent cipher they can't predict bytes from just seeing a part of the key
<whyrusleeping>
jbenet: also, #1200, RFCR
<whyrusleeping>
and 1199
<domanic>
like if you generated the key stream by doing N hash(N
<domanic>
oops
<domanic>
N = hash(N-1 + key) //with a good hash that didn't have length extention
<jbenet>
whyrusleeping: neither of those sgtm-- bitswap should just not be help up by messages it sends. if it sends out a message and the network layer delays or conn dies, all that should happen for bitswap is that connection is gc-ed until it appears again. (ledger + wantlist remains intact. not being a live connection should prevent messages from being sent to it
<jbenet>
for the while)
<domanic>
(but i figure that would be slow compared to a proper cipher)
<whyrusleeping>
jbenet: okay, then we run into goroutine explosion again
<whyrusleeping>
where we have to have one goroutine per message we send
<jbenet>
i have to re-page all of this in, but i think the nonce being plaintext is fine-- many protocols actually have deterministic nonces.
<jbenet>
whyrusleeping: or queues. like-- what could happen is that a bitswap worker tries to send to the peer, but if the outgoing queue is full it moves onto another
<jbenet>
(can use a `chan struct{}{}` to signal how many messages are waiting to be sent out, like a semaphore?)
<jbenet>
not sure.
<jbenet>
this may be more complicated than it's worth
<jbenet>
but i think there probably is a way of keeping the bitswap workers making forward progress on unclogged peers, and yeah maybe a goroutine waiting per hung peer.
<whyrusleeping>
jbenet: queues per peer?
<whyrusleeping>
that could potentially work
<whyrusleeping>
but at what point is it okay to drop a message to a peer?
ei-slackbot-ipfs has quit [Remote host closed the connection]
ei-slackbot-ipfs has joined #ipfs
<domanic>
jbenet, right but what are the padding / plaintext problems? tbh I have only studied the handshake/setup so far, but not the content
<jbenet>
depends on where this rate limiting is being done-- if on bitswap side, the worker could "take next work item from heap, try sema down, send. if sema down fails, put work back on heap (assuming we have a way of selecting the others or whatever). and on disconnection, we clear the heap of outgoing crap.
<jbenet>
domanic: maybe take a look at all the padding attacks on TLS. we may be ok, but i doubt it
<whyrusleeping>
alright, i'll try out queues, i just dread having yet another goroutine per peer
<domanic>
jbenet, use salsa20 because it doesn't need padding
<domanic>
jbenet, so far it looks like tls has padding attack vulns when it is degraded to legacy model
<domanic>
modes
Tv` has quit [Quit: Connection closed for inactivity]
<jbenet>
in part removing non AEAD ciphers, static key exchange, and adding 1-RTT mode
<domanic>
jbenet, I think the key is good primitives - just like unit testing
<domanic>
make sure the individual parts don't have bugs, and then you don't have to check their interactions
<domanic>
all the TLA crypto standards have bugs, or have to be used just right (SHA, RSA, AES)
therealplato has quit [Read error: Connection reset by peer]
<domanic>
jbenet, how do ipfs peers find each other & each other's keys?
therealplato has joined #ipfs
<jbenet>
they _are_ the keys. if you want to have other id, use a key signing hierarchy
<domanic>
okay so my question should be how do they find their ip addresses?
<jbenet>
whyrusleeping: more thoughts on bitswap?
<domanic>
relocating
<whyrusleeping>
jbenet: i'm just gonna go with the queue thing
<whyrusleeping>
it will probably work
<jbenet>
whyrusleeping: maybe let's sketch out the change before sinking into it? what would it be?
<jbenet>
domanic: the routing system
domanic has quit [Ping timeout: 244 seconds]
mildred has joined #ipfs
www has quit [Quit: Leaving.]
www1 has joined #ipfs
<whyrusleeping>
jbenet: a 'peer manager' type object that manages goroutines for each bitswap partner
<whyrusleeping>
bitswap would send messages to it, and it would put the message in each peers outbox
<whyrusleeping>
if multiple wantlist type messages collect in the outbox they will be combined
<whyrusleeping>
it will receive the notifications for peer connected and peer disconnected from the net
<whyrusleeping>
and close out/start up goroutines per peer that way
<jbenet>
(scanning the outbox for consistency is also good-- receiving a cancel should remove things to be sent)
<whyrusleeping>
yeap! an outgoing cancel counts as a wantlist message
<whyrusleeping>
which would be 'combined' by removing that key if present
tilgovi has quit [Remote host closed the connection]
nemik has quit [Ping timeout: 245 seconds]
nemik has joined #ipfs
Kuba has quit [Ping timeout: 255 seconds]
Kuba has joined #ipfs
Kuba has quit [Ping timeout: 255 seconds]
Kuba has joined #ipfs
RzR has quit [Excess Flood]
RzR has joined #ipfs
Daboloskov has joined #ipfs
<whyrusleeping>
jbenet: really weird... the sharness tests didnt fail locally
<whyrusleeping>
and i have no idea why they would have failed at all
<whyrusleeping>
we dont test with files greater than 8MB
<whyrusleeping>
(in sharness)
<whyrusleeping>
i mean, the EXPENSIVE tests do, but nothing that travis does
<cryptix>
gmooorning
<whyrusleeping>
cryptix: good wednesday to you!
<whyrusleeping>
as it is very recently wednesday...
<cryptix>
haha i guess
<whyrusleeping>
i think my neighbors hate me
<whyrusleeping>
i havent turned my music down since 9
<whyrusleeping>
there are apparently really big javascript meetups in seattle
<whyrusleeping>
held at google freemont
<jbenet>
i've been there!
<ipfsbot>
[go-ipfs] jbenet closed pull request #1198: make comments match the code in secure channel implementation (master...master) http://git.io/vJHnN
atomotic has joined #ipfs
<whyrusleeping>
jbenet: is it cool?
<whyrusleeping>
its really close to my place
mildred has quit [Ping timeout: 272 seconds]
<jbenet>
yeah it was nice
<whyrusleeping>
they probably wont let me talk about ipfs...
<whyrusleeping>
unless we have cool javascript things
awrelll has joined #ipfs
mildred has joined #ipfs
chriscool has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<whyrusleeping>
wtf fuse
<whyrusleeping>
maybe its not fuse?
<whyrusleeping>
but the trash directories from sharness are getting created as owner root
<whyrusleeping>
group tooy
<whyrusleeping>
root*
* whyrusleeping
blames linux 4.0
<whyrusleeping>
its broken! turn back!
chriscool has quit [Ping timeout: 276 seconds]
<ipfsbot>
[go-ipfs] whyrusleeping pushed 1 new commit to fix/blocksize-calc: http://git.io/vJHNo
<ipfsbot>
go-ipfs/fix/blocksize-calc 942a403 Jeromy: update hash for bigfile
<whyrusleeping>
plsplsplswork
<ipfsbot>
[go-ipfs] whyrusleeping pushed 1 new commit to feat/bsrefactor: http://git.io/vJHA7
<ipfsbot>
go-ipfs/feat/bsrefactor 7551bd7 Jeromy: address comments from CR
<ipfsbot>
[go-ipfs] whyrusleeping force-pushed feat/bsrefactor from 7551bd7 to 9049dae: http://git.io/vJQID
<ipfsbot>
go-ipfs/feat/bsrefactor 0324b4b Jeromy Johnson: mild refactor of bitswap
<ipfsbot>
go-ipfs/feat/bsrefactor 9049dae Jeromy: address comments from CR
<jbenet>
whyrusleeping: oh interesting. could have a specific type of object for that too. (bundling objects to move together will go a long way. i.e. hosts that know you'll want multiple objects together)
<whyrusleeping>
exactly!
<whyrusleeping>
on the extreme end they turn into torrent files
<whyrusleeping>
i think
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<whyrusleeping>
yeah, a torrent file contains the hashes of all the blocks in that torrent
<whyrusleeping>
resisted the urge to use scp on a 30MB log file
<whyrusleeping>
used ipfs instead
<whyrusleeping>
was pleasantly surprised to get 4Mbit download speeds on my 5Mbit internet
<awrelll>
the ipfs-bot got them gitter messages up in here
<jbenet>
awrelll: a bit confused. so-- i see (a) `ipfs-bot` fwding whyrusleeping and my comments from gitter to irc. (b) i see your account proxying stuff too (in gitter's irc server), (c) i cant tell where you typed that just now
<whyrusleeping>
if you see it in irc he typed it in irc
<jbenet>
oh i see, so `ipfs-bot` is gitter -> irc, and `awrelll` is irc -> gitter (+ awrell's comments)
<pinbot>
now pinning QmfAQfqrCxJYLucTbgk4w9vDm65mAoaiQ9Wp4zqEe1AbXC
<whyrusleeping>
check out that stack trace jbenet
<whyrusleeping>
all literally 5 million lines of
<ipfs-bot>
(jbenet) awrelll, i didnt get the "this can be linked to other accounts too" message on irc. ipfs-bot is not relaying your gitter messages to irc.
<pinbot>
failed to pin QmfAQfqrCxJYLucTbgk4w9vDm65mAoaiQ9Wp4zqEe1AbXC: pin: context deadline exceeded
<whyrusleeping>
pinbot: just deal with it okay?
<jbenet>
okay way too many notifs. im closing gitter
notduncansmith has quit [Read error: Connection reset by peer]
rschulman has quit [Quit: Leaving.]
jabroney has joined #ipfs
jabroney has left #ipfs [#ipfs]
tilgovi has quit [Remote host closed the connection]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
inconshreveable has quit [Ping timeout: 246 seconds]
inconshreveable has joined #ipfs
m0ns00n has joined #ipfs
<m0ns00n>
Hoi!
<m0ns00n>
Good evening!
zooko has left #ipfs ["#tahoe-lafs the cryptographically-protected, decentralized storage system"]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
Daboloskov has joined #ipfs
Daboloskov has quit [Client Quit]
<wking>
jbenet, whyrusleeping: Following up with the Waffle folks on my "Only collaborators can update cards" issue, and one of their shot-in-the-dark hypotheses was that we've blocked third-party application access but not whitelisted Waffle.
notduncansmith has quit [Read error: Connection reset by peer]
<wking>
cryptix, dPow, other go-ipfs-team members: Can you drag cards on https://waffle.io/ipfs/ipfs ? Or do you also see the "Only collaborators can update cards" warning?
pfraze_ has joined #ipfs
pfraze has quit [Ping timeout: 264 seconds]
zorun has joined #ipfs
ipfs-bot has quit [Remote host closed the connection]
rschulman has joined #ipfs
m0ns00n has quit [Quit: Leaving]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
anshukla_ has quit [Ping timeout: 256 seconds]
patcon has joined #ipfs
anshukla has joined #ipfs
anshukla has quit [Remote host closed the connection]
inconshreveable has quit [Ping timeout: 265 seconds]
compleatang has quit [Quit: Leaving.]
neoteo has quit [Quit: Be back later ...]
<substack>
is there any writeup of how the mutable publishing part of ipfs works?
inconshreveable has joined #ipfs
<substack>
dominictarr pointed out a denial of service attack against BEP44 by flooding a portion of the DHT with keys generated to be close to a target key
<substack>
so I want to see if ipfs is vulnerable to the same attack
<posix4e>
Could it be viewed as simple as writing a file or reading a file?
<whyrusleeping>
posix4e: im confused what youre asking
<posix4e>
Ipfs is a file system right. You can read and write files
<posix4e>
Sorry to be dense
<whyrusleeping>
ah, yes. ipfs is a filesystem
<whyrusleeping>
you can read and write files through the fuse interface
zooko has left #ipfs ["#tahoe-lafs the cryptographically-protected, decentralized storage system"]
<whyrusleeping>
adding files to ipfs locally is pretty fast, depends on the speed of your HDD mostlu
<whyrusleeping>
reading local files back out is mostly the same story
<posix4e>
What if I'm totally without storage
<whyrusleeping>
if a file doesnt exist locally, once you find it you can get decent transfer speeds >~9 Mbps depending on your transfer speed
<whyrusleeping>
thats going to increase really soon
<posix4e>
Why is that
<whyrusleeping>
im working on some code to make transfers faster
notduncansmith has joined #ipfs
<posix4e>
So no quorum reads or writes
notduncansmith has quit [Read error: Connection reset by peer]
<whyrusleeping>
you dont write data to other nodes
<whyrusleeping>
data is hosted only by nodes who choose to host it
<whyrusleeping>
similar to bittorrent in that aspect
<okket_>
"pull not push"
<whyrusleeping>
^
<posix4e>
Hmm let me ponder that zen riddle
<vaelys>
like rope.
<posix4e>
So
<posix4e>
Say I'm a new user
<vaelys>
welcome to the internet, it is great.
<posix4e>
And my Buddy says here's a file uri (no idea if this exists)
<posix4e>
So I grab it over a fuse Mount
<posix4e>
And then it is wrong so he updates it
<posix4e>
Reverses the entire file
<posix4e>
How long before I can download the new content
<ipfsbot>
[webui] krl opened pull request #45: make full path default in object view (fixes #42) (master...ipns-raw) http://git.io/vJbmV
<posix4e>
And what's the looking cost
<posix4e>
Finding cost rather
<posix4e>
Sorry in more familiar with distributed filesystem's like hdfs or ceph so perhaps my question makes no sense
<_fil_>
in ipns you mean…
<vaelys>
well, the new file would hash differently, so you'd get a new object.
<posix4e>
I'm assuming one giant federated namespace
<posix4e>
Do you hash the metadata as well? Or content addressing
kyledrake has joined #ipfs
<kyledrake>
Yo.
<substack>
ahoy
<okket_>
best case: just a query from your node to your friends (if you are already connected), worst case: endless failing lookups on the DHT because your friends internets connection decided it's time for pause
<kyledrake>
substack hi! glad you survived your week w/captain ron
<whyrusleeping>
kyledrake: hey!
<whyrusleeping>
posix4e: if he updates the file, its now a different file
<substack>
kyledrake: captain ron?
<whyrusleeping>
to find the update, it takes a dht lookup and a request
<kyledrake>
substack dominic
<substack>
ah
<kyledrake>
whyrusleeping hihi
<kyledrake>
Starting on Neocities implementation of IPFS within a few days. Took a few days to implement our new stats UI and the education site. Figured it would give a few days for the new DB and JSON lava to cool a bit
<whyrusleeping>
kyledrake: you should help me pressure jbenet into writing the dht record spec
<kyledrake>
Anyone know if the ipfs add -r stuff JSON problem got plugged?
<whyrusleeping>
kyledrake: either cryptix or tperson was working on it
rschulman has quit [Quit: Leaving.]
<kyledrake>
jbenet all the cool kids are doing it, while simultaneously snowboarding, smoking and listening to blink 182
<substack>
kyledrake: rad~!
zooko has joined #ipfs
* zooko
is looking for jbenet
<whyrusleeping>
zooko: hes more absent than usual due to traveling
<whyrusleeping>
kyledrake: tperson has a working solution, but its ugly so we havent merged it
grncdr has quit [Quit: Leaving.]
<kyledrake>
substack starting with just ipfs hashes linked via our SQL DB, but eventually private keys w/ipns for all the sites, so they can publish w/o us.
<substack>
that sounds fantastic
<kyledrake>
whyrusleeping ah, ok.
<zooko>
whyrusleeping: thanks.
<kyledrake>
substack the end of centralization... begins.
<whyrusleeping>
kyledrake: i'm really excited for neocities on ipfs
<kyledrake>
It's totally the same problem as bitcoin web wallets. I control the private key, or I encrypt it in a way that I can't read it, or people use standalone apps
<substack>
ah I was just about to ask about your plans for key storage
<kyledrake>
whyrusleeping me too. :)
<whyrusleeping>
also, just an fyi, if you notice large transfers being obnoxiously slow or "hanging", im fixing that by this weekend
<kyledrake>
substack: Probably the "bitcoin bank" model for starts. Newbs will undoubtedly forget their passwords.
* substack
has been doing more crypto research for keyboot v2
rschulman has joined #ipfs
<kyledrake>
whyrusleeping thanks. Is that a DB store change or does it not affect the stored data
Tv` has joined #ipfs
<substack>
kyledrake: it's less of a problem when there's not money on the line
<kyledrake>
substack: yeah, my thoughts exactly.
<substack>
it actually makes much less sense for a bitcoin bank to do that than neocities
<kyledrake>
The DNS link gives you that "central override" incase someone steals the private key for, say, the New York Times or something.
<kyledrake>
substack I haven't put a single thought into how to store open private keys safely in our DB though. It's a little beyond me what best practices there are.
inconshreveable has quit [Remote host closed the connection]
notduncansmith has joined #ipfs
atrapado has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
nemik has quit [Read error: No route to host]
<kyledrake>
substack Makes me wonder if we won't get into multisig territory at some point, or mmenonic strings that turn into the private key. A lot of bitcoin wallets do that now (with HD wallets)
nemik has joined #ipfs
<kyledrake>
The hacky version is to just use a Shamir's shared secret
<substack>
kyledrake: I think a good balance would be to offer 3 choices for key storage: 1) offer to store the keys on your end, unencrypted (maximally recoverable) 2) offer to store the keys on your end, encrypted (password not recoverable, only data) 3) store keys in indexeddb, ideally on a different domain that can't be upgraded easily (maximum security)
<substack>
option 3 might also involve inputting a public key directly
<kyledrake>
Yeah, that makes sense.
<jbenet>
Heyo o/ working on records spec atm. :)
<kyledrake>
heyo!
<substack>
I also don't think option 3 needs to necessarily be terribly advanced for end users, there are a lot of tricks to make this easier and transparent
<substack>
it's just mad science territory at the moment
<jbenet>
Without having read the whole thread-- substack: also want the option to delegate to key agents. Some people don't trust apps with their keys, and that's a good thing
<whyrusleeping>
kyledrake: that wont affect already stored data,its purely something in the transport
<kyledrake>
whyrusleeping cool.
<whyrusleeping>
although a change pushed last night affects transfer rate of individual files larger than 8MB
<substack>
jbenet: yes exactly, I'm building a keyring system right now
<whyrusleeping>
but not hugely
<substack>
jbenet: another approach for content-addressable data would be to automatically generate a new key for each html payload hash
<jbenet>
substack: cool! Would be useful to make a thread with lots of notes of how others have made keyrings and what we can learn.
<substack>
and then the user can choose at any time to link those generated hashes with their known public identity
<substack>
but it would be asynchronous, so linking (and unlinking) can happen at any time
<substack>
this is a generalization for asymmetric keys for how single sign on with centralized providers already works in the more advanced use cases
<substack>
sign in with {twitter,google,facebook,password}, identities may have one or more of these identity schemes
<substack>
and schemes can be discarded or split at a later date too
<substack>
jbenet: one part of this that I was asking about yesterday is related to multihash, but for signatures
<kyledrake>
As for web-of-trust side, I was hoping the keybase ppl could maybe implement something. Talked with them a bit about that.
<substack>
since people may have many different types of keys in use, and signatures should probably include that metadata
<kyledrake>
The takeaway from the bitcoin stuff is that you have to make key stuff super easy to use, otherwise people don't get it.
<jbenet>
What do you mean by for signatures? Just specifying the types of keys / Algos used for generating the sig?
<substack>
jbenet: yes
<substack>
which also makes this scheme more upgrade friendly
<substack>
like dominic is doing with ssb where signatures end with $HASH.blake2s.hmac
<jbenet>
Yeah that should happen though most things (like pgp) already out this info in metadata I think
notduncansmith has quit [Read error: Connection reset by peer]
<Tv`>
the thing that keeps bothering me about trying to name a hash "simply"
<Tv`>
is that blake2 etc has personalized, salted hashes
<Tv`>
which i think are a honking great idea
anshukla has joined #ipfs
<Tv`>
but now you can't just describe my hash as "blake2b"
<whyrusleeping>
blake2b+tv'skey
<Tv`>
more like bazilChunkHash
<Tv`>
at which point, it's just less clear what the value was
<kyledrake>
substack I was very impressed with bip39's approach of using something you can write down. It only works w/ECC though: https://github.com/weilu/bip39
<kyledrake>
They like koblitz and bernstein curves. The other curves are kindof mystery meat.
pfraze has quit [Read error: Connection reset by peer]
<Tv`>
well nacl is 32 bytes, so you're looking at 24 random english words
<vaelys>
p. sure djb and co's curves are also mystery meat.
pfraze has joined #ipfs
<Tv`>
i'm not sure i'd call that memorizable ;)
krl has quit [Quit: WeeChat 0.3.8]
<kyledrake>
vaelys agree strongly. I pinged Bruce Schneier on it, his advice "you have to trust the author".
<vaelys>
yes
<kyledrake>
I kindof hate ECC curves for that reason, honestly. But you know, smaller and stuff.
<kyledrake>
s/ECC curves/ECC
<vaelys>
well, I wouldn't go that far
<kyledrake>
Only RSA is going into my doom bunker. Right next to the barrel of creamed corn.
<vaelys>
delicious delicious creamed corn.
pfraze_ has joined #ipfs
pfraze has quit [Ping timeout: 240 seconds]
mildred has quit [Quit: Leaving.]
<whyrusleeping>
i'd also have a barrel of rice
<whyrusleeping>
because a barrel of rice could last me years and years and years...
notduncansmith has joined #ipfs
pfraze_ has quit [Ping timeout: 264 seconds]
notduncansmith has quit [Read error: Connection reset by peer]
zooko has left #ipfs ["#tahoe-lafs the cryptographically-protected, decentralized storage system"]
<jbenet>
at this point, i'd be hard pressed to trust anyone more than djb-- he's been fixing things left and right for years. though then again m night shama(lama)+ may be directing this movie
<kyledrake>
ha
<kyledrake>
Satoshi went with Koblitz. Your mileage may vary.
<whyrusleeping>
or even better, you dont need your old GOPATH
<jbenet>
whyrusleeping: that's much worse -- the user who's downloading your code now has to change their path
<jbenet>
whyrusleeping: gb makes this automatic, which may be ok for the most part-- but without that you've now left the user wondering why the hell it wont compile on her machine if she doesn't have the same version of the packages
* whyrusleeping
sighs
<whyrusleeping>
i thought it was cool!
<vaelys>
rip sunrise.png
<Tv`>
jbenet: but then they're gonna ask "why did it not build", and the faq can tell them to "just run xxx"
<jbenet>
Oh it is very cool, just it can hurt if not used properly
notduncansmith has quit [Read error: Connection reset by peer]
<Tv`>
then you can have faq tell people to see whether "ipfs version" says "dev" or something else, for troubleshooting
pfraze has joined #ipfs
<jbenet>
Tv` doc is not a viable solution-- it causes fits and starts and looses people. we need to start taking the UX of library adoption more seriously. make damn conversion funnels if we need to. --- in the good parts of node, "everything just works" out of the box. it's why people are very productive reusing each others code, because people go to great lengths
<jbenet>
to make sure everything is ... nice. that it takes you under 10 seconds to run an example.
<jbenet>
(and props to substack et al for making it so)
<Tv`>
*shrug*
<Tv`>
i still have that mental note to not argue vendoring with you
<whyrusleeping>
well, whatever we change, we have to get people to adopt it
<whyrusleeping>
even if we make our own package manager, people are going to be upset
<Tv`>
even more so, in that case ;)
<jbenet>
hahaha yeah, but there's worse things.
<substack>
just make sure you get people writing the best stuff to use your package manager
tilgovi has joined #ipfs
rk[1] has quit [Ping timeout: 256 seconds]
<jbenet>
i hope to convince dave that libraries need to vendor too
notduncansmith has quit [Read error: Connection reset by peer]
anshukla has quit [Remote host closed the connection]
anshukla has joined #ipfs
therealplato1 has joined #ipfs
therealplato has quit [Ping timeout: 265 seconds]
Wallacoloo has joined #ipfs
Wallacoloo has quit [Client Quit]
Wallacoloo has joined #ipfs
<whyrusleeping>
just go to localhost:5001/webui
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<jbenet>
whyrusleeping: i updated the gateways to latest and now they crash less but they get stuck in bad states more. meaning i have to reboot them more often.
therealplato has joined #ipfs
step21_ has joined #ipfs
<whyrusleeping>
what weird states?
therealplato1 has quit [Read error: Connection reset by peer]
step21 has quit [Ping timeout: 256 seconds]
step21_ is now known as step21
<jbenet>
lovely slowness
<ipfsbot>
[webui] krl opened pull request #46: fix lint test command using 'standard', also for .jsx (master...lint-fix) http://git.io/vJbhD
zen|merge_ has quit [Remote host closed the connection]
<jbenet>
hey Tv` -- is the width of the flat repo good for perf? or would it be better to use multiple directories? not sure how this performs across a variety of filesystems, but do know git uses smaller levels (2 hash chars per level)
<jbenet>
("width = char width per level")
<Tv`>
jbenet: that really depends on what the underlying fs is.. e.g. ext4 should be able to tolerate 50k entries without a second thought
<Tv`>
though i set that number *before* the commit stripping the prefixes was added, so now it's bytes 2..5 not including the /, or something
<Tv`>
with ext4 or something, the first thing to choke is ls
<jbenet>
4 byte prefix ? or 4 hex char prefix?
<Tv`>
4 byte
<jbenet>
so a max of 256^4 entries?
<Tv`>
ls ~/.ipfs/blocks => 1220fd4e etc
<Tv`>
which is 4 bytes of raw data
<Tv`>
the number 4 was chosen by observing the non-uniformity of actual ipfs keys
<jbenet>
since the hashes are normally distributed-- we'll get a lot of entries fast. oh the multihash prefix is included, so that's -- for the most part -- just 2 bytes
<jbenet>
err uniformly*
<Tv`>
at that time, that lead to about 300-way split, or something
<Tv`>
but then someone added a commit on top that mangled the keys
<jbenet>
" observing the non-uniformity of actual ipfs keys" ? you mean with the mutihash prefix? cause they should be uniformly distributed
<Tv`>
and... that probably widens it by *256
<Tv`>
yes
<jbenet>
ls ~/.ipfs/blocks | wc == 6497 for me.
<Tv`>
anyway, it's a tunable, on purpose
hellertime has quit [Quit: Leaving.]
<Tv`>
ipfs keys are "/", 2 bytes from multihash, and then uniform distribution
<Tv`>
when i picked that value 4, that lead to a little higher than 256 way split
<Tv`>
after that, a commit was added that stripped the "/" (and made assumptions i wasn't ready to make; if there ever is a key without that, it screws up; and if it's always there, what's the value of)
<Tv`>
so now the sharding data is 2 non-random bytes, 2 random bytes
<Tv`>
actually, let's see whether it shards before or after the skipping of "/" was added...
<Tv`>
after
<Tv`>
so what i said above holds, afaik
<Tv`>
so by that logic, now the tunable 4 should result in up to about 65k top-level dirs in ~/.ipfs/blocks
<Tv`>
which is fine, the dirs will be a little empty for small setups, which is mostly a waste of inodes, but not a huge concern
<Tv`>
jbenet: splitting out the mostly-uniform multihash part does nothing to the spread
<Tv`>
that just shuffles the files from ~/.ipfs/blocks/ to ~/.ipfs/blocks2/<binary version of sha256, 32>/
<jbenet>
Tv` yeah i know, but it trades one inode for dir entry size (unless they're mostly fixed size? i guess it makes sense for them to be fixed size in most platforms)
<Tv`>
i can't think of a modern filesystem that would pad directory entry names to constant size
<jbenet>
so then, saving the 4 bytes per dir entry * 65k would help? (256k, i guess not a big deal)
<Tv`>
yeah that's not gonna be interesting, the overhead of an extra dir level might cost way more
<Tv`>
i mean, at the minimum that's a couple of pointer indirections
<jbenet>
mm doubt it, if it's hot will likely stay in the buffer caches
<jbenet>
\o/ we need benchmarks.
<Tv`>
anyway, this is an optimization effort both without a workload or ground truth
* jbenet
looks at build.golang.org longingly again
<Tv`>
you're already hurting elsewhere worse, based on the quick profiles i did
inconshreveable has quit [Remote host closed the connection]
<jbenet>
yeah agreed
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
inconshreveable has joined #ipfs
<G-Ray>
pinning is not possible from the webui ATM ?
<jbenet>
G-Ray think so? not sure. cc krl tperson -- it may be fixed and merged but not yet published. i want to do a release of webui with 0.3.4 (once we fix some outstanding bugs)
<G-Ray>
I don't see any button to pin files
<G-Ray>
I pinned one hash from the CLI, it does not appear in the webui
ei-slackbot-ipfs has quit [Remote host closed the connection]
ei-slackbot-ipfs has joined #ipfs
<whyrusleeping>
jbenet: what bad states are the gateways getting into?
<jbenet>
whyrusleeping: not sure, usually the gateways are snappy for the most part. today they've been slow and had to reboot.
<whyrusleeping>
they are too damn long
<whyrusleeping>
are they still running in docker?
<jbenet>
whyrusleeping: blame evernote and their crap. once we have s3 pinning im dumping everything into ipfs
<whyrusleeping>
or can i manually invesigate?
<jbenet>
yeah they are, but you can attach to a docker container
<jbenet>
docker exec -it [container-id] /bin/bash
therealplato has quit [Ping timeout: 256 seconds]
lgierth_ has joined #ipfs
<G-Ray>
jbenet: nice ;)
kyledrake has quit [Ping timeout: 246 seconds]
<G-Ray>
jbenet: I read IPFS could be use a storage backend, like Tahoe. I assume performance would be far better than tahoe if we pin content on some node ?
lgierth_ has quit [Client Quit]
lgierth has quit [Ping timeout: 264 seconds]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<whyrusleeping>
jbenet: oh yeah, except the gateways dont run the ipfs daemon
therealplato has joined #ipfs
<whyrusleeping>
they run some gimped hack of it
<jbenet>
there's reason for it-- they do other things. and i predict we'll see more of that as we clean up the shell api and make it easier to make special purpose ipfs nodes with their own special programming / semantics
<whyrusleeping>
yeah, but i cant actually do anything with them
<whyrusleeping>
i have zero ability to debug this
<jbenet>
ah cause there's no profile?
<whyrusleeping>
theres no daemon
<whyrusleeping>
no rpc
<whyrusleeping>
no CLI
<whyrusleeping>
i cant do 'ipfs swarm peers'
<whyrusleeping>
theres also no output
<jbenet>
it should expose an api
<jbenet>
aaaand it doesn't. lovely.
<whyrusleeping>
yeah
<whyrusleeping>
it feels wrong running different code on the gateways
<whyrusleeping>
we should be running everything homogenous
<whyrusleeping>
less surface area to debug
<jbenet>
yeah that's very fair.
<jbenet>
we can switch them to the daemon
<jbenet>
we havent used the blocklist
<whyrusleeping>
and the blocklist stuff needs to be wired into the normal daemon anyways
<jbenet>
yep.
inconshreveable has quit [Remote host closed the connection]
<whyrusleeping>
do you have time to work on switching that over soonish?
<whyrusleeping>
i'd really like to get the bitswap stuff fixed
rschulman has joined #ipfs
inconshreveable has joined #ipfs
<G-Ray>
what the current status of filecoin ?
<jbenet>
yeah i can. do you still want to remove docker? am no longer having problems with it. it does have the benefit that it also runs other services (like nginx), so it's nice to have a uniform container manager
<jbenet>
(attaching works well for all cases)
patcon has quit [Ping timeout: 256 seconds]
<jbenet>
G-Ray it's coming, no ETA. we're way too busy making ipfs really good right now.
<jbenet>
(the more people help us, the faster we'll get to it)
chriscool has joined #ipfs
<whyrusleeping>
;w
<whyrusleeping>
dammit, wrong window
<jbenet>
;w looks like a weird duck.
<G-Ray>
jbenet: There is no public repo for filecoin ?
<whyrusleeping>
hahaha
<whyrusleeping>
quack!
<doublec>
Does ipfs work behind a NAT? For distributing as well as retrieving content?
<whyrusleeping>
doublec: it depends on what kind of NAT traversal your router supports
therealplato has quit [Quit: Leaving.]
<whyrusleeping>
full symmetric NAT is currently not supported
<whyrusleeping>
one way NAT traversal via upnp and NAT-PMP works
neoteo has joined #ipfs
<doublec>
whyrusleeping: if the NAT doesn't allow incoming will ipfs partially work in any way?