stebalien changed the topic of #ipfs to: Heads Up: To talk, you need to register your nick! Announcements: go-ipfs 0.4.19 and js-ipfs 0.34 are out! Get them from dist.ipfs.io and npm respectively! | Also: #libp2p #ipfs-cluster #filecoin #ipfs-dev | IPFS, the InterPlanetary FileSystem: https://github.com/ipfs/ipfs | Logs: https://view.matrix.org/room/!yhqiEdqNjyPbxtUjzm:matrix.org/ | Forums: https://discuss.ipfs.io | Code of
piti has joined #ipfs
The_8472 has joined #ipfs
piti has quit [Ping timeout: 255 seconds]
piti has joined #ipfs
piti has quit [Excess Flood]
piti has joined #ipfs
piti has quit [Excess Flood]
Ekho has quit [Quit: An alternate universe was just created where I didn't leave. But here, I left you. I'm sorry.]
piti has joined #ipfs
Ekho has joined #ipfs
mikerah has joined #ipfs
<mikerah>
Hi
<mikerah>
I work on a Javascript/Typescript ETH2.0 client called Lodestar.
<mikerah>
We are currently implementing gossipsub in js and as such, are architecting the integration of libp2p into our client.
<mikerah>
I am currently seeking to understand how libp2p was integrated in js-ipfs.
<mikerah>
Are there are any docs for this?
kakra has joined #ipfs
piti has quit [Ping timeout: 252 seconds]
hurikhan77 has quit [Ping timeout: 245 seconds]
piti has joined #ipfs
ctOS has joined #ipfs
Xenguy has quit [Ping timeout: 240 seconds]
Xenguy has joined #ipfs
Fessus has quit [Read error: Connection reset by peer]
Fessus_ has joined #ipfs
Xesxen has quit [Remote host closed the connection]
Xesxen has joined #ipfs
piti has quit [Ping timeout: 252 seconds]
nighty- has joined #ipfs
reit has joined #ipfs
lostfile has joined #ipfs
Jesin has joined #ipfs
}ls{ has quit [Quit: real life interrupt]
piti has joined #ipfs
reit has quit [Ping timeout: 252 seconds]
piti has quit [Excess Flood]
gmoro has quit [Remote host closed the connection]
piti has joined #ipfs
Belkaar has quit [Ping timeout: 252 seconds]
Belkaar has joined #ipfs
Belkaar has joined #ipfs
placer14 has quit [Quit: placer14]
placer14 has joined #ipfs
Jesin has quit [Quit: Leaving]
placer14 has quit [Client Quit]
Jesin has joined #ipfs
The_8472 has quit [Ping timeout: 252 seconds]
The_8472 has joined #ipfs
user_51 has quit [Ping timeout: 252 seconds]
stoopkid has quit [Quit: Connection closed for inactivity]
screensaver22 has quit [Remote host closed the connection]
mowcat has quit [Remote host closed the connection]
ravencrank[m] has joined #ipfs
nighty- has joined #ipfs
ddahl has joined #ipfs
Trieste has quit [Ping timeout: 245 seconds]
Trieste has joined #ipfs
nighty- has quit [Quit: Disappears in a puff of smoke]
mikerah has quit [Ping timeout: 252 seconds]
Trieste has quit [Ping timeout: 268 seconds]
ygrek has joined #ipfs
Trieste has joined #ipfs
<lordcirth__>
redfish, one block at a time. Node A sees block X on B's wantlist, and just starts sending it. When that stream ends, B hashes it, and if it matches something on the wantlist, marks it as done.
ctOS has joined #ipfs
<lordcirth__>
At least, that is my understanding
mikerah has joined #ipfs
Sine2496 has quit [Ping timeout: 244 seconds]
mandric has joined #ipfs
Guest80369 has quit [Remote host closed the connection]
MDude has quit [Ping timeout: 240 seconds]
MDead has joined #ipfs
MDead is now known as MDude
jimtron has quit [Read error: Connection reset by peer]
jimtron has joined #ipfs
matt-h has joined #ipfs
MDead has joined #ipfs
MDude has quit [Ping timeout: 268 seconds]
MDead is now known as MDude
MDead has joined #ipfs
MDude has quit [Ping timeout: 246 seconds]
MDead is now known as MDude
tangdou[m] has joined #ipfs
<AphelionZ>
Is there any issue with using the bootstrap list as a "friends" list?
<AphelionZ>
like if i have an app where specific peers with specific data are connected to repeatedly by users is there any harm in adding those peers to the bootstrap list?
<placer14>
qq: We're writing a migration which updates our CIDs...however due to the fact that migrations can run anytime in the future we would want to freeze/vendor the current hashing behavior so the migration never changes once the hash method eventually does... so the question: how do you guys protect that the migration behavior is frozen well into the future without vendoring a whole dependency tree of IPFS just for it's CID hashing?
<lordcirth__>
AphelionZ, I wouldn't replace the normal bootstrap peers, but adding should be harmless
MDead has joined #ipfs
<lordcirth__>
placer14, can't you specify the CID format explicitly?
boredmanicrobot has joined #ipfs
<placer14>
I'm not sure. I'm looking at this problem from a 10k foot view and don't have intimate knowledge of the internal interfaces to know whether I can or can't. :(
MDude has quit [Ping timeout: 250 seconds]
MDead is now known as MDude
<AphelionZ>
I believe there was a proposal to migrate to CID1
<AphelionZ>
but I'm not sure which is which. We use `dag-cbor` in the underlying OrbitDB oplog currently
<Mikaela>
Is there any use in having dhtclient peers as bootstrap nodes?
<placer14>
IIRC, I think our project adopting v1 before IPFS was ready to and did not move as quickly as we did. So now we're unifying our CIDs with IPFS's default....
mandric has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
mandric has joined #ipfs
<placer14>
It seems that the hash algo is coupled with content chunking and serialization...
renich_ has joined #ipfs
tangdou[m] has left #ipfs ["User left"]
<lordcirth__>
placer14, yes, IPFS does chunking. I think the ipfs libraries will do a lot of this stuff for you.
boredmanicrobot has quit [Quit: boredmanicrobot]
<AphelionZ>
yeah if you're already using v1 with the `zdpu` prefix I **think** you're OK but you really need to double check with the official folks
renich has quit [Ping timeout: 252 seconds]
<placer14>
It does, but we dont' want it to change our migration later when IPFS updates its behavior. The migration should always be the same.
MikeFair has quit [Read error: Connection timed out]
MikeFair has joined #ipfs
matth has joined #ipfs
dimitarvp has joined #ipfs
placer14 has quit [Quit: placer14]
matt-h has quit [Ping timeout: 252 seconds]
mowcat has quit [Remote host closed the connection]
MikeFair has quit [Read error: Connection timed out]
MikeFair has joined #ipfs
jesse22 has joined #ipfs
mikerah has joined #ipfs
mandric_ has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
mandric has joined #ipfs
mandric has quit [Client Quit]
mandric has joined #ipfs
pecastro has joined #ipfs
lidel` has joined #ipfs
placer14 has joined #ipfs
lidel has quit [Ping timeout: 245 seconds]
lidel` is now known as lidel
mandric has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
mandric has joined #ipfs
mandric has quit [Client Quit]
mandric has joined #ipfs
jesse22_ has joined #ipfs
jesse22 has quit [Ping timeout: 252 seconds]
ygrek has joined #ipfs
kapil____ has quit [Quit: Connection closed for inactivity]
jimtron has quit [Read error: Connection reset by peer]
jimtron has joined #ipfs
mandric has quit [Read error: Connection reset by peer]
mandric has joined #ipfs
ddahl has quit [Ping timeout: 255 seconds]
jimtron has quit [Read error: Connection reset by peer]
jimtron has joined #ipfs
dimitarvp has left #ipfs ["Bye"]
jimtron has quit [Remote host closed the connection]
<MikeFair>
Has anyone meaingfully discussed extending MultiHash to support an array of multiple hash algorithms?
<MikeFair>
For example, sha1 + adler32
<MikeFair>
Sha1 is 160 bits; and Adler32 is 32 bits
<MikeFair>
I don't even know how to compute the likelihood of a collision; but it's obviously super low
<MikeFair>
Or, perhaps what I'm suggesting is better asked as; has anyone discussed adding a "CheckSum" component to MultiHash Addresses such that an address is both a Hash Algo + a CheckSum algo
<lordcirth__>
MikeFair, what would be the objective?
<MikeFair>
And using a significantly smaller keyspace
<MikeFair>
The problem wiht small hash algos is higher collisions
jimtron has joined #ipfs
<MikeFair>
While I can't "prove it"; I suspect that mutiple data blocks creating a crypto hash collision would be exceedingly unlikely to also collide on a decent CheckSum algo
<MikeFair>
I'd suspect you could get down to ~64 bits
<lordcirth__>
MikeFair, having a weak hash like adler32 backing up a strong one like sha2 seems pointless
<lordcirth__>
Also, sha1 is deprecated
<MikeFair>
agreed, the point was it just being smaller
<lordcirth__>
If you want 256 bits of security, there's no way to avoid using a 256bit CID. Anything less must be weaker.
<MikeFair>
I agree, but 64-bits is enough bits to count more than the number of atoms in the known universe
<MikeFair>
So as an "Address" 256 bits is crazy wasteful
<MikeFair>
We don't _need_ 256 bits of security
<MikeFair>
But I think 256 bits is used mostly to make the "what about collisions" problem effectively moot
<lordcirth__>
MikeFair, a 64bit hash is trivial to brute force, even without a hash algo weakness.
jimtron has quit [Ping timeout: 244 seconds]
<lordcirth__>
128 is the absolute minimum
jimtron has joined #ipfs
<MikeFair>
lordcirth__: It'd be a NN-bit hash plus a YY-bit validator
<lordcirth__>
MikeFair, X bit hash + Y bit hash isn't going to be better than a X + Y bit hash, unless you are trying to mitigate a weakness in one hash.
<MikeFair>
I guess i'm just thinking about "useful data" and ignoring the "random noise" attack
<MikeFair>
I agree with the theory; but that's for arbitrary random data; which isn't what we actually have
mikerah has quit [Ping timeout: 252 seconds]
<lordcirth__>
Not sure what you mean. Hash algorithms approximate a random oracle. Or do you mean the input data being non-random? That doesn't matter.
<MikeFair>
input data being non-random (It's very "clustered" in the input space) which means certain segments of the hash space will always go totally unused
<MikeFair>
It's like having a 1,500 room mansion and only using 100 of them
ZaZ has joined #ipfs
<MikeFair>
Data Size + CheckSum + Hash seems much harder to try and "brute force" attack
<MikeFair>
Because you have to fit your attack in "multiple dimensions"
<lordcirth__>
inputs map randomly to outputs, and there are an infinite number of possible inputs for the same output (you'll just never find one) so I don't think that's relevant
<lordcirth__>
Having data size wouldn't hurt, I guess, except by making CIDs bigger
<MikeFair>
Not exactly true; I beleive it could absolutely be that there is no collision for Hash XYZ that is also Size NN and CheckSum 123
maxzor has joined #ipfs
<MikeFair>
There is a Collision in each item individually
<MikeFair>
And the input space isn't infinite
<MikeFair>
Size and Checksum constrain the input space
<lordcirth__>
But you would get just as much extra security by using those bits for a larger hash output.
<lordcirth__>
If SHA2 256 isn't enough, use SHA3 512bit.
<MikeFair>
No you don't because a single algorithm can't also check data size and use a second algorithm at the same time
<MikeFair>
A single Sha3 512 key has an infinite input attack space
<lordcirth__>
A secure hash with output size X provides the maximum amount of security that it is possible to put into an identifier of size X.
jesse22_ has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
<lordcirth__>
It cannot have more, because that's nonsensical, and if it has less, that means there's a vulnerability in the hash.
<MikeFair>
In an infinite random input space ; yes
<MikeFair>
But when you constrain the limits; it changes
<lordcirth__>
In any input space; if it had different properties depending on the space, that would also be considered a weakness
jimtron has quit [Ping timeout: 250 seconds]
jimtron has joined #ipfs
<lordcirth__>
One of the required properties of a secure hash is that no pattern in the input can have any predictable effect on the output
<MikeFair>
For example if size is constrained to 1k; then there are only 256^1024 possible inputs; and of the ones that collide with your crypto hash; they fail the checksum
<MikeFair>
Agreed
<MikeFair>
Which is why it's a "Second Test" not a change to the algorithm
jesse22 has joined #ipfs
<lordcirth__>
And the second test would help; but it is simpler and more useful to just increase the hash size
<MikeFair>
Except for the amount of built infrastructure that requires everyone to upgrade their existing systems to handle "bigger"
<MikeFair>
People are hard coding their systems to accept 256 bits as the size of an IPFS address
<MikeFair>
they don't care that's wrong
<MikeFair>
Mostly because they have a hard enough time understanding one crypto algorithm let alone MultiHash
<MikeFair>
I'd rather see a solution that goes "multidimensional" rather than always linearly growing along a second dimension
<lordcirth__>
MikeFair, if they are hardcoding a given CID format, I don'
<lordcirth__>
*t see how changing the format differently is going to help?
<MikeFair>
It makes the existing size last longer
jimtron has quit [Read error: Connection reset by peer]
ddahl has joined #ipfs
jimtron has joined #ipfs
<MikeFair>
A single 256 bit value is just insanely huge; it actually just seems like an inefficient brute forced design answer; 4 64-bit values seems better or 2 128 bit values
<MikeFair>
It makes better use of the space
jimtron has quit [Read error: Connection reset by peer]
<MikeFair>
Maybe I should be more concerned about random data attacks; but I'm just not
<lordcirth__>
preimage attacks don't care about what the target data looked like, generally.
<lordcirth__>
256 should last us a while; but 128 is dangerous
<MikeFair>
Here's something to think about; what it was 128 bits + "The count of 1 values in the data"
jimtron has joined #ipfs
<lordcirth__>
That would likely be more expensive to compute than sha256 (though maybe not, with the right optimizations) and couldn't be more secure.
<MikeFair>
I guess it depends on what you mean by "more secure"
<lordcirth__>
Attempts required to generate a collision.
<MikeFair>
I'm sacrificing security on random message attacks; to decrease the collision likelihood on "signal data"
<MikeFair>
Specifically to makethe key value smaller (use fewer bits)
<lordcirth__>
I don't think you *are* decreasing the likelihood on non-random data.
<lordcirth__>
If that was how hashes worked, at all, then there would be systems that worked that way.
<MikeFair>
Could be, like I said, I couldn't prove this; but for example say the Hash value was 128 bits; and the Hash Function was you hash the data forward to get the first value; then hash it backward to get the second value
<MikeFair>
I think crypto hash function authors are focused ona different problem
<MikeFair>
nm, that example is most likely identical to the single 256 bit function
<lordcirth__>
It by definition cannot be better than 256, and could be worse.
<MikeFair>
the premise of what I'm describing is that you can change/shape the distribution of the output space; for example by making more collisions in the "random noise" area, but fewer in the "signal space" to better distribute the use of your output keys
<lordcirth__>
But there is no such areas. Inputs map randomly to outputs.
xcm has quit [Ping timeout: 252 seconds]
<MikeFair>
Not in my design they don't
<MikeFair>
they partially randomly map
<lordcirth__>
Any reduction in the randomness of mapping is a *weakness*, not a strength.
<MikeFair>
For an evenly distributed input space I'd agree
<MikeFair>
But we don't have that
jimtron has quit [Read error: Connection reset by peer]
<lordcirth__>
For anything.
<MikeFair>
Not true
<MikeFair>
If your input space is 8 bytes
<MikeFair>
then anything larger than 8 bytes ipointless
jimtron has joined #ipfs
<lordcirth__>
Ok, sure, any input space not confined to be smaller than the hash output.
zeden has quit [Quit: WeeChat 2.4]
<MikeFair>
And if all your input data was for example "Ascii"
<MikeFair>
(Assume only 7-bit values)
jimtron has quit [Read error: Connection reset by peer]
<MikeFair>
Then there are valid inputs in your input space that you will never use but take up space in your output spsace
xcm has joined #ipfs
<MikeFair>
In that case you could write an alogrithm that says "ignore the highest bit when generating the hash"
<MikeFair>
and you'd get better usage of your output key space
jimtron has joined #ipfs
zeden has joined #ipfs
<MikeFair>
It's true that inputs map randomly to outputs, but it is a deterministic algorithm
<MikeFair>
And it's true that a larger hash "better protects" the input space; I just don't see how a deterministic algorithm is going to evenly distribute the output results when you don't actually make use of all the possible input values
<MikeFair>
(the algorithm is expecting 0-255 and you only ever feed it 0-128)
<MikeFair>
err 127
<MikeFair>
So there are collisions with your 7-bit input values that include values 128-255; all of them are invalid "signal carrying" inputs
boredmanicrobot has quit [Quit: boredmanicrobot]
<MikeFair>
For example adding a single bit that represented "input contains a value in the 128-255 range" would immediately invalidate all such collision
<lordcirth__>
MikeFair, a deterministic hash does evenly distribute outputs, since in practice the number of possible inputs far exceeds number of possible outputs.
<MikeFair>
lordcirth__: But not when the input space isn't randomly assinged
<lordcirth__>
the input space being even slightly larger dwarfs any such effect.
mandric_ has joined #ipfs
<MikeFair>
it very well could be; like I said, I can't prove whether or not having 255 bits of Hash + 1 of "Does the input contain invalid values" is more protective than 256 bits of hash
xcm has quit [Remote host closed the connection]
<lordcirth__>
If you're really worried about any effect of patterns in the input, compress your input first.
<MikeFair>
but IPFS doesn't do that
<lordcirth__>
No, compression has downsides and isn't needed. But nothing's stopping you from gzipping your stuff before adding it to IPFS. It might even be good for performance in certain cases.
<lordcirth__>
I err on the side of not compressing, so as to have a better chance of deduplicating with someone else
mandric has quit [Ping timeout: 255 seconds]
jimtron has quit [Read error: Connection reset by peer]
<MikeFair>
The question is more pedantic/theory based; if you have input constraints (size of a block being a really good one) then I believe you reduce the valid preimage attack surface
<MikeFair>
But maybe I'm mistaken
jimtron has joined #ipfs
xcm has joined #ipfs
<MikeFair>
Perhaps being able to choose from an infinite space is equally secure to forcing attackers to choose from a finite set
ygrek has quit [Ping timeout: 250 seconds]
<MikeFair>
For example I'd optimize the algorithm to make collisions more likely as the data got bigger
<MikeFair>
because human beings use way more "small sections of data" than they do "massive libraries"
<MikeFair>
actually, library is the wrong word;
<MikeFair>
because that's a composite of smaller data
<MikeFair>
lordcirth__: Thanks for chatting about it though
jesse22 has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
matt_ has joined #ipfs
jesse22 has joined #ipfs
matt_ is now known as Guest28348
jimtron has quit [Read error: Connection reset by peer]
jimtron has joined #ipfs
matth has quit [Ping timeout: 250 seconds]
xcm has quit [Read error: Connection reset by peer]
<MikeFair>
lordcirth__: Maybe a more general way to say it is something like "The input attack surface of finding any random data that matches H(m) is significantly reduced if 'm' must also pass other constraints (e.g. also be a valid GIF encoded image)"
xcm has joined #ipfs
<MikeFair>
lordcirth__: I'm thinking/suggesting that the bits added to the hash output which demand this kind of "data shape"/"data structure" validation could be "worth more" in security terms than simply adding the same number of bits to the output hash
<MikeFair>
But that's not something I can see how to "prove"
brabo is now known as brb
brb is now known as brabo
varl[m] has joined #ipfs
<MikeFair>
This matters because we humans aren't in the habit of storing and retrieving arbitrary random data
<MikeFair>
I used "CheckSum" as an psuedo-arbitrary proxy for "file type"
arthuredelstein has quit [Ping timeout: 245 seconds]
xcm is now known as Guest43306
Guest43306 has quit [Killed (weber.freenode.net (Nickname regained by services))]
arthuredelstein has joined #ipfs
xcm has joined #ipfs
spinza has quit [Quit: Coyote finally caught up with me...]
jimtron has quit [Read error: Connection reset by peer]
aerth has quit [Remote host closed the connection]
aerth has joined #ipfs
<MikeFair>
Okay, I just stumbled on this awesome tidbit: FIPS 180-4 specifies that the message has a ‘1’ bit appended, and is then padded to a whole number of 512-bit blocks, including the message length (in bits) in the final 64 bits of the last block
<MikeFair>
So Sha2 does include message length as part of the hash already; it's simply not something we can use to validate the IPFS fetch
spinza has joined #ipfs
jesse22 has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
MikeFair has quit [Ping timeout: 245 seconds]
jesse22 has joined #ipfs
Fessus has joined #ipfs
ZaZ has quit [Read error: Connection reset by peer]
Fessus has quit [Remote host closed the connection]
jimtron has joined #ipfs
Fessus has joined #ipfs
jimtron has quit [Remote host closed the connection]
jimtron has joined #ipfs
Biopandemic has quit [Ping timeout: 240 seconds]
Fessus has quit [Remote host closed the connection]
Fessus has joined #ipfs
Ai9zO5AP has quit [Quit: WeeChat 2.4]
Fessus has quit [Remote host closed the connection]