calavera has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
Senji has quit [Ping timeout: 244 seconds]
ggoZ has quit [Ping timeout: 240 seconds]
patcon has quit [Ping timeout: 264 seconds]
M-ryanwilliams has quit [Quit: node-irc says goodbye]
Guanin_ has quit [Ping timeout: 260 seconds]
<ipfsbot>
[js-ipfs] noffle opened pull request #86: upgrades peer-id to 0.6.0 (master...upgrade-peer-id) https://git.io/va45y
voxelot has quit [Ping timeout: 260 seconds]
<jbenet>
zignig: yes, too much :)
<ipfsbot>
[go-ipfs] whyrusleeping created feat/fd-limit-check (+1 new commit): https://git.io/va4dI
<ipfsbot>
go-ipfs/feat/fd-limit-check 2009d6f Jeromy: try to raise ulimit if its too low...
<ipfsbot>
[go-ipfs] whyrusleeping opened pull request #2466: try to raise ulimit if its too low (master...feat/fd-limit-check) https://git.io/va4ds
IlanGodik has quit [Quit: Connection closed for inactivity]
reit has joined #ipfs
Guanin_ has joined #ipfs
M-sivachandran has quit [Quit: node-irc says goodbye]
reit has quit [Read error: Connection reset by peer]
M-Peer2Peer has quit [Quit: node-irc says goodbye]
sivachandran has joined #ipfs
nicolagreco_ has quit [Quit: nicolagreco_]
<sivachandran>
whyrusleeping: are you around for discussion about generic key-pair dht?
r04r is now known as zz_r04r
Kane` has joined #ipfs
voxelot has joined #ipfs
inconshreveable has quit [Read error: Connection reset by peer]
patcon has joined #ipfs
<whyrusleeping>
sivachandran: i am!
dignifiedquire has quit [Quit: Connection closed for inactivity]
<tmg>
is there a plan to pull the commands package out of go-ipfs repo?
ipfsrocks has quit [Ping timeout: 244 seconds]
ygrek has quit [Ping timeout: 252 seconds]
<whyrusleeping>
tmg: yeah, we want to
<whyrusleeping>
the tricky part will be how to make it not rely on the main ipfs codebase
<whyrusleeping>
currently it has a few fields internally that are ipfs specific and we'd like to have it be its own thing, agnostic of ipfs
<whyrusleeping>
but, if the first iteration of it being extracted still depends on ipfs, my feelings wont be too hurt
nicolagreco_ has joined #ipfs
leer10 has joined #ipfs
<voxelot>
daviddias: ping
<voxelot>
err 2am in portugal
micxjo has joined #ipfs
<whyrusleeping>
voxelot: lol
ipfsrocks has joined #ipfs
ipfsrocks has quit [Ping timeout: 248 seconds]
jfred has joined #ipfs
<sivachandran>
whyrusleeping: I was looking at the dht implementation of public key(/pk/) and ipns(/ipns/). Wondering how the implementation prevents someone circulating wrong record for public key.
<sivachandran>
I understand that public key DHT records are signed but I don't understand why can't some X circulate wrong public key for Y? the DHT record could be signed by X own's private key so the signature check will pass.
<whyrusleeping>
public keys are checked
<whyrusleeping>
their contents must hash to the key
<nicolagreco_>
whyrusleeping: are we going to have a /pk/ ?
<sivachandran>
I couldn't find where we are checking the public key records are signed by the same person. May be I am missing something.
<whyrusleeping>
routing/record/validation.go
<whyrusleeping>
nicolagreco_: /pk/ is a dht key prefix
<nicolagreco_>
nice, is there a list of root folders ? /ipfs, /ipns.. ?
<sivachandran>
The implementation uses the Author field to retrieve the public key and verifies the signature. I don't where we do the Key == Author check.
<whyrusleeping>
nicolagreco_: theyre not really folders, theyre only namespaces
<whyrusleeping>
sivachandran: routing/record/validation.go line 81ish
<nicolagreco_>
I didn't know how to call them :) is there a list of namespaces?
<whyrusleeping>
theres currently just /ipns/ and /pk/
<sivachandran>
whyrusleeping: Is the ValidatePublicKeyRecord function called even while retrieving the value? I mean GetValue
<whyrusleeping>
sivachandran: yeap
TheWhisper_ has joined #ipfs
<nicolagreco_>
why are /pk/ in their own namespaces? could they not be just /ipfs or they have different implementation semantics (just wondering)
tilgovi has quit [Ping timeout: 240 seconds]
<sivachandran>
oh, then I need understand how it is getting called. I was wondering if I know the system I can very well inject wrong public key records without this check. But if we also do the validation at the time of retrieval then it is fine.
<voxelot>
anyone know how to create in memory at runtime buffer in the browser?
<sivachandran>
whyrusleeping: I am not sure how the generic key-value pair DHT should be. Any node can publish a key-value pair in the DHT. how do we decide which one we should pick.
<sivachandran>
I was thinking similar approach of what we do for the public key records
TheWhisper has quit [Ping timeout: 240 seconds]
<sivachandran>
Prefixing with the node's public key. So the retriever decides which record he wants.
<sivachandran>
What do you think about this approach?
dedgecumbe has quit [Read error: Connection reset by peer]
dedgecumbe has joined #ipfs
<whyrusleeping>
sivachandran: hrm... i don't know about that so much
<whyrusleeping>
at that point, we might as well just use ipns
<whyrusleeping>
so you have one record per key
<sivachandran>
yes, that is exactly I was also thinking
<sivachandran>
why can't we extend the ipns entries to have paths
<whyrusleeping>
they do have paths, you just publish a directory
<sivachandran>
something like /ipns/QmTkzDwWqPbnAh5YiV5VwcTLnGdwSNsNTn2aDxdXBFca7D/name1 /ipns/QmTkzDwWqPbnAh5YiV5VwcTLnGdwSNsNTn2aDxdXBFca7D/name2
<sivachandran>
with directory I need to republish everything
<whyrusleeping>
then what happens if i do `ipfs ls /ipns/<hash>`
<sivachandran>
hmm... yes that could be a problem when we publish directory on the peer ID
<sivachandran>
but if we stick with the rule that we don't publish anything on the root/peer hash then this will work
<whyrusleeping>
eh...
<whyrusleeping>
i'm not a fan of that
<sivachandran>
so we will never publish something on '/ipns/<hash>' but allow publishing like '/ipns/<hash>/<name>'
<whyrusleeping>
then that kill discoverability
<whyrusleeping>
i have no way of just looking at a peers ipns stuff
<sivachandran>
yes, you are correct
<sivachandran>
then I am not sure how to do this generic dht
leer10 has quit [Quit: Leaving]
<whyrusleeping>
i'm still not sure what youre trying to accomplish
<voxelot>
whyrusleeping: do you know what would be the case of a merkle dag node being empty? or the data field not containing data
<whyrusleeping>
whats the issue with republishing a directory when its changed?
<whyrusleeping>
voxelot: uhm, that should be valid
<sivachandran>
a side question, assuming a directory and its files are available in multiple nodes. can a peer publish the directory on its hash/peerID and go offline but still can other nodes resolve the peerID and retrieve the directory?
<voxelot>
that is unfortunate for me atm
<whyrusleeping>
voxelot: `ipfs object new` creates such an object
<whyrusleeping>
voxelot: why?
<voxelot>
poop
<whyrusleeping>
sivachandran: yes
<voxelot>
well in our current js implementation, if we try to unmarshal such an object, we at runtime create an empty buffer
<voxelot>
not sure how to do that in the browser
<sivachandran>
whyrusleeping: oh, then I think it solves my requirement
<whyrusleeping>
uhm, doesnt `new Buffer` do that?
<sivachandran>
I basically wanted to store some name-to-hash mapping within the network
<whyrusleeping>
sivachandran: oh, woo!
<voxelot>
maybe brfs can run new Browser?
<voxelot>
lemme check
<whyrusleeping>
sivachandran: yeah, the record you publish will be valid for 24 hours
<sivachandran>
I don't want it to keep the mapping in one node as it might go down
<voxelot>
s/new Browser/ new Buffer
<sivachandran>
So now I am thinking I will put the mapping in a file(e.g. JSON) and publish it with the node hash
<sivachandran>
and once the mapping file is cached in nodes within the network then anyone can provide the mapping. am I right?
<sivachandran>
whyrusleeping: thanks for your help in understanding this. will do some experiments and check whether it satisfies my requirement.
<whyrusleeping>
sivachandran: awesome, let me know if you need anything
<tperson>
Trying to figure out why Webpack isn't shimming itself thoguh
<voxelot>
tperson: it's just weird, the module was not throwing the error until i ran standard and was just cleaning up things that shouldn't affect it and them boom, buffer is not defined
<tperson>
Do you have a SHA of the module prior to running standard?
<voxelot>
i did before i made that last commit lol
<voxelot>
i could revert it if you want
<tperson>
No need to revert it
<voxelot>
thanks btw
O47m341 has joined #ipfs
pfraze has joined #ipfs
<tperson>
hmm this is strange
Aeon is now known as Aeonwaves
pfraze has quit [Remote host closed the connection]
Magik6k has quit [Ping timeout: 276 seconds]
Kubuxu has quit [Ping timeout: 250 seconds]
dedgecumbe has quit [Ping timeout: 276 seconds]
<voxelot>
tperson: just going to use that hack until dignfiedquire yells at me, it works for now
Magik6k has joined #ipfs
Kubuxu has joined #ipfs
<tperson>
Okay, I'm going to work on this a bit more
qbg has quit [Quit: Konversation terminated!]
_rht has joined #ipfs
tmg has quit [Ping timeout: 264 seconds]
tmg has joined #ipfs
<voxelot>
awesome!
frabrunelle has joined #ipfs
flapjack0811 has joined #ipfs
frabrunelle has quit [Client Quit]
frabrunelle has joined #ipfs
sivachandran has quit [Quit: Connection closed for inactivity]
<daviddias>
voxelot: tperson whyrusleeping (now, after reading the whole discussion) We've hit that bug in the past, in fact, Webpack 1.X gets weird with arrow functions and doesn't shim things properly, Webpack 2.X however, does the right thing, that is why we use it in all the other repos -> https://github.com/ipfs/js-ipfs/blob/master/package.json#L68
copernicusMogley has joined #ipfs
<dignifiedquire>
good morning everyone :)
<daviddias>
mornin' dignifiedquire :)
ulrichard has joined #ipfs
ulrichard has quit [Read error: Connection reset by peer]
<dignifiedquire>
daviddias: how is life?
<daviddias>
I'm out of coffee beans
<daviddias>
life is terrible
<daviddias>
:P
<dignifiedquire>
:D
<daviddias>
j/k I'm not like that xD
tmg has joined #ipfs
<daviddias>
how are you?
<dignifiedquire>
great, all things are moving in the right direction, I just had an excellent workout at the gym and now I'm thinking about a formal type specification for IPLD
<daviddias>
'formal type'?
<dignifiedquire>
representing the graph of an IPLD object in a type system, and definining operations on it, in a way that it gurantees that a mapping onto another graph is 1:1 and invertible
<dignifiedquire>
yes that's what I'm basing my work on
Encrypt has joined #ipfs
ulrichard has joined #ipfs
ulrichard has quit [Read error: Connection reset by peer]
<jbenet>
daviddias: i have a nespresso pod machine. it's surprisingly good.
<jbenet>
dignifiedquire: that's fantastic. btw, we may be able to ask cleichner and davidar (on github) for help on this. im sure they'd be interested in the problem.
<daviddias>
jbenet agree, I like them as well, in fact I have one too
* M-davidar
catches up
<daviddias>
but for a morning wake up routine, grinding beans gives you an extra kick
<M-davidar>
dignifiedquire: sounds interesting
ulrichard has joined #ipfs
Guest52 has joined #ipfs
<dignifiedquire>
jbenet: M-davidar yes happy to get some help on this, but want to have a prototype/draft first so it's to discuss
<dignifiedquire>
*easier
* dignifiedquire
typing skills ran away
ulrichard has quit [Read error: Connection reset by peer]
___ has joined #ipfs
* M-davidar
is planning on integrating IPLD with Haskell at some point, so I imagine there'd be some overlap
<dignifiedquire>
M-davidar: I'm very sure there will be overlap, will ping you when I have sth to show
<M-davidar>
cool
copernicusMogley has quit [Ping timeout: 240 seconds]
hellertime has joined #ipfs
martinkl_ has joined #ipfs
ulrichard has joined #ipfs
ulrichard has quit [Read error: Connection reset by peer]
<dignifiedquire>
voxelot: why are you still awake??
<daviddias>
voxelot: zombies, amiright
<voxelot>
nah its 5:30 here, been working out in the mornings
<voxelot>
bout to get ready to leave for the office
* dignifiedquire
impressed by voxelots work ethic
<copernicusMogley>
daviddias: I'm always planning some kind of trouble ;)
<r0kk3rz>
dignifiedquire: without fully understanding what your independance thing is about, i just want you to be aware that this was tried before https://www.youtube.com/watch?v=pcYJTIbhYF0
martinkl_ has quit [Ping timeout: 276 seconds]
O47m341 has quit [Ping timeout: 252 seconds]
ulrichard has joined #ipfs
martinkl_ has joined #ipfs
zorglub27 has quit [Ping timeout: 244 seconds]
jaboja has joined #ipfs
ulrichard has quit [Read error: Connection reset by peer]
<r0kk3rz>
or copernicusMogley rather ^^
mats_ has joined #ipfs
<copernicusMogley>
r0kk3r: It's a podcast that has interviewed me and the other founder of Alexandria before. They are very interested in IPFS and reached out to me for another interview, this time with the focus on the IPFS part of our system.
<copernicusMogley>
I'm not a fan of blockchain law/governance, personally, and tend to avoid that sort of thing.
<r0kk3rz>
its an interesting talk regardless, mostly focusing on lessons we can learn from the dawn of the internet
<daviddias>
voxelot: nice! :)
<r0kk3rz>
and how people then thought it was all going to turn out
<r0kk3rz>
including a 'declaration of independance' which was recorded in 1996
cemerick has joined #ipfs
reit has joined #ipfs
<copernicusMogley>
That is interesting. watching the video now.
<whyrusleeping>
daviddias: lol, but i cant do the russian accent required to sound portugese :P
zootella has quit [Quit: zootella]
pfraze has joined #ipfs
zootella has joined #ipfs
<noffle>
whyrusleeping: that's a thing of real beauty, right there. <3
<whyrusleeping>
:D
<noffle>
that's where people go to become happy
<noffle>
and not asleep
<whyrusleeping>
the best recent addition is the electric kettle that you can set a temperature on
<whyrusleeping>
and it has a 'hold' button
<whyrusleeping>
so you can say 'get the water to 201 degrees, and just hold it there until i'm ready'
<redfish>
energy star certified: warmimg the earth one kettle at a time
<noffle>
nice
<noffle>
coffee is no joke
<whyrusleeping>
redfish: my house is consistently at a cozy 62f
<whyrusleeping>
so im okay leaving the kettle on while my fingers freeze
<whyrusleeping>
in other news; single pane windows, while definitely stylish and rustic, do not make for a happy heating bill
<whyrusleeping>
and the landlord refuses to replace them because 'theyre part of the house'
<Kubuxu>
and here we are planning to introduce laws that all new houses must be build passive, which means that they should be warm only on normal energy usage, w/o heating.
martinkl_ has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
nicolagreco_ has joined #ipfs
<whyrusleeping>
i own two personal space heaters. does that count as 'normal energy usage' ?
ygrek has joined #ipfs
<whyrusleeping>
or should i start mining bitcoin in my basement
<Kubuxu>
the latter
nicolagreco_ has quit [Quit: nicolagreco_]
nicolagreco_ has joined #ipfs
zootella has quit [Quit: zootella]
nicolagreco_ has quit [Client Quit]
zootella has joined #ipfs
chriscool1 has joined #ipfs
nicolagreco_ has joined #ipfs
<dignifiedquire>
good morning whyrusleeping
<whyrusleeping>
dignifiedquire: can i get a quick feature request on randor?
<dignifiedquire>
whyrusleeping: yes but not immediately
<whyrusleeping>
okay, i was just wanting the debug mode of randor to be a little less verbose when printing out file contents
Encrypt has joined #ipfs
hellertime has quit [Quit: Leaving.]
<richardlitt>
xicombd: you around?
zootella has quit [Quit: zootella]
<xicombd>
richardlitt: on the tube heading home, give me 5 mins :)
<richardlitt>
xicombd: cool. No rush. :)
hellertime has joined #ipfs
<dignifiedquire>
whyrusleeping: how exactly? what do you not want to be printed?
zootella has joined #ipfs
<whyrusleeping>
dignifiedquire: it would be nice if the entire contents of the file didnt get printed
<voxelot>
dignfiedquire: repo.keys should be an object containing all of the data blocks plus a get function correct?
<whyrusleeping>
maybe just the first 20 characters and some elipses, or something
<dignifiedquire>
whyrusleeping: DEBUG=randor:*,-randor:generate IPFS_EXEC=$GOPATH/bin/ipfs bin/randor run -l 100 -p 10
<dignifiedquire>
this removes the generate output
<voxelot>
ohh keys is the private key
<dignifiedquire>
whyrusleeping: is that enough, or do you need the short args to be still printed?
<whyrusleeping>
that doesnt appear to have worked...
<dignifiedquire>
what happend?
<dignifiedquire>
maybe you need to add quotes
<dignifiedquire>
whyrusleeping: DEBUG="randor:*,-randor:generate" IPFS_EXEC=$GOPATH/bin/ipfs bin/randor run
calavera has joined #ipfs
<voxelot>
dignifiedquire: when we create the repo in our web tests, there is a step that goes through all of the blocks and loads them into an array, does this not get passed to the datastore?
nicolagreco_ has quit [Quit: nicolagreco_]
<dignifiedquire>
voxelot: no idea
<voxelot>
logging, repo.datastore is an empty object with the createreadStream and such functions
<dignifiedquire>
xicombd: and daviddias can probably help there
<daviddias>
because indexedDB uses paths as keys for the buckets
<xicombd>
richardlitt: just got to the computer, what's up?
<voxelot>
i see where the paths are supposed to be getting travsered and loaded into the repoData array. is there a way to console.log indexedb to be sure i'm loaded?
<daviddias>
you can open chrome dev tools
<daviddias>
and see it loaded there
<daviddias>
on the Resources tab
<voxelot>
ahh i see ty
<daviddias>
do you see the IndexedDB icon there?
<daviddias>
awsm :)
<dignifiedquire>
also whyrusleeping important question, what kind of grinder do I need?
<voxelot>
hmm i haz all the blocks, why no finds when tests :(
hellertime has quit [Quit: Leaving.]
<daviddias>
voxelot: make sure that you are seeking on the right bucket
<richardlitt>
xicombd: working on object patch commands
<richardlitt>
I feel bad for the friend of the friend of my cousin whose password and account I use without him knowing to access internet in this cafe
<richardlitt>
Also, `patrick`, you really shouldn't use a five character name of what I assume is your wife as your password
<dignifiedquire>
whyrusleeping: so I can get this one "QuickMill Orione 3000" and "Rancilio Silvia" for nearly the same price together with the above mentioned grinder any thoughts which one?
<whyrusleeping>
dignifiedquire: i get no output from randor when i do that
<dignifiedquire>
whyrusleeping: that makes no sense, can you show me your terminal, also what shell are you using?
<richardlitt>
xicombd: So that it doesn't throw a syntax error
<whyrusleeping>
fish
<richardlitt>
apiary format is weird.
<dignifiedquire>
whyrusleeping: could you try in bash for me?
<whyrusleeping>
hrm, thats better. must be something different about the '*' character being set in the env vars
pfraze has quit [Remote host closed the connection]
<xicombd>
richardlitt: ahaha cool
pfraze has joined #ipfs
TheWhisper_ is now known as TheWhisper
s_kunk has quit [Ping timeout: 244 seconds]
nicolagreco_ has quit [Quit: nicolagreco_]
<ipfsbot>
[go-ipfs] RichardLitt created feature/object-patch-set-data-docs (+1 new commit): https://git.io/vaETI
<ipfsbot>
go-ipfs/feature/object-patch-set-data-docs ba4c7e9 Richard Littauer: Edited object patch set-data doc...
micxjo has quit [Quit: Connection closed for inactivity]
nicolagreco_ has joined #ipfs
mid has quit [Ping timeout: 248 seconds]
M-2617 has quit [Quit: node-irc says goodbye]
patcon has quit [Ping timeout: 244 seconds]
palkeo has joined #ipfs
edrex_ is now known as edrex
ulrichard has joined #ipfs
jokoon has quit [Quit: Leaving]
Encrypt has quit [Quit: Quitte]
rendar has quit [Ping timeout: 252 seconds]
inconshreveable has joined #ipfs
rendar has joined #ipfs
martinkl_ has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
Protocol_under has quit [Quit: Teleported with a cheezecake]
<whyrusleeping>
apparently, if i fill up all my ram by file bombing my /tmp directory, it makes vim print out the file i'm trying to save instead of actually saving it
<whyrusleeping>
weirdest bug ever
<whyrusleeping>
also, on that note: dignifiedquire, could i get a flag to make randor use a custom path for its 'temp' ?
<redfish>
any resources/thoughts about feasibility of an indexed database on top of IPFS, presumably, a B-tree? Difficulty is in insert/delete. Thx.
<redfish>
in many respects IPFS is just like an HD -- for efficiency, gotta minimize the lookups (analogously to reads), so same solutions should be applicable
<M-mubot>
in many respects ipfs is just like an hd has -1 points, -1 of which is for efficiency, gotta minimize the lookups (analogously to reads), so same solutions should be applicable.
<whyrusleeping>
M-mubot: i'm going to kick you
<whyrusleeping>
redfish: a few different people have looked into implementing databases on top of ipfs
<ipfsbot>
[go-ipfs] whyrusleeping tagged v0.4.0-rc3 at master: https://git.io/vaEKP
<redfish>
whyrusleeping: thx. fascinating. so, CRDT is approaching the problem from a totally different perspective. IIUC, a bunch of nodes form a pubsub group, they are replicas, they maintain a DB.
<redfish>
the other perspective is simpler -- IPFS = storage layer for an *unreplicated* DB, so no consistency problem. Anybody can "mutate" the db, just like anybody can commit on top of a git root.
<redfish>
perhaps, the question I should have asked is: what are efficient methods for searching across merkel trees?
<redfish>
s/merkel trees/merkel tree/
pfraze has quit [Remote host closed the connection]
<redfish>
general content search, that is, not lookup of hash, of course
<whyrusleeping>
redfish: well, just treat the merkletree like you would a B-tree
ugjka has quit [Ping timeout: 240 seconds]
<redfish>
yeah, it's a game of indexing
<redfish>
index is entangled with data in a B-tree (my feeling), but with IPFS, it needs to be a super-clean isolated companion
_rht has quit [Quit: Connection closed for inactivity]
<whyrusleeping>
yeah, you will very likely want the index separate from the data itself
<redfish>
yep. ok, thanks a lot for the thinking material, i'm glad I asked (else stuck in own box of thoughts)
<whyrusleeping>
which frees you up to make some interesting design choices on making a B-Tree like datastructure *just* for indexes
<whyrusleeping>
(and possibly other metadata)
martinkl_ has joined #ipfs
martinkl_ has quit [Max SendQ exceeded]
<redfish>
say we restrict the problem scope: e.g. append-only DB with an append-only index, more doable and still very useful
<redfish>
philosophical.. when does an index become data, index of index of ... index of data...
martinkl_ has joined #ipfs
ashark has quit [Ping timeout: 244 seconds]
<whyrusleeping>
lol
<whyrusleeping>
redfish: if you come up with an append-only DB implementation, i'd love to see it
<achin>
months ago someone in here linked to a handy webpage that had a nice graphical layout of your current time, and what that would be locally for every other timezone. does anyone happen to know what i'm talking about and have that URL handy?
<redfish>
correction: db traversal wouldn't have much to do with merkel tree traversal, the DB would be its own tree, just linking hahes together, without the Merkle property (obviously...)
<redfish>
yeah, i'll come back if any concrete ideas crystalize
<daviddias>
you need to change it to be one of those require buffers from WebPack
<voxelot>
if i try to require('...txt.data') it doesn't pick up the other blocks in the data-store
<daviddias>
or use a browserify transform such as brfs
<voxelot>
right
<voxelot>
so take 1.2MiB.txt.. it is like 6 blocks, how should i require all of them
<daviddias>
that specific line is just requiring one of the blocks
<daviddias>
so ~(256Kb + protobuf stuff)
<daviddias>
even if you were loading into memory 1.2MiB, it would be ok
<voxelot>
somehow, dagnode.size() equals the entire dag, not just one node
<voxelot>
loading 1.2MiB into mem wont pick up each chunks proto and is too smal
<daviddias>
dagnode.size() will equal to the entire dag
<daviddias>
cause it sums the sizes of the nodes it points too
<daviddias>
that is the correct behavior
<voxelot>
right, so to test against the blocks, i need to load all of the nodes
<voxelot>
and test against the created node.size() from our data-importer, just not sure how to load all of the blocks
<daviddias>
The way that the tests are written is:
<daviddias>
1) files get loaded and added to IPFS using the importers, which create the respective unixfs nodes (dir and file nodes)
<daviddias>
2) the root node from that import (and some of the leaves) is compared to matching node that go-ipfs generated (those .block on the test-data folder)
<daviddias>
does this helps to make sense out of it?
<voxelot>
no haha, i understand that part
<voxelot>
i just dont understand how readFileSynce on one block, then unmarshal that buf leads to the entire dag
<daviddias>
because it doesn't :)
<voxelot>
does unmarshal read the links then pik up the other blocks too
<voxelot>
then how is the size equal to the entire dag?
<daviddias>
the math is that if you compare the root node, and check its size, it's data field and the links it is pointing too
<daviddias>
you are pretty damn sure that the thing is the same underneath
<daviddias>
you can check if the tree is the same, if the root hash is the same
<daviddias>
this was actually the first use case for MerkleTrees:)
<daviddias>
what happens after is that I pick one of the leafs and compare it as well, but that is there because when I was building it, I caught some bugs and started testing the creation of every leaf to make sure they were right too
<daviddias>
I can jump into a call and talk this through, it might help :) are you available ?
<voxelot>
yeah that would be great, Uncaught AssertionError: expected 1258318 to equal 251
<daviddias>
voxelot: that might be because of the encoding
<daviddias>
of how the files are being buffered into memory