<gozala>
lgierth: it’s ok to do additional things when I start up my ipfs node to enable that
<lgierth>
pubsub and the dht -- the dht has various prefixes, and peer routing is just one. there could be a metadata prefix for such thing. and with pubsub you could to request/response over a topic which derived from their peerid
<lgierth>
but not currently any API for something like that :(
bwn has joined #ipfs
<lgierth>
for that ^ meaning for the dht prefix. pubsub you can already do
<gozala>
lgierth: so if understood your pubsub suggestion, you’d sub on peer id
<lgierth>
yeah
<gozala>
and send on that peer id something like giveMeMetadata
<lgierth>
but: you might be opening the thing as ipfs://, i.e. a specific version snapshot
<lgierth>
for that case i was thinking a reverse-ipns lookup would be useful
<lgierth>
tell me ipns entries that point or pointed to hash X
jkilpatr has quit [Ping timeout: 268 seconds]
<gozala>
lgierth: yeah that would be really cool
<gozala>
lgierth: another thing I would love to have is if ipns was a log rather just a mapping
<gozala>
so I could see all the versions of it assuming someone still holds them
<lgierth>
you can put a history data structure in there
jkilpatr has joined #ipfs
<lgierth>
and the codec bit of the CID your ipns entry points to could identify what kind of history data format it is
<lgierth>
putting a whole versioning scheme into IPNS would be a bit much
<lgierth>
treat it as a primitive for mutability
<lgierth>
here's the mindblowing thing: i think dat there could be a dat implementation on top ipfs, complete with compatible hashes and data formats
<lgierth>
*i think that ;)
MikeFair has joined #ipfs
<lgierth>
about a log in specific: there can be multiple writers for an ipns key - you need to avoid conflicts and rewriting history
<MikeFair>
lgierth, I only saw that last line; but to clarify I know what you mean, you're saying that since ipns is generally updated by a private/public keyfile; and that keyfile can be distributed to many parties; that distributed set of parties can do bad things to each other if they are coordinated properly right?
jkilpatr has quit [Remote host closed the connection]
<MikeFair>
s/are/aren't/ coordinated
<lgierth>
we were talking about versioning history as part of IPNS
<lgierth>
and my point was IPNS should be treated as just the mutability primitive, and that versioning history can be a data structure that IPNS points to
paralin has joined #ipfs
maxzor has quit [Remote host closed the connection]
<lgierth>
it's also a matter of keeping room for experimentation. if versioning were to be made part of IPNS itself, it'd likely tend to be treated as set-in-stone once its first version has been implemented and is in use
<MikeFair>
but without some kind of additional data on the IPNS record itself, wouldn't a third party have a hard time telling if the pointed to thing is out of date?
<lgierth>
expiry/TTL is a different matter, IPNS already has metadata to tell which record is newer
<lgierth>
time based, and counter based
<MikeFair>
And I guess there's some argument where counter+timestamp != versionid?
<lgierth>
and once the TTL is up, records are generally evicted
<MikeFair>
oh right!! That was the problem I kept having (my IPNS records would disappear after 24 hours)
<lgierth>
oh yeah tons of. counter+timestamp is the most dead simple versioning scheme
<lgierth>
there's tons of more sophisticated versioning schemes
<lgierth>
CRDTs are one :)
<paralin>
on pubsub I'm getting a lot of connection reset errors, and 100% cpu, even though my node isn't subbed to anything - are you guys aware of this? i would think my node would not bother to process any pubsub streams it is not actively participating in.
<lgierth>
git, couchdb's scheme, etc.
<lgierth>
paralin: is that in the browser or go-ipfs?
<paralin>
lgierth: go-ipfs, latest release
<MikeFair>
Oh I agree; I was just thinking of coordinating for the record itself -- I'm with you on the pointed to stuff is where the "smart" version are; but also believe that some kind of basic record id is needed to help the smart stuff figure things out
<lgierth>
but then you'd also want these historic records to stick around
<lgierth>
and then you're redoing a small part of ipfs, in ipns :)
<lgierth>
ipns is just "i'll tell you what the latest thing is"
<lgierth>
and the timestamp/counter is purely for deciding what the latest thing is, not for keeping track of history
<MikeFair>
I'm not sure ipns needs handle keeping the historic around; that should be a log of some kind in "the latest stuff"
* MikeFair
nods.
jdevoo has quit [Remote host closed the connection]
jdevoo has joined #ipfs
<MikeFair>
I think of ipns as a psudeo equivalent to a hard drive block; when you ask for the block id, you get back the current dataset
<MikeFair>
when you write to it, it updates the data block; if you want a recoverable filesystem from that, that's an application
<lgierth>
or the partition table saying a certain file is currently at a certain sector
<MikeFair>
exactly
<lgierth>
with emphasis on currently
ashark has joined #ipfs
<lgierth>
and there's also the IPLD part, where you can treat ipfs as an immutable graph database
<lgierth>
(that's not the FS part in ipfs anymore)
<MikeFair>
I'm trying to remember what challenges I ran into using IPLD...
<MikeFair>
Oh right! I had to know where the db partitions were, which I couldn't know in advance
<MikeFair>
So I couldn't look up say: /addressbook/contacts/lgierth/email/home
<MikeFair>
From the base ID that contained addressbook
<lgierth>
right, that index view kind of access isn't included, need to build it yourself
<lgierth>
might be included as a "recommended data structure" or so in the future
<lgierth>
we're all only starting to figure out how to model data in a content-addressed manner
infinity0_ has joined #ipfs
ashark has quit [Ping timeout: 248 seconds]
infinity0_ has quit [Changing host]
infinity0_ has joined #ipfs
infinity0 is now known as Guest54735
Guest54735 has quit [Killed (card.freenode.net (Nickname regained by services))]
infinity0_ is now known as infinity0
<MikeFair>
I commented on some of the GitHub threads talking about the challenges of that kind of resolution last year
<MikeFair>
agreed, it's definitely a mindwarp ;)
<lgierth>
know how every basic database class in the beginning teaches you de-normalization?
* MikeFair
nods.
<lgierth>
we need a similar concept and practice for content addressed graphs
<lgierth>
there's probably something in that direction in the general graph database world
<MikeFair>
I think the answer is going to be in some kind of slick "naming index"
<lgierth>
maybe there's also something about the directed acyclic subset
<MikeFair>
One thing I love about CAS, is that when I ask for an ID, I can be assured (and test) that the data I receive is in fact the data I asked for
<gozala>
lgierth: sorry I had to be away for sometime. I understand your point about treating ipns more like cell and that’s reasonable. I guess I wish there was something else then which would be ipvs if you like
<MikeFair>
I think one place that will make a huge difference is authenticated distributed code execution. (If I ask for a piece of code from ipfs and run it, I can be assured it is what I asked for and not some hacked up version of it.)
<MikeFair>
gozala: Have you looked at Orbit-Db?
<gozala>
unless it’s part of the ipfs ecosystem then nothing in the ecosystem will know how to deal with it
<gozala>
MikeFair: I glanced at it in the past I think
<lgierth>
gozala: oh! speaking about versioning, lol, i always forget in these situations: we can natively address Git and resolve through history, branches and directories
<gozala>
MikeFair: lgierth what I mean is today if I’m listing some site
<lgierth>
all that while preserving original hashes
<gozala>
there is no way of looking at what it looked like yesterday
<lgierth>
so you can basically use git as a versioning protocol over ipfs/ipns
<gozala>
or a month ago, unless site develops it’s own thing to allow that
<gozala>
I would love if this was possible in distributed web case
<lgierth>
we can `git push` to an ipfs remote
<MikeFair>
gozala: Sure there is, just not natively, you post the ID to a log somewhere.
<lgierth>
it's just not yet implemented in js
<MikeFair>
gozala: There's no gaurantee ipfs will maintain old versions of the site anyway
<lgierth>
it does if you want it to. that's enough
jkilpatr has joined #ipfs
<gozala>
MikeFair: that’s totally fine though
<gozala>
lgierth: +1
<MikeFair>
You have to take on making sure all that history stays intact; so even if you could reference the existance of an older ID, you won't necessarily be able to see it
<gozala>
lgierth: I’m not totally following on git as versioning mechanism here though
<gozala>
MikeFair: yeah I understand it old version will be available as long as there is interest (peers who pinned it)
<lgierth>
you're wanting a versioning scheme in a social context, i'm just bring up git as an example of a versioning scheme on top of ipfs. (a mindblowing on too lol)
<lgierth>
the CRDTs used in peerpad are another example
<gozala>
lgierth: but that seems same as wikipedia allowing me to see previous version
<gozala>
I’d like it to be at web scale
<gozala>
hence build-into protocol
<gozala>
does that makes sense ?
<MikeFair>
gozala: For example, in the top level directory of your website, put a ".revisions" file. Inside that file put a list of IPFS CIDs that represented the site, one per row with a datetime stamp and the CID
<MikeFair>
gozala: This is similar to the Interplanetary PasteBin I was helping with ;)
<gozala>
MikeFair: but that is again, per opt-in
<gozala>
problem is each site will come up with own way of doing it so browser won’t be able to know how to present versions
<MikeFair>
Yes, convention is a powerful tool
<gozala>
and many sites will even ignore that
<gozala>
if it was part of the protocol it becomes feature of the protocol
<gozala>
meaning browser then can show you changelog regardless where you navigate to
<MikeFair>
But the requirements of maintaining past revisions of things that don't require past revisions is bloat problem don't you think?
ashark has joined #ipfs
<MikeFair>
gozala: what if a website doesn't want to help you get access to past revisions?
<gozala>
MikeFair: I’d say it’s not trustworthy then, but that’s probably a not a good answer.
<gozala>
MikeFair: but really it could be opt-out feature or even opt-in, but again something more than just convention
<MikeFair>
gozola: If I'm undestanding what you're looking for; it's something like when you look up an IPNS record, there's a reference to the prior IPNS record
ashark has quit [Ping timeout: 248 seconds]
<MikeFair>
gozala: And I could ask the network for a specific CID version of that IPNS record
<gozala>
MikeFair: exactly although it does not has to be a ipns per say
<MikeFair>
gozala: IPNS is the only thing that has an updatable CID as far as I know
<MikeFair>
I guess that's why you were saying "IPVS" which had this extra semantic
<gozala>
MikeFair: What I meant is as lgierth pointed out IPNS is just a cell
<gozala>
maybe there can be something like IPNS were it’s more like secure ledger
<MikeFair>
gozala: From a practical standpoint IPNS is "fixed address in space" and a cell makes a very good metaphor; IPNS is a reference to something else, like a DNS lookup translates a fixed string of characters to a changeable numeric address
<MikeFair>
gozala: You're saying we should be able to lookup last year's versions of the DNS database system
<MikeFair>
So www.ipfs.io points to place W today; I have no clue what A-V were
<MikeFair>
Nor can I find out using the DNS system itself
<MikeFair>
IPNS means I can publish a fixed address that every one can have and keep to mean "mike's website"
<MikeFair>
I can then update the CID that thing points to so people can see changes over time
<gozala>
MikeFair: I think another way to think about it is
<gozala>
there is ipfs so I can have ipfs://${v1} ipfs://${v2}
<MikeFair>
lgierth: Do IPNS have a CID that can be referenced right? As in, the entry that hosts the IPNS record has a CID?
<gozala>
and there is also ipns:// where it can point to arbitrary ipfs://${vx}
<gozala>
but then there is no way to draw any connection between ipfs://${v1} and ipfs://${v2}
<gozala>
even if there is a relation
<MikeFair>
well there is, just not a native one; for example if ${v1} and ${v2} were different blocks in a file, they can be related
<gozala>
my claim is it would be extremely valuable to have something in the protocol that could express relationships
<MikeFair>
gozala: There is, but you're relating two revisions of the conceptually same object -- opposed to relations within the object state itself
<MikeFair>
for example, a directory of files relates all those files as siblings to each other; and ipfs handles that relation quite well
<MikeFair>
a website is a related group of files
<MikeFair>
files are a related group of data blocks
<MikeFair>
Traversing revisions of objects through time is the missing relation
<MikeFair>
Currently relations are handled as "spatial" or "positional" and consistent in the same "moment"
<MikeFair>
i.e. what is the current state of this object "now"
<gozala>
MikeFair: ok but I’m not sure where you going with this
<gozala>
you are right that I’m talking about different states of conceptually same identity
<gozala>
file relations with in that identity isn’t really interesting for me in this context
wmoh has joined #ipfs
ashark has joined #ipfs
<MikeFair>
right, in the CAS storage paradigm, the "state" or "content" is the identity
<MikeFair>
So every historical state has a "CID"
<gozala>
yes
<MikeFair>
you're lamenting the loss of CID.1 when you update IPNS to CID.2
<gozala>
yes
<MikeFair>
What I'm thinking is if you tracked the CID for the prior IPNS entry on the block that holds the current CID reference, you can build what you want kind of easily, and unless I'm mistaken, all the data fields and semantics are there to add that quite easily
<MikeFair>
But, I still agree with lgierth that historical revisions isn't a thing for IPNS to natively track
<MikeFair>
But in this case, I could probably be swayed
ashark has quit [Ping timeout: 240 seconds]
<gozala>
MikeFair: again I’m fine if it’s not ipns
<gozala>
it’s just ipfs content supports ipns links and would relsove
<MikeFair>
well the CIDs are totally different things
<gozala>
it whatever that other thing maybe called, I would like it to be also be supported
<MikeFair>
There's no way to identify "revision -1 of this thing"
<MikeFair>
We can call it IPVS
<gozala>
MikeFair: I”m thinking of linked-list of CIDS
<MikeFair>
Hehe - that's a file in ipfs ;)
<gozala>
rather than just single CID record which is what in my understanding IPNS is
<MikeFair>
or IPLD
<MikeFair>
I'd disagree with making IPNS anything but a reference to a single CID; though adding a single additional CID of the prior IPNS record is something I could see the value (and little harm) in doing
<gozala>
MikeFair: I’m afraid I need to run, but if there is anything else I can read or write to pursue this please point out and I’ll try to get to it
<MikeFair>
I'd start by assuming you're going to fracture the space into "things browsers can see now" (limited to only the latest, current version); and a semantic protocol for prior versions
<gozala>
MikeFair: two CID’s would be a linked list though no :)
<MikeFair>
exactly
<MikeFair>
it requires very little extra data, and gets you what you want; but only so long as the old IPNS entries stick around
<MikeFair>
(something has to pin them)
<gozala>
MikeFair: internet archive for example
<gozala>
or any bookmark in any ipfs supporting browser :)
<MikeFair>
exactly, could be anything really
<MikeFair>
but something must
<MikeFair>
ipns entries are designed to be purged; for good reasons too
<MikeFair>
frankly, I'm more in favor of just keeping track of the historical CIDs that website referenced
<MikeFair>
but adding a single extra CID to each IPNS node is a bit more efficient i think
<MikeFair>
(it would then be a flag when updating the IPNS entry whether or not to track/update the prior version record)
jdevoo_ has joined #ipfs
<MikeFair>
(He; but I gaurantee the next thing ppl will want is the ability to modify that CID)
<lgierth>
that's more of a UI porcelain than a protocol though
<MikeFair>
:) -- LMK if I'm comprehending right that the IPNS data entry has a CID that could be referenced
<lgierth>
but here's again why i brought up git earlier: it's an example of versioning data structure (the `git` cli is just plumbing and porcelain around that data structure)
wmoh has quit [Read error: Connection reset by peer]
<lgierth>
it would be interesting if we can make git.js (if that's the git library nowadays) cooperate with ipfs, just like https://github.com/magik6k/git-remote-ipld does for regular git
<MikeFair>
From the git reference it sounded like you were saying I could type something at a CLI that could actually pull a GIT hash
<lgierth>
gozala: i can imagine there's also cases where you want a page/app to have multiple writers, e.g. for comments or a forum
<MikeFair>
or were simply making the parallel that the CRDTs mimic the structure of git
<lgierth>
where you don't neccessarily want everyone to contribute by "fork"
<lgierth>
that's where you'd use a CRDT (as in peerpad)
wmoh has joined #ipfs
<lgierth>
i guess what i'm saying is, versioning systems do exist, it's just that not terribly many people have used them nicely yet :)
ashark has joined #ipfs
<lgierth>
MikeFair: yes IPNS can point to any CID, so it doesn't have to be a unixfs structure -- can be anything
<lgierth>
you could have ipns://git.mike.com point directly to the original git data structure
panda0 has joined #ipfs
<lgierth>
that means it can be the remote. and if your ipfs node has the IPNS key for that, it's a writable origin
panda0 has quit [Client Quit]
<MikeFair>
lgierth: I was asking if the entry for the IPNS record itself has a CID I can pull (to see the raw IPNS record)
AkhILman has quit [Ping timeout: 240 seconds]
panda0 has joined #ipfs
<lgierth>
ah i see! no not currently
<lgierth>
it's not ipfs content in the strict sense
<lgierth>
only gets written to the dht
<MikeFair>
okay, maybe not a CID, but a DHT address then
panda0 has quit [Client Quit]
panda0 has joined #ipfs
<MikeFair>
Or does an updated IPNS entry overwrite the same DHT location?
wmoh has quit [Ping timeout: 240 seconds]
ashark has quit [Ping timeout: 248 seconds]
<lgierth>
i am not a 100% sure
<MikeFair>
I was just thinking that if an IPNS entry contained the DHT address of the prior version of itself, that'd be most of the requirement. Though frankly I think the best way to handle this is a convention where IPNS points to a file or directory that contains a list of CIDs, the top one being first
<MikeFair>
But it does require a change to IPNS resolution
<MikeFair>
So that's why I was calling IPVS
<lgierth>
either a) other nodes you ask just won't give you ipns entries that they know are outdated, or b) the older entries get overwritten by the new ones
<MikeFair>
Well I wouldn't ask for it by IPNS id; I'd ask for it by DHT address
* MikeFair
has another "Damn IPFS is just soo cool!" moment. ;)
<MikeFair>
Is there a "short version" of major updates since early last year?
<lgierth>
no, there should be one
<lgierth>
unless someone beats me to it i'll make one and let people at a meeting in 2 weeks add missing ones
<MikeFair>
It's kind of the wild west when I look at all the projects and at the time the project was reorganized repositories so it was difficult to know exactly what I should be reading/focusing on to see the current stuff
dominic__ has joined #ipfs
<MikeFair>
s/reorganized/reorganizing
<lgierth>
i sometimes get 300 github emails a day :)
<whyrusleeping>
^ thats a common thing over here
<lgierth>
maybe something that'd make it easier to follow... have a
<lgierth>
... have a "watchers" team on each org, that's added to every repo with "read" perms. than people can enable their "auto-watch new repos of mine" and ask to be invited to that team
<lgierth>
and bam, you now get notifications for 200 or so repos
<lgierth>
gl&hf
<lgierth>
of course regular update posts are better
wmoh has joined #ipfs
<MikeFair>
Well for example, what's the difference between something like IPLD and IPRS?
<MikeFair>
I think gozala should be pointed to iprs: Signed, based on ancestry (chain)
<MikeFair>
This record type uses digital signatures (and thus a PKI) and merkle-links to other, previous records. It establishes that the "most recent" record (merkle-ordered) is the most valid. This functions similar to a git commit chain.
<lgierth>
IPLD is a data model, it describes how different data structures can link to each other and how one can resolve through a heterogenous graph. and IPLD Selectors will be a query language for that. IPRS is a data structure, and a set of defined functions on that data structure
<lgierth>
right! gozala: ^^ IPRS :)
dudebbn_ has quit [Ping timeout: 268 seconds]
_whitelogger_ has joined #ipfs
doodlemania has joined #ipfs
yangwao has joined #ipfs
ygrek has quit [Ping timeout: 268 seconds]
shoogz has joined #ipfs
shoogz has quit [Max SendQ exceeded]
JohnX has joined #ipfs
A124 has quit [Ping timeout: 260 seconds]
vimclippy has quit [Ping timeout: 260 seconds]
shoogz has joined #ipfs
A124 has joined #ipfs
vimclippy has joined #ipfs
SevenTimes has joined #ipfs
dominic__ has quit [Ping timeout: 260 seconds]
lgierth has joined #ipfs
jaboja has joined #ipfs
AntonLatukha[m] has joined #ipfs
dgrisham has joined #ipfs
robattila256 has joined #ipfs
lord| has joined #ipfs
synthmeat has joined #ipfs
dudebbn has joined #ipfs
lemmi has joined #ipfs
tglman1 has joined #ipfs
ashark has joined #ipfs
teej has joined #ipfs
jaboja has quit [Ping timeout: 264 seconds]
ashark has quit [Ping timeout: 240 seconds]
Poefke has joined #ipfs
newhouse has quit [Read error: Connection reset by peer]
jaboja has joined #ipfs
jnes has joined #ipfs
notdaniel has joined #ipfs
fredthomsen has joined #ipfs
Anton_ has quit [Ping timeout: 248 seconds]
Anton_ has joined #ipfs
ashark has joined #ipfs
fredthomsen has quit [Quit: Leaving.]
ashark has quit [Ping timeout: 260 seconds]
vsimon has quit [Remote host closed the connection]
<Renich>
MikeFair: yep, that worked. The site was updated. Just testing, though. But the request time sucks for now... no idea why
<MikeFair>
Renich: I noticed the same problem with request time when I was testing a year ago
domanic has quit [Ping timeout: 264 seconds]
<Renich>
how did you love it?
<MikeFair>
I had a few challenges, among them was the fact ipns would purge out every 24 hours (and I didn't know that was by design at the time)
<MikeFair>
I preferred using the pubkey based ipns publishing over using the local peer node's address which made the commands more cumbersome
notdaniel has quit [Quit: Leaving]
<MikeFair>
And I quickly figured out that publishing any kind of web API was going to require a rethink
<MikeFair>
(like letting user's log in, or use a web form)
paralin has left #ipfs ["Using Circe, the loveliest of all IRC clients"]
paralin has joined #ipfs
<MikeFair>
So a huge +1 on ipfs based browser download; but I needed some extra love to get where I wanted to go
<MikeFair>
For example, it just occurred to me that using PubSub/OrbitDB might be a way to get client sites connected to a "backend"
}ls{ has quit [Ping timeout: 240 seconds]
<Renich>
cool
<MikeFair>
err client browsers (not sites)
<Renich>
My test site is just html... and very basic. I just think that almost 2 minutes of request time is ridiculous. I must have something wrong somewhere
<MikeFair>
I used to think that too; but I don't think so
<MikeFair>
It seems to be after updating an existing ipns entry
<MikeFair>
For instance, a few moments ago I was able to pull your test.woralelandia.com almost instantly, now it took almost a minute
}ls{ has joined #ipfs
robattila256 has quit [Ping timeout: 264 seconds]
robattila256 has joined #ipfs
ashark has joined #ipfs
ashark has quit [Ping timeout: 260 seconds]
dandevelo has joined #ipfs
paralin has quit [Ping timeout: 264 seconds]
rodolf0 has joined #ipfs
lupine has quit [Remote host closed the connection]
paralin has joined #ipfs
dandevelo has quit [Ping timeout: 240 seconds]
<Renich>
can I mount some peer's err... objects?
<Renich>
For example, a friend and I share peer hashes. How do we share objects?
lupine has joined #ipfs
paralin has quit [Remote host closed the connection]
<MikeFair>
not sure the distinction
paralin has joined #ipfs
<MikeFair>
A simple method is to put the object data in a file, put the file in ipfs, the CID = the object id
rodolf0 has quit [Ping timeout: 268 seconds]
<MikeFair>
If you want to have distributed editing over a common repository of data, that's a bit different
<MikeFair>
First off is to really grind it into your brain that the "state" of the object defines its name
<MikeFair>
You change one byte, you change its name
<MikeFair>
(aka its CID)
<MikeFair>
It's Content Addressed Storage; and it means it
<MikeFair>
If however you just want to locally copy from of their data, you can pin their content to your local node
<MikeFair>
s/from/some/
letmutx has joined #ipfs
wking_ is now known as wking
ashark has joined #ipfs
l2d has quit [Remote host closed the connection]
l2d has joined #ipfs
<gozala>
lgierth: tnx, will read about IPRS
ashark has quit [Ping timeout: 240 seconds]
Neomex has joined #ipfs
Ellenor is now known as Reinhilde
chriscool1 has joined #ipfs
<haad>
MikeFair: o/ thanks for the ping on the couchdb<-->orbitdb issue. I'll get back to properly at some point but the short is, I just haven't had the time to work on it and it hasn't been a priority :/ we're in an agreement what it *should* be though :)
<haad>
gozala: it sounds to me https://github.com/orbitdb/ipfs-log might help you achieve what you're trying to do. it doesn't handle message propagation (ie. PubSub) and if you need more "out-of-the-box" tool, OrbitDB might be worth checking
<haad>
that said, ymmv and there's a lot happening in IPFS atm so there's plenty of other options too :)
* MikeFair
shrugs. "Not sure I can say; I don't really use the JS API"
<codedread>
for instance, in a bit torrent client, I can see how big the file is, which chunks are present and which chunks need to be transferred
<MikeFair>
Well, I'm running it now and I can see chrome is receiving !57/sec
<MikeFair>
err 57k/sec
dandevelo has quit [Ping timeout: 265 seconds]
<codedread>
how are you seeing that?
<MikeFair>
performance monitor
<MikeFair>
network utilization
ashark has joined #ipfs
<MikeFair>
At 57k/sec it'll take ~25 minutes to process 87MB
ashark has quit [Ping timeout: 260 seconds]
nob has quit [Remote host closed the connection]
Betsy has joined #ipfs
astrofog has joined #ipfs
chriscool1 has joined #ipfs
<JCaesar>
Hm. I have a directory with text files that all come with a fairly big common header or footer. Is there a way to intelligently chunk that?
cherry-xw has joined #ipfs
cherry-xw has quit [Client Quit]
cherry-xw has joined #ipfs
KraemeReinharD[m has joined #ipfs
cherry-xw has quit [Client Quit]
AkhILman has joined #ipfs
hipboi__ has quit [Ping timeout: 248 seconds]
hipboi__ has joined #ipfs
lemmi has quit [Ping timeout: 248 seconds]
trqx has quit [Ping timeout: 272 seconds]
AndrewMalik is now known as AndreaMalik
ashark has joined #ipfs
aw- has joined #ipfs
McJuicy has quit [Quit: Pce]
ashark has quit [Ping timeout: 240 seconds]
aw- has left #ipfs [#ipfs]
MaybeDragon has joined #ipfs
lemmi has joined #ipfs
chriscool1 has quit [Ping timeout: 248 seconds]
dandevelo has joined #ipfs
hipboi__ has quit [Ping timeout: 256 seconds]
codedread has quit [Quit: Page closed]
zautomata has quit [Ping timeout: 265 seconds]
trqx has joined #ipfs
Betsy has quit [Ping timeout: 260 seconds]
McJuicy has joined #ipfs
dandevelo has quit [Ping timeout: 240 seconds]
letmutx has quit [Quit: Connection closed for inactivity]
leeola has quit [Quit: Connection closed for inactivity]
ashark has joined #ipfs
ashark has quit [Ping timeout: 256 seconds]
<MikeFair>
JCaesar: In what way?
<MikeFair>
If you made a header and footer as separate files and deleted the contents from the rest of the files it will store more compactly
<JCaesar>
Duh.
<JCaesar>
I'd like: the header gets stuffed into a chunk, the footer gets stuffed into a chunk, a file consists of the common header block, a (small) content block, and the common footer block.
<MikeFair>
But unless i'm misunderstanding, you really won't be able to get a resuable CID entry for the header, and especially not the footer (alignment issues)
<JCaesar>
So I can't just concat blocks together to make a file?
<MikeFair>
Blocks are of a fixed size
<MikeFair>
So you can, but they have to be blocksize aligned
<MikeFair>
(again, unless I misunderstand how it works; not an expert for sure)
<MikeFair>
Then I believe you'd also have to construct hte DHT entries by hand
<JCaesar>
Hm. I thought that rabin chunker would create variably sized chunks… :/
<MikeFair>
But isn't that just within the same file?
<MikeFair>
(repitive content within the same file)
<MikeFair>
Like if you tarred up all the files but didn't zip them
<MikeFair>
then the header and footers would be repeating patterns within the same file
<JCaesar>
Yeah, but that means that they're variably sized and re-used at different offsets. So what I want should work, at least in theory.
<MikeFair>
Right, but I don't see how/why the algorithm would detect to chunk the header/footer across multiple files at those points
<MikeFair>
Since there's a lack of a pattern I assumed it'd just stuff as much as possible into a block
<MikeFair>
(creating block alignment issues)
letmutx has joined #ipfs
<JCaesar>
I assume "the algorithm" isn't written yet, at least not for IPFS…
aw- has joined #ipfs
<MikeFair>
fair enough
<MikeFair>
I could see a use case for a "tar first" parameter which increases the opportunity for cross file resuability
<JCaesar>
I'd like the thing to be browsable…
<MikeFair>
The DHT?
<MikeFair>
It kind of is
<JCaesar>
the files. I'd like them to stay as separate files.
<MikeFair>
oh you mean on my "flag"; yeah I did mean it'd actually store it as a tar file
<MikeFair>
err didn't
aw- has left #ipfs [#ipfs]
<MikeFair>
Just that it'd construct the files "as if" they were part of a single file archive
<MikeFair>
So the likelihood of finding the repeating data sections across multiple files increases
<MikeFair>
Frankly I think the CIDs likely take up more space than they save doing sub-file repeat finding though (except in special isolated cases)
cherry-xw has joined #ipfs
<JCaesar>
That's why --raw-leaves exists, no? Or did I miss something?
robattila256 has quit [Ping timeout: 248 seconds]
cherry-xw has quit [Client Quit]
<MikeFair>
Could be, but that doesn't exactly address what I'm saying. It takes 32 bytes to point at the CID (you need the leaf entry in the DAG)
AndreaMalik is now known as Ellenor
astrofog has quit [Quit: Quite]
<MikeFair>
It consumes 32 bytes in every file that uses that leaf
<MikeFair>
So if the leaf contains 32 bytes or less of "reusable data" it doesn't actually save space
robattila256 has joined #ipfs
<JCaesar>
Yes. These headers are a few kb. For 30k files.
<MikeFair>
Then there's the "wasted space" in other blocks; while I believe the block can store variable sized chucks the node in the DAG is fixed size
cherry-xw has joined #ipfs
<MikeFair>
(with the difference between chunk size and DAG block size being "wasted")
cherry-xw has quit [Remote host closed the connection]
cherry-xw has joined #ipfs
<MikeFair>
So when it's all taken together I'm guessing the cases where we can get a win for misaligned shared data across separate files is kind of low
cherry-xw has quit [Read error: Connection reset by peer]
<MikeFair>
Unless there's some kind of special helper algorithm for storing particular kinds of files
<MikeFair>
I suppose there could be a published "favorite hashes" list that the add algorithm could consider... A list of file chunks that exist in the network expected to hit a higher intrafile hit ratio....
droman has joined #ipfs
vsimon has quit [Remote host closed the connection]
vsimon has joined #ipfs
girlhood has joined #ipfs
VitalyLipatov[m] has quit [Quit: idle on matrix for more than 30 days]
jebdj[m] has quit [Quit: idle on matrix for more than 30 days]
synth[m] has quit [Quit: idle on matrix for more than 30 days]
kkr[m] has quit [Quit: idle on matrix for more than 30 days]
pacharanero[m] has quit [Quit: idle on matrix for more than 30 days]
aquentson[m] has quit [Quit: idle on matrix for more than 30 days]
joose[m] has quit [Quit: idle on matrix for more than 30 days]
johngrimes[m] has quit [Quit: idle on matrix for more than 30 days]
matrixjc[m] has quit [Quit: idle on matrix for more than 30 days]
abhii_n[m] has quit [Quit: idle on matrix for more than 30 days]
timthelion[m] has quit [Quit: idle on matrix for more than 30 days]
Ferdinandhi[m] has quit [Quit: idle on matrix for more than 30 days]
Gryffins[m] has quit [Quit: idle on matrix for more than 30 days]
gubbes[m] has quit [Quit: idle on matrix for more than 30 days]
LustTeufel69[m] has quit [Quit: idle on matrix for more than 30 days]
paranoix[m] has quit [Quit: idle on matrix for more than 30 days]
AkhILman1 has left #ipfs ["Kicked by @appservice-irc:matrix.org : idle on matrix for more than 30 days"]
ElWood[m] has quit [Quit: idle on matrix for more than 30 days]
nonce911[m] has quit [Quit: idle on matrix for more than 30 days]
Jico[m] has quit [Quit: idle on matrix for more than 30 days]
Woef[m] has quit [Quit: idle on matrix for more than 30 days]
gzgy[m] has quit [Quit: idle on matrix for more than 30 days]
garikson[m] has quit [Quit: idle on matrix for more than 30 days]
hueso[m] has quit [Quit: idle on matrix for more than 30 days]
Dirtatron[m] has quit [Quit: idle on matrix for more than 30 days]
Jarekpl[m] has quit [Quit: idle on matrix for more than 30 days]
t9840[m] has quit [Quit: idle on matrix for more than 30 days]
signalkraft[m] has quit [Quit: idle on matrix for more than 30 days]
M-oddvar has quit [Quit: idle on matrix for more than 30 days]
ZeroMe[m] has quit [Quit: idle on matrix for more than 30 days]
Choyes[m] has quit [Quit: idle on matrix for more than 30 days]
ctresc[m] has quit [Quit: idle on matrix for more than 30 days]
jaredthirsk[m] has quit [Quit: idle on matrix for more than 30 days]
Bob[m]3 has quit [Quit: idle on matrix for more than 30 days]
kants[m] has quit [Quit: idle on matrix for more than 30 days]
ZerataX[m]1 has quit [Quit: idle on matrix for more than 30 days]
cnygaard[m] has quit [Quit: idle on matrix for more than 30 days]
xelra[m] has left #ipfs ["Kicked by @appservice-irc:matrix.org : idle on matrix for more than 30 days"]
jpf137[m] has quit [Quit: idle on matrix for more than 30 days]
elmago[m] has quit [Quit: idle on matrix for more than 30 days]
trust-tech[m] has quit [Quit: idle on matrix for more than 30 days]
janke99[m] has quit [Quit: idle on matrix for more than 30 days]
houyunlong[m] has quit [Quit: idle on matrix for more than 30 days]
M-3630 has quit [Quit: idle on matrix for more than 30 days]
auditcranium[m] has quit [Quit: idle on matrix for more than 30 days]
k37[m] has quit [Quit: idle on matrix for more than 30 days]
Lucas[m] has quit [Quit: idle on matrix for more than 30 days]
Guest307124[m] has quit [Quit: idle on matrix for more than 30 days]
Riot_User_123[m] has quit [Quit: idle on matrix for more than 30 days]
stech[m] has quit [Quit: idle on matrix for more than 30 days]
cutie has quit [Quit: idle on matrix for more than 30 days]
harlock[m] has quit [Quit: idle on matrix for more than 30 days]
tombusby has quit [Ping timeout: 272 seconds]
chenzhengyi[m] has quit [Quit: idle on matrix for more than 30 days]
asyncsec[m] has quit [Quit: idle on matrix for more than 30 days]
equaliser[m] has quit [Quit: idle on matrix for more than 30 days]
chenlee1116[m] has left #ipfs ["Kicked by @appservice-irc:matrix.org : idle on matrix for more than 30 days"]
JEFFJ[m] has quit [Quit: idle on matrix for more than 30 days]
gobelin[m] has quit [Quit: idle on matrix for more than 30 days]
Peter[m]2 has quit [Quit: idle on matrix for more than 30 days]
TheWhisper[m] has quit [Quit: idle on matrix for more than 30 days]
Guest198477[m] has quit [Quit: idle on matrix for more than 30 days]
tombusby has joined #ipfs
patto_overflow[m has quit [Quit: idle on matrix for more than 30 days]
atenorio[m] has quit [Quit: idle on matrix for more than 30 days]
soberao[m] has quit [Quit: idle on matrix for more than 30 days]
Ineentho[m] has quit [Quit: idle on matrix for more than 30 days]
Xiang[m] has quit [Quit: idle on matrix for more than 30 days]
redviking[m] has quit [Quit: idle on matrix for more than 30 days]
egalano[m] has quit [Quit: idle on matrix for more than 30 days]
callil[m] has quit [Quit: idle on matrix for more than 30 days]
adam123[m] has quit [Quit: idle on matrix for more than 30 days]
M0spark[m] has quit [Quit: idle on matrix for more than 30 days]
M-amdtel has quit [Quit: idle on matrix for more than 30 days]
rederekt[m] has quit [Quit: idle on matrix for more than 30 days]
huaixiaoxiao[m] has quit [Quit: idle on matrix for more than 30 days]
christopherdebee has quit [Quit: idle on matrix for more than 30 days]
Encrypt has joined #ipfs
M6AA4FD[m] has quit [Quit: idle on matrix for more than 30 days]
gvimrh[m] has quit [Quit: idle on matrix for more than 30 days]
remediface[m] has quit [Quit: idle on matrix for more than 30 days]
AntoineM[m] has quit [Quit: idle on matrix for more than 30 days]
mxmaiastone[m] has quit [Quit: idle on matrix for more than 30 days]
kantengri[m] has quit [Quit: idle on matrix for more than 30 days]
genr8r[m] has quit [Quit: idle on matrix for more than 30 days]
Tectest123[m] has quit [Quit: idle on matrix for more than 30 days]
fti7[m] has quit [Quit: idle on matrix for more than 30 days]
nobot4[m] has quit [Quit: idle on matrix for more than 30 days]
HerroKitty[m] has quit [Quit: idle on matrix for more than 30 days]
alldigital[m] has quit [Quit: idle on matrix for more than 30 days]
huda[m] has quit [Quit: idle on matrix for more than 30 days]
wazaundtechnik[m has quit [Quit: idle on matrix for more than 30 days]
adampaigge[m]1 has quit [Quit: idle on matrix for more than 30 days]
Conspiracy[m] has quit [Quit: idle on matrix for more than 30 days]
waz2[m] has quit [Quit: idle on matrix for more than 30 days]
q3y4r[m] has quit [Quit: idle on matrix for more than 30 days]
Xanza[m] has quit [Quit: idle on matrix for more than 30 days]
clunk-dike[m] has quit [Quit: idle on matrix for more than 30 days]
peakj[m] has quit [Quit: idle on matrix for more than 30 days]
augoeides[m] has left #ipfs ["Kicked by @appservice-irc:matrix.org : idle on matrix for more than 30 days"]
NathanBraswell[m has quit [Quit: idle on matrix for more than 30 days]
chevdor[m] has quit [Quit: idle on matrix for more than 30 days]
kyoo[m] has quit [Quit: idle on matrix for more than 30 days]
mmy[m] has quit [Quit: idle on matrix for more than 30 days]
addd34[m] has quit [Quit: idle on matrix for more than 30 days]
JuliusCaesar[m] has quit [Quit: idle on matrix for more than 30 days]
homulilly[m] has quit [Quit: idle on matrix for more than 30 days]
aekym[m] has quit [Quit: idle on matrix for more than 30 days]
spoonless[m]1 has quit [Quit: idle on matrix for more than 30 days]
dryajov[m] has quit [Quit: idle on matrix for more than 30 days]
nine-h[m] has quit [Quit: idle on matrix for more than 30 days]
Cargo[m] has quit [Quit: idle on matrix for more than 30 days]
testimonlkjd[m] has quit [Quit: idle on matrix for more than 30 days]
wsdookadr[m] has quit [Quit: idle on matrix for more than 30 days]
leonardo77[m] has quit [Quit: idle on matrix for more than 30 days]
rubencordeiro[m] has quit [Quit: idle on matrix for more than 30 days]
Sonata has quit [Quit: idle on matrix for more than 30 days]
daneroo[m] has quit [Quit: idle on matrix for more than 30 days]
PowerOlive[m] has quit [Quit: idle on matrix for more than 30 days]
random_user[m] has quit [Quit: idle on matrix for more than 30 days]
rhizo[m] has quit [Quit: idle on matrix for more than 30 days]
MrAxilus[m] has quit [Quit: idle on matrix for more than 30 days]
thinkmassive[m] has quit [Quit: idle on matrix for more than 30 days]
jsyn[m] has quit [Quit: idle on matrix for more than 30 days]
gugyygy00[m] has quit [Quit: idle on matrix for more than 30 days]
Novitsh[m] has quit [Quit: idle on matrix for more than 30 days]
alphard[m] has quit [Quit: idle on matrix for more than 30 days]
dopeman[m] has quit [Quit: idle on matrix for more than 30 days]
robertomurta[m] has quit [Quit: idle on matrix for more than 30 days]
jux[m] has quit [Quit: idle on matrix for more than 30 days]
ztl8702[m] has quit [Quit: idle on matrix for more than 30 days]
sizeofcat[m] has quit [Quit: idle on matrix for more than 30 days]
yodata[m] has quit [Quit: idle on matrix for more than 30 days]
belie[m] has quit [Quit: idle on matrix for more than 30 days]
omomom[m] has left #ipfs ["Kicked by @appservice-irc:matrix.org : idle on matrix for more than 30 days"]
paralin has quit [Ping timeout: 248 seconds]
Armagedda[m] has quit [Quit: idle on matrix for more than 30 days]
ManTou[m] has quit [Quit: idle on matrix for more than 30 days]
bilowan[m] has quit [Quit: idle on matrix for more than 30 days]
liaowu[m] has quit [Quit: idle on matrix for more than 30 days]
anomie____[m] has quit [Quit: idle on matrix for more than 30 days]
trouty[m] has quit [Quit: idle on matrix for more than 30 days]
chriscool1 has joined #ipfs
jsnng[m] has quit [Quit: idle on matrix for more than 30 days]
bitgulf[m] has quit [Quit: idle on matrix for more than 30 days]
OrangeOrange[m] has quit [Quit: idle on matrix for more than 30 days]
RuanMedGar[m] has quit [Quit: idle on matrix for more than 30 days]
ricburton[m] has quit [Quit: idle on matrix for more than 30 days]
hoomx[m] has quit [Quit: idle on matrix for more than 30 days]
Jacomo[m] has quit [Quit: idle on matrix for more than 30 days]
johnwhitlow[m] has quit [Quit: idle on matrix for more than 30 days]
yuh96bbn[m] has quit [Quit: idle on matrix for more than 30 days]
laptop_nativ[m] has quit [Quit: idle on matrix for more than 30 days]
juanbernabo[m] has quit [Quit: idle on matrix for more than 30 days]
DavidP[m] has quit [Quit: idle on matrix for more than 30 days]
edejong[m] has quit [Quit: idle on matrix for more than 30 days]
hajak[m] has quit [Quit: idle on matrix for more than 30 days]
crapman[m] has quit [Quit: idle on matrix for more than 30 days]
CryptoPunkin[m] has quit [Quit: idle on matrix for more than 30 days]
gylpm[m] has quit [Quit: idle on matrix for more than 30 days]
xiaonanwen[m] has quit [Quit: idle on matrix for more than 30 days]
Andrew[m]1 has quit [Quit: idle on matrix for more than 30 days]
SethDavis[m] has quit [Quit: idle on matrix for more than 30 days]
BobWhite[m] has quit [Quit: idle on matrix for more than 30 days]
metbit[m] has quit [Quit: idle on matrix for more than 30 days]
MawKKe[m] has quit [Quit: idle on matrix for more than 30 days]
M-kalmi has quit [Quit: idle on matrix for more than 30 days]
Rmesil8O4b[m] has quit [Quit: idle on matrix for more than 30 days]
zhikin[m] has quit [Quit: idle on matrix for more than 30 days]
roderik[m] has quit [Quit: idle on matrix for more than 30 days]
TempMan[m] has quit [Quit: idle on matrix for more than 30 days]
confusedrobot[m] has quit [Quit: idle on matrix for more than 30 days]
stapper[m] has quit [Quit: idle on matrix for more than 30 days]
thelonelyghost has quit [Quit: idle on matrix for more than 30 days]
zhangyijun[m] has quit [Quit: idle on matrix for more than 30 days]
zie[m] has quit [Quit: idle on matrix for more than 30 days]
linux8bit[m] has quit [Quit: idle on matrix for more than 30 days]
TFX[m] has quit [Quit: idle on matrix for more than 30 days]
adamierymenko[m7 has quit [Quit: idle on matrix for more than 30 days]
snow-frog[m] has quit [Quit: idle on matrix for more than 30 days]
shepner[m] has quit [Quit: idle on matrix for more than 30 days]
bauer1j[m] has quit [Quit: idle on matrix for more than 30 days]
unlmtd has quit [Quit: idle on matrix for more than 30 days]
frelars[m] has quit [Quit: idle on matrix for more than 30 days]
sbdbd[m] has quit [Quit: idle on matrix for more than 30 days]
wktian[m] has quit [Quit: idle on matrix for more than 30 days]
bananabread[m] has quit [Quit: idle on matrix for more than 30 days]
cyberpepe[m] has quit [Quit: idle on matrix for more than 30 days]
MaybeDragon has quit [Remote host closed the connection]
tehciolo[m] has quit [Quit: idle on matrix for more than 30 days]
Aerielli[m] has quit [Quit: idle on matrix for more than 30 days]
landakram[m] has quit [Quit: idle on matrix for more than 30 days]
machete[m] has quit [Quit: idle on matrix for more than 30 days]
IsaacDunlap[m] has quit [Quit: idle on matrix for more than 30 days]
kvakes[m] has quit [Quit: idle on matrix for more than 30 days]
jig[m] has quit [Quit: idle on matrix for more than 30 days]
Phyra[m] has quit [Quit: idle on matrix for more than 30 days]
jeremycahill[m] has quit [Quit: idle on matrix for more than 30 days]
testuser54[m] has quit [Quit: idle on matrix for more than 30 days]
hollowzen[m] has quit [Quit: idle on matrix for more than 30 days]
M-gavinjohnnoona has quit [Quit: idle on matrix for more than 30 days]
dandevelo has joined #ipfs
Guest218640[m] has quit [Quit: idle on matrix for more than 30 days]
Chris[m]4 has quit [Quit: idle on matrix for more than 30 days]
Hongar[m] has quit [Quit: idle on matrix for more than 30 days]
M107262[m] has quit [Quit: idle on matrix for more than 30 days]
zaengerlein[m] has quit [Quit: idle on matrix for more than 30 days]
hilariousandorig has quit [Quit: idle on matrix for more than 30 days]
anon2[m] has quit [Quit: idle on matrix for more than 30 days]
coldsauce[m] has left #ipfs ["Kicked by @appservice-irc:matrix.org : idle on matrix for more than 30 days"]
mave99a[m] has quit [Quit: idle on matrix for more than 30 days]
nop[m] has quit [Quit: idle on matrix for more than 30 days]
aronvanammers[m] has quit [Quit: idle on matrix for more than 30 days]
ash[m] has quit [Quit: idle on matrix for more than 30 days]
Fess has quit [Ping timeout: 255 seconds]
shabeeraly[m] has quit [Quit: idle on matrix for more than 30 days]
TheGillies has quit [Quit: idle on matrix for more than 30 days]
mikecohen[m] has quit [Quit: idle on matrix for more than 30 days]
snsnsbsb[m] has quit [Quit: idle on matrix for more than 30 days]
jiawei[m] has quit [Quit: idle on matrix for more than 30 days]
a123123123[m] has left #ipfs ["Kicked by @appservice-irc:matrix.org : idle on matrix for more than 30 days"]
mattylar12[m] has quit [Quit: idle on matrix for more than 30 days]
aikidouke has left #ipfs ["Kicked by @appservice-irc:matrix.org : idle on matrix for more than 30 days"]
GuillaumeREMBERT has quit [Quit: idle on matrix for more than 30 days]
ArthurNorthwode[ has quit [Quit: idle on matrix for more than 30 days]
nullptr_[m] has quit [Quit: idle on matrix for more than 30 days]
dreuter[m] has quit [Quit: idle on matrix for more than 30 days]
Olinkl[m] has quit [Quit: idle on matrix for more than 30 days]
at2500[m] has quit [Quit: idle on matrix for more than 30 days]
jjw[m] has quit [Quit: idle on matrix for more than 30 days]
homelessrobot[m] has quit [Quit: idle on matrix for more than 30 days]
Rad[m] has quit [Quit: idle on matrix for more than 30 days]
aravindrs[m] has quit [Quit: idle on matrix for more than 30 days]
cyolovin[m] has quit [Quit: idle on matrix for more than 30 days]
shellkr[m] has quit [Quit: idle on matrix for more than 30 days]
StackerRobot[m] has quit [Quit: idle on matrix for more than 30 days]
j24[m] has quit [Quit: idle on matrix for more than 30 days]
halle[m] has quit [Quit: idle on matrix for more than 30 days]
polytomous[m] has quit [Quit: idle on matrix for more than 30 days]
dandevelo has quit [Ping timeout: 246 seconds]
redsteakraw[m] has quit [Quit: idle on matrix for more than 30 days]
obi[m] has quit [Quit: idle on matrix for more than 30 days]
carbonosity[m] has quit [Quit: idle on matrix for more than 30 days]
pik[m] has quit [Quit: idle on matrix for more than 30 days]
cubemonkey[m] has quit [Quit: idle on matrix for more than 30 days]
Kinch[m] has quit [Quit: idle on matrix for more than 30 days]
hypercube[m] has quit [Quit: idle on matrix for more than 30 days]
TheKysek[m] has quit [Quit: idle on matrix for more than 30 days]
lisky[m] has quit [Quit: idle on matrix for more than 30 days]
xiaolai[m] has quit [Quit: idle on matrix for more than 30 days]
willhite[m] has quit [Quit: idle on matrix for more than 30 days]
VegBerg[m] has quit [Quit: idle on matrix for more than 30 days]
pacman99[m] has quit [Quit: idle on matrix for more than 30 days]
GotoDengo[m] has quit [Quit: idle on matrix for more than 30 days]
kurtis[m] has quit [Quit: idle on matrix for more than 30 days]
stevetbm[m] has quit [Quit: idle on matrix for more than 30 days]
M-mlt has quit [Quit: idle on matrix for more than 30 days]
solatis[m] has quit [Quit: idle on matrix for more than 30 days]
laosijijulebu[m] has quit [Quit: idle on matrix for more than 30 days]
Geeky[m] has quit [Quit: idle on matrix for more than 30 days]
Lin[m] has quit [Quit: idle on matrix for more than 30 days]
fiber[m] has quit [Quit: idle on matrix for more than 30 days]
leerspace has quit [Quit: idle on matrix for more than 30 days]
encns[m] has quit [Quit: idle on matrix for more than 30 days]
visim[m] has quit [Quit: idle on matrix for more than 30 days]
rubenmcosta[m] has quit [Quit: idle on matrix for more than 30 days]
xfhg[m] has quit [Quit: idle on matrix for more than 30 days]
moellus[m] has quit [Quit: idle on matrix for more than 30 days]
M-jfred has quit [Quit: idle on matrix for more than 30 days]
melanargia[m] has quit [Quit: idle on matrix for more than 30 days]
d-0504[m] has quit [Quit: idle on matrix for more than 30 days]
M[m]19 has quit [Quit: idle on matrix for more than 30 days]
kobajagi[m] has quit [Quit: idle on matrix for more than 30 days]
Antharia[m] has quit [Quit: idle on matrix for more than 30 days]
RichardCranium[m has quit [Quit: idle on matrix for more than 30 days]
nimtiazm[m] has quit [Quit: idle on matrix for more than 30 days]
RedFire[m] has quit [Quit: idle on matrix for more than 30 days]
gulimu[m] has quit [Quit: idle on matrix for more than 30 days]
KillaZA[m] has quit [Quit: idle on matrix for more than 30 days]
deepmind[m] has quit [Quit: idle on matrix for more than 30 days]
MariaP[m] has quit [Quit: idle on matrix for more than 30 days]
chatmasta[m] has quit [Quit: idle on matrix for more than 30 days]
ravita1967[m] has quit [Quit: idle on matrix for more than 30 days]
iarebatman[m] has quit [Quit: idle on matrix for more than 30 days]
henri_nourel[m] has quit [Quit: idle on matrix for more than 30 days]
Anastiel[m] has quit [Quit: idle on matrix for more than 30 days]
followers[m] has left #ipfs ["Kicked by @appservice-irc:matrix.org : idle on matrix for more than 30 days"]
hanwudizz[m] has quit [Quit: idle on matrix for more than 30 days]
Orpugi[m] has quit [Quit: idle on matrix for more than 30 days]
M3v[m] has quit [Quit: idle on matrix for more than 30 days]
tx[m] has quit [Quit: idle on matrix for more than 30 days]
mrpx[m] has quit [Quit: idle on matrix for more than 30 days]
imtoken[m] has quit [Quit: idle on matrix for more than 30 days]
dwesima[m] has quit [Quit: idle on matrix for more than 30 days]
Vaultpls[m] has quit [Quit: idle on matrix for more than 30 days]
plexigras has quit [Ping timeout: 252 seconds]
TrickOrTreat[m] has quit [Quit: idle on matrix for more than 30 days]
gragotta68[m] has left #ipfs ["Kicked by @appservice-irc:matrix.org : idle on matrix for more than 30 days"]
kostya[m] has quit [Quit: idle on matrix for more than 30 days]
shirian[m] has quit [Quit: idle on matrix for more than 30 days]
DiCE1904[m] has quit [Quit: idle on matrix for more than 30 days]
mike_new[m] has quit [Quit: idle on matrix for more than 30 days]
firemound[m] has quit [Quit: idle on matrix for more than 30 days]
risingsun2100[m] has quit [Quit: idle on matrix for more than 30 days]
fightentropy[m] has quit [Quit: idle on matrix for more than 30 days]
chae4aish9[m] has quit [Quit: idle on matrix for more than 30 days]
stevendai[m] has quit [Quit: idle on matrix for more than 30 days]
wjanklow[m] has quit [Quit: idle on matrix for more than 30 days]
M[m]15 has quit [Quit: idle on matrix for more than 30 days]
fhs[m] has quit [Quit: idle on matrix for more than 30 days]
vihfvuuv[m] has quit [Quit: idle on matrix for more than 30 days]
plexigras has joined #ipfs
johanherman[m] has quit [Quit: idle on matrix for more than 30 days]
Sabrina[m] has quit [Quit: idle on matrix for more than 30 days]
yellowbelly[m] has quit [Quit: idle on matrix for more than 30 days]
wanglei0095[m] has quit [Quit: idle on matrix for more than 30 days]
mirek1337[m] has quit [Quit: idle on matrix for more than 30 days]
nedzadhrnjica[m4 has quit [Quit: idle on matrix for more than 30 days]
cybervking[m] has quit [Quit: idle on matrix for more than 30 days]
R1911[m] has quit [Quit: idle on matrix for more than 30 days]
M-trashrabbit has quit [Quit: idle on matrix for more than 30 days]
ltq520[m] has quit [Quit: idle on matrix for more than 30 days]
screamerbeetle[m has quit [Quit: idle on matrix for more than 30 days]
aaronye[m] has quit [Quit: idle on matrix for more than 30 days]
berot3[m] has quit [Quit: idle on matrix for more than 30 days]
sk23[m] has quit [Quit: idle on matrix for more than 30 days]
jaybee[m]1 has quit [Quit: idle on matrix for more than 30 days]
marcusdunlop[m] has quit [Quit: idle on matrix for more than 30 days]
qben[m] has quit [Quit: idle on matrix for more than 30 days]
balhd[m] has quit [Quit: idle on matrix for more than 30 days]
sanday[m]1 has left #ipfs ["Kicked by @appservice-irc:matrix.org : idle on matrix for more than 30 days"]
astronavt[m] has quit [Quit: idle on matrix for more than 30 days]
Chelovek[m] has quit [Quit: idle on matrix for more than 30 days]
slowtoaster[m] has quit [Quit: idle on matrix for more than 30 days]
conorohiggins[m] has quit [Quit: idle on matrix for more than 30 days]
zebburkeconte[m] has quit [Quit: idle on matrix for more than 30 days]
henriqueqc[m] has quit [Quit: idle on matrix for more than 30 days]
Mario[m]1 has quit [Quit: idle on matrix for more than 30 days]
warrenchan[m] has quit [Quit: idle on matrix for more than 30 days]
hyde_stevenson[m has quit [Quit: idle on matrix for more than 30 days]
blinkwang[m] has quit [Quit: idle on matrix for more than 30 days]
ipnoyl[m] has quit [Quit: idle on matrix for more than 30 days]
mntr[m] has quit [Quit: idle on matrix for more than 30 days]
yl[m] has quit [Quit: idle on matrix for more than 30 days]
benk[m] has quit [Quit: idle on matrix for more than 30 days]
gomarashi[m] has quit [Quit: idle on matrix for more than 30 days]
TheLugal[m] has quit [Quit: idle on matrix for more than 30 days]
bbcnetwork[m] has quit [Quit: idle on matrix for more than 30 days]
tzuntzai[m] has quit [Quit: idle on matrix for more than 30 days]
martinrusev[m] has quit [Quit: idle on matrix for more than 30 days]
mortn[m] has quit [Quit: idle on matrix for more than 30 days]
G16357300[m] has quit [Quit: idle on matrix for more than 30 days]
zekaf[m] has quit [Quit: idle on matrix for more than 30 days]
anon9876[m] has quit [Quit: idle on matrix for more than 30 days]
mrcrowly[m] has quit [Quit: idle on matrix for more than 30 days]
Walletor[m] has quit [Quit: idle on matrix for more than 30 days]
fzzr[m] has quit [Quit: idle on matrix for more than 30 days]
JZA has left #ipfs ["Konversation terminated!"]
halohaL[m] has quit [Quit: idle on matrix for more than 30 days]
fetlock_coco_fis has quit [Quit: idle on matrix for more than 30 days]
jessiedella[m] has quit [Quit: idle on matrix for more than 30 days]
vnadgir[m] has quit [Quit: idle on matrix for more than 30 days]
M[m]20 has quit [Quit: idle on matrix for more than 30 days]
SeaSon[m] has quit [Quit: idle on matrix for more than 30 days]
miso[m] has quit [Quit: idle on matrix for more than 30 days]
M7865nfnjjfjf[m] has quit [Quit: idle on matrix for more than 30 days]
rasmuserik[m] has quit [Quit: idle on matrix for more than 30 days]
mailtorajeshmr[m has quit [Quit: idle on matrix for more than 30 days]
Anthony[m] has quit [Quit: idle on matrix for more than 30 days]
caveboner[m] has quit [Quit: idle on matrix for more than 30 days]
mingray[m] has quit [Quit: idle on matrix for more than 30 days]
ot8[m] has quit [Quit: idle on matrix for more than 30 days]
testing_matrix47 has quit [Quit: idle on matrix for more than 30 days]
orenwf[m] has quit [Quit: idle on matrix for more than 30 days]
Atrus[m] has quit [Quit: idle on matrix for more than 30 days]
Symen[m]1 has quit [Quit: idle on matrix for more than 30 days]
MadcapJake[m] has quit [Quit: idle on matrix for more than 30 days]
josejuanisaac[m] has quit [Quit: idle on matrix for more than 30 days]
Luguber[m] has quit [Quit: idle on matrix for more than 30 days]
t[m] has quit [Quit: idle on matrix for more than 30 days]
doronbehar[m] has left #ipfs ["User left"]
H-Lo[m] has quit [Quit: idle on matrix for more than 30 days]
Encrypt has quit [Quit: Quit]
doronbehar[m] has joined #ipfs
droman has quit [Quit: WeeChat 2.0.1]
doronbehar[m] has left #ipfs ["User left"]
random_dork[m] has joined #ipfs
bomb-on has quit [Quit: SO LONG, SUCKERS!]
newhouse has joined #ipfs
MikeFair has quit [Ping timeout: 248 seconds]
girlhood has quit [Ping timeout: 260 seconds]
fredthomsen has joined #ipfs
newhouse has quit [Read error: Connection reset by peer]
muvlon has joined #ipfs
fredthomsen has quit [Client Quit]
letmutx has quit [Quit: Connection closed for inactivity]
Encrypt has joined #ipfs
dandevelo has joined #ipfs
jaboja has joined #ipfs
uzzrroOOOO has joined #ipfs
xnbya2 has quit [Ping timeout: 248 seconds]
xelra has quit [Remote host closed the connection]
xnbya has joined #ipfs
xelra has joined #ipfs
}ls{ has joined #ipfs
Vladislav has joined #ipfs
dudebbn has quit [Read error: Connection reset by peer]
dudebbn has joined #ipfs
doronbehar[m] has joined #ipfs
Neomex has quit [Remote host closed the connection]
shizy has joined #ipfs
jaboja has quit [Ping timeout: 276 seconds]
jaboja has joined #ipfs
jdevoo has joined #ipfs
jdevoo has quit [Client Quit]
shizy has quit [Ping timeout: 255 seconds]
jaboja has quit [Ping timeout: 256 seconds]
uzzrroOOOO has quit [Ping timeout: 240 seconds]
XgpcX has joined #ipfs
vflyson has quit [Ping timeout: 264 seconds]
wmoh has quit [Read error: Connection reset by peer]
reit has joined #ipfs
wmoh has joined #ipfs
vflyson has joined #ipfs
atrapado_ has joined #ipfs
whittymatt[m] has joined #ipfs
wmoh has quit [Read error: Connection reset by peer]
wmoh has joined #ipfs
randomstrangerb has quit [Ping timeout: 240 seconds]
randomstrangerb has joined #ipfs
vflyson has quit [Ping timeout: 265 seconds]
atrapado_ has quit [Quit: Leaving]
NullbutC00L has joined #ipfs
XgpcX_ has joined #ipfs
vflyson has joined #ipfs
XgpcX has quit [Ping timeout: 260 seconds]
wmoh has quit [Ping timeout: 248 seconds]
geir_ has quit [Ping timeout: 272 seconds]
wmoh has joined #ipfs
geir_ has joined #ipfs
jkilpatr has quit [Ping timeout: 256 seconds]
cwahlers_ has joined #ipfs
cwahlers has quit [Ping timeout: 255 seconds]
trqx has quit [Ping timeout: 272 seconds]
leeola has joined #ipfs
onabreak has quit [Ping timeout: 260 seconds]
Anton_ has quit [Ping timeout: 248 seconds]
jkilpatr has joined #ipfs
espadrine has quit [Ping timeout: 260 seconds]
Anton_ has joined #ipfs
trqx has joined #ipfs
ONI_Ghost has joined #ipfs
jkilpatr has quit [Ping timeout: 248 seconds]
<fiatjaf>
I'm trying to use libp2p for a small demo in which I connect to known peers and send text messages to them.
<fiatjaf>
I've tried dialing on them directly (using their b58 ids), but that failed because no transports between the two were available.
<fiatjaf>
if I listen for all the discovered peers, however, I get the peers I want to connect (I just check the id of the discovered peer against the list of known peers I have)
randomstrangerb has quit [Ping timeout: 268 seconds]
<fiatjaf>
(I'm running this on two browsers side by side on my computer)
randomstrangerb has joined #ipfs
<fiatjaf>
then I call .dial() and get a connection on the dialer's side. I proceed to write some text to the connection (using pull-streams as per the chat example) and nothing happens.
<fiatjaf>
on the listener's side, there's an event "peer:connect" when the dialer dials
<fiatjaf>
but then I get an error on .handle()
<fiatjaf>
the error says nothing, it is just the specified protocol name
<fiatjaf>
what can I do?
<fiatjaf>
oh, also, after that happens, the process continues indefinitely. the both sides keep getting the same list of peers discovered over and over, and both try to dial to each other, get the error, repeat forever.
<fiatjaf>
is that normal?
wmoh has quit [Read error: Connection reset by peer]
<fiatjaf>
(one correction: the "peer:connect" event is not always from the know peer trying to dial, it may be from any other peer, I guess -- that shouldn't be important)
jkilpatr has joined #ipfs
wmoh has joined #ipfs
<fiatjaf>
ok, I thought the callback to .handle() would get (err, conn), but actually it gets (protocolSignature, conn), so maybe everything is working and I don't know.
doronbehar[m] has left #ipfs ["User left"]
doronbehar[m] has joined #ipfs
rozie_ has quit [Quit: Lost terminal]
<fiatjaf>
yeah, everything was working besides these mistakes.
<fiatjaf>
ok, my problem remains, however, on the discovery side. why does it keep discovering peers, the same peers, over and over again? is that normal?
<fiatjaf>
should I trust the connection object I get on .dial() and on .handle() to last?
<fiatjaf>
why can't I just keep a list of peers and dial to them directly?
<fiatjaf>
how do I encrypt messages between peers?
<fiatjaf>
and, finally, why can't I read the public key of the automatically discovered peers?
NullbutC00L has quit [Ping timeout: 260 seconds]
ONI_Ghost has quit [Read error: Connection reset by peer]
rozie has joined #ipfs
ONI_Ghost has joined #ipfs
wmoh has quit [Read error: Connection reset by peer]
wmoh has joined #ipfs
wmoh has quit [Read error: Connection reset by peer]
wmoh has joined #ipfs
wmoh has quit [Read error: Connection reset by peer]
Encrypt has quit [Ping timeout: 246 seconds]
wmoh has joined #ipfs
wmoh has quit [Read error: Connection reset by peer]
paralin has joined #ipfs
wmoh has joined #ipfs
Fess has joined #ipfs
synthmeat has quit [Quit: WeeChat 2.0.1]
newhouse has joined #ipfs
synthmeat has joined #ipfs
clemo has joined #ipfs
Encrypt has joined #ipfs
kaotisk has quit [Read error: Connection reset by peer]
kaotisk has joined #ipfs
onabreak has joined #ipfs
ONI_Ghost has quit [Ping timeout: 264 seconds]
ONI_Ghost has joined #ipfs
ONI_Ghost has quit [Read error: Connection reset by peer]
jkilpatr has quit [Ping timeout: 248 seconds]
randomstrangerb has quit [Ping timeout: 265 seconds]
randomstrangerb has joined #ipfs
paralin has quit [Ping timeout: 252 seconds]
maxzor has joined #ipfs
Encrypt has quit [Quit: Quit]
raynold has quit [Quit: Connection closed for inactivity]
jkilpatr has joined #ipfs
ONI_Ghost has joined #ipfs
rcat has quit [Remote host closed the connection]
jkilpatr has quit [Excess Flood]
jkilpatr has joined #ipfs
newhouse has quit [Read error: Connection reset by peer]
jkilpatr has quit [Ping timeout: 248 seconds]
rendar has quit [Quit: std::lower_bound + std::less_equal *works* with a vector without duplicates!]
apiarian has quit [Read error: Connection reset by peer]
apiarian has joined #ipfs
<dryajov>
whyrusleeping: fiatjaf: I’m here now… taking a look
Encrypt has joined #ipfs
<dryajov>
fiatjaf: > ok, my problem remains, however, on the discovery side. why does it keep discovering peers, the same peers, over and over again? is that normal?
<dryajov>
not sure about this one… doesn’t sound right, but I need to double check the scenario - do you have a repo to reporoduce this?
robattila256 has quit [Ping timeout: 265 seconds]
robattila256 has joined #ipfs
maxzor_ has joined #ipfs
maxzor has quit [Ping timeout: 240 seconds]
XgpcX_ has quit [Ping timeout: 268 seconds]
ONI_Ghost has quit [Read error: Connection reset by peer]
ONI_Ghost has joined #ipfs
witten has quit [Ping timeout: 265 seconds]
MikeFair has joined #ipfs
NullbutC00L has quit [Ping timeout: 260 seconds]
kyanardag_ has quit [Ping timeout: 240 seconds]
kyanardag_ has joined #ipfs
witten has joined #ipfs
<fiatjaf>
I'll get the code into a state where you won't go mad by looking at it and publish somewhere
ONI_Ghost has quit [Read error: Connection reset by peer]
ONI_Ghost has joined #ipfs
<gozala>
haad: took another look at the orbit it does have a lot more docs now makes it a lot easier to grasp what it’s about
<gozala>
haad: that being said I’m not entirely sure in what context you suggested it. Is it for versioned sites stuff that I was talking about ?
<fiatjaf>
I don't know if it is ok to share my personal server URL here
<gozala>
MikeFair: lgierth IPRS does seem relevant although it’s yet another thing that I have hard time understanding how does it fit with rest of the ecosystem
<gozala>
with other being IPLD
<fiatjaf>
dryajov: what about the direct-dialing? on the chat example that is on the libp2p repo, for example, there's a direct dial, but that's on node.js. I'm trying on the browser. is that something?
ONI_Ghost has quit [Read error: No route to host]
Ronsor_ has quit [Ping timeout: 240 seconds]
<fiatjaf>
(please ignore the errors on https://three-noise.surge.sh/, just wait a little and the errors will stop appearing and you'll see just the logs for repeated peers being discovered over and over again)
<fiatjaf>
(also, sourcemaps are available on that page, so I hope it is good enough)
XgpcX_ has joined #ipfs
witten has quit [Ping timeout: 260 seconds]
Mateon3 has joined #ipfs
rodolf0 has joined #ipfs
webdev007 has quit [Quit: Leaving]
croll has joined #ipfs
rodolf0 has quit [Client Quit]
Ronsor has joined #ipfs
Mateon1 has quit [Ping timeout: 256 seconds]
Mateon3 is now known as Mateon1
Ronsor is now known as Guest95580
<MikeFair>
gozala: I suggested Orbit; and yes, it was for a database to store the revision history graph of the websites; however there might better solutions
<gozala>
MikeFair: oh I guess I got it mixed who mentioned it :)
<whyrusleeping>
gozala: IPRS is a system for being able to determine whether or not a given piece of information you have is valid
<whyrusleeping>
in the broad sense
<whyrusleeping>
we will be using it for naming (IPNS), but it can be used for many other things
<MikeFair>
gozala: I asked a similar question re IPRS vs IPLD; my understanding is IPRS is a specific record type descriptor where IPLD is more of a interoperable "structure"; lgierth put it much more eloquently (the impression I was left with was IPLD is more for interoperable and compatible implementations of a distributed linked db; IPRS is a specific recommendation for recording bytes to implement its described functions)
<whyrusleeping>
someone even was thinking about how you could define validity rules for an IPRS record that would end up creating a blockchain
<gozala>
whyrusleeping: yeah that’s more or less what I’d love to do I guess
wmoh has quit [Remote host closed the connection]
<gozala>
whyrusleeping: MikeFair What I meant by not understanding how pieces fall together is.
<MikeFair>
gozala: As a lead maintainer for Orbit, haad chimed in because we mentioned it earlier ;)
<gozala>
It seems to me that all these are low level building blocks and what I’m actually looking for is something as replacement for web apps
<whyrusleeping>
gozala: then what youre looking for is all the js-ipfs crdt stuff thats been going on
<whyrusleeping>
you've seen peerpad right?
<gozala>
whyrusleeping: yeah
<gozala>
whyrusleeping: let me try to rephrase it
<MikeFair>
gozala: So not the same thing as versioned website releases?
<gozala>
whyrusleeping: I really like how beaker browser utilizes dats for creating self-contained sites / apps
<gozala>
all of them have verifiable versioning
<gozala>
and only api exposed is to of to read / write dats
<gozala>
whyrusleeping: MikeFair consequence is, it’s really easy to put things togather
<gozala>
ande every dat:… site has things I would p2p web to have
<gozala>
one being versioning
<gozala>
which is to say you could look at the changelog of any site you visit
<gozala>
in fact then visit that specific version
<gozala>
if you pin / bookmark specific version
<MikeFair>
gozola: I think you're misunderstanding that beaker browser must be doing that by convention, not protoocl
<gozala>
it’s there forever for everyone in a state it was bookmarked
<gozala>
MikeFair: not really dat is build that way
<MikeFair>
(THis is the first I'm hearing of it)
wmoh1 has joined #ipfs
<gozala>
dat has ledger
<gozala>
MikeFair: let me send you link
<MikeFair>
gozala: The dat has a strucutre (a convention for a file format)
<whyrusleeping>
right, so they have a append only log built into the top level datastructure
<MikeFair>
gozala: It's as a layer on top of http, it's not inside http
<MikeFair>
gozala: You could accomplish the same simply by pubishing the dat inside ipfs and having your IPNS reference the dat file
<gozala>
MikeFair: dat isn’t using http it uses p2p networking as well
<MikeFair>
From the first line, they are saying SLEEP is an upgraded version of TAR
<gozala>
whyrusleeping: MikeFair anyway I’m not saying dat is perfect, it’s not there are certain things that I think are so much better in IPFS
<whyrusleeping>
gozala: I would be interested to define some conventions around a similar data structure for ipfs
<MikeFair>
the data in a register is a binary append only feed
<gozala>
whyrusleeping: what I was trying to say bundle of things like dat is viable alternative for building next web
<whyrusleeping>
gozala: yeah
<gozala>
whyrusleeping: with ipfs and friends it’s bag of things that you could build something like that
<whyrusleeping>
so, back to your question of how IPLD relates to all of this
<whyrusleeping>
in that DAT doc, they lay out their datastructure as a protobuf, and then define semantics around hashing things different ways
<gozala>
but it’s hard at least for me to see how one could connect all these technologies to do that
neuTHral_ has joined #ipfs
<whyrusleeping>
with ipld, you can just draw out that datastructure however you like, and the hashing and serialization and linking is all taken care of
<MikeFair>
whyrusleeping: It came up earlier that, for example, that the same header and footer reused across many files could be split more intelligently to have their blocks reused if there was an easy to construct a custom file
<whyrusleeping>
MikeFair: yeah, i saw that and was thinking about it
<whyrusleeping>
MikeFair: One intermediate step would be a command that create a file from other files
<whyrusleeping>
so you could do: `ipfs merge QmHeader QmBody QmFooter`
<MikeFair>
whyrusleeping: That's what I was thinking, like "+" operator as a parameter to add; or the "tar first" flag as I mentioned
* MikeFair
nods.
<whyrusleeping>
I think that would probably get a lot of use
<whyrusleeping>
gozala: does my ipld analogy make sense?
<gozala>
whyrusleeping: I’m not sure I totally get it, but I think I did somewhat
<whyrusleeping>
think of IPLD as a merkledag aware protobuf/json/[insert favorite serialization format here]
<MikeFair>
gozala: This dat file thing is something different than altering the structure of ipfs such that all files are inherently versioned
<gozala>
whyrusleeping: yeah I understand that and reading IPLD paper it all made sense
<gozala>
I guess part that was not clear for me how would I use that
<MikeFair>
gozala: You would take SLEEP, and stick it in IPLD
<MikeFair>
SLEEP is a file format convention for a linked list of specifically formatted records; IPLD is a structure for storing linked and arbitrary data records
<MikeFair>
So each record in a SLEEP file would become an entry in an IPLD database
<whyrusleeping>
so basically, if you implement a linked list in IPLD, it does the same thing as sleep
<MikeFair>
Once you have that, you then need to teach browser how to read it
<MikeFair>
But there might be some magic sauce where ipfs constructed a single file for the whole dat for you if you requested it the right way
<gozala>
MikeFair: they changed the docs so I’m not exactly sure where the part i was trying to point to is but they store metadata as append only log with each entry being signed by dat owners key
<gozala>
so that every version can be verified to be of the author
<MikeFair>
whyrusleeping: Has anyone cracked the code on making an append only file structure something less than rewriting the entire IPLD tree back to the root yet?
<gozala>
and each version contains prev version hash as well to make sure that it can’t be altered
<whyrusleeping>
MikeFair: you can do it easily in log(n)
<MikeFair>
gozala: Right, so picture each "block" as an entry in IPLD that has references to other IPLD entries
<whyrusleeping>
there are interesting algorithms that give you amortized constant operations for appends
<whyrusleeping>
but that ends up being a series of N constant time operations, and the occasionally you have to do a O(n) operation
<MikeFair>
whyrusleeping: How's that? Doesn't each entry need to be updated to point at the new child (which just changed addresses because of CAS)
<gozala>
MikeFair: whyrusleeping I get what you’re telling me. Which is also what I was saying - I’m looking for something higher level that uses bunch of technologies from ipfs family
<fiatjaf>
yes, daviddias, I'm following those, but apparently all but one are node.js examples, so maybe there's a difference in behavior (probably)
<MikeFair>
gozala: Okay; I'm not sure what that means. The highest level I can envision is you make a DAT; you stick it in IPFS, you get the CID, and you point an IPNS record at it
jaboja has joined #ipfs
<whyrusleeping>
MikeFair: yeah, he wants that, but with nice apis
<MikeFair>
gozala: Then you change the dat; you stick it in ipfs again; you update IPNS again
<MikeFair>
whyrusleeping: I think a revisioned history of IPNS entries (IPRS) would finish most of the job
<whyrusleeping>
MikeFair: yeah, me too
<gozala>
MikeFair: what I was trying to say is the in terms of abstraction level Dat is more or less what you want for writing web apps and ipfs provides ton of little things that are too low level
<MikeFair>
And I think what I(we) are trying to explain is that you're poking at a gap in the current toolset and we're attempting to explain the existing toolset; and what has been proposed to address the gap
<gozala>
anyway just my impression and admittedly I can’t connect all the pieces so I might be overlooking things
<MikeFair>
Or you're genuinely poking at a "That doesn't exist yet" gap. ;)
<gozala>
MikeFair: ok I see
<MikeFair>
But I do think you are missing the key distinction between CAS/IPFS CIDs; and an IPNS entry
<gozala>
MikeFair: I was under impression that you were telling me if you connect IPNS+IPRS+IPDL+IPFS you’ll be able to get that
<MikeFair>
With a DAT, you append some data to a file, and users can download the same file but with more data in it
<MikeFair>
gozala: That's not too far from the truth (but IPRS replaces IPNS and gives you revision history; IPLD gives you lists of records instead of single files; and IPFS is kind of orthogonal to IPLD, but IPFS gives you files whereas IPLD gives you records (JSON Objects))
<gozala>
MikeFair: Given that I have no idea what CAS is likely so :)
<MikeFair>
CAS = Content Addressed Storage
whyrusleeping has quit [Changing host]
whyrusleeping has joined #ipfs
<MikeFair>
It means, the file name is determined by the contents
<gozala>
MikeFair: oh yeah I see
<MikeFair>
You change the file and you change the contents
<gozala>
MikeFair: I think you’re overlooking some of the dat stuff though
<MikeFair>
whyrusleeping: I answered my earlier question about "append only"; the answer is you rewrite the algorithm so it's a prepend only operation
<gozala>
it is build on CAS as well
<MikeFair>
gozala: I only quickly read through the front page you linked; what it looks like to me is SLEEP is an archiver file format that bundles a bunch of revision history into a single file
<gozala>
MikeFair: yeah it’s used for dat metadata
<MikeFair>
gozala: The CAS stuff is then baked inside that metadata
<gozala>
MikeFair: DAT also contains actual data files
<gozala>
let me see if I can find a link to that
<MikeFair>
right, treat DAT as if it's an encapsulated ipfs
<gozala>
MikeFair: I think I understand what your telling me
<MikeFair>
There's a proposal (at least I think it's still a proposal in that there's no code implementing it) called IPRS which should replace IPNS; that includes a feature to do that
<gozala>
another ipfs container that would have say current and last entries
<gozala>
and current being current version and last being the former thing IPNS pointed to
<MikeFair>
right
<MikeFair>
THough you can't put the current reference in the container because you can't know your own address in advance
<gozala>
MikeFair: that’s a fair point, but kind of convention
<MikeFair>
To some extent, it's all convention; it's merely a function of how often and easy it is to use that convention
<MikeFair>
From my brief read, IPRS looks like IPNS++
<gozala>
MikeFair: in other words than my browser would not be implementing ipns://… protocol
<MikeFair>
Get more, lose nothing, basically the same thing
<gozala>
but rather ipns+://...
<MikeFair>
I suspect they'd make an effort to keep the ipns protocol; but if they can't they'd make iprs:// and tell the world to stop using ipns asap
<MikeFair>
I'd drop ipns in a heartbeat if iprs was available
<MikeFair>
But that's the gap you're pointing at; currently IPNS doesn't track historical values. It is expected that if you want that, then you store the historical value somewhere else
<MikeFair>
(Back to my DNS analogy earlier)
<whyrusleeping>
MikeFair: IPRS will upgrade IPNS
<MikeFair>
whyrusleeping: So the expectation is that ipns:/ will remain; and older code will just misbehave on newer records until they get upgraded?
<whyrusleeping>
ipns:/ will remain
* MikeFair
nods.
<whyrusleeping>
at the protocol level, the two won't interfere with eachother
<whyrusleeping>
so older nodes just won't be able to resolve IPRS records
<whyrusleeping>
so, right
<whyrusleeping>
what you said
<MikeFair>
makes sense
<MikeFair>
gozala: This DAT archive thing seems like a really good use case for encouraging people to upgrade their systems :)
<MikeFair>
In my mind's eye; the only major thing that really needs to do be done is make the dat file format "prepend only" instead of "append only" to beeter match ipfs semantics versus regular semantics
ulrichard has quit [Remote host closed the connection]
dimitarvp` has quit [Quit: Bye]
Encrypt has quit [Quit: Quit]
<MikeFair>
gozala: Now that I think on it a bit more; I suspect it's mostly that way already
<MikeFair>
gozala: BTW, IPNS does support using custom keys; so you can already create one IPNS per file archive
<MikeFair>
whyrusleeping: How difficult is it to add a custom file format to ipfs? Specifically I'm thinking of sparse files
<MikeFair>
whyrusleeping: So the file structure would have start/end byte offsets and a CID that stores those offsets
<MikeFair>
err the data in those offset
<MikeFair>
whyrusleeping: It's very close to the current structure, but it doesn't assume the file is 100% contiguous
<daviddias>
you can just do ipfs.dag.get('<the CID in question>/addressbook/friends/daviddias/email/home', (err, result) => { console.log('David personal address is', result)})
<daviddias>
the jsipfs one?
yuhl has joined #ipfs
<MikeFair>
It was several months ago, I think it was go
<daviddias>
you can do jsipfs dag get that path too
<daviddias>
I see
espadrine has joined #ipfs
<daviddias>
did you ever had the chance to follow those examples?
witten has joined #ipfs
<MikeFair>
I don't think I followed the ones you just provided, but I followed others; and the answer was consistent that the query retrieval wouldn't automatically follow linked data items
<MikeFair>
I wanted to build a distributed repository where different people controlled different areas of the repo but was accessible via a common root
<MikeFair>
And so not being able to start at the common root id and traverse into different linked areas was a bit of a usability blocker
<daviddias>
MikeFair: "onsistent that the query retrieval wouldn't automatically follow linked data items" that was certainly a bug because dag.get is designed to support full paths
<daviddias>
sorry about the hassle
<daviddias>
did you get to open an issue?
<MikeFair>
no worries; it was/is new tech; I haven't given up ;)
<MikeFair>
daviddias: No, but I commented on a related issue; having summary info in the parent object that then resolved down for "more details"
<daviddias>
thank you :)
<MikeFair>
In that example, for instance, having access to first/last name in the parent object is useful to prevent traversal to the lower node just to get those two data items
<MikeFair>
If you picture 1,000 contacts; being able to "push up" a couple data items into the parent list would prevent 1,000 add'l node retrievals
<daviddias>
yeah, you can and should shard your graph to optimize the usage pattern of it
<daviddias>
IPLD doesn't make that decision for you
<daviddias>
you can have a giant JSON blob and signal where you want it to be shared with `/` and when you call flush, it will slice it there and replace by the respective CID
<MikeFair>
But not really; if I have a "DavidDias" contact; I can't put first/last name in the parent object (unless I specifically make and extra copy of it) and the rest of the data in the DavidDias Object
<MikeFair>
That's a cool tool
<MikeFair>
I think the best description of what I'm talking about is something 'upper level data caching'
<MikeFair>
It's a practical point on UX implementations
<MikeFair>
If I have a top level "Contacts" object; it will contain 1,000 links to actual contacts
<MikeFair>
But if I want to present those contacts in a list to the user, some basic data to identify the contact (like first/last name) would be useful so the user can see what contact they are clicking on
<MikeFair>
The only way IPLD allows me to get that information is to retrieve the child node
<MikeFair>
(so the whole contact is retrieved; then the first/last name are returned)
<MikeFair>
The semantic I was reporting was a syntax that allowed making a link with some of that data in the link itself
<MikeFair>
reporting/promoting
yuhl has quit [Remote host closed the connection]
<MikeFair>
I then absolved IPLD of having to deal with user's putting inconsistent data in; IPLD would use a least effort approach to give the currently best answer it's aware of
<MikeFair>
So if the first and last name entries were somehow different from the summary values in the upper node; IPLD would give the upper answer until it actually traversed the link and pulled the child node, then give the new answer after that
<MikeFair>
(If the child node should get purged from cache, it would revert to giving the earlier upper answer)
<daviddias>
MikeFair: I believe the problem you are describing is not related to IPLD itself but rather it is how you structure your database
<daviddias>
your data in your database*
<MikeFair>
In this case I'm putting the same data point in the IPLD structure twice
<MikeFair>
contact/dd/first_name exists in two places
<daviddias>
you would get the same issues if you were sharding your data the same way in another db
<daviddias>
and note
<daviddias>
in soon™ future
<daviddias>
there will be a new thing in IPLD land called selectors
<daviddias>
so that you can send queries to other nodes and get just the data you need which can be from one or more nodes
<daviddias>
nevertheless, it is still more a data modeling problem and not an IPLD per se
<MikeFair>
I totally agree the situation has issues; and so far that ambiguity has been a question/holdback to allowing the feature; so I'm simply chiming in to say the feature's utility outweighs the ambiguity
<MikeFair>
daviddias: I disagree on the IPLD vs data modeling part because I can't currently create a partial object in IPLD or put the same data in two places.... here's an example
<daviddias>
I'm thinking that one good way to implement a design like you are describing is by having Indexes that use selectors underneath
<MikeFair>
But that requires more "effort" to retrieve the missing data
<MikeFair>
The point I'm making is the speed and conservation of effort
<MikeFair>
So if that's the MikeFair contact object, it's easy to put a link to that in a higher object
<MikeFair>
That higher object has a 1,000 or more of those object links
<MikeFair>
But the only data it has about those objects is the reference address (MikeFair in the example) and link id itself
<MikeFair>
If I wanted to make a UI that presented those connected objects in a list, by pulling the first/last name from each object; the system has iterate over all of them and pull each child node (in essence it has to open each file to get the data)
<MikeFair>
If however, in the link reference itself, we could stuff some data directly in the upper node, say only the first/last name for each link for instance; then the system can skip having to dereference all 1,000 contacts and instantly disply the list
<MikeFair>
The reason this becomes a potential IPLD thing, is that when I go to update the MikeFair contact, and I change the first_name to "Michael" what happens to the data in the upper node
sim590 has joined #ipfs
<MikeFair>
Obviously the CID for the MikeFair contact changes and the upper node's link address changes; but what I'd want is that any overlapping cached data items would also get updated
vsimon has quit [Read error: Connection reset by peer]
vsimon has joined #ipfs
<MikeFair>
The "feature" then is the chance to cache lower level data directly in the higher nodes; without having to explicitly create a secondary pathname to reference it
<MikeFair>
In the link reference we're expressing that "this link caches the first_name and last_name fields of the referenced node"
<MikeFair>
What fields are cached and why is then a data modelling question
<MikeFair>
This is the age-old "summary/detail view" thing
robattila256 has quit [Ping timeout: 255 seconds]
matthiaskrgr has joined #ipfs
<MikeFair>
In almost all hierarchical storage systems, the UIs present some kind of summary/basic identification information from the lower detail level to help the user identify the lower level object, or prevent retrieval of the full object by giving oft-referenced data items