<Icefoz_>
I'm *basically* trying to solve The IPNS Problem and a bunch of other semi-related problems
<Icefoz_>
Most of which can be summed up by having a database you can either tell "here is a new document you should index" or ask "what documents have relationship X to document Y?"
gts has quit [Ping timeout: 248 seconds]
<lgierth>
"index that" is a write operation on a part of your data, and "relation X to Y" is an index graph that needs to be structured so you can go /ipfs/QmDocY/docs-by-X/ :)
<lgierth>
well not neccessarily exactly /ipfs/QmDocY/docs-by-X... just an example
<lgierth>
if you don't expect millions of documents with that relationships, it can be a simple directory `"X": [{"/": "QmX1"}, {"/": "QmX2"}]" in your Y document
<lgierth>
or a separate node/document that you link to from Y if that's data you only need sometimes
<lgierth>
s/directory//
leavehouse has quit [Ping timeout: 246 seconds]
<lgierth>
one big drawback with this ^ is that your document's hash would change everytime a relationship is added
<lgierth>
so this relationship index should probably be objects whose sole purpose is expressing these relationshipe
jaboja has quit [Ping timeout: 255 seconds]
<lgierth>
basically, any kind of lookup or query that is not fetching exactly one specific document, is an index read (as in any relational DB too btw!)
<Icefoz_>
Indeed.
<Icefoz_>
This sort of comes down to building a relational DB that just stores its data chunks in IPFS, it sounds like. :-P
<lgierth>
and with IPLD you have complete freedom in how you structure these indices, except you can't have circular dependencies, and IDs change everytime data changes
<lgierth>
think of it more like a graph db
jkilpatr has quit [Ping timeout: 246 seconds]
<lgierth>
but one with the big immutability constraint
<Icefoz_>
The immutability constraint for documents is exactly what I'm trying to work around. :-P
<lgierth>
there's also nothing stopping you of putting /ipns links in your data. IPLD won't understand them and they'd be just strings, but whatever reads your docs can make sense of them
<lgierth>
if you really can't get around making your data one big immutable graph with one (or more) roots
<Icefoz_>
Well part of the problem is that IPNS doesn't seem like a good solution for things that might change often.
<Icefoz_>
Or involve dynamic content.
<Icefoz_>
(Like database query responses.)
<lgierth>
you make it sound like you use multiple /ipns identities to do RPC (request/response) over ipfs?
<Icefoz_>
I don't.
<lgierth>
IPLD shines when you express your queries as graph traversals
<Icefoz_>
RPC is part of this thing but I don't have a super clear feel for what form it takes yet.
<lgierth>
sorry about the handwaiving, i haven't actually built a lot of graph data structures so far
<Icefoz_>
I mean yes, a graph data structure is great for data structures that are easily represented by graphs. :-P
<Icefoz_>
No worries, I'm really just feelig around for what is possible and how things fit together.
<lgierth>
right :)
<lgierth>
don't worry about adding a ton of ipld nodes that aren't actual documents in your application
<lgierth>
these are totally okay to have if they support the case of traversing the data
<Icefoz_>
What is IPLD used for? Describing connections between IPFS nodes, such as for storing directory trees?
<lgierth>
yes unixfs directories are one format that IPLD can make sense of
<Icefoz_>
So a "directory" is an IPLD object that contains links to all the contents, with the IPLD object stored in IPFS?
<Icefoz_>
Okay.
<ChrisMatthieu>
We're looking for a good IPFS database or queuing system for computes.com too. Otherwise, we are planning to build one.
<Icefoz_>
I seeeeee.
<lgierth>
i.e. directory membership of a file is expressed as a link from the directory node to the file node
<ChrisMatthieu>
IPLD looks promising too
<lgierth>
and file chunking is just a file node => file chunks link set
<Icefoz_>
Suddenly it all makes sense! :D Thank you.
<lgierth>
hah :)
<lgierth>
if you do `ipfs block get QmSomeDirectory` you see a protobuf structure called merkledag (or dag-pb) which has Links and Data, and in that a second protobuf structure called unixfs (which basically says this is a dir)
<ChrisMatthieu>
@haad has a bunch of updates to orbitdb queued on his localhost
<ChrisMatthieu>
I was also looking at interface-datastore today
<lgierth>
dag-pb is kinda deprecated and we're moving more stuff to "native ipld", i.e. with links embedded right in the document (instead of Links/Data)
<lgierth>
but IPLD can make sense of dag-pb just fine
<lgierth>
ChrisMatthieu: nice!
<lgierth>
i'd love to chat more but unfortunately i need to get up early tomorrow :)
<lgierth>
the command for working dag-pb nodes is `ipfs object`, and the unixfs commands (ipfs add, get, cat, files) work on top of that
<lgierth>
at some point soon we want to make a unixfs-v2 which would be native IPLD
<lgierth>
native as in, the standard default format of how IPLD is serialized into a block is CBOR
<lgierth>
and native as opposed e.g. blocks that are imported from git, ethereum, or torrent
<lgierth>
*opposed to
<Icefoz_>
No worries.
<Icefoz_>
Sleep well!
<lgierth>
stebalien: yeah maybe "native IPLD" is a good term for these cbor nodes that aren't some other stuff like dag-pb, git, or btc)
<lgierth>
thanks! good night
<ChrisMatthieu>
It looked like IPLD in ipfs-api was not ready and IPLD in js-ipfs was missing remove
<ChrisMatthieu>
thanks @lgierth
<lgierth>
and again, view this as a denormalization and careful index/view planning -- unfortunately not much precedent yet but that means you can be one of the first to write down how you did it ;)
<lgierth>
ChrisMatthieu: try `ipfs block rm` or so instead
<ChrisMatthieu>
ok, i was looking at ipfs dag
<lgierth>
deletion is hairy and we don't want to it on the upper layers, only on the block layer
<lgierth>
because it's a repo/datastore thing, and that part is only concerned with blocks, not objects/nodes/directories
<ChrisMatthieu>
thanks
dimitarvp has quit [Quit: Bye]
<lgierth>
if the file you're trying to delete has children (e.g. chunks or other links) you need to find them and delete them too i guess
<lgierth>
`ipfs refs -r` does that
<lgierth>
(because blocks don't know anything about links, only the layers above that do)
erictapen has quit [Ping timeout: 260 seconds]
nikivi has quit [K-Lined]
leavehouse has joined #ipfs
nikivi has joined #ipfs
droman has quit [Quit: WeeChat 1.9.1]
engdesart has joined #ipfs
kvda has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
kvda has joined #ipfs
kvda has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
kvda has joined #ipfs
kvda has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
_rht has joined #ipfs
athan has quit [Remote host closed the connection]
kvda has joined #ipfs
johnytiago has joined #ipfs
johnytiago has quit [Ping timeout: 240 seconds]
kvda has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
MDude has joined #ipfs
kvda has joined #ipfs
dhruvbaldawa has joined #ipfs
asyncsec has quit [Quit: asyncsec]
athan has joined #ipfs
kvda has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
wxss has joined #ipfs
kvda has joined #ipfs
sleepdeficit[m] has joined #ipfs
kvda has quit [Client Quit]
dhruvbaldawa has quit [Remote host closed the connection]
kvda has joined #ipfs
chris613 has quit [Quit: Leaving.]
wxss has quit [Remote host closed the connection]
chris613 has joined #ipfs
kvda has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
niekie has joined #ipfs
kvda has joined #ipfs
NullConstant has quit [Ping timeout: 248 seconds]
_whitelogger has joined #ipfs
kvda has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
kvda has joined #ipfs
kvda has quit [Client Quit]
CHIPPY has joined #ipfs
kvda has joined #ipfs
kvda has quit [Client Quit]
}ls{ has quit [Quit: real life interrupt]
_rht has quit [Quit: Connection closed for inactivity]
dhruvbaldawa has joined #ipfs
dhruvbaldawa has quit [Ping timeout: 240 seconds]
m0ns00n has joined #ipfs
dhruvbaldawa has joined #ipfs
CHIPPY has quit [Ping timeout: 248 seconds]
redfish has quit [Quit: WeeChat 1.9.1]
CHIPPY has joined #ipfs
CHIPPY has quit [Max SendQ exceeded]
CHIPPY has joined #ipfs
CHIPPY has quit [Max SendQ exceeded]
CHIPPY has joined #ipfs
CHIPPY has quit [Max SendQ exceeded]
gts has joined #ipfs
CHIPPY has joined #ipfs
CHIPPY has quit [Max SendQ exceeded]
CHIPPY has joined #ipfs
CHIPPY has quit [Max SendQ exceeded]
CHIPPY has joined #ipfs
CHIPPY has quit [Max SendQ exceeded]
CHIPPY has joined #ipfs
CHIPPY has quit [Max SendQ exceeded]
rendar has joined #ipfs
infinity0_ has joined #ipfs
infinity0 is now known as Guest61270
infinity0 has joined #ipfs
infinity0_ is now known as infinity0
infinity0 has quit [Changing host]
Guest61270 has quit [Ping timeout: 240 seconds]
leavehouse has quit [Ping timeout: 255 seconds]
edulix has joined #ipfs
leavehouse has joined #ipfs
kirby__ has joined #ipfs
johnytiago has joined #ipfs
edulix has quit [Ping timeout: 246 seconds]
akkad has quit [Quit: Emacs must have died]
edulix has joined #ipfs
inetic has joined #ipfs
johnytiago has quit [Ping timeout: 240 seconds]
captain_morgan has quit [Remote host closed the connection]
captain_morgan has joined #ipfs
niekie has quit [Remote host closed the connection]
joja has joined #ipfs
smuten has joined #ipfs
jrahmy_ has joined #ipfs
cxl000 has joined #ipfs
jojaxyz has joined #ipfs
jrahmy_ has quit [Quit: Leaving]
joja has quit [Remote host closed the connection]
jojaxyz has quit [Quit: Leaving]
gts has quit [Remote host closed the connection]
Alpha64 has quit [Quit: Alpha64]
ylp has joined #ipfs
smuten has quit [Quit: leaving]
smuten has joined #ipfs
maxlath has joined #ipfs
leavehouse has quit [Ping timeout: 240 seconds]
maxlath has quit [Quit: maxlath]
maxlath has joined #ipfs
edulix has quit [Ping timeout: 260 seconds]
screensaver has joined #ipfs
screensaver6 has quit [Remote host closed the connection]
edulix has joined #ipfs
gts has joined #ipfs
pedrovian has quit [Read error: Connection reset by peer]
igorline has joined #ipfs
johnytiago has joined #ipfs
girrrrrrr2 has quit [Ping timeout: 240 seconds]
arpu_ has joined #ipfs
johnytiago has quit [Remote host closed the connection]
<theodor488>
Trying to maybe get some answers faster :)
girrrrrrr2 has joined #ipfs
dhruvbaldawa has joined #ipfs
<theodor488>
anyone previously used js-ipfs client-side to upload ?
mib_kd743naq has joined #ipfs
girrrrrrr2 has quit [Ping timeout: 248 seconds]
<mib_kd743naq>
so I have a question ( and please don't take it the wrong way - I need to know this for my own planning ): is it correct to assume that all dev resources are currently focused on low level stuff like libp2p and/or the store and/or the filecoin side of things?
<mib_kd743naq>
in other words: folks like myself who want to get some issues fixed within the unixfs part, before going on to using it at scale - is it reasonable for me to assume that if I don't find time to fix them, it is unlikely someone else will?
<mib_kd743naq>
Kubuxu: ^^
maxlath has joined #ipfs
NullConstant has joined #ipfs
<Kubuxu>
mib_kd743naq: not at all, we have full or even more resources in go-ipfs right now than we had few months ago
<Kubuxu>
s/go-ipfs/ipfs itself/
<mib_kd743naq>
oh! well this is great to hear then ;)
<Kubuxu>
mib_kd743naq: what is the issues you are thinking about?
<mib_kd743naq>
specifically https://github.com/ipfs/go-ipfs/issues/4286 is a blocker for me - depending on whether this is fixable or not will decide how I structure a ~500gb dag
<mib_kd743naq>
( there are other lower level things, but that one literally doesn't let me go forward, because custom-chunking that much data is a very very long process )