jaboja has quit [Remote host closed the connection]
devbug has joined #ipfs
avastmick_ has joined #ipfs
apiarian has quit [Quit: zoom]
Oatmeal has quit [Ping timeout: 250 seconds]
slothbag has joined #ipfs
A124 has quit [Remote host closed the connection]
wallacoloo has joined #ipfs
A124 has joined #ipfs
pfista has joined #ipfs
avastmick_ has quit [Quit: Page closed]
<slothbag>
quick question about working with gx.. if you want to tweak a package that is referenced as a gx package what is the best way? do you modify the files at /gx/ipfs/QmAbCd? Or do you change your import references to use a local src/github.com/name/repo temporarily?
<whyrusleeping>
slothbag: easiest way is to do `gx-go rewrite --undo PACKAGENAME`
<whyrusleeping>
it should work in most cases
<whyrusleeping>
when youre done, you can do `gx-go rewrite` to set everything back to normal
Oatmeal has joined #ipfs
<slothbag>
whyrusleeping: thanks, i'll try it out
M-drwasho has joined #ipfs
kerozene has joined #ipfs
<M-drwasho>
hey folks
matoro has quit [Ping timeout: 250 seconds]
<ipfsbot>
[go-ipfs] pfista opened pull request #2810: Adds aliases to stats command (master...2680-stats-alias-commands) https://git.io/voJok
<A124>
Kubuxu: Oh yeah. Senn the talk around that last days. It is little different from what I meant, but I guess would be applicable, if relevant interface in binary.
<A124>
Nope. Someone please pin, not sure if pinned already. zero lenght data.
<Kubuxu>
A124: I think same policy as we took in the cjdns is better, it is up to application to use compression, it might be just overhead in some cases.
<Kubuxu>
But there could be some parameter to enable the compression on a channel.
<A124>
Kubuxu You do not want to compress a lot of stuff maybe.. pages scripts, etc. So I would say lz4 should be there in ipfs.
<A124>
Relative to encryption the performance hit is negligible but on a lot of data the compression does have great effect.
rgrinberg has quit [Ping timeout: 244 seconds]
<A124>
Just to be clear I meant the chunk transfer between nodes.
<Kubuxu>
You don't want to compress videos that are huge part of current internet traffic.
* A124
really things it would improve overall speed of the network. If in feature pages and everything shall be served, we do not want to use javascript to do compression, do we? Also storing stuff compressed might not be best idea. Transfers of smaller pages, html under 256k, javascripts and all would speed up significantly.
<A124>
Yeah, I did ... think about videos. Yes. lz4 does few hundred MB/s. The IPFS nodes could have index of compressible or noncompresible chunks.
<A124>
*few hundred on single desktop core, servers are around gigabyte.
<A124>
Well, the CPU powerful ones, the low power 1.6-1.8 we use here would be same.
<Kubuxu>
But it does few hundred MB/s maxing a core, are you sure what
<Kubuxu>
we, in the middle there
<Kubuxu>
I got to go.
<A124>
Alright, see you. Most nodes would not have really fast connections. It should be settable per node, with default compressed.
lars887 has joined #ipfs
<M-leer10>
Does anyone know what happened to ipfs-board?
<A124>
M-leer10 Nope, tried to run it last time, no success. It is internet, if only one person works on something, and no maintainers, it is doomed to fail eventualy.
<A124>
I guess it has to be some API changes or something. Or maybe my setup.
<M-leer10>
Ah what a shame :/
lars887 has quit [Quit: jmIrc destroyed by the OS]
<A124>
M-leer10 Did you try running it with 0.4?
<M-leer10>
Yeah but it was a task to get different clients to find each-other's profiles and commenting on other's posts didn't work
<A124>
M-leer10 I guess there would have to be some ipns type of synchronization so if a node knows of someone having a board, it would include them in the list, and that list would of course propagate to others, so in the end it would be list of ipns IDs that do have a board, unless they never announced their existence to anyone.
reit has quit [Ping timeout: 264 seconds]
<A124>
Also if you get any updates on this or other board or stuff, and you remember, let me know. I am currently interested in more stuff (I could be a large team leader and they would still have work to do, haha) .. but currently more about data transfer, compression, and when it concerns services atop of IPFS, markdown, blog for information sharing and archival, and a shift to the new paradigm.
<A124>
M-leer10 He seems to be occasionaly active, try to get in ouch with him.
<ipfsbot>
[js-ipfs] dignifiedquire deleted greenkeeper-ipfs-bitswap-0.4.1 at 7850dbb: https://git.io/voUPh
<92AAAXHIO>
[js-ipfs] diasdavid force-pushed greenkeeper-babel-runtime-6.9.1 from 5e87d75 to 730b5a5: https://git.io/voUXe
<92AAAXHIO>
js-ipfs/greenkeeper-babel-runtime-6.9.1 730b5a5 greenkeeperio-bot: chore(package): update babel-runtime to version 6.9.1...
<7YUAA947Y>
[js-ipfs] diasdavid force-pushed greenkeeper-interface-ipfs-core-0.2.2 from a6bd51d to c10a193: https://git.io/voUXT
<7YUAA947Y>
js-ipfs/greenkeeper-interface-ipfs-core-0.2.2 c10a193 greenkeeperio-bot: chore(package): update interface-ipfs-core to version 0.2.2...
<ipfsbot>
[js-ipfs] diasdavid pushed 1 new commit to update/files-add-interface: https://git.io/voU18
<ipfsbot>
js-ipfs/update/files-add-interface 1cdea62 David Dias: add createAddStream feature
stoogenmeyer has joined #ipfs
<ipfsbot>
[js-ipfs] diasdavid closed pull request #308: Update interface-ipfs-core to version 0.2.2
matoro has joined #ipfs
mdangear has quit [Quit: mdangear]
mdangear has joined #ipfs
<ipfsbot>
[js-ipfs] diasdavid force-pushed update/files-add-interface from 1cdea62 to ea0eb40: https://git.io/voUyh
<ipfsbot>
js-ipfs/update/files-add-interface d704742 Stephen Whitmore: Make ipfs.files.add return DAGNodes.
<ipfsbot>
js-ipfs/update/files-add-interface 3e1f563 Stephen Whitmore: Rename unixfs output from 'stream' to 'content'.
<ipfsbot>
js-ipfs/update/files-add-interface b083649 Stephen Whitmore: Mitigate test timing issues....
eleuwhavercamp has joined #ipfs
<ipfsbot>
[js-ipfs] diasdavid force-pushed update/files-add-interface from ea0eb40 to 9bb79f6: https://git.io/voUyh
<ipfsbot>
js-ipfs/update/files-add-interface 9bb79f6 David Dias: add createAddStream feature
<eleuwhavercamp>
I've tried the mounting option with ipfs cmd line tool following https://ipfs.io/docs/commands/#ipfs-mount docs. It seems to mount correctly but if I ipfs files ls / I don't see any files. However there are definitely files listed via localhost:5001. I'm just wondering if anyone else has experienced this issue and possibly has a solution.
<ipfsbot>
[js-ipfs] diasdavid force-pushed update/files-add-interface from 9bb79f6 to 903022c: https://git.io/voUyh
<ipfsbot>
js-ipfs/update/files-add-interface 903022c David Dias: add createAddStream feature
mdangear has quit [Quit: mdangear]
<Kubuxu>
eleuwhavercamp: ipfs mount and ipfs files ls are totally different currently, we will be bringing them together soon.
<Kubuxu>
ipfs mount exposes two mountpoints: /ipfs/ and /ipns/ which are addressed further by hashes.
mdangear has joined #ipfs
mdangear has quit [Client Quit]
<eleuwhavercamp>
Thanks for the response. okay got it. yes got ipfs and ipns mounted. I've tried to ls directly on these mounts but ipfs mounts without read access. I tried changing the permissions but without success. I was wondering if I should use the --writable flag?
<Kubuxu>
eleuwhavercamp: you can't ls the /ipfs/ mount, although you can access files under it.
<noffle>
A124: do you have ${GOPATH}/bin in your $PATH?
<A124>
Hey! Back. Yes, I did.
<A124>
It does compile and do stuff, but in the end it does not appear in gopath/bin folder, checked. So just downloaded the tarball.
<A124>
However the tarball has issue also, when trying to fetch it inits node and just hangs.
<A124>
noffle Well, not sure about how suitable or crude that would be, but a regexp on domain, (?:http|https)://<some tested tld matching regexp>/ip(?:f|n)s/Qm[<bitcoin alphabet>]#.*
<A124>
I am not sure if one can supply query strings to javascript and stuff, so maybe # is out of place. But even if it did recognize the "official" two gateway (sub)domains, plus localhost with any port it would be beneficial a lot.
* A124
note there is always this thing with gateways that do not use /ip(f|n)s/
<noffle>
A124: the tarball is an older version fyi
<noffle>
I'm still working on this release
<A124>
Well, when you figure out where the binary disappears let me know. But I could likely scan the whole system also xD
<noffle>
really appreciate the feedback. I can ping you when these are addressed if you feel like taking another stab at it
<noffle>
ha
<A124>
Sure, always do, unless gone. Thanks.
<noffle>
if you 'cd' into the ipget/ipget dir and 'go build' do you get a binary in that dir?
<A124>
Yes, getting "panic: pprof: NewProfile name already in use: fds" .. will check config.
<Kubuxu>
noffle: as you are working on ipget, if there is any official release done, ping me on IRC or GH, I am packaging it for Arch
<noffle>
might be me -- I'll dig some more after sync
<noffle>
Kubuxu: awesome! I will :) is it in aur or in the main repos?
<A124>
I believe in aur, as it's not stable.
<Kubuxu>
AUR, I am not trusted user and it isn't voted high enough.
<A124>
If I was trusted I would not include that at this point anyway.
<Kubuxu>
Ture, unstable :p
<A124>
Is there a way for people to "pin" stuff onto gateways?
<Kubuxu>
Not pin, you can just access it which will keep it there for some time
<A124>
Define some time please.
<Kubuxu>
Until servers doesn't run out of space.
<Kubuxu>
Weeks
<noffle>
A124: it's not a standardized thing, so don't set expectations
<Kubuxu>
But lgierth has more info on that: how long gateways keep objects.
<A124>
Ah. .. hm. Got idea. There should be a way to temporarily pin (with TTL) small things onto gateways.
<Kubuxu>
Yeah, pin as a service
<A124>
So I can say, pin this javascript file for half a year.
<A124>
And of course can renew it later if wanted, or it will just decay.
<Kubuxu>
Problem with service like that is limiting it somehow.
<A124>
It would have two major benefits a) speed, b) avilability, that it would not vanish of ipfs if author disappears, dies or anything.
sydbarrett74 has joined #ipfs
<noffle>
A124: we have ideas on this in the pipeline :)
<A124>
I both have no problems of hosting stuff, but some as libraries and common small assets would benefit, while I do not want to keep potentialy outated stuff undefinitely.
<Kubuxu>
My idea is that it would be best to standardize an API for that.
<Kubuxu>
Also I am willing to start again with hosting whole cdnjs (21GB of js libs) when the memory leak is fixed.,
<noffle>
richardlitt: sync is now, right? or in an hour? I'm pretty sure it's still 0900h PST...
<A124>
Yes, maybe. As a service on top of ipfs. .. What memory leak?
<Kubuxu>
I think in an hour.
<Kubuxu>
Or now.
<Kubuxu>
or I didn't help you
<noffle>
Kubuxu: :D
<noffle>
Kubuxu: we're the only ones who posted sprint updates
<Kubuxu>
Yeah, and there isn't much there :/
<A124>
Some times are just better, some times you got none. It eventualy evens out, not to be worries about.
<haad>
I thought it was supposed to be now. Also, I think richardlitt is off the grid atm.
<A124>
... ipfs node ramped to 500MB yeah, did restart and now only 16MB
<Kubuxu>
On my dedi it is on 900MB :p
jonnycru_ has joined #ipfs
<noffle>
haad: the calendar says in an hour -- huh. I swear it was now in the past
<A124>
Well I got very small set of pinned data, only the very much used like examples, and stuff.. which brings me to... there could be some gateway output of most requested stuff, plus a list of "stuff you should pin" like the geoip
sydbarrett74 has quit [Ping timeout: 240 seconds]
<Kubuxu>
I think it might be something with Summer Day Light Saving.
<A124>
We have here CET and CEST, most people are confused and has no idea what "CEST" timezone is and that it is bull, but the second means "Central European Summer Time" .. Avoids this problems.
<A124>
... Umm do not want to be the guy that always has something, but I encountered yet another thing.
<lgierth>
Kubuxu A124: the gateways keep stuff until they reach the GC watermark, then they run GC -- so "it depends"
<A124>
IPFS API Webui. a) The world view is laggy (remote connection, not sure about cause), b) if I enter hash to the "top search bar" nothing happens, c) The webui should have a way to pin stuff, not only to upload.
<ipfsbot>
[go-ipfs] Kubuxu created feature/ping-api (+1 new commit): https://git.io/voTvq
<ipfsbot>
go-ipfs/feature/ping-api 155714d Jakub Sztandera: Improve ping API a bit by returning failure in case of only failure...
<Kubuxu>
World view is going away.
<lgierth>
PSA: We were supposed to have the bi-weekly sprint sync hangout, but we messed up timezones again, and also half the team is traveling, so we're cancelling this one -- sorry!
M-oyajiru has joined #ipfs
<ipfsbot>
[go-ipfs] Kubuxu opened pull request #2813: Improve ping API a bit by returning failure in case of only failure (master...feature/ping-api) https://git.io/voTvC
<A124>
Kubuxu When I do search from the "DAG" the search works, but there should definitely be a "Pin/Unpin" button. ... Umm.. why it is going away? I like it. But would like a 2D one
<Kubuxu>
A124: it is resource heavy, the team is getting annoyed as it chewed through their notebooks' batteries
<noffle>
hey haad can I do anything to help with impending orbit demo this week?
<Kubuxu>
the flat map seems to be better idea
<lgierth>
s/demo/demos/ :)
<lgierth>
there's no formal demo but lots of showing it around
<haad>
noffle: not really. waiting for add and cat but afaik daviddias is working on those while flying.
<Kubuxu>
I wouldn't be able to, right away it would turn out I am missing something from the internet :p
zorglub27 has joined #ipfs
<noffle>
daviddias: yeah I saw -- I'll try and sync with him; I *was* working on those as of friday, but it looks like he went running with them over the weekend..
<noffle>
er, haad:
<A124>
Kubuxu Sorry to annoy you about that, but there will be a flat replacement? My major "catch" on the world view today is I cannot see whole world. Plus the CPU load of course. ...On other note I did test WebAssembly on Linux yesterday, the assets took long to load, but after that the difference against javascript is ... just have to see for self.
<Kubuxu>
I don't know what would running 3D game demo prove.
<A124>
A lot. a) performance, b) native instructions
<A124>
Which when using that in IPFS to transform data and for scientific, compression, decompression and all... might not only transform ipfs but the web as we use it.
<A124>
There is emscripten compiled asm.js version, back then when asm.js was not part of browsers, it did "ok" on relatively small images. Did not test lately on larger ones, but with WebAssembly and native instructions (it comes from HEVC, so really can use them) it would be really niche.
<Kubuxu>
This + IPLD WA transforms could seamlessly replace image fomrats
<A124>
No I did not see that. Not sure what mipmap pyramid is. But I guess you would still want blobs with compressed images, unless I am wrong.
<Kubuxu>
It is where you save image in consecutive smaller resolutions.
<A124>
Though ... my point was we could use the transforms both in ipfs and in browser, which means more code reuse and stronger ecosystem, which is a strong argument.
<Kubuxu>
Start with 1024x1024 it is level 0, level 1 is 512x512 and so on.
<A124>
Oh so binary pyramid, alright. Yeah, thanks.
<Kubuxu>
You can preload with with 4x lower resolution in 16 times lower time, and then load image in full as user wants to view it.
<A124>
What I... proposed silenty in my head is that there should be a web standard to distribute images. And unless it is original camera amateur jpeg fotage, there should be metadata to lossless image. As we all know how images look when you compress them repeatedly with different jpeg compressors.
<Kubuxu>
There is `derivedFrom` filed in that format.
<A124>
Kubuxu FLIF can do this kind of loading, plus lazy loading javascript, for example in PageSpeed Proxy, does that inlining small thumbs and lazy loading larger ones.
<Kubuxu>
Best part is: it is IPFS, linking it costs nothing, and you will be able to download it in most cases.
<A124>
Yeah, I get it. We also need to get that "on the web".
<noffle>
A124: just to double check, after 'go install'ing ipget, you don't see that binary in ${GOPATH}/bin ?
<A124>
noffle IDK, I had there the tarball one, but it did appear in the local folder where main.go is.
<Kubuxu>
also nice thing is, storing all mipmaps of an image increases its total size by 33%.
<noffle>
A124: I've pushed a fix. if you feel like giving it a shot, let me know how it goes :)
<A124>
noffle Feature request: Enhancement: let there be visible version in help of the ipget command.
<A124>
Will give it a shot.
<noffle>
A124: good idea
jonnycru_ has joined #ipfs
jonnycru_ has quit [Client Quit]
rgrinberg has quit [Ping timeout: 250 seconds]
jonnycru_ has joined #ipfs
jonnycru_ has left #ipfs [#ipfs]
jonnycrunch has joined #ipfs
<A124>
noffle install seems to work, but same error, not sure what it means exactly. panic: pprof: NewProfile name already in use: fds
<A124>
Need the trace or you know? .. Could it be interference with the old version? Does it create any files?
<noffle>
what imports show up in the github.com/ipfs/ipget/ipget/main.go ?
<noffle>
is there an ipfs hash before fallback-ipfs-shell?
<noffle>
if not, then 'go get' maybe didn't actually grab the latest
<A124>
Nope it did not, is there a way to force grabbing latest?
<noffle>
A124: I wonder if 'go get -d' doesn't try to overwrite what you have
<noffle>
try nixing the github.com/ipfs/ipget dir and go get'ing again
<A124>
noffle I rm -rf and did try again, oviosuyl had to get the source and it works, great job.
ligi_ has quit [Ping timeout: 244 seconds]
<noffle>
yay
<noffle>
thanks a lot for helping me test :)
<A124>
The help message is ... not consistent with the current usage, of course, which is minor detail. And seen there is already some version. Excellent.
s_kunk has quit [Ping timeout: 272 seconds]
<noffle>
ah, good catch
ligi has quit [Ping timeout: 244 seconds]
<noffle>
version string is wrong of course ;)
<A124>
Cheers! No problem. ... ipget is lovely, as .. the performance problems of running a daemon and the b/w is what a lot of people cannot offer.
s_kunk has joined #ipfs
jsrockss has joined #ipfs
<A124>
... and ipfs pin unpin add cat .. when you mix them few times without running daemon, it just breaks, ... which might be a bug, but did not ponder more around it, if its expected behaviour or not.
rgrinberg has joined #ipfs
<A124>
Out of the topic thread: gateway DNS, I am not profficient at DNS, but I believe there is a way to pick geographically closer gateways. When I am in Europe I would prefer to pick any that is also in Europe, the intercontinental latency and TCP ramp up affects smaller files a lot.
<lgierth>
yeah amazon route53 does that
<lgierth>
we wanna make use of it but haven't gotten to it yet
jonnycrunch has quit [Quit: leaving]
<A124>
.. Not even when, if I wanted to file github issue (not), would do that. So please fill the issue unless it exists.
<A124>
Great to hear that.
<lgierth>
i have it written down somewhere
<lgierth>
definitely not gonna be forgotten
<lgierth>
until then you can pick the gateway with the lowest latency from `dig +short A ipfs.io` and put that into your /etc/hosts
ansuz has quit [Ping timeout: 260 seconds]
Encrypt has quit [Remote host closed the connection]
jaboja has joined #ipfs
matoro has quit [Ping timeout: 276 seconds]
eleuwhavercamp has left #ipfs ["Ex-Chat"]
s_kunk has quit [Ping timeout: 246 seconds]
matoro has joined #ipfs
jaboja has quit [Ping timeout: 240 seconds]
holst has quit [Quit: bomb, kill, genocide, terrorism]
ashark has quit [Ping timeout: 264 seconds]
stoogenmeyer has quit [Remote host closed the connection]
stoogenmeyer has quit [Remote host closed the connection]
mildred has joined #ipfs
warptangent has quit [Quit: No Ping reply in 180 seconds.]
warptangent has joined #ipfs
bNARWOYV has joined #ipfs
bNARWOYV has left #ipfs [#ipfs]
stoogenmeyer has joined #ipfs
<ipfsbot>
[js-ipfs] dignifiedquire tagged v0.12.0 at 8aac907: https://git.io/voTE3
Guest7831 is now known as special
special has quit [Changing host]
special has joined #ipfs
ashark_ has quit [Ping timeout: 260 seconds]
UlLA has joined #ipfs
UlLA has left #ipfs [#ipfs]
ELLIOTTCABLE_ is now known as ELLIOTTCABLE
jokoon has quit [Quit: Leaving]
palkeo has joined #ipfs
rardiol has joined #ipfs
chriscool1 has joined #ipfs
zorglub27 has joined #ipfs
disgusting_wall has joined #ipfs
mbrock has joined #ipfs
rgrinberg has quit [Ping timeout: 252 seconds]
reit has quit [Ping timeout: 260 seconds]
taaem has joined #ipfs
cketti has quit [Ping timeout: 246 seconds]
ashark has quit [Ping timeout: 240 seconds]
ashark has joined #ipfs
cketti has joined #ipfs
cketti has quit [Changing host]
cketti has joined #ipfs
mdangear has joined #ipfs
Vytek has joined #ipfs
<Vytek>
hello everyone!
<Vytek>
can I ask a legal/techinal question about ipfs?
<brimstone>
Vytek: sure!
<Vytek>
thank you @brimstone
<Vytek>
If I make a node of ipfs nerwork
<Vytek>
and some ask an illegal file
<Vytek>
the block pass througt my node?
<Kubuxu>
no
<Vytek>
And I am legal responsable?
<Kubuxu>
Your node only stores and passes what you explicitly use/pin/download
<Kubuxu>
no
<Vytek>
so only if iI download someone to other node that I know is illegal..
<kyledrake>
lgierth the best way to do that is an anycast network. I'm working on some stuff that you could use for that if you wanted to go that route! In the interim I'd recommend a poor man's anycast using something like rage4.com, which is DNS geo with TTL set to one second. Not perfect but pretty close. You could also try shoving the whole thing behind cloudflare
<kyledrake>
but that will only work for the HTTP gateway.
<kyledrake>
a124 ^^
<lgierth>
Vytek: note that if you make the gateway or api publicly accessible, anyone can make ipfs fetch stuff on your behalf
<kyledrake>
The really interesting thing would be to try Anycast with IPFS directly, since it's UDP and Anycast works best for UDP (and not well for TCP)
mdangear has quit [Quit: mdangear]
<mythmon>
Kubuxu: what if the node that has some content I want and my node can't make a direct connection? is it relayed through parts of the network?
<lgierth>
kyledrake: hey thanks for the pointers!
<Kubuxu>
mythmon: currently it isn't, but it is planned to allow user to explicitly enable such feature
<Vytek>
I don't understand @lgierth
<lgierth>
mythmon: content is not relayed -- your connection to the other host might be relayed if you enable relaying
<Vytek>
Sorry for my bad english ;-)
<lgierth>
Vytek: the gateway and api can fetch content from the network (i.e. download it)
<lgierth>
so if e.g. your gateway is publicly accessible (like the one on ipfs.io), random people can make requests, and the gateway will download it
<Vytek>
ohhh...ok...How can i close gateway and api to open access??
<kyledrake>
lgierth: I'll send an email with more infos
Oatmeal has quit [Ping timeout: 240 seconds]
avastmick has joined #ipfs
<Kubuxu>
Vytek: it is closed by default
<Vytek>
@Kubuxu @lgierth thank you very much for your info. I start a node right now.
<lgierth>
Vytek: gateway and api are only listening on localhost by default -- so that should be sage
<lgierth>
*safe
<lgierth>
kyledrake: thanks that's super useful!
<Vytek>
Do you have seen the use of ipfs for metaverse? I think that is fantastic!!
<lgierth>
that VR thing?
<lgierth>
i have to admit VR doesn't get me that excited -- probably i'm just a bit ignorant :)
<A124>
I know what it is about, just reading how it is implemented. I had the idea of ubiqitous filesystem, basically BTRFS + IPFS Augmented and more stuff.
<A124>
Thanks, I thought there is another branch or so and the org name confused me. Not sure what happens when you send a file that does not exist via API.
<kevina>
A124: Sending a file that doesn't exist will fail outright.
<A124>
Alright. kevina, do you know BTRFS?
<kevina>
Somewhat.
<A124>
Good. Then you know about COW. If a file has COW mode on (if filesystem is COW, then unless direct IO, it is) you can just reference blocks directly.
ruby32 has quit [Remote host closed the connection]
<A124>
Which would mean, blocks would not become invalid. Also there is a secondary tree for checksums that is currently crc32, but has space for up to 256 bits. Which means SHA, so ... not sure about the sizes of the chunks but if SHA is added to the FS, they would be compatible and create really interesting outcome.
<A124>
Of course one can just skip that and use the blocks, I mean pages. Just a thing to think about.
<kevina>
I am not sure what you are getting at. Are you trying to say with BTRFS you would not need something like the filestore?
<kevina>
One problem I see is that IPFS stores data with a header that might mess up any block level deduplication.
<kevina>
That is a file in the filestore is a protobuf where the file contents is combined with other data.
<A124>
kevina I am saying that a) Copy-on-Write would mean that while the file is referenced by filestore, it just cannot change unless someone f'up administrative commands or there is data corruption. So it should always retain the version. Unpinning would decrement reference count of the extents. (When 0 btrfs garbage collects it) Same thing is used for snapshots and clones.
stoogenmeyer has joined #ipfs
<whyrusleeping>
yeah, even with a COW filesystem, you will still need something like the filestore
<A124>
So filestore would be needed, but data retention would work without duplication. BTRFS has online and ofline dedup (online offline in filesystems means inline and manual). So one could just foolishly use BTRFS and sometimes runn dedup, but that does not sound much "Pro" Also it would dedup the raw blocks, or files unless boundaries differ. Not knowledgable around that.
<whyrusleeping>
no. it still wouldnt work
<whyrusleeping>
btrfs would have a hard time deduplicating
<A124>
kevina And the b) you could use the filesystem directly, or a subvolume to store the meta for filestore, unless there is smarter way. And if SHA is added to BTRFS, one can just rely on the filesystem integrity then.
<whyrusleeping>
fileoffsets wouldnt match expectations, and the framing around blocks would really throw it off
<A124>
Yes, that is what I meant, which goes to b), using the filesystem on lower level to achieve efficiency.
<kevina>
A124, whyrusleeping: My filestore could also be adopted to just store raw blocks without the header info, therefore allowing btrfs to dedup. I think there was even an issue an on this.
stoogenmeyer has quit [Remote host closed the connection]
stoogenmeyer has joined #ipfs
<kevina>
Even without COW, A clever use of hard links will prevent most accidental f'ups.
<A124>
.. This thing was what did bring me back to ipfs. I was sketching out my own filesystem. And after some thoughts it came to me it looks like BTRFS on steroids + IPFS with tons of stuff, and some more. Convergence could achieve a lot here.
<whyrusleeping>
A124: cool, let me know if you write a spec
<kevina>
I now determined my original idea can be built on top of the filestore. The client copies or links files into place and then send the new path to the filestore.
<A124>
whyrusleeping Sure, will. Given my health and other stuff, I did decide to shift my focus improving ecosystem about the only thing I know so far could make a dent in my vision. The spec might never be complete. Disk encryption with forward secrecy, data integrity, snapshots, clones, compression, multihash checksumming in separate btree (BTRFS does it, but crc32), P2P transfers with secure auth [...]
<A124>
[...] and sharing, with different levels of sharing (public, auhenticated, auth+encrypted), and a safe manner to estabilish if two clients have some files in common.
<A124>
One mans work when the man is fragile ain't much future proof. I am very looking forward to WebAssembly and distributed data analytics.
<A124>
kevina chattr +i
<A124>
If that is somehow used in the filestore or manualy by user, that will change almost heavens.
mbrock has quit [Quit: Connection closed for inactivity]
<kevina>
A124: What does "chattr +i" do?
<A124>
Makes files immutable.
<A124>
Aka even if you are root, and if the filesystem supports it, and unless you deliberately remove that flag, you can't do anything with it, except read and execute. (Not sure about moving)
<A124>
kevina ^
<kevina>
I see, I was not aware of that. That might be helpful, but it looks a bit too restrictive. It man page says "no link can be created to this file".
<kevina>
By idea was to create a hard link and than chmod a-w.
<kevina>
The hard link guards against removal or moving, the chmod guards against writes.
<A124>
kevina Maybe you can create the link before setting it +i. I did not use it with links so I never encountered that.
<A124>
(I did use symbolic, though)
<kevina>
Thanks for the idea, I will keep it mind.
<kevina>
The basic idea behind my idea is the user can keep using the file as normal, while also letting the file be shared on IPFS.
stoogenmeyer has quit [Remote host closed the connection]
<A124>
Great. About what I wanted to say, you are the boss with it, but having more tools ain't never bad. Yes, that was my idea also. I am aware of the rename restriction, when it comes down to permanent data I rarely do it.
<A124>
... The mentioned copy file, but do not modify for the dedup does sound good also. Would of course work with ZFS too.
<kevina>
Yeah with a dedup filesystem I could just copy the file and then still allow the user to modify it.
<A124>
I did compile the filestore, but ipfs filestore yells at me.
<kevina>
How so?
<A124>
Error: Unknown Command "filestore"
jedahan has joined #ipfs
stoogenmeyer has joined #ipfs
<kevina>
So what does "ipfs filestore --help" give you. Very very strange.
<A124>
And yes, I did alias the binary. Do I have to alias more then that?
<kevina>
No. Just the binary. What branch are you using?
<A124>
I am using global and local go paths, had ipfs installed, so I did alias the binary, thought that will solve it. Master.
<A124>
Will do the kevina/
<kevina>
Master should be fine. It is the same content as kevina/filestore.
<A124>
Oh then rm -rf the go get does not refresh it seems.
Pent has quit [Quit: Leaving]
<A124>
kevina master does not work, when I did manual clone and checkout it does do something.
cketti has quit [Quit: Leaving]
matoro has quit [Ping timeout: 244 seconds]
matoro has joined #ipfs
<A124>
kevina and when I did add file when node offline, official gw does not return, and local does return folder, but not the file.