stebalien changed the topic of #ipfs to: Heads Up: To talk, you need to register your nick! Announcements: go-ipfs 0.4.18 and js-ipfs 0.33 are out! Get them from dist.ipfs.io and npm respectively! | Also: #libp2p #ipfs-cluster #filecoin #ipfs-dev | IPFS, the InterPlanetary FileSystem: https://github.com/ipfs/ipfs | Logs: https://view.matrix.org/room/!yhqiEdqNjyPbxtUjzm:matrix.org/ | Forums: https://discuss.ipfs.io | Code of Con
trqx has quit [Remote host closed the connection]
cratliff has joined #ipfs
trqx has joined #ipfs
jhulten has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
aldenp has joined #ipfs
jhulten has joined #ipfs
cratliff has quit [Ping timeout: 250 seconds]
jhulten has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
jhulten has joined #ipfs
T3rran has joined #ipfs
jhulten has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
Papa__Alpaka has joined #ipfs
Papa__Alpaka has quit [Remote host closed the connection]
Papa_Alpaka has quit [Ping timeout: 245 seconds]
jhulten has joined #ipfs
jhulten has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
jhulten has joined #ipfs
jhulten has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
gritzko has quit [Ping timeout: 246 seconds]
jhulten has joined #ipfs
jakobvarmose has quit [Quit: ZNC 1.6.5+deb1 - http://znc.in]
gritzko has joined #ipfs
jesse22 has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
hcs^ has quit []
abaiste has joined #ipfs
gritzko has quit [Ping timeout: 268 seconds]
gritzko has joined #ipfs
xcm is now known as Guest21039
Guest21039 has quit [Killed (livingstone.freenode.net (Nickname regained by services))]
xcm has joined #ipfs
gritzko has quit [Ping timeout: 244 seconds]
gritzko has joined #ipfs
ctOS has joined #ipfs
<ctOS>
Hi, say I’ve got two directories (dir1 an dir2) that contains all the same files but dir1 contains one additional file. If i put both directories into dir3 IPFS object; how do I get the accurate size for dir3? Considering that both dir1 and dir2 contain mostly the same content hashes almost all data is deduplicated. `ipfs object stat $dir3_hash` shows me the cumulative non-deduplicated size of the two directories.
jhulten has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
<Swedneck>
Wouldn't it be basically nothing
clemo has quit [Ping timeout: 272 seconds]
aldenp has quit [Ping timeout: 246 seconds]
jesse22 has joined #ipfs
gritzko has quit [Ping timeout: 246 seconds]
<ctOS>
Swedneck: was that in answer to my question?
gritzko has joined #ipfs
jesse22 has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
<Kolonka[m]>
yeah, deduplication should make it nothing other than some headers I think
hurikhan77 has quit [Ping timeout: 252 seconds]
hurikhan77 has joined #ipfs
<ctOS>
Kolonka[m]: but how do I get the deduplicated size of dir3?
detran has joined #ipfs
<Kolonka[m]>
alas, I do not know
MDude has quit [Ping timeout: 250 seconds]
hurikhan77 has quit [Ping timeout: 252 seconds]
hurikhan77 has joined #ipfs
<deltab>
you'd need to get all the block hashes of the files into a set, and total the size of the blocks
<deltab>
which I believe is what du does with files
T3rran has left #ipfs [#ipfs]
<postables[m]>
ctOS: you won't be able to get the deduplicated size of dir3, since you're not actually viewing dir3 as it exists on your disk or whatever. You're viewing dir3 which has a link to the files that already exist.
<postables[m]>
at leas thats what i think should be the case.
<postables[m]>
whenever you add something to IPFS you're generating a `merkle-dag` and I believe in the example you gave you'll be generating a `merkle-dag` for dir3 that has a link to the duplicated content you're talking about
<postables[m]>
as opposed to generating a `merkle-dag` which is re-adding the contents and creating a larger `merkle-dag` that's pointing to re-added data
silotis has quit [Ping timeout: 250 seconds]
<ctOS>
I found out how to do it, but it’s not pretty and requires you to know the chunk size in advance.
<ctOS>
Summary of why I was interested in this issue: “This would be very useful to pinning services; all of which currently overcharge their customers for the CumulativeSize of pinned objects and not the actual storage space used. The IPFS community would benefit from more accurate storage accounting and cheaper pin storage services.”
<ctOS>
Plus you’d be able to tell how much disk space it takes to store any given object.
xnaas has joined #ipfs
zane has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
<postables[m]>
hey now my pinnning service is the cheapest there is 😁 but this would be dope as fuck and let me make it even cheaper. I'll investigate into whether or not its possible to clean up your dirty solution so i can add that functionality 🔥
<ctOS>
postables[m]: which one is that?
Belkaar has quit [Read error: Connection reset by peer]
xnaas has quit [Client Quit]
xnaas has joined #ipfs
<postables[m]>
are you in the your-projects channel? rather not clutter up the ipfs channel with my project
<postables[m]>
*edit:* ~~are you in the your-projects channel? rather not clutter up the ipfs channel with my project~~ -> are you in the your-projects channel? rather not clutter up the ipfs channel with talk of my project
<ctOS>
postables[m]:I sent you a DM
<postables[m]>
does your solution work with the default chunking size
<postables[m]>
alas i am on the proprietary discord and can't receive your message 😦
shoku has joined #ipfs
<postables[m]>
if it works with the default chunking size, or even a semi-configurable chunking size i can probably write a quick golang helper function
Belkaar has joined #ipfs
Belkaar has joined #ipfs
}ls{ has quit [Ping timeout: 268 seconds]
thomasan_ has joined #ipfs
<ctOS>
postables[m]: please join me in #whatwaswrongwithirc :P
postables[m]1 has joined #ipfs
<postables[m]>
actually i realize i have a matrix account sent you a pm. IRC requires me to open ports on my DC firewall 😛
thomasan_ has quit [Remote host closed the connection]
<postables[m]>
and i'd rather not open that port nor disable suricata from blocking that kind of stuff lol
<postables[m]>
when i went to that website noscript yelled at me, https everywhere was not happy, and neither was ublockorigin.
jacksap[m] has joined #ipfs
ylp has joined #ipfs
ikari` has joined #ipfs
marcocastignoli4 is now known as marcocastignoli2
marcocastignoli2 is now known as marcocastignoli6
Novitsh has quit [Ping timeout: 250 seconds]
Novitsh has joined #ipfs
gritzko has quit [Ping timeout: 244 seconds]
marcocastignoli has joined #ipfs
marcocastignoli has quit [Client Quit]
marcocastignoli has joined #ipfs
marcocastignoli has quit [Quit: Leaving]
marcocastignoli has joined #ipfs
marcocastignoli has quit [Client Quit]
shakalaka has quit [Ping timeout: 252 seconds]
BeerHall1 has joined #ipfs
BeerHall has quit [Ping timeout: 246 seconds]
littleebee has joined #ipfs
clemo has joined #ipfs
vandemar has joined #ipfs
gmoro has joined #ipfs
spinza has quit [Quit: Coyote finally caught up with me...]
gmoro has quit [Remote host closed the connection]
gmoro has joined #ipfs
as_ has joined #ipfs
spinza has joined #ipfs
ctOS has quit [Quit: Connection closed for inactivity]
as_ has quit [Ping timeout: 264 seconds]
as_ has joined #ipfs
as_ has left #ipfs [#ipfs]
chiui has joined #ipfs
bakerio has joined #ipfs
<bakerio>
Hi guys, I'm trying to implement a simple IPFS Client for adding and fetching files on a Nucleo Board with RIOT. For this I have an IPFS node running on my computer to which the board is connected with an ethernet shield. Can I directly interact with the nodes' API using UDP/TCP without HTTP?
<Swedneck>
riot as in the matrx client or something else?
<nickjohnson>
lidel: That page confuses me, though; it says "If it is 46 characters long and starts with Qm..., it's a CIDv0. Decode it as base58btc and continue to step 2."
<nickjohnson>
But earlier on the same page an example CID is given, which starts with zb2
<nickjohnson>
Eh, never mind, that's a CIDv1
<lidel>
ipfs cid --help
<lidel>
may also be useful
<seba->
isn't everything a cid then hm
<nickjohnson>
Yeah, I was asking about CIDv1, since v0 is just multihash.
<nickjohnson>
But since the difference is explicit, I guess we can just use v0 in our text format.
<seba->
:>
<seba->
nickjohnson would you want to do a few tests with me regarding IPFS performance?
<nickjohnson>
Ping me on Monday? I don't have time today, sorry.
<seba->
oh it's ok
<seba->
np :)
Lieuwex has quit [Quit: ZNC 1.8.x-git-91-b00cc309 - https://znc.in]
Caterpillar has quit [Quit: You were not made to live as brutes, but to follow virtue and knowledge.]
Lieuwex has joined #ipfs
Guest29947 has joined #ipfs
Guest29947 has quit [Excess Flood]
Guest29947 has joined #ipfs
Guest29947 has quit [Max SendQ exceeded]
Guest29947 has joined #ipfs
Guest29947 has quit [Read error: Connection reset by peer]
<postables[m]>
@ctOS: I think I found a way to replicate your github issue without actualyl needing new functionality
<ctOS>
The expected sizes mentioned in the issue is unpacked blocks. IPFS doesn’t pack storage blocks, so everything is done in blocks of 262158 bytes. (Or whatever the blocksize of the object was set to.)
<ctOS>
I may have forgotten to include the IPFS directory objects in my totals.
pep7 has quit [Remote host closed the connection]
zeden has quit [Read error: Connection reset by peer]
zeden has joined #ipfs
florianH_ has quit [Quit: Connection closed for inactivity]
jesse22 has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
jesse22 has joined #ipfs
<postables[m]>
hmm i see, i think the solution i mocked up works though no? it's pretty darn accurate with your base line and its not expecting chunk sizes or anything
<postables[m]>
oi thats some intense shell piping lol
<ctOS>
gets the job done.
<postables[m]>
hmm that won't run in the shell for me (i replaced CID with the example hash from your github issue. Did you take a look at the sample code i posted?
<ctOS>
postables[m]: missing a trailing ' at the end
<postables[m]>
nvm got it to work
<postables[m]>
yup that matches with the cumulative data size from my golang version of this
<postables[m]>
🔥
renrutnnej[m] has joined #ipfs
<ctOS>
postables[m]: so all you’re missing is to add every pin a customer makes into one IPFS object per customer and run that function of your's on that object. Queue triumphant music.
<postables[m]>
yea thats pretty easy, would just need to look up in the database what's been pinned by said customer and we'll be golden
<postables[m]>
i've thought about using IPLD objects to track this kind of stuff, but from a privacy end of things that seems a little.... iffy
<ctOS>
postables[m]: you can bill that development time under GDPR data-portability too.
<postables[m]>
i mean AFAIK (legal conversations under way to ensure we're good) since we just store our stuff in Canada we're fine
<postables[m]>
GDPR only applies if the data is hosted out of the EU ya?
<ctOS>
postables[m]: GDPR applies if the data subject (aka. the customer) is an European Economic Are (EEA) citizen regardless of where the company (or the data subject) is operating from.
<postables[m]>
ah cool good to know i'll double up on our research before we talk with the lawyers next time.
<ctOS>
postables[m]: as a hosted storage provider, you should be sure to comply with Article 20. that's the one your customers are likely to get annoyed by and start filing complaints.
* ctOS
is not a lawyer
<postables[m]>
that's why we've got our talks with the lawyer started about all this 😄
Algonkian has joined #ipfs
chopchapa[m] has joined #ipfs
arthuredelstein has quit [Ping timeout: 245 seconds]
lnostdal has joined #ipfs
arthuredelstein has joined #ipfs
dimitarvp` has quit [Quit: Bye]
chopchapa[m] has left #ipfs ["User left"]
silverfisk[m] has joined #ipfs
maxzor has joined #ipfs
bongozig has joined #ipfs
shakalaka has joined #ipfs
<TimMc>
If two customers pin the same object, who gets charged?
<TimMc>
Maybe both, but you kindly offer a discount? :-)
<postables[m]>
why would there be a discount? that's entirely unfair to the person who originally pinned.
<postables[m]>
the issue ctOS is talking about is if someone has already pinned something, and then they pin another object which references the previous object they pinned. In that case, they would not be charged for data-size of the previous pin
<postables[m]>
so user A pins objects, A,B,C.
<postables[m]>
At a later date they pin object D, which references object A+B. Without a check for the issue ctOS has pointed out, you would calculate the total cumulitive size of object D referencing A+B. The patch I suggest on the github issue would only calculate cost of D, without the reference to A+B
<postables[m]>
well that patch in and of its self wouldn't, some additional processing is needed to lookup the references objects, and check a users previous pinset
JustMaier[m] is now known as JustMaier7893[m]
JustMaier7893[m] is now known as JustMaier[m]
<postables[m]>
for example TimMC, say you pinned a 10GB object and you were charged full price. Then ctOS comes along and pins the same 10GB object, but gets a 25% discount because you pinned it. That's better for ctOS because he gets a discount, but you get the short end of the stick
purisame has quit [Ping timeout: 268 seconds]
<ctOS>
TimMc: both get charged, but my point has been that you don’t charge the same customer multiple times for referencing the same block in the same IPFS object or even from another IPFS object. For an IPFS hosting services, I expect to get the benefits of IPFS.
purisame has joined #ipfs
<ctOS>
postables[m]: the revised version of my script also works for any block size. I didn’t think to recursively count the deduplicated object DataSize. but that should indeed work for any block size.
<TimMc>
postables[m]: It's not a recurring charge>
<TimMc>
*charge?
JustMaier[m] is now known as JustMaier7893[m]
JustMaier7893[m] is now known as JustMaier[m]
<postables[m]>
ctOS: sweet, this was a real quick fix
<TimMc>
I agree that this is a straightforward improvement when the same object is transitively pinned by multiple roots by the *same* customer.
<postables[m]>
TimMC: what do you mean by recurring?
<ctOS>
TimMc: what do you mean? you usually pay per gigabyte/day or per gigabyte/month. I’m arguing in
<TimMc>
In a day (or month) when two users are pinning the same thing, you could charge each of them half the cost for that pin.
<TimMc>
In a time period where only one person is asking for it to be pinned, charge that one person full price. :-)
<TimMc>
I dunno, it was just a thought. It's something that might look weird from the outside if you charge both people full price at the same time.
<Obo[m]1>
Something to consider guys: Is your fix handling the edge case where say the users pins Objects A,B,C and then later pin D, which references A + B. And then after your system marks the size for object D (effectively D - A - B), that user then deletes objects A & B?
<Obo[m]1>
I'd hate to see people effectively scamming you.
<TimMc>
ctOS: Yeah, I get the context. I'm familiar with it from other deduplicated storage systems, including Tarsnap backups and Clojure's persistent data structures.
<Obo[m]1>
To elaborate, if they delete A & B, is your system going back and retroactively changing the size of D?
<postables[m]>
based off how pinning works without doing any kind of test, if A+B are pinned by user A, and A+B are pinned recursively by another user through object D, the first user can remove the pin from their account, but they will still persist because they are referenced by object D from the other user
PyHedgehog has quit [Ping timeout: 252 seconds]
<Obo[m]1>
Correct, but does your database account for that?
<Obo[m]1>
I'm not saying the data will be deleted from IPFS (that's handled natively like you mentioned)
codewaffle has quit [Quit: bye]
<Obo[m]1>
I'm saying be careful with the accounting side of things. It could be a weird edge case (I don't know your codebase very well, just something I thought of)
PyHedgehog has joined #ipfs
<postables[m]>
I guess if you let people run raw removal commands that would be a concern but we don't do that. Backend logic that runs once a day handles removal of pins
<TimMc>
It shouldn't be hard to do correctly with a naive approach where you take a nightly snapshot of the pins, walk the DAGs, and charge each person for the set of objects seen during the walk.
<TimMc>
and I don't know that there's any need for something more clever than that...
<ctOS>
If anyone wants to play with this, you can have a go with the test object provided in https://github.com/ipfs/go-ipfs/issues/5910 <- it has some duplicated files and even a file consisting of duplicated blocks found in the other files.
codewaffle has joined #ipfs
<ctOS>
Your billed 36 MB at Eternum.io and Pinata.cloud for 15 MB of actual data blocks, which is what set of this whole discussion.
<ctOS>
Use case: pinned directory of blog posts. then add a new post and regenerate some index files, and pin the directory again. most data will be unchanged yet you’re billed twice instead of just the cost of the new and regenerated files. This is not how IPFS is supposed to work (and definitely not how I want to be billed for it!)
aldenp has quit [Quit: Leaving]
<Obo[m]1>
Could a cleaner way to do this be utilizing ipfs object patch?
<ctOS>
Obo[m]1: sure. make one object with everything the customer pins / unpins and count the deduplicated blocks.
<Obo[m]1>
just brainstorming right now
<ctOS>
Yeah, that is the solution I’m suggesting at the end of the blog post.
<ctOS>
I’m just a disgruntled overcharged customer :P
vyzo has quit [Quit: Leaving.]
<Obo[m]1>
lol you're fine. you bring up some really good points
<ctOS>
Eternum, Pinata cloud, and postables[m]'s upcoming service have all been really positive to the concept of fixing their storage accounting.
vyzo has joined #ipfs
<Obo[m]1>
I'm just trying to make sure I don't rush into a fix that will open us up for abuse or slow our system to a crawl
<ctOS>
Hopefully pinning services will soon actually behave like an IPFS storage provider and not just a legacy dumb storage provider.
hurikhan77 has joined #ipfs
Aaron101- has left #ipfs [#ipfs]
Aaron101- has joined #ipfs
Aaron101- has left #ipfs [#ipfs]
<ctOS>
Storing an old version of a directory with hundreds of megabytes should only cost a few kilobytes when most of the blocks are duplicated in a newer version of that directory.
spinza has quit [Quit: Coyote finally caught up with me...]
smoku[m] has joined #ipfs
<Obo[m]1>
Something I'm curious about here, is there a reason you can't just delete the old copy?
<Obo[m]1>
for example, @ctOS, would it be beneficial for us to have a "replace" option for our uploads?
<Obo[m]1>
or a "patch" option?
<ctOS>
Obo[m]1: how can IPFS be "the permanent web" if you delete old copies?
<ctOS>
go-ipfs pin update $old_cid $new_cid
<ctOS>
None of the current pinning services have an API for that, though.
<Obo[m]1>
would something like that be useful for you?
<ctOS>
Obo[m]1: but yes. you can’t unpin because of DNSLink or IPNS caching.
roygbiv has joined #ipfs
<ctOS>
Clients will cache for several hours plus the DNS TTL caching.
<ctOS>
So you need to keep the old pin around for hours before unpinning it
<ctOS>
If I remember correctly, the IPFS Companion browser extension caches up to 128 entries for 12 hours.
<ctOS>
And the storage cost of keeping it around is supposed to be negligible.
<Obo[m]1>
Wouldn't your old pinned objects still be contained in the new root?
<ctOS>
So you have to schedule a deletion hours n the future at the earliest.
jhulten has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
<ctOS>
Obo[m]1: no.
cratliff has left #ipfs [#ipfs]
hexfive has joined #ipfs
jhulten has joined #ipfs
<ctOS>
say you add file5 to dir1 and rehash it. you’d have a new hash, and a new object: dir2.
<ctOS>
The references in dir1 and dir2 are deduplicated in IPFS so it costs a few bytes to keep dir1 around.
<Obo[m]1>
ahh, it's the root hash you're concerned about staying around
<ctOS>
well, any hash that I’m distributing to the world, really.
<ctOS>
Obo[m]1: have a look at this object. https://cloudflare-ipfs.com/ipns/media.ctrl.blog/ <- up to 256 dirs each containing up to 256 sub dirs each containing a dir which contains a bunch of image files.
<ctOS>
I’m adding about a megabyte to the file and to my local ipfs node repo. However, pinning services would charge me 590 MB for that 1 additional MB.
<ctOS>
The three new directory objects_ root, changed_dir_level_1, and changed_dir_level_2, plus all the new image files.
<ctOS>
Keeping the old directory pinned would ensure all the indirectly pinned child objects remained accessible.
<Obo[m]1>
That definitely makes sense
<Obo[m]1>
I'll keep brainstorming solutions to this. I want to make sure any solutions we come up with aren't rushed
<Obo[m]1>
thanks for your input today
<ctOS>
Anyone with a blog or static site generator would run into this.
<Obo[m]1>
really appreciate you bringing this all up
<ctOS>
Plus this is important to achieve one of the IPFS project goals "the permanent web"
<ctOS>
The permanent web can’t be expensive. :P
<Obo[m]1>
haha everything comes at a cost
<Obo[m]1>
But yes, we definitely can get our costs down
<ctOS>
Obo[m]1: do you operate a pinning service?
alexgr has quit [Quit: Leaving]
alexgr has joined #ipfs
<Obo[m]1>
I'm the CTO of pinata.cloud lol
<Obo[m]1>
we've been conversing on a few different platforms today
Papa_Alpaka has joined #ipfs
<ctOS>
Obo[m]1: oh, hi.
maxzor has quit [Remote host closed the connection]
<ctOS>
Obo[m]1: do you take feature suggestions? :P I talked with postables[m] about this one yesterday for his pinning service : A DNSLink mirroring service. Your customers would enter a DNSLink enabled domain name on your website (e.g. media.ctrl.blog), and you check that DNSLink record when it's time-to-live (TTL) expires and automatically pins the new content ID.
purisame has quit [Ping timeout: 272 seconds]
alpha has joined #ipfs
<ctOS>
Makes it easier for folks to deploy IPFS enabled websites with automated pinning. No need to mess around with your APIs or web portal.
purisame has joined #ipfs
<ctOS>
(Would obviously depend on either hosting at least 1 old IPFS object for free, or deduplicated storage accounting.)
spinza has joined #ipfs
<naszam[m]>
Any way to get an IPFS laptop sticker?
alpha has quit [Remote host closed the connection]
<naszam[m]>
Ok, Thanks!
<naszam[m]>
👍
<ctOS>
naszam[m]: you’ll find sticker printing services on the web. about 4 usd for an uncut sheet the last time I looked
<ctOS>
Cutting is more expensive, but I’m sure you have scissors. :P
goiko has quit [Ping timeout: 250 seconds]
aldenp has joined #ipfs
<ctOS>
Obo[m]1: check your mailbox from time to time. I actually sent you an email about an update/replace API some day late in December. I just remembered.
roygbiv has quit [Quit: ™]
goiko has joined #ipfs
jhulten has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
mischat has joined #ipfs
jhulten has joined #ipfs
aldenp has quit [Ping timeout: 250 seconds]
jhulten has quit [Client Quit]
jhulten has joined #ipfs
<Obo[m]1>
Well shoot. Sorry about that. Apparently it ended up in the spam folder
<Obo[m]1>
I wish I would have seen that too
<ctOS>
Obo[m]1: since you didn’t respond to that one, I didn’t reach out to you regarding the storage accounting issue either.
<Obo[m]1>
completely understandable
<Obo[m]1>
I've added you to my contacts
<Obo[m]1>
so hopefully any future emails will come through
aldenp has joined #ipfs
kaminishi has quit [Remote host closed the connection]
<ctOS>
Obo[m]1: maybe auto-whitelist every email found in the customer database? :-P