aschmahmann changed the topic of #ipfs to: Heads Up: To talk, you need to register your nick! Announcements: go-ipfs 0.7.0 and js-ipfs 0.52.3 are out! Get them from dist.ipfs.io and npm respectively! | Also: #libp2p #ipfs-cluster #filecoin #ipfs-dev | IPFS: https://github.com/ipfs/ipfs | Logs: https://view.matrix.org/room/!yhqiEdqNjyPbxtUjzm:matrix.org/ | Forums: https://discuss.ipfs.io | Code of Conduct: https://git.io/vVBS0
bluto[m] has joined #ipfs
tech_exorcist has quit [Quit: tech_exorcist]
arcatech has joined #ipfs
arcatech has quit [Ping timeout: 245 seconds]
arcatech has joined #ipfs
arcatech has quit [Ping timeout: 252 seconds]
bengates has joined #ipfs
bengates has quit [Ping timeout: 268 seconds]
mowcat has quit [Remote host closed the connection]
baojg has joined #ipfs
drathir_tor has quit [Remote host closed the connection]
drathir_tor has joined #ipfs
baojg has quit [Remote host closed the connection]
baojg has joined #ipfs
psyk has quit [Quit: Leaving]
jcea has quit [Ping timeout: 276 seconds]
MDude has quit [Read error: No route to host]
upekkha has quit [Quit: upekkha]
upekkha has joined #ipfs
jrt is now known as Guest30737
Guest30737 has quit [Killed (tepper.freenode.net (Nickname regained by services))]
jrt has joined #ipfs
<Discordian[m]>
Rome Silvanus: so I'm currently suffering through a lot of files being removed from blockstore too ... I wonder if I can get away with just removing offset 0 references, which should lighten the load a lot for large files...
<Discordian[m]>
* Rome Silvanus: so I'm currently suffering through a lot of blocks being removed from blockstore too ... I wonder if I can get away with just removing offset 0 references, which should lighten the load a lot for large files...
<Discordian[m]>
* Rome Silvanus: so I'm currently suffering through a lot of blocks being removed from filestore too ... I wonder if I can get away with just removing offset 0 references, which should lighten the load a lot for large files...
<Discordian[m]>
I mean long-term, it's for the best anyways I guess...
i_am_hari_seldon is now known as HariSeldon[m]
derpadmin has joined #ipfs
ipfs-stackbot has quit [Remote host closed the connection]
ipfs-stackbot has joined #ipfs
mindCrime_ has quit [Ping timeout: 240 seconds]
jrt is now known as Guest7091
jrt has joined #ipfs
LiftLeft has quit [Ping timeout: 246 seconds]
Guest7091 has quit [Ping timeout: 252 seconds]
jesse22 has joined #ipfs
kesenai has joined #ipfs
opa7331 has quit [Ping timeout: 246 seconds]
opa7331 has joined #ipfs
arcatech has joined #ipfs
arcatech has quit [Ping timeout: 250 seconds]
mindCrime_ has joined #ipfs
ash91 has joined #ipfs
raed has joined #ipfs
ash91 has quit [Client Quit]
mindCrime_ has quit [Ping timeout: 268 seconds]
arcatech has joined #ipfs
arcatech has quit [Quit: Be back later.]
joocain2 has quit [Remote host closed the connection]
joocain2 has joined #ipfs
mindCrime_ has joined #ipfs
mindCrime_ has quit [Ping timeout: 246 seconds]
Adbray has quit [Quit: Ah! By Brain!]
jesse22 has quit [Ping timeout: 246 seconds]
zeden has quit [Quit: WeeChat 3.0.1]
MrSon[m] has joined #ipfs
bengates has joined #ipfs
bengates has quit [Ping timeout: 246 seconds]
maggotbrain has quit [Remote host closed the connection]
maggotbrain has joined #ipfs
plntyk has quit [Quit: Leaving]
abueide[m] has joined #ipfs
k1ltzman has quit [Ping timeout: 240 seconds]
<abueide[m]>
two questions: is there any research being done to make content resolution faster (e.g. finding peers faster while also being scalable)? where could I find some leads on approaches to solving this? correct me if i'm wrong, but in my past experience finding content that wasn't pinned to some cdn like cloudflare or ipfs gateway was extremely slow to resolve.
<abueide[m]>
Second question: If i have a list of libp2p peers, or maybe a list of ip addresses and a public port with an ipfs node on it, is there an easy way to skip the dht and ask x peer for y content first, and then fall back to the dht?
<abueide[m]>
Second question: If i have a list of libp2p peers, or maybe a list of ip addresses and a public port with an ipfs node on it, is there an easy way to skip the dht and ask that list of "trusted" peers first so i can kind of play around with building my own content resolution via social graphs instead of dhts
<abueide[m]>
* two questions: is there any research being done to make content resolution faster (e.g. finding peers faster while also being scalable)? where could I find some leads on approaches to solving this? correct me if i'm wrong, but in my past experience finding content that wasn't pinned to some cdn like cloudflare or ipfs gateway was extremely slow to resolve.
<abueide[m]>
i'm thinking about building an app on top of the matrix protocol that goes something kinda like this: I want to download niche cat picture on ipfs. Instead of crawling dht, I first ask the cats.org matrix homeserver if they know of any peers that are seeding this ipfs hash, and then fall back to dht.
<abueide[m]>
* i'm thinking about building an app on top of the matrix protocol that goes something kinda like this: I want to download niche cat picture on ipfs. Instead of crawling dht, I first ask the example.cats matrix homeserver if they know of any peers that are seeding this ipfs hash, and then fall back to dht.
<abueide[m]>
* i'm thinking of building a matrix plugin which can detect ipfs links in matrix rooms and by downloading it, the plugin will automatically "react" to the ipfs link message. when you stop seeding it, it automatically unreacts to the post. so now you have a list of peers to ask instead of resorting to the dht
bengates has joined #ipfs
Adbray has joined #ipfs
bengates_ has joined #ipfs
bengates has quit [Remote host closed the connection]
pecastro has joined #ipfs
The_8472 has quit [Ping timeout: 276 seconds]
bengates_ has quit [Remote host closed the connection]
bengates has joined #ipfs
baojg has quit [Remote host closed the connection]
baojg has joined #ipfs
The_8472 has joined #ipfs
nsh has quit [Max SendQ exceeded]
kesenai has quit [Ping timeout: 240 seconds]
nsh has joined #ipfs
nsh has quit [Max SendQ exceeded]
jarodit has joined #ipfs
PendulumSwinger4 has joined #ipfs
eaterof has joined #ipfs
treora_ has joined #ipfs
mzal_ has joined #ipfs
PendulumSwinger has quit [Quit: Ping timeout (120 seconds)]
treora has quit [Quit: No Ping reply in 180 seconds.]
<RomeSilvanus[m]>
Can’t you just run that in the background ?
<RomeSilvanus[m]>
Since it doesn’t really block iPad from working
<RomeSilvanus[m]>
* Since it doesn’t really block ipfs from working
royal_screwup21 has joined #ipfs
royal_screwup21 has quit [Quit: Connection closed]
royal_screwup21 has joined #ipfs
fling has quit [Ping timeout: 268 seconds]
royal_screwup21 has quit [Ping timeout: 260 seconds]
tech_exorcist has joined #ipfs
royal_screwup21 has joined #ipfs
jcea has joined #ipfs
royal_screwup21 has quit [Quit: Connection closed]
AnotherFoxGuy[m] has joined #ipfs
royal_screwup21 has joined #ipfs
royal_screwup21 has quit [Ping timeout: 246 seconds]
fling has joined #ipfs
zeden has joined #ipfs
<Discordian[m]>
<RomeSilvanus[m] "Since it doesn’t really block iP"> bad filestore does prevent certain operations from working correctly, actually : /. I'll keep investigating into better solutions. Maybe removing just offset 0 references would allow the gc to get rid of the rest at a later time, but still allow everything else to work correctly.
nikolayclfx has joined #ipfs
Encrypt has joined #ipfs
jokoon has quit [Quit: Leaving]
zeden has quit [Quit: WeeChat 3.0.1]
luxuride[m] has joined #ipfs
Nact has joined #ipfs
jokoon has joined #ipfs
avakand has joined #ipfs
<AmariyahWhite[m]>
Halo
<AmariyahWhite[m]>
hi
zeden has joined #ipfs
royal_screwup21 has joined #ipfs
<McSinyx[m]>
o/
sergiupopescu[m] has joined #ipfs
drathir_tor has quit [Ping timeout: 240 seconds]
royal_screwup21 has quit [Quit: Connection closed]
royal_screwup21 has joined #ipfs
drathir_tor has joined #ipfs
royal_screwup21 has quit [Ping timeout: 260 seconds]
pakxo has quit [K-Lined]
royal_screwup21 has joined #ipfs
mowcat has joined #ipfs
theseb has joined #ipfs
royal_screwup21 has quit [Quit: Connection closed]
royal_screwup21 has joined #ipfs
kiltzman has joined #ipfs
zeden has quit [Ping timeout: 260 seconds]
royal_screwup21 has quit [Client Quit]
royal_screwup21 has joined #ipfs
LiftLeft has joined #ipfs
LiftLeft has quit [Max SendQ exceeded]
LiftLeft has joined #ipfs
zeden has joined #ipfs
royal_screwup21 has quit [Quit: Connection closed]
royal_screwup21 has joined #ipfs
ipfs-stackbot has quit [Remote host closed the connection]
zeden_ has joined #ipfs
royal_screwup21 has quit [Ping timeout: 240 seconds]
ipfs-stackbot has joined #ipfs
zeden has quit [Ping timeout: 246 seconds]
Encrypt has quit [Quit: Quit]
jesse22 has joined #ipfs
jokoon has quit [Ping timeout: 246 seconds]
nikolayclfx has quit [Ping timeout: 260 seconds]
<eleitl[m]>
My little server is still chugging along. 2.51 TiB of 5.65 TiB pinning complete so far, with some 53% memory usage. Looks like it could actually complete within another day or two.
kn0rki has quit [Quit: Leaving]
mowcat has quit [Remote host closed the connection]
AnonzNFT has joined #ipfs
mowcat has joined #ipfs
jokoon has joined #ipfs
mowcat has quit [Remote host closed the connection]
drathir_tor has quit [Ping timeout: 240 seconds]
drathir_tor has joined #ipfs
mowcat has joined #ipfs
jcea has quit [Ping timeout: 250 seconds]
fling has quit [Ping timeout: 260 seconds]
jokoon has quit [Quit: Leaving]
AnonzNFT has quit [Quit: Connection closed]
zeden_ has quit [Quit: WeeChat 3.0.1]
<McSinyx[m]>
what is it eleitl? sorry my client display all join/leave events and my internet is awful to scroll
zeden has joined #ipfs
Acacia has quit [Ping timeout: 252 seconds]
ylp has quit [Quit: Leaving.]
Chicken has joined #ipfs
Chicken has quit [Client Quit]
Acacia has joined #ipfs
piraces_[m] has quit [Quit: Idle for 30+ days]
nikolo22[m] has quit [Quit: Idle for 30+ days]
Bart[m]2 has quit [Quit: Idle for 30+ days]
mowcat has quit [Remote host closed the connection]
Nact has quit [Quit: Konversation terminated!]
SirPHOENiX1 has joined #ipfs
<eleitl[m]>
I'm just pinning a million of files on a 4 GB RAM 1.6 GHz Atom D510 toaster. To see whether it is possible.
fling has joined #ipfs
mindCrime_ has joined #ipfs
impossiball is now known as MrSparkle
bengates has quit [Remote host closed the connection]
supercoven has joined #ipfs
theseb has quit [Quit: Leaving]
royal_screwup21 has joined #ipfs
<RomeSilvanus[m]>
Reeeeeeeee
<RomeSilvanus[m]>
Why isn't there a way to just reset the filestore.
mowcat has joined #ipfs
<Discordian[m]>
I'm going to investigate into offloading most of the work to the gc
<Discordian[m]>
I just got home tbh
<Discordian[m]>
Because you're right, the removal time is insane
<RomeSilvanus[m]>
I just have a grip with most oss project leaving out completely standard features.
<Discordian[m]>
I found out it's actually faster to first unpin everything you don't want pinned, run a full gc, then do cleanup
<RomeSilvanus[m]>
I did unpin everything.
<Discordian[m]>
Yeah full gc too might be worth it too tbh, it's way faster than manually removing the blocks.
<RomeSilvanus[m]>
Unless you mean delete everything under 'files' and not under 'pins'
<Discordian[m]>
No I mean pins, then run `ipfs repo gc`
theseb has joined #ipfs
<Discordian[m]>
I too, don't want to remove my precious MFS
<RomeSilvanus[m]>
I did that. It's till slow.
<Discordian[m]>
Wow that's brutal
theseb has left #ipfs [#ipfs]
<Discordian[m]>
I was looking at possibly hours, ran GC instead, and was up-and-running like like 10mins
<RomeSilvanus[m]>
It did add a few tens of millions files before though. But there is tbh literally no reason to not remove these when you unpin it.
<Discordian[m]>
* I was looking at possibly hours, ran GC instead, and was up-and-running in like 10mins
<Discordian[m]>
Right?! I feel that, I think this is why filestore is still in experimental
<Discordian[m]>
It's these types of things they're probably working on improving
<Discordian[m]>
At least with ipfs-sync it's easier to stress it, and test it out, just sucks a bit for now
dpl has joined #ipfs
Newami has joined #ipfs
Newami has quit [Remote host closed the connection]
jcea has joined #ipfs
bengates has joined #ipfs
royal_screwup21 has quit [Quit: Connection closed]
royal_screwup21 has joined #ipfs
Encrypt has joined #ipfs
bengates has quit [Ping timeout: 246 seconds]
ZaZ has joined #ipfs
tech_exorcist has quit [Remote host closed the connection]
tech_exorcist has joined #ipfs
royal_screwup21 has quit [Ping timeout: 260 seconds]
tech_exorcist has quit [Remote host closed the connection]
tech_exorcist has joined #ipfs
<franc1s>
<eleitl[m] "My little server is still chuggi"> What are the specs of your server @eleit?
mowcat has quit [Remote host closed the connection]
<franc1s>
<franc1s "What are the specs of your serve"> Just read it. Scrolling is slow today.
<RomeSilvanus[m]>
Hmm weird
<RomeSilvanus[m]>
I removed the mfs stuff ipfs-sync did and then restarted and now it's just hanging at the `2021/03/30 19:31:17 Removing blocks that point to a file that doesn't exist from filestore...` step
<RomeSilvanus[m]>
* I removed the mfs stuff ipfs-sync did (so I have no pins and no files) and then restarted and now it's just hanging at the `2021/03/30 19:31:17 Removing blocks that point to a file that doesn't exist from filestore...` step
<RomeSilvanus[m]>
Previously it started removing instantly
<Discordian[m]>
Oh that hopefully won't take long, it's running `filestore/verify`, and filtering for `no-file` AKA status 11. If it doesn't find status 11, it won't remove blocks, but it will receive the whole list that command gives out.
<RomeSilvanus[m]>
So it's only because I removed the mfs stuff?
<Discordian[m]>
Idk IPFS must have removed the filestore references
<Discordian[m]>
I've been struggling myself to figure out what causes it to clean up automatically
<Discordian[m]>
That's why it runs `filestore/verify` now, but that command is slow because I can't ask it for status 11, I need to receive EVERY listing in filestore, and filter those.
<RomeSilvanus[m]>
I see the problem. ipfs-sync didn't respect the exit signal and is still running
<Discordian[m]>
Ah, I exit just by terminating the process
bengates has joined #ipfs
<Discordian[m]>
I could probably add in proper signal handling, should do that eventually.
<Discordian[m]>
I DMd you BTW
<RomeSilvanus[m]>
I run it on Cronicle so idk what cronicle sends to the process
<RomeSilvanus[m]>
Also no, it's still hanging
<RomeSilvanus[m]>
Hmm
<Discordian[m]>
Could do `ps -A` to get the process ID, then just `kill -9` it
<Discordian[m]>
or `pkill -9 ipfs-sync`
<RomeSilvanus[m]>
I just hopped into the tty and killed it there
<RomeSilvanus[m]>
Also I didn't mean the process is hanging but the step its on
<RomeSilvanus[m]>
Unless its parsing soemhing but there's no output
<RomeSilvanus[m]>
* Unless its parsing something but there's no output
<Discordian[m]>
Yeah that step runs a whole `filestore/verify` if it doesn't find bad blocks, it doesn't output
<Discordian[m]>
It's just slow because it needs to receive the entire filestore, or at least most of it, it seems.
<Discordian[m]>
It only looks at the status code though, unless it's 11.
bengates has quit [Ping timeout: 246 seconds]
<RomeSilvanus[m]>
Oh, so because it removed so many already it's kinda just going through all the good ones until it finds a bad one. Whereas before it had bad ones right at the star?
<Discordian[m]>
Yeah probably
<Discordian[m]>
Not sure how long it'll take for you, if I have no bad blocks, takes between 10-40s usually.
<Discordian[m]>
Yeah, there really should be an index on the status codes, so that way I could quickly return just status 11 filestore references.
royal_screwup21 has joined #ipfs
<Discordian[m]>
Going to have to look into how filestore works and submit a PR me thinks.
<RomeSilvanus[m]>
Maybe I should just nuke my ipfs folder
<Discordian[m]>
oof, I mean, if you have nothing useful stored, it'll be faster
<Discordian[m]>
Man, the fact a full verify scan takes so long for you too, even without bad blocks, that's just painful to hear
<Discordian[m]>
I'm wracking my head here to think of optimisations on that front.
Arwalk_ has joined #ipfs
<RomeSilvanus[m]>
I'll try it
Arwalk has quit [Ping timeout: 240 seconds]
Arwalk_ is now known as Arwalk
<RomeSilvanus[m]>
I was right at giving up on ipfs because after using it for like half an hour I found it absolutely not scaleable. That's when I joined here and you mentioned your application.
<Discordian[m]>
I mean, it'll scale, just maybe not all on one machine. ipfs-cluster spread out to several machines would accomplish the goal. Just wouldn't be the same setup
<Discordian[m]>
It's also just young, the problems you have are solvable, they just haven't been solved yet
drathir_tor has quit [Ping timeout: 240 seconds]
jarodit has joined #ipfs
<Discordian[m]>
Also filestore is still experimental unfortunately, as cool and useful of a feature as it is. I'll be investigating into writing a PR though, because an index would be SO useful rn.
<RomeSilvanus[m]>
I already nuked my ipfs folder :v
<Discordian[m]>
It's cool, this needs to be done anyways
<Discordian[m]>
Even when you do eventually finally get your add done, verification will happen sometimes, it'd be nice if that didn't take 1 trillion years
<Discordian[m]>
I can build an index, I found the code that sets file status. I can output the index, found the code that does verify. Just need to modify the HTTP API, and we're golden.
<Discordian[m]>
Suppose I also need to look into how the DB is used to see how I'm going to store the index 🤔. However this is much cleaner than I expected, I'm impressed.
<Discordian[m]>
I see now, verify actually does a stat on every file in the filestore, and generates the status on the fly, it's not actually storing much rn that I see.
royal_screwup216 has joined #ipfs
<Discordian[m]>
Doesn't store status anyways ...
<Discordian[m]>
So ... what I can do instead, is when a file is going to be added, I can verify just the first block it points to, and if it's good, then continue with the add.
<RomeSilvanus[m]>
Why don't they just remove the ref when you remove the file?
<Discordian[m]>
I assume because it wasn't an issue without filestore
<RomeSilvanus[m]>
D:
pakxo has joined #ipfs
is_null has joined #ipfs
arcatech has joined #ipfs
<Discordian[m]>
You're right though, IMO the last abspath should be the one used, not the first
<Discordian[m]>
Or store both somehow, and use whichever one worked last
royal_screwup21 has quit [Quit: Connection closed]
royal_screwup216 has quit [Quit: Connection closed]
<Discordian[m]>
* Or store both somehow, and use whichever one worked last (like the verifier works well enough, it could do this)
royal_screwup21 has joined #ipfs
<Discordian[m]>
Anyways, I think I have a workable solution without modifying upstream for now, just gotta test out a couple ideas to make it as fast as humanly possible.
<Discordian[m]>
Also I like that solid analysis of the 4 steps to break filestore I wrote above, so maybe I can open an issue and get more expert opinions.
royal_screwup21 has quit [Ping timeout: 240 seconds]
<RomeSilvanus[m]>
At this point it was that Rome realized he made a grave mistake in issuin 'rm -rv' on the '.ipfs' directory.
<RomeSilvanus[m]>
I bet it takes as long to finish as if I had just run repo gc
<RomeSilvanus[m]>
* At this point it was that Rome realized he made a grave mistake in issuing 'rm -rv' on the '.ipfs' directory.
<Discordian[m]>
Lmfao how? Doesn't it just unlink?
<RomeSilvanus[m]>
At like 5 files per second
<RomeSilvanus[m]>
There's a lot of files in /blocks/
<Discordian[m]>
Lmao hot damn
<Discordian[m]>
My blockstore has under 600 files, over 1k dirs
<RomeSilvanus[m]>
There's a lot of .data files, which I assume to be the hashes of the files
<Discordian[m]>
I think `.data` usually contains actual data. I've always assumed `datastore` contains the hashes (has leveldb files)
<RomeSilvanus[m]>
Hm, idk, maybe it's some leftover from when I didnt have filestore enabled
<Discordian[m]>
I'm thinking maybe this removal of the ipfs dir is for the best
<RomeSilvanus[m]>
Probably
<Discordian[m]>
If I don't have time tonight, I will tomorrow to get a patch rolling to skip verification altogether forever.
<Discordian[m]>
It'll just verify files when they get added to IPFS, if it find a bad block after the add, just add again if the file exists.
<Discordian[m]>
* It'll just verify files when they get added to IPFS, if it find a bad block after the add, just add again if the file exists (after removing the bad block(s) obviously).
<RomeSilvanus[m]>
You said it causes all sorts of problems earlier though
<Discordian[m]>
It would if I didn't remove the blocks, but I can remove them on a case-by-case basis, it'll just slow the add step down a bit.
<Discordian[m]>
It's a new idea I got reading through the source code
<Discordian[m]>
Coupled with an idea where I may only need to remove one block per file, we'd be laughing compared to the speed now
<RomeSilvanus[m]>
Idk, why this is even a problem in the first place. If the daemon gets a request for a file it has to look in in's Db anyway and then it could also just check if the file exists.
<RomeSilvanus[m]>
* Idk, why this is even a problem in the first place. If the daemon gets a request for a file it has to look in in's Db anyway and then it could also just check if the file exists and just say no, I dont to the other peers if not.
<Discordian[m]>
But the issue is it refers to the first abspath stored, once that abspath no longer exists, there's absolutely no mechanism to detect it happens until a problem arises. Because the second abspath wasn't stored (it at least isn't used if it is), you get errors.
<Discordian[m]>
It's just an oversight IMO, caused by the feature being experimental, with little feedback (the devs have asked for feedback).
teaman has quit [Read error: Connection reset by peer]
<RomeSilvanus[m]>
Well then! At least you should have enough feedback now!
theseb has quit [Quit: Leaving]
<Discordian[m]>
Lmao yeah, that's why I opened that issue, to give the feedback
mowcat has joined #ipfs
arcatech has quit [Quit: Be back later.]
usopu[m] has joined #ipfs
supercoven has quit [Ping timeout: 268 seconds]
Encrypt has joined #ipfs
ZaZ has quit [Read error: Connection reset by peer]
jeanlouie has joined #ipfs
wompt[m] has joined #ipfs
arcatech has joined #ipfs
royal_screwup21 has quit [Quit: Connection closed]
royal_screwup21 has joined #ipfs
arcatech has quit [Ping timeout: 276 seconds]
royal_screwup21 has quit [Ping timeout: 252 seconds]
afx__ is now known as afx
b-m-f[m] has joined #ipfs
joocain2 has quit [Ping timeout: 240 seconds]
joocain2 has joined #ipfs
arcatech has joined #ipfs
PhilippMayr[m] has joined #ipfs
arcatech has quit [Ping timeout: 252 seconds]
PhilippMayr[m] is now known as PMTH[m]
royal_screwup21 has joined #ipfs
dpl has quit [Ping timeout: 240 seconds]
dpl has joined #ipfs
royal_screwup21 has quit [Ping timeout: 260 seconds]