aschmahmann changed the topic of #ipfs to: Heads Up: To talk, you need to register your nick! Announcements: go-ipfs 0.7.0 and js-ipfs 0.52.3 are out! Get them from dist.ipfs.io and npm respectively! | Also: #libp2p #ipfs-cluster #filecoin #ipfs-dev | IPFS: https://github.com/ipfs/ipfs | Logs: https://view.matrix.org/room/!yhqiEdqNjyPbxtUjzm:matrix.org/ | Forums: https://discuss.ipfs.io | Code of Conduct: https://git.io/vVBS0
Guest1714 has quit [Killed (card.freenode.net (Nickname regained by services))]
jrt has joined #ipfs
Nukien2 has joined #ipfs
Nukien2 has quit [Ping timeout: 246 seconds]
MDude has quit [Quit: Going offline, see ya! (www.adiirc.com)]
mindCrime_ has joined #ipfs
mindCrime has quit [Ping timeout: 265 seconds]
D_ has quit [Quit: 💨]
D__ has joined #ipfs
chachasmooth has quit [Ping timeout: 258 seconds]
ipfs-stackbot has quit [Remote host closed the connection]
ipfs-stackbot has joined #ipfs
chachasmooth has joined #ipfs
Mike7 has joined #ipfs
jcea has quit [Ping timeout: 252 seconds]
nicbk[m] has joined #ipfs
mindCrime_ has quit [Ping timeout: 268 seconds]
maggotbrain has joined #ipfs
<Discordian[m]>
Hmm wondering, if I can make curl requests to my local IPFS daemon, and control it, what's stopping a rogue website from doing the same thing?
<Discordian[m]>
Alarmingly, seems like nothing 🤔
chachasmooth has quit [Ping timeout: 265 seconds]
chachasmooth has joined #ipfs
nicbk[m] is now known as nicbk
nicbk has quit [Quit: authenticating]
nicbk1 has joined #ipfs
Mike7 has left #ipfs [#ipfs]
nicbk1 is now known as nicbk_m
nicbk_m has quit [Client Quit]
nicbk_m has joined #ipfs
nicbk_m is now known as nicbk
<Discordian[m]>
Apparently there are CORS checks, hmm, might need to play with it more. But I think most websites appear to be localhost anyways when loaded over IPFS, which I figured would make CORS happy, hmm. Going to bed
opa7331 has joined #ipfs
opa has quit [Ping timeout: 276 seconds]
nicbk is now known as sdfjsdf
sdfjsdf is now known as nicbk
nicbk has quit [Quit: authenticating]
nicbk has joined #ipfs
drathir_tor has quit [Ping timeout: 240 seconds]
drathir_tor has joined #ipfs
arcatech has joined #ipfs
maxxxed69 has quit [Quit: Leaving]
arcatech has quit [Quit: Be back later.]
johnvl has joined #ipfs
johnvl has quit [Client Quit]
LiftLeft has quit [Ping timeout: 240 seconds]
arcatech has joined #ipfs
arcatech has quit [Client Quit]
LinusCDE has quit [Ping timeout: 250 seconds]
LinusCDE has joined #ipfs
Ringtailed_Fox has joined #ipfs
Ringtailed-Fox has quit [Ping timeout: 265 seconds]
obensource has quit [Ping timeout: 245 seconds]
clemo has joined #ipfs
clemo has quit [Ping timeout: 265 seconds]
kn0rki has joined #ipfs
Ringtailed-Fox has joined #ipfs
Ringtailed_Fox has quit [Ping timeout: 240 seconds]
Ringtailed_Fox has joined #ipfs
Ringtailed-Fox has quit [Ping timeout: 246 seconds]
<jlj_[m]>
<Piddle "> <@piddle:copperfox.xyz> you're"> All working! Running a bit warm (~60°C), but all within tolerances, as I understand it. :-)
baojg has quit [Remote host closed the connection]
baojg has joined #ipfs
baojg has quit [Remote host closed the connection]
baojg has joined #ipfs
baojg has quit [Remote host closed the connection]
baojg has joined #ipfs
baojg has quit [Remote host closed the connection]
baojg has joined #ipfs
baojg has quit [Remote host closed the connection]
baojg has joined #ipfs
baojg has quit [Remote host closed the connection]
baojg has joined #ipfs
baojg has quit [Remote host closed the connection]
baojg has joined #ipfs
baojg has quit [Remote host closed the connection]
baojg has joined #ipfs
baojg has quit [Remote host closed the connection]
baojg has joined #ipfs
baojg has quit [Remote host closed the connection]
baojg has joined #ipfs
baojg has quit [Remote host closed the connection]
baojg has joined #ipfs
baojg has quit [Remote host closed the connection]
tech_exorcist has joined #ipfs
luna_is_here has quit [Ping timeout: 258 seconds]
luna_is_here has joined #ipfs
luna_is_here has quit [Read error: Connection reset by peer]
luna_is_here has joined #ipfs
luna_is_here has quit [Ping timeout: 276 seconds]
luna_is_here has joined #ipfs
Nukien2 has joined #ipfs
p3rs0_ has quit [Ping timeout: 268 seconds]
Nukien2 has quit [Ping timeout: 246 seconds]
luna_is_here has quit [Ping timeout: 246 seconds]
luna_is_here has joined #ipfs
luna_is_here has quit [Ping timeout: 246 seconds]
luna_is_here has joined #ipfs
luna_is_here has quit [Ping timeout: 246 seconds]
luna_is_here has joined #ipfs
Nukien2 has joined #ipfs
Nukien2 has quit [Ping timeout: 246 seconds]
chronomark921 has quit [Quit: Ping timeout (120 seconds)]
chronomark921 has joined #ipfs
chronomark921 has quit [Remote host closed the connection]
chronomark921 has joined #ipfs
XORed has quit [Ping timeout: 240 seconds]
p3rs0 has joined #ipfs
XORed has joined #ipfs
edperry has quit [Quit: Connection closed]
luna_is_here has quit [Ping timeout: 265 seconds]
edperry has joined #ipfs
<AmariyahWhite[m]>
yo J here
zeden has joined #ipfs
p3rs0 has quit [Remote host closed the connection]
__jrjsmrtn__ has joined #ipfs
luna_is_here has joined #ipfs
raed has quit [Ping timeout: 250 seconds]
_jrjsmrtn has quit [Ping timeout: 240 seconds]
luna_is_here has quit [Read error: Connection reset by peer]
luna_is_here has joined #ipfs
edperry has quit [Quit: Connection closed]
edperry has joined #ipfs
<edperry>
Troubeshooting Question, I am setting up a private cluster and swarm based on the Docker-Compose from the Cluster IPFS. The swarm appears to be working fine, but my desktop on another PC never discovers all the peers unless I specify them as Bootstrap. Anything you can help me with focusing to focus my troubleshooting would be great
royal_screwup21 has joined #ipfs
bsm1175321 has joined #ipfs
royal_screwup21 has quit [Quit: Connection closed]
royal_screwup21 has joined #ipfs
arcatech has joined #ipfs
<eleitl[m]>
Question: the current network will not be able to automatically derive different hashes for the same content pinned, right?
luna_is_here has quit [Ping timeout: 246 seconds]
luna_is_here has joined #ipfs
royal_screwup21 has quit [Quit: Connection closed]
royal_screwup21 has joined #ipfs
Nukien2 has joined #ipfs
<eleitl[m]>
E.g. --hash=sha2-256 will not be found if we look for CID added with --hash=blake2b-256 unless it's been also been explicitly pinned so as well, right?
astronavt has quit [Remote host closed the connection]
astronavt has joined #ipfs
astronavt has quit [Remote host closed the connection]
luna_is_here has quit [Read error: Connection reset by peer]
astronavt has joined #ipfs
luna_is_here has joined #ipfs
astronavt has quit [Remote host closed the connection]
astronavt has joined #ipfs
edperry has quit [Quit: edperry]
edperry has joined #ipfs
LiftLeft has joined #ipfs
royal_screwup21 has quit [Quit: Connection closed]
royal_screwup21 has joined #ipfs
theseb has joined #ipfs
royal_screwup21 has quit [Ping timeout: 246 seconds]
chachasmooth_ has joined #ipfs
chachasmooth has quit [Ping timeout: 245 seconds]
koo5 has joined #ipfs
LevelUp[m] has joined #ipfs
jcea has joined #ipfs
arcatech has quit [Quit: Be back later.]
matt-h has quit [Ping timeout: 245 seconds]
<DokterBob>
Hi everybody, some weekends ago I wrote a simple util to download Youtube videos straight to IPFS, for archival purposes: https://pypi.org/project/youtube2ipfs/
<DokterBob>
Let me know what you think of it :)
ipfs-stackbot has quit [Remote host closed the connection]
<DokterBob>
<eleitl[m] "Question: the current network wi"> That is correct. Deriving any information on one hash from another should be practically impossible for any cryptographic hash. ;)
ipfs-stackbot has joined #ipfs
matt-h has joined #ipfs
<eleitl[m]>
One could use a lingua franca hash internally, but that would add extra overhead -- and obviously counteract to the intention of picking a particular hash.
koo5 has quit [Ping timeout: 240 seconds]
dpl_ has quit [Ping timeout: 268 seconds]
arcatech has joined #ipfs
arcatech has quit [Client Quit]
arcatech has joined #ipfs
Nact has quit [Quit: Konversation terminated!]
tech_exorcist has quit [Remote host closed the connection]
tech_exorcist has joined #ipfs
supercoven has joined #ipfs
supercoven has quit [Read error: Connection reset by peer]
MrSparkle has quit [Read error: Connection reset by peer]
Rokas[m] has quit [Ping timeout: 245 seconds]
icaruszDiscord[m has quit [Ping timeout: 245 seconds]
adinbDiscord[m] has quit [Ping timeout: 245 seconds]
MrSparkle has joined #ipfs
Rokas[m] has joined #ipfs
icaruszDiscord[m has joined #ipfs
adinbDiscord[m] has joined #ipfs
royal_screwup21 has joined #ipfs
justanotherdude has quit [Quit: RAGEQUIT]
royal_screwup21 has quit [Ping timeout: 246 seconds]
ylp has quit [Quit: Leaving.]
D__ is now known as D_
neo_trix[m] has quit [Quit: Idle for 30+ days]
bmurray[m] has quit [Quit: Idle for 30+ days]
questaetang[m] has quit [Quit: Idle for 30+ days]
mburns[m] has quit [Quit: Idle for 30+ days]
pixelgeek[m] has quit [Quit: Idle for 30+ days]
wickhere[m] has quit [Quit: Idle for 30+ days]
chuggiesmuggies[ has quit [Quit: Idle for 30+ days]
fm[m] has quit [Quit: Idle for 30+ days]
ErichSpaker[m] has quit [Quit: Idle for 30+ days]
dospitercios[m] has quit [Quit: Idle for 30+ days]
yafai[m] has quit [Quit: Idle for 30+ days]
dopeman[m] has quit [Quit: Idle for 30+ days]
patte8[m] has quit [Quit: Idle for 30+ days]
zant[m] has quit [Quit: Idle for 30+ days]
drpytho[m] has quit [Quit: Idle for 30+ days]
tema3210[m] has quit [Quit: Idle for 30+ days]
AeioMuch[m] has quit [Quit: Idle for 30+ days]
mattelekharris[m has quit [Quit: Idle for 30+ days]
gasha[m] has quit [Quit: Idle for 30+ days]
Abdullah[m] has quit [Quit: Idle for 30+ days]
stevefloyd[m] has quit [Quit: Idle for 30+ days]
chaoticcapybara[ has quit [Quit: Idle for 30+ days]
claircognizance[ has quit [Quit: Idle for 30+ days]
kamil445[m] has quit [Quit: Idle for 30+ days]
M0x1d[m] has quit [Quit: Idle for 30+ days]
lecro[m] has quit [Quit: Idle for 30+ days]
MrSparkle is now known as impossiball
arcatech has quit [Quit: Be back later.]
royal_screwup21 has joined #ipfs
<RomeSilvanus[m]>
Nice
arcatech has joined #ipfs
<RomeSilvanus[m]>
How does it handle hashes it already made, or if I switch between both modi?
<Discordian[m]>
It'll replace them like before
<Discordian[m]>
And re-add them all to IPFS
<Discordian[m]>
However now that initial hashing stage shouldn't take 1 month before it begins adding to IPFS now
<RomeSilvanus[m]>
Hmm... why not keep have both in the db and just use the one set by the config?
<RomeSilvanus[m]>
I had this idea that you could do hashes at the initial run and then switch to modtime. And if motime changes it still does the has part. Basically it always does hashes at any change but when you restart it it just does modtime to get upto-date faster.
<RomeSilvanus[m]>
This is very confusing and makes no sense probably.
<Discordian[m]>
Well that kinda makes sense, do a sorta double-check, but then the initial hashing is still required
jesse22 has joined #ipfs
arcatech_ has joined #ipfs
<Discordian[m]>
The whole goal is to avoid the initial hashing, as you said it'd take about a month to xfer all the data into memory
<RomeSilvanus[m]>
I just though it adds some kind of security into it because the has is precise. But modtime is fast. Would be only really useful on archives that don't chnage much.
arcatech has quit [Ping timeout: 252 seconds]
<RomeSilvanus[m]>
My gripe was actually more with the case that if you restart ir for any reason it will take a month again. One time initial hashing is not so critical since it's only one time.
<RomeSilvanus[m]>
* My gripe was actually more with the case that if you restart it for any reason it will take a month again. One time initial hashing is not so critical since it's only one time.
<RomeSilvanus[m]>
Not demanding you put that in. Just something I had time to think about last night.
jesse22_ has joined #ipfs
<Discordian[m]>
A dual mode could be an option for added stability, but like we talk about before, it's probably very unlikely a file will change, be the same size, and have no modtime change
<Discordian[m]>
* A dual mode could be an option for added stability, but like we talked about before, it's probably very unlikely a file will change, be the same size, and have no modtime change
<Discordian[m]>
And dw, I never take you as demanding, just valuable feedback 🙂
<RomeSilvanus[m]>
🐴
jesse22 has quit [Ping timeout: 268 seconds]
<RomeSilvanus[m]>
How much faster is it actually now? Did you do any benchmarksß
mowcat has joined #ipfs
<Discordian[m]>
I only did a little test using like 0.5GB data, took about 8s to process before, took under 1s after.
<RomeSilvanus[m]>
Nice
<RomeSilvanus[m]>
Eh, I'll try
<RomeSilvanus[m]>
If it's slow I will cry
<Discordian[m]>
It literally takes the filesize in int64, and modsize in int64, then just concatenates those 8 bytes, and stores them in LevelDB
<Discordian[m]>
Uses bitmasking and bitshifting to do it, so it should be extremely fast.
<RomeSilvanus[m]>
He says, while using secret alien government tech SSDs with petabyte/second write speeds
<Discordian[m]>
Haha just an NVME, I happen to be lucky and had a mobo that has the slot haha
<Discordian[m]>
And it's your read speeds that are killing me, probably the seek times specifically
<Discordian[m]>
No more month long start-up times, even my fiance feels bad you have to wait a month for anything, and she doesn't even know you.
<RomeSilvanus[m]>
Well, the server has a lot of r/w from other applications too. It's at least a 4 HHD raid so it's at least faster than a single HDD.
<eleitl[m]>
I see 7+ GB/s reads on my 1 TB NVMe. At least in some aspects we're living in the future.
<Discordian[m]>
Right? I can't go back from SSD. No more stupid HDD failures to play with, no more seek times destroying my speeds.
<Discordian[m]>
I only use HDDs to store some large data that rarely changes, and a couple massive drives holding my 4K HDR movies.
<eleitl[m]>
I don't want to get back to HDDs either. But my wallet seems to disagree, when looking at two-digit TB values.
royal_screwup21 has quit [Quit: Connection closed]
royal_screwup21 has joined #ipfs
<Discordian[m]>
Just think about this whenever you're considering using an HDD as a primary: How often does SMART really warn you before a failure?
royal_screwup21 has quit [Ping timeout: 260 seconds]
<eleitl[m]>
All my HDDs but scratch disks are in zfs arrays. These can be surprisingly resilient.
echoekhi[m] has joined #ipfs
<Discordian[m]>
I had another traumatic failure recently, so I'm still jaded from that
<Discordian[m]>
I thought it was about to fail, but SMART told me it's okay, I decided to backup the next day ... never spun up again.
<Discordian[m]>
* I thought it was about to fail, but SMART told me it's okay, I decided to backup the next day ... never spun up again. (well, it spin loops)
<RomeSilvanus[m]>
2021/03/29 18:55:15 Removing reference from filestore:
<RomeSilvanus[m]>
Okay now it is gonna remove a few million files !
arcatech has quit [Ping timeout: 245 seconds]
<Discordian[m]>
Wait if they're in filestore, then it must have gotten past the hashing step! Unless you mean a directory that completed, in which case, sorry about that heh.
<Discordian[m]>
Or if it's doing cleanup, tbh, you'll want that
<Discordian[m]>
Performance will skyrocket, and random bugs will vanish
<RomeSilvanus[m]>
No it just started doing that right after I ran it
<Discordian[m]>
Oh yeah that's filestore cleanup, if you see removals, that's very good
<Discordian[m]>
It does a clean on boot now, to ensure your filestore won't be totally broken when it tries to pin/add to MFS
<eleitl[m]>
I'm doing a 1 million file 6 TB filestore pin on low-end hardware right now. Seems to take about 3 days.
<Discordian[m]>
He's using `ipfs-sync`, which does a couple extra steps before the add, and the add might not be as efficient as it could be rn.
<RomeSilvanus[m]>
Nice
<eleitl[m]>
The filestore with ~/.ipfs is on a 128 GB SSD, the data is on a 8 TB WD shuck. The spindle speed determines the pin speed. The CPU is not quite maxed out, but getting closer than with the 2 TB test pin.
<Discordian[m]>
Oh nice, sounds like that'd be fast
<RomeSilvanus[m]>
Shame they stopped makign these 15000rpm HDDs. Shoved in your face in every gameing ad in the mid 2000s.
<eleitl[m]>
The 8 TB starts with some 200+ MB/s, but then falls down to about 150 MB/s towards the end.
<Discordian[m]>
I've only tested with ~4GB of 18k files, and it takes under 1min to hash it all.
<eleitl[m]>
I still have a large raidz with 10k WD Raptors 300 GB.
<Discordian[m]>
<eleitl[m] "The 8 TB starts with some 200+ M"> Dayum that's nice, I figured seek would have slowed that down, clearly I use shit drives.
<eleitl[m]>
The 200 MB/s is raw device benchmark. I don't know how fast the ipfs add works. I'm running a time and will compute the result afterwards.
<Discordian[m]>
Ahh
<Discordian[m]>
If you ever wanted to try `ipfs-sync` out, we could use someone else who really pushes the extremes lol
bengates_ has quit [Remote host closed the connection]
royal_screwup21 has joined #ipfs
<Discordian[m]>
Also FWIW, I believe the IPFS daemon when you do an add on a directory, will add that whole directory, and all it's structures all at once, then you can pin the returned CID, and add it to MFS. What ipfs-sync does is instead 1 add per file, and adds each file to MFS, with the same structure. It's probably a bit of a slowdown for massive data sets, because it does more HTTP connection opening/closing.
bengates has joined #ipfs
theseb has left #ipfs ["Leaving"]
bengates has quit [Ping timeout: 246 seconds]
lawid has quit [Quit: lawid]
astronavt has quit [Remote host closed the connection]
astronavt has joined #ipfs
Jesin has quit [Quit: Leaving]
<codepy69>
Hello
royal_screwup21 has quit [Quit: Connection closed]
royal_screwup21 has joined #ipfs
royal_screwup21 has quit [Ping timeout: 240 seconds]
zeden has quit [Quit: WeeChat 3.0.1]
koo5 has joined #ipfs
reit has joined #ipfs
<VoidSoul[m]>
Nube question here.
<VoidSoul[m]>
Does ipfs sites works similar at https. If one created website to store stuff in it will those stuff would be hosted on ipfs protocol too and become immortal?
<VoidSoul[m]>
Or there other strategy for it.
arcatech has joined #ipfs
royal_screwup21 has joined #ipfs
arcatech has quit [Ping timeout: 276 seconds]
<Discordian[m]>
Maybe a couple misconceptions here:
<Discordian[m]>
- Data must be added to a writable node you have access to, however you plan to accomplish that is up to you
<Discordian[m]>
- Data on IPFS is not immortal
dpl_ has joined #ipfs
arcatech has joined #ipfs
arcatech has quit [Client Quit]
arcatech has joined #ipfs
royal_screwup21 has quit [Quit: Connection closed]
royal_screwup21 has quit [Ping timeout: 246 seconds]
aLeSD_ has joined #ipfs
aLeSD has quit [Ping timeout: 260 seconds]
Ringtailed-Fox has joined #ipfs
Ringtailed-Fox has quit [Read error: Connection reset by peer]
Ringtailed-Fox has joined #ipfs
Ringtailed_Fox has quit [Ping timeout: 246 seconds]
MDude has quit [Quit: Going offline, see ya! (www.adiirc.com)]
Newami has joined #ipfs
andrew2 has joined #ipfs
lawid has joined #ipfs
Newami has quit [Quit: Leaving]
andrew2 has quit [Quit: Connection closed]
mowcat has quit [Remote host closed the connection]
mowcat has joined #ipfs
icebear2 has joined #ipfs
MDude has joined #ipfs
mowcat has quit [Ping timeout: 240 seconds]
mindCrime_ has quit [Ping timeout: 240 seconds]
arcatech has quit [Quit: Be back later.]
<RomeSilvanus[m]>
I can see it right now, removing all these will take as long as adding them before took D:
<Discordian[m]>
If it was at the initial hashing step, it wasn't adding to IPFS though 🤔, now I'm wondering exactly how far your daemon has been getting
<RomeSilvanus[m]>
It's doing about 1-2 files per second
<Discordian[m]>
Also it's only removing blocks that have filestore references which are genuinely bad
<RomeSilvanus[m]>
There's a ot of garbage in there from before I used ipfs-sync
<RomeSilvanus[m]>
* There's a lot of garbage in there from before I used ipfs-sync
<Discordian[m]>
Ah, honestly I just totally cleaned up my daemon, and performance is waayyy better
<Discordian[m]>
I also removed every extra pin, and did a GC
<Discordian[m]>
Also the reason it takes so long on the removals, is because it removes as it goes through the list, so some of the waiting might be just reading through blocks
<RomeSilvanus[m]>
I did remove all the pins. But as I said before, at one point ipfs add was almost finished before it died because these people don't know that you don't make a program exit just because an error occurs. So all these tens of millions of files are stil in there somewhere.
<RomeSilvanus[m]>
* I did remove all the pins. But as I said before, at one point ipfs add was almost finished before it died because these people don't know that you don't make a program exit just because an error occurs. So all these tens of millions of files are still in there somewhere.
<Discordian[m]>
Ah, so any of those, it'll only remove the ones that broke in filestore
<Discordian[m]>
It'd have to be done at some point anyways, or the final result will be all sorts of broken : /
<Discordian[m]>
Perhaps there should be an index on filestore status in the IPFS daemon, so I can quickly just grab that list.
<Discordian[m]>
Specifically a sorted list, only returning bad blocks
<RomeSilvanus[m]>
Or ipfs should just nuke it when you remove the pin. Not at some arbitrary point in time.
<Discordian[m]>
Like, it already tracks the filestore status, so it could remove the blocks automatically. Not sure why it doesn't
<Discordian[m]>
But that's why ipfs-sync does it for you, because it needs to happen anyways
<Discordian[m]>
Also I tried to just fix blocks referenced in error messages, but unfortunately the error checking doesn't always get the errors on add, files/cp, pin/add, or pin/update, so it wasn't reliable at all.
<Discordian[m]>
Because it just wouldn't dump an error at all, however when you tried to access the files over IPFS, they wouldn't function at all.