stebalien changed the topic of #ipfs to: Heads Up: To talk, you need to register your nick! Announcements: go-ipfs 0.4.22 and js-ipfs 0.35 are out! Get them from dist.ipfs.io and npm respectively! | Also: #libp2p #ipfs-cluster #filecoin #ipfs-dev | IPFS, the InterPlanetary FileSystem: https://github.com/ipfs/ipfs | Logs: https://view.matrix.org/room/!yhqiEdqNjyPbxtUjzm:matrix.org/ | Forums: https://discuss.ipfs.io | Code of
uniquerockrz has joined #ipfs
mowcat has joined #ipfs
uniquerockrz has quit [Read error: Connection reset by peer]
awk1314 has joined #ipfs
uniquerockrz has joined #ipfs
awk1314 has quit [Quit: Leaving]
uniquerockrz has quit [Read error: Connection reset by peer]
hqdruxn08_ has joined #ipfs
hqdruxn08 has quit [Ping timeout: 246 seconds]
woss_io has quit [Read error: Connection reset by peer]
uniquerockrz has joined #ipfs
thomasanderson has joined #ipfs
thomasanderson has quit [Ping timeout: 240 seconds]
stavros has joined #ipfs
<stavros>
Hello
<stavros>
I have an IPFS node in a cluster, I added a second node to the cluster, waited for the pinset to be transferred and then took the original node out. My purpose is to transfer all pinned objects from one node to the other, but it's taking days. Is there a way I can do this more quickly?
<jehaverlack[m]>
Yo
<stavros>
Basically I need to take all pinned objects from one node to the other, and to make ipfs-cluster aware that things are pinned
<jehaverlack[m]>
I've not had good luck with this in IPFS yet.
<stavros>
Hm right :/
<jehaverlack[m]>
In theory that's what pinning does.
<stavros>
Which part exactly?
<stavros>
In theory, yes, but it's been a week and it still hasn't managed to transfer 30 GB
thomasanderson has quit [Ping timeout: 265 seconds]
<stavros>
I can't see that message, it's too long for the relay
<jehaverlack[m]>
Also, I would think in theory that if you know the list of URI's that are supposed to be pinned, you could stop ipfs daemon on each host and try to ipfs cat each file.
<jehaverlack[m]>
So you have many many files you want pinned.
<stavros>
I have many many files I *have* pinned
<stavros>
I want to transfer the pinned objects to a different IPFS node
<jehaverlack[m]>
Disclosure: I'm still green at IPFS, not an expert by any means.
<stavros>
And make ipfs-cluster aware of the status on the new node
<stavros>
Ah, okay
<jehaverlack[m]>
Well, what I understand that you want to do is certainly something that IPFS is supposed to excel at from what I've read.
<jehaverlack[m]>
My attempts to proof of concept that have not been super successfull.
<stavros>
It's supposed to, but it doesn't look like it very much wants to
<jehaverlack[m]>
On the odd chance you are just looking for a way to replicate data between hosts you might checkout https://syncthing.net/
<jehaverlack[m]>
But if you question is specific to IPFS you are on the right track. I just have not been able to get IPFS to replicated accross nodes as I understand it should.
<stavros>
Syncthing is great, but this is IPFS-specific, yeah
<jehaverlack[m]>
I've seen huge latency with IPNS also.
<jehaverlack[m]>
Keeping in mind that IPFS is not really ready for production yet.
<CopenBra[m]>
Hello
<jehaverlack[m]>
Sorry I don't have a straight answer for you.
<CopenBra[m]>
Those IPFS peer addresses I've seen connected to me, could some of those have their own files if I just opened one if them randomly?
<jehaverlack[m]>
That is my understanding.
<jehaverlack[m]>
But I don't think there is a way for you to know what files are on what other nodes.
<jehaverlack[m]>
You can on access content by the hash for a given file.
xcm has quit [Remote host closed the connection]
uniquerockrz has quit [Read error: Connection reset by peer]
xcm has joined #ipfs
Encrypt has quit [Ping timeout: 276 seconds]
uniquerockrz has joined #ipfs
carldd has joined #ipfs
}ls{ has quit [Ping timeout: 240 seconds]
}ls{ has joined #ipfs
uniquerockrz has quit [Read error: Connection reset by peer]
uniquerockrz has joined #ipfs
zettelding has joined #ipfs
uniquerockrz has quit [Read error: Connection reset by peer]
ygrek__ has joined #ipfs
zettelding has left #ipfs [#ipfs]
uniquerockrz has joined #ipfs
RamRanRa has quit [Read error: Connection reset by peer]
lmariscal has quit [Quit: I'm Out!]
uniquerockrz has quit [Read error: Connection reset by peer]
lmariscal has joined #ipfs
uniquerockrz has joined #ipfs
uniquerockrz has quit [Read error: Connection reset by peer]
uniquerockrz has joined #ipfs
thomasanderson has joined #ipfs
DavidPH has joined #ipfs
thomasanderson has quit [Ping timeout: 265 seconds]
CCR5-D32 has joined #ipfs
stoopkid has quit [Quit: Connection closed for inactivity]
uniquerockrz has quit [Read error: Connection reset by peer]
verin0x4 has joined #ipfs
verin0x has quit [Ping timeout: 240 seconds]
uniquerockrz has joined #ipfs
Soo_Slow has quit [Quit: Soo_Slow]
mowcat has quit [Remote host closed the connection]
uniquerockrz has quit [Read error: Connection reset by peer]
hqdruxn08_ has quit [Ping timeout: 246 seconds]
hqdruxn08 has joined #ipfs
user_51 has joined #ipfs
jcea has quit [Remote host closed the connection]
jcea has joined #ipfs
stavros has quit [Quit: Connection closed for inactivity]
user_51_ has quit [Ping timeout: 252 seconds]
thomasanderson has joined #ipfs
KempfCreative has joined #ipfs
thomasanderson has quit [Ping timeout: 240 seconds]
uniquerockrz has joined #ipfs
kakra has joined #ipfs
uniquerockrz has quit [Read error: Connection reset by peer]
hurikhan77 has quit [Ping timeout: 240 seconds]
jcea has quit [Quit: jcea]
uniquerockrz has joined #ipfs
uniquerockrz has quit [Read error: Connection reset by peer]
Belkaar_ has quit [Ping timeout: 250 seconds]
Belkaar has joined #ipfs
Belkaar has joined #ipfs
uniquerockrz has joined #ipfs
_whitelogger has joined #ipfs
uniquerockrz has quit [Read error: Connection reset by peer]
uniquerockrz has joined #ipfs
Adbray has quit [Remote host closed the connection]
MrAureliusR has joined #ipfs
Adbray has joined #ipfs
<MrAureliusR>
is it just me or does ipfs seem to use a lot of RAM when adding large files? I tried to add a 2.4gb file and almost crashed my system, was swapping like crazy
<MrAureliusR>
also, I deleted that file after adding it, but the repo size is still 2.4gb. is it possible to clear that?
Arwalk has quit [Read error: Connection reset by peer]
uniquerockrz has quit [Read error: Connection reset by peer]
uniquerockrz has joined #ipfs
thomasanderson has joined #ipfs
arky has joined #ipfs
uniquerockrz has quit [Read error: Connection reset by peer]
thomasanderson has quit [Ping timeout: 240 seconds]
uniquerockrz has joined #ipfs
uniquerockrz has quit [Read error: Connection reset by peer]
uniquerockrz has joined #ipfs
M981031046badd[m has quit [Quit: User has been idle for 30+ days.]
LaPingvino has left #ipfs ["Kicked by @appservice-irc:matrix.org : User has been idle for 30+ days."]
Methos[m] has left #ipfs ["Kicked by @appservice-irc:matrix.org : User has been idle for 30+ days."]
ictius[m] has quit [Quit: User has been idle for 30+ days.]
}ls{ has quit [Ping timeout: 276 seconds]
Phenomite[m] has quit [Quit: User has been idle for 30+ days.]
}ls{ has joined #ipfs
scio has left #ipfs ["Kicked by @appservice-irc:matrix.org : User has been idle for 30+ days."]
terichadbourne[m has quit [Quit: User has been idle for 30+ days.]
vpei[m] has left #ipfs ["Kicked by @appservice-irc:matrix.org : User has been idle for 30+ days."]
alyssa0 is now known as Zalyssa
uniquerockrz has quit [Read error: Connection reset by peer]
uniquerockrz has joined #ipfs
stavrosk[m] has joined #ipfs
<stavrosk[m]>
Test
<stavrosk[m]>
The bot keeps kicking me
ffl^ has joined #ipfs
<stavrosk[m]>
<MrAureliusR "is it just me or does ipfs seem "> @mraurMrAureliusR yes it does consume a lot of ram, you can clear files with `ipfs repo gc` IIRC
RamRanRa has joined #ipfs
arky has quit [Ping timeout: 240 seconds]
Encrypt has joined #ipfs
uniquerockrz has quit [Read error: Connection reset by peer]
arky has joined #ipfs
woss_io has joined #ipfs
uniquerockrz has joined #ipfs
arky has quit [Ping timeout: 240 seconds]
uniquerockrz has quit [Read error: Connection reset by peer]
uniquerockrz has quit [Read error: Connection reset by peer]
thomasanderson has joined #ipfs
thomasanderson has quit [Ping timeout: 240 seconds]
gmoro has joined #ipfs
uniquerockrz has joined #ipfs
dolphy has joined #ipfs
uniquerockrz has quit [Read error: Connection reset by peer]
reit has quit [Ping timeout: 268 seconds]
uniquerockrz has joined #ipfs
uniquerockrz has quit [Read error: Connection reset by peer]
uniquerockrz has joined #ipfs
uniquerockrz has quit [Read error: Connection reset by peer]
Clarth has quit [Ping timeout: 276 seconds]
JorropoDiscord[m is now known as Jorropo4529[m]
<Jorropo4529[m]>
Yes, there is an known leak of memory, its thé peerstore, libp2p store each known node in ram, that a lot of informations for a lot of nodes
<Jorropo4529[m]>
In the future less information will je stored and memory will just be a cache for the disk.
uniquerockrz has joined #ipfs
<Jorropo4529[m]>
If tour nodes use 2gb of ram (while doing nothing, no pin, no get, ...) Its likly 1gb to 512mb is the peerstore
crest has quit [Read error: Connection reset by peer]
crest has joined #ipfs
stoopkid has joined #ipfs
gmoro has quit [Ping timeout: 240 seconds]
CCR5-D32 has joined #ipfs
uniquerockrz has quit [Read error: Connection reset by peer]
uniquerockrz has joined #ipfs
ffl^ has quit []
tsrt^ has joined #ipfs
uniquerockrz has quit [Read error: Connection reset by peer]
uniquerockrz has joined #ipfs
woss_io has quit [Read error: Connection reset by peer]
}ls{ has quit [Quit: real life interrupt]
woss_io has joined #ipfs
harai has joined #ipfs
fifeefo has joined #ipfs
uniquerockrz has quit [Read error: Connection reset by peer]
arky has quit [Quit: Ex-Chat]
thomasanderson has joined #ipfs
gmoro has joined #ipfs
thomasanderson has quit [Ping timeout: 246 seconds]
uniquerockrz has joined #ipfs
<pusherDiscord[m]>
hey all. Im aware --progress shows the number of nodes found/queried when running an ipfs pin, wondering here whether there is a command to retrieve all of the ipv4 or ipv6 hosts that may host a file on the DHT. So TLDR is there a simple way to return the number of seeds for a given ipfs_hash?
<pusherDiscord[m]>
also Jorropo i think that memory leak in later versions is not as bad, or has been patched. I had some real problems in previous versions of our pin management software
<pusherDiscord[m]>
performance seems a bit better now, and i dont have to be so harsh with systemd and security limits et al. Ultimately if you have created your own systemD and have set a sensible AS, the worst that will happen is it will restart. My solution, anyway.
<ZipperSKDiscord[>
Hi does someone try to make some kind of b-tree in IPFS to index data by some property?
plimeriz has joined #ipfs
uniquerockrz has quit [Read error: Connection reset by peer]
<Jorropo4529[m]>
pusher with `ipfs dht findprovs <hash>` you can find peers hosting a file and with `ipfs dht findpeer <peer id>` you can find addrs for each peers (and bash for loop will automate that)
<Jorropo4529[m]>
pusher also, use ipfs cluster for pin management that just way better (it allow async pinning easly (better than `nohup ipfs pin <ipfs hash> &`))
<Jorropo4529[m]>
it also have name and metadata
uniquerockrz has joined #ipfs
<pusherDiscord[m]>
hey man, thanks a lot for responding. I had forgotten about findprovs.
<pusherDiscord[m]>
I was actually having issues with pin management previously and felt that ipfs-cluster might not be what I wanted. For us its for automation, and ipfs cluster requires a lot of configuration that seems complex at runtime, with control plane knowing about other clustered nodes. We actually rolled our own operating system and image to deal with this management... but I have been looking at the native IPFS api, and
<pusherDiscord[m]>
wondering if ipfs cluster might be better.
<pusherDiscord[m]>
I told my co-developer not to try and roll our own API on our control plane, for this, just because I am still not yet sure whether to go with ipfs-cluster or continue to use our own developed toolset. Naturally my tool suite can control variable filesize conditions, large file threshold, large hold timeout, for the pin lists we get from other blockchains.
<pusherDiscord[m]>
Naturally, i didnt like my solution since it meant our automation relied on a service we provide of hash lists, and so on. So I've rewritten it all to support all native execution. both ipfs chain and others. I do expect we will be taking a look at ipfs-cluster properly soon, before we write any more of our control plane. I don't want to redesign the wheel when it has already been done, but yeah i did have some fury
<pusherDiscord[m]>
when dealing with pin management. I guess I have mixed feelings both for and against native pin management or not. Presumably though ipfs-cluster is a tool set that forms around ipfs, rather than a completely different dist of the binary
gmoro has quit [Ping timeout: 240 seconds]
uniquerockrz has quit [Read error: Connection reset by peer]
<ZipperSKDiscord[>
How does the objects count (items in DSHT) affect performance of IPFS? I know that there is some log compexity, but does every node have to download the whole hashtable?
<Jorropo4529[m]>
no, its kadamelia, and kadamelia have a way to order nodes so you only store a small part (and you are capable to find wich node store what kind of data (for the dht only, because dht store pointer to nodes storing files))
<ZipperSKDiscord[>
thak you, I will study kadamelia than first, and find out how it works
<Jorropo4529[m]>
the french article on wikipedia about dht explain it well (even if that a bit simpler than how it works in reality)
<Jorropo4529[m]>
* the french article on wikipedia about dht explain it well (even if that a simpler than how it works in reality)
<ZipperSKDiscord[>
hmm, I dont have time to study french but thank you 😄 I find this article on wikipedia in english https://en.wikipedia.org/wiki/Kademlia
uniquerockrz has quit [Read error: Connection reset by peer]
<stavrosk[m]>
Ah yes, that's an old issue
uniquerockrz has joined #ipfs
valka[m] has joined #ipfs
shizy has joined #ipfs
Newami has joined #ipfs
Newami has quit [Remote host closed the connection]
CCR5-D32 has quit [Quit: ZZZzzz…]
kakra has joined #ipfs
uniquerockrz has quit [Read error: Connection reset by peer]
hurikhan77 has quit [Ping timeout: 265 seconds]
uniquerockrz has joined #ipfs
thomasanderson has joined #ipfs
thomasanderson has quit [Ping timeout: 265 seconds]
The_8472 has quit [Ping timeout: 248 seconds]
The_8472 has joined #ipfs
uniquerockrz has quit [Read error: Connection reset by peer]
uniquerockrz has joined #ipfs
harai has quit [Ping timeout: 260 seconds]
Frog_ has joined #ipfs
Frog_ is now known as Guest80335
uniquerockrz has quit [Read error: Connection reset by peer]
uniquerockrz has joined #ipfs
mowcat has quit [Remote host closed the connection]
Ard1t has joined #ipfs
uniquerockrz has quit [Read error: Connection reset by peer]
uniquerockrz has joined #ipfs
uniquerockrz has quit [Read error: Connection reset by peer]
uniquerockrz has joined #ipfs
charley has joined #ipfs
uniquerockrz has quit [Read error: Connection reset by peer]
uniquerockrz has joined #ipfs
thomasanderson has joined #ipfs
luizirber has quit [Ping timeout: 245 seconds]
silotis_ has joined #ipfs
thomasanderson has quit [Ping timeout: 240 seconds]
silotis has quit [Ping timeout: 245 seconds]
aug has joined #ipfs
dolphy has quit [Ping timeout: 265 seconds]
lidel has quit [Remote host closed the connection]
aug has quit [Remote host closed the connection]
luizirber has joined #ipfs
uniquerockrz has quit [Read error: Connection reset by peer]
uniquerockrz has joined #ipfs
CCR5-D32 has joined #ipfs
uniquerockrz has quit [Read error: Connection reset by peer]
reit has joined #ipfs
seba- has joined #ipfs
seba-- has quit [Ping timeout: 240 seconds]
tsrt^ has quit []
uniquerockrz has joined #ipfs
DavidPH has quit [Quit: Leaving]
bengates has joined #ipfs
bikes has joined #ipfs
uniquerockrz has quit [Read error: Connection reset by peer]
bikes has quit [Remote host closed the connection]
uniquerockrz has joined #ipfs
charley has quit [Remote host closed the connection]
charley has joined #ipfs
charley has quit [Remote host closed the connection]
charley has joined #ipfs
The_8472 has quit [Ping timeout: 252 seconds]
mowcat has joined #ipfs
The_8472 has joined #ipfs
uniquerockrz has quit [Read error: Connection reset by peer]
grumble has quit [Read error: Connection reset by peer]
grumble has joined #ipfs
}ls{ has joined #ipfs
uniquerockrz has joined #ipfs
lidel has joined #ipfs
bengates has quit [Ping timeout: 276 seconds]
Clarth has joined #ipfs
tsrt^ has joined #ipfs
charley has quit [Remote host closed the connection]
charley has joined #ipfs
Soo_Slow has joined #ipfs
zazikiDiscord[m] has left #ipfs ["User left"]
charley has quit [Ping timeout: 250 seconds]
uniquerockrz has quit [Read error: Connection reset by peer]
jcea has joined #ipfs
uniquerockrz has joined #ipfs
rendar has quit []
woss_io has quit [Read error: Connection reset by peer]
uniquerockrz has quit [Read error: Connection reset by peer]
uniquerockrz has joined #ipfs
mowcat has quit [Remote host closed the connection]
PyHedgehog has quit [Quit: Connection closed for inactivity]
mowcat has joined #ipfs
uniquerockrz has quit [Read error: Connection reset by peer]
mowcat has quit [Remote host closed the connection]
uniquerockrz has joined #ipfs
thomasanderson has joined #ipfs
cheet has quit [Quit: ZNC 1.8.x-nightly-20190530-13450c7f - https://znc.in]
cheet has joined #ipfs
charley has joined #ipfs
thomasanderson has quit [Ping timeout: 246 seconds]
charley has quit [Ping timeout: 245 seconds]
uniquerockrz has quit [Read error: Connection reset by peer]
uniquerockrz has joined #ipfs
Guest80335 has quit [Read error: Connection reset by peer]
uniquerockrz has quit [Read error: Connection reset by peer]
Ecran has joined #ipfs
Ecran has quit [Client Quit]
uniquerockrz has joined #ipfs
dolphy has joined #ipfs
uniquerockrz has quit [Read error: Connection reset by peer]
charley has joined #ipfs
kappadeus[m] has joined #ipfs
CCR5-D32 has quit [Quit: ZZZzzz…]
Jybz has quit [Quit: Konversation terminated!]
charley has quit [Remote host closed the connection]
charley has joined #ipfs
dolphy has quit [Remote host closed the connection]
uniquerockrz has joined #ipfs
Jesin has quit [Quit: Leaving]
uniquerockrz has quit [Read error: Connection reset by peer]
Jesin has joined #ipfs
uniquerockrz has joined #ipfs
rardiol has quit [Read error: Connection reset by peer]
MrAureliusR has left #ipfs ["linker error: AirCombat.so.1 not found"]
thomasanderson has joined #ipfs
thomasanderson has quit [Ping timeout: 240 seconds]
Clarth has quit [Remote host closed the connection]
uniquerockrz has quit [Read error: Connection reset by peer]
plexigras has joined #ipfs
drocta has joined #ipfs
uniquerockrz has joined #ipfs
drocta has quit [Remote host closed the connection]
plexigras has quit [Quit: WeeChat 2.4]
uniquerockrz has quit [Read error: Connection reset by peer]
thomasanderson has joined #ipfs
thomasanderson has quit [Ping timeout: 240 seconds]
ygrek__ has joined #ipfs
uniquerockrz has joined #ipfs
KempfCreative has joined #ipfs
uniquerockrz has quit [Read error: Connection reset by peer]
uniquerockrz has joined #ipfs
humbleElitist_Di has joined #ipfs
<humbleElitist_Di>
noob question : If I'm not going to be using this frequently yet, are there settings (or maybe just updating to latest version?) for the go-ipfs where I can just set the daemon to run at startup, and not worry about wanting to close it despite not actively using it yet?