stebalien changed the topic of #ipfs to: Heads Up: To talk, you need to register your nick! Announcements: go-ipfs 0.4.22 and js-ipfs 0.35 are out! Get them from dist.ipfs.io and npm respectively! | Also: #libp2p #ipfs-cluster #filecoin #ipfs-dev | IPFS, the InterPlanetary FileSystem: https://github.com/ipfs/ipfs | Logs: https://view.matrix.org/room/!yhqiEdqNjyPbxtUjzm:matrix.org/ | Forums: https://discuss.ipfs.io | Code of
zopsi has quit [Quit: Oops]
zopsi has joined #ipfs
soju has quit [Remote host closed the connection]
pecastro has quit [Ping timeout: 276 seconds]
soju has joined #ipfs
redfish has quit [Ping timeout: 252 seconds]
soju has quit [Ping timeout: 276 seconds]
}ls{ has quit [Quit: real life interrupt]
AbsolutelyJ has quit [Ping timeout: 265 seconds]
daMaestro has joined #ipfs
j197788 has quit [Read error: Connection reset by peer]
j197789 has joined #ipfs
j197789 is now known as j197788
bsm1175321 has quit [Ping timeout: 245 seconds]
niyawe has quit [Ping timeout: 246 seconds]
niyawe has joined #ipfs
voltar13 has quit [Ping timeout: 240 seconds]
Am3nadiel has quit [Quit: FAMOUSamosfolks]
RamRanRa has quit [Read error: Connection reset by peer]
Newami has quit [Remote host closed the connection]
hqdruxn08 has quit [Ping timeout: 246 seconds]
captain_morgan20 has quit [Ping timeout: 240 seconds]
captain_morgan20 has joined #ipfs
<swedneck2>
is recursive add supposed to work for URLs?
<PhoenixDiscord[m>
I don't think so?
blallo has quit [Quit: No Ping reply in 180 seconds.]
<swedneck2>
alright good, nothing's broken then
<PhoenixDiscord[m>
If you have a mountable filesystem with a URL maybe you could use a FUSE tool to mount something like FTP or WebDAV to add it to IPFS?
<swedneck2>
i'll open an issue for adding it to go-ipfs
<swedneck2>
it would be extremely handy to be able to run `ipfs add -r https://example.com`
<PhoenixDiscord[m>
how would it discover what resources are located at a HTTP address? are you imagining go-ipfs would include a HTML parser that can navigate through every file referenced in every document and crawl through it?
<PhoenixDiscord[m>
a better tool for that is probably to use wget's mirror function
<swedneck2>
maybe it could call wget directly?
<swedneck2>
not sure, i just know it's a bit annoying to have to first mirror the entire site with wget and then add it to ipfs
<PhoenixDiscord[m>
not all platforms have wget installed as a standard utility, for instance, macOS and Windows don't come with wget, and it's not guarenteed on linux
<PhoenixDiscord[m>
it would still need to do that
<PhoenixDiscord[m>
and it wouldn't be perfect, a web scraper can't discover dynamic resources where the URL is generated by a script, which is pretty common
<PhoenixDiscord[m>
it's hard to imagine how ipfs would integrate with wget any better than just doing the mirror first, then adding it once it's downloading, it's unlikely to be faster or in any way really better
<swedneck2>
well, for one it could skip files it already has downloaded
<swedneck2>
or at least, immediately deduplicate the data once it realizes it's identical
blallo has joined #ipfs
hqdruxn08 has joined #ipfs
hqdruxn08 has quit [Max SendQ exceeded]
hqdruxn08 has joined #ipfs
<PhoenixDiscord[m>
how would it know it had already downloaded them, before downloading them?
<PhoenixDiscord[m>
it would need the full contents of the file in order to hash it and check
hqdruxn08 has quit [Max SendQ exceeded]
rendar has joined #ipfs
<PhoenixDiscord[m>
maybe something could be done along these lines with something like SSH, because the IPFS app could issue commands to ask OpenSSL or other commonly installed utilities to hash the contents of a file before deciding if it should download the data, but at that point, you might as well just run an IPFS node on the server that has the content
fling has joined #ipfs
xtarget has joined #ipfs
xtarget has quit [Client Quit]
pecastro has joined #ipfs
psychologist has quit [Read error: Connection reset by peer]
hqdruxn08 has joined #ipfs
hqdruxn08 has quit [Max SendQ exceeded]
hqdruxn08 has joined #ipfs
hqdruxn08 has quit [Max SendQ exceeded]
loxleo[m] has joined #ipfs
seba-- has joined #ipfs
Wall`e has joined #ipfs
Wall`e has quit [Remote host closed the connection]
aha[m] has left #ipfs ["Kicked by @appservice-irc:matrix.org : 30 day idle timeout."]
mapetik has joined #ipfs
alxev[m] has left #ipfs ["Kicked by @appservice-irc:matrix.org : 30 day idle timeout."]
azk64[m] has quit [Quit: 30 day idle timeout.]
BeatRupp[m] has left #ipfs ["Kicked by @appservice-irc:matrix.org : 30 day idle timeout."]
Caramba_3000[m] has quit [Quit: 30 day idle timeout.]
GeorgeAntoniadis has quit [Quit: 30 day idle timeout.]
geriatricjacob[m has quit [Quit: 30 day idle timeout.]
hiima[m] has quit [Quit: 30 day idle timeout.]
irongomme[m] has left #ipfs ["Kicked by @appservice-irc:matrix.org : 30 day idle timeout."]
jak_wolf[m] has left #ipfs ["Kicked by @appservice-irc:matrix.org : 30 day idle timeout."]
kremator[m] has quit [Quit: 30 day idle timeout.]
legendplays93[m] has quit [Quit: 30 day idle timeout.]
linuslee[m] has quit [Quit: 30 day idle timeout.]
logoilab[m] has quit [Quit: 30 day idle timeout.]
mack[m] has quit [Quit: 30 day idle timeout.]
michaelrao[m] has left #ipfs ["Kicked by @appservice-irc:matrix.org : 30 day idle timeout."]
miseenplace[m] has left #ipfs ["Kicked by @appservice-irc:matrix.org : 30 day idle timeout."]
mapetik has quit [Remote host closed the connection]
mapetik has joined #ipfs
nemon[m] has quit [Quit: 30 day idle timeout.]
NetluxeTV[m] has left #ipfs ["Kicked by @appservice-irc:matrix.org : 30 day idle timeout."]
noonereally[m] has left #ipfs ["Kicked by @appservice-irc:matrix.org : 30 day idle timeout."]
okhosting[m] has quit [Quit: 30 day idle timeout.]
palantir22[m] has quit [Quit: 30 day idle timeout.]
<SuikaDiscord[m]>
Hopefully a quick question: I have an ipfs node that has filestore enabled and indexed ~1tb of data with about 1M files. And iirc the hashes and what not are stored in `datastore/`.
<SuikaDiscord[m]>
Just now I noticed that my `blocks` folder ist ~90G big and the webui tells me that I host 86G of data. While data store if configured to `"StorageMax": "1GB"`
<SuikaDiscord[m]>
Feels like something is going wrong? Since datastore should be cleaned up if it gets too big. Or am I missing something?
<SuikaDiscord[m]>
*filestore enabled and data added with no-copy
sandmage[m] has quit [Quit: 30 day idle timeout.]
soapoperator[m] has left #ipfs ["Kicked by @appservice-irc:matrix.org : 30 day idle timeout."]
soren-rademacher has left #ipfs ["Kicked by @appservice-irc:matrix.org : 30 day idle timeout."]
Swedneck3 has left #ipfs ["Kicked by @appservice-irc:matrix.org : 30 day idle timeout."]
tasrev[m] has left #ipfs ["Kicked by @appservice-irc:matrix.org : 30 day idle timeout."]
thoth-el999[m] has left #ipfs ["Kicked by @appservice-irc:matrix.org : 30 day idle timeout."]
TionisNagir[m] has quit [Quit: 30 day idle timeout.]
xswe3456ui[m] has quit [Quit: 30 day idle timeout.]
zumar[m] has quit [Quit: 30 day idle timeout.]
<voker57>
SuikaDiscord[m]: data store is only cleaned when GC is run
<voker57>
automatic gc run is disabled by default
<voker57>
also, pinned data is not touched by gc
<SuikaDiscord[m]>
Ok, I need to run the whole thing with `--enable-gc`
KempfCreative has quit [Remote host closed the connection]
<swedneck2>
it makes so much more sense it feels physically pleasant
jcea has quit [Ping timeout: 245 seconds]
charley has quit [Remote host closed the connection]
charley has joined #ipfs
soju has joined #ipfs
jcea has joined #ipfs
Fessus has joined #ipfs
charley has quit [Ping timeout: 240 seconds]
soju has quit [Ping timeout: 265 seconds]
mapetik has quit [Remote host closed the connection]
Fessus has quit [Remote host closed the connection]
Fessus has joined #ipfs
luminiferousaeth has joined #ipfs
soju has joined #ipfs
jcea has quit [Ping timeout: 250 seconds]
soju has quit [Ping timeout: 240 seconds]
mloki has quit [Ping timeout: 246 seconds]
charley has joined #ipfs
tg has quit [Ping timeout: 250 seconds]
thomasanderson has joined #ipfs
charley_ has joined #ipfs
tg has joined #ipfs
<postablesDiscord>
Unpopular question, is the IPFS ecosystem approaching a critical point in time? Lots of people seem to be upset with the current state of the ecosystem and are getting frustrated. Across the board through GitHub issues, Twitter threads, etc.. it's quite clear something needs to change.
thomasanderson has quit [Ping timeout: 265 seconds]
<postablesDiscord>
I think as a community we need to not be hostile towards these feelings, and we need to understand why people have them, and come up with viable solutions
charley has quit [Ping timeout: 268 seconds]
soju has joined #ipfs
charley_ has quit [Ping timeout: 265 seconds]
nereusx has quit [Quit: Leaving]
soju has quit [Ping timeout: 240 seconds]
<lordcirth__>
postablesDiscord, I am a huge fan of IPFS, and I want it to succeed, but I can't justify using it in production even for personal use (for my use cases)
<postablesDiscord>
What are the reasonings for that?
<lordcirth__>
Because I can't get it to connect and stay connected reliably. Probably due in part to my bad internet connection.
<lordcirth__>
Also, I have an ipfs.service file that kills it and restarts it when it hits 6GiB of memory use, because of leaks
<lordcirth__>
But I'm still working on it. I'm in the middle of a PR to nixpkgs to update their ipfs-cluster version
zeden has quit [Quit: WeeChat 2.6]
soju has joined #ipfs
soju has quit [Ping timeout: 265 seconds]
soju has joined #ipfs
thomasanderson has joined #ipfs
mapetik has joined #ipfs
mapetik has quit [Remote host closed the connection]
thomasanderson has quit [Ping timeout: 240 seconds]
mapetik has joined #ipfs
Jybz has quit [Ping timeout: 265 seconds]
soju_ has joined #ipfs
soju has quit [Read error: Connection reset by peer]
<postablesDiscord>
Fair points
soju_ has quit [Remote host closed the connection]
Ai9zO5AP has quit [Quit: WeeChat 2.5]
jcea has joined #ipfs
Newami has joined #ipfs
Newami has quit [Remote host closed the connection]