aschmahmann changed the topic of #ipfs to: Heads Up: To talk, you need to register your nick! Announcements: go-ipfs 0.7.0 and js-ipfs 0.52.3 are out! Get them from dist.ipfs.io and npm respectively! | Also: #libp2p #ipfs-cluster #filecoin #ipfs-dev | IPFS: https://github.com/ipfs/ipfs | Logs: https://view.matrix.org/room/!yhqiEdqNjyPbxtUjzm:matrix.org/ | Forums: https://discuss.ipfs.io | Code of Conduct: https://git.io/vVBS0
<aschmahmann>
I guess one more thing I'd recommend is if you're freshly adding your data to consider doing it with `--cid-version=1` it'll make all the CIDs used in your DAG be more browser friendly and it'll make the leaf nodes of your files be encoded as raw bytes (i.e. it implies `--raw-leaves`) instead of wrapped with some protobuf.
<aschmahmann>
It's not a huge deal and they both work fine but just an FYI
<aschmahmann>
for sure, although they'll have to do it periodically to catch up. All depends on your constraints
<kallisti5[m]>
yeah, the repinning thing is a big issue
musalbas has joined #ipfs
mamu has joined #ipfs
mindCrime_ has joined #ipfs
<kallisti5[m]>
aschmahmann: so from a badger standpoint, is there anything that could be done to help prevent stuff like the issue I saw? I think the most likely issue was a hard reboot from the hosting company or something. I checked the logs and I provisioned the server 3 days ago... uptime is 2 days.
<kallisti5[m]>
I don't think I was adding files or anything then... but who knows. It feels like the CID got to disk, but the data in it didn't
Arwalk has quit [Read error: Connection reset by peer]
dsrt^ has quit [Ping timeout: 252 seconds]
<kallisti5[m]>
most databases are transnational to avoid stuff like this (setting a "complete" flag or something after disks have been flushed). I see people talking about badger v3, do you think there are any scenarios which could be improved here?
<kallisti5[m]>
I mean... from my perspective the data was working 100%. The only way I even noticed the issue was the pin silently not completing
dsrt^ has joined #ipfs
Arwalk has joined #ipfs
<kallisti5[m]>
I guess badgerdb isn't an IPFS thing but a more general thing
<kallisti5[m]>
aschmahmann: Thanks for the help! I blew away our repo and rebuilt it on a Rasberry Pi 4 at my house (more ram than the silly Vultr node lol). Things seem to be working now. Went with "flat file" over badger since data integrity on the "central seeding node" is more important that on nodes pinning content.
<kallisti5[m]>
<shameless plug>, please pin (and occasionally repin) ```/ipns/hpkg.haiku-os.org``` if you have a bit of extra space (~20 GiB today) and want to help host Haiku's package repositories + artifacts on IPFS :-) </shameless plug>
Soni has joined #ipfs
<Soni>
how do you make an ipfs mutable?
<McSinyx>
you don't, you make a new one
<McSinyx>
think of git
<McSinyx>
btw i have some data pinned by pinata, which is reachable via public gateways but not through the local ones (timed out) on a CI
<Soni>
dat allows things to evolve
<McSinyx>
<McSinyx "btw i have some data pinned by p"> would explicitly connecting the local node to the gateway helps the latency and how can I find the multiaddr to `ipfs swarm connect`
<McSinyx>
you can try ipfs mfs if i understand correctly what you meant by evolving
jrvlima has joined #ipfs
<aschmahmann>
Soni: There are a variety of ways to build mutable systems that link to immutable IPFS data. Some of the ones off the top of my head are IPNS, DNSLink, and Textile Threads
royal_screwup21 has quit [Quit: Connection closed]
voker57 has quit [Quit: No Ping reply in 180 seconds.]
voker57 has joined #ipfs
<Discordian[m]>
<McSinyx "btw i have some data pinned by p"> I wonder, I would imagine you could add some pinata nodes to your bootstrap to assist with this, but I can't find where you'd get a pinata multiaddress to bootstrap with.
hooway_ has joined #ipfs
hooway has quit [Read error: Connection reset by peer]
<McSinyx>
that'd be even better, but I couldn't find that documented anywhere either
<McSinyx>
how about the public gateways, can i connect to them directly?
<kallisti5[m]>
you can access files on public gateways, you really can't push files via public gateways
<kallisti5[m]>
One hard lesson is IPFS isn't magic. Everyone temporarily pulls data from your nodes when requested.
<McSinyx>
that's not wat I meant, kallisti5 (@kallisti5:matrix.org)
<McSinyx>
let me try to rephrase, I already have the files pinned locally and on pinata
<McSinyx>
since I want to have the checksum done by ipfs I want to access it via the local gateway
<kallisti5[m]>
If you have your data pinned locally, then you can connect to any gateway to access it.
<Discordian[m]>
<McSinyx "how about the public gateways, c"> I think so, but I'd personally try emailing them. I mean, they are selling a service of course ;p
<McSinyx>
yes, kallisti5 (@kallisti5:matrix.org), but I also need to run the thing on a CI
<kallisti5[m]>
(keeping in mind the speed will be dependent on the gateway "finding and pulling" the data from your node)
<McSinyx>
Discordian, yea that sounds like a good idea, let me try
<kallisti5[m]>
McSinyx: You need a "IPFS server" to host the data then.
<McSinyx>
I do
<kallisti5[m]>
and push from CI to that server to host it
<kallisti5[m]>
(making sure to securely expose the API from your IPFS node to the CI system
<McSinyx>
the data is on pinata (well my local bandwidth is shit so let's not count on it) and I cannot afford a dedicated machine with decent bandwidth and domain at the moment
<kallisti5[m]>
ok, then Pinata is your "IPFS server"
<kallisti5[m]>
Push to pinata, then you can access from any gateway
<McSinyx>
yep but it is slow, so I'm asking if I can explicitly connect to pinata or a public gateway to speed it up
<McSinyx>
(by slow I meant after a few minutes the content's still not found)
<McSinyx>
https would warrant the connection, not the actual content
<McSinyx>
driving the data through a ipfs node I can control would verify the integrity of the data
<kallisti5[m]>
That kinda goes back to my original statement. "IPFS isn't magic" You need to convince more folks to pin your data (and re-pin it) The more folks who pin it, the faster your data will be located in IPFS
<McSinyx>
eventually, yes
<kallisti5[m]>
I wasn't being a jerk saying that... it was a hard learned lesson on my part early on :-)
<McSinyx>
it's like the chicken an egg problem
<kallisti5[m]>
yup
<kallisti5[m]>
I have a ton of fast moving data I want to try and host on IPFS
<kallisti5[m]>
I stick it out there... but have to convince people to pin it (and re-pin it) so it becomes reliable
<kallisti5[m]>
also keep bandwidth in mind. You're going to have "inbound and outbound" bandwidth on pretty much every IPFS node / gateway
<McSinyx>
my use case is providing a software repository which would change less often
<McSinyx>
i kinda expect when it's used it'd behave similar to peertube and stuff will balance themselves and eventually dropping the need for a pinning service in favor of people running their own mirrors