<achin>
i also need to do some research to see what kind of perf you can get from zfs with a ZIL and l2arc
<whyrusleeping>
achin: Kubuxu just get a 1TB m2 nvme :)
<achin>
oooh
<whyrusleeping>
thats whats in my laptop
<whyrusleeping>
it screams
Atrus6 has joined #ipfs
<Kubuxu>
OVH has servers with NVMEs in their offer :p
<whyrusleeping>
kyledrake: sharding *is* in a release
<kyledrake>
Ah, good to know
<whyrusleeping>
though theres some fixes to viewing sharded things on the gateway that will be in the next release
Atrus6 has quit [Remote host closed the connection]
<Kubuxu>
Nice, just 44E additional for 2x1.2T NVME over HDD price
<kyledrake>
RE storage, there's a weird thing where you can use SSDs as a caching layer for a spinner disk array. I'm not sure what they use for it but I've seen it in the wild.
<Kubuxu>
kyledrake: I have my PC set up like this
<whyrusleeping>
same
<whyrusleeping>
my NAS, rather
<whyrusleeping>
zfs l2arc
<Kubuxu>
6TB of spinning disks with NVME backed bcache on top
<Kubuxu>
6TB of usable (12 TB of RAID disks)
SalanderLives has quit [Quit: Leaving]
<kyledrake>
I have a Ceph cluster right now in a datacenter that's 72TB (24TB usable). I can max it to 72TB usable if I need to. I'm working on getting IPFS on top of it this week.
<whyrusleeping>
kyledrake: <3
<achin>
:D
<kyledrake>
The only thing stopping me from fulling it up is the cost.. it's going to cost about $5500 to fully load it with hard drives
<Kubuxu>
do you need it right away and can you expend it in future?
leeola has quit [Quit: Connection closed for inactivity]
* Kubuxu
doesn't know a lot about state Ceph right now.
<emunand[m]>
i got a weird idea; how about attaching a raspberry pie connected to an ssd, a solar battery, and satellite internet, install IPFS on it, then strap it to a rocket and launch it into space?
<achin>
ipfs in space!
<achin>
interplanetary!
<emunand[m]>
inter earth orbit file system
<emunand[m]>
IEOFS
<Kubuxu>
emunand[m]: cubesats are almost that, just the hardware inside is a bit different
<kyledrake>
Kubuxu we're using 1.31% of it right now as-is, so there's plenty of room already. You can expand in the future, yeah
<Kubuxu>
you have to use processors with bigger transistors to prevent damages due to cosmic rays
<Kubuxu>
or use FPGAs and re-route around damaged areas
<whyrusleeping>
Kubuxu: not entirely accurate
<kyledrake>
Ceph is drinking from the firehose but once you get the hang of it it's the best thing going for data durability
<whyrusleeping>
larger transistors are used in space because the fabs that make space grade CPUs are quite a bit behind intel
<achin>
i've heard some not-so-good things about performance with ceph
<kyledrake>
Once you get ceph set up right and with the correct hardware you need, it screams.
<kyledrake>
You need to have an SSD just for the write journals for example. I also had slowness related to a bug in a 4.4 kernel driver that was solved by upgrading.
<kyledrake>
But once you get your head around it, it opens up a lot of doors. You could make an Internet Archive petabox with it no problem.
<kyledrake>
But it's insanely complicated and byzantine to get started with. Just use ZFS unless you have a very compelling reason to use Ceph. :)
<achin>
yeah, that's where i am
<achin>
i don't have the funds (or even the need) for large multi-machine storage clusters
<achin>
but damn if ceph isn't cool as hell
<kyledrake>
I'm running it in a datacenter for a web hosting service with a completely redundant high-availability architecture, for example.
<whyrusleeping>
can anyone ping QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ ?
<achin>
whyrusleeping: yes
<achin>
Pong received: time=82.21 ms
<Kubuxu>
whyrusleeping: that isn't entirely accurate, many radiation hardening processes are much harder to do with smaller transistors and smaller transistors require more rad hardening.
<Kubuxu>
and here they were running 4 non-hardened PowerPC processors in lockstep and they are in 500nm process
<Kubuxu>
whyrusleeping: a lot of yamux goroutines stuck in send function
<Kubuxu>
hmm
<Kubuxu>
scratch that,
<Kubuxu>
whyrusleeping: nothing really
<Kubuxu>
500 peers, a lot of bitswap sessions
<whyrusleeping>
hrm
<whyrusleeping>
i'm having trouble interacting with that node
<Kubuxu>
there are some yamux Read routines that haven't been touched in hours
<Kubuxu>
they might be locked
<Kubuxu>
so probably underlying connection failed but yamux didn't propagate the error
<Kubuxu>
whyrusleeping: ^^
<whyrusleeping>
hrm... I think it might be okay if reads havent moved in hours
<whyrusleeping>
just means the node on the other end isnt doing anything
<kevina>
whyrusleeping: I see what I can do about reviewing 3786 although I am really not good at code reviews for something I don't have a vested interest in
<kevina>
when I have a vested interest I know what to look for :)
<kevina>
like keks command rewrite code
<kevina>
also got several p.r. I need to work on
Akaibu has joined #ipfs
citizenErased has joined #ipfs
<Kubuxu>
Ok, I underestimated English wiki a bit, it is 600G total. Expected time for add is 14 days
<Kubuxu>
:P
<e337>
Does that include all edit history?
<achin>
Kubuxu: :D
<achin>
e337: no, just a static dumb
<achin>
dump*
<Kubuxu>
e337: no, just articles and assets
<e337>
How did you get the dump?
<Kubuxu>
We have extracted zim file from kiwix dumps
<emunand[m]>
i think it was an archive on dump.wikimedia.org
<emunand[m]>
the kiwix version
<e337>
Ah
chatter29 has joined #ipfs
<chatter29>
hey guys
<chatter29>
allah is doing
<chatter29>
sun is not doing allah is doing
<chatter29>
to accept Islam say that i bear witness that there is no deity worthy of worship except Allah and Muhammad peace be upon him is his slave and messenger
chatter29 has left #ipfs [#ipfs]
<emunand[m]>
ramadan is coming soon, but you should not force others to convert
<whyrusleeping>
i swear, we get this once a day
<achin>
that person (or more likely bot) is well known on freenode
<achin>
though i suspect there are a few copycats
therealklanni[m] has joined #ipfs
<therealklanni[m]>
oi
tilgovi has joined #ipfs
<emunand[m]>
well, his name is chatter
gmcabrita has quit [Quit: Connection closed for inactivity]
caiogondim has joined #ipfs
reit has quit [Quit: Leaving]
reit has joined #ipfs
_whitelogger has joined #ipfs
brendyn has joined #ipfs
_whitelogger has joined #ipfs
Foxcool has joined #ipfs
tilgovi has quit [Ping timeout: 240 seconds]
caiogondim has quit [Quit: caiogondim]
huxehuxe[m] has joined #ipfs
huxehuxe[m] has left #ipfs ["User left"]
chris613 has quit [Quit: Leaving.]
palkeo_ has joined #ipfs
Oatmeal has joined #ipfs
infinity0 has joined #ipfs
tilgovi has joined #ipfs
athan has joined #ipfs
brendyn has quit [Ping timeout: 240 seconds]
qjkxkcd has quit [Ping timeout: 245 seconds]
Aranjedeath has quit [Quit: Three sheets to the wind]
qjkxkcd has joined #ipfs
tilgovi has quit [Quit: No Ping reply in 180 seconds.]
tilgovi has joined #ipfs
akkad has quit [Quit: Emacs must have died]
Foxcool has quit [Ping timeout: 260 seconds]
asyncsec has quit [Quit: asyncsec]
<walle303>
Man, that stupid islam bot is even affecting here?
<alu>
ive seen it on other networks
<walle303>
Just ban the nick, everyone says it uses the same nick, at least today
<walle303>
it's been in quite a few channels
<walle303>
Exact same message, it's definitely a bot
<whyrusleeping>
i'm mostly curious about the why
kerozene has quit [Remote host closed the connection]
asyncsec has joined #ipfs
dignifiedquire has quit [Quit: Connection closed for inactivity]
john2 has joined #ipfs
john2 has quit [Ping timeout: 240 seconds]
kerozene has joined #ipfs
Guest20366 has joined #ipfs
palkeo_ has quit [Quit: Konversation terminated!]
mildred has quit [Read error: Connection reset by peer]
mildred2 has quit [Read error: Connection reset by peer]
mildred has joined #ipfs
mildred2 has joined #ipfs
CMCDragonkai has joined #ipfs
CMCDragonkai is now known as Guest91804
asyncsec has quit [Quit: asyncsec]
Guest20366 has quit [Ping timeout: 260 seconds]
johnhenry[m] has joined #ipfs
rendar has joined #ipfs
tilgovi has quit [Quit: No Ping reply in 180 seconds.]
spilotro has quit [Ping timeout: 268 seconds]
tilgovi has joined #ipfs
spilotro has joined #ipfs
CMCDragonkai has joined #ipfs
CMCDragonkai is now known as Guest195
Guest91804 has quit [Ping timeout: 260 seconds]
kerozene has quit [Remote host closed the connection]
cehteh has quit [Ping timeout: 260 seconds]
kerozene has joined #ipfs
dignifiedquire has joined #ipfs
Caterpillar has joined #ipfs
btmsn has joined #ipfs
cehteh has joined #ipfs
digitalism has quit [Ping timeout: 245 seconds]
mahloun has joined #ipfs
<whyrusleeping>
!pin QmWNU1cRjuBGk8K6B4N3LKasgjqME8eiJpnUkUpupjxVLC dists with 0.4.9-rc1
<xelra>
I've been eyeing this feature for my home- and small company network for some time.
soesoe has joined #ipfs
<xelra>
Because sync solutions are problematic because of the required disk space on each device and other distributed filesystems were more pain than gain.
<xelra>
Especially over WAN.
soesoe has quit [Ping timeout: 240 seconds]
Foxcool has quit [Ping timeout: 240 seconds]
shizy has joined #ipfs
sprint-helper has quit [Remote host closed the connection]
sprint-helper has joined #ipfs
Foxcool has joined #ipfs
victorbjelkholm has quit [Ping timeout: 264 seconds]
voldyman has quit [Ping timeout: 264 seconds]
bbielsa has quit [Ping timeout: 264 seconds]
bergie has quit [Ping timeout: 264 seconds]
dlight has quit [Remote host closed the connection]
Sigma has quit [Ping timeout: 240 seconds]
tiago has quit [Ping timeout: 240 seconds]
bcavileer has quit [Ping timeout: 264 seconds]
shakalaka has quit [Ping timeout: 264 seconds]
riveter has quit [Ping timeout: 264 seconds]
caseorganic has quit [Ping timeout: 264 seconds]
poga has quit [Ping timeout: 264 seconds]
akemot has quit [Ping timeout: 240 seconds]
pent has quit [Ping timeout: 240 seconds]
dsal has joined #ipfs
bmpvieira has quit [Ping timeout: 255 seconds]
jzelinskie has quit [Ping timeout: 264 seconds]
voxelot has quit [Ping timeout: 264 seconds]
[0__0] has quit [Ping timeout: 241 seconds]
redfish has quit [Ping timeout: 260 seconds]
bigbluehat has quit [Ping timeout: 276 seconds]
gmcabrita has quit [Ping timeout: 276 seconds]
Ekho has quit [Ping timeout: 255 seconds]
bergie has joined #ipfs
zignig has quit [Ping timeout: 240 seconds]
gorhgorh has quit [Quit: No Ping reply in 180 seconds.]
mmuller has quit [Ping timeout: 264 seconds]
Guest35394 has quit [Ping timeout: 264 seconds]
taravancil has quit [Ping timeout: 264 seconds]
vasile has quit [Ping timeout: 264 seconds]
NeoTeo has quit [Ping timeout: 264 seconds]
oleavr has quit [Ping timeout: 255 seconds]
lohkey has quit [Ping timeout: 255 seconds]
ashark has quit [Ping timeout: 276 seconds]
dignifiedquire has quit [Ping timeout: 276 seconds]
robogoat_ has quit [Ping timeout: 240 seconds]
Captain_Beezay has quit [Ping timeout: 264 seconds]
dvim has quit [Ping timeout: 264 seconds]
kumavis has quit [Ping timeout: 264 seconds]
jnes has quit [Ping timeout: 240 seconds]
asui has quit [Ping timeout: 258 seconds]
sickill has quit [Ping timeout: 255 seconds]
gorhgorh has joined #ipfs
riveter has joined #ipfs
pent has joined #ipfs
jnagro has quit [Ping timeout: 276 seconds]
Intensity has quit [Ping timeout: 258 seconds]
ckwaldon1 has joined #ipfs
tiago has joined #ipfs
tiago has joined #ipfs
tiago has quit [Changing host]
brixen has quit [Ping timeout: 264 seconds]
anderspree_ has quit [Ping timeout: 264 seconds]
jnes has joined #ipfs
zignig has joined #ipfs
ckwaldon has quit [Ping timeout: 255 seconds]
ckwaldon1 is now known as ckwaldon
bigbluehat has joined #ipfs
qjkxkcd has quit [Ping timeout: 240 seconds]
ebarch has quit [Ping timeout: 240 seconds]
manveru has quit [Ping timeout: 264 seconds]
Akaibu has quit [Ping timeout: 255 seconds]
caseorganic has joined #ipfs
NeoTeo has joined #ipfs
anderspree_ has joined #ipfs
jnagro has joined #ipfs
mmuller has joined #ipfs
wedowmaker__ has quit [Ping timeout: 240 seconds]
vasile has joined #ipfs
trn has quit [Ping timeout: 255 seconds]
pedrot has quit [Ping timeout: 264 seconds]
Sigma has joined #ipfs
Captain_Beezay has joined #ipfs
qjkxkcd has joined #ipfs
oleavr has joined #ipfs
bbielsa has joined #ipfs
shakalaka has joined #ipfs
poga has joined #ipfs
brixen has joined #ipfs
Ekho has joined #ipfs
sickill has joined #ipfs
pedrot has joined #ipfs
ashark has joined #ipfs
jonnycrunch has joined #ipfs
joelburget has quit [Ping timeout: 255 seconds]
Intensity has joined #ipfs
Guest35394 has joined #ipfs
bmpvieira has joined #ipfs
robogoat has joined #ipfs
jonnycrunch has quit [Client Quit]
victorbjelkholm has joined #ipfs
voldyman has joined #ipfs
Guest74377 has quit [Ping timeout: 268 seconds]
gmcabrita has joined #ipfs
redfish has joined #ipfs
bcavileer has joined #ipfs
voxelot has joined #ipfs
jzelinskie has joined #ipfs
lohkey has joined #ipfs
dvim has joined #ipfs
manveru has joined #ipfs
akemot has joined #ipfs
asui has joined #ipfs
solvoid[m] has joined #ipfs
mahloun has quit [Ping timeout: 258 seconds]
solvoid[m] has left #ipfs [#ipfs]
physiognomize has quit [Ping timeout: 240 seconds]
Foxcool has quit [Read error: Connection reset by peer]
solvoid[m] has joined #ipfs
trn has joined #ipfs
vasile has left #ipfs [#ipfs]
Foxcool has joined #ipfs
Guest223053[m] has joined #ipfs
citizenErased has joined #ipfs
Foxcool has quit [Ping timeout: 246 seconds]
espadrine has quit [Ping timeout: 240 seconds]
joelburget has joined #ipfs
<lgierth>
whyrusleeping: yeeah that was a cjdns hickup...
<lgierth>
!pin QmWNU1cRjuBGk8K6B4N3LKasgjqME8eiJpnUkUpupjxVLC dists with 0.4.9-rc1
<pinbot>
now pinning on 8 nodes
<whyrusleeping>
lgierth: <3
<dsal>
whyrusleeping: Good day, sir.
<whyrusleeping>
dsal: Hey!
<dsal>
A friend of mine told me I absolutely must try IPFS.
<whyrusleeping>
Well that sounds like a pretty solid recommendation
citizenErased has quit [Ping timeout: 255 seconds]
<whyrusleeping>
lgierth: i think cjdns is broken
<lgierth>
meh :/
<whyrusleeping>
dsal: you can grab a prebuilt binary over here: https://dist.ipfs.io
<lgierth>
will downgrade to latest release
galois_d_ has joined #ipfs
<dsal>
Yeah, I got it running last night.
<whyrusleeping>
or build from source
<whyrusleeping>
ah, nnice
<dsal>
Did one of each.
<dsal>
Haven't exactly figured out what it does for me just yet.
<whyrusleeping>
dsal: well that depends on what you actually want to do with it
<dsal>
One thing that might be useful is my home camera system. For that, I capture and then upload to a GAE app that has authentication to pull blobs.
spossiba has quit [Ping timeout: 260 seconds]
<dsal>
For this to solve that problem (which would be nice), I'd need a) some kind of blob privacy and b) some kind of auto-external pinning.
[0__0] has joined #ipfs
reit has joined #ipfs
<whyrusleeping>
dsal: for privacy, you can use an ipfs private network. Your nodes willl only communicate with eachother, using a shared psk for encryption
galois_dmz has quit [Ping timeout: 258 seconds]
<whyrusleeping>
for auto-external pinning, theres nothing built in yet, but you could use pubsub to publish the latest hashes from your cap server, and then subscribe and pin those from your backup server
<dsal>
Oh neat.
<dsal>
I'd also like to spread that across multiple machines.
<dsal>
Did you ever see CBFS? That solved most of my problems pretty directly. :)
<whyrusleeping>
CBFS? I'm not sure if i'm familiar with it
<dsal>
It's the thing I wrote a long time ago to solve my problems.
<dsal>
Basically, you could just take a machine, add it to the (LAN) cluster, and the whole cluster would agree to ensure a minimum number of replicas of all your data.
<dsal>
Self healing and all that.
<dsal>
It also had expiration which was nice because much of the data I stored I didn't need for a long time.
<dsal>
ipfs-cluster might be solving the problem I solved in cbfs. That'd be super nice.
<dsal>
Effectively, I added both capacity and redundancy by just plugging in a new machine. You could turn off any machine, and the whole thing would just heal around it. Turn off two machines and you only have to be nervous for like, an hour.
<dsal>
Of course, after I left couchbase, they had a "data loss" in CBFS, which I declared impossible until I found out they turned off like, five or so machines at approximately the same time.
* whyrusleeping
reading
<whyrusleeping>
thats pretty cool
<whyrusleeping>
I think ipfs-cluster is what youre looking for on that front
<whyrusleeping>
if you want to solve the same problem using ipfs
<whyrusleeping>
How did the expiration work?
<dsal>
Yeah. I really liked cbfs, but I'm happy to use something else if I can make it work.
<dsal>
cbfs had a logically centralized metadata service that supported expiration. All objects were keyed by user-specified names and the names could expire.
<dsal>
When an object had no names, it was subject to gc
<lgierth>
in ipfs it's pins
<dsal>
There was a backup mechanism that stored snapshots of all the metadata in cbfs itself, and it would consider any object that appeared in a backup as live. It was super hard to lose data.
<whyrusleeping>
interesting, I'd love to find a nice way to make that metadata service type thing work over ipfs
<whyrusleeping>
aka, decentralized
spossiba has joined #ipfs
<dsal>
Yeah. Using couchbase for that made huge sense - that was my company. It had great benefits, too, but there were things I couldn't do.
<dsal>
Mainly, I just want to figure out how to spread storage across machines. E.g., I'd like to not store stuff on my arm boxes, but ensure anything I did store was safe.
<whyrusleeping>
That should be easy enough with ipfs-cluster
<whyrusleeping>
though the only replication strategy right now is mirror on all machines
<whyrusleeping>
(its pluggable, we just havent gotten around to more interesting strategies)
<dsal>
Yeah, cbfs was really good about utilization concerns. Each node had a maximum amount it would allow and the system would just work it out.
<whyrusleeping>
mmm, that is something we want
<dsal>
About given object could have a minimum of N replicas and a maximum of Y. Periodic scans would find underreplicated or overreplicated objects and fix them up.
<dsal>
(things that don't make sense are my phone's fault)
<whyrusleeping>
irc on your phone? is it 2017 already?
<dsal>
Overreplication occurred when an object was requested from a node that didn't have it. There was a "make a copy here" probability.
<whyrusleeping>
hrm... was it just a polling type mechanism? or did it have some consensus among nodes?
<dsal>
Popular objects were highly replicated while they were popular.
<dsal>
There were tasks each node would perform and global tasks. Global tasks worked by racing for a lock and then examining metadata
<dsal>
Nodes verified integrity of all their blobs and that they were registered as holding them. Global tasks would ask nodes to add or remove replicas.
togart has joined #ipfs
<dsal>
Expiration helps with this kind of thing.
<dsal>
Locks all expired, e.g.
blipz has left #ipfs ["Be back later..."]
conway has joined #ipfs
<dsal>
I imagine you could do a pretty good job of this with a single centralized process running a cluster that just balanced the pinning about.
Guest74377 has joined #ipfs
<lgierth>
just need to take into account that ipfs is dealing with graphs
<dsal>
I'm not sure that matters for my concern.
<lgierth>
i.e. if something gets unpinne,d some subgraph might still be pinned regardless
<dsal>
Oh, I see what you mean.
<dsal>
Yeah, all of my objects were just explicitly there or not. pinning itself wasn't a concept.
<dsal>
I guess it makes it a little harder to stay balanced, but being balanced is hard anyway.
anewuser has joined #ipfs
tilgovi has joined #ipfs
chris613 has joined #ipfs
<lgierth>
well i'd love to see ipfs-cluster pick up some usage :)
<dsal>
Yeah, it looks neat. I've also got the problem of only having one actual computer with a hard drive running at home. heh
asyncsec has quit [Remote host closed the connection]
asyncsec has joined #ipfs
<dsal>
Is there a reasonable way to store my objects with privacy on the existing network?
<dsal>
I guess I could make a little web server thing that does decryption or something.
<dsal>
It'd be nice if I could do that in the browser somehow.
<lgierth>
right now you'd need to encrypt them yourselves, but we'd love to see that within ipfs itself
<dsal>
It'd be a great replacement for my GAE app, but I suppose it could still be the storage for my GAE app.
<dsal>
Though it's not totally obvious to me how to have just a plain directory I'm adding stuff to yet.
Guest74377 has quit [Ping timeout: 240 seconds]
<dsal>
My app is basically [location]/[timestamp].{json,jpg,avi,mp4}
<victorbjelkholm>
lgierth: something weird is happening all over the place, having the same issues :/ Multiple websites are responding weirdly
mahloun has joined #ipfs
<lgierth>
right now there's two way to keep a directory synced into ipfs: the cmd/ipfswatch tool in go-ipfs which is inotify based, and the FUSE mounts
dignifiedquire has joined #ipfs
<lgierth>
victorbjelkholm: more inf pls?
taravancil has joined #ipfs
<lgierth>
what does weirdly mean
<dsal>
Does FUSE automatically pin?
galois_d_ has quit [Remote host closed the connection]
<lgierth>
noot sure
<dsal>
What I want is for objects I write to be pinned somewhere, but pretty much never where I'm actually placing them. :)
galois_dmz has joined #ipfs
<lgierth>
sigh i'm gonna go back home, this network is unbearable
anewuser has quit [Quit: anewuser]
kumavis has joined #ipfs
wedowmaker__ has joined #ipfs
<victorbjelkholm>
lgierth: websites being down, probably some cloudflare downtime or silly things like that
<dsal>
whyrusleeping: So is it safe to say that "pin management" is basically not a concern of IPFS proper? i.e. my concerns for data safety fall outside the main bits.
<whyrusleeping>
dsal: yeah, right now pin management isnt a 'core feature'
<whyrusleeping>
You can pin and unpin things
<whyrusleeping>
but the interfaces there are far from ideal
<dsal>
The whole thing mostly makes sense to me.
akkad has joined #ipfs
<dsal>
What is gateway.ipfs.io? Sounds expensive.
<dsal>
It's got my site on it now, so get ready to raise those serving bills.
<whyrusleeping>
Heh, we can deal
<whyrusleeping>
Our traffic has been steadily picking up lately
<whyrusleeping>
our setup is still kinda overkill
<dsal>
So, ipns -- is there any kind of magical pin thing I can do to ensure that all the things are pinned somewhere?
<whyrusleeping>
not yet, thats something we're planning on doing, but it actually turns out to be rather difficult
<dsal>
yeah, I'd think so
<dsal>
I just want to know how to think about my data being safe.
<xSkyripper>
hey guys, any update on the DHT & Relay integration on js-ipfs ?
<whyrusleeping>
xSkyripper: Hrm... i'm not sure. Havent paid much attention to the js side of things in a while
<whyrusleeping>
dsal: yeah, if data is pinned on a reliable host (one whose disks you trust not to die) then your data is 'safe'
<xSkyripper>
so there isn't an approximate time until it will be up ?
strykerkkd has joined #ipfs
<whyrusleeping>
xSkyripper: i have no idea, you would have to ask dryajov and daviddias
<xSkyripper>
thanks
rendar has joined #ipfs
rendar has joined #ipfs
rendar has quit [Changing host]
john4 has joined #ipfs
john2 has quit [Ping timeout: 240 seconds]
<dsal>
How does gc work, btw?
john2 has joined #ipfs
bsm1175321 has joined #ipfs
<dryajov>
xSkyripper: circuit is mostly coded but needs thorough review, so I expect it to be ready soon ish, at least as soon as daviddias reviews it
Guest223250[m] has joined #ipfs
<dryajov>
Not sure about dht
Guest223250[m] has left #ipfs [#ipfs]
john4 has quit [Ping timeout: 255 seconds]
<whyrusleeping>
dsal: it does a typical mark and sweep garbage collection using the pins and their descendants as the colored set
<xSkyripper>
circuit is the mechanism that will put off the use of the webrtc signalling server ?
<dsal>
whyrusleeping: So anything not pinned is eligible for gc?
<whyrusleeping>
correct
Foxcool has joined #ipfs
<whyrusleeping>
i'm very interested in more complex journal based caching systems though
<whyrusleeping>
where things that are commonly or recently used won't be gc'ed
<whyrusleeping>
even if they arent pinned
<dsal>
I ended up with an "and must be at least an hour old" in cbfs to cover a few cases, such as things that are just being added and haven't been pinned yet.
<dsal>
But yeah, recently used would be nice to keep.
<whyrusleeping>
yeah... its really a metadata problem at this point
<whyrusleeping>
unsure how to store info about object age
<xSkyripper>
dryajov: or is that DHT ?
maxlath has quit [Ping timeout: 240 seconds]
jonnycrunch has joined #ipfs
<dryajov>
Circuit is relay :) sorry we've been using them interchangeable :)
<dryajov>
Circuit-relay will allow connecting nodes that can't otherwise reach them self.
<xSkyripper>
I've run 2 instances on 2 devices on a somewhat "local" network, more exactly 2 routers in the local reach
<xSkyripper>
but only the first one that finished initializing the js daemon could see the slower one
<xSkyripper>
will circuit solve that ? :D
<dryajov>
Well, the use case is more along the lines of having two nodes that don't speak the same transports, I've TCP node trying to dial a browser node directly
<xSkyripper>
these 2 devices use js-ipfs in "browser" environment, more exactly webview (android)
<xSkyripper>
on the same router they were connected and they were able to sub / pub on the same topic
<xSkyripper>
but with 2 different routers in the same building, they couldn't
<dryajov>
Say that node only talks TCP and knows nothing about websockts or webrtc... And needs to dial to a web only node that talks websockets. The TCP node would use a third node as a relay to reach the browser node
Oatmeal has quit [Quit: Suzie says, "TTFNs!"]
jonnycrunch has quit [Quit: leaving]
<dryajov>
The cool thing is that, circuit addresses are disable, so your browser node can have a phisical address now that you can dial it on, which will go over either a specific relay or an arbitrary one.
<kythyria[m]>
Disable?
<xSkyripper>
assuming that I'm only interested on the browser nodes
<xSkyripper>
and my future app won't really care about TCP nodes
<xSkyripper>
I'd like to have automatic discovery between these browser nodes, without using the signalling server
<xSkyripper>
did I understand smth wrong ? :D
<whyrusleeping>
xSkyripper: you'll want both the dht and the relay
<kythyria[m]>
Why would you be able to rule out any possibility of a non-JS node?
<xSkyripper>
well, yes. that's what I'm looking for and I'm kinda nervous on when these updates will come
<xSkyripper>
because the node will live inside a webview, always
<xSkyripper>
it's a mobile app
<dryajov>
Yeah, in that scenario you still need a relay node, the reason is because you can't dial from browser to browser
<xSkyripper>
aaand, that's what the webstar does at the moment, right ?
<kythyria[m]>
xSkyripper: mobile app with no servers whatsoever?
<xSkyripper>
it dials for you
<xSkyripper>
yes ..
<dryajov>
There is no transport that allows you to dial a browser directly, the browser is the one initialing connections, bottom like, the browser doesn't allow you to create sockets
<kythyria[m]>
And under no circumstances will you ever have any server or non-JS client? Quite a bold statement.
<whyrusleeping>
we're still going to have signalling servers
<xSkyripper>
well, the main idea was that if the app user couldn't reach the global internet connectivity
<whyrusleeping>
but any ipfs node will be able to be a signalling server in the future
<xSkyripper>
it could use the IPFS to reach other nodes
<dryajov>
Well no, webstart is a webrtc thing, so your circuit can use webrtc as a transport. Circuit would use webrtc as a transport
<kythyria[m]>
xSkyripper: You'll need to do that bit outside your webview, I guess.
<dryajov>
So yeah, we would still have webrtc signaling servers
<xSkyripper>
kythyria[m]: supposing that node.js works on mobile ... ?
<xSkyripper>
so there's no way to create something like what I imagined .. ?
<victorbjelkholm>
supposedly, you can have each cell phone act like a signal server if you run a android background activity / ios equivalent
<kythyria[m]>
You can't listen from inside a browser yet, so anything depending on listening will need you to go outside the webview.
engdesart has quit [Quit: yes]
<victorbjelkholm>
there is also wifip2p in android, that might be useful for finding any local signaling servers
<xSkyripper>
victorbjelkholm: so each device can act like a node AND a signalling server ? that way will allow other nodes to find it "locally" ?
<victorbjelkholm>
xSkyripper: please note that these are just ideas, not something that already been implemented. But yeah, node + signal. The signaling part would still have to be outside the webview though
btmsn has quit [Quit: btmsn]
<xSkyripper>
kythyria[m]: problem is I've planned to use a js framework for android. ATM I know that pubsub works fine on go-ipfs and js-ipfs. The only API that could use pubsub is js-ipfs-api but that doesn't work in browsers
<xSkyripper>
victorbjelkholm: I see. Supposing that it would work that way, each node would swarm on what address ?
caiogondim has quit [Quit: caiogondim]
<xSkyripper>
my final goal is to achieve that "no internet backbone" feature so my app would use that :D
Encrypt has joined #ipfs
fleeky has quit [Read error: No route to host]
caiogondim has joined #ipfs
jonnycrunch has joined #ipfs
maxlath has joined #ipfs
<xSkyripper>
another option would be to bring go-ipfs on that device and use it. But I'd need pubsub to work on js API ..
<xSkyripper>
and it doesn't AFAIK, right ?
<dryajov>
xSkyripper: not sure about pub/sub in go… whyrusleeping ?
<whyrusleeping>
pubsub in go works
<whyrusleeping>
whats the question?
<dryajov>
in a local network, you might have a device that works as a circuit, in which case you can rely on it “route” all your traffic internally
<xSkyripper>
using js-ipfs-api in webview (practically browser env ? ) to use the go-ipfs's pubsub
<xSkyripper>
dryajov: kinda kills my app's purpose :(
<xSkyripper>
whyrusleeping: js-ipfs-api pubsub feature will work for go-ipfs ?
<whyrusleeping>
yeah, it should
<xSkyripper>
in a webview ?
<whyrusleeping>
you should also be able to pubsub with just js-ipfs
<whyrusleeping>
and that should communicate with go-ipfs just fine
<xSkyripper>
yes but using only js-ipfs doesn't satisfy all my needs
<xSkyripper>
especially in the browser
<dryajov>
xSkyripper: yeah, if i understand you correctly you basically want direct browser to browser communication, without using anything external?
<dryajov>
or atleast, have the third thing confined to the device your browser runs on?
<xSkyripper>
webview to webview counts just like browser to browser
<xSkyripper>
right ?
<xSkyripper>
yeap
<xSkyripper>
let me explain it shortly: I imagined an app that could help people localize victims of natural disasters
fleeky has joined #ipfs
<xSkyripper>
assuming those victims have local network connectivity but no internet backbone connectivity
<xSkyripper>
.. does that makes sense in some way ? :D
<xSkyripper>
so far I've learnt from you that I cannot have automatic discovery only using the daemon in the webview
<xSkyripper>
keeping in mind the idea of no internet backbone connectivity
<dryajov>
so, internet is not strictly required… what is needed is a third node that will serve either as a relay, or if you have webrtc, with an internal signaling server it should work
morantron has joined #ipfs
<dryajov>
say for example that a bulding is cut off from the internet, but you still want to “ping” the residents and see if they respond - pub/sub.
<xSkyripper>
yes
<dryajov>
you don’t really need a browser to solve that scenario, but lets say that you only have access to a browser...
jjanzic has joined #ipfs
<dryajov>
with both, js-ipfs(node) and go-ipfs you can listen on different transport protocols, such as TCP, WebRTC and WebSockets, where the go and js implementations only share a subset of those transport, say TCP
<dryajov>
you don’t really need a backbone to discover nodes, there are different discovery mechanists that IPFS supports, one of them is mDNS (bonjour)
<dryajov>
so nodes can discover other nodes using that…
<dryajov>
once several nodes are connected they can talk to each other, if the spek the same protocol
<dryajov>
ok, now back to the browser,
espadrine has joined #ipfs
<dryajov>
you want a browse to be able to subscribe to some nodes in your local “meshy” network, without access to the backbone…
chungy has joined #ipfs
<xSkyripper>
that's right
espadrine_ has joined #ipfs
<dryajov>
that is possible, (tho i’m not sure if mdns works in the browser) but regardless… you need several things to make that happen
<dryajov>
1) you need your browser node to connect to other peers - it can do that with either WebRTC or WS
jonnycrunch has quit [Quit: leaving]
Foxcool has quit [Ping timeout: 240 seconds]
<dryajov>
2) you need your browser node to connect to other peers that don’t speak either WebRTC or WS
<dryajov>
the first part is implicitelly taken care of if the browser node is trying to connect to a node that speaks either of those protos
<dryajov>
the second one is trickier, since if two nodes don’t speak the same transports they can’t talk to each other
<dryajov>
ie: browser node and go node that only speaks TCP
<xSkyripper>
isn't my case only reduced to browser node - browser node ?
sirdancealot has joined #ipfs
<dryajov>
I’ll get there in a sec :{
<dryajov>
:)
<xSkyripper>
okay :D
<dryajov>
so Browser to Browser is not possible without an intermediary node
<dryajov>
that is the gist of it…
<dryajov>
but that doesn’t preclude you from building your app...
<morantron>
I think I was going to ask the same question lol
<dryajov>
that is where circuit comes in… it allows two otherwise unreachable nodes, like browser to browser for example to communicate directly
<dryajov>
but it requires a circuit node in between… (or several)
<xSkyripper>
and what if we have no circuit nodes ?
espadrine` has joined #ipfs
<dryajov>
tough luck
<dryajov>
:)
espadrine has quit [Ping timeout: 276 seconds]
<dryajov>
again, the main problem here is that browsers don’t do sockets...
<dryajov>
it might be possible to build something with bluetooth maybe
<xSkyripper>
the range is too short compared to wifi ..
<dryajov>
which essentially allows two browsers to communicate directly
<emunand[m]>
bluetooth p2p?
onabreak has quit [Ping timeout: 260 seconds]
<emunand[m]>
i think that exists on android
<emunand[m]>
you can send files to other phones through bpuetooth
<xSkyripper>
yes but I'm limited to the webview ..
<dryajov>
its a pretty nifty project, and it allows doint http/websockets from browser to browser,
<victorbjelkholm>
xSkyripper: hold out for Bluetooth 4, supposed to help with range
<dryajov>
basically an embeded webserver
<dryajov>
+1
<victorbjelkholm>
xSkyripper: are you using cordova/phonegap or similar? You might be able to write a plugin that exposes a interface for js-ipfs to use, then you can use sockets
<morantron>
total noob here: I want to make a distributed webapp with IPFS. Two things that I don't quite get: are you supposed to use the public gateways, or should you host your own IPFS nodes? Also, is it possible to go browser-only, or would you need some nodes that are able to persist files in IPFS?
<dryajov>
good point… cordova is a much more permissive environment than pure browser
<xSkyripper>
victorbjelkholm: my app will use cordova/phonegap as a foundation
shizy has quit [Ping timeout: 258 seconds]
<xSkyripper>
i forgot to specify that, sorry
<emunand[m]>
morantron: host your own nodes and after js-ipfs, yes
<victorbjelkholm>
xSkyripper: then you have the possibility for a lot more control, look into exposing sockets via cordova and having a ipfs-cordova-sockets-transport or something like that
<xSkyripper>
so the main idea would be to use a "lower level"
<xSkyripper>
instead of using a webview created by cordova in which i start i daemon, i could start the daemon inside a cordova plugin ?
<xSkyripper>
a daemon *
mahloun has quit [Ping timeout: 255 seconds]
<morantron>
i wonder if providing a desktop version of the app is a good idea ( desktop app users would serve as nodes )
<morantron>
#ihavenoideawhatimdoing
<victorbjelkholm>
xSkyripper: no, still using a daemon in the webview, but you write a transport for sockets that you expose via cordova. Imagine https://github.com/libp2p/js-libp2p-tcp but instead of tcp, it uses whatever interface your plugin exposes
espadrine` has quit [Ping timeout: 240 seconds]
<victorbjelkholm>
almost want to try writing that plugin myself... Would be a really nice module to have
<xSkyripper>
so there isn't a way easier than this ?
<ashark>
I tried to move a bunch of stuff onto IPFS last weekend but ran into a couple problems: 1) committing to exact, semi-permanent filenames/structures is painful if I want to add whole directories, but 2) adding single files or more atomic directories makes it hard to treat those as a collection for software that wants directories of stuff, and to keep track of what I've got pinned. Anyone have advice for
<ashark>
overcoming those issues?
<victorbjelkholm>
xSkyripper: signal server as background activity is the only other way I can think of...
<xSkyripper>
what about fetching the go-ipfs, initing a daemon with it and using the js-ipfs-api in the cordova webview ?
<xSkyripper>
would that work ?
<xSkyripper>
assuming that I also need the pubsub feature ..
<victorbjelkholm>
xSkyripper: yeah, been some experiments of running go-ipfs on android, on iOS I have no idea. https://github.com/ligi/IPFSDroid is the one I know off
<victorbjelkholm>
maybe you're better off writing a native app instead of with cordova
<xSkyripper>
why would that be better ?
<xSkyripper>
cordova / phonegap has shell exec that would suffice to start the daemon .. right ?
dlight has joined #ipfs
<victorbjelkholm>
oh, didn't know you could run shell commands via cordova. Probably a android only feature no?
<xSkyripper>
android would be enough for me. And I thought this could work, especially now that dryajov explained why only using js-ipfs in a browser (webview) env won't allow direct connections atm
<xSkyripper>
let's say the go-ipfs would start and everything is fine. will the js-ipfs-api pubsub feature work properly in the cordova webview ?
<dryajov>
might work either out of the box or with a tin layer on top of js-libp2p-tcp
<xSkyripper>
I do not fully understant the thing behind this plugin idea :-s
<dryajov>
victorbjelkholm, Skyripper
<xSkyripper>
understand*
<xSkyripper>
so my daemon inside the webview would use cordova sockets to directly communicate with other nodes ?
<dryajov>
that would allow you to run a full js-ipfs node in cordova that can also communicate over tcp without the need of a third node
<dryajov>
pretty much what you’re looking for
Oatmeal has joined #ipfs
<dryajov>
I haven’t tried it myself, but it would be something I’d give a shot
<xSkyripper>
the node would still live inside the cordova's webview ?
<xSkyripper>
or within a plugin ?
<victorbjelkholm>
dryajov: yeah, that's what I'm thinking too! So, the plugin above (or your own) would expose a JS API for interacting with native sockets, then you can write a cordova-transport for libp2p and inject that when starting your js-ipfs node in the webview
<victorbjelkholm>
all traffic would go via the plugin, but the node itself lives inside the webview
<dryajov>
yeah, that be pretty sweet to try out
<victorbjelkholm>
yeah, my fingers are itching to try implementing it as well!
<dryajov>
if the api is the same, coual actually work out of the box
<dryajov>
:)
<dryajov>
same as node’s api
shizy has quit [Ping timeout: 264 seconds]
<xSkyripper>
I'm trying to fully grasp this :D
<dsal>
ipfs fuse is not being awesome right now. heh
* dsal
comes back later
<whyrusleeping>
dsal: yeah... we havent worked on the fuse code in a while. The last time i tried working on it fuse froze my kernel
<whyrusleeping>
so i've been boycotting it
jkilpatr has quit [Ping timeout: 260 seconds]
<dsal>
I trusted you!
<dsal>
I did a bunch of work on camlistore fuse a few years ago
<dsal>
It basically worked.
<dsal>
I've been heavily using another fuse filesystem lately I keep about a TB and a half in.
<whyrusleeping>
huh, you don't run into any weird fuse issues?
<dsal>
Not lately.
<whyrusleeping>
i just kinda assumed fuse itself was flakey
<dsal>
Except this one. :)
<dsal>
Yeah, it's been good to me lately.
<whyrusleeping>
well maybe i'll have to find a renewed interest in making fuse work better
<whyrusleeping>
are you able to run fuse tests in CI?
<whyrusleeping>
this is the 'front page':https://ipfs.io/ipns/tr.wiki.ipfs.ovh/wiki/Anasayfa.html
<dryajov>
xSkyripper: so there are a few things, if the both the node and cordova socket interfaces match, we should be able to use js-libp2p-tcp without any modifications to it...
<dryajov>
the only thing we’d need to do is change how we detect environments and tweak the js-ipfs-libp2p-node/browser to set up correctly under cordova
<emunand[m]>
whyrusleeping: i've also noticed that whenevr i try to open the wiki locally, i get an error saying "no link named 'wiki' under <the hash>"
<whyrusleeping>
emunand[m]: interesting...
<whyrusleeping>
oooh
<whyrusleeping>
thats because its using sharding
<whyrusleeping>
you'll need to run 0.4.9-rc1
<emunand[m]>
i enabled sharding too
<emunand[m]>
in 0.4.8
<dryajov>
worst case we might even have a js-ipfs-libp2p-cordova or something, which is a pretty tin layer that setups libp2p and swarm...
<xSkyripper>
tin layer ?
<emunand[m]>
oh, i require 0.4.9?
<whyrusleeping>
Yeah, viewing sharding through the gateway didnt work quite right in 0.4.8
<xSkyripper>
dryajov I don't think it's exactly the same
<whyrusleeping>
(this is why its an 'experimental' feature ;) )
<dryajov>
this steps up libp2p to run in the browser or node… if they are the same, we might use the node version in cordova…. if they are different, it would be a mater of putting together a new js-libp2p-ipfs-cordova…
<whyrusleeping>
emunand[m]: gx is a content addressed package manager
<dsal>
Yeah, alright. This is a relatively obvious way to do things, but going to be hard with a ton of things. My first attempt to play with ipfs was my 90k images I use for training data for this machine learning thing I was doing.
<whyrusleeping>
agreed
<whyrusleeping>
we're trying to figure out how to make it work more scalably
maxlath1 has joined #ipfs
maxlath has quit [Ping timeout: 260 seconds]
maxlath1 is now known as maxlath
<emunand[m]>
is there an english version of the ipfs wikipedia yet?
jonnycrunch has joined #ipfs
<whyrusleeping>
emunand[m]: working on it
<whyrusleeping>
its *much* larger
SalanderLives has joined #ipfs
<emunand[m]>
yeah, i remember you guys saying it was over 1TB or something
<dsal>
You guys loading wikipedia into ipfs?
<dsal>
I've probably got some good tooling for that.
<emunand[m]>
yeah, so far they have the turkish wikipedia online
Caterpillar has quit [Quit: You were not made to live as brutes, but to follow virtue and knowledge.]
The_8472 has quit [Ping timeout: 260 seconds]
shizy has quit [Ping timeout: 240 seconds]
xSkyripper has quit [Read error: Connection reset by peer]
asyncsec has quit [Quit: asyncsec]
maxlath has quit [Quit: maxlath]
rovdyl has quit [Ping timeout: 258 seconds]
MDude has joined #ipfs
cxl000 has quit [Quit: Leaving]
The_8472 has joined #ipfs
<e337>
If ipfs is content-addressed storage, then is ipns analogous to location-addressed storage?
espadrine_ has quit [Ping timeout: 240 seconds]
asyncsec has joined #ipfs
<e337>
If at least one ipfs node pins a given block, then is it always possible to cat the block from any other node?
<achin>
assuing the two nodes can talk to each other, yes
<e337>
Do they need to talk directly to each other?
<achin>
today, yes
<achin>
in the future, data will be able to be relayed between nodes
Jesin has joined #ipfs
<e337>
So if only one node provides a block, then the requestor will simply get it directly from the provider?
<e337>
Once the second node gets the block in its cache, will it then immediately be able to provide the block to other nodes, or will the second node need to pin the block?
<achin>
you don't need to pin a block to make it available
<achin>
pinning it just means that your node won't garbage collect it
<achin>
so as soon as the second node gets the block, the rest of the swarm would be able to get it from both the first and second nodes
<e337>
Ah
infinity0_ has joined #ipfs
infinity0_ has quit [Changing host]
infinity0 has joined #ipfs
infinity0 is now known as Guest61055
Jesin has quit [Quit: Leaving]
infinity0 has quit [Remote host closed the connection]
<e337>
So if there is a very popular block, say for some viral video, then once many nodes have the block in their cache, would that mean it's generally very fast for a new node to get the block?