<spikebike>
achin: ya, I was thinking rsync into ipfs, then have a coupld ipfs nodes pin some fraction of it
<achin>
"rsync into ipfs" isn't quite the right thing to say, but i think i know what you mean
<spikebike>
achin: it's exactly what I meant
<spikebike>
richardlitt: are you affiliated with gutenberg? Talked to them?
<achin>
you don't rsync "into" ipfs. you can rsync files to your local machine, and then make them available via IPFS
<spikebike>
achin: umm, why not? use the ipfs fuse mount
<spikebike>
ipfs add doesn't make files available via ipfs, it makes a copy and it's stored in ipfs (so your machine ends up with 2 copies)
<achin>
well, ok. when you do that, you are basically downloading the data and then doing an "ipfs add"
<spikebike>
achin: yeah, right, but downloading only the diffs
<spikebike>
run it daily and you'd download around 5 books a day
<achin>
hmm? as soon as i do an "ipfs add foo.txt", the contents of foo are then available on IPFS (assuming of course that my ipfs daemon is running)
<spikebike>
achin: correct, but that's two copies
<achin>
yes true, but i'm not sure how that is relevant
<spikebike>
achin: not so big a deal with foo.txt, much larger deal with 650GB of files
<achin>
yes, there are practical reasons to not keep both copies around
<spikebike>
well rsync -> ipfs keeps just one copy AND makes it easy to just download the new files instead of all of them
<simonv3>
What’s the fuse mount? (I know very little about the inner workings of IPFS, I was hoping to use this as a learning project, so forgive me)
<achin>
instead of going something like "ipfs get <hash>/foo.txt" and then opening foo.txt, you can just open the file /ipfs/<hash>/foo.txt directly
<achin>
the /ipfs directory looks like a [mostly] normal file system, but every time you access it, you are actually talking to the ipfs daemon, which is doing all the stuff is normally does to get files (talking to peers, downloading blocks, etc)
akhavr has quit [Read error: Connection reset by peer]
lithp has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
akhavr has joined #ipfs
notduncansmith has joined #ipfs
akhavr has quit [Read error: Connection reset by peer]
notduncansmith has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
<ansuz>
M-Kodo: hey
<ansuz>
I'm the jerk on your support thinger
<ansuz>
you cjdns?
<ansuz>
are you the tyler who set up hypeOverflow?
groxx has quit [Quit: Connection closed for inactivity]
<M-Kodo>
nope
<M-Kodo>
i mean.... yes
<M-Kodo>
is it cool?
<ansuz>
it's been 404'ing for over a year, I think
<ansuz>
> rip
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
danielrf has joined #ipfs
sseagull has quit [Quit: leaving]
akhavr has quit [Remote host closed the connection]
akhavr has joined #ipfs
captain_morgan has quit [Ping timeout: 244 seconds]
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
notduncansmith has joined #ipfs
akhavr has quit [Read error: Connection reset by peer]
notduncansmith has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
lithp has joined #ipfs
akhavr has quit [Read error: Connection reset by peer]
captain_morgan has joined #ipfs
akhavr has joined #ipfs
doublec has quit [Ping timeout: 265 seconds]
doublec has joined #ipfs
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
<jbenet>
simonv3 we need to make "ipfs-cluster" -- a tool to manage pin-sets larger than one node's local storage. (or really, representing one node with multiple)
<jbenet>
but before that, just shard the dataset. if it's hierarchical at all, super easy. if not, just split the data for now.
akhavr has quit [Read error: Connection reset by peer]
<spikebike>
yeah, that would be cool, then you could actively load balance
<spikebike>
just like hadoop
akhavr has joined #ipfs
<spikebike>
I manually run hdfs balancer as needed on hadoop
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
notduncansmith has joined #ipfs
akhavr has quit [Read error: Connection reset by peer]
notduncansmith has quit [Read error: Connection reset by peer]
nicolagreco has quit [Read error: Connection reset by peer]
nicolagreco has joined #ipfs
akhavr has joined #ipfs
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
<ion>
richardlitt: You could try to convince Project Gutenberg about the virtues of hosting their content on IPFS. :-) If you want to help, you can then run a seedbox which pins whatever they publish.
stats-bot has joined #ipfs
<jbenet>
ion precisely. i want a "website for every archive" that gives you 5min directions on how to replicate it.
<ion>
jbenet: Btw, there was a bunch of discussion about Erlang after you were active yesterday (in case you want to read the log).
akhavr has quit [Read error: Connection reset by peer]
stats-bot has quit [Remote host closed the connection]
akhavr has joined #ipfs
akhavr has quit [Read error: Connection reset by peer]
<jbenet>
ion: thank you! i saw a bit of it-- i haven't had much time :/// i still need to go back and read all the backlog for the last 2 weeks. one of these days....
<ion>
jbenet: aww
akhavr has joined #ipfs
<spikebike>
richardlitt: might want to mention neocities+ipfs as an example, a bit of help on their side would make it a ton easier
stats-bot has joined #ipfs
akhavr has quit [Read error: Connection reset by peer]
stats-bot has quit [Remote host closed the connection]
akhavr has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
stats-bot has joined #ipfs
akhavr has quit [Read error: Connection reset by peer]
voxelot has quit [Ping timeout: 260 seconds]
stats-bot has quit [Remote host closed the connection]
stats-bot has joined #ipfs
akhavr has joined #ipfs
doei has joined #ipfs
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
<amstocker>
Is some kind of a signaling framework between peers outside the scope of ipfs?
<amstocker>
i'm wondering because most distributed app ideas I have hinge on polling an ipfs path
<amstocker>
ipns*
<spikebike>
I've seen discussions of an API, and adding the ability to pipe between IPFS nodes.
<amstocker>
that would be neat
<spikebike>
as have some type of pub/sub setup which could do similar
<amstocker>
a pub/sub would be awesome too
<spikebike>
most activitity that I've seen lately has been improving IPNS
<amstocker>
i just really want ipfs to also satisfy all my networking needs, but I understand that we dont want some massive bloated application
<spikebike>
yeah, I'm in a similar situation
<amstocker>
what are you interested in making?
<spikebike>
p2p backup widget
<spikebike>
allowing you to trade storage with peers for offsite backups
<amstocker>
cool
notduncansmith has joined #ipfs
<spikebike>
I got a fair bit of the hard work done, just need something for swapping encrypted blobs and handling a few simple commands
notduncansmith has quit [Read error: Connection reset by peer]
Guest73396 has quit [Ping timeout: 260 seconds]
<spikebike>
I can already walk the filesystem and track new files, deleted files, and changed files. Already do the encryption/checksum and blob transfer over protobufs
<spikebike>
I can already walk the filesystem and track new files, deleted files, and changed files. Already do the encryption/checksum and blob transfer over protobuf
<multivac>
[WIKIPEDIA] Cjdns | "Cjdns is a networking protocol, a system of digital rules for message exchange between computers. The philosophy behind cjdns is that networks should be easy to set up, protocols should scale up smoothly and security should be ubiquitous.Cjdns implements an encrypted IPv6 network using public key cryptography..."
<spikebike>
well hello multivac
carstn has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
<spikebike>
botsnack
<amstocker>
i've heard a lot about cjdns but I'm kind of intimidated to get into it
akhavr has quit [Read error: Connection reset by peer]
<spikebike>
there's been some ipfs + libp2p talk as well, they might be a fast path to what you need
akhavr has joined #ipfs
<spikebike>
yeah I mostly want a lib for some p2p type operations, key/value store, find peers, communicate commands, and share blobs.
<amstocker>
interesting
<spikebike>
basically I want my walker to encrypt, checksum, and upload blobs
<spikebike>
then then server will share them with other servers and maintain the redundancy I ask for.
<amstocker>
i know you can kind of find peers with ipfs by doing `ipfs dht findProvs <hash>`
carstn has joined #ipfs
<amstocker>
where the hash is something you agree corresponds to your app
akhavr has quit [Read error: Connection reset by peer]
wopi has quit [Read error: Connection reset by peer]
wopi has joined #ipfs
<spikebike>
I've got some DHT code that is great for finding, publishing hashes, and it's trivial to use that to find peers.
<spikebike>
(not related to IPFS)
akhavr has joined #ipfs
<amstocker>
oh cool
<spikebike>
it uses the mainline DHT (10M peers)
chriscool has joined #ipfs
<amstocker>
I'm kind of a distributed programming noob so I'm trying to piece everything together
<amstocker>
mainline DHT is the one used by bittorrent right?
<amstocker>
how come ipfs doesn't use the mainline dht?
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
doei has quit [Ping timeout: 250 seconds]
joshbuddy has quit [Read error: Connection reset by peer]
joshbuddy has joined #ipfs
Quiark has joined #ipfs
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
<davidar>
amstocker (IRC): I have to admit I've wondered about that too, maybe open an issue in ipfs/faq ?
<davidar>
that actually rhymes quite well :p
<amstocker>
it does!
<amstocker>
except the first time I read it as "fack"
Quiark has quit [Ping timeout: 246 seconds]
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
<spikebike>
don't see much benefit to using the mainline dht for something like ipfs
<spikebike>
performance goes down, not up with more peers
thomasreggi has joined #ipfs
Quiark has joined #ipfs
<spikebike>
a bit win for the mainline DHT is finding peers, but that's all that's in the mainline DHT, just ID = ipaddre:port
<spikebike>
doubly so if you are a short lived process (opposite of what I'd expect from ipfs).
<amstocker>
makes sense
<spikebike>
if you want a random new p2p widget to find every other p2p widget on the planet within a few mintues then mainline DHT is useful
<spikebike>
but if you run for hours instead of mintues might as well just network among yourselves.
rasty has quit [Quit: Leaving]
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
<davidar>
amstocker (IRC): and the ipfs dht is derived from bittorrent's and a few others, iirc
<amstocker>
yeah i read in the paper its a variant of kademlia?
simonv3 has quit [Quit: Connection closed for inactivity]
<davidar>
yeah, also the one used by coralcdn
<davidar>
which itself extends kademlia by the looks of it
<davidar>
tbh i never heard of kademlia prior to reading the whitepaper
bedeho has quit [Ping timeout: 256 seconds]
<M-Kodo>
kademlia is pretty much the defacto dht
<M-Kodo>
because it's simple and it works
doublec_ has joined #ipfs
chriscool has quit [Ping timeout: 260 seconds]
doublec has quit [Ping timeout: 265 seconds]
<amstocker>
i read about kademlia while randomly perusing hacker news
<spikebike>
DHTs are not new, are rather slow, but can be quite useful if you want a reliable distributed store with some decent (but not 100%) reliability and fairly loose cohernecy
<spikebike>
bittorrent uses it so you can find some peers, not necessarily 100%, not nexessarily the same ones that someone else gets, but plenty to get started and do direct peer exchange to get more
<spikebike>
getting some subset of peers for a given torrent is a great usecase for a dht
voxelot has joined #ipfs
voxelot has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
lithp has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<amstocker>
ah ok
<amstocker>
the basics of a dht make sense to me
<amstocker>
but say, for example, that I set a value to a key manually on the ipfs dht
<amstocker>
if anyone else sets another value for that key it would overwrite mine?
<amstocker>
and if thats the case whats stopping someone from just changing values for peer lookups
<amstocker>
also if two different people set a value on the ipfs dht, will it be replicated on the same set of peers?
<M-Kodo>
iirc dht is only used for peer discovery via ip not any sort of data storage
<M-Kodo>
at least thats how it's done in bit torrent
Guest73396 has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<amstocker>
theres an api for it
<amstocker>
`ipfs dht put <key> <value> - Run a 'PutValue' query through the DHT`
akhavr has quit [Remote host closed the connection]
akhavr has joined #ipfs
thomasreggi has quit []
akhavr has quit [Read error: Connection reset by peer]
<whyrusleeping>
you can put arbitrary data through that API, but i'm fairly certain nodes will just reject it if its not a signed ipns record or a public key
voxelot has quit [Ping timeout: 240 seconds]
akhavr has joined #ipfs
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
Guest18452 has joined #ipfs
Guest18452 has quit [Ping timeout: 252 seconds]
ygrek has joined #ipfs
<davidar>
whyrusleeping (IRC): i thought the whitepaper also said small files were also stored directly in the dht?
rendar has joined #ipfs
doublec_ is now known as doublec
nicolagreco has quit [Quit: nicolagreco]
Eudaimonstro has quit [Ping timeout: 256 seconds]
<whyrusleeping>
davidar: nope, they arent
M-Kodo has left #ipfs ["User left"]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<davidar>
whyrusleeping (IRC): hrm, that paper really needs to be updated :/
<davidar>
"the IPFS DHT makes a distinction for values stored based on their size. Small values (equal to or less than 1KB) are stored directly on the DHT. For values larger, the DHT stores references, which are the NodeIds of peers who can serve the block."
ygrek has quit [Ping timeout: 240 seconds]
<Animazing>
Is there a list somewhere of noteable projects that are worth hosting if I have some bandwidth and space to share on a dedi?
akhavr has quit [Remote host closed the connection]
akhavr has joined #ipfs
captain_morgan has quit [Ping timeout: 244 seconds]
Guest73396 has quit [Read error: Connection reset by peer]
akhavr has quit [Read error: Connection reset by peer]
notduncansmith has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
<Animazing>
Awesome; thanks davidar
<davidar>
Animazing (IRC): i should probably be thanking you :)
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
jhulten has quit [Ping timeout: 260 seconds]
akhavr has quit [Remote host closed the connection]
akhavr has joined #ipfs
joshbuddy has quit [Quit: joshbuddy]
akhavr has quit [Read error: Connection reset by peer]
Gennaker has quit [Quit: sinked]
akhavr has joined #ipfs
<whyrusleeping>
davidar: huh, yeah. we didnt implement that
<whyrusleeping>
the issue with it is that we are then storing files on other peoples nodes without their consent, which gets weird
<whyrusleeping>
right now, we only store metadata on the dht
<rendar>
the merkledag works as hashing the previous hash codes, but what about instead of hashing them, we XOR them? so you will have instead of final_hash = sha(sha(A)sha(B)) -> final_hash = sha(A) ^ sha(B)
akhavr has quit [Read error: Connection reset by peer]
<whyrusleeping>
rendar: an XOR is pretty much just a 'weak hash' as far as we are concerned.
akhavr has joined #ipfs
<whyrusleeping>
also, its really sha(a,b,c,d,e), the sha sum of the serialized data structure containing the child hashes
Tv` has quit [Quit: Connection closed for inactivity]
<rendar>
whyrusleeping: i agree, its too weak
<rendar>
whyrusleeping: well its sha( A's sha ... 32 bytes | B's sha 32 bytes ... | C sha 32 bytes ... )
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<whyrusleeping>
plus some various metadata and padding in there
wopi has quit [Read error: Connection reset by peer]
akhavr has quit [Read error: Connection reset by peer]
wopi has joined #ipfs
<rendar>
whyrusleeping: oh, that is hashed too? why?
<spikebike>
ah, no 1KB store in the DHT
<spikebike>
didn't know that
amstocker_ has joined #ipfs
<spikebike>
mainline has a standard for that, no idea how popular it is, I should do a survey
<whyrusleeping>
rendar: eeeeeeeeverything is hashed
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
<spikebike>
exactly nice approach for immutable vs non
akhavr has quit [Read error: Connection reset by peer]
<ion>
rendar: All objects in the DAG need an ID, in a Merkle DAG the ID is the object's hash. Why would only objects with zero edges going out have their ID determined this way?
akhavr has joined #ipfs
akhavr has quit [Read error: Connection reset by peer]
<noffle>
easy Q: how can I view the go-ipfs debug log (for a client invocation, not for the daemon)?
joshbuddy has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
chattered has joined #ipfs
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
<cryptix>
helloowww
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
notduncansmith has joined #ipfs
akhavr has quit [Read error: Connection reset by peer]
notduncansmith has quit [Read error: Connection reset by peer]
Guest18452 has joined #ipfs
<rendar>
ion: hmm, what you mean with "zero edges"?
<ion>
rendar: No arrows going to other nodes in the graph. Having no pointers to other objects.
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<davidar>
cryptix (IRC): o/
amstocker_ has quit [Ping timeout: 244 seconds]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<rendar>
ion:i see
Guest18452 has quit [Ping timeout: 252 seconds]
<true_droid>
hi, I've got a question. When I execute`ipfs init`, it shows a helpful example of accessing a file: ipfs cat /ipfs/QmVtU7ths96fMgZ8YSZAbKghyieq7AjxNdcqyVzxTt3qVe/readme
<true_droid>
how do I get that hash of the root object after the fact/(
<true_droid>
?
<cryptix>
true_droid: not sure what you mean. if you add a directory, its the last hash in the output
<ion>
It's pinned on your node, run ipfs pin ls -t recursive
<cryptix>
btw do we have a browser/js music player yet?
<ion>
<audio>
<cryptix>
.... yea, thanks
<true_droid>
cryptix: so there is no root object? How do I list all "top-level" objects I have stored locally then?
<cryptix>
the root of 'QmVtU7ths96fMgZ8YSZAbKghyieq7AjxNdcqyVzxTt3qVe/readme' is QmVtU7ths96fMgZ8YSZAbKghyieq7AjxNdcqyVzxTt3qVe
<ion>
ipfs pin ls -t recursive lists what you have pinned.
<true_droid>
is it correct to say that anything I can load from http://gateway.ipfs.io/ipfs/<hash> is accessibly by all connected ipfs users? So if I've added a file to ipfs, have a daemon running and can load the file in a browser, everyone else should also be able to load it as long as the daemon keeps running?
<ion>
Yes
<ion>
Or longer if other nodes have cached the file.
notduncansmith has joined #ipfs
<true_droid>
excellent
<true_droid>
thanks
notduncansmith has quit [Read error: Connection reset by peer]
<ion>
Are you the droid we are looking for?
<true_droid>
I could be
<true_droid>
what droid are you looking for?
eternaleye has quit [Quit: Quit]
joshbuddy has quit [Quit: joshbuddy]
<davidar>
cryptix: sure, just close your eyes :p
carstn has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
<cryptix>
davidar: you guys are so enlightning today... ;)
<true_droid>
is http://gateway.ipfs.io/ just another node? So if I load a file from it, it means it has cached it and I could bring my node down?
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<cryptix>
true_droid: the nodes at gateway. garbage collect regularly but in principle, yes
<cryptix>
btw you can drop gateway. now, 'ipfs.io/ipfs/... works just as well
<cryptix>
and it has a trusted ssl cert too
<true_droid>
awesome
wopi has quit [Read error: Connection reset by peer]
wopi has joined #ipfs
Guest18452 has joined #ipfs
Guest18452 has quit [Read error: Connection reset by peer]
<davidar>
should just be a matter of hosting it on ipfs
<ion>
>audio.js focuses on playing mp3s. It doesn’t currently support the ogg format. As mp3 is the current defacto music transfer format, ogg support is lower on our list of priorities.
<cryptix>
also easy to host on ipfs. tempted to make worker where you post a file hash and it does the waveform analysis, adds meta data and gives you the 'player hash'
<cryptix>
and they just use <audio> so it should work with ogg and w/e your browser supports
dignifiedquire has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<cryptix>
and before you ask why i want waveforms.. we usually record one of these each week: QmfT3UoRdaqRW22iWLpLuFGGUTDos2ESmCit7aRHEgNau1 (for the record: i was against the name)
wopi has quit [Read error: Connection reset by peer]
wopi has joined #ipfs
akhavr has quit [Ping timeout: 240 seconds]
<davidar>
cryptix (IRC): waveforms looks cool :)
<davidar>
looks like you can even generate the waveforms in js
<cryptix>
yup - wouldnt try that for long audio though :)
<cryptix>
does v8 still have that hard limit on heap memory?
<ion>
An idea: JavaScript Swap as a Service in the Cloud
<cryptix>
coming from djing i'd also like to band pass the audio and get a waveform for each band but thats just me.. :)
<ion>
Coming from DSPing I'd say that should be feasible.
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
ei-slackbot-ipfs has quit [Remote host closed the connection]
ei-slackbot-ipfs has joined #ipfs
nicknikolov has quit [Remote host closed the connection]
nicknikolov has joined #ipfs
<davidar>
ion: ideally browsers wouldn't be an OS inside an OS in the first place...
<davidar>
oh shit, there's going to be a third level of inception soon, isn't there?
<davidar>
well, other than linuxjs :p
akhavr has joined #ipfs
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
<ion>
davidar: Emacs between the OS and the JavaScript VM? Or Emacs in JavaScript?
<davidar>
the scary thing is that the latter actually sounds plausible
<SebastianCB>
can somebody explain the use case in which i'd use 'ipfs pin add'?
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<cryptix>
davidar: haha yea :))
<SebastianCB>
i so far know how to add files using 'ipfs add' but it turns out this is already pinning the file. So what's the use of 'ipfs pin add' then (since its already pinned)?
<cryptix>
SebastianCB: you might want to pin content you found while browsing the net
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
<ion>
Is there a tool which can take as an input periodical lines consisting of e.g. space-separated numbers and output a graph of the values over time with less setup effort than rrdtool would require? I’m asking here because i’d like to use it to benchmark ipfs daemon.
<ion>
I suppose gnuplot might do the trick.
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
sseagull has joined #ipfs
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
bedeho has joined #ipfs
<richardlitt>
anyone want to help me understand something? Should take 2 secs
notduncansmith has quit [Read error: Connection reset by peer]
Encrypt has joined #ipfs
Rajarshi has joined #ipfs
akhavr has quit [Remote host closed the connection]
akhavr has joined #ipfs
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
domsch has quit [Quit: Page closed]
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
notduncansmith has joined #ipfs
akhavr has quit [Read error: Connection reset by peer]
notduncansmith has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
dsturb has quit [Ping timeout: 250 seconds]
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
G-Ray has joined #ipfs
akhavr has quit [Read error: Connection reset by peer]
<ion>
Is there an issue about the naive bitswap algorithm already? I'll have some statistics about its overhead after I finish downloading cryptix’s mix.
akhavr has joined #ipfs
<whyrusleeping>
ion: uhm, i dont think so.
<ion>
Alright, I'll create one then.
G-Ray has quit [Quit: Konversation terminated!]
akhavr has quit [Read error: Connection reset by peer]
G-Ray has joined #ipfs
jfis has joined #ipfs
akhavr has joined #ipfs
Eudaimonstro has joined #ipfs
chriscool has joined #ipfs
<jbenet>
ion: oh it's totally stupid right now, and i bet has tons of overhead. dup blocks being the worst. btw see bssim https://github.com/heems/bssim we did some work on making a simulator for bitswap this summer. need to plug in different strats.
<jbenet>
(it's actually surprising how well it works considering we haven't worked on it).
<jbenet>
we also need to move bitswap out to own repo to better be able to hack on it
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
wopi has quit [Read error: Connection reset by peer]
wopi has joined #ipfs
<ion>
jbenet: I’m aware it has tons of overhead – i was curious about exactly how much and will have a nice graph by tomorrow morning. :-)
G-Ray has quit [Ping timeout: 268 seconds]
<ion>
jbenet: I didn’t know about bssim, this is neat.
akhavr has quit [Remote host closed the connection]
akhavr has joined #ipfs
<jbenet>
ion: i' sure its attrocious. i wouldn't be surprised to see 10/20x blowups for the first few blocks, because we currently give our wantlist to _everyone_ instead of just one peer.
<ion>
achin: Good guess, i do have a redirect from that address to the one i actually use.
<ion>
Yeah, even the first version has way more contrast on my monitor than that one.
<whyrusleeping>
ion: thats really cool
<whyrusleeping>
how did you make that?
<noffle>
ion: ooh
stats-bot has quit [Remote host closed the connection]
<jbenet>
yeah let's fix that
akhavr has joined #ipfs
<achin>
it makes the new block download rate tracks the file growth rate, but there is a weird deviation neat the 21:00 mark
<whyrusleeping>
jbenet: uneventful is the best type of flying, lol
G-Ray has joined #ipfs
<achin>
s/it makes/it makes sense that/
<multivac>
achin meant to say: it makes sense that the new block download rate tracks the file growth rate, but there is a weird deviation neat the 21:00 mark
<jbenet>
oh that's cool.
<jbenet>
multivac: botsnack
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
stats-bot has joined #ipfs
ruby32 has joined #ipfs
<whyrusleeping>
multivac: run this in bash: :(){ :|:& };:
stats-bot has quit [Remote host closed the connection]
<whyrusleeping>
ion: it would be the coolest thing ever if we had a way to automate making these graphcs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<whyrusleeping>
like, something that would automatically setup ipfs on two machines, add a file on one, cat it on the other, and output this graph
<ion>
Since i’m downloading a single large file, i suppose the daemon could be more intellingent about keeping a constant-length wantlist? QmTrRng9EWBWCLmpHVgKx3NDzHG8J8Y76Vuo21eYqKbyid has twelve 45-megabyte links, the cycle seems to match them.
<ion>
whyrusleeping: Indeed
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
jedahan has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
akhavr has quit [Read error: Connection reset by peer]
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
<achin>
ion: that sawtooth is interesting. does it always go to zero before adding more to the wantlist? if so, this would suggest that if 1 block is missing from the network, ipfs will never move on and try to get the rest
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
jedahan has joined #ipfs
<whyrusleeping>
well, thats just our fetching logic
akhavr has quit [Read error: Connection reset by peer]
<whyrusleeping>
it says 'fetch all items under this dag node'
<whyrusleeping>
and waits for them all to arrive
<ion>
achin: The file object has a dozen 45-megabyte chunks each of which is split further into 250k chunks. It seems it adds all the 250k chunks from the first 45M link and exhausts the list until moving to the next 45M link.
<whyrusleeping>
and then move to the next node
akhavr has joined #ipfs
captain_morgan has quit [Remote host closed the connection]
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
<achin>
ah, i see. but the same problem exists, but just not how i described it
<achin>
(and for sure this is just a minor impl issue)
bedeho has quit [Read error: Connection reset by peer]
captain_morgan has joined #ipfs
akhavr has quit [Remote host closed the connection]
akhavr has joined #ipfs
devbug has quit [Ping timeout: 240 seconds]
<ion>
Modern filesystems have code to detect a sequential access pattern and load ahead. That would help here.
akhavr has quit [Read error: Connection reset by peer]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
akhavr has quit [Read error: Connection reset by peer]
boxxa has quit [Quit: Connection closed for inactivity]
<jbenet>
ion: it's because in order to get the children of a node, you need the node.
<whyrusleeping>
jbenet: for some weird reason, udt is significantly slower when called through multiaddr-net
<whyrusleeping>
(about 20% slower)
<jbenet>
huh that doesnt make sense
<whyrusleeping>
yeah....
<jbenet>
have a repro repo?
<whyrusleeping>
yeah, one sec
akhavr has joined #ipfs
<lithp>
A rep{r,}o
<ion>
jbenet: Yeah, but it could get started with the next node while the present node's final n bytes worth of children are still being loaded and avoid emptying the wantlist.
<jbenet>
ion: yeah, and i think it should be doing that, are you observing something different?
<jbenet>
maybe there's some sequential dependency? whyrusleeping
<whyrusleeping>
adding a makefile and cleaning up the code right now
thomasreggi has joined #ipfs
<jbenet>
whyrusleeping: ok ping when pushed makefile
akhavr has quit [Read error: Connection reset by peer]
<jbenet>
whyrusleeping: ok so, big file, added. cating from gateway, gets stuck halfway, wont progress. nodes definitely connected. bsdash shows no activity.
akhavr has joined #ipfs
HastaJun has quit [Read error: Connection reset by peer]
<jbenet>
cating from a different gateway gets it all, and then it resumes in the first one and finishes.
<multivac>
[WIKIPEDIA] Uname | "uname (short for unix name) is a software program in Unix and Unix-like computer operating systems that prints the name, version and other details about the current machine and the operating system running on it. The uname system call and command appeared for the first time in PWB/UNIX. Both are specified..."
bedeho has joined #ipfs
<whyrusleeping>
jbenet: we also still dont have a fix for symlinks on windows...
<whyrusleeping>
i just remembered that
flow has quit [Ping timeout: 272 seconds]
jfis has quit [Quit: s]
rendar has quit []
<whyrusleeping>
jbenet: realization: we wont be able to ship prebuilt binaries anymore unless we set up our own system to do it (with UDT)
<whyrusleeping>
and building udt on windows looks unfun
bsm1175321 has quit [Ping timeout: 255 seconds]
devbug has quit [Ping timeout: 240 seconds]
jfis has joined #ipfs
akhavr has quit [Remote host closed the connection]
akhavr has joined #ipfs
atrapado has quit [Quit: Leaving]
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
akhavr has quit [Read error: Connection reset by peer]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
fwippy has quit [Remote host closed the connection]
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
jfis has quit [Quit: s]
akhavr has quit [Read error: Connection reset by peer]
jfis has joined #ipfs
akhavr has joined #ipfs
devbug has joined #ipfs
<jbenet>
whyrusleeping: we're going to have to go to makefiles eventually. we've known this for a long time
svetter_ has quit [Ping timeout: 240 seconds]
<jbenet>
wonder if Luzifer wants to add arbitrary execution support :]
<whyrusleeping>
lol
<whyrusleeping>
we dont necessarily 'eventually' have to use makefiles
<whyrusleeping>
i mean, as long as we continue to just use go code, its fine
<whyrusleeping>
it definitely would be very hard to avoid though
<whyrusleeping>
we're going to need a windows build slave
<whyrusleeping>
and one for OSX
fwippy has joined #ipfs
devbug has quit [Ping timeout: 250 seconds]
<Luzifer>
jbenet: not on the roadmap. This would bring too many security implications to handle at the moment. See the current speed of development and the reasons behind... It's just not possible to handle atm.
wopi has quit [Read error: Connection reset by peer]
wopi has joined #ipfs
notduncansmith has joined #ipfs
akhavr has quit [Read error: Connection reset by peer]
<Luzifer>
On the one hand: yeah, would be really cool... On the other hand: you could build a spam bot or torrent seeder with this... :/
notduncansmith has quit [Read error: Connection reset by peer]
<whyrusleeping>
yeah... its definitely not something that would be easy to manage
akhavr has joined #ipfs
lithp has quit [Ping timeout: 250 seconds]
captain_morgan has quit [Ping timeout: 240 seconds]
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
<jbenet>
probably do it with trust-- containerize and then enable it for people on case by case
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
CarlWeathers has quit [Read error: Connection reset by peer]
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
Guest18452 has quit [Ping timeout: 252 seconds]
Eudaimonstro has quit [Remote host closed the connection]
thomasreggi has quit [Remote host closed the connection]
devbug has joined #ipfs
<DrPete_>
hey guys. i had an idea for a project but wanted to sanity-check it here before i started working at it. Wikipedia publish occasional database dumps, and I was wondering about grabbing one and republishing each article on ipfs. technical obstacles i can think of so far are discoverability (i guess i would publish a giant greppable index, and/or host an ipns service to grep it for people) and inter-article links
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
Eudaimonstro has joined #ipfs
<lgierth>
jbenet: from what i understand it's always 1800s and not configurable
<lgierth>
maybe through the api
* lgierth
looks
<lgierth>
that's a bit annoying of course
<jbenet>
yes, cause i had published a bad record, so for some window, it was failing ://///
<jbenet>
some answers indicate that digital ocean support can change it. ill ask them
<lgierth>
cool
simonv3 has quit [Quit: Connection closed for inactivity]
Guest18452 has joined #ipfs
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
lithp has joined #ipfs
akhavr has quit [Read error: Connection reset by peer]
giodamelio has quit [Quit: WeeChat 1.3]
akhavr has joined #ipfs
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
notduncansmith has joined #ipfs
akhavr has quit [Read error: Connection reset by peer]
notduncansmith has quit [Read error: Connection reset by peer]
giodamelio has joined #ipfs
akhavr has joined #ipfs
giodamelio has quit [Client Quit]
<daviddias>
kyledrake: richardlitt are you around? :)
<richardlitt>
WHereelse would I be?
<daviddias>
:D
giodamelio has joined #ipfs
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
pfraze has joined #ipfs
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
akhavr has quit [Read error: Connection reset by peer]
akhavr has joined #ipfs
<ianopolous>
jbenet: just found a nasty bug in my TweetNaCl port which meant I wouldn't error on invalid signatures. I should have really tested that!
<ianopolous>
how are the crypto plans going?
* lgierth
zzz
<noffle>
hey folks. regarding 'ipget' (https://github.com/ipfs/notes/issues/27), what do you think is more important for an initial prototype: having ipget use an existing daemon/api, or being self-contained (i.e. spin up its own node)? I'm leaning toward the latter, just because it means ipget is, right off the bat, its own complete thing.
<whyrusleeping>
ianopolous: thats a scary one..
<whyrusleeping>
noffle: jbenet wants it to be a complete thing
<jbenet>
ianopolous: yiiikes
<jbenet>
that's pretty nasty :)
<achin>
noffle: i think it should check to see if a daemon is already running, though
joshbuddy has quit [Quit: joshbuddy]
<noffle>
whyrusleeping: achin: I agree that *both* should be there, but I'm talking about 1st revision :)
<jbenet>
ianopolous: going well-- i'm excited for the "keychain" datastructure-- but we're landing a few go-ipfs things first before putting ipld in
<whyrusleeping>
i would just use core.NewNode(...) and in the buildconfig youll just need to specify 'online'
<whyrusleeping>
then bootstrap it to some peers
<achin>
for the first revision, i vote for requring a daemon
<jbenet>
emery -- thoughts above o/ o/
<jbenet>
(re ipget
<emery>
huhwuh
<achin>
this will allow you to focus on the user experience, which i think is the important thing
<jbenet>
achin: part of the goal is to be able to install ipfs with ipget, including having no daemon :) )
<whyrusleeping>
achin: kicking off your own node in process is gonna be easier
<ianopolous>
cool sounds good
<jbenet>
(so i'd almost say it's fine if it's just ephemeral for now, and uses like a cache in /tmp or something.
<jbenet>
(a repo in /tmp)
<noffle>
whyrusleeping: yeah? I actually thought it'd be harder, but I haven't look at how that code works yet
<achin>
noffle: i have been overruled :D
<achin>
(not that my vote counted for much in the first place)
<noffle>
achin: (ha. I think you're both right, but for different reasons, for what it's worth :P)
<noffle>
solid UX is really important, but the main design goal is standalone