<kvda>
it prints really badly on osx/preview. I wonder if anyone else complained about it?
<achin>
whyrusleeping: given my system, the following error makes sense. but feedback nonetheless
<achin>
ERROR: could not move old binary: rename /storage/home/achin/devel/gopath/bin/ipfs /storage/home/achin/.ipfs/old-bin/ipfs-old: invalid cross-device link
<achin>
if the move fails, do you think a copy would be OK?
<lgierth>
davidar: mind if i kill your ipfs daemon on castor? it's eating like 10G ram, i think. or is it busy?
<lgierth>
davidar: you can work with the api on 127.0.0.1:5001 anyhow
<whyrusleeping>
achin: yeap, thats what i'm wrigin right now, lol
<achin>
:D
<ion>
Nice. I'm in bed, but I'll try ipfs-update tomorrow.
<whyrusleeping>
achin: there we go
<whyrusleeping>
update that and try again
<whyrusleeping>
ion: woo :)
<achin>
whyrusleeping: lookin good
Matoro has joined #ipfs
<ion>
Unless it and go-ipfs already work on Android ;-)
<whyrusleeping>
not quite yet
<lgierth>
someone had it working half a year ago
<lgierth>
was streaming 4K videos to tablets
<achin>
ipfs-update revert looks useful, but i'm worried that by only keeping the previous version, it's not saving enough old versions
<achin>
ipfs-update fetch doesn't chmod+x things. maybe it should?
<whyrusleeping>
achin: oh, yeah.
<whyrusleeping>
it probably should
<whyrusleeping>
and re: number of stashed versions
<whyrusleeping>
i'm not sure how best to do the UX on that
<achin>
i'm not either. git-stash comes to mind, but that's not a very good UX
<whyrusleeping>
updated on the fetch setting +x
<whyrusleeping>
yeah, git stash has terrible UX
<whyrusleeping>
and i'm normally one to defend gits UX
<achin>
i started with a build of 0.4.0-dev, and now that version has been deleted (obviously i built it from source, so i can get it back). it might be nice to be able to get back to where you started, if at all possible
<whyrusleeping>
achin: hrm... okay
<whyrusleeping>
i'll try a couple things...
<whyrusleeping>
after food
TheWhisper is now known as Whispery
<achin>
cool
ygrek has joined #ipfs
<achin>
whyrusleeping: two more comments: maybe ipfs-update fetch shouldn't overwrite files? i dont feel very strongly either way here
<achin>
and after an ipfs-update-install, if an old daemon is still running, perhaps print a warning to the effect of "please rememeber to restart your daemon before continuing"
ygrek_ has quit [Ping timeout: 240 seconds]
tsenior`` has joined #ipfs
<whyrusleeping>
both very good points
<whyrusleeping>
thanks!
<zignig>
whyrusleeping: sorry afk, you might want to merge 2-to-3 of fs-repo-migrations onto master.
<zignig>
I followed the instructions and it gets an old version.
<zignig>
...
Matoro has quit [Quit: Matoro]
<whyrusleeping>
zignig: ah, yeah. its not ready to merge yet
<zignig>
perhaps a note on the version warning then.
<whyrusleeping>
what does it do now?
<zignig>
it just barfs on -to=3 with index out of range.
* zignig
is taking mfs for a spin.
cojy has left #ipfs [#ipfs]
<zignig>
if 0.4 going to stay in it's own branch or is there date for master merge ?
<zignig>
*is
<whyrusleeping>
ah, gotcha
<whyrusleeping>
its going to merge into master, hopefully by december
ricmoo has quit [Ping timeout: 246 seconds]
Whispery is now known as TheWhisper
tsenior`` has quit [Ping timeout: 265 seconds]
<davidar>
lgierth (IRC): hrm, odd, I don't think I'm doing anything with it atm
hellertime has quit [Quit: Leaving.]
<lgierth>
davidar: yeah it seems to be something else
<lgierth>
trying to track it down
doublec_ has joined #ipfs
tsenior`` has joined #ipfs
doublec_ has quit [Read error: Connection reset by peer]
doublec has quit [Ping timeout: 272 seconds]
reit has joined #ipfs
doublec has joined #ipfs
jhulten_ has joined #ipfs
jhulten has quit [Ping timeout: 240 seconds]
<lgierth>
anyone got a good idea what might be consuming memory, apart from processes?
<lgierth>
free -m tells me some 14G are used (excluding buffers/caches) and the processes don't nearly add up to that
joeyh has quit [Ping timeout: 260 seconds]
<achin>
i would think adding up the 'VIRT' column from `top` will get you the right number. that's not the case?
doublec_ has joined #ipfs
doublec has quit [Ping timeout: 265 seconds]
ygrek_ has joined #ipfs
<lgierth>
achin: that's maybe half
dPow has quit [Ping timeout: 240 seconds]
<lgierth>
maybe zfs
joeyh has joined #ipfs
ygrek has quit [Ping timeout: 260 seconds]
doublec_ is now known as doublec
__konrad_ has quit [Ping timeout: 260 seconds]
<achin>
ah, maybe
M-prosodyContext has quit [Ping timeout: 240 seconds]
__konrad_ has joined #ipfs
Senji has quit [Quit: Back to the void.]
dPow has joined #ipfs
<kbtombul>
scrollback goto 1
M-prosodyContext has joined #ipfs
kbtombul has quit [Quit: leaving]
<pjz>
whyrusleeping: if you're taking feature requests, a way to get recursive sizes from 'pin ls' might be nice
Matoro has joined #ipfs
* pjz
had to write something higher level
<achin>
what about "ipfs object stat" ?
<pjz>
achin: ahh.. interesting
<pjz>
achin: does it do de-duping?
<pjz>
achin: like if the tree has two links to the same content, does that content get counted twice?
<achin>
in the total size? yes i believe so
<pjz>
ugh. I'm looking for something that's accurate to on-disk size
<achin>
you might have to roll your own tool for that. even if there was no duplicate blocks, ipfs-object-stat is likely going to overestimate the on-disk size
<pjz>
ah, I did it by recursing through ipfs.refs to get the set of all blocks and then doing an ipfs-block-stat on them all and adding it up
<achin>
even ipfs-block-stat is not going to give you a true on-disk size
tsenior`` has quit [Ping timeout: 272 seconds]
<pjz>
afaict it's the closest
<pjz>
it's a little less than actual on-disk b/c it doesn't include the ipfs header, but that's a constant size (14) so I can multiply that by the number of blocks and add to the total
<achin>
you mean more than the on-disk size?
<pjz>
achin: why won't ipfs-block-stat be true on-disk size?
<achin>
it'll return size=14, but it's really a 6-byte file
r04r is now known as zz_r04r
<pjz>
achin: why?
Senji has joined #ipfs
<achin>
it's returning the size of the block data (the thing in ~/.ipfs/blocks). so it has to include the unixfs serialization and the pbnode serialization
<pjz>
ah! but that's the on-disk size I'm looking for
<achin>
ahh. you're not looking for the size of the thing when you go ipfs-get ?
<pjz>
no, I'm looking for the size its going to take up when I do ipfs add
<pjz>
well, actually, the size it'll take up if I do ipfs-pin
<achin>
sorry, i misunderstood. you found the thing you're looking for!
<pjz>
well, the duplicate block problem is still a problem
<pjz>
if you mkdir /tmp/foo ; echo "foo" > /tmp/foo/one ; cp /tmp/foo/one /tmp/foo/two ; ipfs add -r /tmp/foo ... then I think the size you get won't be accurate
<pjz>
b/c there's one block that's pointed to twice by the directory
<pjz>
so I think it gets double-counted by whatever computes size
tsenior`` has joined #ipfs
<pjz>
yup, ipfs object stat double-counts
<pjz>
just verified it
kvda has quit [Remote host closed the connection]
reit has quit [Quit: Leaving]
reit has joined #ipfs
jabberwocky has joined #ipfs
jabberwocky has quit [Ping timeout: 265 seconds]
reit has quit [Quit: Leaving]
reit has joined #ipfs
reit has quit [Client Quit]
reit has joined #ipfs
ygrek_ has quit [Ping timeout: 250 seconds]
reit has quit [Quit: Leaving]
reit has joined #ipfs
reit has quit [Client Quit]
reit has joined #ipfs
guest234234 has joined #ipfs
voxelot has quit [Ping timeout: 246 seconds]
guest234234 has quit [Read error: Connection reset by peer]
reit has quit [Read error: Connection reset by peer]
tsenior`` has quit [Ping timeout: 272 seconds]
gunn has joined #ipfs
domanic has quit [Ping timeout: 250 seconds]
pfraze has quit [Remote host closed the connection]
pfraze has joined #ipfs
simonv3 has quit [Quit: Connection closed for inactivity]
pfraze has quit [Remote host closed the connection]
chriscool has quit [Quit: Leaving.]
chriscool has joined #ipfs
pfraze has joined #ipfs
chriscool has quit [Ping timeout: 244 seconds]
reit has joined #ipfs
tensorjack has quit [Quit: Leaving]
tinybike has joined #ipfs
pfraze has quit [Remote host closed the connection]
pfraze has joined #ipfs
Soft has quit [Read error: Connection reset by peer]
ygrek_ has joined #ipfs
Soft has joined #ipfs
pfraze has quit [Remote host closed the connection]
ricmoo has joined #ipfs
<ricmoo>
Poking the chatroom again... Does anyone know anything about the wire protocol on port 4001 and mayhems how to decode it, handshake and all that good stuff?
merlijn_ has joined #ipfs
merlijn_ has quit [Ping timeout: 264 seconds]
<M-davidar>
ricmoo: whyrusleeping probably does
<M-davidar>
pjz (IRC): whyrusleeping is working on object diffs, which might help?
amstocker has joined #ipfs
<ricmoo>
Heya!
<ricmoo>
Awesome... I've been banging my head looking at this binary data trying to figure out if it is protocol buffers, or whatnot...
<ricmoo>
Is whyrusleeping out there? :)
<davidar>
He's probably sleeping :p
<davidar>
.ask whyrusleeping to talk to ricmoo about the wire protocol
<multivac>
davidar: I'll pass that on when whyrusleeping is around.
<tperson>
davidar: Powers out lol
<ricmoo>
Awesome! Thanks. If I'm not on here, can you pass my e-mail along? me@ricmoo.com
<davidar>
ricmoo (IRC): a lot of stuff is currently protobuf, yes
jabberwo_ has joined #ipfs
<ricmoo>
I'm trying to implement IPFS in Objective-C, or at least reading.
<davidar>
ricmoo (IRC): post an issue to ipfs/faq
<amstocker>
ricmoo there isnt exactly a single wire protocol
<amstocker>
but let me give you some links
<ricmoo>
Maybe you can help? So, right now, I'm trying to make sense of what I get... Looks like maybe \r delimited? First 12 bytes look like some useful version, or something: '\x00\x00\x00\x04\x00\x00\x01|\x00\x00\x01|'
<ricmoo>
Oh! That's a great place to start, thanks. I found it earlier, by gripping for handshake, but my go is less than rusty... I'm been poking around what js I could find...
<ricmoo>
I found the propose pb earlier, but haven't tried parsing against it... But this go code gives me a much better idea of the direction to go in.
merlijn_ has quit [Ping timeout: 244 seconds]
<tperson>
Ya, if you open a connection to 4001 the first thing you will get is that protobuf
mildred has joined #ipfs
<ricmoo>
I think I don't know how to parse the pb yet then. :) I'm experimenting in Python first...
<davidar>
ricmoo (IRC): python has protobuf compilers
mildred1 has joined #ipfs
mildred has quit [Ping timeout: 276 seconds]
<ricmoo>
Yeah, I've got those working... But if I just pass the bytes in as is, it doesn't parse. I've got some substrings parsing now...
<ricmoo>
Is it maybe multiple buffers concatenated?
<ricmoo>
p.ParseFromString(a) => "google.protobuf.message.DecodeError: Tag had invalid wire type." (which makes sense... As I am more-or-less reverse engineering right now... I expect things to explode on me. :))
mildred1 has quit [Client Quit]
<ricmoo>
(still rummaging through the go and js you sent me...)
mildred has joined #ipfs
<ricmoo>
Success!
<ricmoo>
I need to discard the first 12 bytes...
<ricmoo>
Thanks davidar!
<tperson>
Yep... strange
<ricmoo>
I'm sure it will make sense once I look through the source code again... I just made a for loop to try all possible substrings :p
<tperson>
lulz
<ricmoo>
And it is suspicious that it is the length up to the first \r... So, probably meaningful once I dive in more.
<tperson>
aaah I wonder if multistream is sitting in front
<tperson>
What are the first 12 bytes in ascii?
indigane has left #ipfs ["Leaving"]
<amstocker>
ricmoo, tperson, i think its because all messages are length delimited
<tperson>
insecureM is the writer which is a msgio ReadWriter
<M-davidar>
amstocker (IRC): kandinski wanted to talk to you about py-ipfs
<amstocker>
yeah I saw the message a few days ago
<amstocker>
I tried to catch him on IRC but he wasn't here... maybe I can find him on github
<ricmoo>
Awesome! Thanks! :) Now to learn how that works... Because I would think \0\0\0\4\0\0\1|\0\0\1| would actually be a bunch of small 0 length messages...
<amstocker>
ricmoo: I think the lengths are varint encoded
<ricmoo>
You guys are awesome. I can't wait for the python port. Thanks.
<ricmoo>
Does it use the same variant format as Bitcoin?
<amstocker>
yeah that would be great! We haven't really started yet tho
<amstocker>
but basically I think the plan is to try to build a python implementation of libp2p, which is basically the network stack of ipfs
<amstocker>
which they are going to abstract into its own spec
<ricmoo>
Yeah, looking through the go it seems like basically there is p2p and then, bam out of the woodwork IPFS just falls out.
<amstocker>
then we would build the rest of the ipfs protocol on top of that
besenwesen has quit [Quit: ☠]
<amstocker>
paging kandinski
besenwesen has joined #ipfs
besenwesen has quit [Changing host]
besenwesen has joined #ipfs
<ricmoo>
Awesome. I'll post to the issue. Thanks again guys!
<amstocker>
ricmoo np
merlijn_ has joined #ipfs
<ricmoo>
As a side note... Does anyone have any stats on any stats? Like current capacity, current storage used, durability and object loss, propagation delay, etc? It'd be nice to have an Ethereum-style stats board...
Tv` has quit [Quit: Connection closed for inactivity]
rendar has joined #ipfs
tendie has joined #ipfs
<tendie>
Good morning
<chriscool>
Good morning tendie and all!
s_kunk has quit [Ping timeout: 255 seconds]
tendie has quit [Ping timeout: 240 seconds]
ianopolous has quit [Ping timeout: 244 seconds]
amstocker has quit [Ping timeout: 246 seconds]
dignifiedquire has joined #ipfs
elima has joined #ipfs
tinybike has quit [Ping timeout: 240 seconds]
tinybike has joined #ipfs
mildred has quit [Ping timeout: 240 seconds]
amstocker has joined #ipfs
<davidar>
ricmoo: think lgierth has some stats going somewhere, haven't checked it out myself though :/
ziggypup has joined #ipfs
ziggypup has left #ipfs [#ipfs]
<tperson>
So I think I figured out why there are those 12 bytes. I believe secio is double length deliminated
<amstocker>
im in PST @ 1am so i'm about to head to bed
<kandinski>
actually, +11 with Dailight Savings
<kandinski>
well, good night to you. I'm excited about starting.
<tperson>
same ^_^ I just had to figure out why secio was double deliminated
s_kunk has joined #ipfs
mildred has joined #ipfs
<amstocker>
thanks kandinski! I will be around on IRC/github and we will get going with py-ipfs too
<amstocker>
tperson yeah I have no idea, I can't read go that well
<tperson>
I'll bug whyrusleeping tomorrow
dignifiedquire has quit [Read error: No route to host]
dignifiedquire has joined #ipfs
<tperson>
Unless daviddias is getting up soon
amstocker_ has joined #ipfs
amstocker has quit [Ping timeout: 246 seconds]
timgws has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
dignifiedquire has quit [Quit: dignifiedquire]
amstocker_ has quit [Ping timeout: 255 seconds]
<haadcode>
did a massive update to https://github.com/haadcode/anonymous-networks yesterday and released the UI source code along with it. if anyone's interested, go check it out and let me know if you have any issues.
<cryptix>
gmroning
<tperson>
cryptix you know much about the networking and and secio?
<cryptix>
wrt ipfs codebase?
<cryptix>
only superficial to be honst honest
<tperson>
Ya, go-ipfs
<cryptix>
sorry - maybe some specific details but no big picture
<tperson>
The first 4 (00 00 00 04) is a length of 4, followed by 4 bytes (00 00 01 7c) which is a value of 380, then the same sequence repeats (00 00 01 7c), 380 again. Then the protobuf actually starts. The length of the ptotobuf is 380, and if you strip those first 12 bytes you can decode it correctly.
pfraze has quit [Ping timeout: 240 seconds]
<cryptix>
this smells like a redundant wrap of msgio - i thought that would be gone in 0.4.0
<cryptix>
i'm not sure what go-libp2p is based on
<cryptix>
(i.e. pre 0.4 ipfs code base or newer)
<tperson>
Pretty sure that is exactly what it is.
<cryptix>
i remember a bug report about a double wrap, maybe you found it again :)
<cryptix>
tperson daviddias should know as well - not the go code maybe but the protocol specifics
<daviddias>
Morning :)
<daviddias>
We do multistream handshakes through multicodec
<daviddias>
Each time we want to talk a different protocol in the wire
<daviddias>
After the dial, we do a multistream handshake to secio
<davidar>
Multi-turtles all the way down!
<daviddias>
Then another one to spdy
Encrypt has joined #ipfs
<daviddias>
And then in each of the spdy streams, we do another one for the protocol that is going to be spoken there ( DHT, bitswap, ping, and so on )
<daviddias>
And each time a multistream handshake starts, we send /multistream/1.0.0 multicodec as well
<daviddias>
And that is prefixed by a varint
<daviddias>
davidar: ahah :D
<ipfsbot>
[js-ipfs-api] Dignifiedquire created greenkeeper-request-2.66.0 (+1 new commit): http://git.io/v4rGS
<ipfsbot>
js-ipfs-api/greenkeeper-request-2.66.0 0447030 greenkeeperio-bot: chore(package): update request to version 2.66.0...
<ipfsbot>
[js-ipfs-api] Dignifiedquire deleted greenkeeper-request-2.66.0 at 0447030: http://git.io/v4rZA
shyamsk has quit [Ping timeout: 240 seconds]
shyamsk has joined #ipfs
wemeetagain has quit [Ping timeout: 240 seconds]
zz_r04r is now known as r04r
NeoTeo has joined #ipfs
ygrek_ has quit [Ping timeout: 250 seconds]
voxelot has joined #ipfs
dignifiedquire has joined #ipfs
<kandinski>
I've read the code of go-multihash, and I don't understand the point of making a multihash.Sum with a length lower than the default. This behaviour is not in the spec. Are other multihash implementations expected to also support custom hash lengths?
chriscool has quit [Read error: No route to host]
chriscool has joined #ipfs
NightRa has quit [Quit: Connection closed for inactivity]
chriscool has quit [Ping timeout: 240 seconds]
merlijn_ has quit [Ping timeout: 255 seconds]
<M-davidar>
kandinski (IRC): hrm, not sure - can you tell if it's used anywhere in go-ipfs?
voxelot has quit [Ping timeout: 250 seconds]
wemeetagain has joined #ipfs
<daviddias>
kandinski: Not sure what you mean with length lower then default
<daviddias>
There is js-multihashes and js-multihashing
<davidar>
I think that's what kandinski is talking about
<daviddias>
Hmm.. multihash has an header, that header is not part of the hash itself
<daviddias>
It is a way to make it a self describable data format
<daviddias>
And why it needs to be truncated in order to be used by other programs that don't understand multihash
<daviddias>
Am I getting the question right ?
<davidar>
daviddias: I think he's asking if implementations of multihash have to implement truncation functionality like that
<davidar>
and if so, where does it say this in the multihash spec
<daviddias>
Ah
jabberwo_ has joined #ipfs
<dignifiedquire>
daviddias: I give up on cloning the registry :( I restarted the process after restarting my computer and now it’s starting from scratch :’(
<daviddias>
The spec defines how the data is structured and why
<daviddias>
Dignifiedquire it is not starting from scratch
<daviddias>
It will check the modules they you have
<daviddias>
When it gets there
<daviddias>
You will see a lot of "[0] checking"
<dignifiedquire>
hmm but it says from the numbers that it started from scratch 2291/478010]
<daviddias>
It won't download what it already has again
<dignifiedquire>
hmm ok
<dignifiedquire>
lets see
<daviddias>
:)
<daviddias>
I'm already pulling it to Castor also, that makes a third node with the registry
<dignifiedquire>
nice
<daviddias>
And the one we will grab the ipns pointer
<daviddias>
Then, it will be all of the sweetness of installing from IPFS directly with no clone necessary :)
<dignifiedquire>
by the way new follow-registry was released yesterday with my speed patch
<dignifiedquire>
looking forward to it, finally fast npm deploys
<dignifiedquire>
by the way does it pull non existing packages automatically when they are requested?
jabberwo_ has quit [Ping timeout: 244 seconds]
Senji has quit [Ping timeout: 250 seconds]
merlijn_ has joined #ipfs
Encrypt has quit [Quit: Quitte]
sbruce has joined #ipfs
hellertime has joined #ipfs
guest234234 has joined #ipfs
Senji has joined #ipfs
AlienCQ has joined #ipfs
reit has quit [Ping timeout: 246 seconds]
<kandinski>
davidar, daviddias: never mind, I found where jbenet mentions truncating the hashes. Let me show you the comment in an issue for the multihash spec: https://github.com/jbenet/multihash/issues/1
<kandinski>
yes, I think implementations need to implement truncation
<kandinski>
and it should be in the spec
<kandinski>
otherwise, other implementations won't be able to verify/check all possible multihashes emitted by the reference go-multihash implementation
<kandinski>
also, I think the spec needs to define not only the format of the multihash, but the behaviours of compliant implementations
<richardlitt>
I can't get it working for some reason.
merlijn_ has quit [Ping timeout: 260 seconds]
<cryptix>
richardlitt: the UI somewhat loads but i cant get a provider for QmbZJx5eArexPYDp4D44YAkcjME8rFczdZiHn3SyugxcaH
<richardlitt>
cryptix: thanks
<richardlitt>
me neither
merlijn_ has joined #ipfs
hoony has joined #ipfs
hoony has quit [Client Quit]
border0464 has joined #ipfs
chriscool has joined #ipfs
Encrypt has joined #ipfs
reit has joined #ipfs
elima has quit [Ping timeout: 240 seconds]
reit has quit [Read error: Connection reset by peer]
guest234234 has quit [Ping timeout: 260 seconds]
erikj` has quit [Quit: node-irc says goodbye]
M-erikj has joined #ipfs
elima has joined #ipfs
border0464 has quit [Quit: sinked]
e-lima has joined #ipfs
<ion>
whyrusleeping: ipfs-update worked for me. Nice work.
elima has quit [Ping timeout: 244 seconds]
Smilex has joined #ipfs
e-lima has quit [Ping timeout: 255 seconds]
<Smilex>
When I run a IPFS daemon, do I help out transferring files around for everybody?
<Smilex>
I have a server I'm not using, and am thinking about maybe running some IPFS server on there
voxelot has joined #ipfs
<ipfsbot>
[go-ipfs] lgierth force-pushed discovery-cjdns from 245a1a3 to 772931d: http://git.io/v4K3P
<ipfsbot>
go-ipfs/discovery-cjdns 772931d Lars Gierth: cjdns: query admin api for possible peers...
<lgierth>
Smilex: you only provide blocks that you requested yourself
<lgierth>
Smilex: you help with the dht though (i.e. content routing)
<lgierth>
go for it!
Matoro has quit [Ping timeout: 252 seconds]
<Smilex>
lgierth: content routing?
<lgierth>
finding a peer that provides what someone is looking for
<lgierth>
your node will be part of the DHT so it helps a bit
<Smilex>
oh right
<haadcode>
daviddias, dignifiedquire: I'm seeing this as of recently (first noticed it today) http://ipfsbin.xyz/#Qmehwu1HxYuB3rkmLCoKCHqPFUCdAFvZEpbBzDGbVLzXbp. I'm trying to ipfs.cat large video file. ipfs-api v2.7.6. it used to work (2.6.1 I think) and looking at the commit logs after that, the streaming has been reworked. any ideas what's happening?
<daviddias>
would you mind branching js-ipfs-api, add a test with a large file (add and cat)
<daviddias>
and see if it breaks there
<daviddias>
also, which IPFS version are you using?
<daviddias>
0.3.8 was weird
<haadcode>
daviddias: 0.3.9
<daviddias>
adding the test would help us identify the bug + make sure it doesn't break for the future
<haadcode>
daviddias: will look into it more closely tonight after work and add some tests.
<daviddias>
thank you haadcode :)
<daviddias>
and sorry for the troubles
<haadcode>
no worries
voxelot has quit [Ping timeout: 246 seconds]
Xe has quit [Ping timeout: 240 seconds]
<dignifiedquire>
richardlitt: still around?
Xe has joined #ipfs
jabberwocky has joined #ipfs
<dignifiedquire>
whyrusleeping: daviddias are you seeing ifps@0.4 eating a lot of memory as well? been running it since this morning and am now at 8gb memory :(
<jbenet>
you should ask whyrusleeping about getting a good profile while doing this
<jbenet>
it will slow it down, but will give much useful data to fix bugs.
<dignifiedquire>
fuck my terminal is hanging because I tried to copy and paste everything
<jbenet>
:(
nicknikolov2 has joined #ipfs
<jbenet>
i typically redirect stderr on my daemon with tee so i save stack traces
merlijn_ has quit [Ping timeout: 276 seconds]
<dignifiedquire>
I’m not that clever, and tbh it’s the first time I generate a stack trace for ipfs..
tendie has joined #ipfs
roguism has joined #ipfs
<jbenet>
if you two could harvest some stack traces as you do this, it would be hugely useful to go-ipfs. add them to ipfs and post them for whyrusleeping
<dignifiedquire>
jbenet: not sure if you do sth with this, that’s the first 200 lines I got before my terminal decided to quit on me
<pinbot>
now pinning /ipfs/QmUgRq7QfmRbPw5kXqwSs1TRtPDBXMoDNiYwJQgQ1kodNY
border0464 has joined #ipfs
pinbot has quit [Ping timeout: 252 seconds]
Tv` has joined #ipfs
<jbenet>
pinbot: srs?
tendie has joined #ipfs
Matoro has joined #ipfs
<tendie>
Hey not sure if this is the right place to ask this and most likly my router is just garbage but everytime I start ipfs my routers wifi cuts out and then my laptop's mac-address seems to be blocked for a good few minutes
merlijn_ has joined #ipfs
<tperson>
Does it only drop your laptop, or does tour wifi drop for all connected devices?
<jbenet>
tendie that sucks. probably has some anti-p2p crap.
<Stskeeps>
(see the looong rule set by Luzifer on July 10)
mildred has quit [Ping timeout: 255 seconds]
<tendie>
I torrent a lot. I have also used other things like i2p before. Who knows. Thank you for the quick response guys. I will checkout that link.
anticore has joined #ipfs
<jbenet>
tendie: we're shipping utp support soon so that will cut down on addresses dialed
<Stskeeps>
cool :)
<jbenet>
errr tcp ports*
<mmuller_>
cd ..
merlijn_ has quit [Read error: Connection reset by peer]
<mmuller_>
ls
<mmuller_>
(sorry)
<ion>
rm -fr ~
<mmuller_>
ion: unfazed. $HOME = /ipfs
voxelot has joined #ipfs
voxelot has quit [Changing host]
voxelot has joined #ipfs
<tendie>
That is sweet. Hey I'd really like to help the project out but I am still learning and even then I am learning C. Does golang have a ton of differences from C? If so I guess I am screwed then.
pfraze has joined #ipfs
amade has joined #ipfs
<ion>
Some but not a ton. In the grand scheme of things, it is firmly within the C/Lisp/*Script triangle.
<ion>
It has garbage collection, goroutines (green threads) with IPC, a degree of type inference etc.
<haadcode>
daviddias: found this while googling for the error mentioned earlier: https://github.com/nodejs/node/issues/3175, 256MB seems to be the max size for strings in nodejs, so I'm hoping request in not converting the full buffer into a string somewhere... need to investigate more
<daviddias>
that is why we need the streams to be working nicely :)
merlijn_ has joined #ipfs
ricmoo has quit [Ping timeout: 246 seconds]
tyho has joined #ipfs
pinbot has joined #ipfs
<tyho>
Yet another security issue with Lastpass, folowed by yet again suggestions to use an open source one, the problem of distributing the db between PC's laptops and phones has not been solved without using a company like lastpass
<tyho>
is IPFS the solution?
<dignifiedquire>
jbenet: I’m back to 1gb after an hour or so, is that enough for a useful stack trace?
pfraze has quit [Remote host closed the connection]
<dignifiedquire>
jbenet: not working in combination with tee :( ^\[1] 16339 exit 2 IPFS_PATH=/Volumes/npm/.ipfs ./ipfs daemon 2>&1 |
<dignifiedquire>
16340 quit tee ipfs.stdio
<ion>
The SIGQUIT is probably sent to the tee process.
NightRa has joined #ipfs
<dignifiedquire>
ion: any idea how to run it to fix that?
<ion>
You could pkill -QUIT ipfs
pfraze has joined #ipfs
<dignifiedquire>
okay will try
<ion>
Or avoid the tee and tail -F the log in another terminal.
<dignifiedquire>
pkill worked like a breath, thanks ion
simpbrain has quit [Quit: Leaving]
<ion>
tyho: I would love a secure password manager which synchronizes between your devices over IPFS, but nobody has implemented that yet.
simpbrain has joined #ipfs
ricmoo has joined #ipfs
<dignifiedquire>
daviddias: any idea if wreck is usable in the browser, didn’t find anything on their homepage
merlijn_ has quit [Ping timeout: 276 seconds]
<daviddias>
I don't see any specific reason why it couldn't be :) gotta try it to be sure
<dignifiedquire>
daviddias: they use karma, as they should :)
<daviddias>
ahaha nice! :D
<daviddias>
well, let's open a issue, moving to wreck is not urgent
<daviddias>
but I learned that using request is a considerable perf blocker (from folks that do performance reviews for a living :))
<dignifiedquire>
with the latest?
<dignifiedquire>
(cause I know a lot of changed in request in the last year)
<tendie>
what do you guys think of the morph.is project?
<daviddias>
I'll ping to know that
border0464 has quit [Ping timeout: 240 seconds]
border0464 has joined #ipfs
<dignifiedquire>
there is also the option of using two different request modules, on for the browser and one for node if we don’t find anything which fits all requirements
<dignifiedquire>
daviddias: will do a webpack build tonight and see if I can reduce bundle size with that, and after that test out wreck
Senji has joined #ipfs
pfraze has quit [Remote host closed the connection]
<daviddias>
sounds great! :D thank you
<haadcode>
daviddias: quick tests on the streaming issue makes me think something funky is going on with the streams and request. it buffers the full file (befeore even piping it) and somewhere down the line I suppose request converts the buffer to a string and that's causing the error. gonne head home now and look more into it, try to find what exactly is happening.
<haadcode>
daviddias: btw. what was the reason behind moving from http to request in js-ipfs-api?
<dignifiedquire>
haadcode: that would be my faut
<dignifiedquire>
*fault
pfraze has joined #ipfs
<haadcode>
dignifiedquire: the reason for moving to request or the bug? ;)
<haadcode>
no worries, we'll fix it :) it's pretty mucha breaking bug as it makes streaming large videos(/files) impossible atm.
Matoro has quit [Ping timeout: 265 seconds]
<dignifiedquire>
haadcode: for sure, it really sucks
<haadcode>
I'm back to it in about 30mins, gotta get a move on now
<ion>
Just 50 more and you almost have one bit.
<dignifiedquire>
ion: i like small letters :P
<ion>
“/Dignifiedquire/”
<dignifiedquire>
where Oo
<dignifiedquire>
haadcode: I wanted to move to request because of other bugs in http.request which it fixed (and I’m a big fan of not reimplementing all this logic in every module again) actually looked at supergagent and fetch as well but they didn’t provide enough proper streaming support in the browser
<dignifiedquire>
bugs not in http.request, but rather how it was used
<ipfsbot>
[js-ipfs] diasdavid created docs/roadmap-status (+1 new commit): http://git.io/v46Ri
<ipfsbot>
js-ipfs/docs/roadmap-status 61655c7 David Dias: add status bagdges, branch out libp2p roadmap to libp2p repo
<jbenet>
jcd: the right way to do acls will be with capabilities (encrypting objects, etc)
<jbenet>
we need a good capabilities lib.
Matoro has quit [Ping timeout: 265 seconds]
<jcd>
jbenet: I'm still getting my head around all of this, but we are considering using a XACML based ACL solution for our distributed system, and am trying to figure out if IPFS could play in that world directly, or if we have to have a private network and put that logic in the app layer only. :(
<jbenet>
jcd: im not familiar with XACML. maybe if you write up a rough note here in https://github.com/ipfs/notes/ we can look it over, etc.
<jbenet>
ipfs can be used in private networks too
<jcd>
also wondering if there is any info avail about a data sovereignty solution. (This data must not be duplicated except to these nodes (or nodes in this geo-fenced area) )
devbug has joined #ipfs
<jcd>
btw I realize this is still a young project and not expecting all of the things out of the gate, just these are issues we need to confront, and curious if there happens to already be an idea of how they can be addressed with IPFS (if at all)
<jcd>
were "we" is the project I'm working on.
<jcd>
not necessarily IPFS
<jbenet>
hey yeah totally we want to build support for such policies
<jbenet>
one think to do here is simple support for programmable sharing policies, once that is in then people can program checks like, make sure requesting peer carries an authorized capability, or check ip addr is geofenced, etc.
<fazo>
jbenet: that's really cool! Could also be used in my discussion board apps to make private boards possible
<jbenet>
yeah
<ion>
With the major object format change that’s coming with IPLD, will chunking large IPLD objects be covered from the beginning?
<jbenet>
ion: you mean implementing sharding natively?
<jbenet>
that's a good idea.
<jbenet>
and no it wasn't there already.
<jbenet>
its a bit tricky, because sharding is very use case specific.
<jbenet>
but we could implement a general case sharding struct, that others derive from.
<ion>
I was thinking of e.g. small updates to a huge flat directory of files.
<ion>
Just applying Rabin chunking to the IPLD object might be good enough.
<jcd>
jbenet: yep. a plugin based architecture to define if sharing is allowed would be perfect. We want to control duplication of data to only our nodes (potentially further restricted by geo) and restrict retrieval of said data based on an external ACL application.
NeoTeo has quit [Quit: ZZZzzz…]
<jbenet>
jcd: we should do some prep work to figure out how to hook these decisions in. they could be hooked into bitswap.
NeoTeo has joined #ipfs
ekaron has joined #ipfs
<jcd>
Am I right in thinking of IPNS as similar to sym links?
<achin>
that's probably not a bad mental model, at least on the surface
<fazo>
jcd: yes, you can also think of them like pointers
<fazo>
pointers is probably better
s_kunk has quit [Ping timeout: 244 seconds]
master8787 has joined #ipfs
<ion>
jcd: If you think of them as mutable symlinks into immutable files which reside in a different namespace, I guess.
<Smilex>
cryptix: the file is not coming through IPFS
e-lima has joined #ipfs
devbug has quit [Ping timeout: 260 seconds]
ianopolous has joined #ipfs
ianopolous2 has joined #ipfs
devbug has joined #ipfs
ianopolous has quit [Ping timeout: 240 seconds]
e-lima has quit [Ping timeout: 240 seconds]
dignifiedquire_ has quit [Quit: dignifiedquire_]
devbug has quit [Read error: Connection reset by peer]
ygrek_ has joined #ipfs
devbug has joined #ipfs
<whyrusleeping>
cryptix: :(
<multivac>
whyrusleeping: 2015-11-18 - 07:06:35 <davidar> ask whyrusleeping to talk to ricmoo about the wire protocol
<whyrusleeping>
multivac, dont tell me what to do
<whyrusleeping>
i have power again guys
Matoro has joined #ipfs
e-lima has joined #ipfs
bedeho has joined #ipfs
ZioFork has joined #ipfs
<dignifiedquire>
welcome back to the power whyrusleeping :)
<Smilex>
cryptix: I never got the file wie IPFS, and the one in the github.com link fails for the same reasons the previous one failed
<ansuz>
oh, I heard seattle was having issues
<ansuz>
big storm, eh?
<Smilex>
actually no, just the code.google.com link fails
devbug has quit [Ping timeout: 252 seconds]
dignifiedquire_ has joined #ipfs
Alien961 has joined #ipfs
AlienCQ has quit [Ping timeout: 260 seconds]
Matoro has quit [Ping timeout: 265 seconds]
Matoro has joined #ipfs
Encrypt has quit [Quit: Quitte]
jabberwocky has joined #ipfs
jabberwocky has quit [Ping timeout: 255 seconds]
Alien961 has quit []
<lgierth>
daviddias: have you stopped downloading?
master8787 has quit [Quit: Leaving]
s_kunk has joined #ipfs
devbug has joined #ipfs
dignifiedquire_ has quit [Quit: dignifiedquire_]
voxelot has quit [Ping timeout: 250 seconds]
Soft has quit [Read error: Connection reset by peer]
merlijn_ has joined #ipfs
rendar has quit [Ping timeout: 260 seconds]
devbug has quit [Read error: Connection reset by peer]
bedeho has quit [Ping timeout: 246 seconds]
rendar has joined #ipfs
devbug has joined #ipfs
<dignifiedquire>
daviddias: I’m nearly done porting ipfs-api to wreck, request was pissing me of
Soft has joined #ipfs
revolve has quit [Ping timeout: 244 seconds]
pfraze has quit [Remote host closed the connection]
jcd has quit [Quit: Ex-Chat]
merlijn_ has quit [Ping timeout: 250 seconds]
devbug has quit [Read error: Connection reset by peer]
devbug has joined #ipfs
chriscool has quit [Ping timeout: 240 seconds]
<haadcode>
dignifiedquire, daviddias: found the problem in streaming and fixed it. still need to write tests but might not have time for today.
chriscool has joined #ipfs
<dignifiedquire>
haadcode: cool, what was the issue (dropping request right now, so you know ;))
<haadcode>
dignifiedquire: wait what, are you switching the library now?
<dignifiedquire>
yeah request was pissing me off :(
<haadcode>
oh wow. so this fix is useless?
<dignifiedquire>
I have no idea what you fixed
<dignifiedquire>
or rather how you fixed
<dignifiedquire>
it might still be needed
<dignifiedquire>
all node tests passing with wreck now :)
<haadcode>
the problem was request. if you have the end callback (err, res, body), request internally converts its buffer to string (I guess for the body) before calling it, and if buffer.toString().size > 256MB, it throws an error.
<dignifiedquire>
and the code is much nicer
<dignifiedquire>
ah I should have known, too much magic in request..
<haadcode>
the fix was to listen to the 'end' event on the response you get from request instead of passing the onEnd as a callback
<haadcode>
dignifiedquire: so hold on, are you saying you're gonna now swap the library in the official version and there's no need for PR for this fix?
<dignifiedquire>
well we still need a test to make sure it doesn’t happen
<dignifiedquire>
and if you already have the code I’m happy to pull it in, as it sounds you are already done with it
<haadcode>
not gonna do a PR if you're gonna change the lib on the near future, wasted work
<haadcode>
if not, I'm happy to do a PR
devbug has quit [Ping timeout: 240 seconds]
chriscool has quit [Ping timeout: 276 seconds]
devbug has joined #ipfs
<dignifiedquire>
haadcode: I’m going to push my work and if you have a good way of testing I’d appreciate if you could try if it fixes the issue as well
<whyrusleeping>
javascripters, question
* dignifiedquire
runs away
<whyrusleeping>
when youre writing an app, and youre also writing a module that is used by that app
<whyrusleeping>
whats that workflow like?
<whyrusleeping>
do you write the module in one dir, publish it, then import it into your app?
<ipfsbot>
[js-ipfs-api] Dignifiedquire opened pull request #125: [WIP] Wreck and webpack (master...webpack) http://git.io/v4PTQ
<dignifiedquire>
whyrusleeping: that depends on if I know I want to publish it later
<whyrusleeping>
hrm, okay
<dignifiedquire>
whyrusleeping: but if I plan on publishing it on its own, I would create two folders, “app” and “module”, and use `npm link` to pull in the local module version into app
<whyrusleeping>
ah, okay.
<whyrusleeping>
that link command is what i was after then
<haadcode>
dignifiedquire: ok. will test it in a bit.
<dignifiedquire>
haadcode: thanks :)
<dignifiedquire>
and sorry :(
pfraze has joined #ipfs
chriscool has quit [Ping timeout: 265 seconds]
<haadcode>
dignifiedquire: if you want to test it yourself, the test case is simple: try to request a file > ~256MB from via the js-ipfs-api (I had 1GB file)
<dignifiedquire>
haadcode: okay thanks
<haadcode>
dignifiedquire: there was another another funky thing with the request implementation: the file was first fully buffered before the piping started, so it had a long lead up time before the stream started on the receivin end. fixed that too, but you probably want to make sure that the response piping starts immediately after first chunk is received from the ipfs api.
<dignifiedquire>
haadcode: yeah, I really need to find a way to have an automated test to make sure that’s happening
<haadcode>
dignifiedquire: I took a look at the wreck version and there's the onEnd callback, so should check it behaves as expected. can easily be tested with large files.
<haadcode>
gotta go now, be back later
devbug has quit [Read error: Connection reset by peer]
<dignifiedquire>
haadcode: will test soon thanks for all the help!
devbug has joined #ipfs
<tperson>
dignifiedquire: you just blew my mind with npm link. I always knew it existed, I just never thought to use it in that way.
<dignifiedquire>
tperson: :)
<dignifiedquire>
without it developing multiple modules really sucks
<tperson>
Ya, I'm going through that now. I just have a directory and use relative pathing. The refactor work to move everything into separate modules and publish is going to suck, but I can probably do that work slowly now and just npm link things till I'm ready to publish.
zugz has quit [Ping timeout: 272 seconds]
zugz has joined #ipfs
voxelot has joined #ipfs
voxelot has quit [Changing host]
voxelot has joined #ipfs
devbug has quit [Read error: Connection reset by peer]
devbug has joined #ipfs
ygrek_ has quit [Ping timeout: 240 seconds]
ianopolous2 has quit [Ping timeout: 246 seconds]
<dignifiedquire>
daviddias: good news, looks like with request -> wreck and browserify -> webpack I got the minified version down to 400kb
<tperson>
Is that the js-ipfs-api?
<tperson>
That seems absolutely massive
devbug has quit [Read error: Connection reset by peer]
<kandinski>
daviddias: what do you compare multihashes of different lengths? (for instance, when needing to find the Kademlia distance between a sha1 and a sha3 multihash) Have you considered the case?
<kandinski>
s/what/how/
devbug has joined #ipfs
Encrypt has joined #ipfs
<dignifiedquire>
tperson: yes it is, at the moment we are at 800
<dignifiedquire>
and it’s not that big, we have to bring in all the request handling fun
<dignifiedquire>
jbenet: daviddias whyrusleeping is there a way to remove a file from my local ipfs node?
<dignifiedquire>
(not pinned)
<dignifiedquire>
haadcode: finished my pr and tested it fixes the streaming issues :)
<tperson>
You can gc, but I assume you want to just gc a single object?
<dignifiedquire>
tperson: yeah
<dignifiedquire>
I just added it for testing but want to clean it up cause it’s pretty large
<dignifiedquire>
something like ipfs remove would be nice :(
<whyrusleeping>
dignifiedquire: thats a thing i dont know if we can do right now
<dignifiedquire>
so repo gc would clean up everything not pinned?
<dignifiedquire>
well I just tried and the file is still available
<dignifiedquire>
-.-
* dignifiedquire
tells himself to never add files to ipfs he might not want on there
<tperson>
I mean... you could delete the blocks on disk. Just never request them, who knows what the daemon will do lol
<dignifiedquire>
tperson: yeah I just ran ipfs init -f
<tperson>
It may think they exists, maybe it will just fail to read and request them from the network
<dignifiedquire>
because ipfs repo gc did not work :(
<dignifiedquire>
luckily I didn’t have anything important added
<tperson>
You should checkout whyrusleeping cli util
<tperson>
That way you can separate your own daemon with whatever one you want to play with.
<kpcyrd>
dignifiedquire: if you put it into ipfs it stays there until everybody stopped seeding, which happens after requesting it. it's possible to discover files from the dht, as far as I understand, haven't tried that tho.
NeoTeo has quit [Quit: ZZZzzz…]
<whyrusleeping>
lol, this hash: QmQmqbqoqpJoYyUHqA9VKDreoMpEUHEy23gpxxqoyHhWs8
<whyrusleeping>
i thought i miscopied something
ianopolous2 has joined #ipfs
<tperson>
lol
<tperson>
A lot of `q`s in there.
<haadcode>
dignifiedquire: sounds good! can't test the PR tonight but can verify that it works later
devbug has quit [Ping timeout: 250 seconds]
ygrek has joined #ipfs
r04r is now known as zz_r04r
devbug has joined #ipfs
border0464 has quit [Quit: sinked]
zz_r04r is now known as r04r
roguism1 has joined #ipfs
roguism has quit [Ping timeout: 240 seconds]
<dignifiedquire>
haadcode:
tendie has quit [Ping timeout: 272 seconds]
<daviddias>
That is really rad dignifiedquire ! Re: js-ipfs-api
<daviddias>
kandinski: well, we don't have that in place and have decided on anything
<daviddias>
But the more simple and probably the cleanest, would be to hash the sha1 to sha3 and compare
<daviddias>
Well, actually no
<daviddias>
If we upgrade from sha2-256 to sha3 for example, we need to hash the blob again
amade has quit [Quit: leaving]
fazo has joined #ipfs
<fazo>
I'm facing the problem of implementing user discovery for my service. What would be the best way to do it assuming I'm using js-ipfs-api?
<fazo>
I thought about having all users add somefile then query the dht. Maybe there's a better way though
<kandinski>
daviddias: yeah, but that would require you to get every blob. Expensive if you're only routing.
<kandinski>
daviddias: Or impossible if the reason you're computing distances is that you're *searching* for the blob!
devbug has quit [Ping timeout: 252 seconds]
<kandinski>
so options are:
<kandinski>
- rehashing to address (or other application-dependent) hash lenght
<kandinski>
- rehashing with 0s
<kandinski>
I mean, padding with 0s
<kandinski>
- padding with itself, which doesn't add entropy but helps with dispersal