JustinDrake has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
JustinDrake has joined #ipfs
maxlath has joined #ipfs
elico has quit [Quit: Leaving.]
<_mak>
is there a way to see the outgoing traffic of the daemon?
espadrine has joined #ipfs
ygrek has quit [Ping timeout: 244 seconds]
slothbag has quit [Quit: Leaving.]
palkeo has quit [Quit: Konversation terminated!]
palkeo has joined #ipfs
JustinDrake has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
jedahan has joined #ipfs
<lgierth>
_mak: you can start the daemon with --disable-transport-encryption and use wireshark or pcap. you won't be able to dial other nodes on the network though
<lgierth>
_mak: so you'd need to have two daemons with that options so there's any network traffic
<_mak>
lgierth: I see... do you know of any other third party app I could use to see the daemon's traffic?
<_mak>
netstat will only show me connections
<lgierth>
there is none
<lgierth>
yeah
<lgierth>
the traffic is encrypted
<_mak>
I'm interested only in the total amount
<lgierth>
so you can technically see it but it's just a cipherstream
jedahan has quit [Ping timeout: 260 seconds]
<lgierth>
ah i see
<_mak>
like, how much was 'seeded' from that particular daemon
<lgierth>
we don't have an exact number for that
jholden has joined #ipfs
<_mak>
ok, thanks mate
<lgierth>
you can check `ipfs stats bw` but that won't show you "bitswap blocks sent"
anonymuse has joined #ipfs
<_mak>
nice, I'll check it out as soon as 'ipfs-see-all lost-pins' finishes :)
<_mak>
not sure if there's anything happening, I've ran it like 7 minutes ago and got no feedback message so far
<lgierth>
got a lot of data in the repo?
<lgierth>
check `top` when in doubt, it should show it spinning wild
jholden has quit [Ping timeout: 240 seconds]
<_mak>
yeah, it is doing something indeed :)
anonymuse has quit [Ping timeout: 250 seconds]
JustinDrake has joined #ipfs
infinity0 has quit [Ping timeout: 244 seconds]
infinity0 has joined #ipfs
JustinDrake has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
infinity0 has quit [Remote host closed the connection]
ianopolous has quit [Ping timeout: 258 seconds]
maxlath has quit [Ping timeout: 268 seconds]
infinity0 has joined #ipfs
ianopolous has joined #ipfs
chriscool has quit [Ping timeout: 256 seconds]
soloojos has quit [Ping timeout: 245 seconds]
maxlath has joined #ipfs
Anduck_ has joined #ipfs
Anduck_ has quit [Client Quit]
infinity0 has quit [Ping timeout: 250 seconds]
ianopolous has quit [Ping timeout: 245 seconds]
infinity0 has joined #ipfs
Encrypt has quit [Quit: Quit]
bastianilso____ has quit [Quit: bastianilso____]
yoosi has quit [Remote host closed the connection]
IlanGodik has quit [Quit: Connection closed for inactivity]
ianopolous has quit [Ping timeout: 268 seconds]
ianopolous has joined #ipfs
mguentner has joined #ipfs
ianopolous_ has joined #ipfs
chriscool has quit [Quit: Leaving.]
chriscool has joined #ipfs
ianopolous has quit [Ping timeout: 246 seconds]
chriscool has quit [Read error: No route to host]
chriscool has joined #ipfs
bastianilso____ has joined #ipfs
anonymuse has joined #ipfs
ianopolous_ has quit [Ping timeout: 240 seconds]
soloojos has joined #ipfs
anonymuse has quit [Ping timeout: 244 seconds]
maxlath has quit [Ping timeout: 260 seconds]
ianopolous_ has joined #ipfs
mguentner has quit [Ping timeout: 248 seconds]
mguentner has joined #ipfs
jager has quit [Ping timeout: 260 seconds]
mib_kd743naq has joined #ipfs
<mib_kd743naq>
with the cidv1 stuff - is it feasible that ipfs will ever support sha1 as the hashing algorithm?
Encrypt has joined #ipfs
<mib_kd743naq>
or minimum of 256bit hashes will be enforced as a matter of principle?
<mib_kd743naq>
the reason I ask is that I am experimenting with super-deep-trees, and the 32byte(before overhead) is hurting me as my links are too fat
maxlath has joined #ipfs
<mib_kd743naq>
1.5gb(!) spent on links alone ( describing a total of ~40k chunks )
<Mateon1>
Uhh, I think I either found a bug in cid parsing or I'm doing something wrong
cwahlers has joined #ipfs
<Mateon1>
I'm encoding <01 aa 11 bb 90909090> into b58 (intentional hash length mismatch to dump parsed multihash). I get <H9UXvgykvs>, but when I `ipfs get zH9UXvgykvs`, I get `Error: multihash length inconsistent: &{187 144 [144 144 144]}`. Have I missed a field in Cid encoding? This looks fine to me - <cid version> <codec> <sha1 multihash> <hash length> <hash bytes...>
<Mateon1>
It seems that something is shifted by half a byte
<Mateon1>
Oh, actually, it might be varints, let me check <0x80
<Mateon1>
I couldn't remember whether valid codecs were 0x60..0x62 or 0x70..0x72
<Mateon1>
Oh, I'm totally wrong either way
<Mateon1>
I need to stop being lazy and to patch go-multibase to actually support multiple bases. It would be much easier to mess around if 'f' (hex) encoding worked
sprint-helper has quit [Remote host closed the connection]
sprint-helper has joined #ipfs
anonymuse has quit [Remote host closed the connection]
<mib_kd743naq>
whyrusleeping: ^^ you mention in this backlog that you will open some github issues to track the "non-sha2" work
<mib_kd743naq>
whyrusleeping: but searching didn't yield much: could you point me to the issues in question so I could subscribe?
chris613 has joined #ipfs
elico has joined #ipfs
<graffen>
What's the best way to keep the ipfs daemon running? I just have 'ipfs daemon &' running under a passwordless user on one of my VPSes (to serve my website from) but it seems to die every now and then. What do you guys do?
espadrine has joined #ipfs
cubemonk1y has joined #ipfs
<lgierth>
Mateon1: multibase PRs welcome :) we definitely want more bases and there's a few issues open for them
<lgierth>
mib_kd743naq: same for multihash and sha1 :) nobody has wanted it badly enough to contribute it :)
<Mateon1>
graffen: Just have a cronjob running a script that checks whether ipfs is running, if not: run `ipfs repo fsck` (to remove locks) and `ipfs daemon &`
herzmeister has quit [Quit: Leaving]
<Mateon1>
ipfs seems to randomly die on my raspberry pi, but it never crashes on my Windows machine. Not sure why
herzmeister has joined #ipfs
<mib_kd743naq>
lgierth: I would truly consider doing the sha1 part, but I need to know what is the stance on shorter-than-256bit hashes: if this work never makes it into go-ipfs proper, then it's moot (as I do need full interoperability)
<lgierth>
Mateon1: crash dumps!
<Mateon1>
lgierth: Where might I find them?
anonymuse has joined #ipfs
<Mateon1>
lgierth: Also, note that it doesn't crash often, it crashed twice in the entire time I used it on my pi, and I had to kill it a couple of times due to it eating all memory.
<lgierth>
mib_kd743naq: yeah i hear you. if multihash supports sha1, ipfs supports it. it just will never use sha1 by default :) sha1 is close to being completely broken
<lgierth>
(schneier estimated it to be broken by the end of 2016)
sprint-helper has quit [Remote host closed the connection]
sprint-helper has joined #ipfs
<lgierth>
Mateon1: crash dumps would be on the console where you started it
<mib_kd743naq>
lgierth: multihash does support sha1 fwiw, but it seems ipfs itself doesn't do anything other than sha256
<lgierth>
mib_kd743naq: ah right bitswap. yeah we're *definitely* going to fix that
<lgierth>
i.e. bitswap MUST be agnostic as to which kind of multihash it is
<mib_kd743naq>
well... even bitswap aside, say I want to raw-store a blob:
<mib_kd743naq>
~$ echo "foo" | ipfs block put -f raw zdvgqDePc8mcVqHh8MwZ8fbWWHkNxqjXqVDn1G5xGfwyyuNvw
<mib_kd743naq>
lgierth: ^^ there is no option to say e.g. "store this as a cidv1-sha3"
<mib_kd743naq>
it just assumes sha256 and that's that
<Mateon1>
lgierth: Also, I have a question. If I request a shorter multihash (say 20 bytes of sha256 instead of 32), will the person who has the longer matching hash ever respond to the short query?
pfrazee has joined #ipfs
<lgierth>
Mateon1: nope
<lgierth>
20 bytes is never a valid sha256 hash
<Mateon1>
Uhh... then why does multihash have a length field?
<lgierth>
mib_kd743naq: mh yeah i hear you. some kind of --hash=sha1 option eh?
<lgierth>
Mateon1: there are hash algos with a variable length
<Mateon1>
I'd think it was for truncating/expanding hashes to other lengths
<Kubuxu>
lgierth: no, length allows for truncation
<lgierth>
iirc sha256 isn't truncatable
<lgierth>
Kubuxu: yeah but does sha256 allow for truncation?
<lgierth>
i thought that was just sha512
<mib_kd743naq>
lgierth: that's where I enter "don't know" territory, as I am still not entirely sure how the go-ipfs plumbing fits together
<Kubuxu>
nope, you just get the hash and truncate it
<lgierth>
mh ok
<lgierth>
others still won't respond though
<Kubuxu>
probably no
<lgierth>
mib_kd743naq: with raw blocks it's super easy to make the multihash yourself though
<lgierth>
as there's no wrapping data structure
<Mateon1>
Well, right now, definitely not. I actually can't request a cid version of a multihash with just 0170 prepended. (unchanged otherwise)
<lgierth>
is that what you meant earlier? ^
<Kubuxu>
Mateon1: you can't as the other node doesn't know about block like that
<Kubuxu>
even though it might just store block like that with CIDv0
<Kubuxu>
the blockstore differentiates between those two
<Mateon1>
Unfortunate. /ipns/ipfs.io currently resolves to /ipfs/QmaYRrBpZQRjonxBpBBSLvYmy893ySPpXpHCakhomaQtkU, but I can't request /ipfs/zdj7WhdgDctQsuLgWG7Yv94DJPLbLLVgyoxQ8gjFZ6Ub1ynvQ
<Kubuxu>
Mateon1: yes this will be the case
<lgierth>
oh right becaue there's noone really who can translate between these hashes
<Kubuxu>
it might be worth adding translation layer so CIDv0 hashes can be accessed as CIDv1 dag-pb and reverse.
<lgierth>
#wantlistexplosion
sprint-helper has quit [Remote host closed the connection]
sprint-helper has joined #ipfs
<Mateon1>
By the way, when does IPFS switch to using CIDv1 by default? (in ipfs add)
<lgierth>
in the next release
<mib_kd743naq>
hm hm hm this talk of partial hashes makes me wonder if specifying truncated hashes in links will actually work
* mib_kd743naq
tries...
<haad>
I've been recently installing various different versions of ipfs, so I made a little tool to help installing IPFS easily: https://github.com/haadcode/ivm. if you find yourself in need to switching between versions quickly or installing new ones constantly, this might help you.
anonymuse has quit [Remote host closed the connection]
herzmeister has quit [Quit: Leaving]
herzmeister has joined #ipfs
ianopolous_ has quit [Ping timeout: 258 seconds]
<mib_kd743naq>
nope... doesn't work
<mib_kd743naq>
hm hm hm wonder if it is worth digging into go-ipfs and trying to figure out why a link to a 224-bit sha256 portion doesn't work
<mib_kd743naq>
I suspect it is the same cidv0 vs v1 problem: there is nothing in the index that points to the existence of the linked block under a shortened hash
<mib_kd743naq>
Kubuxu: ^^ about right?
<Mateon1>
mib_kd743naq: Probably because it's not added as such to the datastore, and there is no handling for truncated hashes
<mib_kd743naq>
oh well, that was a fun thought experiment ;)
cubemonk1y has quit [Ping timeout: 265 seconds]
cubemonk1y has joined #ipfs
<lgierth>
haad: isn't that was ipfs-update does? :)
robattila256 has quit [Quit: WeeChat 1.6]
robattila256 has joined #ipfs
anonymuse has joined #ipfs
<Kubuxu>
mib_kd743naq: I think yes
<mib_kd743naq>
I see various hints on the web that the blockstore might be something else other than the fs
ianopolous has joined #ipfs
<mib_kd743naq>
has any of this been implemented, or as of today .ipfs/blocks is the only thing that actually works?
<Kubuxu>
we used to have s3 backed one
<Kubuxu>
but it was very inefficient\
<mib_kd743naq>
aye
<mib_kd743naq>
I am looking to try to see in reality how this 40m links file will actually behave, but running into fs limitations ;)
<lgierth>
might try a ceph blockstore in the future
* mib_kd743naq
will provision a new subblock-allocating fs for tests tomorrow
Qwertie has quit [Ping timeout: 245 seconds]
Qwertie has joined #ipfs
IlanGodik has joined #ipfs
anonymus_ has joined #ipfs
anonymuse has quit [Read error: Connection reset by peer]
espadrine has quit [Ping timeout: 260 seconds]
anonymus_ has quit [Remote host closed the connection]
cubemonk1y has quit [Ping timeout: 248 seconds]
cubemonk1y has joined #ipfs
rgrinberg has joined #ipfs
ylp has quit [Quit: Leaving.]
rgrinberg has quit [Remote host closed the connection]
rgrinberg has joined #ipfs
ianopolous_ has joined #ipfs
ianopolous has quit [Ping timeout: 256 seconds]
anewuser has joined #ipfs
anonymuse has joined #ipfs
cubemonk1y has quit [Ping timeout: 248 seconds]
ylp has joined #ipfs
herzmeister has quit [Quit: Leaving]
herzmeister has joined #ipfs
anonymuse has quit [Remote host closed the connection]
<mib_kd743naq>
grrrrr I am getting lost in the source again
<mib_kd743naq>
where do I find how the file structure under .ipfs/blocks/... is laid out?
<mib_kd743naq>
I want to look in this dir directly before calling the API (much more overhead)
<Kubuxu>
go-ds-flatfs
chriscool has joined #ipfs
<mib_kd743naq>
Kubuxu: thanks
anonymuse has joined #ipfs
<Kubuxu>
keys are base32 encode
<Kubuxu>
no padding
anonymuse has quit [Remote host closed the connection]
<mib_kd743naq>
Kubuxu: where is this happening? just asking for my own sake...
JustinDrake has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
herzmeister has quit [Quit: Leaving]
herzmeister has joined #ipfs
ylp has quit [Quit: Leaving.]
bastianilso____ has quit [Ping timeout: 250 seconds]
<Mateon1>
lgierth: Fittingly, IPFS crashed on my raspi about 2h ago, at 16:19:58Z, where can I find the crash logs? (I can give some scrollback too)
<Mateon1>
Actually, the scrollback is the crash dump, so it was probably a panic... Except there are so many goroutines I can't see the panic
Boomerang has joined #ipfs
<Mateon1>
Gah, I can't see anything about this crash, my netdata log only goes back one hour
<pfrazee>
AFAICT the only way to identify whether a DAGNode is a directory or a file is to fetch the data and unmarshal it to get the type attribute, correct?
Boomerang has quit [Ping timeout: 250 seconds]
<pfrazee>
is there another way to do that? Because that requires me to load the entire data chunk into memory, which is bad for large files
<Kubuxu>
pfrazee: we chunk into 256k chunks
Boomerang has joined #ipfs
Encrypt has quit [Quit: Quit]
* Mateon1
learns lesson, and starts ipfs piping into tee
<pfrazee>
Kubuxu: so I'm using the js api, if I call object.data() I just get one big marshalled chunk. If I can something like file.get, it'll start streaming, and then... do I unmarshal each chunk, and check the type on the first chunk?
<Kubuxu>
no idea
<Kubuxu>
I don't know about files API.
<Kubuxu>
s/files API/js-ipfs-api
<mib_kd743naq>
Kubuxu: um what about dags with custom chunking (like the stuff I am playing with)
<mib_kd743naq>
surely the "check if 256k" will fail in these cases
<Mateon1>
mib_kd743naq: Limited to 2MB by network layer
<Kubuxu>
he won't be checking if it is == 256k because you can add smaller files too
<Kubuxu>
'and they won't be 256k
Boomerang has quit [Ping timeout: 245 seconds]
<mib_kd743naq>
indeed, didn't think it through ;)
Boomerang has joined #ipfs
Boomerang has quit [Ping timeout: 268 seconds]
tmcothran[m] has left #ipfs ["User left"]
Boomerang has joined #ipfs
infinity0 has quit [Remote host closed the connection]
<pinbot>
now pinning /ipfs/QmUU4oG28exPFxvtb8YSAUd7qPyxjeyCN2eRzrmvKvZSAb
Qwertie has quit [Ping timeout: 250 seconds]
<pinbot>
[host 3] failed to grab refs for /ipfs/QmUU4oG28exPFxvtb8YSAUd7qPyxjeyCN2eRzrmvKvZSAb: Post http://[fc4e:5427:3cd0:cc4c:4770:25bb:a682:d06c]:5001/api/v0/refs?arg=/ipfs/QmUU4oG28exPFxvtb8YSAUd7qPyxjeyCN2eRzrmvKvZSAb&r=true&encoding=json&stream-channels=true&: dial tcp [fc4e:5427:3cd0:cc4c:4770:25bb:a682:d06c]:5001: getsockopt: connection timed out
JustinDrake has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
Qwertie has joined #ipfs
mguentner has joined #ipfs
dignifiedquire has quit [Quit: Connection closed for inactivity]
anonymuse has quit [Remote host closed the connection]
Encrypt has quit [Quit: Quit]
dignifiedquire has joined #ipfs
ulrichard has joined #ipfs
anewuser has quit [Quit: anewuser]
lkcl has joined #ipfs
arpu has quit [Ping timeout: 250 seconds]
JustinDrake has joined #ipfs
Boomerang has joined #ipfs
<Mateon1>
Wait... /api/v0/refs ?... So !pin doesn't pin, but caches?
Boomerang has quit [Ping timeout: 252 seconds]
<lgierth>
Mateon1: first refs, then pin/add
<Mateon1>
Hm, okay. That's not immediately clear
<lgierth>
i dunno, but github.com/whyrusleeping/pinbot is only like 200 lines :):)
<Mateon1>
Maybe. I didn't look at the source.
<Mateon1>
Also, I found a nice little lib for arbitrary big number base conversion. Just need to splice it in multibase somehow, and I can then throw out the b58 code. (By the way, multibase isn't gxified)
<Kubuxu>
Mateon1: most formats we are interested in are already implemented in stdlib or implemented for go-ipfs
<Mateon1>
Oh, okay, I need to copy the char for b32 from the spec then, the current file only includes b1, binary, octal, dec, hex and two variants of b58
<Kubuxu>
you don't need to add them all at the same time
Boomerang has joined #ipfs
maxlath has quit [Ping timeout: 245 seconds]
chriscool has quit [Ping timeout: 248 seconds]
Boomerang has quit [Ping timeout: 240 seconds]
Boomerang has joined #ipfs
soloojos has quit [Ping timeout: 245 seconds]
Boomerang has quit [Ping timeout: 250 seconds]
Boomerang has joined #ipfs
<Mateon1>
Actually, I think I might need to change a couple things, because there can be many chars for one multibase base
soloojos has joined #ipfs
Boomerang has quit [Ping timeout: 250 seconds]
Qwertie has quit [Ping timeout: 265 seconds]
Qwertie has joined #ipfs
ulrichard has quit [Quit: Ex-Chat]
screensaver has quit [Ping timeout: 268 seconds]
zanadar has joined #ipfs
anonymuse has joined #ipfs
maxlath has joined #ipfs
PrinceOfPeeves has quit [Quit: Leaving]
A124 has quit [Ping timeout: 260 seconds]
anonymuse has quit [Remote host closed the connection]
anonymuse has joined #ipfs
rgrinberg has joined #ipfs
Leptolepidae has quit [Ping timeout: 248 seconds]
ulrichard has joined #ipfs
ulrichard has quit [Client Quit]
everywhen has joined #ipfs
JustinDrake has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
anonymuse has quit [Remote host closed the connection]
everywhen has quit [Ping timeout: 258 seconds]
anonymuse has joined #ipfs
Wurzburger has joined #ipfs
Wurzburger has quit [Excess Flood]
PseudoNoob has quit [Remote host closed the connection]
palkeo has quit [Ping timeout: 260 seconds]
palkeo has joined #ipfs
palkeo has joined #ipfs
palkeo has quit [Changing host]
mguentner has quit [Ping timeout: 245 seconds]
anonymuse has quit [Remote host closed the connection]
JustinDrake has joined #ipfs
CrashOverride has joined #ipfs
maxlath has quit [Quit: maxlath]
pseudobulb has joined #ipfs
palkeo has quit [Quit: Konversation terminated!]
reit has quit [Ping timeout: 260 seconds]
<Mateon1>
I just PRed my go-multibase changes, haven't implemented base 2,8,10 but everything else seems to work
Boomerang has joined #ipfs
<Mateon1>
Also, this is the first Golang code I've ever written :) Took a while + a bunch of Google, but I'm happy anyway
Boomerang has quit [Ping timeout: 250 seconds]
IlanGodik has quit [Quit: Connection closed for inactivity]
infinity0 has quit [Ping timeout: 245 seconds]
JustinDrake has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
infinity0 has joined #ipfs
ianopolous__ has joined #ipfs
ianopolous_ has quit [Ping timeout: 244 seconds]
CrashOverride has quit [Ping timeout: 256 seconds]