CMCDragonkai2 has quit [Ping timeout: 260 seconds]
<kyledrake>
Any ideas for speeding up ipfs add? There's some sort of CPU bottleneck going on
<kyledrake>
It really chews up a few cores when large files are added
<whyrusleeping>
kyledrake: using that version i sent you?
CMCDragonkai2 has joined #ipfs
<kyledrake>
I tried 0.4.8 on my local machine too just to make sure it wasn't specific to that.
<whyrusleeping>
kevina: lets just wait for keks to respond
<whyrusleeping>
kyledrake: hrm... How fast are the adds going?
<whyrusleeping>
it could just be that youre adding things as fast as your cpu can sha256 hash
<kyledrake>
openssl rand -out sample.txt -base64 $(( 2**30 * 3/4 )) <- I'm running that until about 350MB and CtrlCing it
<whyrusleeping>
kyledrake: are you running the add with --raw-leaves ?
juchemane has quit [Ping timeout: 260 seconds]
<kyledrake>
Nope
<whyrusleeping>
I would use that, should reduce resource consumption a little
<whyrusleeping>
not significantly though, just useful
<whyrusleeping>
while the add is running, could you run `curl localhost:5001/debug/pprof/profile > ipfs.cpu`
<whyrusleeping>
that should take 30 seconds and create a cpu profile
<whyrusleeping>
once you have that, you can send me that and the exact ipfs binary your daemon is running, and i can see exactly what is causing the cpu usage
<whyrusleeping>
e337: it also looks like the network is under some heavy load right now...
<whyrusleeping>
our gateways are fronting 80 requests per second, where we normally have 3-4
<kyledrake>
Mind you, this may just be a general thing, I don't think it's specific to me. I would be surprised if it was bottlenecking from sha256.. I can shasum this in 0.045 seconds, but the import is taking 30-40s
<kyledrake>
(shasum -a 256)
<kyledrake>
wait, I'm dumb.
<kyledrake>
6 seconds to shasum -a 256 it. So maybe.
<whyrusleeping>
Yeah... Theres some overhead there too
<whyrusleeping>
we need to parallelize things
<whyrusleeping>
and also switch to blake2b for hashing
<whyrusleeping>
kevina: if you've got time, i'd love to have some more eyes reviewing bitswap sessions
<whyrusleeping>
getting that merged in should really reduce idle bandwidth usage a good deal
<kyledrake>
It's fairly parallel. It seems to be eating up 3 cores, and I've got 4 here for it to work with.
crankylinuxuser has quit [Disconnected by services]
conway has joined #ipfs
leeola has joined #ipfs
Boomerang has joined #ipfs
jkilpatr has quit [Ping timeout: 260 seconds]
jkilpatr has joined #ipfs
CMCDragonkai has joined #ipfs
CMCDragonkai is now known as Guest45034
CMCDragonkai4 has quit [Ping timeout: 252 seconds]
anewuser has quit [Ping timeout: 240 seconds]
anewuser has joined #ipfs
ashark has joined #ipfs
anewuser has quit [Ping timeout: 255 seconds]
anewuser has joined #ipfs
cjb has quit [Remote host closed the connection]
m3tti has quit [Ping timeout: 240 seconds]
stoopkid has quit [Quit: Connection closed for inactivity]
Guest45034 has quit [Ping timeout: 260 seconds]
Guest45034 has joined #ipfs
soesoe has quit [Ping timeout: 240 seconds]
shizy has joined #ipfs
cjb has joined #ipfs
cjb has quit [Remote host closed the connection]
cjb has joined #ipfs
btmsn has quit [Ping timeout: 245 seconds]
btmsn has joined #ipfs
Guest45034 has quit [Ping timeout: 260 seconds]
dlight has joined #ipfs
rovdyl has quit [Read error: Connection reset by peer]
rovdyl has joined #ipfs
cjb has quit [Quit: Ping timeout (120 seconds)]
galois_dmz has joined #ipfs
TheGuyWho has quit [Ping timeout: 255 seconds]
arkimedes has joined #ipfs
TheGuyWho has joined #ipfs
<oed>
hey, I'm wondering about the ipfs filestore feature. is it only possible to use when doing ipfs add, or is it possilbe to use the feature when doing ipfs get?
anewuser has quit [Ping timeout: 240 seconds]
SchrodingersScat has quit [Quit: WeeChat 1.4]
SchrodingersScat has joined #ipfs
mildred has quit [Ping timeout: 252 seconds]
mildred2 has quit [Ping timeout: 245 seconds]
anewuser has joined #ipfs
<sprint-helper>
The next event "IPFS All Hands Call" is in 15 minutes.
<whyrusleeping>
oed: only with add right now. It should be possible to make it work with get though, PRs accepted ;)
<dgrisham>
whyrusleeping: regarding that bitswap PR from last week (regarding the 'bytes_sent' bug), is it possible the same issue exists in js-ipfs? I haven't looked myself, and haven't run js-ipfs yet to check
<whyrusleeping>
dgrisham: i don't think js-ipfs even has that accounting stuff in it
<sprint-helper>
Correct usage: sprint-helper: announce <args> | next | now | tomorrow | help
<richardlitt>
^ See above.
anewuser has joined #ipfs
<lgierth>
zoom just died on me
<PureTryOut[m]>
guys I was wondering (as someone new to IPFS), will there ever be a way to run applications using server-side code (think PHP, game servers, databases, etc) on PHP? IPFS definitely sounds awesome, but I'm not sure if it'll ever become "big" if you can't run such applications on it
warner` has joined #ipfs
CMCDragonkai has joined #ipfs
<whyrusleeping>
PureTryOut[m]: would you call https://orbit.chat an application?
CMCDragonkai is now known as Guest20366
<PureTryOut[m]>
I don't know that program, but does it use serverside code?
<lgierth>
the goal is to not need server-side code
<whyrusleeping>
PureTryOut[m]: no, theres no server
<lgierth>
we're not 100% there yet for all use cases, but getting there
<whyrusleeping>
PureTryOut[m]: what sorts of applications are you wanting to build?
<Kubuxu>
The link my be be working on not depending on DNS caches
barnacs_ has joined #ipfs
<whyrusleeping>
Kubuxu: somethings a bit weird there
<PureTryOut[m]>
whyrusleeping: well not me necessarily, but think of a webshop. it will need a database and such
tom1 has joined #ipfs
<Kubuxu>
whyrusleeping: why?
<whyrusleeping>
PureTryOut[m]: have you seen open bazaar?
<PureTryOut[m]>
no
<Kubuxu>
it might be resolving or not, or failing to resolve half way
<Kubuxu>
as for DNS propagation
<whyrusleeping>
Kubuxu: hrm... Yeah, it might be half failing then
<whyrusleeping>
some things are failing to resolve when i refresh the page
<Kubuxu>
yeah, DNS is still propagating
<whyrusleeping>
PureTryOut[m]: open bazaar is a fully peer to peer webstore built on ipfs
pawalls_ has joined #ipfs
m_anish__ has joined #ipfs
Guest45034 has quit [*.net *.split]
mahloun has quit [*.net *.split]
rovdyl has quit [*.net *.split]
tsetair has quit [*.net *.split]
aedigix has quit [*.net *.split]
warner has quit [*.net *.split]
Akaibu has quit [*.net *.split]
kerozene has quit [*.net *.split]
esph has quit [*.net *.split]
rodarmor has quit [*.net *.split]
kpcyrd has quit [*.net *.split]
athan has quit [*.net *.split]
ofdm has quit [*.net *.split]
m_anish has quit [*.net *.split]
xelra has quit [*.net *.split]
Confiks has quit [*.net *.split]
lassulus has quit [*.net *.split]
barnacs has quit [*.net *.split]
c0dehero has quit [*.net *.split]
pawalls has quit [*.net *.split]
flyingkiwi has quit [*.net *.split]
FrankPetrilli has quit [*.net *.split]
pawalls_ is now known as pawalls
Nycatelos has quit [*.net *.split]
tom1 is now known as mahloun
aedigix has joined #ipfs
flyingkiwi has joined #ipfs
ofdm has joined #ipfs
kpcyrd has joined #ipfs
rodarmor has joined #ipfs
Nycatelos has joined #ipfs
keroberos has joined #ipfs
athan has joined #ipfs
Akaibu has joined #ipfs
bsm117532 has joined #ipfs
Confiks has joined #ipfs
<PureTryOut[m]>
huh. with a database and all?
<lgierth>
not a traditional relational database
xelra has joined #ipfs
<PureTryOut[m]>
obviously I guess
<lgierth>
it's more like, every user is their own database
<PureTryOut[m]>
huh...
lassulus has joined #ipfs
tsetair has joined #ipfs
<lgierth>
fetches data from others as needed
<lgierth>
and there's pubsub for low level realtime messaging
<PureTryOut[m]>
but how exactly do applications "search" for info/data/files on the network?
<jbenet>
super cool
rovdyl has joined #ipfs
<jbenet>
kubuxu, flyingzumwalt: we may want to plan out the arc of the demo before re-recording it. like what to show, in what order, etc.
<flyingzumwalt>
agreed
<flyingzumwalt>
the cjdns point is a powerful one, but needs careful explanation for people to understand it
<PureTryOut[m]>
and in the case of OpenBazaar, will a seller still be found if he/she is offline and no potential buyer has found their sale yet?
<PureTryOut[m]>
cause from what I understand, data will only be replicated over the network as long as multiple other people "download" the data using IPFS right?
<jbenet>
flyingzumwalt: can i get a ping when you make ipfs/wikipedia so i can translate those issues ^ ? (i can also make it)
<PureTryOut[m]>
and OpenBazaar definitely looks good, but it seems more to be a replacement for Ebay/Craigslist than a proper webshop selling goods from their store
rovdyl has quit [Read error: Connection reset by peer]
<achin>
jbenet: you pinged me a day or two ago about the wikipedia issue on the archives repo. i'm around for about 20 minutes if you wanted to chat
rovdyl has joined #ipfs
galois_dmz has quit []
mahloun has quit [Ping timeout: 246 seconds]
brendyn has quit [Ping timeout: 268 seconds]
<jbenet>
hey achin -- yeah take a look at the allhands call today. you'll see something cool :)
<flyingzumwalt>
achin jbenet I'm almost done rendering the recording of the all-hands
<jbenet>
thanks flyingzumwalt! -- i guess we no longer auto-stream to YT, right? i recall that breaking at some point
<achin>
sweet. i'm about to disappear into 2 hours of meetings, but i'll for sure checkout the notes later today
<flyingzumwalt>
Yeah. I'm not sure why we stopped using it, but I've been recording on zoom. Might just be because only a few people have a setup that can host the streaming (kubuxu, dignifiedquire) and they can't always make it.
<achin>
is there some github issue to track the wikipedia-on-ipfs stuff? (i think the ipfs/archives one is a bit stale)
<flyingzumwalt>
I will be creating a repository for it
<achin>
\o/
<flyingzumwalt>
lgierth or kubuxu please pin QmTpfmaZhTATHWGMDXCEsb5KqAmgQgtcrdHeFXZf95Zuug 2017-05-01 12.04.14 IPFS All Hands Call
<Kubuxu>
pinning
rovdyl has quit [Read error: Connection reset by peer]
<flyingzumwalt>
kubuxu what's the peer id I should connect to -- I don't think my daemon is bootstrapping properly
<flyingzumwalt>
nevermind -- my internet connection had droped
ianopolous has quit [Read error: Connection reset by peer]
<achin>
flyingzumwalt: thanks. downloading it now, and will watch later today
<flyingzumwalt>
It's pretty awesome that I can give you that link before I've even "uploaded" the files to anywhere. You can stream it from my machine while kubuxu pins it. The link doesn't change after we've moved the data to a more stable location.
<achin>
<3 ipfs
Aranjedeath has joined #ipfs
jonnycrunch has quit [Quit: jonnycrunch]
<oed>
whyrusleeping: ok, tnx
athan has quit [Remote host closed the connection]
<whyrusleeping>
i might need to rebase it on latest master if you want to use that same node to view the wikipedia stuff though
<achin>
assuming no merges, i can rebase it locally
<achin>
after running this PR, should i just try to download something really large?
<whyrusleeping>
Yeah, it should reduce wasted bandwidth a good deal
<whyrusleeping>
it doesnt yet do better want sheduling
<whyrusleeping>
but that will come as a v2 PR
<achin>
any objective measurements or metrics i should take?
<flyingzumwalt>
kubuxu are you done pinning? I need to relocate but don't want to close my laptop until you're done grabbing a copy of the video
<whyrusleeping>
achin: watch your `ipfs stats bw` output
<Kubuxu>
yes, the transfer just finished
PureTryOut[m] has left #ipfs ["User left"]
<flyingzumwalt>
great
<whyrusleeping>
especially `ipfs stats bw -t /ipfs/bitswap/1.1.0`
<whyrusleeping>
I expect those outbound numbers to be significantly lower
<achin>
whyrusleeping: ok. give me 1ish day
<whyrusleeping>
SGTM
bsm117532 has quit [Killed (barjavel.freenode.net (Nickname regained by services))]
galois_dmz has quit []
mildred2 has joined #ipfs
mildred has joined #ipfs
maxlath has quit [Ping timeout: 260 seconds]
jonnycrunch has joined #ipfs
<achin>
whyrusleeping: trying to build 8ded923eceddb9046bebcd13b665840d978cf562 yields a build error: core/commands/bitswap.go:67: not enough arguments in call to bs.CancelWants
ZaZ has quit [Read error: Connection reset by peer]
<whyrusleeping>
achin: heh, sorry. Fixed
galois_dmz has joined #ipfs
<achin>
thanks
ashark has joined #ipfs
gde33 has quit [Read error: Connection reset by peer]
<achin>
i tried to rebase on top of master, but the result doens't build
<whyrusleeping>
Yeah, i noticed theres a merge conflict
<whyrusleeping>
i'll try and get to that today
FrankPetrilli has joined #ipfs
esph has joined #ipfs
FrankPetrilli is now known as Guest35394
c0dehero has joined #ipfs
jonnycrunch1 has joined #ipfs
<achin>
the merge conflict was just in some include, but even after solving that i saw some build problems
<achin>
but far from an expert i am
gde33 has joined #ipfs
SalanderLives has joined #ipfs
SalanderLives has quit [Client Quit]
SalanderLives has joined #ipfs
captain_morgan has quit [Remote host closed the connection]
captain_morgan has joined #ipfs
<achin>
hmmm, a crapload of too many open files in my ipfs daemon control
<whyrusleeping>
:/
maxlath has joined #ipfs
<achin>
(where "crapload" means like 200 messages per second)
<whyrusleeping>
>.>
<whyrusleeping>
what
<whyrusleeping>
constantly?
<achin>
yeah
<whyrusleeping>
can i get a short copy paste of some?
<whyrusleeping>
achin: what operations are you running?
warner` is now known as warner
<whyrusleeping>
was that a fetch?
<achin>
ipfs get
<achin>
of a large 60GB dataset
<achin>
my wantlist keeps on growing. currently 1013
<whyrusleeping>
hrm...
<achin>
i restarted the daemon, and while i'm still seeing the occational too many open files, the rate is much much lower
Guest83448 has joined #ipfs
<achin>
(wantlist length is now 2300)
cjb has joined #ipfs
cjb has quit [Remote host closed the connection]
cjb has joined #ipfs
john____ has quit [Ping timeout: 240 seconds]
maxlath has quit [Ping timeout: 240 seconds]
maxlath has joined #ipfs
<achin>
after interrupting the ipfs-get and waiting about 5 minutes, the length of the wantlist is unchanged (i would have expected it to go down slowly)
john2 has joined #ipfs
Guest83448 has quit [Ping timeout: 255 seconds]
OstlerDev has joined #ipfs
MDude has joined #ipfs
Foxcool has quit [Ping timeout: 245 seconds]
Guest20366 has quit [Ping timeout: 260 seconds]
wak-work has joined #ipfs
ianopolous has joined #ipfs
<whyrusleeping>
achin: are you running 0.4.8?
<whyrusleeping>
wantlists should clear entirely once the request that started them completes
<achin>
i'm running f39ac4ef14
<achin>
which identifies itself as "0.4.9-dev"
<whyrusleeping>
okay
<whyrusleeping>
achin: is this a public gateway that youre using?
<achin>
do you mean if my node configured as a public gateway? if so, then no
<whyrusleeping>
cool thanks, i'll get back to you in a bit
<achin>
sounds good
* achin
meanders back to $DAYJOB
keroberos has quit [Quit: reboob]
kerozene has joined #ipfs
<achin>
whyrusleeping: i'm happy to leave notes on the PR if that's easier for you
<emunand[m]>
didnt someone make a wiki software for ipfs? i think its listed in ipfs-awesome
<Kubuxu>
article directory is moved to be named wiki, we plan on adding index page and shifting articles to their own directories so they don't have .html extension
<whyrusleeping>
Yeah, ward cunningham hangs out in this channel occasionally
<whyrusleeping>
i think he was working on federated wiki stuff
<achin>
flyingzumwalt: cool. what did the directory sharding stuff get added?
<achin>
s/what/when/
<whyrusleeping>
achin: 0.4.7 i think
<achin>
oh neat, dignifiedquire is using a fork on my zim extractor code
<Kubuxu>
flyingzumwalt: can you fix quotes in this comment, google docs added special backticks
<flyingzumwalt>
<3
<Kubuxu>
yeah, we based it off your work
<Kubuxu>
achin: before I forget, mind adding license to it?
<Kubuxu>
MIT would be the best but if you prefer something use that.
<achin>
nope. the standard rust license is both Apache and MIT, so i'll go with that
Mateon1 has quit [Remote host closed the connection]
<dignifiedquire>
achin: yeah thanks for that code :)
<dignifiedquire>
I went ahead and updated some things over the weekend :)
<achin>
glad it was useful!
Encrypt has joined #ipfs
<dignifiedquire>
very, the basic decoding stuff is all working great
Akaibu has quit [Quit: Connection closed for inactivity]
fleeky has joined #ipfs
ashark has quit [Ping timeout: 240 seconds]
Encrypt has quit [Quit: Quit]
Encrypt has joined #ipfs
ashark has joined #ipfs
fnlintred887 has joined #ipfs
unpermeable has quit [Ping timeout: 240 seconds]
digitalism has joined #ipfs
nekomune has joined #ipfs
ashark has quit [Ping timeout: 260 seconds]
ianopolous has quit [Read error: Connection reset by peer]
ianopolous has joined #ipfs
Encrypt has quit [Quit: Quit]
jonnycrunch has quit [Quit: leaving]
jonnycrunch1 is now known as jonnycrunch
soesoe has quit [Ping timeout: 240 seconds]
jonnycrunch has quit [Quit: jonnycrunch]
jonnycrunch1 has joined #ipfs
ashark has joined #ipfs
jonnycrunch1 has quit [Quit: jonnycrunch1]
btmsn has quit [Ping timeout: 260 seconds]
kook_ has quit [Ping timeout: 260 seconds]
anewuser has quit [Quit: anewuser]
<Atrus[m]>
I haven't followed the sharding stuff too much. Is the Wikipedia mirror a nice test of it?
rendar has quit [Ping timeout: 240 seconds]
<lgierth>
because it's one huge directory with millions of files in it
<lgierth>
and file chunking only applies to file content, not a directory's links to its files
<whyrusleeping>
Atrus[m]: yeah, the wikipedia stuff is a nice test
<lgierth>
so you'd previously end up with huuuge single directory objects
<whyrusleeping>
i think we have a single directory with something like 17 million items in it
<lgierth>
now just need an improvement to the standard directory listing
<whyrusleeping>
yeah, once we get that to work in a streaming manner, this will be really slick
<lgierth>
coreapi.Unixfs.LsAsync()
galois_d_ has joined #ipfs
Caterpillar has quit [Quit: You were not made to live as brutes, but to follow virtue and knowledge.]
galois_dmz has quit [Ping timeout: 245 seconds]
rendar has joined #ipfs
seagreen has joined #ipfs
soesoe has joined #ipfs
hacman has joined #ipfs
chungy has joined #ipfs
rovdyl has quit [Ping timeout: 240 seconds]
aeternity has quit [Ping timeout: 245 seconds]
aeternity has joined #ipfs
rovdyl has joined #ipfs
jkilpatr has quit [Ping timeout: 268 seconds]
rendar has quit [Quit: std::lower_bound + std::less_equal *works* with a vector without duplicates!]
ashark has quit [Ping timeout: 255 seconds]
chris613 has joined #ipfs
galois_d_ has quit [Remote host closed the connection]
galois_dmz has joined #ipfs
robattila256 has joined #ipfs
shizy has quit [Ping timeout: 268 seconds]
jkilpatr has joined #ipfs
tocky has joined #ipfs
The_8472 has quit [Ping timeout: 260 seconds]
Boomerang has quit [Remote host closed the connection]
conway has quit [Ping timeout: 246 seconds]
espadrine has quit [Ping timeout: 260 seconds]
The_8472 has joined #ipfs
akkad_ is now known as akkad
tocky has quit [Quit: My MacBook Air has gone to sleep. ZZZzzz…]
<kyledrake>
I think you could roll your own sharding too if it's just to split up the directory tree. I ended up doing this for site storage.
<kyledrake>
CephFS doesn't have dir sharding ready yet so I was kindof forced into it...
tocky has joined #ipfs
<Kubuxu>
And add process of English wikipedia dump was started.
<emunand[m]>
can't wait to see it
soesoe has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
<lgierth>
kyledrake: ipfs's sharding should be good to try out now :)
<kyledrake>
I'll give it a go when it's in a released version
<kyledrake>
Throw 140k sites into it and watch the sparks fly
maxlath has quit [Quit: maxlath]
tocky has quit [Quit: My MacBook Air has gone to sleep. ZZZzzz…]
<lgierth>
hah cool
dimitarvp has quit [Quit: Bye]
<Kubuxu>
It took 5min to just ls this directory
<Kubuxu>
ipfs haven't started adding it yet
<Kubuxu>
it still tries to read the listing
jonnycrunch has joined #ipfs
jonnycrunch has quit [Client Quit]
<Kubuxu>
ipfs add command instance takes 1.3GiB of ram
<Kubuxu>
I will create issue about it tomorrow
<Kubuxu>
we should improve this a bit
<Kubuxu>
12999178 articles with redirects
<Kubuxu>
just under 13M
asyncsec has joined #ipfs
<lgierth>
yeah we need async reading and returning of the links
<lgierth>
s/async/streaming/
infinity0_ has joined #ipfs
infinity0_ has quit [Changing host]
infinity0 has joined #ipfs
infinity0 has quit [Killed (hitchcock.freenode.net (Nickname regained by services))]
infinity0 has quit [Remote host closed the connection]