pawnbox_ has quit [Remote host closed the connection]
shellac has quit [Quit: Computer has gone to sleep.]
tcrawley-away is now known as tcrawley
shellac has joined #jruby
pawnbox has joined #jruby
tcrawley is now known as tcrawley-away
drbobbeaty has joined #jruby
vtunka has joined #jruby
etehtsea has joined #jruby
shellac has quit [Quit: Computer has gone to sleep.]
drbobbeaty has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
etehtsea has quit [Ping timeout: 272 seconds]
tcrawley-away is now known as tcrawley
yfeldblum has quit [Ping timeout: 250 seconds]
tcrawley is now known as tcrawley-away
<GitHub174>
[jruby] chrisseaton pushed 1 new commit to master: https://git.io/voi4q
<GitHub174>
jruby/master 99de0da Chris Seaton: [Truffle] Tag spec that fails in CI.
<GitHub193>
[jruby] brometeo opened issue #3975: Running with flag --enable-frozen-string-literal crashes stdlib/jar_dependencies.rb https://git.io/voi43
skade has quit [Quit: Computer has gone to sleep.]
sebstrax has joined #jruby
rsim has joined #jruby
drbobbeaty has joined #jruby
tcrawley-away is now known as tcrawley
skade has joined #jruby
vtunka has quit [Quit: Leaving]
tcrawley is now known as tcrawley-away
sandelius has joined #jruby
skade has quit [Quit: Computer has gone to sleep.]
skade has joined #jruby
pawnbox has quit [Remote host closed the connection]
<headius>
chrisseaton: that's how we found locking bottleneck in the sqlite jdbc driver...sped up cremes' app 7x
<headius>
and celluloid I gather?
Aethenelle has joined #jruby
thedarkone2 has joined #jruby
<headius>
cremes: do you know what takes all that time on MRI? Might be illustrative too
<headius>
I mean I know we're awesome, but that seems a bit too awesome for something DB-heavy
<headius>
oh I guess we do have threading edge
<headius>
8-way machine?
<enebo>
yeah 8-way
<enebo>
parallel speedup looks reasonable
<enebo>
I wonder too though whether we are doing something poorly still though
<kares>
what's an 8-way machine :) ?
<enebo>
kares 8 real processors
<enebo>
not hyperthreads
<kares>
aaah - the slangs!
<enebo>
since cremes said he was on 8 core intel he probably does have 16 locical cores with hyperthreading
<enebo>
I have never really seen those threads do much though
<enebo>
err fake cores
<enebo>
:)
<chrisseaton>
They help a lot if you are going far for memory, otherwise they don't help as much
<kares>
yes exactly my observations
<enebo>
chrisseaton: yeah that makes sense
<chrisseaton>
So, doing matrix multiply in C, they don't help, but chasing boxed values in Ruby, they help tons
<enebo>
chrisseaton: I guess I should say I do not see it helping in JRuby
<enebo>
chrisseaton: observationally probably because we are reading so much memory things are never getting any locality
<enebo>
chrisseaton: whereas I can see it being easier to see in a more controlled library in C
<chrisseaton>
But that sounds like it should help them - you don't have any locality, so might as well run another thread in the gap it takes to fetch memory
<enebo>
chrisseaton: then perhaps cremes should try to up his parallelism a little bit and see if there is a pay out
<enebo>
although I suspect he has already
<chrisseaton>
More threads does mean more contention on VM structures though
<enebo>
and JRuby and JVM do have other active threads as well
<chrisseaton>
Does anyone know if we can read whether or not Maven is in quiet mode, inside a pom.rb?
<headius>
there's probably a property somewhere
<cremes>
chrisseaton: the workload is pretty simple. Once per month I batch process about 25 CSV files. First step is reading in the data and building a giant hash using the first field as a key. Second step I write out a separate file for each key (I do this for IO parallelism and to ease troublehsooting when something goes wrong). Third, I CSV.parse each of these new files (:converters => :all) and write the rows to individual databases (one per
<cremes>
all of this uses celluloid and futures so I’ve written it using the usual syncrhonous/imperative style and leave the fiber/threading stuff to celluloid to manage.
<chrisseaton>
Hmmm sounds like something I could pull benchmark kernels out from, until Truffle can run the sqlite C extension - is it open source?
<cremes>
I typically see step one run at 600-700% on my 8-core VM. Step two runs at 200-300%. Step 3 runs at 200-300%.
<cremes>
Step 1 takes approximately 5 minutes (wall clock). Step 2 takes about 55m. Step 3 takes 5m.
<cremes>
chrisseaton: I can send you a zip file of the source along with a zip of some sample data. The sample data can’t be distributed but the source is nothing special.
<chrisseaton>
yes please! chris.seaton@oracle.com
<chrisseaton>
can't promise I'll ever get around to it, but I'll try
<headius>
chrisseaton: cremes has filed a few perf issues that are independent of sqlite also, like the csv converters/ignored exceptions thing
<headius>
that prompted the rescue nil optimization
<chrisseaton>
Yeah, I copied that at the time
<cremes>
chrisseaton: I’ll send you an email today sometime. The data file will be a link to dropbox since the data is 500MB compressed.
<chrisseaton>
thanks
<headius>
cremes: if you could copy enebo and I too that would be great
<cremes>
sure
<headius>
I'm sure there's more we can improve in JRuby classic
<cremes>
hmmm, if I’m sending you all this stuff then maybe you can do the profiling runs…
<headius>
I can, but it will be different than an 8-way
enebo_ has joined #jruby
enebo has quit [Ping timeout: 276 seconds]
enebo_ is now known as enebo
<headius>
my four year old, four-way i7 is going to behave differently than whatever behemoth you're running on
<headius>
I need a PAAS that's as easy to use as AWS but actually gives me bare metal
<cremes>
I’m on a intel core i7 5960x which has 8 real cores. I run everything under VMWare workstation, so my OSX instance where I do most of this work has 8 cores allocated. My other small VMs all have 1 core allocated. Yes, I overallocated cores but vmware is pretty smart and perf is great.
<enebo>
cremes: does this do much logging?
<enebo>
cremes: we still have non-native date/format
<enebo>
although I think eregon did optimize some part of it?
<cremes>
It prints dots and ‘q’s and ‘w’s as it progresses. not much text. hard to have usable logging to stdout with parallelism :)
<headius>
push messages to a queue and have a single logging thread
<enebo>
cremes: well I meant to a log file
<headius>
but you wouldn't want to be logging or recording metrics all the time at runtime anyway
<headius>
unless you really want your memory/hd cache to thrash forever :-D
<cremes>
enebo: nope, no logging.
<enebo>
cremes: ok
<cremes>
you’ll see how it works soon… it might not be the best design but it’s far superior to my prior versions that used my hand-rolled threading with mutexes. celluloid majorly cleaned it up.
<cremes>
and the ‘writing a bunch of files out only to reread them’ step might seem superfluous but I have found it useful for debugging problems particuarly with corrupted CSV files and things. A few extra steps and more work saves lots of debugging time.
<cremes>
give me 20m and I’ll have this sent to your emails.
<headius>
yeah that seems fine to me
<cremes>
enebo: private msg me your email
<cremes>
headius: do the same, pls
<headius>
you might do better writing to some structured form other than csv, so you don't have to pay as much re-parsing it
<headius>
msgpack or protobuf or something like that
<headius>
maintain some structure on the way out so you don't have to rebuild it again
<headius>
cremes: headius@headius.com always
<headius>
I get so much spam after 20 years of having the same email it really doesn't matter anymore
<cremes>
headius: your idea is a good one. if the code didn’t already pretty much work, I’d probably make that change. I’ll add it to a ‘to-do’
<headius>
sure, one hour per month is already pretty lean
<headius>
making that 30 min wouldn't do a great deal
<cremes>
yep
<cremes>
nope
<cremes>
ok, got this stuff packaged up. uploading to drop box… expect emails shortly.
<headius>
cool
<headius>
so yeah, exception fixes and sqlite-jdbc fixes came out of this...great wins both
camlow32_ has joined #jruby
camlow325 has quit [Ping timeout: 258 seconds]
shellac has quit [Quit: Computer has gone to sleep.]
<cremes>
headius, enebo, chrisseaton: check your email. Should have 1 tarball attached and a link to the dropbox datafile.
<headius>
yup just saw it
<headius>
thank you
<enebo>
cremes: thaniks
<headius>
dont know if we will find anyting but it has been a good load to test
<cremes>
I am just blown away by that. I really don’t do anything special. These are really just glorified perl scripts for doing text processing and some simple database insertion.
<headius>
right, that tells me we may already have hit the big items
<headius>
especially given 62 min to do all that for a 500MB data set
<headius>
but we shall see :-)
<cremes>
good. I’ll still try to rerun here with the profiler and send you the data. I’ll do 9k & 7.x
<headius>
there's always opportunities
<headius>
great, thanks!
<headius>
that wiki page is a WIP but it's the steps I usually follow to find bottlenecks
<headius>
the allocation profiling is really expensive, so that part you'd want to break up into the three phases and measure separately
<headius>
expensive as in 100x slower but you only want to run for long enough to get e.g. one iteration of allocations...we would want relative numbers, not total numbers
<enebo>
cremes: I see you create a StringIO in parse_json and then iterate over chars
<headius>
enebo: parsing like that is another point for doing some specialized String instances that have no byte[]
<cremes>
enebo: that’s a one-off script for making rubinius’ streamed profiler output easier to read.
<enebo>
cremes: is that to avoid block overhead with each_char?
<headius>
e.g. pack a couple bytes into header and not create ByteList or byte[] at all
<enebo>
haha ok
<enebo>
cremes: so not part of the run
<cremes>
yeah, wrote that in 10m. not an exemplar of good code :)
<enebo>
cremes: I was just skimming files
<cremes>
enebo: the meat of it is in lib/split_options_file.rb
<cremes>
could use some more refactoring but it works well enough for a utility script.
<enebo>
cremes: I agree with headius that msgpack or some other protocol might give you a little more over using csv as intermediary format
<cremes>
enebo: sure, but it’s super useful to be able to open the file in excel and look at it. that’s why I kept it in csv.
<headius>
write it out as xls using poi :-D
<cremes>
it’s about my convenience, not the program’s convenience. :)
<headius>
I kid...that would probably be a lot more overhead than csv
<enebo>
cremes: yeah I guess if you are happy with speed now too
<cremes>
so, can you guys allocate 40GB heaps? :)
<enebo>
not without swap :)
<headius>
hah
<headius>
yeah, there's another limiting factor for us profiling it :-)
<cremes>
this might be a good exercise to figure out how to slim things down. JRuby takes about 80% more heap than MRI for this specific work.
<headius>
alloc profile may show some places to trim
<headius>
I have a branch for small arrays that cuts 1 and 2-element array size 60%
<headius>
lots of libraries use small arrays as intermediate data
<headius>
a heap dump of the app at each phase would be useful too... jmap command can pull a heap dump from any running process
<enebo>
uniq works here because the dates are generally already sorted right?
<headius>
if that's a big data structure, .lazy could be a quick win
<headius>
uniq and sort and map and the final map of dates would not create intermediate arrays
<enebo>
I am guessing that is not a big list based on how it is used
<enebo>
I should haven oticed the puts of the date ranges
<headius>
yeah, .lazy is an app optimization also...I'm more interested in JRuby stuff :-)
<headius>
ok, back to vacay for me
<headius>
ttfn
<cremes>
enebo: yes, the @dates array will typically only have one entry. I left the logic there in case the input/source data changes and multiple dates show up per file.
<cremes>
it’s probably a YAGNI
<enebo>
cremes: yeah I figured it you did a puts on the data it would not be big
<enebo>
cremes: I just missed seeing that puts at first :)
<cremes>
headius: I hope your vacay includes a bevy of excellent brews.
<headius>
that's the plan today...we're up at Union Pier about an hour north of you on eastern shore of Lake MI
<cremes>
enebo: ping me if you need me. going for lunch.
<headius>
bit chilly today for beach so it's brewery hopping
<cremes>
nice
<enebo>
cremes: yeah so far I am just coming up with ideas for tiny tweaks which is not really a JRuby thing
<enebo>
cremes: like sprintff for the file_date making
camlow32_ has quit [Remote host closed the connection]
<GitHub114>
[jruby] chrisseaton pushed 1 new commit to master: https://git.io/voPBM
<GitHub114>
jruby/master a37c753 Chris Seaton: Make the JRuby build system optionally more quiet.
camlow325 has joined #jruby
camlow325 has quit [Remote host closed the connection]
norc has joined #jruby
<headius>
cremes: oh hey, I know you're at lunch but I'd love to hear how the celluloid optimization is going for gsoc
<headius>
that would be really easy to profile on JRuby and find some good wins
<headius>
and you'd be running with JIT so it would actually reflect real workloads
camlow325 has joined #jruby
<headius>
that could be good for jruby+truffle too since it's all ruby concurrency stuff
camlow325 has quit [Read error: Connection reset by peer]
camlow32_ has joined #jruby
bbrowning is now known as bbrowning_away
skade has quit [Quit: Computer has gone to sleep.]
rsim has quit [Quit: Leaving.]
shellac has joined #jruby
thedarkone2 has quit [Quit: thedarkone2]
camlow32_ has quit [Remote host closed the connection]
camlow325 has joined #jruby
camlow32_ has joined #jruby
camlow32_ has quit [Remote host closed the connection]
camlow32_ has joined #jruby
camlow325 has quit [Ping timeout: 252 seconds]
camlow32_ has quit [Remote host closed the connection]
camlow325 has joined #jruby
<GitHub99>
[jruby] nirvdrum pushed 3 new commits to master: https://git.io/voPro
<GitHub99>
jruby/master a8e2a80 Kevin Menard: [Truffle] Introduced an EncodingManager to better collect encoding state.
<GitHub99>
jruby/master 5ee58e6 Kevin Menard: [Truffle] Added some delegates to EncodingManager to limit direct access to the JRuby runtime.
<GitHub99>
jruby/master d5c6b40 Kevin Menard: [Truffle] Faster lookup from jcodings Encoding -> Ruby encoding (2nd attempt).
camlow325 has quit [Remote host closed the connection]
camlow325 has joined #jruby
shellac has quit [Quit: Computer has gone to sleep.]
rcvalle has joined #jruby
subbu is now known as subbu|afk
camlow325 has quit [Remote host closed the connection]
camlow325 has joined #jruby
bbrowning_away is now known as bbrowning
<cremes>
enebo: I was using sprintf to make the date, but it turned out to be a spinlock bottleneck in rubinius so I rewrote it to its current form.
<headius>
ah
<enebo>
cremes: yeah I tried it
<headius>
spinlock in sprintf?
<headius>
whatever for?
<enebo>
cremes: sprintf is slower on MRI and JRuby too
<cremes>
headius: the student is doing nice work fleshing out the specs. no real perf improvements yet
<enebo>
cremes: not due to locking
<headius>
ahh, ok
<headius>
well let us know if jruby metrics would be useful
<enebo>
cremes: slower than what you wrote
<enebo>
cremes: our margin is quite a bit closer between the two than MRI though
<cremes>
headius: in rbx the format string is added to a LRUCache protected by a mutex. all threads trying to print using that format cause major contention on that lock. Only see it with lots of threads. :)
<enebo>
locks are funny that way
<cremes>
I have an easy fix in mind for rbx but in the menatime manually building the string is the easiest play.
<enebo>
cremes: and is it about the same speed before/after with a single thread?
<enebo>
cremes: you may still be faster with this double split + couple of size checks
<headius>
cremes: weird
<headius>
seems like the locking cache would be more expensive than reparsing, but I don't know how expensive sprintf parsing is in rbx
<cremes>
enebo: I don’t remember if it’s same or faster with single thread. not my use-case :)
<cremes>
headius: it’s a case of premature optimization. heh.
<enebo>
cremes: no but I am just wondering if it optimized as well
<headius>
fair enough
<enebo>
cremes: for us we did not but we did a better job than MRI
<headius>
we don't cache anything in sprintf
<headius>
might make it faster than what you have if we did, I dunno
camlow325 has quit [Remote host closed the connection]
<cremes>
enebo, headius: remember that you are going to run into lots of Ruby code that could be written better or smarter. You need to make dumb code (even mine) run fast regardless. Sucks to be you! ;)
<headius>
indeed
<headius>
enebo: sure, that's similar to truffle approach anyway
<headius>
more specialization at the site once we know what it is
<headius>
in truffle it just splices their nodes in and runs with it
<enebo>
headius: yeah to some degree
<enebo>
headius: I think of it more as bytecode specialization for a site
<enebo>
headius: less partial evaluation and more just like array copy stubs in hotspot
<headius>
yeah
<headius>
I still have faith that indy + more IR could compete with IR inlining if we can wire it up right
<enebo>
headius: we already have annotations on native methods
<headius>
I keep meaning to play with an indy call site that forces IR to re-emit just for that site
<headius>
so that polymorphic code looks monomorphic and hotspot inlines right through
<enebo>
headius: so I think it would be that those methods would have a emitSpecialized() or something
<headius>
brewery-hopping time...I'll be on IM
<headius>
ttfn
<enebo>
cya
<GitHub80>
[jruby] nirvdrum pushed 2 new commits to master: https://git.io/voP1s
<GitHub80>
jruby/master 9a0e74d Kevin Menard: [Truffle] Minor clean-up.
<GitHub80>
jruby/master 134293f Kevin Menard: [Truffle] Removed some dead code.
<GitHub132>
[jruby] chrisseaton pushed 2 new commits to master: https://git.io/voP1o
<GitHub132>
jruby/master c68c0b4 Chris Seaton: Let's just not support Truffle in the Shell launcher.
<GitHub132>
jruby/master 47120e1 Chris Seaton: Rename -X-T to be -Xclassic...
<GitHub141>
[jruby] chrisseaton pushed 1 new commit to master: https://git.io/voP1A
<GitHub141>
jruby/master 71b1e09 Chris Seaton: Debug code!
<GitHub110>
[jruby] chrisseaton pushed 1 new commit to truffle-head: https://git.io/voPMn
<GitHub110>
jruby/truffle-head b965b0c Chris Seaton: Merge branch 'master' into truffle-head...