<headius>
chrisseaton: frustrating responses from MRI but not surprising :-(
<headius>
they don't want to implement a modern VM for Ruby...they want to implement a unicorn VM for what Ruby might be some day maybe
<headius>
like all this discussion about whether shared memory is even right for Ruby...yeah, basically just shoot the whole discussion in the head right there
<headius>
maybe Ruby should just remove threads and you can stay in the one-core ghetto with JS and Python
rcvalle has quit [Quit: rcvalle]
pawnbox has quit [Ping timeout: 250 seconds]
<headius>
from ko1: "At least I don't believe I can implement it.
<headius>
(Of course, it is because of my low skill. Somebody should be able to implement it)"
<headius>
many decisions get made for all of Ruby solely because MRI can't do something and nobody knows how to make it do that thing
<headius>
that's why we're the only Ruby to have defined a memory model
<headius>
chrisseaton: I'm sorry about sounding too negative...there's just a lot of open questions that get handwaved with "we'll get to that later"
<chrisseaton>
well it's a research project, if we knew exactly what we'll be doing, it wouldn't be worth doing
<headius>
indeed...but a solution that's not implemented is not a solution
<headius>
it blunts conversation to try to claim it will be without any evidence
<headius>
anyway...we all have our crosses to bear
<headius>
chrisseaton: what's the perf like for sulong stuff right now?
<headius>
I've been worried that LLVM IR does not convey enough information for Truffle to really eat it up
<headius>
not enough semantic/structural information once it has been reduced down to IR, etc
<headius>
same goes for bytecode...it's super generic
<chrisseaton>
But byte code includes precisely all the information that C2 has to work with
<headius>
(and I have to say being able to run LLVM IR on a JVM *at all* is miraculous...so take these questions as real interest)
<chrisseaton>
And LLVM IR includes precisely all the information that the compiler backend has to work with
<chrisseaton>
I can't find public benchmark results right now
<chrisseaton>
But glad to hear you're positive about sulong as well!
<headius>
yes, but LLVM IR generally works with unmanaged memory
<chrisseaton>
So can Sulong
<headius>
I guess I'm confused how you can do MRI C exts if you have to copy in and out of unmanaged memory then
<headius>
I mean, do them efficiently
<headius>
e.g. rbx's copying in and out all thr
<chrisseaton>
You can have both managed and unmanaged memory, no copying needed
<headius>
I gues I thought Sulong translated everything into a manged memory space
<headius>
like nestedVM or similar
<chrisseaton>
We showed it worked and was fast on real gems in the TruffleC papers - that did the same thing
<headius>
but that was a C interpreter wasn't it?
<chrisseaton>
The memory was implemented in the same way
<chrisseaton>
Just squint at LLVM IR and you see a C program
<headius>
well, you see an assembly program
<chrisseaton>
Unstructured control flow is the main difference
<headius>
that makes some sense then...it's more along the lines of a different LLVM that can run alongside JVM code than an LLVM for JVM
<headius>
hell, why doesn't it just translate LLVM IR directly to Graal? Would that be much of a leap?
<chrisseaton>
Because we want caching and ability to handle foreign objects
<headius>
makes sense
<chrisseaton>
LLVM thinks it has a char* but really it's a Ruby object
<headius>
the managed/unmanaged interaction
<headius>
and dynamically optimizing calls that can't be statically determined at compile time
<headius>
chrisseaton: I think what you and the other truffle folks have managed to do is really amazing...it drives us to try to do a better job with our plain-JVM runtime
<enebo>
chrisseaton: good lord Chris how late to you stay up?
<headius>
thanks for pointing out the memory model discussion...I'll try to get through it this evening
<headius>
petr has added a lot to this doc since I last looked
<headius>
chrisseaton: almost all the responses from MRI folks on this thread completely dodge the question
<chrisseaton>
we have an upcoming paper on an idea for concurrency that may be more what they want, but it's not public yet
<headius>
chrisseaton: where is this "Red" proposal written up
<headius>
I see it quoted but I can't find an original docuent
<headius>
"(A) I understand the need to add such abstraction (actors, channels, other?)
<headius>
to Ruby to enable Ruby users to build concurrent applications with ease. For
<headius>
future reference let's call the one future abstraction Red. The Red would then
<headius>
have well documented and defined behaviour in concurrent and parallel execution
<headius>
(This is what I think you are referring to). However providing just one
<headius>
abstraction in standard library (and deprecating Threads) will hurt usability
<headius>
I don't see how shared nothing model is a solution to anything
<headius>
it's no better than process model we have today
<chrisseaton>
well you can pretend to share nothing, but actually share a lot
<enebo>
headius: sounds like ko1 wrote it
<headius>
enebo: I don't know...Petr does say "using block quotes for structure" above it
<headius>
I couldn't find key sentences from that stuff anywhere but on this bug
camlow32_ has quit []
<enebo>
seems like an out of band thing included
<headius>
chrisseaton: so bottom line here seems to be they're not even interested in discussing this before ruby 3
<enebo>
perhaps matz has now branded his idea and petr knows the term
<headius>
or at least they're using the possibility that ruby 3 will have a different concurrency model as an excuse to not do this now
<headius>
MAY have a different concurrency model, maybe, someday
<enebo>
So problem is Matz dislikes threads
<enebo>
and whatever this model they come up with will be an attempt at having users write to this new abstraction which buries any threads into the C implementation
<headius>
we have threads with a bad model...we need a better model...matz doesn't like threads so we don't discuss fixing the model
<enebo>
I think matz is hoping threads will be deprecated :)
<headius>
yes, that's about what it sounds like after reading through
<enebo>
but not supported any better than now
<headius>
one true concurrency model is a hell of a rabbit hole
<enebo>
I think Petr’s point that subsets of things will not be possible in Ruby naturally is true
<headius>
for sure
<enebo>
but I do not think Matz has ever wanted his language to do all things well
<enebo>
if he did we would have single word math and shit :)
e_dub has joined #jruby
<enebo>
anyways this is too much for me tonight
<enebo>
night guys
<headius>
ttfn
enebo has quit [Quit: enebo]
<thedarkone2>
headius: yeah, MRI's response (in that ticket) is somewhat disappointing, and "red model" is just Petr's definition as he tries to come with a structured argument for Koichi
<thedarkone2>
since he brought up Go, we should point out that Go does have a memory model and corresponding concurrency primitives… you can't implement concurrent hash with channels...
<headius>
thedarkone2: yeah am I right that MRI seems to be dodging this issue by handwaving over some completely new, as-yet-unspecified concurrency model?
<thedarkone2>
I feel he just simply thinks threads/shared memory is just a dead end (as in too hard to program in)
<chrisseaton>
thedarkone2: 'you can't implement concurrent hash with channels' channels are a fundamental primitive of concurrency - you can implement anything using channels
<chrisseaton>
won't be lock-free though, which is probably what you mean
<thedarkone2>
chrisseaton: I meant lock free/concurrent
<chrisseaton>
your processor bus is of course a channel
<chrisseaton>
right I'm off
<thedarkone2>
headius: and doesn't want to add anything to Ruby that would encourage shared memory model...
chrisseaton has left #jruby [#jruby]
<headius>
thedarkone2: unless that comes along with a new C API that isolates memory across threads, I don't see it happening
<thedarkone2>
yea, they would also have to drop Threads and most of the existing Ruby code...
<thedarkone2>
as long as GVL stays areound, there is also almost no additional engineering effort involved implementing memory model & cas primitives...
<thedarkone2>
if they get rid of GVL, then yes, memory model & cas will require some effort to do, but it should take something 1% of the work required to get rid of GVL...
<headius>
thedarkone2: I will try to comment later that deferring this discussion until some future model is not reasonable
<headius>
ruby 3 isn't even likely to be available before 2020
<thedarkone2>
on the other hand, while MRI is GVL-ed I don't think, conc-ruby really needs MRI to cooperate… (as long as JRuby, Truffle implement the necessary primitives)
norc_ has joined #jruby
<headius>
yeah kinda back to the situation that we need an answer today, for Ruby 2.x
<headius>
if Ruby 3.x wants to break every piece of threaded code out there, it's a separate discussion
<thedarkone2>
I also wouldn't rule out that this is just lang barrier issue… we might be just talking past each other..
<headius>
yeah that's a hard problem :-(
<headius>
petr has put together some great documentation but it's also pretty dense English prose
norc has quit [Ping timeout: 264 seconds]
<thedarkone2>
yeah, Koichi must have spent hours on that ticket...
<GitHub70>
jruby/master 46d85a2 monkstone: crude fix for range with different numeric types
<GitHub70>
jruby/master 8b1671a monkstone: remove wrong equality test
<GitHub70>
jruby/master aa7f3ed monkstone: @enebo this seems to do it, will add test later
<GitHub125>
[jruby] kares closed pull request #3960: Fix Range#eql? checks that start and end of range are same using MethodNames.EQL (master...master) https://git.io/voONH
<GitHub23>
[jruby] kares closed issue #3956: Range#eql? returns incorrect result for float vs int https://git.io/voIZs
<GitHub59>
jruby/master b338524 Petr Chalupa: [Truffle] add actionpack to CI
<GitHub59>
jruby/master 79bb9ce Petr Chalupa: [Truffle] jtr: runnner correct ruby bin in setup before/after steps
<GitHub59>
jruby/master 864e2eb Petr Chalupa: [Truffle] jtr: fix print_cmd for [hash_env, string] case
pawnbox has quit [Remote host closed the connection]
pawnbox has joined #jruby
pawnbox has quit [Remote host closed the connection]
pawnbox has joined #jruby
e_dub has joined #jruby
Aethenelle has quit [Quit: Aethenelle]
<kares_>
at last - Bundler reached a state when a single Gemfile.lock is no longer shareable between MRI and JRuby ;(
<kares_>
Your Gemfile.lock is corrupt. The following gem is missing from the DEPENDENCIES section: 'ffi'
vtunka has joined #jruby
dsc has quit [Quit: This computer has gone to sleep]
Aethenelle has joined #jruby
<GitHub102>
[jruby] lxp opened pull request #3965: [Truffle] Extend openssl-stubs for Active Merchant (master...extend-openssl-stubs) https://git.io/volMb
Aethenelle has quit [Ping timeout: 240 seconds]
enebo has joined #jruby
pawnbox has quit [Remote host closed the connection]
Aethenelle has joined #jruby
pawnbox has joined #jruby
dsc has joined #jruby
pawnbox has quit [Remote host closed the connection]
pawnbox has joined #jruby
pawnbox has quit [Remote host closed the connection]
<headius>
kares_: there's a bug for that and I strongly encourage you to make noise about it
<headius>
at least one user has had to monkey-patch Bundler to use separate lockfiles for separate platforms
<headius>
and I had a report yesterday about this too
<headius>
it can't even handle multiple platforms of the same gem, but in this guy's case it was that pry's -java gem depends on spoon and the MRI version doesn't
<kares_>
headius: too deep into the issue stack - stumbled upon JRuby incompatibility which I will deal first
<kares_>
but they (Bundler) "officially" do not support JRuby ;( ... at least they do not want to run suite against JRuby
<kares_>
headius: did you have success reporting issues there?
<headius>
kares_: well it won't matter if we get enough people to complain...it also affects MRI ext gems on Linux versus Windows
<headius>
I also told them about the simple change to have separate lockfiles per platform
<kares_>
that sounds like a non backwards compatible change
<headius>
well, it's broken as it is
<dsc>
When a .jar file is required, I assume that's done with a custom ClassLoader ?
<kares_>
headius: what did you propose?
<kares_>
dsc: yy
<headius>
Gemfile.jruby.lock with fallback to Gemfile.ruby.lock or Gemfile.lock
<headius>
so if it's not there it uses the default name
<kares_>
heh thought so :) thx
<headius>
the other fix is big...Gemfile.lock needs to be restructured in a backward compat way to support multiple platforms
<headius>
or at least it needs to handle old and new structure for a while
<kares_>
going to try to extract a simple repro and report
chrisseaton has joined #jruby
<dsc>
Is there any way to see (log) every .jar loaded?
<headius>
kares_: let me find the bug I commented on
<headius>
there's an existing one for the platform issues
<headius>
kares_: that's interesting about bundler...where did you see that?
<kares_>
thanks - subscribed. was wondering if they still use JIRA
<kares_>
headius: trying to reproduce issue I've run into I started paying attention what's going on when I switch rubies compared to different versions
<kares_>
its not in 1.12.4 but is in 1.12.5 so we should start recommending that as a first resolution to JRuby users
<kares_>
still wonder if I manage to reproduce the issue as isolating it turned out difficult
<headius>
I was able to reproduce with pry, because it has different deps on JRuby
<headius>
worth trying with 1.12.5 I guess
<kares_>
headius: it should work
<headius>
testing now
skade has joined #jruby
<chrisseaton>
JRuby is 10x slower when I put my benchmark in a file that is loaded vs the main file. The benchmark is in a block, like in bips, that I .call in a loop. This is because the main file is force compiled, and you don't otherwise compile blocks without the enclosing method, is that right?
<kares_>
so there's a bug with Bundler - it works on resolving platform gems but it seems to be getting into an invalid state when there are multiple
<kares_>
think they should be able to fix that
<chrisseaton>
-X+CIR makes things slower, is that the right flag to get around this?
pil-afk is now known as pilhuhn
lanceball is now known as lance|afk
dsc has quit [Quit: This computer has gone to sleep]
prasunanand has quit [Ping timeout: 250 seconds]
prasunanand has joined #jruby
yfeldblum has joined #jruby
<headius>
chrisseaton: we do compile blocks now
<headius>
I'm surprised it wouldn't generally come out the same...usually it does for me
<headius>
what's the bench?
<chrisseaton>
headius: I can't get my head around this difference in behaviour - try this...
<chrisseaton>
I want all files to be treated as if they're the main one, where the blocks get precompiled or whatever
<headius>
-Xjit.logging
<headius>
it's not jitting for some reason
<headius>
-X+C hasn't been reimplemented because new JIT doesn't really have an AOT mode
<headius>
I'll try here as well
<chrisseaton>
'done compiling target script', no other output
<headius>
wot
<chrisseaton>
this is on master btw
<chrisseaton>
This isn't just mandelbrot - it looks similar on all classic benchmarks
<headius>
something broke then :-\
<headius>
that should be fine
<headius>
oh
<headius>
heh
<headius>
well you know we wait 50 iterations to compile :-)
<headius>
-Xjit.threshold=0
<chrisseaton>
ah, right
<headius>
we have talked about putting some other metric in place, like number of backward branches or something
<chrisseaton>
I'm sure this used to work in benchmarks before... but it was different code so maybe I did something else in the past
<chrisseaton>
so if you don't AOT, what does 'done compiling target script' mean now?
<headius>
that's outer.rb
<chrisseaton>
ok it's working now
<headius>
oh, you mean what's in there
<headius>
the JIT is capable of doign AOT, it just isn't hooked up in the loading process
<headius>
sorry, I wasn't clear
<headius>
AOT-on-load is probably the only thing missing from -X+C
<chrisseaton>
sorry, still not totally clear? you don't do AOT? so why is the benchmark fast when I make it the main file?
<headius>
we do for the target script, not for loading other files
<headius>
that is the default behavior in 1.7 as well, but -X+C will make it AOT every loaded file
<headius>
and we don't AOT every file by default because there's files we can't compile to bytecode
<headius>
and it's slower to AOT compile and load and verify that bytecode than to just interpret
dsc has joined #jruby
<headius>
so...what's missing is making load/require AOT with -X+C
<chrisseaton>
but it's only useful for benchmarks
<headius>
pretty much
<headius>
well, and for just running some script you want to be fast right away
<chrisseaton>
does a low JIT threshold harm benchmarks in any way?
<headius>
not really, it just takes longer to boot and may compile more code than necessary
<chrisseaton>
I guess I just want faster benchmarks if I want to look at warmup then
<headius>
sometimes the compiled version will be slower than interpreted but that's a bug
<headius>
oh, and large methods that compile to JVM bytecode might be too big for JVM to JIT, which is also slower than our interpreter
<headius>
oh, and... :-D
<headius>
there's a complicated set of variables to this madness
<headius>
mostly around balancing hot performance with reasonable startup
<chrisseaton>
i can't win this one - if I change default JIT iterations I can't measure warmup properly, and if I don't some benchmarks could never get compiled
<chrisseaton>
I should get -X+C for load working again - do you think that'd be easy? or might it be easier to adding profiling for backward jumps to your inteprreter?
<enebo>
profiler branch has a bootstrap algo which can detect changes per time
<enebo>
that is not hooked up for JIT yet but it is hoped to stall JITting until after bootstrap is over
<enebo>
with that said compiling first script is still neccesary for benchmarkers/single scripters
<chrisseaton>
Yes my problem is I set up a magic benchmark system that just works, except for this issue
<enebo>
jit.threshold=0 should be good enough
<enebo>
it will compile way too much
<enebo>
but it will be about the same as -X+C
lance|afk is now known as lanceball
<headius>
chrisseaton: well, I don't know which is a better solution, but hooking up AOT in load/require would be much easier
<headius>
the interpreter change should happen though, and I think some of that is there already
<enebo>
it is largely still on the branch but not in JIT logic
<headius>
if -X+C would have been ok for you, though, then jit.threshold=0 is about equivalent
<enebo>
still that logic will not help for loading a single script scope once
<headius>
chrisseaton: benchmarks that have one big method doing a lot of work aren't very good candidates for testing warmup anyway, are they?
<GitHub166>
[jruby] ckpeter opened issue #3966: Incorrect case evaluation when evaluating variable with value of [] https://git.io/vo8Pk
<headius>
I mean, even the hotspot engineers have told me that OSR is mostly in hotspot because of dumb big-method loop benchmarks
<headius>
jitting target script right away is our version of the OSR benchmark dance
<headius>
enebo: no, but it would help chrisseaton's case
<headius>
one method that runs for a couple seconds on JRuby ought to have plenty of profile pings to trigger jit
<enebo>
headius: yeah and fwiw we largely only use thread_poll to know we should be checking stats itself but the number of sites hit from that scope should still indicate it is a scope with action going on
<enebo>
current profiler is focusing on hot sites within the scope for sake of inlining but it is not a big stretch to use that as info for JIT’ing the method itself
<headius>
well thread poll is at the right places
<enebo>
yeah
<enebo>
I think we now have entry to scope and back end of loops
Aethenelle has joined #jruby
<headius>
yes
<headius>
that sounds correct
<enebo>
hmm I wonder if we could kill thread poll and just poll on branches
<enebo>
scope entry is manual poll and then all branches are
<headius>
what do you mean kill thread poll?
<enebo>
not have it as an actual instr
<headius>
oh
<headius>
hmm, I guess
<headius>
but we don't have an explicit "backward jump" instruction, so it would be a flag or something?
<GitHub80>
jruby/master 86ac58b Thomas E. Enebo: Fixing one bug unmasked another. when *[] should be when nil not when []. Exclude these tests until tomorrow
<headius>
I need to remove that jit caching stuff...it might come back but not like this
<headius>
it does seem right
<headius>
hmm
<headius>
ahh yeah
<headius>
this is still the old JIT stuff
<headius>
that NameError may be a red herring
<headius>
er NoMethodError
<headius>
chrisseaton: try just making it call runNormally with scriptNode
<chrisseaton>
still fails on the require_relative - which is really weird as that is in the main script, not the required script, which is what this should impact
<chrisseaton>
I guess some of your Ruby code might be being loaded incorrectly?
<chrisseaton>
Is require_relative written in Ruby?
<headius>
short term workaround might be to not +C while runtime.isBooting()
<headius>
but I'll see if I can figure out why it's breaking this way
<headius>
it starts up too fast, like kernel isn't running at all
<headius>
oh heh
<headius>
yeah that's it, not sure why though
<headius>
wow wtf
tcrawley-away is now known as tcrawley
<headius>
they're all compiling as empty scripts
<headius>
everything coming through this method
<headius>
well that's better now but I get an EBADF after it starts
<headius>
er no, an EPIPE
<headius>
stdout is getting closed
<headius>
hmm, must have been a fluke because I was piping to less
<headius>
chrisseaton: I have it working, will push momentarily
<chrisseaton>
thanks very much
<GitHub185>
[jruby] headius pushed 2 new commits to master: https://git.io/vo4sv
<GitHub185>
jruby/master 93b5851 Charles Oliver Nutter: Remove restrictions on which branches build in Travis.
<GitHub185>
jruby/master 4e98b71 Charles Oliver Nutter: Wire up -X+C to file loads.
<GitHub116>
[jruby] headius pushed 1 new commit to fast_fixnum_case: https://git.io/vo4sf
<GitHub116>
jruby/fast_fixnum_case 5aa73a3 Charles Oliver Nutter: Merge branch 'master' into fast_fixnum_case
<headius>
chrisseaton: no problem...just have been something odd with the pre/post logic
<headius>
I'll refactor this and get "wrap" support back in
tcrawley is now known as tcrawley-away
<headius>
heh, I just thought of one caller() optimization that probably wouldn't help much: cache frames 1 and higher if it's called more than once in a given method
<headius>
then only current line frame needs to be built
<headius>
if course if people are running caller in a hot loop, they have other problems
tcrawley-away is now known as tcrawley
<headius>
hmm, maybe this is actually better than 1.7 -X+C because if it can't compile it will at least interpret
<headius>
-X+C wouldn't run at all if any file couldn't load
<headius>
I mean couldn't JIT
<chrisseaton>
This is working
<chrisseaton>
Now all I have to do is run with -X+C for all benchmarks :(
<GitHub50>
jruby/master 141c3ab Charles Oliver Nutter: HEADIUS
<headius>
chrisseaton: it would be interesting to see if -Xjit.threshold=0 has any perf difference from -X+C
<enebo>
;)
<chrisseaton>
now I wonder if I'm being unfair when I benchmark Rubinius - if that doesn't have OSR does that get stuck in these big methods as well - I don't know if it has jump counters as well
<enebo>
so ‘case *[], false’ in MRI ends up as [false] and the case seems like it must treat that as ‘case false’
<enebo>
err switch case for when
<headius>
it does not have OSR as far as I know
<headius>
and I have no idea how to tweak their JIT to run sooner
<chrisseaton>
there's a threshold I've used
<headius>
you could just run rbx 3
<headius>
then you don't have to worry about the jit
<headius>
fwiw I don't feel like it's unfair because this is already a peculiar situation, running one long-running script after another that are freestanding
tcrawley is now known as tcrawley-away
<chrisseaton>
in this case I'm trying to see what is the best these implementations can possibly do, so I'm happy to turn on this like -X+C - I don't want to trip people up here
<chrisseaton>
if I know an implementation can do something, and I'm getting in the way, that's not what I want to do
<headius>
sure, makes sense ti stay on rbx too for that I guess
<headius>
er, rbx 2
<headius>
also, I don't believe what we can do today is the best we can possibly do :-)
<enebo>
ok good news is we also make [false]
<headius>
enebo: good news!
<enebo>
bad news is we need to unwrap that array so we only look at false