<enebo[m]>
headius: that metaclass issue may be that staticScope inherits the parents scopeType
<enebo[m]>
unless you use it in constructor or call setScopeType
<enebo[m]>
yeah I believe all of block is same info that compiledirmethod had so that will be a main target for me in the morning
<enebo[m]>
staticscope should also save file from irscope when AOTing and doing a setFile on staticscope
<enebo[m]>
unfortunately though we have to figure out why staticscope is not always getting a file set on it now
<headius[m]>
I am persisting the file from the static scope right now
<headius[m]>
And the static scope type but not the IR scope type
<enebo[m]>
yeah and it seemingly is null some portion of the time
<headius[m]>
I just persisted the things on static scope that were actually static
<enebo[m]>
yeah for execution IR Scope type is important
<enebo[m]>
and it is technically static info
<headius[m]>
The IR scope doesn't change, right? So I could just move that type field into the static scope?
<headius[m]>
That might fix the method definition issue
<enebo[m]>
it is there
<enebo[m]>
we have a field now for it but it is goofy in static scope but not in irscope
<headius[m]>
I also did nothing special with cref but it is usually set by the parent scope when it first instantiates the child scope
<headius[m]>
Ok
<enebo[m]>
we sometimes set it at construction and sometimes setter it but seemingly not always
<enebo[m]>
IRRuntimeHelpers depends on when it isn't set getting the parents scopeType so it does not walk up the scope stack
<enebo[m]>
I have no idea how it ended up like this but it is very unintuitive
<headius[m]>
Ok sure
<headius[m]>
I will try that after I eat some dinner
<enebo[m]>
Whatever you persist of the live staticscope for it should be correct though for AOT when you reify
<headius[m]>
Yeah I was surprised how much works already
<headius[m]>
Jefcoed really does not use much from IR scope
<enebo[m]>
I would like to consider "fixing" this somehow. I would like to know what the static scope actually is
<headius[m]>
Wha
<headius[m]>
Jit code
<enebo[m]>
jefcoed
<headius[m]>
This exercise will be good in general
<headius[m]>
I have never had a good grasp of what scope data is truly static and truly needed for execution
<headius[m]>
I think you have a better grasp of that because of your work on the interpreter context stuff
<enebo[m]>
once block is done and aliasmethod and methoddata we just need to debug and set file properly on staticscope so everything can stop asking irscope
<headius[m]>
Yeah, I did not go forward with any changes to blocks because it falls down anyway
<enebo[m]>
require_relative is just a derivative of file at this point
<enebo[m]>
well prepareBlock with indy may be a little more challenge
<enebo[m]>
headius: so prepareBlock does new CompileIRBlockBody but ultimately we want this constructor to change to include 'id', 'line', and 'argumentDescriptors'.
<enebo[m]>
right now there is a handle to cope with not being able to just push StaticScope onto the stack because indy is not that way
<enebo[m]>
we would not want to make n handles to pass all this info right?
<headius[m]>
Anything static can just be passed as an argument or encoded into a number or string
<headius[m]>
We are probably not going to get this to native compile for fosdem because there's a lot of invoke dynamic use in the jit right now
drbobbeaty has joined #jruby
<headius[m]>
It can all be replaced, but we are doing a bunch of it by default to reduce bytecode size
<headius[m]>
That won't be as big a deal with a native compile and if we cache things in final fields we know graal can optimize it like a constant
<enebo[m]>
but if I can pass that we can eliminate and AOT can end up not referencing IRScope...this will work with appcds right?
<headius[m]>
I can keep plugging away at it until our talk but there's a lot of little indy uses to replace
<headius[m]>
This will all work fine with AppCDS
<enebo[m]>
yeah I am really only concerned with appcds personally
<headius[m]>
Some limited invokedynamic may also work since they have to support lambda
<headius[m]>
Purely static indy sites should work fine
<enebo[m]>
I guess the other interesting thing I notice is closure body handles are in IRClosure and not a map like StaticScope
<headius[m]>
That is only a compiler artifact
<headius[m]>
It's so I can look up the handle in WrappedIRClosure
<headius[m]>
It could just as easily be a map on the visitor
<enebo[m]>
yeah it will have to be to eliminate IRClosure from existing
<enebo[m]>
looks easy enough since emitClosure makes a mangled name
nirvdrum has quit [Ping timeout: 258 seconds]
<headius[m]>
yeah just need a place to stick the handle, it's no problem
<headius[m]>
the closures get compiled in an initial pass through the method and then the handle is used later for wrapped instr
<headius[m]>
but it shouldn't matter in the compile
<headius[m]>
there's a number of things I use from IRScope* during compile that won't be needed during execution
<enebo[m]>
so to add line I can just add an 'int line' to Boortrap.prepareBlock in both sig() and the actual bootstrap called and then add it to IRByteCodeAdapter6.prepareBlock's invokedynamic call?
<headius[m]>
oh wait
<headius[m]>
is there a separate static scope type?
<headius[m]>
ah I see
<enebo[m]>
there is but it is not useful
<headius[m]>
type and scopeType
<headius[m]>
that's not confusing at all
<enebo[m]>
yeah I don't really recall what we use type for. I think it came into being when you combined BlockStaticScope and LocalStaticScope into a single class
<headius[m]>
yeah could be
<headius[m]>
redundant now
<enebo[m]>
So likely simple variable scoping assignment
<headius[m]>
8 usages
<enebo[m]>
Real issue is that parser uses it before IR builds it
<headius[m]>
only one of consequence
<enebo[m]>
so to remove them we would need to make scopeType at parse time
<enebo[m]>
did you see my question above?
<headius[m]>
the answer is yes
<headius[m]>
the args passed to the invokedynamic are sent to the bootstrap method where it creates CompiledIRBlockBody
<headius[m]>
so you just need to stitch it through the various signatures there
<enebo[m]>
Ok I think I can make Block not use IRScope in the morning reasonably quickly
<headius[m]>
well it happens every time an object is constructed the first time
<headius[m]>
it just works so well we don't notice
<enebo[m]>
I put a println on it and rarely saw it happen
<headius[m]>
it just needs to be able to walk scopes and find a list of instance vars accessed
<enebo[m]>
I can just encode it as a string...interestinly a few fields in actual type are redundant in where it will be going
<headius[m]>
this will probably go away later anyway once we can have multiple shapes in flight...we'll just start out with an IRubyObject[] version and then evolve to specific shapes later
<headius[m]>
yeah it's static once the scope has been compiled
<headius[m]>
so it could live in static scope too
<headius[m]>
it's like var names, but ivar names written in that scope
<headius[m]>
anyway disabling it works so yay, we can define classes and methods from AOT bytecode
<enebo[m]>
ah yeah it could since we actually have to stand that up but I don't how how it came to be on IRScope but really has nothing to do with IR
<headius[m]>
IRScopeType fixed that
<enebo[m]>
that will make that one much simpler
<headius[m]>
this is all pretty promising already though
<enebo[m]>
yeah I am not concerned with having something which runs something but I feel in 2 days we can run nearly everything
<headius[m]>
top-level scope probably should learn how to bootstrap itself
<headius[m]>
so we don't have to make that static scope and pass it in
<enebo[m]>
I know nearly everything left with a plan for removal
<enebo[m]>
hehe
<enebo[m]>
well good luck with top-level
<enebo[m]>
It sounds like a good idea though
<headius[m]>
just needs to do top level scope factory somethign something
<enebo[m]>
By end of friday I think we commit to what is done and build up deck on that and if we chip more next week we just tweak
<headius[m]>
yeah I'm cool with that
<headius[m]>
then go through all other startup tricks and end with ideas about AOT bytecode + CDS or native going forward
<enebo[m]>
I do really think methoddata, alias, and block can be done tomorrow
<enebo[m]>
file as we said above could be pretty simple if we just properly set it
<headius[m]>
I'll be up early for school drop-off tomorrow so I might call it done for tonight
<headius[m]>
I'll push what I have
<enebo[m]>
cool. Good day over all
<headius[m]>
heh
<enebo[m]>
we also got ir serialization lazy too even if it is only like a 5% gain
<enebo[m]>
yeah that is formulaic too...so methoddata on that should be just another string between a ';'
<headius[m]>
yeah
<enebo[m]>
At this point I will try and stay focused on eliminating more of IRScope...I would really like it gone by end of week
<enebo[m]>
I think we will be able to run nearly anything if so
<headius[m]>
top level is a priority for me mostly so we can say "here's a class file, let's run it"
<headius[m]>
right now that top level scope still requires parsing the file
<headius[m]>
I will fiddle with method data for a bit before I call it quites
<headius[m]>
quits
<enebo[m]>
ah cool...yeah that one should fit into staticscope with what you have other than having to write the encoder/decoder
<enebo[m]>
and not having it on IRScope will be nice too
<headius[m]>
yeah
<headius[m]>
I will have everything go through StaticScope and a field there
<headius[m]>
the logic for gathering it will stay on IRMethod for the moment but it could live anywhere
<enebo[m]>
headius: yeah ignore the move it odd thing...it is lazy and it really does just walk IR so it does belong there
<enebo[m]>
actually though it would probably not be too hard to bake this into the parser and add to staticScope
<enebo[m]>
It would mean all StaticScope have this field I guess?
<enebo[m]>
anyways no point doing much more than minimum to the task at hand
<enebo[m]>
compiler does have IRScope so it is fine to live there
<headius[m]>
baking in would be nice
<headius[m]>
it has to live somewhere
<headius[m]>
well "has to" because reification is pretty dumb right now
<enebo[m]>
yeah I mean it would just be another array on staticscope and null on non-method scopes
<enebo[m]>
I don't think staticscope is a big memory thing or not
<enebo[m]>
we could also add it to lvar array on creating method body
<enebo[m]>
but then we would have some special thing...anyways I think when I come up with ideas like that I need to do something else
<enebo[m]>
weirdly I am sleepy at 7pm
<headius[m]>
yeah
<headius[m]>
I have it returning EMPTY_LIST for non-method scopes
<headius[m]>
the field will always be there though
<headius[m]>
too much coffee?
<headius[m]>
I've been drinking way too much now
<enebo[m]>
I did have a single coffee today but it was full caffeine and not half caf
<headius[m]>
maximum strength
<headius[m]>
woot, it works
<headius[m]>
I'll push this too
<headius[m]>
easy peasy
<enebo[m]>
nice
<enebo[m]>
going to windows land...
<headius[m]>
ok I think that's pretty good progress for today
<headius[m]>
once we get blocks working we'll be most of the way to fully AOT bytecode
<headius[m]>
hmm occurs to me most of the handles we use in JIT probably will work with native compile since they're just direct handles to static methods
<headius[m]>
just need to wrap them in some interface like a lambda and they'll precompile in native image 👍
<headius[m]>
I'm not sure if this "dry run" is really useful anymore, but in this case dry run prevented bytecode compile because it doesn't set the IRScope into the StaticScope (e.g. in IRMethod constructor)
<headius[m]>
there's always one person ready to pour cold water on your fun
_whitelogger has joined #jruby
<lopex>
wowsers
<enebo[m]>
headius: so compile_ir originally was an entry point for jruby_visualizer and then later started getting used by compiler itself. Ultimatrely, we should be able to support compiling IR with recording all relationships but this change is ok. jruby_visualizer is not working
<headius[m]>
ok, my change didn't fix everything though
<headius[m]>
see jruby suite...something is still getting a StaticScope with a null IRScope
<headius[m]>
That should probably be ok in the future, once StaticScope knows how to reconstitute its IRScope, but right now it breaks stuff going through those JRuby.compile variants
<headius[m]>
the jruby suite failures may be coming from jrubyc so I'll look into that now
<headius[m]>
for native image we'll want to rewire CompiledIRMethod to call through some callable interface rather than MethodHandle directly...then the precompiled Ruby code can just statically boot up a lambda equivalent and that goes into the dynamic method object
<headius[m]>
remains to be seen if we can also use that callable interface in invokedynamic sites and still get the same amount of inlining...I would hope so, but we'll see
<enebo[m]>
the other option is to just have instantiated type which extends CompiledIRMethod
<enebo[m]>
That would eliminate exact and varargs handles from the equation too
<headius[m]>
yeah that is an option but it would mean a class per method body
<headius[m]>
that will be much harder to wire up with lazy scopes
<headius[m]>
well, harder, maybe not much harder
<headius[m]>
on instantiation you'd pass in a parent scope anyway
<headius[m]>
we have lots of material for our talk...doesn't have to be implemented stuff as long as we have an interesting roadmap going forward
<enebo[m]>
yeah
<enebo[m]>
I was thinking ultimately lambda forms for a single use will end up just being generated code as well so maybe less classes that route internally
<headius[m]>
I think a big point to talk through will be why we interpret to begin with
<headius[m]>
and then why we're re-interested in AOT bytecode compiles now
<enebo[m]>
I guess perhaps they are not "types" but I don't know if that matters or how different it is in actuality
<headius[m]>
it will also be interesting for background to show how our startup improves on successive runs in same VM
<headius[m]>
"it's not our fault"
<enebo[m]>
heh
<headius[m]>
enebo: are you able to run rake test:jruby?
<headius[m]>
I get errors because it's expecting a junit.jar to live in test/target/junit.jar and I don't know how that gets there
<headius[m]>
spec/java_integration/fixtures/ThrowExceptionOnCreate.java:3: error: package junit.framework does not exist
<headius[m]>
import junit.framework.Test;
<headius[m]>
it looks like there's maven crud to copy the jar in there but rake test:jruby doesn't trigger mvn to build anything
<headius[m]>
maybe I've always run -Ptest once before running rake targets in the past?
<headius[m]>
hmm no that didn't copy it either
<headius[m]>
test/pom.rb hasn't changed since July
<headius[m]>
and it works on travis...what's up with my env
<headius[m]>
huh ok -Pbootstrap did it
<headius[m]>
I swear I did that before but must have cleaned since and wiped it out
<enebo[m]>
you know it would be great to change the output of some of these runs
<enebo[m]>
I think 2 errors at the bottom takes like 5 minutes to figure out what they actually are
<headius[m]>
yeah that is annoying...there's some formatting option we're not passing
<headius[m]>
several of these suites were flipped to verbose output so we could see hangs...turning that off would probably clean it up
<enebo[m]>
is this minitest default? what is this
<headius[m]>
not to mention speeding up travis's gigantic page renders
<enebo[m]>
oh verbose
<enebo[m]>
I am running test:jruby atm and it works
<enebo[m]>
this commit is what causes erb to not compile
<headius[m]>
yeah that was just trying to get it working
<headius[m]>
or at least that was patching a previous failure that was also null scope
<enebo[m]>
but erb.rb works the commit before this according to bisect
<headius[m]>
turning off dry run maybe caused new errors, but what am I supposed to do here
<headius[m]>
the IR you get from compile_ir should not have null IRScopes
<headius[m]>
in the case of jrubyc it's clearly not a dry run because it needs to persist a working IR
<enebo[m]>
I need to make sure I fully remember but dry scope will guarantee a field gets all lexical relationships whether they are runtime meaningful or not...jrubyc is likely compiling down and it perhaps is using that somehow
<headius[m]>
all it does is persist
<headius[m]>
compile_ir plus persist
<headius[m]>
if I put the dry run back it fails differently because static scopes now have null IRScope
<enebo[m]>
dry run came years before persist and was only intended for debugging but it likely represents a different name
<enebo[m]>
// List of all scopes this scope contains lexically. This is not used
<enebo[m]>
// for execution, but is used during dry-runs for debugging.
<enebo[m]>
private List<IRScope> lexicalChildren;
<headius[m]>
every time I try to get IR to compile something I feel like I'm doing it wrong
<enebo[m]>
So I don't really get why we would see any null anything because of this but without dry run this attribute is not written at all
<headius[m]>
do compiler passes set some of this up?
<headius[m]>
is there something else I should be calling before expecting to persist IR?
<enebo[m]>
This comment is likely wrong in that jrubyc probably does a lexical_children
<headius[m]>
jrubyc only calls persist
<enebo[m]>
oh well perhaps that is an issue then
<headius[m]>
literally just those three lines I pasted... compile_ir, persist into a ByteArrayOutputStream
<headius[m]>
yeah wtf
<headius[m]>
that line in DefineClassInstr just returns the body field it was constructed with
<headius[m]>
so something's creating that instr with a null body
<headius[m]>
which is never valid
<headius[m]>
we have an assertion, I'll try to get a trace
<enebo[m]>
yeah dryRun is misnamed but required for persistence
<headius[m]>
required?
<headius[m]>
I don't understand that
<enebo[m]>
It walks from the root os the scripts and calls getLexicalScopes
<headius[m]>
we don't dryRun in IRTranslator but then persist just fine
<enebo[m]>
we normally do not reflect the entire lexical structure since we never need it for execution
<enebo[m]>
So the name sucks. It should be 'preserveLexicalInformation' or something but we have not root down method
<enebo[m]>
other than this one
<enebo[m]>
I did work on a patch once to combine getClosures and getLexicalChildren so they would get stored in the same collection
<enebo[m]>
then perhaps always having it on would hardly use any more memory
<headius[m]>
but persistence should end up forcing child scopes anyway, no?
<enebo[m]>
the IR builder makes all the scopes and persist needs to walk them
<enebo[m]>
dry run pretty much just allows getChildren to be populated
<headius[m]>
ok now I'm raising a hard error if body is null and it won't raise
<enebo[m]>
we don't to prevent a hashmap per IRScope normally as we said IRScope memory was too big
<enebo[m]>
hahahah I also see why the null happens
<enebo[m]>
if (lexicalChildren == null) lexicalChildren = new ArrayList<>(1);
<headius[m]>
wtf is that
<enebo[m]>
now assign of 0 there but why was it ever like that
<headius[m]>
oh it adds scope immediately after though
<enebo[m]>
so why was dryRun needed to be removed
<headius[m]>
the 1 is just initial capacity
<headius[m]>
if you put dryRun back, the top scope's StaticScope doesn't have an IRScope
<enebo[m]>
ah so it is still isEmpty?
<headius[m]>
I don't know... StaticScope.irScope == null
<headius[m]>
I assumed it was because dryRun prevents scopes from setting themselves into their own StaticScope
<enebo[m]>
ok...well dryRun should be renamed and we have to fix that another way
<headius[m]>
look at IRScriptBody constructor for example
<headius[m]>
why does it prevent setting IRScope into StaticScope
<headius[m]>
I do not understand
<headius[m]>
this is generally broken because much more stuff goes through StaticScope to get IRScope stuff now
<enebo[m]>
reading that epic commit comment associated with it
<headius[m]>
but because of dryRun (and apparently other things) that's sometimes null
<enebo[m]>
ok so dryRun is fundamental to irpersist but really only the lexicalChildren part...this other conditional I have no idea why it is like that
<enebo[m]>
I mean I cannot fathom why it matters
<headius[m]>
I will try reverting dryRun and moving the irscope sets outside those conditions
<headius[m]>
also
<headius[m]>
if (staticScope != null) {
<headius[m]>
staticScope.setIRScope(this);
<headius[m]>
}
<headius[m]>
how can it be valid for a scope to have a null StaticScope
<headius[m]>
I mean we want to cut that connection but StaticScope has to get the IRScope somehow
<enebo[m]>
well oriringally dryScope was only for printing out stuff about IR and not executing anything
<headius[m]>
I don't see why any of these scope constructors have this dryRun conditional
<headius[m]>
all they ever do is populate simple fields with existing objects
<enebo[m]>
then when serialize came into being we/i/gsoc person noticed it preserves lexical children relationships which are needed for walking root down
<headius[m]>
I mean really...
<headius[m]>
if (getManager().isDryRun()) {
<headius[m]>
} else {
<headius[m]>
this.body = null;
<headius[m]>
why
<headius[m]>
this is in IRClosure constructor
<headius[m]>
why would it ever want to null out the body
<enebo[m]>
It is debugging IR and compiling would explode vs printing out useful info
<enebo[m]>
That would be my guess
<enebo[m]>
this is not for executing ruby original but getting a diag dump when IR was primal and super messed up
<headius[m]>
ok I guess in this case it's not a huge deal...body = block body and not needed for non-execution modes
<headius[m]>
so this one is one less object
<headius[m]>
but it also refuses to set staticscope.irscope
<enebo[m]>
but specifically if it tried to do that when something was totally not working you would not be able to get any info past that point
<enebo[m]>
but since it does not execute it did not need to set staticscope
<headius[m]>
I guess this is because END has same scope as parent but that broke something too
<headius[m]>
I had to modify it to clone parent's static scope
<headius[m]>
I mentioned that above also
<enebo[m]>
yeah I saw that...that is a good idea
<headius[m]>
END can't have a static scope that points at some other IRScope
<enebo[m]>
it totally solves that issue
<headius[m]>
I see setupLexicalContainment now
<headius[m]>
that one I generally get
<enebo[m]>
so here is a summary of the situation as I see it
<enebo[m]>
dryRun was originally a tool for debugging without executing
<enebo[m]>
At smoe point when jruby_visualizer existed as a gsoc project we tried to leverage it so we could print out data
<enebo[m]>
we are past needing dryRun for debugging and jruby_visualizer never made it past POC stage
<enebo[m]>
so I think perhaps we add a name 'preserveLexicalInformation' or something like that and then remove these dryRun checks from the setStaticScope bits
<headius[m]>
I only found those two valid uses of dryRun
<headius[m]>
I'll show you patch once I confirm things are working
<headius[m]>
at this point I wonder if it's even worth it...all it prevents is a BlockBody object per IRClosure and an ArrayList of children per IRScope
<headius[m]>
oh this explains why my POC of testing persistence in spec:compiler broke
<headius[m]>
I got weird NPEs there too
<enebo[m]>
well we were pants on fire on memory for IR about 5 months ago so I think we could have our cake and eat it to to just combine closures and children with filters for each
<enebo[m]>
I am amazed you could do anything with persistence without dryRun
<headius[m]>
yeah surely we can assemble the list of children from other structures we already maintain
<enebo[m]>
probably only stuff without children would persist
<headius[m]>
well I couldn't
<enebo[m]>
ah yeah makes sense
<headius[m]>
I am still confused why persistence works in IRTranslator though
<enebo[m]>
getChildren would return something empty with it off
<headius[m]>
spec:jruby, spec:ji, spec:compiler, and test:jruby pass with that (and my previous dryRun commit reverted
<headius[m]>
I still question why StaticScope would be null in any of these cases but that's a different rabbit hole
<enebo[m]>
ah it is setupLexicalContainment
<headius[m]>
pushed
<enebo[m]>
it checks to see if IR_WRITING it set
<headius[m]>
ohhh
<enebo[m]>
so there is possibly a path by using that and not dryRun
<headius[m]>
too many conditionals
<headius[m]>
that comes full circle then
<headius[m]>
when I just used persistence directly I'd get NPEs
<headius[m]>
because of this scope thing
<headius[m]>
when I ran with WRITING it worked
<headius[m]>
well maybe because of this scope thing, I guess I can't be sure, but clearly either dryRun or WRITING must be set or persistence is broken
<headius[m]>
persist should probably just call what it needs to call to set up
<enebo[m]>
That record lexical hierarchy flag I think may have been in the original gsoc project for persist
<headius[m]>
-probably
<enebo[m]>
'chen' did it and I have no idea who that is
<headius[m]>
given an IRScope, persist should always work
<headius[m]>
dryRun mode or no
<enebo[m]>
well the bottom line is until we save all lexical children it will never work
<headius[m]>
we only save closures right now?
<headius[m]>
I mean in non-dry mode
<enebo[m]>
saving them all the time is fine for now but if we merge closures (as they are also lexical children) with all others then the cost will be minimal
<enebo[m]>
yeah we only preserve runtime usable children closures being also eval and begin/end
<headius[m]>
yeah I don't want to add back a lot of memory use but this seems like more trouble than it's worth
<enebo[m]>
believe it or not I did land a commit merging them together and something was not quite right and I reverted
<headius[m]>
I hope when we abandon JRuby we do a better job of it than this
<enebo[m]>
I mean all OSS sticks around so I am not just sure
<enebo[m]>
we have like 7x as many stars
<enebo[m]>
not that stars really mean anything
<enebo[m]>
My mind was blown when they released and did that coding demo...it was amazing how many things you had to edit to do anything
<enebo[m]>
but they had PR and media push on it so if they had made it I would expect a huge number of stars or something :)
<headius[m]>
oh yeah the demo was a riot
<headius[m]>
IT'S SO EASY
<enebo[m]>
and Spring already had a reasonable framework for DI: SpringMVC
<headius[m]>
while doing a mind-numbingly complicated dev process
<enebo[m]>
If you were into String that is
<headius[m]>
I never bought into aspects
<headius[m]>
horrible idea
<enebo[m]>
AOP has never resonated to me but a large part of it loss of locality to what is actually happening (with no source hint that it will be)
<headius[m]>
these look like good errors 👍
<headius[m]>
yeah exactly
<headius[m]>
what a great idea to have code injected into your execution path based on some configuration file over there
<headius[m]>
debugging alone becomes impossible
<enebo[m]>
The notion of forming an onion is appealing at some level but not an invisible one
<headius[m]>
invisible onion
<headius[m]>
that sums it up
<headius[m]>
it's the tor of OOP
<enebo[m]>
hahaha you sort of don't want tor of code
<enebo[m]>
You will never figure out where that code is...you made it safe from prying eyes
<headius[m]>
perfect
<headius[m]>
enebo: I pushed nil change and I made IRClosure.body just be lazy rather than ever letting it be null
<headius[m]>
still 7 failures in persistence specs
<headius[m]>
nil commit is big because I also removed IRManager.getNil and all consumers now just do Nil.NIL
<headius[m]>
there's at least a few places that only had manager to get nil so there's more cascading cleanup possible
<enebo[m]>
ok
<rwilliams[m]>
Can one of you explain to the Java noob the implications of all the recent work you have been doing? Seems super exciting if I'm understanding it right.
<headius[m]>
Boolean could be another singleton option...that one is also not a singleton now
<headius[m]>
rwilliams: we are cleaning up the divide between the runtime execution state for a given script and the compiled code+structures that represent it... this will allow us to precompile Ruby code into JVM bytecode with just what it needs to execute, excluding a lot of state only needed by the interpreter
<headius[m]>
it's also just a lot of good cleanup of IR
<enebo[m]>
Mostly though the realization that using IR data for execution was not the right separation in the JIT
<headius[m]>
once we can precompile to JVM bytecode we can start to do things like precompile all of stdlib and let JVM cache it, or native compile the entire precompiled app + libraries using something like GraalVM
<enebo[m]>
once any IR data structures are removed from what is used for bytecode generation then making stuff like full AOT becomes simpler
<enebo[m]>
I would not say we have cleaned up IR as much as we just stopped using it for runtime stuff
<rwilliams[m]>
Good stuff
<headius[m]>
for a user it should eventually mean faster startup and the possibility of precompiling entire apps to native code
<enebo[m]>
My take from last couple of weeks is we have some pretty weird internals revolving around state in staticscope
<enebo[m]>
we have a file field which seemingly is sometimes populated. we have weird scopeType which get inherited from its parent...or it doesn't
<enebo[m]>
we were inheriting the parent IRScope in staticscope but seemingly removing it did nothing
<enebo[m]>
All of which is to say code has history and all of this stuff either made sense or got twisted into what it is today...it is good to revisit stuff
<rwilliams[m]>
How feasible is it to get the full jvm bytecode for a rails app with all the metaprogramming
<headius[m]>
metaprogramming does not affect it
<headius[m]>
metaprogramming is mostly just twiddling method tables, which remain the same structure
<headius[m]>
eval of strings would not get precompiled obviously and would never leave the interpreter
<rwilliams[m]>
Ahh
<enebo[m]>
in any case of halting problem where same stuff is not executed and therefore not AOTd we still can interpret and JIT
<headius[m]>
though there's ways to work around that (e.g. if it's an eval of a method, and we can eventually jit that method, we can save the jit result)
<headius[m]>
right
<headius[m]>
"full AOT" would still ship with interpreter for unexpected code or string evals
<headius[m]>
enebo: though we could cut a pretty tiny little JRuby if we were able to strip out parser + compiler + interpreter
<headius[m]>
I went ten pages into google results and no sf.net hits
<headius[m]>
so yeah nobody visits this project
<enebo[m]>
Java::JavaLang::BootstrapMethodError: call site initialization exception
<headius[m]>
enebo: wha
<headius[m]>
are you current?
<headius[m]>
branch is green right now
<enebo[m]>
we never run spec:ruby:fast:jit or it is never green
<enebo[m]>
I have been running it before most changes to see
<headius[m]>
yeah I'm not sure it has ever been green
<enebo[m]>
but it has been non-green for all of 9k a tiny bit
<headius[m]>
but probably should remedy that
<headius[m]>
no reason it can't be
<enebo[m]>
yeah the actual faiilures are not very serious
<headius[m]>
we pass a bunch of other suites in force jit mode now
nirvdrum has quit [Ping timeout: 260 seconds]
<enebo[m]>
hmmm I would naturally think this involves proc to method conversion optimization but for JIT?
<enebo[m]>
could be...hmm...anyways I will spend some time looking
<headius[m]>
ok
<headius[m]>
I never got to my lunch so doing that now
<enebo[m]>
ok
<headius[m]>
enebo: looking at those failures now...is this file the only one that fails this way?
<enebo[m]>
in spec:ruby:fast yes
<headius[m]>
ghmm
<enebo[m]>
define_method does have specialness in the native impl
<headius[m]>
it does
<headius[m]>
I'm leaning toward this being a problem with preparing the block
<headius[m]>
it gets prepared in the jit as a proper CompiledIRBlockBody and then flipped to a method by define_method, which probably conflicts
<enebo[m]>
This commit largely only removes name + _IRScope
nirvdrum has joined #jruby
sagax has quit [Read error: Connection reset by peer]
<headius[m]>
digging into the bytecode
sagax has joined #jruby
<lopex>
headius[m]: you think that AOT would support sinatra/sequel like deps in the shorter run ?
<lopex>
I think rails would be an overkill at first shot
<headius[m]>
yeah sure
<headius[m]>
mostly it's just a matter of telling the native image compiler what classes and jars it needs to keep available
<lopex>
I have a pretty minimalistic deps and quite a bit of functional apps in my framework
<enebo[m]>
To me whether it is possible or not also requires it not taking minutes+ to make
<lopex>
my apps start in 0.4 sec in mri
<enebo[m]>
although I guess it depends on the benefits for that to factor in
<enebo[m]>
lopex: zippy
<lopex>
and like two / three secs for jruby on a good machine
<lopex>
enebo[m]: most startup i sequel cache read, and my own processing
<headius[m]>
for local dev I don't see AOT compiling dependencies as useful
<lopex>
like verifying relations, column existence and the like
<lopex>
headius[m]: I have my own reloading feature, so I dont care avout that
<headius[m]>
main limitation for partial AOT is JRuby extensions in deps
<lopex>
by local you mean dev ?
<headius[m]>
yeah
<lopex>
I dont care
<lopex>
actually, reload is faster on jruby than mri in my case
<headius[m]>
like we could native compile JRuby + stdlib and let the rest interpret and probably still have better startup than JVM, but the libraries with deps wouldn't be able to load
<lopex>
no idead why
<headius[m]>
there's supposed to be a JVM interpreter for Truffle coming that might make it possible to bridge tht gap
<headius[m]>
truffle really is a weird wart on the side of a native image though
<headius[m]>
like let's compile everything so it's native, but then put languages on there that still have to parse and interpret and optimize and compile at runtime
<headius[m]>
weird
<lopex>
truffle aot ? I thought substrate was the only way
<lopex>
oh wait
<headius[m]>
there's no truffle AOT
<lopex>
right
<headius[m]>
truffleruby et al just native compile their java parts
<headius[m]>
so they get almost none of the benefit of native image unless you also pre-boot the app in that image (e.g. get those truffle ASTs in memory and running before you create native image)
<lopex>
ifonly that AOT had any control on granulation
<headius[m]>
well it's still such a band-aid
<lopex>
like, I dunno, first require pass, and then aot ?
<lopex>
subsequent load and the lke could be interpreted
<headius[m]>
the right answer would be a JVM that can save its optimized code across runs and expand its closed world dynamically
<headius[m]>
substrate ONLY works with a moment-in-time closed world
<headius[m]>
something changed? you have to recompile the world
<lopex>
but you mentioned the evals
<lopex>
so evals, loads etc
<headius[m]>
evals currently have no unique identification so we can't cache or compile anything
<headius[m]>
loads can be uniquely identified and we can say "oh this file was precompiled as X"
<headius[m]>
and tell substrate about X
<headius[m]>
but evals certainly could do that too...it's just never been a priority
<lopex>
so the idea of "merging" two aoted worlds is like mad mans dream ?
<headius[m]>
I don't think so, but it's not possible with substrate right now
<headius[m]>
OpenJ9's "AOT" basically just saves off jit results over time so it can boot into those faster the next time
<headius[m]>
but it's not closed-world optimization and those jit artifacts are not as optimized as what you might get from a long run
<lopex>
so like PGO almost
<headius[m]>
yeah
<lopex>
like static profiles etc
<headius[m]>
my current thinking for JRuby + AOT + substrate would be smaller services...compile all the .rb to .class, tell substrate about all the JRuby extensions, and profile out a list of Java classes to make reflectable
<lopex>
which isnt a bad idea even from c++ perspective
<headius[m]>
there's also PGO for substrate but it's not free
<headius[m]>
or Free
<headius[m]>
so that's a nonstarter
<headius[m]>
enebo: seems to need to be invoked twice to fail
<headius[m]>
which would explain why it's a call site linking error with no other Java lines in trace
<headius[m]>
it's already bootstrapped and then something breaks
<headius[m]>
these are tricky to debug but it's not an effect of my code really
<enebo[m]>
so what about that commit exposed it?
<headius[m]>
if you make the define_method chunk be inside a before :all it passes
<headius[m]>
what changed I'm not sure yet
<enebo[m]>
yeah something about that change seems to have made it fail...I never saw these fail and I ran that suite dozens of times before that
<headius[m]>
I think I have a repro
<enebo[m]>
I am really curious on this one because it is indy and I feel I will learn something
<enebo[m]>
yeah that 1:1 relationship is important
<headius[m]>
there's probably an underlying ClassCastException or something that breaks the call site
<headius[m]>
swallowed somewhere
<enebo[m]>
and we did know this was not always 1:1 so I guess we should keep that in our heads for the next weird error
<enebo[m]>
cool
<headius[m]>
yeah all passes now
<headius[m]>
👍
<headius[m]>
I'll commit and push
<enebo[m]>
fantastic
<headius[m]>
jeez, where did the day go
<enebo[m]>
debugging is fatal
<enebo[m]>
Nearly have all IRBlockBody not using IRScope except for proc to method conversion itself
<enebo[m]>
It is all just at CompiledIRBlockBody constructor so last part will be hooking up the fields I need in JIT
Antiarc has quit [*.net *.split]
ang-st has quit [*.net *.split]
<enebo[m]>
assuming that is ok then file and aliasmethod are maybe last significant users
<headius[m]>
that's excellent
<headius[m]>
I will help with jit for sure, I know that part is a bit involved
<enebo[m]>
well it weirds me out to see invokedynamic(...., arg1, arg2, arg3)
<headius[m]>
I have been realizing as we work on this how important a modular jit is becoming
<enebo[m]>
I mean those must all be constants to work properly?
<headius[m]>
we need to be able to turn off only part of indy, or truly turn off all of it, or whatever
<enebo[m]>
Is that why that works?
<headius[m]>
yeah
<enebo[m]>
ok then this should be super easy to finish for me
<headius[m]>
so integral types, strings, and static method handles
<enebo[m]>
knocks on wood
<headius[m]>
and method types
<headius[m]>
anything in constant pool
<headius[m]>
hell, more snow?
<enebo[m]>
I am excited to put a print in getIRSCope on staticscope tomorrow afternoon and never see it execute
<headius[m]>
coming down thick now
<headius[m]>
yeah that will be awesome
<headius[m]>
we may actually be able to AOT a simple app and boot it with CDS
<enebo[m]>
yeah I am excited to see if it paid off or not
<headius[m]>
that change is pushed now, I was not up to date
<enebo[m]>
I assume it will
<enebo[m]>
but I am still happy regardless
<headius[m]>
whether it does or not it will be interesting for the talk
<enebo[m]>
My only reservation is having to reconsider how inliner works but I THINK I will just use staticScope and lazily grab IRScope
<headius[m]>
if it does not then the next question is why, and I suspect it would be that we're throwing a lot of cold bytecode at the JVM versus our hot interpreter
<enebo[m]>
and honestly I am content with AOT not doing inliner for a while
<enebo[m]>
it is not even enabled by default
<headius[m]>
which online simple AOT could solve
<enebo[m]>
I assume we will see some significant win from no parsing
<headius[m]>
well I'm not sure what AOT + inliner would even look like
<headius[m]>
it becomes the same problem as trying to save C2 output
<headius[m]>
maybe we can get a research grant
<enebo[m]>
cold executing being slower than our interpretation would be a little surprising
<enebo[m]>
AOT itself is just a stable state JIT
<headius[m]>
surprising but it makes for an interesting case for openj9 which does have JIT caching
<headius[m]>
alibaba folks are also adding taht feature to openjdk
<headius[m]>
all things are happening
<enebo[m]>
holding the newversion of the code is an issue but not too weird
<headius[m]>
I'll be offline for a bit and back working this evening
<enebo[m]>
right now we use methods as opaque wrappers which invoke handles...so I would guess an inline on a JITTed method would be saving the AOT handle and using the inline handle
<enebo[m]>
on deopt go back to AOT one
<enebo[m]>
ok
<headius[m]>
we're close
<enebo[m]>
I will work until 5ish and maybe spend an hour tonight on some slides
<lopex>
HOW CLOSE
<headius[m]>
we might want to consider some sort of reverse null check in setIRScope
<headius[m]>
like if it's already set, error out
<headius[m]>
StaticScope.setIRScope
<headius[m]>
this is now two places where it has broken something
<headius[m]>
lopex: 🙏
<lopex>
I wonder how does the whole jit machinery pollute the cache so high that a more naive code would run faster
<enebo[m]>
headius: yeah that is a good call
<headius[m]>
yeah I've wondered that too
<headius[m]>
I know our early churn from interp to jit really messes with JVM's head
<enebo[m]>
this branch removed the weird inherit from parent logic already
<headius[m]>
things optimize quick for interp methods and then we change it up
<enebo[m]>
which I may cp over to master along with a fix to refinements I made
<headius[m]>
but if this works we can lazily save off .class
<headius[m]>
so the first ever progressive AOT for Ruby
<headius[m]>
give me money
<headius[m]>
ok bbl
<enebo[m]>
hahah I'll take some cabbage too :)
<headius[m]>
greenbacks all around
<lopex>
ah those veggies
shellac has joined #jruby
ang-st has joined #jruby
Antiarc has joined #jruby
nirvdrum has quit [Ping timeout: 268 seconds]
Antiarc has quit [Quit: ZNC 1.7.4+deb7 - https://znc.in]