jmalves has quit [Read error: Connection reset by peer]
jmalves_ has joined #jruby
<kares>
hey enebo, what's the release hold up? waiting for refinements or more Rails 6 testing?
drbobbeaty has joined #jruby
drbobbeaty has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
enebo has quit [Ping timeout: 245 seconds]
enebo has joined #jruby
jmalves_ has quit [Remote host closed the connection]
jmalves has joined #jruby
jmalves has quit [Read error: Connection reset by peer]
jmalves has joined #jruby
jmalves has quit [Remote host closed the connection]
rdubya has quit [Ping timeout: 240 seconds]
rdubya has joined #jruby
jmalves has joined #jruby
xardion has quit [Remote host closed the connection]
xardion has joined #jruby
lucasb has joined #jruby
<headius>
kares enebo: the bulk of refinement work is done with a green build
<enebo>
yay
<headius>
there are a handful of things still tagged, mostly surrounding prepend/include combined with refinements; those are mosly systemic problems with how we handle super from modules, and I've got another branch where I'm attempting to fix that
<enebo>
headius: you remember the '
<enebo>
include module.new'
<headius>
it's called refinements2 right now but basically it starts using the MRI tactic of capturing the module a method came from and passing it through to use for super
<headius>
I made that change for most lookup paths and only failed a few tests
<enebo>
Is that perhaps part of the same systemic issue? since I believe that is about changing cref
<headius>
enebo: I don't think that would behave any differently
<headius>
not sure which include you mean exactly though
<enebo>
headius: I was just wondering if that is similar in nature to prepend issues with refinements
<enebo>
include Frogger.new where Frogger is a module
<headius>
it's general to all module includes and prepends
<headius>
doesn't matter if it's duped or anonymous or whatever
<enebo>
yeah
<enebo>
anyways I am distracting with that question
<headius>
duped modules are probably still not working exactly as MRI
<headius>
but I think this lookup mechanism would fix that in a different way
<headius>
enebo: in terms of DynamicMethod.call this basically starts *actually* using the passed in klazz
<headius>
enebo: so we should chat here or gitter about this threading thing
<headius>
now that we've gotten through some discovery
<enebo>
headius: yeah looks good but I think that I changed getImplementationClass().getMethodLocation() not so long ago for some bug
<headius>
yeah I think using it there was just a smell
<headius>
it's trying to find the right place, but with the new logic it gets the right place to begin with
<headius>
we can circle back to that in a minute thiough
<enebo>
headius: yeah I assume
<headius>
enebo: have you been able to produce errors with finalization off?
<enebo>
well I am still trying to see an error
<enebo>
I maybe have run like 15 times
<enebo>
which is a record
<headius>
I'm pretty sure that *most* of the errors you saw were from this, but can't guarantee all of them were
<enebo>
so I think finalization is a path
<headius>
this path could be helped by attempting to shut down Ruby threads at teardown
<enebo>
90% of the cases stopped when I disabled selectorpool
<enebo>
I did disable just finalization early on and it did not seem to have any effect
<headius>
you remember we had someone who wanted this for embedding too
<headius>
well unless you can show me otherwise, I don't see any way select logic would close an IO
<headius>
finalization definitely does
<enebo>
yeah but I literally had everything commented out up to the selectorpool and it was always failing
<headius>
I suspect you have been seeing two issues all along
<enebo>
I will try again
<enebo>
selector pool back on with all finalization off
<headius>
if you see anything it should be closed selectors
<headius>
but not the closed stream
<enebo>
headius: yep yep I only see closedselector
<enebo>
I think part of my problem before was I had also modified notifier.rb so I was not seeing things as they were originally noticed
<headius>
makes sense
<headius>
I think we should revert the selector closing...if there's threads waiting on it they should ideally be picked up when we decide how to handle live threads
<headius>
and we should institute the same thread-nuking that MRI does, probably only for threads started from within JRuby
<enebo>
well I am with you on second part in spirit
<headius>
short term I think the only solution for rb-inotify is to handle those errors
<enebo>
but I am nervous on risks
<headius>
I'm very nervous on risks of thread killing
<headius>
MRI does it with impunity
<enebo>
if they added their own explicit resource manage via at_exit they do not even need the error handling
<headius>
this is where I started to realize we need to have a couple levels of shutdown
<enebo>
but they would need more code to notice
<headius>
well, maybe
<headius>
if they just kill the threads in at_exit they'd be mimicking what we'd be doing
<headius>
it would be reasonably clean but they might have to have a blanket rescue for weird errors that come up from killing a running thread
<enebo>
at_exit will run before finalizers so I think if they can know they are shutdown by end of at_exit then cool but I guess at_exit is on another thread so more complicated
<headius>
they might also be able to set @handle.close_on_exit = false
<enebo>
I was not thinking of a kill either as much as toggling state
<headius>
that should leave it open during teardown but I don't know if other oddities might happen
<enebo>
oh interesting
<headius>
yeah kill pretty much is the toggle
<headius>
in both JRuby and MRI it just enqueues a magic event that tells the thread to shut down
<enebo>
it is definitely more terse than writing explicit closing logic :P
<headius>
you could give it a try
<headius>
obviously the thread killing is a much larger concern we need to discuss
<enebo>
yeah I will see
<headius>
close_on_exit would be a nice short-term fix for them
<headius>
sorry I think that's autoclose = false
<headius>
yeah I think autoclose=false should fix the "closed stream" issues
<enebo>
yeah it seems to only be selector issues now
<headius>
ace
<headius>
I think setting autoclose = false isn't even that odd since they create that fd through other means
<headius>
it's not a normal IO
<enebo>
yeah I am at 7 runs and every one is a selector issue
<headius>
for_fd etc
<enebo>
yeah the to_io method itself is reason enough for it
<headius>
right
<enebo>
I did think that it was originally a bug of something calling to_io and closing it explicitly from within Rails or notify
<enebo>
until I started reading how it was used
<enebo>
but it is a sensible precaution since someone could do it
<headius>
they do have init logic, so it would be reasonable to have teardown logic
<headius>
the debate is whether making them do it in at_exit is appropriate versus us triggering thread kills
<enebo>
headius: so fix on our side for now is to nuke getSelectorPool().cleanup();
<headius>
well, don't do that
<headius>
it should clean up the selectors nobody's using
<enebo>
for this week I am happy trying to get them to accept on sensible line
<headius>
for selector issue just revert bbrowning commit
<headius>
IO issue we can open and discuss
<enebo>
whether we push further on more explicit management does seem reasonable but I think a 1-liner will be a quick path to a new release
<enebo>
ok I will revert and make sure we don't see the universe explode
<headius>
I think bbrowning patch papered over actual problems with leaking selectors
<enebo>
possibly since fixed!
<headius>
and that case would likely be solved by shutting down threads too
<headius>
since it would interrupt selects, allowing those selectors to get back into pool before teardown
<enebo>
hey there you go...
<headius>
oh there's another angle that might make you feel better about the thread killing
<enebo>
it was a blocked accept() and it can interup
<headius>
it's not actually calling an async java.lang.Thread.throwException or anything
<headius>
all it does is place this kill event into the events for the thread
<headius>
if, say, it's a Java thread that is not doing Ruby stuff anymore, it will never be harmed
<headius>
if it's a Java thread that's doing Ruby stuff, it will see a JRuby kill propagate out of the Ruby stuff
<enebo>
yeah a bit only we look at when we are a point not executing
<headius>
yeah...so the impact to non-Ruby threads should be minimal
<headius>
MRI also waits for the threads to shut down...I think we'd need to ensure the threads actually were started in Ruby code, but that's doable
<headius>
so basically...send kill event to all known threads in the runtime, but only wait for ones we know we started
<headius>
that would roughly match MRI
<enebo>
one bit for terminate and one for restarted
<enebo>
so we still know whether restarted will stop though?
<headius>
restarted?
<headius>
we can't know it will stop obviously...narrowing the waiting to only Ruby-spawned threads would make it more likely
<headius>
it's possible we might split the teardown into two pieces for embedders, so they can ruby.terminate and ruby.finalize as separate steps or something
<headius>
enebo: I'll revert the selector patch
<headius>
oh nevermind, you did it
<enebo>
vroom vroom
<enebo>
I thought you suggested I do that though
<headius>
I was just trying to wrap up my involvement in this thing :-)
<headius>
the other half should get a new issue and we'll talk through it
<headius>
link to previous issue(s) about teardown and threads+io
<enebo>
ok
subbu|lunch is now known as subbu
<headius>
enebo: so if I'm done helping on this one chat with me for a minute about refinements
<headius>
what I've got on "refinements" branch is green and passes a lot more refinement stuff than before
<headius>
but it was fairly invasive
<headius>
my issue now is prioritization
<headius>
I think either way we need to punt this to 9.2.7
<headius>
but I'm not sure if I should continue down the rabbit hole of trying to fix all unresolved supers
<enebo>
yeah .7 for sure
<enebo>
when .6 is out we land that and kares sprintf rework
<headius>
if I we don't fix that there are real world refinement use cases (like those in sequel) that won't work
<headius>
but I don't know how long the tail is
<enebo>
and then look into require via ruby....probably land your PR at that point as well?
<headius>
require via ruby, what's that?
<headius>
the autoloaf thing?
<enebo>
yeah let me pull up the issue you wrote a PR for it
<headius>
heh
<headius>
I think I'm going to call it autoloaf from now on
<headius>
we can land refinements branch any time you want
<headius>
extra work is on refinements2
<enebo>
yeah I think right after .6 is out which for me is when rb-inotify issue is resolved enough
<enebo>
which may be by tomorrow
<enebo>
so .7 will largely be autoload/refinements
<enebo>
and a few miscellaneous things
<enebo>
so in that vein we should figure out if that jives well with Rails 6
<enebo>
and related deps
<enebo>
dr-itz has really made short work of AR work
<headius>
the autoload thing looks like it's probably pretty good
<headius>
the only real failure I can see in the PR is some spec that didn't expect require to get called but it was called zero times
<headius>
that may be a side effect of how we're managing the loaded features, which is a whole other piece of work to be done
<headius>
and multiple requires is not a problem in practice
<enebo>
headius: I don't remember if I added the bootsnap stuff to that
<headius>
I'll rebase and push and see how it looks
<headius>
ah the bootsnap thing is another matter
<headius>
I like my solution for that best
<enebo>
headius: but how we handle native exts relative to MRIs handling of cext could make us fit into it but it is another strand in the sweater of load
<headius>
it's not native exts in general, it's just the built-in ones
<enebo>
oh right yeah I already forgot
<headius>
"my solution" = stop making them built-in and add .rb files that boot them
<enebo>
yeah not unreasonably but it may caost a little IO time
<enebo>
not sure it matters though
<headius>
so the only stuff that would go through require would be those .rb files, and they'd use JI to boot the actual exts
<enebo>
we don't have that many
<headius>
since we don't want to make them into jars
<headius>
the ones loading through jars should be fine in bootsnap
<enebo>
and they only get loaded if we use them
<headius>
right
<headius>
it should actually reduce startup, but builtins are all super lazy right now anyway
<headius>
it's an easy fix though
<enebo>
headius: seems reasonable to me. I suppose the only aspect is new internals should follow same convention
lucasb has quit [Quit: Connection closed for inactivity]
<headius>
right...the "builtin" registration stuff should just go away
<enebo>
headius: so maybe we should also update our wiki on making internal extensions of something?
<headius>
oh, it will eliminate one step during load service searching too
<enebo>
ah yeah
<enebo>
so Java will be simpler
<headius>
nobody doing third-party exts is using this
<headius>
it has never been documented anywhere and just grandfathered in from pre 2004 code
<enebo>
makes me think back to thoughts on pre-indexing jar contents so we don't need to search
<headius>
I could fix it in an hour if you want
<enebo>
well it is for .7 but if it is an hour then .6 seems reasonable. I don't really see a risk
<headius>
ok so this comes back to my prioritization
<enebo>
how about changing load path so something else loads
<headius>
happy to do this or happy to work on super
<headius>
but I think rails-supporting things should be priority for now
<enebo>
as far as I am concerned this is more important but only because I don't know if we need super refinements for rails 6
<headius>
there are some non-refinement failure cases for super but they're nothing new
<headius>
yeah for the moment I don't believe we do
<enebo>
I agree with you though Rails is our big ticket for a while
<headius>
and all non-include/prepend refinement logic should be working now
<headius>
with refinements branch
<headius>
actually most include/prepend logic works too...but prepends and includes *into* a refinement are aproblem
<enebo>
so let's take our classic approach and stop once we move past current obstacle and take up new thing until we hit our goal
<enebo>
boiling the ocean of a feature is a nice luxury and may save some time in the long run but working with what people use trumps that purity
<enebo>
and I suspect you are getting sick of refinements
<headius>
well having the MRI structure in place was the big hurdle
<headius>
there may be a better way, and I have not tried to optimize this year, but it's at least guaranteed to be able to match their features
<headius>
beyond that we need to be able to pass caller's scope through to any Java-based core method, and then I think it's just a matter of creatively caching
<headius>
enebo: does master compile for you?
<headius>
I'm getting error in SelectorPool from your revert
<enebo>
uh oh
<enebo>
:)
<headius>
ah yeah
<headius>
you didn't remove the openSelectors.remove line
<enebo>
lopex: you know this guy practiced this talk
<lopex>
enebo: who cares ?
<enebo>
lopex: he does
<lopex>
enebo: did you know about this feature in perl6 ?
<enebo>
lopex: no
<enebo>
lopex: did he say what parser tech it is?
<lopex>
yeah, one could explain that in like two slides
<enebo>
lopex: like LALR?
<lopex>
I dont recall
<headius>
enebo: I've pushed the bootsnap builtin changes
<enebo>
lopex: you are thinking about this from standpoint of knowing how to do it anyways
<enebo>
headius: to master?
<headius>
no_builtins branch
<enebo>
ok
<headius>
if it's green, though, it should be fine
<enebo>
I am still hunting down the finalizer causing us woes
<enebo>
yeah it seems very reasonable
<enebo>
I think though a wiki page and a comment in builtins area of Ruby.java may help future builtin work
<headius>
hmmm
<headius>
--dev -e 1 went from 1.37s to 1.1s
<enebo>
fuck yes
<enebo>
:)
<headius>
big enough I worry I broke something
<enebo>
you found our dark matter
<headius>
but test:jruby and spec:ruby:fast pass
<enebo>
lopex:^^^ numberz
<lopex>
enebo: yeah, somewhat limited use, since you have so much tools that are reasonably easy to use
<lopex>
headius: NUMBERS
<headius>
maybe you can confirm this
<headius>
it uses WAY less system time
<headius>
like 0.5s
<lopex>
but hwy ?
<enebo>
lopex: yo dawg we put Perl code in your Perl code to make some other Perl code
<lopex>
enebo: in even more insane syntax
<enebo>
lopex: you get all the power of Perl in your impl at the speed of Rakudo
<enebo>
lopex: sounds like something someone who loves perl would be into
<lopex>
enebo: and, as you said, how it compares to plain parser combinators
<enebo>
more power to them. If I was perl guy and this made it easy for me to make DSLs or something I would be into it
<enebo>
lopex: I am surprised parser combinators are not more of a thing
<lopex>
afaik it's performance ?
<enebo>
lopex: yeah but the Haskell people supposedly made up much of that ground (if I remember the "claims")
<enebo>
lopex: but good luck finding anything online about whether that is true
<lopex>
you can shift/reduce for combinators efficiently ?
<enebo>
lopex: but performance is not everything
<enebo>
lopex: and it may be good enough and get benefits like testing partial grammars of incremental parsing (for compilers/IDEs
<lopex>
and purity too right ?
<enebo>
lopex: if you say so :P
<lopex>
just guessing
<enebo>
lopex: but yeah they are pure functions or should be
<lopex>
always has hard time wrapping my head route perf for like (parser1 + parser2) { action }
<enebo>
lopex: but I don't know if you would be forced to do that unless you were in a language which forced purity
<lopex>
*around
<enebo>
lopex: I cannot recall the name but there is a technique for recompiling n expression over the same string so long as they have "properties" so things like "foo".map { ... }.map { ... } can end up as a single traversal
<lopex>
btw onigmo, has waken up
<enebo>
lopex: so who knows how much stuff happens in rich type systems internally
<lopex>
how does the type system really help ?
<enebo>
lopex: I don't know...just figuring out it is always two strings and the filters of the maps are somehow congruent...waves hands
<enebo>
lopex: just knowing types generally is a path to doing more stuff...where types may be static or just determined by the runtime as having something in common
<enebo>
knowing a string is always a sequence of bounded bytes means you can iterate and understand storage constraints etc...
<headius>
well that's one way to break the build...I forgot to include stringio.rb
<enebo>
woo hoo who needs it!
<enebo>
ok well I am done tonight. I will figure this finalizer first thing in morning
<lopex>
enebo: hmm, it can help productions, but not strings themselves ?
<enebo>
lopex: I am going away now
<enebo>
lopex: but I don't really understand your question but I recommend looking for optimizing parser combinators
<enebo>
lopex: I read a little bit when I was trying to learn Haskell but very little
<enebo>
lopex: I was interested in them and people said they were slow and Haskell people said they made them faster???
<enebo>
so there are some papers on this or at least web pages
<lopex>
enebo: I dont want to
<enebo>
I was mostly bullshitting above by trying to illustrate another technique where people reduce the k of O(kn)
<enebo>
night
<lopex>
night
<lopex>
headius: so why is the startup so much faster ?
<headius>
lopex: beats me
<headius>
I suspected it might be slightly faster because it doesn't have to register all the "builtin" libraries, but that's just a bit of hash manipulation
<headius>
I did clean up what gets loaded when so maybe I reduced what loads by default
<headius>
and several libraries no longer do a load service search...they go straight to the native bits