claudiuinberlin has quit [Ping timeout: 258 seconds]
drbobbeaty has joined #jruby
<rtyler>
headius: double checked with a colleague, the code is sent over to each worker node with the `spark submit`, so the jars exist everywhere. I _think_ in this case where I've got my Ruby defined class, the only problem is that it is serialized with a name that doesn't exist elsewhere
<headius[m]>
rtyler hmm there is a flag that switches JRuby from using generated code to using a java.lang.Proxy to impl interfaces, maybe that's another path
<headius[m]>
Might require tweaks, but fewer tweaks
<rtyler>
reflected.handles=true?
<headius[m]>
No there's one about proxy
<headius[m]>
I wonder if it didn't get into the options output
<rtyler>
oh, I'm reading the wiki page
<rtyler>
I tried that setting with consistent.hashing to no effect fwiw
<rtyler>
interfaces.useProxy=true looks like what you're referring to, trying
<rtyler>
I'm thinking the best next step would be consistent class naming between serialize/deserialize
xardion has quit [Remote host closed the connection]
rusk has quit [Remote host closed the connection]
xardion has joined #jruby
<headius[m]>
The problem is probably not consistent naming as much as it is being able to find the class
<headius[m]>
we generate these interface implementations into their own classloaders, so there's nowhere to look it up
<headius[m]>
I have been talking with enebo about using fewer classloaders recently
<lopex>
something like per callsite group/unit ?
<headius[m]>
lopex: maybe, but that would still need to be consistent for this serialization issue
<headius[m]>
a single classloader for all interface impls would work but what's the risk?
<lopex>
Kernel::load might be one, since it can redefine lots of things
<headius[m]>
we generally have used isolated classloaders for all code generation that happens at runtime
<headius[m]>
lopex: currently we key the name of the generated class off the name of the Ruby class plus a hashcode of the interface names
<headius[m]>
if the name of the class and the set of interfaces is the same it shouldn't regenerate a new one
<rtyler>
headius: I'm out of town riding a bicycle next week, but after that I can probably find some time to jump on a Google Hangout and pair for a little while if you're in one location :)