00:00 <lopex> scala is just mangling

00:01 <rtyler> indeed: /

00:01 <lopex> scala doesnt any weird classloader trick

00:01 <lopex> *doesnt do

00:01 <lopex> jruby does though

00:03 <rtyler> I need to apply beer to this problem, bbiab

00:03 <lopex> as usual

02:25 <rtyler> alright, beered up

02:42 <rtyler> still keep coming back to this classloader/classpath looking issue

02:42 <rtyler> but unable to prove it :(

02:48 <rtyler> java.lang.ClassNotFoundException: org.jruby.gen.BeeFilter_799252494

02:49 <rtyler> this looks more promising! I tried a different function passed through, and this came out the other end, which looks like a much more clear class lookup error on the deserialization side of things

03:22 <headius[m]> ClassNotFoundException is likely because it is not able to fine the generated class

03:22 <headius[m]> I hate stuff that relies on serialization

03:23 <headius[m]> It's not impossible to make the generated classes match on both sides but there's work required

03:23 <headius[m]> A short term change might be to give users a way to specify the name of the generated class

03:24 <headius[m]> Ideally as long as you have the same code on both sides it would work

03:24 <headius[m]> What's on the other side?

03:25 <rtyler> sorry, was in transit

03:25 <rtyler> the otherside is just the same spark environment

03:26 <rtyler> in local testing it's actually just another thread in the same process, but to replicate the real-world environment it's likely connecting over TCP to a different classloader,etc

03:26 <rtyler> in "the real thing" the spark cluster would be a JVM of N different instances

03:29 <rtyler> headius[m]: I'm not sure specifying a generated class name would work terribly well, unless that was something that would be predictable for every load of the jar (for example)

03:29 <rtyler> so I guess class method like java_serialize_as "MyClass_Ser'

03:29 <rtyler> or something, i don't know

03:33 <headius[m]> I'm actually now a bit surprised that it is serializing as the generated class...Ruby objects should serialize using marshalling, which would just use the normal class name from Ruby

03:33 <headius[m]> That might be the right thing to do in either case since you can't add interfaces later

03:36 <rtyler> and org.jruby.gen.BeeFilter_799252494 is definitely a generated class?

03:36 <rtyler> the deserialization stacktrace for what it's worth: https://gist.github.com/rtyler/bfd264a5ef0e9974864f49612dc1b3eb

03:37 <rtyler> this train wifi tends to sever my SSH connection so I might pop in and out

03:37 <headius[m]> Yes, the numbers are a hash of a collection of the Java interfaces it implements and BeeFilter is the name of the Ruby class

03:38 <headius[m]> But that's where maybe it makes sense to just use the Ruby name

03:38 <headius[m]> And to generate them into a classloader structure that would allow looking it up the same way on the other side

03:40 <rtyler> you're not going to find your way in the SF bay area any time soon? might be interesting to hack this out

03:41 <rtyler> let me commit some of these changes and push them with an example on how to test in case you're curious

03:45 <rtyler> https://github.com/jruby-gradle/redspark/blob/master/simple.rb#L3 fwiw

03:46 <rtyler> in between work things tomorrow, I think I'll be able to connect the remote debugger and set some breakpoints for the serialization side to see what it's sending and what the deserialization is getting on the other side

03:49 <rtyler> if you do decide to poke at it, you'lll want to tweak run.sh to add a reference to your own spark download

04:18 <rtyler> it looks like some of these other attempts skipped this problem by simply taking the path that pySpark takes which is to spin up a custom python process that receives pickled objects (or something like that) for the data transport, but otherwise doesn't attempt to serialize objects

04:19 <rtyler> i.e. function or class objects

10:27 <kares[m]> hey! CI on Travis is passing (last couple builds) but if I open details a MRI stdlib target gets terminated (due to a hang) ... not sure its still the right call to call the build green :)

10:27 <kares[m]> checking latest or the one before: https://travis-ci.org/jruby/jruby/builds/534921947

13:46 <headius[m]> Ugh

13:47 <headius[m]> Oh it is marked as allow because of that weird hanging at the end

16:00 <rsim> headius: Hi! I found one critical regression in JRuby 9.2.7.0 regarding loading of compiled Ruby class files. Could you take a look? https://github.com/jruby/jruby/issues/5740

16:18 <enebo> rsim: I see the issue...will be fixed shortly

16:25 <rsim> enebo: I was looking at the change history and identified that probably this change introduced the problem https://github.com/jruby/jruby/commit/fc4797ad8b7c3a82c4c83710ff6a2342d2080136

16:25 <rsim> enebo: Or do you already found the cause?

16:25 <enebo> rsim: yeah it is not encoding the isPotentiallyRefined on encode() but is asking for it on decode()

16:26 <enebo> jbvjb

16:27 <rsim> enebo: BTW what are your plans when JRuby 9.2.8.0 will be released? I am now thinking if we should stay on 9.2.6.0 for some while or wait 9.2.8.0

16:27 <enebo> rsim: we originally said by end of month but I think we need to discuss that a little bit

16:28 <enebo> we wanted zeitwerk working as one bigger item so we need to think on how important that item is now since it is off by default in rails 6 for us already

16:28 <rsim> ok, then we will stay on 9.2.6 now and will test again when 9.2.8 will be released

16:29 <enebo> actually the encode has more wrong than that boolean but simple enough to fix

16:29 <enebo> most time will be making sure we have a test for asString

16:50 <rtyler> o/

16:54 <lopex> numbers ?

19:44 <rtyler> lopex: was that question directed my way?

19:44 <lopex> no, it's our old banter word here

20:01 <rtyler> I saee