<rtyler>
headius[m]: I made some progress, it looks like things are blowing up because somewhere along the line a RubyBasicObject is serializing a singleton, any environment variables I can set to get some more debugging output here? I haven't yet figured out how to attach a debugger to a Spark instances running this code
_whitelogger has joined #jruby
krainboltgreene has joined #jruby
<krainboltgreene>
What can I do to reduce the amount of backtrace noise in my jruby application?
<krainboltgreene>
Also, while I'm here, don't suppose anyone knows what causes this exception? Sequel::DatabaseConnectionError: Java::JavaNioChannels::UnresolvedAddressException:
<rtyler>
broken DNS or a bad hostname?
<krainboltgreene>
Ah, so network issue?
<rtyler>
most likely
<krainboltgreene>
Cloud SQL forces me to use unix sockets, so me misconfiguring is likely the cause
<rtyler>
O_o
<krainboltgreene>
Apparently it's faster, but likely not why they do it.
<krainboltgreene>
Alright, I tried it with the cruby pg gem and it worked. I think this might be a bug with jruby-pg.
<rtyler>
quite possibly, doing unix sockets in JRuby is a little awkward because the JVM hates unix sockets :)
<rtyler>
I believe it requires a jnr library to do a passthrough to native APIs
bga57 has quit [Quit: Leaving.]
bga57 has joined #jruby
Liothen has quit [Ping timeout: 258 seconds]
Liothen has joined #jruby
rusk has joined #jruby
shellac has joined #jruby
drbobbeaty has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
ur5us has joined #jruby
ur5us has quit [Quit: Leaving]
drbobbeaty has joined #jruby
shellac has quit [Quit: Computer has gone to sleep.]
shellac has joined #jruby
krainboltgreene has quit [Quit: Connection closed for inactivity]
<rtyler>
interestingly it looks like serialization of procs might be what's causing this blog up
<rtyler>
blog up? more like blow up
rusk has quit [Remote host closed the connection]
<headius[m]>
rtyler: that would make sense. procs don't marshal
<rtyler>
but class objects and things like that doo correct?
xardion has quit [Remote host closed the connection]
<headius[m]>
rtyler: oh now I'm putting some things together
<headius[m]>
you mentioned making lambdas from Ruby
<headius[m]>
this is an interesting flaw in our conversion of procs to interface impls...I guess this is why they had to make lambda state serializable
<headius[m]>
not saying this is why but I'm guessing
<rtyler>
this is where I get confused, I'm aware of some of the lambda magic that happens with procs, is there any way to define what would be consider a JVM native lambda to try to stuff this through the serializer?
<headius[m]>
@kar
<headius[m]>
bleh
<headius[m]>
kares: enebo got some thoughts on JI improvement we need
<headius[m]>
in the bug I just looked at I had to recommend someone use .java_class to get access to a field broken by the accessibility check
<headius[m]>
I've put off for a long time that we need to make JavaClass and JavaField etc and the equivalent methods just return java.lang.reflect stuff
<headius[m]>
I think most of it can duck-type but there's a lot of classes to fiddle with
<headius[m]>
rtyler: so currently the only easy way would be to make a Ruby class that implements the interface
<headius[m]>
because methods get attached to a class and objects can serialize by pointing at a class
<rtyler>
both are pretty old
<headius[m]>
procs can't serialize because there's no way to point at the code associated with them once they get to the other side
<rtyler>
I see
<headius[m]>
it's just this anonymous ball of code
<headius[m]>
lambdas get around this because they're compiled as hidden static methods on the same class
<rtyler>
whaaaaaa
<headius[m]>
invokedynamic stitches that method into an implementation of the interface at runtime
<rtyler>
wacky
<rtyler>
I think where I was getting caught up with the put-the-method-on-a-class problem was because the interface that needs to be implemented is generic
<rtyler>
manpage says that no more records in database, or the buffer being too small are the only two errosr
<headius[m]>
rtyler: generic interface shouldn't cause any problems
<rtyler>
spark begs to differ :P java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ of type scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD
<rtyler>
which is interesting, because serializing the proc is causing a very different exception, but this makes me wonder if one is getting further than the other
<headius[m]>
looks like it implemented a different interface
<headius[m]>
rtyler: when you pass a Ruby object or proc to a Java method that takes an interface, we just implement that interface...so I'm not sure where this would be coming from
<headius[m]>
could be a bug in how we do that but in general we just implement what the method wants
<rtyler>
well, the method is overloaded, lemme pull up the java doc
<rtyler>
I can easily imagine this getting confused
<rtyler>
I'm wondering if this FilterFunction<t> versus scala.Function1<T, Object> could be confusing (assuming this is where things are getting hung up)
<rtyler>
I think this is actually further along than before though, sincne the stack trace is from the readObject part of the tree, so it must be going through the 'WriteObject' cleanly, so perhas it's not the interface at all, but some other ruby objects that are being deserialized and looking confusing
<headius[m]>
hmmm
<headius[m]>
ah well I could definitely see this all being the same issue with a proc implementing an interface and not serializing right
<headius[m]>
and then it's some data object on the other side that no longer fits into that field
<rtyler>
ah, could be I spose
<rtyler>
like I mentioned yesterday, I still haven't figured out a sane way of attaching a debugger to this process
<headius[m]>
I'm not sure how we can make procs serializable portably 🤔
<headius[m]>
rtyler: Ruby or Java debugger?
<rtyler>
what do you mean by "portably"
<rtyler>
Java debugger
<headius[m]>
well we could marshal a proc with some JRuby-specific pointer to the compiled code, or even JRuby-specific serialized compiled code
<rtyler>
well, attached the debugger in intellij, but will take me a while to figure out how to set a breakpoint here :P
<lopex>
headius[m]: it happens when ldap port is a string
<lopex>
the question is if those other issues are related to that
<lopex>
headius[m]: and it works on windows too as a string
<lopex>
so it goes down to a socket and there's a difference for alpine only ?
<lopex>
I'm confused
<rtyler>
iirc alpine uses musl and debian is using glibc
<rtyler>
we've seen tons of issues with different applications which boil down to wacky things with musl and decided to just say "fsck this" and based all our images off of centos
<rtyler>
(at $workplace)
<lopex>
but the port thing indeed fixed the issue
<lopex>
and only on alpine
<lopex>
er, only alpine affected
<headius[m]>
that page took forever to load
<headius[m]>
there was a comment from me on that other bug that newer arch switched to glibc at some point, dunno if that's true but arch is probably configurable both ways
<headius[m]>
I am annoyed by musl only because it's annoying that glibc didn't export all the standard posix symbols from day 1
<headius[m]>
we've had this problem with stat too...glibc doesn't have any exported "stat" or "fstat" symbols, they're macros around internal __xstat64 crap
<lopex>
it was hell of misleading error too
<lopex>
so maybe that exception could provide more info like service name etc ?
<headius[m]>
yeah that could be improved
djellemah has joined #jruby
subbu is now known as subbu|lunch
subbu|lunch is now known as subbu
subbu is now known as subbu|lunch
<headius[m]>
I hate this code
djellemah has quit [Ping timeout: 250 seconds]
<lopex>
jnr ?
subbu|lunch is now known as subbu
<rtyler>
heh
drbobbeaty has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
KarolBucekGitter has quit [Ping timeout: 252 seconds]
KarolBucekGitter has joined #jruby
<rtyler>
headius[m]: to give you an update from some poking between other work, it looks like some other objects serialize just fine (e.g. String) but a custom defined object (not implemneting an interface) fails on serialization, while a custom object which implements an interface serializes, but something goes awry deserializing
Puffball has joined #jruby
<lopex>
rtyler: why are those procs such an issue ?
<rtyler>
preliminary digging indicates that this error I pasted earlier about a ClassCastException might relate to a mismatch on the classes available in the classloader
<rtyler>
lopex: spark serializes objects to send over to worker nodes and then aggregates them back together
<lopex>
but you mentioned about sarialisable
<lopex>
ah
<lopex>
ah, so whole part of heap is serialized ?
<rtyler>
the specifics of it I genuinely do not understand
<lopex>
like what are the rules ?
<lopex>
is it like a java session thingy ?
<rtyler>
I'm really not sure to be honest, there are some lines in the sand I'm tripping over here, and I certainly don't see them yet
<lopex>
but spark docs might mention some things must be serializable ?
<rtyler>
based on what I'm seeing on stackoverflow about this ClassCastException, it looks like things might be serializing properly, but on deserialize the right jruby jars are not in the class path
<lopex>
so it's not a ruby thing right ?
<lopex>
well
<rtyler>
correct
<lopex>
so like, jruby either created some proxy or runtime interface that intervenes ?
<lopex>
runtime, I mean runtime generated
<lopex>
though according to java spec then shuldnt pose any issues
<lopex>
if an inferface is serializable is should be ok rhight ?
<rtyler>
should be I suppose
<lopex>
hmm
<lopex>
may be, that via invoke dynamic some things pass without general verifier ?
<rtyler>
so it's pretty clearly on he deserialization side, my hunch is that whatever machinery does the deserialization is not finding everythiing it wants in the classloader
<lopex>
pretty vague stack trace
<lopex>
yeah
<rtyler>
I am baking a fat jar with jruby-complete and my code within it, so it's not clear to me how this might be failing
<rtyler>
unless it's just not finding _any_ of that on the destination node
<lopex>
either some lowe lever swallows, or indeed it's a verifier bug
<lopex>
noe idea
<rtyler>
got system property handy for disabling the verifier?
<rtyler>
I can try it out
<lopex>
ah
<lopex>
yeah
<lopex>
jruby libs are by default non verifed
<lopex>
let me recall
<rtyler>
headius[m]: our discussion about the proc interface getting confused mapping into the right interface class, I think that's exactly what's happening! This is the equivalent java code failing to compile https://gist.github.com/rtyler/1984c7c9e62146b9a09aa82842523ea5