<ikus060>
Hello, I have a couple of logstash instances running on JRuby and I notice a lot of non-heap memory usage. Like 1GiB, while I'm uside to Java problem using ~64Mb of non-heap.
<headius[m]>
ikus060: hello there
<ikus060>
Could any body explain if this is expected ?
<headius[m]>
JRuby does use more non-heap, but 1GB might be a bit much depending on the app
<headius[m]>
are we talking all non-heap, like including JVM metaspace, or really non-JVM native memory?
<headius[m]>
I guess I am asking how you determined this and what is included in "non-heap"
<ikus060>
headius[m]: I wish I knew what is include in non-heap. I'm getting this number from logstash metrics API.
<ikus060>
I'm trying to get more data about that non-heap from CLI but a very few tools provide detauls regarding the non-heap
<headius[m]>
do you know what version of JRuby it is using?
<headius[m]>
I would generally say you should contact logstash folks about this... we work to reduce our own use of non-heap memory with every release but I do not know how up to date they are
<ikus060>
headius[m]: I think I have a lead
<ikus060>
it seams related to apache derby running in the JVM
<ikus060>
so unrelated to JRuby
<headius[m]>
aha that is very interesting
<headius[m]>
so it is using some off-heap data store
<headius[m]>
ikus060: is this something special or just a normal part of logstash? We have had others report large off-heap memory in logstash and this is the first time I have heard about Derby
<ikus060>
It's something special in our logstash pipeline
<ikus060>
I've compare a couple of logstash pipeline and non-heap memory usage and I can correlate a bit
<ikus060>
It's related to jdbc_static plugin using Apache derby
<ikus060>
I'm guessing Apache Derby is using alot of non heap
<headius[m]>
Great investigation! I am sorry that it is not something we can fix but at least you know now
<ikus060>
Thanks alot of your help !
<headius[m]>
enebo: I am stuck at dirgra now fyi
<headius[m]>
struggling through module options and getting JRuby to have a proper module-info... need dirgra to be a module to properly include it
<headius[m]>
i may have worked around it by using 'dirgra' as the module name for now
ikus060 has quit [Quit: Leaving]
caleb_land[m] has joined #jruby
<caleb_land[m]>
Hi everyone. I have been struggling trying to create a RubyHash in Java that I can pass back to my Ruby code. I've created hashes with RubyHash.newHash but there doesn't seem to be an equivalent RubySet.newSet static method (in the source I see a newSet method that is an instance method). Am I missing something?
<headius[m]>
caleb_land: hello!
<headius[m]>
I think you mean you are struggling to create a RubySet?
<caleb_land[m]>
@headius Yeah, can't seem to create one from Java.
<headius[m]>
caleb_land: that newSet seems like a mistake... there is no reason that should be static
<headius[m]>
I mean no reason it shouldn't be static
<headius[m]>
so that is a bug-ish
<headius[m]>
or at least there should be another static method
<headius[m]>
if you could file a bug for that... I guess we have not had a need to create a set from Java so far so never noticed it is missing
<caleb_land[m]>
headius Okay, I can file a bug and create a PR. I haven't used Java in 15 years but it shouldn't be too hard
<headius[m]>
for you to work around this... I guess do what the newSet does basically
<headius[m]>
oh but some of those methods are hidden 🤔
<caleb_land[m]>
@headius for now I can pass an empty set in from Ruby, but I'll file that bug
<headius[m]>
the simplest way for you to make this work might be to just run the ruby code... require 'set' and then Set.new
<headius[m]>
Can you explain what you're implementing? These APIs are generally only used if you are writing an extension for JRuby
<caleb_land[m]>
This is crazy, but I'm trying to take an set from a Clojure function and change it into a Ruby set. I'm using a datasource that has a Java API but sometimes returns Clojure data structures
<caleb_land[m]>
(the database is Crux... it's similar to Datomic if you're familiar)
<caleb_land[m]>
But I want to use this datasource from my Rails application
<headius[m]>
aha
<headius[m]>
I am familiar with Datomic, ok
<headius[m]>
well my first suggestion would be to just leave it as whatever set it was originally since we should be able to use it
<caleb_land[m]>
That makes sense, rather than converting back and forth, thanks
<headius[m]>
sure... if you can at least file a bug for that set thing that would make sure we add something for the next person
<headius[m]>
PR is great but I understand if you won't need it yourself now
<caleb_land[m]>
Okay, sounds good. I'll file that since it might be helpful for someone in the future. Thanks for your help!
<headius[m]>
thank you!
<headius[m]>
enebo: so I just tweeted about this but we may have a problem with two classes named the same (case insensitive) as a sibling package
<headius[m]>
org.jruby.Main and org.jruby.runtime.CallSite
<headius[m]>
gonna have to back off on this for now because nobody can explain why I am getting this error
<enebo[m]1>
headius: I released a modularized dirgra this morning
<enebo[m]1>
0.4
<headius[m]>
rad
<enebo[m]1>
I had issues with maven and getting release to work since I always manually did it before
<headius[m]>
I did manage to get further with the automatic names (which are based off jar filename) but that class/package name conflict has stopped my progress
<enebo[m]1>
I do not know I did the module file correctly but I think I did ... it is a single package module
<headius[m]>
I pushed a commit that does all the things... copies dependency jars to lib/modules, changes jruby.bash to use module path, adds a module-info.java that loads the right modules, but then this compile error mystery comes up
<enebo[m]1>
headius: that is a strange requirement of modules
<headius[m]>
nobody I have talked to knows about it