lucasb has quit [Quit: Connection closed for inactivity]
umairsair has quit [Read error: Connection reset by peer]
djellemah has quit [Ping timeout: 244 seconds]
krainboltgreene has joined #jruby
<krainboltgreene>
Hello hello. I'm running jruby (an http server, specifically) in a very memory constrained environment. Currently my usage is between 256mb..512mb, it's hard to tell. Are there any jruby specific ways to (at a high level) analyze memory usage?
<krainboltgreene>
Ah, looks like the default memory for jruby is 500mb, which explains things.
rusk has joined #jruby
krainboltgreene has quit [Quit: Connection closed for inactivity]
drbobbeaty has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
shellac has joined #jruby
umairsair has joined #jruby
umairsair has quit [Ping timeout: 252 seconds]
umairsair has joined #jruby
drbobbeaty has joined #jruby
shellac has quit [Quit: Computer has gone to sleep.]
umairsair has quit [Read error: Connection reset by peer]
umairsair has joined #jruby
<headius[m]>
There's several ways to memory profile but the most useful for your case would be to get a heap dump and see what's in memory
<headius[m]>
I can run a simple rails app in under 500MB but it's a little tight. We do want to reduce that but a large portion is off-heap data the JVM produces
<rdubya[m]>
headius: we're trying to figure out ways to reduce our off heap memory, right now our heap is set at 768 MB but we have to give 2 GB to the container to keep it from getting OOM killed, is that typical? Are there any things that you can think of off the top of your head that I could look at first?
<headius[m]>
Are you running with Indy on?
<headius[m]>
Perhaps this is a good week for us to gather some information from both of your applications. We have spent lots of time profiling the heap, and that seems to hold steady at a decent size, but just recently noticed how large off off-heap
<rdubya[m]>
no, we have it off at this point
<rdubya[m]>
I don't remember what specificly was the issue, but when we were load testing we got better performance without it
<headius[m]>
One thing we suspect is the fact that we generate so many classes into their own class loaders, and that appears to bump metaspace way up because it's allocating too much memory for each class loader
<rdubya[m]>
we had to adjust metaspace because we were running out of room there
<headius[m]>
I forget the numbers that Tom mentioned, but it appears that metaspace assumes you will have four or eight or some number of classes in each classloader and allocates a fixed chunk of memory in preparation for that
<headius[m]>
I am on mobile at the moment, but there are some flags at the jvm level, and also external tools that you could start gathering info from
<rdubya[m]>
cool, yeah I'm not actively digging in right now, just saw there was a conversation about it going on so figured I'd join the topic, whenever you get a chance is cool
<headius[m]>
This is very timely, because Tom saw rubygems.org app consuming something like 1.5 GB for a 400 MB heap and we were a little surprised
<headius[m]>
At first we thought invoke dynamic was the cause, but even with it off it still gets well into 1 GB
<headius[m]>
Good god I just saw your stacktrace flag
<headius[m]>
What's that all about
<headius[m]>
One thing I am going to try is allowing generated classes early in execution to all go in the same class loader, which should cover most generated objects and jitted methods without requiring separate class loaders for each
<headius[m]>
We also are not using the hidden anonymous classloader feature that came with invoke dynamic. I think we need to access unsafe to do it, but it may be a big help
<rdubya[m]>
I forget why we had to do that lol, I think we just made it arbitrarily big because we were running into issues where the stack trace wouldn't include the whole thing when we were trying to track down some well hidden bugs, its probably not necessary anymore
<rdubya[m]>
also, we're unfortunately still running on 9.1.17, we have some gem compatibility issues with 9.2 that we haven't taken the time to dig into yet
<rdubya[m]>
kind of figured we'd deal with that pain when we finally moved from rails 4.2 to 5
<rdubya[m]>
should 9.2 be able to run rails 4.2? If so I can file a bug, otherwise I'll wait until I get to using it with rails 5
djellemah has joined #jruby
djellemah has quit [Ping timeout: 246 seconds]
djellemah has joined #jruby
djellemah has quit [Remote host closed the connection]
<headius[m]>
I don't see why not
<headius[m]>
Unless there's something in Ruby 2.5+ that breaks it
<rdubya[m]>
ok, i'll file a bug, basically when the gem is required it causes a stack overflow
<headius[m]>
Worth a try on 9.2.7+ with reworked super calls
<headius[m]>
That fixed a bunch of places where we could SO due to exotic class structures
<rtyler>
hrm, in 9.2.7.0 is the jruby-openssl bundled the latest version of the gem?
shellac has joined #jruby
<headius[m]>
I believe so
subbu is now known as subbu|away
djellemah has joined #jruby
<enebo[m]>
rdubya: Depending on concurrency needs using jemalloc and tuning arena size down can also help offheap memory (although we are going to try to do something on our side to deal with the eccessive metaspace alloaction)
<enebo[m]>
I think even on newer glibc you can environmentally turn down arena size but not positive. If you plan on using many threads though reducing arena can reduce perf
xardion has quit [Remote host closed the connection]
xardion has joined #jruby
subbu|away is now known as subbu
djellemah has quit [Ping timeout: 246 seconds]
shellac has quit [Ping timeout: 252 seconds]
rusk has quit [Remote host closed the connection]
djellemah has joined #jruby
<headius[m]>
enebo: did you post that off-heap output somewhere?
<headius[m]>
we should look into other tools to get additional info about what all that stuff is
<headius[m]>
I believe async-profiler can track some off-heap stuff
djellemah has quit [Ping timeout: 258 seconds]
<rtyler>
well, I found the issue of my openssl issues, we had a test which was using an "older" JRuby for sanity checking, and the "older" JRuby was 9.1.x which didn't properly ysupport jdk11 yet
<headius[m]>
aha well there ya go
<headius[m]>
enebo: heh I changed the JIT code that uses a one-shot classloader to just use Unsafe.defineAnonymousClass and everything seemed to work the same
<headius[m]>
almost unbelievable
djellemah has joined #jruby
<headius[m]>
one positive thing is that our JIT does quiet down pretty quickly
<rdubya[m]>
headius: looks like there isn't an isssue when I use a clean gemset with the latest rails 4.2 on jruby 9.2 so I'm going to hold off on creating a ticket for now
<headius[m]>
yay, perhaps we fixed whatever it was
<headius[m]>
enebo: if I'm reading this right, puma's "process_client" message has 176 basic blocks?
<headius[m]>
it's not trivial but that seems a bit high
<headius[m]>
hard to know what's included in this count but this IRMethod retains 220kB
<headius[m]>
the next six biggest IRMethod range from 109k to 186k
<headius[m]>
not like this is a big portion of our heap use but they seem really large
<headius[m]>
enebo: Back to the experiment at hand, using anonymous classloading *appears* to work fine...things are still jitting and running and the number of OneShotClassLoader instances has dropped down to single digits from around 1800
<headius[m]>
I see no difference in meatspace
<enebo[m]>
hmm but even with that change it is still using metaspace underneath
<enebo[m]>
were you thinking they may do something more compact?
<headius[m]>
yeah making better use of the space in some way
<headius[m]>
it's so similar it almost looks like the only real savings from anon classloading is avoiding symbol tables and avoiding the wasted size of one-off classloader instances
<enebo[m]>
I know I saw some other explanations but I do not see what I was reading yet
<headius[m]>
ahh ok
<headius[m]>
I do not see that kind of ratio...on OS X the visualvm metaspace graph shows about 75% utilization
<headius[m]>
it's possible this anon classloader thing might have more effect on linux but I'm not sure I actually see the problem to begin with on OS X
<enebo[m]>
headius: that is definitely a possibility but it also seems odd at the same time
<enebo[m]>
That SO has a metaspace tester
<enebo[m]>
or the guy made something to keep making classes to measure
<headius[m]>
yeah but he is also doing one class per classloader...there seems a good possibility that cutting classloader out of the equation altogether might use metaspace differently
<headius[m]>
"A class loader allocates space for metadata from its chunks (a chunk is bound to a specific class loader)"
<headius[m]>
so if there's no classloader?
<headius[m]>
¯_(ツ)_/¯
<headius[m]>
¯_(ツ)_/¯
<headius[m]>
oops
<headius[m]>
stupid backslash arm
<headius[m]>
zero to boot this simple app is at 11k classes loaded
<headius[m]>
a few pages of jit log went by but maybe hundreds of methods, not thousands
<headius[m]>
we still have a class per core method too which is a load of a couple thousand classes at worst
umairsair has quit [Quit: Leaving.]
<headius[m]>
15k classes after 30s of siege
<headius[m]>
there's definitely a meatspace gap
<headius[m]>
the gap is much worse for those new 4000 classes versus the 11k that came before
<headius[m]>
at this point the only places using OneShotClassLoader are relating to reified classes, Java interface impl, stuff like that...which should also be in the hundreds
<headius[m]>
that first plateau is after the first 30s of siege
<headius[m]>
the remainder is the next 30s siege and a little quiet time at the end
<headius[m]>
so it eventually plateaus right at this 101/77 ratio pretty consistently
<headius[m]>
with or without anon classloading, it's almost identical
kiwi_1 has joined #jruby
<kiwi_1>
hi
<headius[m]>
Interestingly, the number of loaded classes blips downward a little during the second siege, but metaspace still goes up
<headius[m]>
kiwi_1: hello there!
<kiwi_1>
hey i want play sound file in jruby but It is not
<kiwi_1>
please help
<headius[m]>
it appears that using anonymous classloading on Java 9+ will require some additional work...
<headius[m]>
2019-05-13T14:11:18.641-05:00 [Ruby-0-JIT-1] INFO JITCompiler : Could not compile; passes run: [org.jruby.ir.passes.LocalOptimizationPass@584ed071, org.jruby.ir.passes.OptimizeDynScopesPass@f61f6f63, org.jruby.ir.passes.DeadCodeElimination@805d30ee, org.jruby.ir.passes.OptimizeDelegationPass@6c7a3ce9, org.jruby.ir.passes.AddCallProtocolInstructions@87f2d916, org.jruby.ir.passes.AddMissingInitsPass@693f2559]: <block>
<headius[m]>
Requirement_CLOSURE_1.Requirement_CLOSURE_1 at /Users/headius/projects/jruby/lib/ruby/stdlib/rubygems/requirement.rb:17 because of: "Host class org/jruby/compiler/JITCompiler and anonymous class Users/headius/projects/jruby/lib/ruby/stdlib/rubygems/requirement are in different packages"
<headius[m]>
so the "host class" needs to be from the same package (at least in 11, but presumably in 9+)
<headius[m]>
kiwi_1: shouldn't be too hard...JVM has sound APIs
<headius[m]>
what kind of sound file?
<kiwi_1>
mp3 or wav
<headius[m]>
I believe OpenJDK supports WAV out of the box
<headius[m]>
MP3 will probably require an external library
<headius[m]>
I'm sure there's lots of them
<enebo[m]>
javafx can play mp3 but javafx is not bundled by default on openjdk now
<headius[m]>
yeah that's one option
<headius[m]>
it does live on as OSS
<headius[m]>
looks like there's a few other options...older libs but then again mp3 is largely a solved problem
<kiwi_1>
Exception in thread "AWT-EventQueue-0" org.jruby.exceptions.RaiseException: (NameError) no method 'getAudioInputStream' for arguments (org.jruby.RubyFile) on Java::JavaxSoundSampled::AudioSystem
<enebo[m]>
java.io.File.new "music.wav"
<enebo[m]>
kiwi_1: File.new is Ruby File and not the Java File class
<headius[m]>
ahh
<headius[m]>
does that API take a java.io.File?
<headius[m]>
the Ruby "File" class is not the same thing...but you can create the Java class with java.io.File.new("music.wav")
<enebo[m]>
is pretty much a single java process after a long time
<enebo[m]>
oh not diff
<enebo[m]>
I did something which just printed it out at that moment
<headius[m]>
yeah I mean when it says that 1GB reserved, was that some aggregate of memory chunks it reserved over time or actual reserved memory at that moment
<headius[m]>
because it was really big
<enebo[m]>
jcmd <pid> VM.native_memory summary
<headius[m]>
it looks like it's moment-in time I guess
<enebo[m]>
well I just got a summary when I ran it but I ran it twice...once when I did a single request and then that one after I ran the benchmarking script
<headius[m]>
so nvermind that
<enebo[m]>
that script is long so yeah moment in time after quite a while
<enebo[m]>
like 35 minutes maybe
<enebo[m]>
of pounding on it
<headius[m]>
so yeah the reserved numbers here may be exaggerated
<enebo[m]>
another thing to ponder is that the script does concurrent requests up to 20
<enebo[m]>
well exaggerated but not unreasonable for a busy server
<headius[m]>
I mean this says reserved size for "Classes" is 1.1GB and the whole process is only taking 1.36GB
<enebo[m]>
we have a JIT thread but just wondering if somehow many threads trying to do same stuff at same time could somehow be making more handles/classes
<enebo[m]>
yeah I noticed that these numbers might not completely add up too
<headius[m]>
in theory multiple threads hitting an indy call site could trigger multiple of the same MH tree to be created and compiled and loaded, but only one would win and the other should GC
<enebo[m]>
but this was 384Mb heap
<enebo[m]>
yeah
<enebo[m]>
just thinking outloud
<enebo[m]>
I mean it is a theory in that maybe meatspace is messing up
<headius[m]>
right...I'd like to see what your visualvm graph looks like
<enebo[m]>
It strikes me that most workloads load all their classes up front in Java
<headius[m]>
because that SO post you linked was showing this same graph and had a much different ratio
subbu|lunch is now known as subbu
<enebo[m]>
Even the use of indy is probably pretty light as well
<headius[m]>
your ratio might be similar to mine looking at the vvm meatspace numbers
<headius[m]>
the number of indy call sites that I see bind is not enormous either
<headius[m]>
hundreds sure, maybe thousands
<headius[m]>
not GB worth
<headius[m]>
I am eager to get some reliable numbers here because if it's not meatspace I dont' want to spin wheels here
<enebo[m]>
yeah I am not sure it matters unless you did both at exact same moment either
<headius[m]>
the anon classloader stuff is still useful for eliminating the classloader fluff but so far that doesn't appear to change metaspace size or usage at all
<headius[m]>
so we can ignore that for the moment
<enebo[m]>
3MB from 92 is not a big difference for the sake of something getting missed
<enebo[m]>
haha 15k classes
<headius[m]>
this is only two 30s sieges but it's a much smaller app too
<enebo[m]>
does that even make sense?
<headius[m]>
oh btw the class count with indy enabled was around the same too
<headius[m]>
15k classes make sense?
<enebo[m]>
yeah
<enebo[m]>
seems like a lot
<headius[m]>
I dunno...booting this up without siege at all is at 11k
<enebo[m]>
but I guess it depends on how many things are getting loaded
umairsair has quit [Quit: Leaving.]
<enebo[m]>
amazing
<lopex>
lots of numbers
<headius[m]>
binding core classes probably eats up 2k classes for the invokers
<headius[m]>
those could be shared when the overloads are identical, or other tricks, but we need to bind something
<enebo[m]>
in the old early JIT days were were loading 4-5k classes but that was a simpler time
<headius[m]>
indy handles would not load any classes unless called
<enebo[m]>
we load more Ruby now for Rails stuff probably than at that point
<headius[m]>
but they are slightly slower to start up still
<headius[m]>
and might not work on substrate
<enebo[m]>
I just remember having the tunable for permgen
<headius[m]>
detail is a separate run so don't correlate numbers
<headius[m]>
the generated invokers are still fastest startup
<headius[m]>
they could be loaded lazily also
<headius[m]>
we have so many ways of doing the same thing
<headius[m]>
0x0000000127877000 - 0x0000000127a77000] reserved 2048KB for Class from
<headius[m]>
lots of that
<headius[m]>
2MB really?
<enebo[m]>
yay pidgin integration stopped working
<headius[m]>
noice
<enebo[m]>
you got me on what any of this means
<headius[m]>
it means so much
<enebo[m]>
I see the 2mb lines but below there are refs to classes and those show mallocs of what you would expect
<headius[m]>
so the ratio I had was roughly 2x larger process than heap max for this simple app
<headius[m]>
so that's 250MB of stuff
<headius[m]>
or so
<enebo[m]>
assuming 92k means something like I would expect (and honestly that seems big but I don't know what other info they are storing or alighment or whatnot)
<headius[m]>
yeah this reserve + commit dance in the logs doesn't mean much to me
<headius[m]>
everythign that gets reserved seems to immediately be committed
<enebo[m]>
some heuristic of not constantly alloc'ing must have made this important at some point
<headius[m]>
I wonder if class data sharing would reduce a meaningful amount of that
<enebo[m]>
one classloader experiment!!!!!!
<enebo[m]>
Lets see how many spaces are made by pangealoader
<headius[m]>
if I'm reading this right my app is using just under 100MB for "Class" space
<headius[m]>
which does seem like too much for an app that can run in 200MB
<enebo[m]>
haha
<enebo[m]>
yeah I mean I guess I don't know what class memory really entails
<enebo[m]>
no doubt metadata and actual generated code
<enebo[m]>
The commit vs reserved also makes me think they must have had issues growing things out in a balanced fashion although I see those in all the numbers on the report. Maybe just a generic issue
<headius[m]>
so as far as I can tell indy improves these numbers across the board
<headius[m]>
I would like to see your metadata numbers again with and without indy
<headius[m]>
the indy thing doesn't sound crazy to me because without indy all call site logic is generated as a method alongside the jitted method
<headius[m]>
so that would be N call sites * whatever the metaspace cost of a method entry is
<headius[m]>
where indy is just invokedynamic call sites
<headius[m]>
enebo: did you try an interpreted run?
<headius[m]>
unsurprisingly metaspace doesn't increase much
<headius[m]>
58/54MB
<headius[m]>
I think it started around 51
djellemah has quit [Ping timeout: 255 seconds]
<headius[m]>
enebo: I just did a run on Java 11 with CDS and about 6600 classes get shared
<headius[m]>
so that's just what we load for -e 1 basically
<headius[m]>
using CDS does improve metaspace usage, from 76MB down to 57MB
<headius[m]>
I assume for a single process that's just moving the deck chairs from metaspace to some shared memory space
<enebo[m]>
yeah probably
<headius[m]>
I guess that's one "little" caveat with the cds stuff...to get any of the memory gains you need to be using more than one process
<enebo[m]>
but maybe it is a bit smarter about packing stuff a bit more since it knows what it needs to save
<headius[m]>
it could be
djellemah has joined #jruby
drbobbeaty has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]