sebstrax has quit [Quit: Connection closed for inactivity]
rsim has quit [Ping timeout: 240 seconds]
<nirvdrum>
I'm in a similar boat :-)
<nirvdrum>
While we're all in TX, we should have a roadmap meeting of sorts. I'm not sure when the 5 of us are going to be in the same location next.
<enebo>
nirvdrum: yeah that sounds good
<enebo>
nirvdrum: do you drink beer?
<nirvdrum>
I do.
<nirvdrum>
But I don't like IPAs, so I get kicked out of all the cool guy bars.
<enebo>
nirvdrum: we usually have a small group of us bring 2-3 beers (more unusual if possible) and do beer tasting
<nirvdrum>
As in you check them in your luggage?
<enebo>
nirvdrum: there are many varieties of beer usually. tarcieri usually brings some big stouts
<enebo>
nirvdrum: we do yeah
<enebo>
nirvdrum: never had a beer leak/break yet
<nirvdrum>
Interesting.
<cpuguy83>
Way back when, I used to drink only imports because there was very little good local beer... now (especially in Tampa) local beer has really hopped up (pun intended)... but I can't drink it anymore as it gives me migraines :(
<enebo>
nirvdrum: but if you don’t check luggage then it is hard
<nirvdrum>
I don't really drink anything too unusual. I'm more of a whisk(e)y guy.
<enebo>
cpuguy83: you going to rubyconf?
<nirvdrum>
I'll have to see if I can track down some Zimas or Seabreezes.
<enebo>
nirvdrum: LAGER MAN
<cpuguy83>
No, I don't get to go to too many confs right now.
<enebo>
cpuguy83: bummer…you could have been a beer tasting specatator :)
<cpuguy83>
Also, RubyConf is happening during DockerCon EU
<enebo>
asarih: you at rubyconf this year right?
<enebo>
oh hmmm he is in EU right now…
<cpuguy83>
Lol, spectator... I'd spectate my way to a migraine, I'm sure :)
camlow32_ has quit [Remote host closed the connection]
camlow325 has joined #jruby
kwando_ has joined #jruby
mkristian has joined #jruby
mattwild_ has joined #jruby
tenderlove has joined #jruby
tenderlo_ has quit [Read error: Connection reset by peer]
projectodd-ci has quit [Ping timeout: 272 seconds]
rsim1 has joined #jruby
kwando has quit [Ping timeout: 272 seconds]
projectodd-ci has joined #jruby
mattwildig has quit [Ping timeout: 272 seconds]
camlow325 has quit [Remote host closed the connection]
blaxter has quit [Quit: foo]
<enebo>
nirvdrum: was eating. I used jvisualvm and took a snapshot and now it is complaining about no source?
camlow325 has joined #jruby
camlow325 has quit [Remote host closed the connection]
<enebo>
nirvdrum: I added a comment. I do not think lib/ruby/stdlib is the right location for this unless there is a reason it needs to be in that dir
<enebo>
nirvdrum: anyways I do not actually see anything about nativeconverter which is holding a reference
brauliobo_ has joined #jruby
camlow325 has joined #jruby
<enebo>
nirvdrum: at end of invocation of a call the ‘session’ is a dead object so should GC and release reference
<enebo>
nirvdrum: the WString itself is not the post thingy so that should not get hit either
camlow325 has quit [Remote host closed the connection]
camlow325 has joined #jruby
<nirvdrum>
enebo: I'm not sure how well jvisualvm tracks off-heap memory anyway.
<enebo>
well I realized I was selecting snapshot and not heapdump :)
<enebo>
nirvdrum: I do see one interesting problem
<enebo>
nirvdrum: CHM will point next -> to the next if in the same bucket
<enebo>
nirvdrum: so a magazine will be point at next magazine and hold a hard ref
<enebo>
nirvdrum: funny thing though is that when I made the heap dump it closed out all BoundedMemoryIO and all TransientNativeMemory instances to report 0
<enebo>
nirvdrum: which may be that there are none
<enebo>
nirvdrum: actually I will rerun this loop for a while and see it hose guys GC or just keep growing
<nirvdrum>
enebo: I'm not sure that matters. The magazine won't have handle to the other magazine's sentinel.
<nirvdrum>
And part of the magazine cleanup is to unregister itself from that CHM.
<enebo>
nirvdrum: yeah ok so all holders to sentinal if GCd should clear up weak ref in Magazine which then makes only a single phantom reference to sentinal so it should clean up
<enebo>
I keep forgetting it is not the magazine itself which is not the reference. the referent is the sentinal
<nirvdrum>
Yeah.
<nirvdrum>
It's wonky.
<nirvdrum>
But I think it's doing what it's supposed to.
<enebo>
so it looks like when I heap dump some finalization must be occurring which cleans it all up
<nirvdrum>
I think in your snapshot you want to be looking for retained TransientNativeMemory instances.
<enebo>
since there is 0 TransientNativeMemory
<enebo>
but there were many right before hitting the heap dump button
<nirvdrum>
This could be similar to the problem things like the imagemagick gem have had.
<nirvdrum>
The object reference itself is very small so it doesn't create GC pressure, but the native allocated memory can be arbitrarily large.
<enebo>
I was wondering if the JNI itself is holding the object alive too
<nirvdrum>
In which case, debugging this is going to be really hard because everything is working the way it's supposed to :-/
<enebo>
nirvdrum: I had that thought but I feel like it should eventually trigger…I guess I could make abunch of rooted garbage
djbkd has quit [Remote host closed the connection]
<enebo>
nirvdrum: I ran some runs for like an hour
<enebo>
nirvdrum: granted it is a lot of tiny garbage being generated in a loop
<enebo>
but eventually those magazines should add up :)
<enebo>
Ihmmm if I do add some pressure I don’t want to add it with a leak
<enebo>
this is surprisingly much harder to observe in tooling than I expected
<enebo>
so seemingly struct itself is fine
<enebo>
since if I remove wstring + native converter the leak is gone
<nirvdrum>
enebo: Keep in mind the magazine size is 8K and only used for objects < 256 bytes.
<nirvdrum>
Although if you were running for hours, you should have churned through some.
<enebo>
nirvdrum: I would have thought enough to GC at least
<nirvdrum>
So if you just allocated WString in a loop, you don't see the leak?
<nirvdrum>
(i.e., cut out nativeconverter)
<enebo>
nirvdrum: I did not try that but it is just a pojo
<enebo>
nirvdrum: so the convert or below has to be the issue
<enebo>
nirvdrum: I even cached the wstring itself and still saw the leak
<nirvdrum>
Yeah. I'm just suggesting the marshal might be a red herring.
<nirvdrum>
I have no idea.
<lopex>
enebo: if 9.0.4.0 is more than week ahead, would you mind releasing new jcodings ?
<enebo>
well I am convinced it is convert down to something lower
djbkd has joined #jruby
<enebo>
we can see the magazines never going away and if I remove wstring usage it does not leak
<enebo>
if I cache wstring as a single instance it still leaks so it does not have anything to do with wstring as an object
<enebo>
lopex: well I am hoping 9.0.4 is a matter of a couple of days at most
<lopex>
enebo: do not then
<enebo>
lopex: but we have a fairly serious memory leak
<lopex>
enebo: yeah, I can see
<enebo>
lopex: how risky are the jcodings changes?
<enebo>
updated some transcoding tables?
mattwild_ has quit [Remote host closed the connection]
<lopex>
enebo: low risk, but I wouldnt risk
<enebo>
lopex: ok fine by me for waiting
<lopex>
enebo: just some cleanups and flags so client code can avoid instanceofs
<lopex>
simple things
<enebo>
lopex: does it actually fix anything semantic?
<lopex>
no
<enebo>
lopex: ah then we can definitely wait :)
pawnbox has quit [Remote host closed the connection]
<enebo>
nirvdrum: bummer
<enebo>
nirvdrum: it is working as designed
<enebo>
nirvdrum: I added a System.gc on every stat call in a loop and there is no leak
<enebo>
nirvdrum: but then I added one every 50,000 and I got like 1200 magazines and then boom down to 200
<nirvdrum>
TransientNativeMemory is calling System.gc() in allocation.
<nirvdrum>
But that seems to not be enough.
<nirvdrum>
Ahh, right.
<enebo>
well I am only staring at Java heap too
<nirvdrum>
So that only calls if memory can't be allocated.
<enebo>
I should make sure commit set is also not growing
<enebo>
oh yeah hah
<nirvdrum>
And that's probably a bit too aggressive.
<enebo>
ok well crud we need something a little more conservative I guess
<nirvdrum>
I think we should have a max pages setting and just look at the CHM size.
<nirvdrum>
It doesn't need to be precise.
<nirvdrum>
Of course, they could actually be used and you'll GC for no gain, so you'd need to check if it actually shrinks or not.
pitr-ch_ has quit [Ping timeout: 244 seconds]
<enebo>
nirvdrum: oh uh
<enebo>
nirvdrum: I am not seeing commit size stop growing
brightball has quit [Read error: Connection reset by peer]
brightball has joined #jruby
johnjoseph has joined #jruby
rsim1 has quit [Quit: Leaving.]
kwando has joined #jruby
kwando_ has quit [Quit: Lost terminal]
<enebo>
nirvdrum: ok so I can see Magazines cleaning up with explicit GCs which means they are seemingly cleaning themselves up BUT
camlow325 has quit [Read error: Connection reset by peer]
<enebo>
nirvdrum: the memory from those magazines do not seem to end up being reclaimed
camlow325 has joined #jruby
<nirvdrum>
enebo: Is that referent cleanup stage happening?
<nirvdrum>
That should free up the pages.
<enebo>
yeah I guess I need to get a breakpoint in there
yfeldblum has joined #jruby
<enebo>
I do not have jnr-ffi on my windows box but I think I might be able to break point un decompiled source
<nirvdrum>
Just System.out :-)
<nirvdrum>
Ahh.
<nirvdrum>
Yeah, if you have IDEA download the sources you can set a breakpoint there.
<enebo>
This is weird though
<nirvdrum>
Just make sure you set the breakpoint suspend to "thread" since it'll be cleaned up by a background thread.
<enebo>
if I set the gc to every 10 stat calls this memory leaks seems to be slower
<nirvdrum>
Set it to 1 and pick up an order of magnitude!
<enebo>
I mean constantly GC’ing will slow down how much is happening but I am confused at how much the leak slowed
<nirvdrum>
Yeah.
<enebo>
finalizeReferent has not fired yet
<nirvdrum>
enebo: I'm afraid to point a finger in yet another location because I've been dead wrong on most of this thus far.
<nirvdrum>
But it looks to me like the memory release call on Windows might be wrong.
jensnockert has joined #jruby
<enebo>
nirvdrum: so when I hover over it it says INVALID
<enebo>
so I am guessing it is not successfully reg'd
<nirvdrum>
According to that, the dwSize param must be 0, but Windows#freePages is passing the allocated memory size.
<nirvdrum>
I guess looking at the return value would be helpful.
<enebo>
well I guess I will compile this after all
<enebo>
I think the commit column is growing too
<enebo>
so maybe either 0 is write or it is the wrong actual request?
<enebo>
write=right
<enebo>
amirite
<nirvdrum>
I still haven't set this up in Windows. I need to get git installed and all that.
<nirvdrum>
enebo: I think it's the right request with the wrong argument.
<nirvdrum>
But the interface defines this to return a boolean instead of an int value. I'm not sure how that's being mucked with.
<enebo>
nirvdrum: which PageManager class is it?
mkristian has quit [Quit: This computer has gone to sleep]
mkristian has joined #jruby
<enebo>
found it
<nirvdrum>
Windows.
<enebo>
I do not love all these inner classes so I skipped right over it :)
<nirvdrum>
Naming it WindowsPageManager would have been a bit more helpful, too.
<enebo>
yeah
subbu is now known as subbu|lunch
rsim has quit [Quit: Leaving.]
kstuart has quit [Read error: Connection reset by peer]
pitr-ch has joined #jruby
kstuart has joined #jruby
<nirvdrum>
enebo: Any luck?
<enebo>
nearly have it all rebuilt
<enebo>
changed to zero
rsim has joined #jruby
rsim has quit [Ping timeout: 240 seconds]
rsim has joined #jruby
<enebo>
nirvdrum: yeah I think that did fix it
<enebo>
nirvdrum: I first ran with the frequent GCs and VMMStat was showing no growth
<enebo>
nirvdrum: then I ran with no explicit GCs and it was still ok
<enebo>
nirvdrum: I looked in with visual VM with both and I could see GC happening after a while in latter case and the other was pretty obvious it was constantly running
skade has joined #jruby
<nirvdrum>
If you wanted to really be sure, you could go back and instrument around the return value and GetLastError.
<enebo>
nirvdrum: I would like someone else to mae this change and also confirm this
<enebo>
:)
<nirvdrum>
Okay. I'll try to get something set up this evening.
<nirvdrum>
Since headius is globetrotting again.
<enebo>
alright. I will commit this as-is since I think it is working so then all you need to do is install
<nirvdrum>
At least I have a better idea of how all this stuff works now.
<nirvdrum>
I had never even heard of phantom references before yesterday.
<enebo>
nirvdrum: I was slowly getting to pagemanager not freeing up but I am not sure if I would have noticed that note in the MSDN docs or not so thank you for that :)
<enebo>
I could see GC doing it’s thing but pages not fgetting freed for reals
<enebo>
I really dislike the weirdness of windows APIs…Unixy APIs do have this stuff like fcntl but it seems like every method in Windows has these weird caveats
rsim has quit [Quit: Leaving.]
<enebo>
nirvdrum: I also love how it just does not work if non-zero
<nirvdrum>
enebo: Well, my original suspicion is that we somehow weren't freeing something that Windows was allocating.
<enebo>
nirvdrum: what is the point of that?
<enebo>
nirvdrum: yeah in retrospect we were only seeing it in windows but with that said this was the only platform using tiny objs for performing stat
shellac has quit [Quit: Computer has gone to sleep.]
<nirvdrum>
Dunno. I'm sure there's a fast path in there that just does a bit operation on the size and doesn't proactively clear for performance.
<enebo>
nirvdrum: but someone is seeing some leaking native memory on ubuntu as well
<enebo>
nirvdrum: so this won’t fix that
<enebo>
nirvdrum: although it is unknown if jnr-posix is to blame
<nirvdrum>
Issue URL?
shellac has joined #jruby
<enebo>
ummm I don’t think so since they never isolated it beyond prop. app leaks
<nirvdrum>
Since these memory methods also return a boolean, we probably shouldn't just be ignoring the value in the page manager.
<enebo>
nirvdrum: I owe you a “nirvdrum-acceptable” beer in a little over a week from now
<nirvdrum>
Heh. It's teamwork!
<enebo>
nirvdrum: even teams need beer bashes
<nirvdrum>
The Windows API do have weird caveats, but at least they're remarkably consistent across versions. I think it was fnctl I was trying to implement and it's behavior changes depending on the Linux kernel version. Then again depending on libc. And Mac is something different yet.
<nirvdrum>
I don't know how anyone can even use it.
<enebo>
that code specifying size is from 2012
<enebo>
so no doubt there were cases where we leaked on windows
<enebo>
but we did not use wstring in stat until about 4 months ago
<enebo>
nirvdrum: yeah fcntl is totally a wild beast across flavors
<nirvdrum>
The world is a vampire.
<enebo>
nirvdrum: probably some variants of sub-functions are consistent but I at least know I should research it a bit
rsim has joined #jruby
<enebo>
Oh almost feaked out…I updated jruby to use newer jnr-posix but forgot it has a direct jffi dep
mkristian has quit [Quit: This computer has gone to sleep]
<enebo>
I want to make sure I do not see a leak in jruby itself in case crazy stuff is happening beyond this
jensnock_ has joined #jruby
jensnock_ has quit [Read error: Connection reset by peer]
jensnock_ has joined #jruby
<nirvdrum>
There were other places using allocateDirect.
<nirvdrum>
But it depends on the allocated object size as well.
jensnockert has quit [Ping timeout: 250 seconds]
<enebo>
holding steady
rsim has quit [Quit: Leaving.]
<enebo>
I will let it run like 30 more minutes to make sure something else does not eventually explose
<enebo>
EXPLOSE
jensnock_ has quit [Remote host closed the connection]
subbu|lunch is now known as subbu
yfeldblum has quit [Ping timeout: 240 seconds]
brightball has quit [Quit: Leaving...]
skade has quit [Quit: Computer has gone to sleep.]
rsim has joined #jruby
whitby has joined #jruby
skade has joined #jruby
skade has quit [Quit: Computer has gone to sleep.]
rsim has quit [Quit: Leaving.]
Guest76723 is now known as Cyrus
Cyrus has quit [Changing host]
Cyrus has joined #jruby
mattwildig has joined #jruby
rsim has joined #jruby
rsim has quit [Client Quit]
cschneid has quit [Ping timeout: 240 seconds]
elia has joined #jruby
bbrowning is now known as bbrowning_away
rsim has joined #jruby
rsim has quit [Quit: Leaving.]
rsim has joined #jruby
cschneid has joined #jruby
rsim has quit [Client Quit]
rsim has joined #jruby
rsim has quit [Client Quit]
rsim has joined #jruby
nateberkopec has quit [Quit: Leaving...]
rsim has quit [Read error: Connection reset by peer]
rsim has joined #jruby
rsim has quit [Client Quit]
yfeldblum has joined #jruby
lanceball is now known as lance|afk
samphippen has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
djbkd has quit [Remote host closed the connection]
rsim has joined #jruby
rsim has quit [Client Quit]
djbkd has joined #jruby
yfeldblum has quit [Remote host closed the connection]
rsim has joined #jruby
yfeldblum has joined #jruby
rsim has quit [Client Quit]
rsim has joined #jruby
rsim has quit [Read error: Connection reset by peer]
rsim has joined #jruby
rsim has quit [Client Quit]
bb010g has quit [Quit: Connection closed for inactivity]
rsim has joined #jruby
thedarkone2 has joined #jruby
rsim has quit [Quit: Leaving.]
rsim has joined #jruby
rsim has quit [Client Quit]
tcrawley is now known as tcrawley-away
elia has quit [Quit: Computer has gone to sleep.]
elia has joined #jruby
mdedetrich has joined #jruby
nateberkopec has joined #jruby
nateberkopec has quit [Ping timeout: 265 seconds]
<Antiarc>
headius: if you're still in the market for a new phone, I'm *really* liking my Nexus 6P
rsim has joined #jruby
shellac has quit [Quit: Computer has gone to sleep.]