<headius>
ChrisBr: I think we should be able to make at least parts of updating atomic
<headius>
packing start and end into a single long might be a first change I make, so both can be updated at once
<headius>
if we can get it down to just a couple pieces of state the atomic logic shouldn't be too complex
<lopex>
long update is guaranteed to be atomic on 64bit ?
<lopex>
on 32 it's not
<headius>
on 64bit native I believe so
<headius>
on 32 it is guaranteed if you are using atomics, I believe
<headius>
"Writes and reads of volatile long and double values are always atomic"
<headius>
so not even atomic
<headius>
just volatile
<lopex>
headius: there's similar trick when passing cr and length both in long for String
<lopex>
but that's just so we can return multiple values at once
<headius>
right
<headius>
we can be doing this more places
<headius>
I'm still of a mind that we could make Array, String, and Hash largely thread-safe with atomics
<headius>
at least internally consistent
<lopex>
are java.util.concurrent.atomic intrinsified and possibly more aggressively allocation eliminated ?
subbu is now known as subbu|afk
<lopex>
because it's just semantics right ?
<headius>
that I'm not sure about
<headius>
I generally end up using either atomic updaters or unsafe methods directly because of the wrapper
<headius>
we definitely don't want a bunch of Atomic* objects for all of these data fields
<headius>
they might fold away fine with good escape analysis but if they have to stay in memory they have to stay in memory
shellac has joined #jruby
<ChrisBr>
headius: what do you mean with single long?
<ChrisBr>
anything I can help out ?
<headius>
start and end are ints, so they could be high/low 4 bytes of a Java long
<headius>
then we'd be able to update start and end atomically with a single write
<headius>
the atomic sequence for such an update would be like...
<ChrisBr>
hmm right
<headius>
get 64-bit value, split out 32-bit values, fiddle with start and end, re-combine as 64-bit, confirm it hasn't changed before updating (compare and swap)
<ChrisBr>
hm smart
<ChrisBr>
so one operation less then?
<headius>
one less operation and no chance of someone else reading half of it
<headius>
I think the NPE we were seeing was likely because someone got their start but someone else's end
<headius>
or vice versa
<headius>
someone = thread
<ChrisBr>
ahh
<headius>
ChrisBr: a short-term thing we could start to do would be to encapsulate as much as possible all accesses to those fields
<headius>
you mostly have that in the internal* impls
<headius>
but anything we can do to make it smaller
<headius>
we want as few reads/writes for everything, ideally down to a point where we can guarantee we can have valid start/end/bins/entries in hand
subbu|afk is now known as subbu
<lopex>
with array memory model you might hope for it
<headius>
right
<headius>
I have not figured out all the bits we need to do but this structure seems easier to make thread-safe than the chained buckets
<headius>
chained buckets at a minimum would have required multiple atomic updates or lock pairs
<headius>
messy
<lopex>
chained buckets give an illusion since the're more consistent internally themselves
<lopex>
is that a right way to put it ?
<ChrisBr>
ok, so if you have a more concrete way in mind let me know. I'm a little lost how to go forward...
<headius>
a good way to put it in fact
<headius>
lopex: frequently we'd see concurrency failure in Hash as a cycle in one of the buckets rather than a crash
<headius>
arguably that's worse than a crash
<lopex>
yeah
<lopex>
the illusion being hash, key being final, and no arrays right ?
<headius>
chained buckets seemed more robust concurrency-wise but it has also had more attention to properly ordering reads and writes
<headius>
yeah
<headius>
well that and the buckets being *actually* separate memory locations
<headius>
so multiple threads have to get lucky and either be changing bucket array size or both working on same slot
<lopex>
but double linked list makes it even worse
<headius>
open addressing has no buckets so all threads are working against the same array
<lopex>
so realloc is more critical
<headius>
we need a better way to separate storage strategy from methods
<headius>
like specialized arrays, alternate hash impl strategies, etc
<lopex>
headius: I think with that fact encoding versions we might solve quote a bit actually
<lopex>
headius: bad thing is that there's lots of by reference comparison spread throughout the code
<lopex>
but
<lopex>
if we know cr is valid we may pass fast encoding for matching
<lopex>
if not, then we pass approximate version
<lopex>
so that fast version doesnt leak back into jruby runtime
<lopex>
cr known / valid, we pass fast version, cr unknown invalid, we pass approx version
<lopex>
we wont blow in broken strings, adn we will be faster for gneeral case
<lopex>
otherwise, can we statically find 'by reference' encoding comparisons ?
<lopex>
they dont end up Object anywhere I would hope
shellac has quit [Quit: Computer has gone to sleep.]
<headius>
The reference compares shouldn't hurt though right? Just prevent some optimizations
subbu is now known as subbu|lunch
<lopex>
the fast encodings will be different instances
<lopex>
so there two options
<lopex>
simpler: pass faster encoding from core methods to the actual routines for length etc, and always keep base encoding in String instances for example
<lopex>
second, keep specialized encodings in core objects, but provide safe comparisons
<enebo>
lopex: especially when I updated the topic in this channel as you were talking
<lopex>
yeah
<enebo>
lopex: and then headius making a bet if we get issue reported in 30 minutes
<headius>
I think I lost
<lopex>
I know why, headius responded just after the topic have been changes
<lopex>
*changed
<enebo>
headius: it is tough to know how fast someone will jump on the horse
<headius>
yeah
<enebo>
This was more likely than not since it was such a long time
<headius>
enebo: I'm goingto make open addressing branch
<enebo>
ok
<headius>
we'll work out with ChrisBr how to collab
<lopex>
the bet is still on ?
<headius>
I'm about 15 min over my 30 bet
<enebo>
unless we change it to metric minutes
<lopex>
or disallow issues!
<enebo>
lopex: sounds like a star trek life hack
<lopex>
enebo: what was that ?
<enebo>
lopex: just Kirk always found the novel way around the situation (well not really he usually just did some horrible fighting scene)
<lopex>
enebo: like plot armor ?
<headius>
jruby date 2394658.5
<ChrisBr>
headius: enebo: congrats on the release
<enebo>
lopex: plot armor is always in play
<enebo>
ChrisBr: thanks...it is good to have it done
<headius>
ChrisBr: thank you!
<enebo>
we just are wondering if we will see anything needing immediate attention now
<enebo>
so far issue tracker is quiet...too quiet
<lopex>
well, still having that bundler issue on windows
<lopex>
but only for update
<lopex>
install works
<headius>
ChrisBr: hey something I could use your help with short term
<headius>
so the old concept of a "small" hash is no longer there because it didn't fit open addressing
<headius>
but I notice in places where we were creating a small hash before we're now creating one with room for 8 entries minimum
<headius>
there are tons of 1- and 2-element hashes obviously for kwargs and such...perhaps we can reinstate some concept of "small" that starts with a smaller entries array?
<headius>
I'm going to look at combining end and start since that's a little fiddlty
<headius>
fiddly
<lopex>
or even specialize it like you did for arrays ?
jmalves has joined #jruby
jmalves_ has quit [Read error: Connection reset by peer]
<lopex>
but then, you'd have to use that same interface consistently
<headius>
lopex: absolutely
<headius>
well so this is one thing Truffle gets those languages
<headius>
they have this amorphous "DynamicObject" that they can mess with internally and then they can apply any storage strategy to it
<lopex>
yeah, I recall
<lopex>
er, remember
<headius>
I'm starting to get there with RubyObject specialization
<headius>
that plus VariableTableManager can be expanded to allow per-allocation shapes
<headius>
the whole thing could be merged with Struct and Array to specialize those the same way
<headius>
and then Hash could come along, although it's a bit more complicated obviously
<lopex>
yeah, hashes could in principle be optimized the same way objects are
<headius>
that was always a special feature of IBM hardware
<lopex>
yeah
<headius>
storage IO would never be your bottleneck
<lopex>
yeah
<headius>
weird filesystems though
<lopex>
headius: they always had a joke in their heads, about how many refrigerators you'd be able to transfer with lamborgini and atrack
<lopex>
*truck
jeremyevans has joined #jruby
<lopex>
but IO is their strongest thing
<lopex>
and timi
<lopex>
and it's from the 80ies
<lopex>
headius: back in 15 years ago or something, our dba's were mostly operating on disk tracks and sectors to optimally lay tablespaces for db2
<jeremyevans>
jruby-dist-9.2.1.0-bin.tar.gz ships with some gems (e.g. rake), but without the gem lib/exe files. so jruby -S rake results in LoadError: no such file to load -- /usr/local/jruby/lib/ruby/gems/shared/gems/rake-12.3.0/exe/rake
<lopex>
and defrag was a manual thing
<headius>
jeremyevans: NOOO
<headius>
enebo: NOOO
<lopex>
headius: more thatn 30 mins
<lopex>
no excuses
<headius>
enebo: I thought you verified against dist tarball
<headius>
maybe did not notice because rails immediately installs rake
<enebo>
I only test installer on windows
<headius>
enebo: which should be based on dist zip right
<headius>
enebo: 9.2.2? :-D
<jeremyevans>
headius: so far I only tested by updating the OpenBSD port
<enebo>
Probably I don't remember
<enebo>
jruby-dist should be jruby-bin and I test that on linux
<headius>
jeremyevans: thank you...we've had a number of issues over the past week that made me realize we need some verification CI against the dist tarballs
<jeremyevans>
but tar ztf shows files like MIT-LICENSE and such, but no lib/exe directories in the gem folder
<headius>
jeremyevans: can you toss this into a bug please
<headius>
I'm walking out the door
<jeremyevans>
headius: sure, just wanted to check here first. Thanks!
<headius>
it was supposed to be fixed in this release but I guess we didn't get all the files
<headius>
sigh
<headius>
enebo: this would be a trivial thing to spin for a 9.2.2
<headius>
jeremyevans: workaround of reinstalling rake works yah?
<jeremyevans>
headius: it should, but I haven't tested that yet
<enebo>
yeah on linux I expand -bin which is just -dist and I ran rails/guantlet
<lopex>
enebo: and no clues about that bundler issue on windows ?
<lopex>
enebo: I could assist
<enebo>
lopex: if you can find out something other than admin that would be great
<lopex>
enebo: but something must have changed in jruby
<enebo>
jeremyevans: I guess one of my verify scripts is gem install rails as first thing which pulls in what is missing
<lopex>
enebo: do you recall what was it ?
<enebo>
lopex: no I assumed you always had to be admin so the fact you can get it to sometimes work as non-admin is confusing...i don't know
<lopex>
enebo: and that is false as far I can tell
<enebo>
lopex: ok well yeah figuring it out would be very helpful
<lopex>
enebo: I have two machines that are consistent with that regression