<GitHub59>
[jruby] r-obert opened issue #4847: YAML parses okay on CRuby, raises exception on JRuby https://git.io/vFgbW
dave__ has joined #jruby
dave__ has quit [Ping timeout: 248 seconds]
eregon has joined #jruby
RickHull has joined #jruby
<RickHull>
Hi, got a question relating to the travis runtime environment. I have a zero-dep library that seems to run significantly slower on jruby-on-travis than MRI-on-travis
<RickHull>
I am guessing it is the travis jruby platform that is slower, not the jruby runtime
<headius>
you will want to switch to scraping that
<RickHull>
what is the source of s3 frustration? just curious
<headius>
the s3 we're using was from a previous employer who allowed us to keep using it...now 5 years later I think they've forgotten and so they delete buckets they're not familiar with and stuff
<headius>
we're trying to get in contact with them
<RickHull>
ah, that's annoying
<headius>
but it's still likely to go away suddenly
Puffball has quit [Remote host closed the connection]
<headius>
RickHull: on my local system JRuby appears to be a clear winner
<RickHull>
nice! i rather expected that
<headius>
actually about a tie with default flags
<headius>
ah yes... -Xcompile.invokedynamic bumps it up
<headius>
so with no flags both JRuby and MRI seem to be around 1000k/s by the end
<headius>
assumimg that's a useful number
<RickHull>
where does jruby turn the corner?
<GitHub143>
[jruby] havenwood opened issue #4848: S3 missing jruby-src-9.1.14.0.zip https://git.io/vF2fH
<RickHull>
(assuming it does-ish)
<RickHull>
on my VM and travis, MRI is solid 500k, constant
<RickHull>
(which is odd because i expect pushes to get slower)
<headius>
an initial optimization bumped it up, then something had to back off and reoptimize
<RickHull>
ah, cool!
<RickHull>
i haven't tested this much beyond a few seconds worth of pushes
<headius>
if I run with -Xcompile.invokedynamic JRuby gets over 2000k at times
<RickHull>
2 meeleeon pushes *pinky to lower lip*
<headius>
haha
<headius>
yes
<RickHull>
i kind of thought maybe my little lib could make a reasonable benchmark suite
<headius>
this appears to be generating a lot of garbage
* RickHull
perks up
<headius>
sure! we love to use real libs as benchmarks
<RickHull>
the rationale for this is to scratch a personal itch -- i just like to implement stuff minimally and simplistically
<RickHull>
as simple as possible and no simpler
<RickHull>
clarity of expression first, performance second
<RickHull>
so it's maybe not a "real lib" ;)
<RickHull>
you might be my first user :)
<headius>
generating lots of garbage for GC I mean
<RickHull>
right
<RickHull>
in the heap case, it should just be a big array of ints
<RickHull>
none of which should be garbage
<RickHull>
so, I'm not sure
<RickHull>
a bug presumably
<RickHull>
in my code
<headius>
oo yeah, this thing loves memory, at least on JRuby
<headius>
up to 1890816K heapon a longer run
<RickHull>
at 2M push /sec, it's just consuming it like mad
<RickHull>
but beyond that, extra consumption?
<headius>
this might be leaking
<headius>
where to these pushed thingies go
<headius>
it seems to be growing endlessly
<RickHull>
Heap.is_a? CompleteTree and CompleteTree.has_a? Array
<headius>
so it's saving all these pushes, ok
<RickHull>
every push is an array append
<headius>
is there a way to clear it after each round or something?
<RickHull>
but there is a #sift_up
<RickHull>
to put the heap back into proper state
<RickHull>
O(log n) n = size of array
<headius>
like near bottom of the loop?
<RickHull>
this script is just designed to accumulate
<RickHull>
but it should be easy to say h = Heap.new
<RickHull>
or whatever
<headius>
ah, ok
<RickHull>
after 2M push / sec, and h = Heap.new, should be plenty for GC to chew on :)
<headius>
ok that's more stable
<headius>
I put it at the bottom of the 100k bit
<RickHull>
yeah -- my goal for this script was to understand how the time-per-push depends on the size of the heap
<RickHull>
so just let it grow
<headius>
looks like it averages out to 1600-1700k
<headius>
ahh sure
<headius>
with the ever-growing heap the later results were just as fast as the warmed-up earlier results
<RickHull>
yeah, that's what I don't get
<RickHull>
#sift_up is not O(1)
<RickHull>
but hey, magic!
<headius>
ahh I see
<headius>
so a peak of 2300k comes fairly early in a 30s run
<headius>
hmm well I see them much later too
<headius>
this is a run that grows forever
<RickHull>
you should notice that #heap? takes longer per heap size
<headius>
I don't like magic :-)
<RickHull>
which btw is not called internally
<headius>
still a heap with 31753335 items? YES - 24.879 sec
<RickHull>
it visits every node
<headius>
ok
<headius>
yeah I dunno...100k push: 0.04251822 s (2351k push / s)
<headius>
that's the last one before it bails
<RickHull>
enough mysteries for tonight :)
<RickHull>
I won't file an issue
<headius>
ok cool
<RickHull>
in the meantime, I can dream of magic
<headius>
yeah if you find out we're not doing something right, do file an issue :-)
<RickHull>
for sure :)
<RickHull>
oh, i meant to ask
<RickHull>
there are some warnings and such on jruby travis runs
<RickHull>
is it worth thinking about maybe helping them optimize their jruby?
<RickHull>
any low hanging fruit?
<RickHull>
e.g. Picked up _JAVA_OPTIONS: -Xmx2048m -Xms512m jruby: warning: unknown property jruby.cext.enabled
<RickHull>
(reasonable?)
<RickHull>
i can imagine they are not using invokedynamic
<RickHull>
any downsides to such?
<RickHull>
downsides to enabling it, i mean
<headius>
yeah that's a 2G heap, plenty
<headius>
you can add in JRUBY_OPTS env for any additional
<headius>
worth seeing how it affects perf
<headius>
another thing that could be affecting it is the random numbers
<headius>
depending on how they're being generated we have sometimes had a problem with exhausting entropy on travis instances, which slows things way down
<headius>
you can install a service like haveged to keel entropy pool full
<RickHull>
oooh
<headius>
keep
<RickHull>
yeah, I think at this point the calls to rand are significant
<RickHull>
I bet I could pregenerate 100 rands, and then every once in awhile shift/unshift a new one
<RickHull>
without introducing too much cyclic behavior
<RickHull>
cheers, thanks!
<headius>
no problem
<headius>
oh yeah it screams without rand
<headius>
as high as 5M/s I think...MRI around 1.75M
<RickHull>
what source of numbers?
<RickHull>
i think cyclic behavior should not be a problem with say a ring/array of shuffled ints
<RickHull>
if you just feed it e.g. i = 5 then it should be pathologically fast i think
<headius>
so there's two aspects here, your non-constant complexity and all the other overhead inherent to the system
<headius>
personally I'm interested in the latter, since I can't do much to help you improve the former
<headius>
but obviously improving the latter will reduce the impact of the former at smaller heap sizes
<headius>
at this point there isn't really a bug to report to JRuby, but we also welcome "bugs" that are really just "I'm trying to get more out of this library...help me"
<headius>
we can set in stone some discussion about how to profile for hot spots, etc
<RickHull>
i definitely agree there is no indication of a jruby bug
<headius>
if you haven't already I recommend you check out the performance pages on the wiki, should be easy to find
<headius>
if you have trouble with the profiling options detailed there, open a bug and we can try to help
<RickHull>
I have a general interest in perf, and different runtimes offer different perspectives, but ...
<RickHull>
jruby is not a focus for me
<headius>
no problem
<headius>
I'm glad we're looking ok
<RickHull>
i still am confuzzled by the constant perf, but it's a happy surprise I guess
akp has joined #jruby
<RickHull>
my assumption is that it's linear with a tiny coefficient
<RickHull>
er, O(log n)
<RickHull>
filling the heap with the same number / Comparable does remove #sift_up from the big-O
<RickHull>
so that is an interesting point of comparison
<RickHull>
anywho
<RickHull>
but then it's just Array#push with some extra constant overhead
<headius>
log n I'd expect
<RickHull>
MRI Array#push is constant I think, amortized
<headius>
yes, same as Hash
<RickHull>
still, we have the #sift_up mystery. maybe we need to leave it running for 5m?
<RickHull>
i think my laptop will barf but I guess I should man up
<RickHull>
;)
<RickHull>
i plan to let my subconscious work on this problem
<RickHull>
honestly, the most likely explanation in the outside view is that #sift_up isn't doing what I think it does and I'm not actually maintaining a heap
<RickHull>
but it sure looks like it and I have tests
<RickHull>
I tried `print '.'` per invocation and got lots of dots
<RickHull>
a ruby-prof run should show the same, log n calls
<RickHull>
and I have that. not sure what the jruby analog is
<RickHull>
i have the ruby-prof, haven't dug into the call count
<headius>
well we could calculate this perhaps
<headius>
worst case distribution at N nodes
<headius>
or however you have this implemented
<headius>
maybe you need billions of entries for it to matter
<RickHull>
i think so -- this is my 'tiny coefficient' hypothesis
<headius>
--profile and related options in JRuby can be a good first go
<headius>
there's JVM flags as well but they're a bit more arcane
<RickHull>
the array for a MaxHeap should have highest at idx 0
<RickHull>
so inserting bigger numbers should mean more #sift_up
<headius>
is it a really wide branching factor perhaps?
<headius>
e.g. array-mapped trie and such things
<RickHull>
no idea there
<RickHull>
my top 3 hypotheses for constant insertion perf, where N = size(heap), are: 1. tiny coefficient 2. bug where there is no actual heap 3. some magic optimization
<RickHull>
#3 would be unlikely across different runtimes
<RickHull>
also, just though it's worth mentioning because it tripped me up
<RickHull>
the Heap data structure, as in heapsort, has little to no relationship between the idea of the Heap vs the Stack
<RickHull>
in terms of C-style programming
<headius>
I was wondering about 2
<RickHull>
the evidence against 2 is 1. eyeballing a heap
<RickHull>
2. tests
<RickHull>
3. code review?
<RickHull>
also, can write an external heap? test
<RickHull>
my heap? could be bugged
<GitHub29>
[jruby] headius pushed 2 new commits to jruby-9.1: https://git.io/vF2mG
<GitHub29>
jruby/jruby-9.1 1a4f888 Charles Oliver Nutter: Force OpenJDK to use /dev/urandom for rng if available....
<GitHub29>
jruby/jruby-9.1 adfd1b3 Charles Oliver Nutter: Remove depreacated -G graal flag.
<kares>
headius: JI got faster and some of the parts more manageable, believe the thing that definitely convinced me was List[] impl
<kares>
on the topic I am still looking for your RubyString implements java.lang.CharSequence for 9.2 :)
dave__ has joined #jruby
dave__ has quit [Remote host closed the connection]
dave__ has quit [Remote host closed the connection]
<headius>
kares: I figured it was probably perf
<headius>
it's probably fair for the stuff you moved because it's very frequently hit
<headius>
the rest of the ruby "kernel" we have I've mostly done heavy, infrequent things
<headius>
like spawning a process...our Ruby overhead isn't going to matter much
dave__ has joined #jruby
dave__ has quit [Remote host closed the connection]
dave__ has joined #jruby
<GitHub117>
[jruby] perlun opened issue #4849: Gem::LoadError: You have already activated rack-cors 1.0.1, but your Gemfile requires rack-cors 1.0.2 https://git.io/vF28i
<GitHub35>
[jruby] 284km opened pull request #4850: Fix a typo (master...fix_typo) https://git.io/vF24v
vtunka has joined #jruby
vtunka has quit [Remote host closed the connection]
vtunka has joined #jruby
shellac has joined #jruby
shellac has quit [Client Quit]
shellac has joined #jruby
samuel02 has joined #jruby
<samuel02>
is there a reason why `cmd` works differently in JRuby compared to MRI?
<samuel02>
I had issues with bundler and finally found the problem
akp has quit [Remote host closed the connection]
<samuel02>
in MRI (2.4.0): `echo 'foo'`.strip == 'foo' => true, in JRuby (9.1.14.0) `echo 'foo'`.strip == 'foo' => false
<samuel02>
in MRI (2.4.0): \`echo 'foo'\`.strip == 'foo' => true, in JRuby (9.1.14.0) \`echo 'foo'\`.strip == 'foo' => false
<samuel02>
for some reason the backticks don't show up
<samuel02>
but there are backticks around the echo command
<samuel02>
is this by design or a bug?
<headius>
huh
<headius>
windows is a tricky beast
<headius>
if you can turn these into some sort of test case it would help us pin down what oddity we're missing
dave__ has quit [Remote host closed the connection]
dave__ has joined #jruby
akp has quit [Ping timeout: 248 seconds]
dave__ has quit [Ping timeout: 240 seconds]
bbrowning_away is now known as bbrowning
dave__ has joined #jruby
olle has quit [Quit: olle]
dave__ has quit [Ping timeout: 240 seconds]
olle has joined #jruby
kotk has joined #jruby
kotk has quit [Client Quit]
dave__ has joined #jruby
dave__ has quit [Ping timeout: 240 seconds]
kotk has joined #jruby
<kares>
cause I'm really having a hard time running any rails tests with the setup
<kares>
there's Bundler (with AR gem) + LOAD_PATH changes (adding AR directly from Rails) than a chdir into Rails (to run its test) and somehow it never boots
<kares>
after initial struggle, last this I got from it was: **File does not exist: active_support**
<GitHub92>
[jruby] gerhardgruber opened pull request #4852: [#3953] Use java.nio for FileTest#symlink? (master...use-java-nio-for-symlink) https://git.io/vF25p
<rdubya>
kares: did you do a bundle install in the rails dir? That's the only thing I can think of
<kares>
rdubya: ok thanks, will check ... are you on 9.1.14.0 arelady?
<rdubya>
no i haven't updated yet
<rdubya>
pretty sure enebo has been running with that though
<kares>
aah oki
<kares>
meh but than Rails fetches the drivers on its own :) ... when bundling
<rdubya>
yeah, not an ideal setup lol
<kares>
starting to recall why I gave up running Rails tests
<kares>
cause one needs to build a whole thing to get this testable with driver updates etc.
dave__ has joined #jruby
<rdubya>
ugg, @connection.query('ROLLBACK') throws an exception because it ultimately uses statement.executeQuery() which throws an error if statement doesn't return any results
shellac has quit [Max SendQ exceeded]
dave__ has quit [Ping timeout: 248 seconds]
<rdubya>
any thoughts on a different way to handle that? the only thing I've come up with is testing if the sql statement starts with "select" and using a different method if it isn't a select statement
<kares>
not sure - but those low level methods never worked well
akp has joined #jruby
<kares>
mean I have always run into issues emulating native pg/mysql2 with JDBC -> thus I ditched code sharing ;(
<kares>
rdubya: undefined local variable or method `max_identifier_length'
<kares>
am able to run mysql/sqlite but not postgres with current master
<kares>
using `rake rails:test:postgres:base_test`
<kares>
are you sure its runnable from master?
<rdubya>
i'm running it from master
<kares>
just wanted to attempt a quick test before releasing driver but I am stuck for hours now doing so
<rdubya>
i have a few changes in my env but i don't think there is anything related to that
<kares>
oukej, so it must be me than
<rdubya>
where's it throwing that exception i don't get any references to it when i search the code
<rdubya>
other than a query that reads that string from the db
<enebo>
headius was figuring out something about nested transactions yesterday that seems to affect both sqlite3 and mysql
akp has quit [Ping timeout: 240 seconds]
<kares>
from its own pieces - as it assumes its there:
<kares>
block in columns at rails/activerecord/lib/active_record/connection_adapters/postgresql/schema_statements.rb:233
<rdubya>
do you have 5.0.6 checked out/installed ?
<rdubya>
i don't see a reference to that in my file
<kares>
rails 5-0-stable
<rdubya>
try going to the 5.0.6 tag
<rdubya>
i think i had a similar issue when i tried to use the stable branch
<rdubya>
which we should probably get it working with that branch but I'm just trying to get it working with the released version right now
<kares>
gotcha - thanks
<kares>
that method (or similar) used to exists in older AR so that is why I got surprised
<enebo>
so I am thinking about what good rails app we should use as a sort of sanity check for these new adapters
<kares>
huginn maybe for mysql
<kares>
spree maybe if its still a Rails app
<enebo>
kares: another "detail" is you need to use minit-test 5.3.3 for testing or it randomizes the test order
<kares>
woot? :) and I do not want that, right?
<enebo>
kares: you definitely want to make sure you are using that
<rdubya>
lol yeah, you get all kinds of fun breakage when they are randomized
<enebo>
kares: for now simplest way is to change our pom.rb to 5.3.3 and nuke all other minitests
<kares>
heh :)
<enebo>
kares: AR calls gem "minitest" and that causes proper one to get loaded but JRuby must have a bug
<enebo>
kares: because we ignore Gemfile.lock
<enebo>
kares: I am wondering if we somehow load it before that point but it is not technically an AR issue I don't think
<enebo>
kares: we are moving quickly for next week ;)
<kares>
have disabled bundler/setup on AR-JDBC;s end otherwise with the chdir I never got Rails's test to boot the proper Gemfile
<enebo>
for the most part sqlite3 and mysql are ready pending thoughts on nested transactions
<enebo>
hmm
<kares>
okey- do you have a failing case?
<kares>
thing I did some work on the previous master ... which is not part of the current master
<enebo>
kares: if you run rails:test:sqlite3 once you have right minitest workign you will see 3 or 4 errors consistently
<kares>
at least it logs BEGIN/COMMIT more aligned with native drivers
<kares>
minitest is not really tx material
<enebo>
kares: they all appear to be a scaffold starting a transaction and then something in a test creating a transaction then the inner one causes a commit and the outer tries to rollback
<enebo>
kares: but I think scaffold may be before and then test so I think it is predictable in this case
<enebo>
kares: headius was working on this problem yesterday...when he is around he would have more details
<kares>
okej definitely want to check that out but I really wanted to update PG's driver
<kares>
... assuming of course it does not break hell for Postgre adapter
<enebo>
kares: I think last I heard it maybe was us not tracking savepoints or sqlite not supporting them directly or something (he was only looking at sqlite3 at this point)
<kares>
sqlite doesn't do that
<kares>
and the JDBC driver is not feature complete
<kares>
so I would not care much about that
<kares>
if it happens on MySQL than its interesting ... and smt worth looking at
<enebo>
kares: yeah I technically could have updated mysql to 6 but fixing the timestamp stuff was tricky enough that I am frightened to for this week
<kares>
also I tihnk I spoted a regression with the latest SQLite driver
<enebo>
kares: however AR does this nesting with native adapter and it works
<kares>
there's a maintainer switch happening and code does not get reviewed that carefully pbly
<enebo>
kares: so I am guessing maybe they keep track another way?
<kares>
oh I know - AR no longer threw FooNotExists.columns under new sqlite 3.20
<kares>
enebo: hmmm well I might be talking about old SQLite --- for sure there were work-arounds in the past
<kares>
as I say its much better to track it down with mysql
<enebo>
kares: so this logic does exist now but perhaps it is not use in the collback
<kares>
since there we can pretty much trust the JDBC API being bullet proof
<enebo>
kares: yeah no doubt mysql will work since we can just use it but something tells me we are just using it now unless I accidentally unconnected something
<enebo>
kares: reviewing this sqlite savepoint code I also added a begin with level which raises sayign we don't support isolation levels
<enebo>
I will say this sqlite3-ruby gem has not been released since january so I feel like some of these adapters are stabilizing
<kares>
they seem to have issues updating to latest sqlite
<kares>
JDBC wraps it all up -> no system dependencies
akp has joined #jruby
<kares>
well actually I think the gem works with whatever version is installed locally
<enebo>
kares: yeah they are dependent on whatever is installed which is amazing it works reasonably
<enebo>
kares: but I meant from a AR -> native gem standpoint the native gem has not needed to be released in 10+ months
<kares>
aaah oki
<enebo>
kares: so I think that set of APIs seems to be becoming pretty stable
<kares>
well yeah that is what I meant thah the JDBC (.java impl) pieces aren't still 100% compliant with the spec
<kares>
in SQLite-JDBC
<enebo>
of course each adapter is maintained by different people so not all of them may be that stable
<kares>
and since it was pretty much one guy who had enough ...
<enebo>
kares: do you know why mysql has a version 8 driver with dnr in the version number string?
<kares>
6.x is dev 8.x must be super-dev or what :)
<kares>
who knows - oracle
<enebo>
kares: yeah I did think about 6.0.6 since it does save timezone in timestamp properly
<enebo>
kares: but did a ton of debugging on 5.x adapter and did the work to adjust the timezone
<enebo>
kares: I could try it today...with prepared stmts set AND an AR test where they temporarily change TZ env we fail 2 things
<enebo>
kares: not manually adjusting for the timezone could maybe fix this since I think we assume getDefault() on TimeZone to make that adjustment for 5.x
<enebo>
I don't know how this all relates to minimum required db either
<enebo>
I think anything we do support allows a property to be set to control this behavior
<kares>
there's a property for that I think
<enebo>
anyways this stuff has lots of wrinkle
<kares>
you could set on the driver
<enebo>
yeah I did set it as a test and other dates/times broke
<kares>
:)
<enebo>
which would mean auditing all time/date/timestamp stuff I guess
<kares>
I think the next AR will do some date/time stuff differently
<enebo>
kares: I do think that is the proper path forward but maybe not for this first release
<enebo>
kares: yeah 2.4 changed something with dates
<kares>
yep
<enebo>
kares: There is detection code in place so I think whether 2.3/2.4 it will do same thing but perhaps internally what is works with will be different
<enebo>
I do plan on seeing how badly 5.1 does before rubyconf too
<enebo>
fwiw running their tests was not horrible
<enebo>
we I think pass >99% I think even on 5.1 right now
<enebo>
but stuff definitely breaks
<kares>
mean you did a great job!
<kares>
* means
<enebo>
bummer discourse only supports postgresql
dave____ has joined #jruby
<kares>
that's actually good - you have a case for PG :)
<enebo>
kares: yeah it is good for verifying it but I would like to verify all three without a ton of work
<kares>
guessing ... 75 errors with postgres means I do not have the right minitest
<enebo>
kares: well maybe maybe not rdubya I think changed the Raketask to run the natrive postgresql adapter tests which pulled in some more tests than we were running a few days ago
<rdubya>
yeah is that 75 errors or 75 errors + failures?
<rdubya>
because if you are combining them, that is around what i am seeing right now
<kares>
too late - new run going on with i_suck_and_my_tests_are_order_dependent!
<kares>
so 75 is fine than?
<enebo>
kares: he had no errors but then discovered that AR tests were not running the postgresql/*_tests
<enebo>
kares: so these are newcomers to the test run
<enebo>
kares: same thing with mysql and sqlite3 but the those both had about 6 or so and largely they are totally unimportant like passing in {timeout: "usa"}
<enebo>
kares: I think yesterday morning he was at 140 or so
olle has quit [Quit: olle]
<rdubya>
yeah, managed to knock down a decent amount yesterday
<rdubya>
seems like a bunch of these are going to have to be considered good enough for now, but there are still some that I know are fixable
vtunka has quit [Quit: Leaving]
<rdubya>
well i was about to commit some more fixes, but then i tried them with prepared statements turned on and about half way through the test run *every* test started throwing exceptions lol
<rdubya>
hopefully its just another thread safety issue...
bruceadams has left #jruby [#jruby]
<kares>
how often do you guys get a hang?
<enebo>
I never do for sqlite nor mysql but I have not run postgresql
<enebo>
kares: do we still have those bench scripts we made many years ago
<enebo>
oh hah I should just look
<kares>
yeah I was using those at some point
<enebo>
kares: to make that useful do you run them
<enebo>
ok
<enebo>
I wonder if the bug in sqlite transactions is always using the same name
<enebo>
I do see mostly active_record_1 but sometimes active_record_2 or 3
<enebo>
perhaps the scaffold and main are using the same name and we are replacing the first with the second
<enebo>
because I print out create_savepoint and around all three failures there is no additional name
<enebo>
HOLY CRAP replacing active_record_1
<enebo>
heh
<enebo>
ok that looks wrong...we create_savepoint over an existing savepoint name
<enebo>
kares: ^ That sounds wrong right?
<kares>
yeah - good find: that was I had in mind
<kares>
have struggled with that in the past to some extent
<kares>
you might find something on support branch but I am not sure how far it went
<kares>
rdubya: please is this ~ what you have or I am still doing it wrong?
<enebo>
so TransactionManager @stack.size is used but we have thing running over this so I half wonder if maybe some path is not using transaction manager
<enebo>
so first transaction is the real one which does a begin_db_transaction and each additional one is a SavepointTransaction which just does create_savepoint
<enebo>
somehow there are cases where two things are using the same size
<enebo>
threads?
<enebo>
hmm good theory tim
<enebo>
err tom sorry I forget your name
<kares>
yeah, I do know it exists but I do not recall
<kares>
what I recall is your reaction it seemed weird on first sight but than somehow find its place to stay in :)
claudiuinberlin has joined #jruby
<enebo>
hmm I wonder how thread-safe Rails even works
<enebo>
since this transaction_manager has zero synch'd code
<enebo>
I am guessing every thread is supposed to get its own connection
<enebo>
perhaps the tests do not do that in a few places
<rdubya>
i'm definitely running into a thread issue, sometimes all the tests break and sometimes they don't :(
<enebo>
ok that never happens to me but I don't know now that you changed which rake task you run perhaps the bootstrapping is different
<enebo>
rdubya: in fact by basically saying you are the native adapter you might be disabling some jruby check logic which puts you into a thread-safe mode
<rdubya>
maybe
<enebo>
rdubya: were you ever seeing that before when you just hacked the postgresql tests into the mix
<rdubya>
it only happens with prepared statements turned on
<kares>
enebo: your guesses are right - connections are never to be shared
<rdubya>
haven't tracked down the actual error yet because the test suite hangs after a while
<enebo>
kares: yeah otherwise this stuff would be horribly broken
<enebo>
kares: so I guess I need to print out threads in those cases and see if they are the same connection instance
<rdubya>
enebo: i'm not sure if it has happened since then or not, i'll try going back to the hacky way of running them and see if that fixes the issue
<enebo>
rdubya: yeah I think that is a good sanity check
<enebo>
ok I will be afk a while I need to pick up Valerie at the airport then likely get lunch with her
<enebo>
for headius when he is around: I see create_savepoint in a few places create the same name and essentially replace the first savepoint. I was going to check to see if they happen to be on two threads sharing same connection (which would be a bad test probably) but I am going dark a while
<enebo>
for sqlite3 tests
<kares>
rdubya: so driver 42.0.0 - same number of fails/errors for me
<kares>
there's a couple of patch releases but I did not test PS and my numbers are close but not the same as yours
<kares>
so here's a branch with the driver updated:
<rdubya>
kares: cool, hopefully I can figure out what this issue is and then i'll be able to test with it…
<kares>
jdbc-postgres-42
<rdubya>
ok
<rdubya>
does minitest run tests in threads? I have a test file where an instance variable that is set in the setup is throwing a warning that the instance variable wasn't initialized in the teardown
<rdubya>
unless its possible that this doesn't actually initialize it:
<rdubya>
enebo: I noticed when you updated the gemfile that you commented out the thread_safe gem, was that intentional?
<kares>
rdubya: thread-safe is replaced by concurrent-ruby
<kares>
minitest does not run in parallel unless told to do so
<kares>
using parallelize_me!
<rdubya>
kares: ok on thread-safe
<rdubya>
any ideas how those instance variables become uninitialized?
<rdubya>
seems like randomly a bunch of them will have that issue
<kares>
sounds like a bug
<kares>
did you isolate it?
<rdubya>
there's one group that has it happen regularly and then occasionally about the whole test suite will have issues with them
<kares>
I mean ... got it on a single test run
<kares>
okej - would check those with MRI
<kares>
which test group is it?
<rdubya>
test/cases/migration_test.rb
<rdubya>
that one is one that regularly has it happen
<rdubya>
seems like every 3 or 4 runs it will happen
<rdubya>
and it doesn't appear to have anything to do with the adaper/arjdbc code
<rdubya>
here's the error:
<rdubya>
test/cases/migration_test.rb:70: warning: instance variable @verbose_was not initialized
<rdubya>
also semi regularly get these:
<rdubya>
test/cases/modules_test.rb:25: warning: instance variable @undefined_consts not initialized
<rdubya>
its another case of the instance variable disappearing by the time it gets to teardown
<rdubya>
it also appears to only happen when prepared statements are enabled
<rdubya>
which I'm not sure why that would matter at all
<kares>
najs - well that is just weird than
<kares>
but there must be an explanation
<kares>
lookedt at those and the base classes but there's no paralellization or suspicious as far as I can tell
<kares>
one obvious thing is mixing method/block style callbacks
<kares>
so you could try avoiding the mix and than it should 'work' e.g.
<kares>
change `teardown do` in migration_test to `def teardown`
<kares>
def setup is handled by minitest but than `teardown do` blocks are after minitest's teardown (which might clear up state)
<rdubya>
i'll give that a shot
<enebo>
rdubya: did something break from removing threadsafe?
<rdubya>
enebo: I don't think so, just was surprised that it was removed after we talked about all the threading issues lol
<rdubya>
i was having threading issues (i think) with it in anyways so I don't think it hurts to take it out
<enebo>
rdubya: ah yeah I can see why that would look weird
<enebo>
rdubya: yeah rails uses ruby-concurrent now so if we do use it for anything beside parallel testing we should definitely place it with that
<enebo>
So I am thinking of a FRIDAY RELEASE! 50.0 of the three adapters
<enebo>
but if I make progress on this savepoint stuff I might change to a SATURDAY RELEASE
<enebo>
I still have not run any rails apps against these at all though
<enebo>
I would really like more than a single scaffold example
<rdubya>
you mean friday as in today?
<enebo>
indeed
<enebo>
it is as scary as halloween but we need to start somewhere and I feel I can get another point out by rubyconf talk if need be
<rdubya>
with all the failures I'm staring at, I'm not entirely comfortable with that from the postgres side, but most of the failures probably don't matter too much
<enebo>
I contend we basically work for almost any normal use and if we don't we are not getting proper coverage from AR tests
<rdubya>
just realized AR 5.0 now supports some of the geometric stuff
<rdubya>
which we don't support that well
<enebo>
rdubya: yeah well we can just open issues for stuff that is known and specific
<enebo>
all software has known problems generally
<rdubya>
i'm about to commit some changes that add some basic support for it
<enebo>
an argument could be made that this should be another preview release but to me that is a great way of getting a lot less testing
<rdubya>
I'll just leave it at that and just try to go through the full list and see if there are any other glaring things
<enebo>
rdubya: I can wait until TUESDAY if need be but I think earlier the more likely we will get some feedback by rubyconf
<rdubya>
if we release it today, its not likely that too many people will have it in production anytime soon anyways
<rdubya>
its not like you can just change your gemfile and be using rails 5 lol
<enebo>
no if we release today it means some interested parties will try it out
<rdubya>
so I guess I'm ok with it
<enebo>
and we still know we need to fix stuff so another point will be coming out probably in the next week to week and a half
<rdubya>
would be nice to have a prioritized bug list to work off of for the next release
<rdubya>
meaning, if we release it now and people start playing with it, maybe by next week we'll have some feedback to know where to focus on the tests that are still broken
<enebo>
I really wish we could have a 51.0 too but I think it is too ambitious even for playing since I know we have extra issues stacked on the 50.0 issues
<rdubya>
yeah guessing it was added to 5.1 and backported, but thought it was amuzing
<enebo>
ok so no extra thread does the create_savepoint
<enebo>
I am now wondering if tests in AR just call create_savepoint without naming it internally and it reused because it uses current_name or whatnot
<enebo>
# Begins a new transaction. Note that nested transactions are not allowed
<enebo>
# by SQLite, so attempting to nest a transaction will result in a runtime