<headius>
many of the posts talked about a problem with TLS versus SSLv3 negotiation and that turning TLS off would fix it, but turning it off in net/http did not seem to help
<headius>
this only seems to have started within the last few weeks
<qrush>
i haven't seen this...does it happen only with jruby?
<headius>
I'm not sure
<qrush>
what OS?
<headius>
OS X
<qrush>
yeesh
<qrush>
is this just a normal `gem install` with jruby?
<qrush>
wondering how we can try to reproduce it
<qrush>
what gem was it?
<headius>
yeah it's hard to reproduce here too
<headius>
it's intermittent
<headius>
nirvdrum on #jruby seems to get it fairly often… I've only seen it once ever
<headius>
Sorry, he's on Linux
<headius>
I think I've seen it on OS X but unsure
<samkottler>
hey, I'm around for a while if you want help debugging this
<samkottler>
qrush: ^^
<headius>
samkottler: I'd really like to know if there's any sign of this on the server side
<headius>
I have client side logs but feel like it's only half of the story
<samkottler>
headius: I can check
<samkottler>
headius: the problem is that we terminate some much SSL traffic that it's hard to debug because of the pure volume
<headius>
yeah I understand
nirvdrum has joined #rubygems
<nirvdrum>
Howdy.
<headius>
nirvdrum: hi there
<headius>
nirvdrum: this has only started happening recently for you too, right?
<samkottler>
nirvdrum: hey! which OS and version of it are you using?
<headius>
like, within the last month
<headius>
?
<nirvdrum>
headius: No, it's been going on a while. I just started hitting it more frequently because I keep testing JRuby 1.7.5 builds and having to bundle from scratch.
<headius>
nirvdrum: like how long? trying to figure out if there was a triggering event
<nirvdrum>
samkottler: On my servers, it's Ubuntu 12.04 amd64.
<nirvdrum>
On my laptop, it's LinuxMint 15, an Ubuntu 13.04 derivative.
<samkottler>
headius: nirvdrum: we replaced a load balancer about a month ago which is the only change we'd have made recently
<samkottler>
nirvdrum: that should have a recent enough version of ca-certificates
<samkottler>
hmmm
<nirvdrum>
headius: Hard to say. I'd peg it at several months at least. But I couldn't say if it was more than 4.
<nirvdrum>
rubygems.org was having problems a while back so I always attributed the problem to that.
<headius>
can you trigger one right now?
<headius>
when I saw it I shrugged it off as being rg.org issues too, or a dropped connection
<nirvdrum>
samkottler: Well, if I retry it'll eventually succeed. So I don't think my root certs are to blame.
<samkottler>
nirvdrum: what region of the world are you in?
<headius>
I have been wholly unsuccessful in reproducing it
<nirvdrum>
samkottler: Suburban Boston.
<nirvdrum>
My servers are in Atlanta.
<samkottler>
nirvdrum: ha, that's funny, I'm outside boston, too :)
<nirvdrum>
Previously they were on EC2, in Virginia.
<headius>
no releases correspond to 4 months ago
<headius>
hmm
<nirvdrum>
headius: That's about the time JRuby 1.7.4 came *shrug*
<samkottler>
nirvdrum: it's not related to handshake times, then
<nirvdrum>
I really can't say if that was the cause though. And I had to skip 1.7.3 entirely because of that Fiber bug.
<headius>
nirvdrum: yeah I guess so…4-5 months ago
<nirvdrum>
Let me blow away my gems and start over. I'm bound to hit it.
<headius>
I'm going to bring up a linux box and see if I can trigger it there
<headius>
fwiw we've only had a handful of reports of this
<nirvdrum>
samkottler: I live in Holliston.
<headius>
so it's weird
<nirvdrum>
headius: I've hit it sometimes with honeybadger.io, too. But I hit it a lot with rubygems.org. It could very well just be a numbers game though.
<nirvdrum>
I've wondered if bundler might just be opening too many connections and rg.org is rejecting one, too.
<samkottler>
I've been thinking about replacing nginx for SSL termination because it's so unreliable
<samkottler>
and you might just be running into nginx problems
<samkottler>
but it's hard to verify
<nirvdrum>
Well, that's a first. I just got: "Could not verify the SSL certificate for https://rubygems.org/.
<nirvdrum>
There is a chance you are experiencing a man-in-the-middle attack, ..."
<samkottler>
nirvdrum: are you going through a proxy?
<nirvdrum>
Nope. Not unless my route tables are screwed up running through a VPN link to my datacenter. I can check that.
<nirvdrum>
Just re-ran bundler and it's going now.
<samkottler>
the really odd part is that some people seem to bear the brunt of the issues
<samkottler>
some are completely fine and others have issues a lot of the time
<nirvdrum>
Maybe only one of the nginx boxes is screwed and I just luck out with round-robin?
<Rotonen>
s3 has new certs at least
<headius>
I set up a script earlier today to repeatedly reinstall gems and bundle a new app and was unable to trigger it
<Rotonen>
triple sure your certs are state of art?
<headius>
but this is a different env than nirvdrum
<nirvdrum>
Rotonen: The distro I'm using right now is only 4.5 months old.
<Rotonen>
and that your rubygems uses the cert store you update?
<nirvdrum>
And I'm using a JVM that was only released like 2 weeks ago. I believe that has its own root store on top of that.
<headius>
I'm certainly willing to accept if it's a JRuby issue but I don't know of anything that would have changed in this timeframe to cause it
<headius>
I'm trying some suggestions on my end that involve limiting protocol to just SSLv3 again
<samkottler>
headius: I don't think that'll fix the issue, just mask it on some implementations
<nirvdrum>
samkottler: Well, it looks like it's failing when connecting to rg.org to get the gem. IIRC, you receive the request and then do a 302.
ged__ is now known as ged
<nirvdrum>
I write to S3 over https a lot and don't recall ever hitting that issue, at least.
<headius>
samkottler: yeah definitely not fix
<samkottler>
nirvdrum: yep, we just issue a 302 via nginx
<headius>
well it's always that first hit to RG that blows up for me
<headius>
(I did manage to reproduce with Oracle JDK on Linux, same setup as nirvdrum)
<nirvdrum>
headius: Not the first hit for me, but it almost always blows up at some point in the process of bundling.
<headius>
I mean the first of the two SSL sessions for a given gem
<headius>
based on SSL logging
<nirvdrum>
Ahh.
<headius>
it's never the larger second one where it's fetching
<headius>
nirvdrum: I've had three full bundles of your Gemfile without failure with my SSLv3-only patch
<drbrain>
headius: are you using ruby 2.0's net/http?
<headius>
no, still 1.9 by default with 1.9 stdlib
<drbrain>
ok
<nirvdrum>
headius: The patch you had me apply earlier?
<headius>
no, one inside SSLContext.java
<drbrain>
ruby 2.0's has SSL session resumption to save negotiation time (50-100ms)
<nirvdrum>
Ahh.
<nirvdrum>
If you have a tarball, I can try on my end. But it sounds like you were finally seeing what I was seeing anyway.
<nirvdrum>
I wonder what the differences are between OpenJDK and Oracle.
<nirvdrum>
If only we knew someone at RedHat :-)
<headius>
hah
* drbrain
chuckled
<headius>
ahh scratch that, 4th time's the charm
<headius>
bad_record_mac back again
<nirvdrum>
Shut it down.
TheComrade has quit [Quit: TheComrade]
DanKnox is now known as DanKnox_away
arthurnn has joined #rubygems
jnimety has joined #rubygems
jnimety has quit [Quit: Computer has gone to sleep.]
headius has quit [Quit: headius]
tbuehlmann has quit [Remote host closed the connection]
tcopeland has quit [Quit: Leaving.]
jcaudle has quit [Quit: jcaudle]
jnimety has joined #rubygems
jnimety has quit [Client Quit]
almostwhitehat has quit [Ping timeout: 264 seconds]
arthurnn has quit [Remote host closed the connection]
almostwhitehat has joined #rubygems
arthurnn has joined #rubygems
ZachBeta has joined #rubygems
DanKnox_away is now known as DanKnox
arthurnn has quit [Ping timeout: 240 seconds]
ZachBeta has quit [Quit: Computer has gone to sleep.]
TheMoonMaster has quit [Remote host closed the connection]
huoxito has quit [Remote host closed the connection]
workmad3 has joined #rubygems
huoxito has joined #rubygems
enebo has quit [Quit: enebo]
huoxito has quit [Ping timeout: 260 seconds]
marpaia has quit [Quit: WeeChat 0.4.1]
luislavena has quit [Remote host closed the connection]
tcopeland has joined #rubygems
arthurnn has joined #rubygems
jaimef has quit [Excess Flood]
huoxito has joined #rubygems
jaimef has joined #rubygems
huoxito has quit [Ping timeout: 256 seconds]
shvelo has quit [Ping timeout: 245 seconds]
fromonesrc has joined #rubygems
imperator has joined #rubygems
fromonesrc has quit [Client Quit]
workmad3 has quit [Read error: Operation timed out]
arthurnn has quit [Remote host closed the connection]
arthurnn has joined #rubygems
huoxito_ has joined #rubygems
arthurnn has quit [Ping timeout: 252 seconds]
huoxito_ has quit [Remote host closed the connection]
huoxito_ has joined #rubygems
huoxito_ has quit [Ping timeout: 264 seconds]
<drbrain>
ha, sweet!
<drbrain>
aww, dang
<drbrain>
almost
<drbrain>
… have on-demand extension building working
jnimety has joined #rubygems
<jnimety>
hello all. I tried to do a cap deploy this afternoon and received the following error "Could not verify the SSL certificate for https://rubygems.org/" using Fedora 18, capistrano, rvm, latest fedora packages are installed. looking for advice. thanks.
<drbrain>
jnimety: install an updated certificate bundle
<jnimety>
drbrain I have the latest provided by fedora
<mpapis>
drbrain, on-demand this sounds great, is it on a branch?
<drbrain>
jnimety: S3's certificates changed, and we don't bundle the updated certificate
<drbrain>
mpapis: yes, on a branch
<drbrain>
I'm trying to figure out why the off-platform install is failing
<mpapis>
I would love to see it, maybe I could help a bit
<jnimety>
drbrain should I bug fedora to update their cert bundle?
<drbrain>
jnimety: we haven't figured out which certificate we need to update in rubygems yet :/
<drbrain>
but, others have reported success updating the certificates
<drbrain>
the less-secure option is to disable HTTPS :/
<jnimety>
yeah, I'm looking to avoid that
<jnimety>
odd that the latest fedora has to offer has issues with s3
<jnimety>
drbrain thanks for the help, I'll see what the fedora folks have to say
<drbrain>
sorry I don't have more to offer yet ☹
<drbrain>
I've been sick and haven't had the time to investigate yet
<imperator>
drbrain, any chance of building a command into rubygems to update certs?
<imperator>
or is that outside of rg's scope?
<jnimety>
no worries, I'll pass along any info I find
<drbrain>
imperator: updating all the certificates is out of scope