arohner_ has quit [Remote host closed the connection]
arohner has joined #rubygems
jwinter has joined #rubygems
redmenace has joined #rubygems
jwinter has quit [Ping timeout: 246 seconds]
phinfone_ has joined #rubygems
phinfonet has quit [Ping timeout: 246 seconds]
tarcieri has joined #rubygems
<tarcieri>
ohai
<tarcieri>
we're hacking away here at Square, heh
einarj has joined #rubygems
<drbrain>
hello
<drbrain>
I mostly want to make myself available to answer questions
<drbrain>
I want to leave security to experts!
pbiggar_ has joined #rubygems
einarj has quit [Ping timeout: 245 seconds]
<pbiggar_>
hey folks. I've been reading the logs since I was here last about the Bundler::Fetcher::CertificateFailureError problems
<pbiggar_>
It looks like fastly has been disabled, and the problems are still being reported
<pbiggar_>
Which matches what we're seeing in customer builds at CircleCI
<pbiggar_>
We had it happen at 10:14am Pacific this morning for one customer, for example
<pbiggar_>
Any thoughts?
bbrowning_away is now known as bbrowning
<lmarburger>
evan: will you be around tomorrow?
<evan>
yeah
<evan>
we have to get fastly to help out
<evan>
I get nearly constant reports of 503s
<pbiggar_>
I thought fastly was disabled?
<evan>
it is.
<lmarburger>
did you re-enable it just now and had to re-disable it?
<pbiggar_>
We're not seeing fastly URLs in the errors today
<lmarburger>
they had downtime a few days ago
<evan>
I disabled it last week
<evan>
it was unrelated to any downtime
<lmarburger>
that's when you disabled it
<lmarburger>
oh then i don't know
<evan>
the reports were not downtime related
<dwradcliffe>
evan disabled it shortly after the fastly downtime
<evan>
they were all 503s hitting the fastly urls
<evan>
not use
<evan>
us
<evan>
fastly had a downtime?!
<evan>
I guess I didn't hear about that.
<lmarburger>
yeah :(
<evan>
and thats not good at all.
<lmarburger>
tyler was going to ping you about it
<evan>
ok
<evan>
i'm not feeling great about fastly atm.
<evan>
because even without the outage
<evan>
i still get 503 reports
<pbiggar_>
So are the 503s related to the certificate errors?
<evan>
no
<evan>
completely unrelated.
<evan>
the 503s I see are "backend timeout"
<pbiggar_>
Yeah, that was my guess. Any thoughts on the certificate errors we're seeing today?
<evan>
it's not an SSL error.
<lmarburger>
honestly, i'd blame the bundler-api backend before blaming fastly
<evan>
which errors?
<evan>
lmarburger: it's not the bundler-api backend
<evan>
well
<pbiggar_>
evan: Bundler::Fetcher::CertificateFailureError: Could not verify the SSL certificate for https://rubygems.org/.
<lmarburger>
how do you know?
<evan>
how do you have fastly configured?
<evan>
isn't in talking to s3 to get .gem files?
<evan>
pbiggar_: not much to go on
<evan>
is that all the time?
<evan>
are you seeing it?
<lmarburger>
it all goes through bundler-api. bundler-api returns a redirect to s3 which fastly follows instead of returning that to the client.
<evan>
lmarburger: fuck.
<evan>
I had no idea
<evan>
I thought it went straight to s3
<evan>
i'm leaving it off
<lmarburger>
no. that's what i'm doing next is moving those redirects into varnish.
<evan>
until we can actually configure it right.
<pbiggar_>
evan: a few complaints a day. Which probably means it happens a few 100 times a day (now would be a good time to have search in our build logs :(
<evan>
i don't see why there are redirects
<lmarburger>
then you can re-eanble it and see if it still 503s. then it's definitely fastly (or our s3 backend configuration)
<evan>
fastly should get files from s3 and cache them
<evan>
no redirects involved.
<lmarburger>
right
<evan>
so basically, bundler-api/heroku are timing out randomly talking to fastly
<tarcieri>
drbrain: we're also having a Hangout/Skype with the TUF people tomorrow
<lmarburger>
and that was part of the warehouse stuff to stand up a bundler/rg.org mirror
<lmarburger>
which came in 1.5 (correct me if i'm wrong, indirect)
<evan>
experiments are ok
<indirect>
evan: it's there specifically to support people pointing directly at the bundler-api URL as a source
<lmarburger>
this shipped in bundler 1.5
<lmarburger>
not really an experiment at this point
<evan>
but I'm getting a little worried because I thought you guys talked about 1.5 using fastly explicitely
<evan>
perhaps I heard/remember wrong
<evan>
lmarburger: what is shipped?
<indirect>
no
<indirect>
1.5 does not use fastly explicitly
<indirect>
and we have never planned on that
<evan>
ok
<indirect>
we want to pull all of rubygems.org up onto a CDN so that everyone can use it
<evan>
what is the code in bundler for then? just an experiment?
<indirect>
yes
<indirect>
well
<evan>
sounds like lmarburger means it to be more than that.
<lmarburger>
1.5 has the ability to configure a source (i forget the name of that option) so CI services can stand up a mirror (bundler warehouse) and reverse proxy cache it
<indirect>
it's there to support us testing/experimenting
<indirect>
oh
<indirect>
yes; but that's incidental to the code in bundler-api
<lmarburger>
that's why those endpoints in bundler-api exist
<evan>
so they're feature slop
<evan>
it's not bundler-api specific at all
<evan>
they were just a place to throw those endpoints
<indirect>
evan: the redirects are there so that the URL will work as a source in bundler
<evan>
thats fine
<evan>
just trying to understand.
<indirect>
yes
<evan>
but no one does that, right?
<evan>
:)
<indirect>
we do while testing :)
<indirect>
but not otherwise, no
<indirect>
so when we were most recently switched to fastly
<evan>
i'm confused now.
<evan>
fastly goes to bundler-api instead of s3
<evan>
right now
<evan>
but where does it go?
<evan>
these endpoints? but these endpoints point to fastly
<evan>
isn't that a loop?
<indirect>
evan: in the production bundler-api, the RUBYGEMS_URL is the production cloudfront bucket
<indirect>
while testing fastly, we set up fastly in front of bundler-api, because that was all we had control over at that point
<lmarburger>
bundler-api returns a redirect to an s3 url (exactly like rg.org does). fastly is configured to see that redirect, follow it, and return the file.
<indirect>
bundler-api needed to return valid gems and gemspecs so that it could be used as a source
<lmarburger>
that way requests for .gem files return a 200 instead of 30x
<evan>
why does it make sense for fastly to hit bundler-api instead of s3?
<indirect>
it doesn't
<evan>
why the extra redirect in the first place?
redmenace has quit [Ping timeout: 246 seconds]
<evan>
but why did you do that to begin with?
<evan>
i'm missing something
<indirect>
because we needed API responses
<indirect>
and only bundler-api can provide those
<indirect>
s3 cannot
<evan>
oh.
<evan>
you wanted to use fastly as a source with API responses
<evan>
directly.
<evan>
I got it now
<indirect>
well, using fastly as a source meant it needed to supply everything
<indirect>
which is our eventual goal
<indirect>
so yes
<indirect>
so the intention was to get fastly talking directly to s3 before rubyconf
<indirect>
but it turned out to be a little more complicated than we thought
<indirect>
and that's why it dragged out to this week
<evan>
is there a chance to back up
<indirect>
heh
<indirect>
sure
<evan>
and point fastly to s3 and break the API stuff
<evan>
because no one is using it anyway
<indirect>
uh
<evan>
(you said no one is using it)
<indirect>
oh
<lmarburger>
why would that break anything?
<evan>
because if you sent fastly an API request
<indirect>
different kinds of no one is using it
<evan>
it would send that to s3
<evan>
and s3 would say "huh?"
<indirect>
yes
<evan>
but the .gem files would work fine
<indirect>
so we can totally make that work
<indirect>
right now
<indirect>
but we need some rubygems.org config cooperation
<indirect>
:)
<evan>
should we?
<evan>
in what way?
<indirect>
the nginx config will need to look like this:
<evan>
it already redirects to fastly only for .gem files
lsegal has joined #rubygems
<indirect>
that is not the behaviour we were seeing
<evan>
or, rather, is configured to do that for mirrors.
<evan>
url?
<evan>
are you seeing it send API traffic to cloudfront right now?
<lmarburger>
evan: wait. .gem files should be hitting fastly now?
<evan>
no.
<evan>
i disabled that.
<indirect>
yeah
<evan>
unless another admin turned it back on without telling me.
<lmarburger>
no. i just installed a gem and it got it from s3.
<indirect>
so
<evan>
indirect: what behavior are you concerned about?
<indirect>
what I would like is for /api/v1/dependency to go to bundler.rubygems.org
<indirect>
and for /*.gem and /*.gemspec to go to Fastly backed by S3
<evan>
that already happens
<lmarburger>
indirect: i'd also suggest that *specs.*.gz go to fastly backed by s3 as well
<evan>
except it goes to CF instead of fastly
<indirect>
with *.gem and *.gemspec going to cloudfront right now instead of fastly?
<indirect>
yeah, okay
<evan>
lmarburger: no
<evan>
we are not doing specs now.
<evan>
that is off the table.
<evan>
let's get .gem's working
<evan>
then we'll look at specs
<lmarburger>
right. obviously after this is proven working.
<evan>
we're in the middle of issues, I want to get them ironned out smoothly before adding more.
jkingdon has quit [Remote host closed the connection]
<evan>
I'm also going to need admin access to the fastly account
<evan>
to do configuration.
<evan>
I may have it already
<evan>
just wanted to be clear
<lmarburger>
you have access
jwinter has joined #rubygems
<evan>
cool
<indirect>
yes, you should have access already
<indirect>
let me know if you have trouble getting in
<evan>
are you ok with me going in there and configuring it the way I want for now?
<lmarburger>
have at it
<evan>
ok
<evan>
i'm going to break using fastly as a direct source then.
<indirect>
yeah
<indirect>
it looks like there's config that tries to avoid the bundler-api hits for gems
<indirect>
but I guess it wasn't working
<indirect>
we have the old versions to use for reference
<indirect>
for later
<lmarburger>
evan: do you know what you're doing with configuring fastly? the configuration for this stuff is spread out all over.
<evan>
probably not, but I can figure it out
<evan>
this isn't my first rodeo
<indirect>
sure
<evan>
given that it's currently dead in the water
<evan>
i'm not too concerned with being overly careful.
<lmarburger>
well i assume you know what you're doing with varnish but fastly has a bunch of extra config
<evan>
I gotta run now, but i'll take a look at this tomorrow.
<indirect>
okay
<lmarburger>
evan: i think i'll go through and delete all the bundler-api backend specific stuff awhile
<indirect>
I will see about getting things set up for the working setup we discussed
<evan>
ok
<indirect>
so that we can test and maybe turn it back on if it does work tomorrow
<samkottler>
tarcieri: yo, we should talk about TUF stuff
<samkottler>
tarcieri: I'd love to get involved
<samkottler>
tarcieri: I've been working on TUF for Tor in Go so I have some experience writing clients
josh-k has quit [Remote host closed the connection]
DanKnox_away is now known as DanKnox
mib_mib has joined #rubygems
<mib_mib>
when running bundle install, i'm getting "https://github.com/jnunemaker/httparty.git (at master) is not checked out. Please run `bundle install`" - is this a github or rubygems issue?
<tarcieri>
samkottler: awesome!
<tarcieri>
samkottler: we're gonna try to get all the code we wrote yesterday pushed to our public github forks today
<samkottler>
tarcieri: cool, just shoot me a link to the repo when you do?
dbussink_ has quit [Remote host closed the connection]
dbussink has quit [Ping timeout: 272 seconds]
dbussink_ has joined #rubygems
dbussink_ has quit [Remote host closed the connection]
<drbrain>
well, it seems to be old ruby license
<drbrain>
but we can qualify under the rename clause
dbussink has joined #rubygems
jkline has joined #rubygems
jkingdon_ has quit [Remote host closed the connection]
jkingdon has joined #rubygems
<jkline>
In my gemspec file how do I indicate I am willing to accept a pre-release version of a gem? I would normally use ~> 1.0 , but in this case I want to allow 1.0.0beta1 also
jkingdon has quit [Read error: Connection reset by peer]
jkingdon has joined #rubygems
bbrowning is now known as bbrowning_away
<drbrain>
jkline: '< 2.0', '>= 1.0.0beta1'
<drbrain>
(assuming semver)
<jkline>
drbrain: ok, the obvious non-magical way. Thanks.
<drbrain>
np!
ZachBeta has quit [Quit: Computer has gone to sleep.]
<tarcieri>
I pushed an initial TUF commit! :O :O :O