asdasdasdasss has quit [Ping timeout: 240 seconds]
x1337807x has joined #rubygems
x1337807x has quit [Client Quit]
asdasdasdasss has joined #rubygems
x1337807x has joined #rubygems
asdasdasdasss has quit [Ping timeout: 240 seconds]
ter_464 has quit [Ping timeout: 244 seconds]
huoxito has joined #rubygems
ter_464 has joined #rubygems
bffff_ has joined #rubygems
elia has quit [Quit: Computer has gone to sleep.]
asdasdasdasss has joined #rubygems
asdasdasdasss has quit [Ping timeout: 240 seconds]
dwradcliffe has quit [Quit: Bye]
x1337807x has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
dwradcliffe has joined #rubygems
tasaif has joined #rubygems
asdasdasdasss has joined #rubygems
<tasaif> hello, when I do gem install X it fails pretty often due to weird connection errors
<tasaif> and they're not the same every time
<tasaif> what am I doing wrong?
asdasdasdasss has quit [Ping timeout: 264 seconds]
tasaif has quit [Quit: Lost terminal]
ter_464 has quit [Quit: Leaving]
asdasdasdasss has joined #rubygems
asdasdasdasss has quit [Ping timeout: 250 seconds]
dwknoxy has quit [Quit: Textual IRC Client: www.textualapp.com]
asdasdasdasss has joined #rubygems
bffff_ has quit [Quit: Connection closed for inactivity]
asdasdasdasss has quit [Ping timeout: 240 seconds]
_whitelogger has joined #rubygems
henrikhodne has joined #rubygems
newUser1234 has joined #rubygems
newUser1234 has quit [Ping timeout: 246 seconds]
asdasdasdasss has joined #rubygems
asdasdasdasss has quit [Ping timeout: 250 seconds]
tbuehlmann has joined #rubygems
redmenace has quit [Ping timeout: 250 seconds]
havenwood has quit [Remote host closed the connection]
redmenace has joined #rubygems
Atttwwww has quit [Ping timeout: 245 seconds]
bhaak_ is now known as bhaak
asdasdasdasss has joined #rubygems
asdasdasdasss has quit [Ping timeout: 272 seconds]
elia has joined #rubygems
workmad3 has joined #rubygems
havenwood has joined #rubygems
havenwood has quit [Ping timeout: 264 seconds]
asdasdasdasss has joined #rubygems
asdasdasdasss has quit [Ping timeout: 255 seconds]
havenwood has joined #rubygems
havenwood has quit [Ping timeout: 264 seconds]
shaileshg has quit [Quit: Connection closed for inactivity]
asdasdasdasss has joined #rubygems
asdasdasdasss has quit [Ping timeout: 260 seconds]
havenwood has joined #rubygems
havenwood has quit [Ping timeout: 264 seconds]
asdasdasdasss has joined #rubygems
asdasdasdasss has quit [Ping timeout: 240 seconds]
havenwood has joined #rubygems
workmad3 has quit [Ping timeout: 246 seconds]
elia has quit [Quit: Computer has gone to sleep.]
bbrowning_away is now known as bbrowning
tcopeland has quit [Ping timeout: 264 seconds]
asdasdasdasss has joined #rubygems
asdasdasdasss has quit [Ping timeout: 250 seconds]
elia has joined #rubygems
havenwood has quit [Ping timeout: 264 seconds]
seanlinsley has quit [Quit: seanlinsley]
workmad3 has joined #rubygems
asdasdasdasss has joined #rubygems
asdasdasdasss has quit [Ping timeout: 240 seconds]
workmad3 has quit [Ping timeout: 240 seconds]
seanlinsley has joined #rubygems
tbuehlmann has quit [Quit: Leaving]
huoxito has joined #rubygems
newUser1234 has joined #rubygems
workmad3 has joined #rubygems
<qrush> ping evan vertis dwradcliffe been getting pingdom/pager duty alerts all morning
<qrush> whats going on?
<qrush> i dont have anyones cell number anymore
<dwradcliffe> qrush: sam and I are looking now
<qrush> cool
<qrush> thanks
<qrush> do you have access to teh status twitter account?
<dwradcliffe> I do
<qrush> please tweet there
<qrush> dwradcliffe: if you can give me the password i'll do it
<dwradcliffe> done
<qrush> thanks
<qrush> any ideas on whats up?
<qrush> db issues?
<dwradcliffe> getting hammered with bogus traffic
<dwradcliffe> app can't keep up
denym_OFF is now known as denym_
<qrush> oh shit :(
<qrush> ddos?
<dwradcliffe> not really
<dwradcliffe> someone has rubygems.org as a source in the `pom.xml` for maven
<qrush> can we just block that UA?
<dwradcliffe> we're going to block the url pattern
asdasdasdasss has joined #rubygems
asdasdasdasss has quit [Ping timeout: 240 seconds]
nirvdrum has joined #rubygems
havenwood has joined #rubygems
<qrush> +1
<dwradcliffe> Update: blocked several url patterns and a UA
havenwood has quit [Ping timeout: 264 seconds]
denym_ is now known as denym_OFF
dwknoxy has joined #rubygems
asdasdasdasss has joined #rubygems
asdasdasdasss has quit [Ping timeout: 245 seconds]
workmad3 has quit [Ping timeout: 244 seconds]
Ainieco has joined #rubygems
<Ainieco> is it just me or it's impossible to install passenger gem via "gem install passenger"?
<Ainieco> Unable to download data from https://rubygems.org/ - no such name (https://rubygems.org/latest_specs.4.8.gz)
workmad3 has joined #rubygems
robotblake has joined #rubygems
tcopeland has joined #rubygems
Ainieco has quit [Quit: leaving]
asdasdasdasss has joined #rubygems
asdasdasdasss has quit [Ping timeout: 245 seconds]
no_wai has joined #rubygems
workmad3 has quit [Ping timeout: 245 seconds]
<no_wai> hi, when bundler gem has been installed i'd have to run 'bundle' or am i missiçÃng sometg?
<no_wai> thing*
rafmagana has joined #rubygems
rafmagana has left #rubygems [#rubygems]
no_wai has quit [Read error: No route to host]
workmad3 has joined #rubygems
someara has joined #rubygems
<someara> y'all being ddosed or something?
<evan> ug
<evan> looks like the DOS is back.
<bbrowning> dwradcliffe: evan: what's the maven DOS look like?
<bbrowning> I ask because jruby builds, which happen quite often, do use maven to fetch rubygems
<bbrowning> and lately some jruby builds have been getting 500 errors from the maven -> rubygems proxy
workmad3 has quit [Ping timeout: 260 seconds]
<evan> sam dealt with it
<evan> but it sounded like it was actually maven thingking rubygems.org was a maven repo
<evan> and asking for maven crap.
<dwradcliffe> yeah
<dwradcliffe> someone was trying to download 1000s of jars
<bbrowning> that sounds like someone has perhaps the maven rubygems proxy as a regular maven repository setup
<bbrowning> thus maven would try to fetch every .jar dep of the project
<headius> yeah that doesn't sound like us
<headius> and it wouldn't be intermittent
<evan> yeah
<bbrowning> yeah - it does sound like a user of the rubygems proxy though - they just have their proxy setup misconfigured
<bbrowning> or maybe someone really did add rubygems.org to their maven repo as a repository? I hope not...
<headius> evan: how frequently is it happening?
<bbrowning> it should just be the proxy getting hit hard when someone has the proxy misconfigured and not rubygems.org though
<evan> I dunno
<evan> sam was dealing with it
<evan> I can look now.
<headius> ok
<evan> I don't see it now
<evan> ok, I do still see it.
<bbrowning> If you run across or dig up any examples of the kinds of URLs and the source IP, I can verify whether that's coming from the proxy we run or not
<evan> 8.25.195.27 - - [07/Aug/2014:17:07:02 +0000] "HEAD /com/twitter/finagle-core/6.9.0/finagle-core-6.9.0-javadoc.jar HTTP/1.1" 404 0 "-" "Artifactory/3.2.0"
<evan> 8.25.195.27 - - [07/Aug/2014:17:07:02 +0000] "HEAD /com/twitter/finagle-commons-stats_2.10/6.19.1-TWOTEN-3/finagle-commons-stats_2.10-6.19.1-TWOTEN-3-javadoc.jar HTTP/1.1" 404 0 "-" "Artifactory/3.2.0"
<bbrowning> hey, like magic!
<evan> i'm considering just blocking that ip
<bbrowning> hmm that's not our proxy ip at least :)
<dwradcliffe> there are several ips
<dwradcliffe> it's not hitting the app server though
<evan> where did sam put the stopper in?
<evan> I don't see it in the nginx config fgile.
<evan> file.
<evan> or is it in nginx on the app server
<evan> i'm looking on the LB
<dwradcliffe> no it's on the lb
<evan> i'm there
<dwradcliffe> I moved it
<evan> i don't see it.
<evan> to where?
<headius> hmmm
<headius> it's a comcast IP
<bbrowning> it looks like an accidental mistake versus an intentional DOS
<headius> in minnesota
<headius> ....
<bbrowning> heh
<evan> headius: did something in your basement get switched on?
<dwradcliffe> evan: /etc/nginx/filters.conf
<headius> whew...it's not mine
<evan> dwradcliffe: ah ok.
<bbrowning> headius: call up enebo and ask him if he just built jruby ;)
<headius> hah
<evan> dwradcliffe: perfect, thanks!
<bbrowning> what is Artifactory, actually?
<bbrowning> that's some hosted build repo tool?
<headius> hmm actually the traceroute continued
<headius> last resolved address is ae52.smf1-er1.twttr.com
<bbrowning> that lines up with finagle
<headius> twttr.com appears to be an alias for twitter.com
<headius> looks like someone at twitter has a wacky config
<evan> what the hel
<evan> hell
<evan> i see a few more ips
<bbrowning> headius: hey look - artifactory is in finagle's pom.xml - https://github.com/twitter/finagle/blob/master/pom.xml
<bbrowning> so someone building finagle is probably triggering it, or at least the ones with Artifactory UA
<evan> ok, I just tweet shammed that ip
<evan> we'll see what happens.
<evan> I see an Artifactory UA hitting normal endpoints
<headius> couldn't get all the way to target IP, but same subnet
<evan> too
<evan> I assume that might be some kind of maven -> rubygems mirror?
<bbrowning> that's not Artifactory's primary purpose, but perhaps that's one of the things it can do
<evan> well
<bbrowning> I don't know how they proxy though
<evan> dwradcliffe: your filter is blocking all uses of Artifactory
<evan> we should probably remove the UA filter.
<dwradcliffe> legit uses?
<evan> 209.208.213.14 - - [07/Aug/2014:17:17:49 +0000] "GET /api/v1/dependencies?gems=rest-client,amqp,cucumber,tiny_tds,rspec HTTP/1.1" 404 162 "-" "Artifactory/3.2.1.1"
<evan> thats a legit request
<evan> arg.
<evan> nevermind
<evan> we have to leave it in
<evan> because these maven repo requests don't all start with /com or /org
<evan> 8.25.195.27 - - [07/Aug/2014:17:18:35 +0000] "HEAD /backtype/jzmq/2.1.0/jzmq-2.1.0-src.jar HTTP/1.1" 404 0 "-" "Artifactory/3.2.0"
<evan> for instance.
<dwradcliffe> yeah, that's why I added it in the first place :(
<evan> yeah
<evan> leave it in for now.
<evan> until this is sorted.
<dwradcliffe> we started with /com/ then /org/ then /.meta/ and it still didn't catch them all
asdasdasdasss has joined #rubygems
<evan> 54.236.124.56 is asking for .meta stuff
<evan> thats an amazon IP
<evan> internet sheriff, my favorite job!
<evan> what the hell are people doing...
asdasdasdasss has quit [Ping timeout: 260 seconds]
<bbrowning> evan: talked to someone over in #finagle and they're looking into things
<evan> did maybe someone release something to use rubygems and maven?
<evan> today
<evan> maybe?
<headius> nothing on our end
<bbrowning> if it just started today, it probably means some active project just recently misconfigured their artifactory :/
<evan> so, checking the log.s
<dwradcliffe> I've seen these requests before (going back several months)
<evan> things have been hitting /.meta/nexus-smartproxy-plugin
<evan> a lot
<evan> in the past
<evan> so thats not new.
<evan> actually
<evan> ditto with everything.
<evan> it's not new.
<evan> 8.25.195.27 - - [14/Jul/2014:11:59:35 +0000] "HEAD /com/twitter/util-codec/6.11.2-20140211235254-32a43be/util-codec-6.11.2-20140211235254-32a43be-javadoc.jar HTTP/1.1" 404 0 "-" "Artifactory/3.2.0"
<evan> for instance.
<bbrowning> good to know, and still twitter gems so probably still related to finagle
<evan> sure
<evan> just wondering why it caused an issue today
<evan> because it's been happening for a while.
<dwradcliffe> so either the volume skyrocketed, or our app suddenly couldn't handle it
<evan> yeah
<dwradcliffe> I deployed app code last night, so could be something changed
<evan> nothing stands out.
kai3x5 has joined #rubygems
<evan> dwradcliffe: btw
<evan> we're not rotating nginx log files fast enough.
<evan> access.log is 15G
<dwradcliffe> yikes, ok
<evan> access.log.1 is 29G
<evan> I think it's getting rotated weekly
<evan> it needs to be daily.
<dwradcliffe> yep
<evan> log file management, the least exciting activity in computers.
<dwradcliffe> it beats working with timezones
<evan> only slighly.
elia has quit [Quit: Computer has gone to sleep.]
<bbrowning> hey! a google search just found an IRC nick tied to your 8.25.195.27 IP
<bbrowning> "vlee" in a dri-devel channel it appears
<bbrowning> if that's in an ISP's IP pool it may not be the same person as now - this was from 2012
<bbrowning> hey! same IP now - /whois vlee
* bbrowning will contact
<evan> probably just someone working at twitter.
<evan> or a bot
<evan> headius: was there something that let maven talk directly to rubygems.org?
<evan> at anytime in the past
<headius> hmm
<headius> well various folks have set up proxies that present a maven server transparently wrapping gems
<headius> but those generally just act as mirrors...they wouldn't be fetching so eagerly
dwknoxy is now known as dknox-bbib
<evan> the fact that their mirrors makes me sad, but yeah, I doubt thats this.
<bbrowning> the proxies I know of act as lazy mirrors, only fetching gems when requested
<bbrowning> the proxy we run just caches metadata and otherwise doesn't even proxy the actual gems
<bbrowning> we just redirect
kai3x5 has quit [Ping timeout: 244 seconds]
<bbrowning> evan: [13:53:43] <sprsquish> bbrowning: confirmed it was us (sorry!). The team has turned it off so it shouldn't happen again.
<bbrowning> from #finagle
<evan> can you ask what it was?
<evan> i'd love to know.
<bbrowning> sure - one sec
<dwradcliffe> still getting requests from other IPs
<evan> yah
<evan> thats why I want to know what the finagle people were running
<bbrowning> he's finding out for me
<evan> might help us narrow does these other ips
<dwradcliffe> 66.35.37.165 72.13.55.7 208.185.253.195
<bbrowning> evan: [14:03:48] <sprsquish> bbrowning: he says. I think I've just fixed the issue on our end by setting an includes pattern on the remote repo for "**/*.gem"
<bbrowning> it makes me wonder if perhaps Artifactory's product is kind of buggy when it comes to rubygems handling out of the box
<bbrowning> you'd think it would be smart enough to only send gem requests to rubygems.org
<evan> ok
<evan> now the big question
<evan> wtf is Artifactory?
<headius> evan: we'd love to have the maven support on rg.org directly, of course :-)
<evan> so something in Artifactory that supports jruby
<evan> tries to treat rubygems.org as a maven repo?
<bbrowning> that's what it looks like, yes
<headius> that's pretty weird
<bbrowning> or twitter just misconfigured their instance to treat rubygems.org as a maven repo
<headius> it doesn't even talk about supporting gems
<evan> based on his comment, i'm guessing
<evan> is wrong.
<evan> I see "Include Pattern" in there defaulting to **/*
<headius> ahh interesting
<bbrowning> some more data is coming in from finagle team - will gist in a minute
<headius> I wonder if artifactory just tries all protocols against all configured repositories or something
<evan> k
<evan> headius: eyah
<headius> so adding rubygems.org would make it try to use it as a maven backend as well as gems
<headius> it doesn't seem to distinguish in that UI
<evan> that UI seems to suggest that rubygems is turned on IN ADDITION to maven support
<headius> yeah exactly
<headius> when it should actually be in place of
<evan> yah
<evan> I suppose it's possible for someone to setup a server that does both
<evan> but since their default is all about using rubygems.org as the backend
<evan> thats plain wrong.
<bbrowning> so the artifactory config person for finagle is around and wants to try turning something on to see if it still hits rubygems.org wrong
<bbrowning> do one of you want to hop in #finagle and talk or should I continue to proxy? :)
<evan> sure
<headius> I will too
<bbrowning> thanks!
asdasdasdasss has joined #rubygems
elia has joined #rubygems
sferik has joined #rubygems
asdasdasdasss has quit [Ping timeout: 240 seconds]
<bbrowning> this reminds me we need to do a "how many 404 requests per second" benchmark
<bbrowning> heh
<bbrowning> "rubygems.org switched from <? - puma? unicorn? passenger?> to foo because it can handle 10x the number of 404 requests per second"
someara has left #rubygems [#rubygems]
bbrowning has quit [Read error: Connection reset by peer]
bbrowning has joined #rubygems
workmad3 has joined #rubygems
asdasdasdasss has joined #rubygems
asdasdasdasss has quit [Ping timeout: 240 seconds]
elia has quit [Quit: Computer has gone to sleep.]
<dwradcliffe> evan: does bluebox still host anything for us? aka can we remove their logo?
<evan> they still host a mirror.
<evan> 20% of our traffic goes to them.
<evan> :q
<evan> hah
<evan> oops.
<dwradcliffe> ah, bb-m
<evan> yeah, weird name but it's there so the SSL certs work.
<dwradcliffe> who has access to the gaug.es account?
<evan> nick and I
<evan> I believe
dknox-bbib is now known as dknox
x1337807x has joined #rubygems
sferik has quit [Ping timeout: 250 seconds]
x1337807x has quit [Max SendQ exceeded]
x1337807x has joined #rubygems
x1337807x has quit [Max SendQ exceeded]
x1337807x has joined #rubygems
x1337807x has quit [Max SendQ exceeded]
x1337807x has joined #rubygems
x1337807x has quit [Max SendQ exceeded]
x1337807x has joined #rubygems
x1337807x has quit [Max SendQ exceeded]
x1337807x has joined #rubygems
x1337807x has quit [Max SendQ exceeded]
x1337807x has joined #rubygems
x1337807x has quit [Max SendQ exceeded]
x1337807x has joined #rubygems
x1337807x has quit [Max SendQ exceeded]
x1337807x has joined #rubygems
elia has joined #rubygems
asdasdasdasss has joined #rubygems
asdasdasdasss has quit [Ping timeout: 245 seconds]
havenwood has joined #rubygems
ged_ is now known as ged
bbrowning is now known as bbrowning_away
x1337807x has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
elia has quit [Quit: (IRC Client: textualapp.com)]
asdasdasdasss has joined #rubygems
asdasdasdasss has quit [Read error: No route to host]
asdasdasdasss has joined #rubygems
x1337807x has joined #rubygems
x1337807x has quit [Max SendQ exceeded]
x1337807x has joined #rubygems
x1337807x has quit [Max SendQ exceeded]
x1337807x has joined #rubygems
x1337807x has quit [Max SendQ exceeded]
x1337807x has joined #rubygems
x1337807x has quit [Max SendQ exceeded]
x1337807x has joined #rubygems
x1337807x has quit [Max SendQ exceeded]
sferik has joined #rubygems
x1337807x has joined #rubygems
x1337807x has quit [Max SendQ exceeded]
x1337807x has joined #rubygems
x1337807x has quit [Max SendQ exceeded]
x1337807x has joined #rubygems
seanlinsley has quit [Quit: seanlinsley]
newUser1234 has quit [Remote host closed the connection]
asdasdasdasss has quit [Quit: Computer has gone to sleep.]
asdasdasdasss has joined #rubygems
asdasdasdasss has quit [Read error: No route to host]
asdasdasdasss has joined #rubygems
workmad3 has quit [Ping timeout: 260 seconds]
asdasdasdasss has quit [Ping timeout: 264 seconds]
sferik has quit [Quit: Textual IRC Client: www.textualapp.com]
x1337807x has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
x1337807x has joined #rubygems
x1337807x has quit [Ping timeout: 260 seconds]
Rennex_ is now known as Rennex
asdasdasdasss has joined #rubygems
huoxito has quit [Remote host closed the connection]
Atttwww has joined #rubygems
havenwood has quit []
seanlinsley has joined #rubygems
<evan> dwradcliffe: you around?
<dwradcliffe> yep
<evan> can you remove the UA filter for Artifactory
<evan> it seems to have calmed down
<evan> and jfrog is working on a fix
<dwradcliffe> ok give me a min and I'll do it
<evan> thanks
<evan> dwradcliffe: I'll go ahead and do it.
<evan> actually, go ahead.
<dwradcliffe> evan: sorry had to take care of something. did you do it?
<evan> not in chef
<evan> just on the server so the guys could try it
<evan> please change it in chef, thanks.
<dwradcliffe> I just restarted nginx and they are hitting the app server now
<dwradcliffe> still a lot of requests
<evan> add .xml
<evan> to the filter
<dwradcliffe> and .pom