<Perceptes>
I feel like some big companies who use ruby should sponsor rubygems, it's necessary for the whole language ecosystem and really needs 24/7 on call support
<Perceptes>
it's not fair to evan, qrush, et al to support such a big thing as volunteers
arohner has quit [Remote host closed the connection]
<Perceptes>
it's kind of absurd when you think about it
huoxito has quit [Ping timeout: 246 seconds]
<Perceptes>
looks like status.rubygems.org only checks if the various endpoints respond to pings
<Perceptes>
hence them showing it as up
huoxito has joined #rubygems
<Perceptes>
qrush probably won't see this until the morning, it's 2 AM east coast
baburdick has joined #rubygems
viranch has quit [Quit: No Ping reply in 180 seconds.]
viranch has joined #rubygems
<bobthecow>
woot! response on the twitters!
<evan>
i'm around now.
<evan>
are things fixed now?
<evan>
or still issues?
sferik has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
<Perceptes>
logrotate doesn't seem to have affected that
<evan>
Perceptes: yep, got the error here.
<Perceptes>
cool
<vertis>
right
<vertis>
well it's not related to the space problem
<Perceptes>
ok
<evan>
i do hate computers.
<evan>
ARG
<vertis>
:D
<qrush>
i dont know the rubygems_status pw...can't reset it either. let's get that updated first :)
<vertis>
unfortunately I have to run
<vertis>
I have a dinner
<evan>
fuck you redis
<qrush>
it would be fun to get that wired up so we dont need to log into it. maybe we can use IFTTT
<evan>
fuck you so hard.
<evan>
2013-12-12_06:55:09.47887 Redis::CommandError - OOM command not allowed when used memory > 'maxmemory'.:
<vertis>
lol
baburdick has quit [Remote host closed the connection]
<Perceptes>
redis is taking a beating this week
<vertis>
I see you have this under control
baburdick has joined #rubygems
<qrush>
thanks vertis.
<qrush>
o/
<vertis>
bye
vertis has quit [Quit: vertis]
arohner has quit [Ping timeout: 272 seconds]
josh-k has joined #rubygems
<evan>
i'm running my redis prune script
<evan>
to get the memory back down.
<qrush>
alright my brain cant brain, need to go
huoxito has quit [Remote host closed the connection]
<evan>
the issue should be fixed
<qrush>
at least the box can fill up with logs and memory can oom and gems keep getting served
<qrush>
thats pretty cool
<evan>
prune is clearing space for new data
<evan>
this was a redis OOM
<evan>
the good kind
<evan>
the one where you configure a good max memory value
<evan>
and redis thusly doesn't tank the whole machine.
<evan>
well, tomorrow i'll be writing a new stats backend for rubygems.org
<evan>
:)
<qrush>
sounds like a cereal. Redis OOM - don't WAIT for it!
<Perceptes>
gem push works again
<Perceptes>
thanks gents
sferik has joined #rubygems
<Perceptes>
on the topic, what service/software would you use to detect and alert on anomalies in business metrics, e.g. "there are usually X gem pushes per hour but there have been none for 3 hours"
sferik has quit [Client Quit]
<qrush>
woot, thanks evan. a quick writeup would be nice before rewriting stuff ;)
<evan>
surely.
<qrush>
Perceptes: we use nagios for this at 37. just a sql query would solve this
<qrush>
some smarter alerts in datadog could do us over here
mootpointer has quit [Ping timeout: 240 seconds]
<Perceptes>
sure, if you want to set the thresholds you're looking for manually
<Perceptes>
it'd be cool to have a system that analyzes historical data to determine what "anomalous" is though
<evan>
I want to build something just like that
<evan>
just in general
<Perceptes>
sort of coincidentally that's what i've been researching for the past week at work
<Perceptes>
sumologic seems to be the closest thing that currently exists
<qrush>
alright, sleep. thanks again folks. o/
<Perceptes>
thanks qrush, good night
<evan>
nite bud
<evan>
hope the kiddo sleeps well
<bobthecow>
thanks all!
jhn has joined #rubygems
josh-k has quit [Remote host closed the connection]
josh-k has joined #rubygems
bf4 has joined #rubygems
josh-k has quit [Ping timeout: 252 seconds]
bf4 has quit [Ping timeout: 240 seconds]
le_gars has joined #rubygems
baburdick has quit [Quit: Leaving.]
mootpointer has joined #rubygems
baburdick has joined #rubygems
josh-k has joined #rubygems
mootpointer has quit [Ping timeout: 264 seconds]
baburdick has quit [Ping timeout: 252 seconds]
Who has joined #rubygems
mootpointer has joined #rubygems
arohner has joined #rubygems
hakunin has quit [Ping timeout: 245 seconds]
arohner has quit [Ping timeout: 260 seconds]
Who has quit [Ping timeout: 272 seconds]
redmenace has joined #rubygems
redmenace has quit [Ping timeout: 246 seconds]
tbuehlmann has joined #rubygems
mootpointer has quit [Quit: ENOCAFFEINE.]
workmad3 has joined #rubygems
arohner has joined #rubygems
baburdick has joined #rubygems
arohner has quit [Ping timeout: 246 seconds]
baburdick has quit [Ping timeout: 260 seconds]
josh-k has quit [Ping timeout: 240 seconds]
josh-k has joined #rubygems
bf4 has joined #rubygems
bf4 has quit [Ping timeout: 272 seconds]
josh-k has quit [Ping timeout: 240 seconds]
josh-k has joined #rubygems
jordia65 has joined #rubygems
arohner has joined #rubygems
redmenace has joined #rubygems
baburdick has joined #rubygems
arohner has quit [Ping timeout: 260 seconds]
redmenace has quit [Ping timeout: 252 seconds]
baburdick has quit [Read error: Operation timed out]
ged has quit [Read error: Connection reset by peer]
ged has joined #rubygems
bf4 has joined #rubygems
bf4 has quit [Ping timeout: 246 seconds]
lsegal has quit [Quit: Quit: Quit: Quit: Stack Overflow.]
josh-k has quit [Ping timeout: 272 seconds]
josh-k has joined #rubygems
arohner has joined #rubygems
arohner has quit [Ping timeout: 264 seconds]
baburdick has joined #rubygems
baburdick has quit [Ping timeout: 264 seconds]
jordia65 has quit [Read error: Operation timed out]
jordia65 has joined #rubygems
jordia65 has quit [Ping timeout: 240 seconds]
jordia65 has joined #rubygems
bf4 has joined #rubygems
luislavena has joined #rubygems
bf4 has quit [Ping timeout: 240 seconds]
arohner has joined #rubygems
redmenace has joined #rubygems
redmenace has quit [Ping timeout: 250 seconds]
baburdick has joined #rubygems
arohner has quit [Ping timeout: 240 seconds]
baburdick has quit [Ping timeout: 272 seconds]
bf4 has joined #rubygems
bf4 has quit [Ping timeout: 264 seconds]
le_gars has quit [Remote host closed the connection]
jhn has quit [Ping timeout: 250 seconds]
le_gars has joined #rubygems
arohner has joined #rubygems
baburdick has joined #rubygems
le_gars has quit [Remote host closed the connection]
Perceptes has left #rubygems [#rubygems]
arohner has quit [Ping timeout: 272 seconds]
baburdick has quit [Ping timeout: 264 seconds]
bbrowning has joined #rubygems
bf4 has joined #rubygems
bf4 has quit [Ping timeout: 240 seconds]
roidrage has joined #rubygems
josh-k has quit [Remote host closed the connection]
almostwhitehat has joined #rubygems
huoxito has joined #rubygems
redmenace has joined #rubygems
redmenace has quit [Ping timeout: 264 seconds]
arohner has joined #rubygems
baburdick has joined #rubygems
arohner has quit [Ping timeout: 240 seconds]
baburdick has quit [Ping timeout: 272 seconds]
swills` is now known as swills
swills has quit [Changing host]
swills has joined #rubygems
jwinter has joined #rubygems
ZachBeta has joined #rubygems
bf4 has joined #rubygems
<roidrage>
hey guys
<dwradcliffe>
roidrage: hi
<roidrage>
we still see quite a few ssl connectivity issues. i keep wondering if those are rubygems.org or possibly s3?
<roidrage>
last week we almost couldn't deploy to heroku because of ongoing ssl connectivity issues, and now we're still seeing the occasional build fail
<roidrage>
anything we can do to help?
arohner has joined #rubygems
baburdick has joined #rubygems
baburdick has quit [Ping timeout: 246 seconds]
arohner has quit [Ping timeout: 240 seconds]
ZachBeta has quit [Quit: Computer has gone to sleep.]
huoxito has quit [Remote host closed the connection]
<evan>
Node attributes are read-only when you do not specify which precedence level to set. To set an attribute use code like `node.default["key"] = "value"'
tbuehlmann has quit [Remote host closed the connection]
graphex has joined #rubygems
graphex_ has quit [Read error: Connection reset by peer]
huoxito_ has quit [Remote host closed the connection]
<samkottler>
evan: hey
<evan>
hey
<evan>
you said you had an instance using chef-cilent
<evan>
I wasn't sure which that ways
<evan>
I thought that may have something to do with this error
<samkottler>
app01 and balancer02
<evan>
ok
<samkottler>
evan: lemme take a quick look
<evan>
I've run knife solo aganist balancer02 yesterday
graphex_ has joined #rubygems
graphex has quit [Read error: Connection reset by peer]
graphex_ has quit [Read error: Connection reset by peer]
graphex has joined #rubygems
<dwradcliffe>
evan: what's the source of that error?