drbrain changed the topic of #rubygems to: RubyGems 2.0.2: http://bit.ly/rubygems-2-0-2 – Latest status: http://twitter.com/rubygems_status and http://status.rubygems.org
<dwradcliffe> redis log is 6.6G
<evan> thats on /var/lib/redis_data
<evan> or it's supposed to be
<dwradcliffe> /var/log/redis/redis-server.log
<evan> oh that log
<evan> fuck
<evan> truncate it.
<evan> i'm on there now too
<dwradcliffe> permission denied
<samkottler> let's move it to /mnt
<evan> i'l do it.
<samkottler> actually I guess it can go on /var/lib/redis_data
<samkottler> k thanks evan
<evan> yeah
baburdick has joined #rubygems
<samkottler> once that's done we can start the DB upgrade
<evan> ok, truncated
<evan> using my favorite truncation method
<samkottler> /dev/null?
<evan> : > log
<samkottler> lol
<samkottler> okay I'm going to put the site into maintenence mode if everyone's cool with that
<evan> we should setup some aggressive policys for all logs written to /var/log
<evan> yep, go ahead
<evan> i'll tweet about it.
<evan> ok, go ahead.
baburdick has quit [Ping timeout: 252 seconds]
<samkottler> evan: where is the maintenence mode enabled? I can't even find that string in /etc/nginx
<evan> I should take the time to also rewrite the redis AOF file
<evan> one sec.
<evan> we might have moved it
<evan> we were previously doing it via changing SetEnv in the apache conf
<evan> i'm not sure we've done maint mode since moving
<evan> give me a sec
<samkottler> ah so it might not be there anymore
<dwradcliffe> I don't remember seeing it
baburdick has joined #rubygems
<evan> yeah, I don't think we have
<evan> do you guys have a recommended way to do it?
<evan> setting RailsEnv to maintanence and restarting unicorn seems the easiest
<samkottler> agreed
<evan> we can use unicorns -E option
baburdick has quit [Ping timeout: 252 seconds]
<samkottler> wanna do that evan?
<samkottler> or should I?
<evan> i'll do it the quick way right now
<evan> one sec.
<samkottler> I'll update the sysvinit script to load the environment from /etc/default/rubygems at some point so it doesn't require too much hacking to do this
<evan> ok, into maint mode
<evan> something is up though.
<evan> it's not responding
<evan> I think -E maintenance isn't working
<samkottler> this is unicorn being ridiculous
baburdick has joined #rubygems
<samkottler> no, it's loading - check out top
<evan> the workers are starting then exitting
<evan> blarg
<evan> uninitialized constant NewRelic
<evan> one sec.
<evan> maint mode got broken.
<samkottler> yeah, I get a 500 now
<evan> ok, fixed it
<evan> we're in maint now
<evan> go about your business
<evan> i'm going to have redis rewrite the AOF file now too.
yashshah has quit [Ping timeout: 252 seconds]
<samkottler> upgrading the packages on dbmaster01
<evan> once AOF is rewritten
<evan> i'm going to change the redis config to turn down the log level
<evan> that log file it's writing is useless
<evan> i've never used it.
<dwradcliffe> evan: sounds good
<dwradcliffe> evan: it was spewing out of memory errors a few minutes ago
baburdick has quit [Ping timeout: 252 seconds]
<evan> "oh hey, the LB incremented hits on the rails gem, yippy kia yae"
* samkottler wonders why the hell pulseaudio is installed on this machine
<evan> dwradcliffe: redis was
<evan> excellent.
<evan> O_o
<evan> we need to fix the stats
<evan> i should be able to dump some RAM from it
<samkottler> did someone install a desktop packagegroup on this box?
<samkottler> it has some gnome shit
<samkottler> and libreoffice
<samkottler> ugh....
<dwradcliffe> wha?
<evan> ha
<evan> we'll schedule another time to fully redo the app server out of chef
<samkottler> "oh look guys, gnome account system is installed on our db master"
baburdick has joined #rubygems
<evan> aah
<evan> thats why the redis log was so big
rhinux1 has joined #rubygems
<evan> it complaining about memory probably
huoxito has quit [Quit: Leaving]
<samkottler> okay DB upgrade is complete
yashshah has joined #rubygems
huoxito has joined #rubygems
rhinux has quit [Ping timeout: 264 seconds]
<samkottler> want to take it out of maintenence evan?
baburdick has quit [Ping timeout: 248 seconds]
<evan> i'm going to do a little more redis cleanup
<evan> almost done
<samkottler> okay no problem
<evan> and i'm going to have redis write it's log to /var/lib/redis_data
<samkottler> evan: awesome - thanks :)
<evan> np
<evan> i'll change it on the server
<evan> and port the changes into chef afterward
<samkottler> works for me
<dwradcliffe> want to deploy datadog tonight so we can start monitoring?
<evan> that would be great
<samkottler> evan: we should have the maintenence page on the balancer instead of the app. I'll work on that this week
<samkottler> dwradcliffe: yep definitely
<evan> i played with datadog about 6 months ago
<evan> looked great
<evan> didn't have the time to dig into it much.
<evan> samkottler: yeah, agreed.
yashshah has quit [Ping timeout: 240 seconds]
baburdick has joined #rubygems
<evan> my prune_redis script is running
<evan> i shouldn't have waited so long to run it :)
<evan> it's got a lot to prune.
xymox has quit [Ping timeout: 256 seconds]
<samkottler> evan: should we just run that as a cronjob each month or something?
<evan> each day
<evan> i'm going to take us out of maint
<evan> this can run concurrently
<dwradcliffe> who has access to the status page?
<evan> it's mostly automated
<evan> I do, a few others
<evan> i can give you cacess
<dwradcliffe> I want to remove the link at the top from that incident
<dwradcliffe> not really current anymore
<samkottler> can I have access, too
xymox has joined #rubygems
<dwradcliffe> heroku right?
baburdick has quit [Ping timeout: 248 seconds]
<evan> yeah
<evan> sure
<evan> unicorn restarting into production now.
<samkottler> site is back
<samkottler> evan++ dwradcliffe++
<evan> samkottler: whats your email to add to status?
<samkottler> evan: shk@linux.com
<evan> k
baburdick has joined #rubygems
<evan> ok, added you both to rubygems-status
<dwradcliffe> evan: thanks
<samkottler> evan: thank you
<evan> i think stat-update is probably my favorite little program I wrote for rubygems
<dwradcliffe> I'll fix the twitter integration on the page too
<evan> uses 1M of RAM resident
<evan> barely gets above 5% CPU ever
baburdick has quit [Ping timeout: 256 seconds]
<evan> replaced like 10 passenger processes running at 80%
<samkottler> :)
<samkottler> one day when I'm less tired this week I'll remove all the desktop crap from the db machine
<evan> heh
<samkottler> because...yeah :P
<evan> prune is about half way done
baburdick has joined #rubygems
baburdick has quit [Ping timeout: 248 seconds]
yerhot has joined #rubygems
<samkottler> k
rhinux has joined #rubygems
baburdick has joined #rubygems
rhinux1 has quit [Ping timeout: 264 seconds]
<samkottler> evan: I'd also like to move redis off the app machine at some point so it's completely ephemeral
<samkottler> the DB box seems like the obvious place for it to go
<dwradcliffe> samkottler: sounds logical
<dwradcliffe> status page is updated
baburdick has quit [Ping timeout: 240 seconds]
<dwradcliffe> we can move it when we rebuild the db server
baburdick has joined #rubygems
<samkottler> dwradcliffe: oh are we planning on doing that?
<dwradcliffe> oh no, that was the app server. nevermind
<dwradcliffe> so, right before we rebuild the app server
rhinux has quit [Ping timeout: 264 seconds]
baburdick has quit [Ping timeout: 252 seconds]
<samkottler> does anyone know some peeps over at pagerduty?
rhinux has joined #rubygems
ZachBeta has quit [Quit: Computer has gone to sleep.]
<dwradcliffe> not I
baburdick has joined #rubygems
<kseifried> cool concept
baburdick has quit [Ping timeout: 256 seconds]
<samkottler> kseifried: pagerduty, you mean?
baburdick has joined #rubygems
<samkottler> dwradcliffe: you deploying the datadog stuff?
<dwradcliffe> I can. I'll need to get the secrets, etc
rhinux has quit [Ping timeout: 264 seconds]
rhinux1 has joined #rubygems
baburdick has quit [Ping timeout: 264 seconds]
<dwradcliffe> samkottler: want to merge it?
<samkottler> dwradcliffe: yep, I'll do that now. I also just updated the typo fix PR
<samkottler> merged
baburdick has joined #rubygems
baburdick has quit [Ping timeout: 248 seconds]
rhinux has joined #rubygems
rhinux1 has quit [Read error: Connection reset by peer]
baburdick has joined #rubygems
* samkottler is heading offline
<samkottler> if you need help rolling that out dwradcliffe I'll be on my cell
<samkottler> OTR message incoming with the #
<dwradcliffe> samkottler: thx
jfelchner has joined #rubygems
jfelchner has left #rubygems [#rubygems]
baburdick has quit [Ping timeout: 252 seconds]
<samkottler> later y'all. nice work tonight
baburdick has joined #rubygems
mando has joined #rubygems
baburdick has quit [Ping timeout: 252 seconds]
<evan> excellent
<evan> got redis down to 1.7G
<evan> from 6.4G
<evan> my work here is done
<evan> :D
baburdick has joined #rubygems
mando has quit [Ping timeout: 248 seconds]
<dwradcliffe> evan: nice!
<dwradcliffe> datadog is runnint on the app server
baburdick has quit [Ping timeout: 256 seconds]
<evan> excellent
<evan> can you add me to the account?
<dwradcliffe> email?
baburdick has joined #rubygems
baburdick has quit [Ping timeout: 252 seconds]
baburdick has joined #rubygems
<dwradcliffe> evan: what email should I send the invite?
baburdick has quit [Ping timeout: 256 seconds]
newUser1234 has quit [Remote host closed the connection]
baburdick has joined #rubygems
Spaceghost|work has quit [Remote host closed the connection]
pipework has joined #rubygems
baburdick has quit [Ping timeout: 256 seconds]
baburdick has joined #rubygems
baburdick has quit [Ping timeout: 256 seconds]
baburdick has joined #rubygems
nezumi has joined #rubygems
baburdick has quit [Ping timeout: 256 seconds]
baburdick has joined #rubygems
baburdick has quit [Ping timeout: 256 seconds]
baburdick has joined #rubygems
ZachBeta has joined #rubygems
baburdick has quit [Ping timeout: 256 seconds]
ZachBeta has quit [Quit: Textual IRC Client: http://www.textualapp.com/]
baburdick has joined #rubygems
baburdick has quit [Ping timeout: 248 seconds]
yerhot has quit [Remote host closed the connection]
baburdick has joined #rubygems
rhinux has quit [Ping timeout: 264 seconds]
baburdick has quit [Ping timeout: 256 seconds]
yashshah has joined #rubygems
baburdick has joined #rubygems
baburdick has quit [Ping timeout: 252 seconds]
baburdick has joined #rubygems
baburdick has quit [Ping timeout: 248 seconds]
baburdick has joined #rubygems
nateberkopec has quit [Read error: Connection reset by peer]
baburdick has quit [Ping timeout: 256 seconds]
baburdick has joined #rubygems
baburdick has quit [Ping timeout: 240 seconds]
baburdick has joined #rubygems
baburdick has quit [Ping timeout: 252 seconds]
baburdick has joined #rubygems
nezumi has quit [Ping timeout: 252 seconds]
baburdick has quit [Ping timeout: 256 seconds]
baburdick has joined #rubygems
baburdick has quit [Ping timeout: 252 seconds]
baburdick has joined #rubygems
wolfeidau has joined #rubygems
newUser1234 has joined #rubygems
baburdick has quit [Ping timeout: 248 seconds]
baburdick has joined #rubygems
wolfeidau has quit [Remote host closed the connection]
baburdick has quit [Ping timeout: 256 seconds]
baburdick has joined #rubygems
baburdick has quit [Ping timeout: 252 seconds]
baburdick has joined #rubygems
baburdick has quit [Ping timeout: 248 seconds]
huoxito has quit [Ping timeout: 248 seconds]
wolfeidau has joined #rubygems
nezumi has joined #rubygems
baburdick has joined #rubygems
huoxito has joined #rubygems
baburdick has quit [Ping timeout: 256 seconds]
baburdick has joined #rubygems
baburdick has quit [Ping timeout: 256 seconds]
kgrz has joined #rubygems
baburdick has joined #rubygems
baburdick has quit [Ping timeout: 248 seconds]
tenderlove has quit [Remote host closed the connection]
baburdick has joined #rubygems
tenderlove has joined #rubygems
baburdick has quit [Ping timeout: 256 seconds]
adkron has joined #rubygems
kgrz has quit [Quit: Computer has gone to sleep.]
adkron has quit [Ping timeout: 245 seconds]
kgrz has joined #rubygems
wolfeidau has quit [Ping timeout: 276 seconds]
tbuehlmann has joined #rubygems
newUser1234 has quit [Remote host closed the connection]
wolfeidau has joined #rubygems
wolfeida_ has joined #rubygems
wolfeidau has quit [Ping timeout: 248 seconds]
kgrz has quit [Quit: Computer has gone to sleep.]
workmad3 has joined #rubygems
wolfeida_ has quit [Ping timeout: 256 seconds]
charliesome has joined #rubygems
wolfeidau has joined #rubygems
lsegal has quit [Ping timeout: 245 seconds]
wolfeidau has quit [Remote host closed the connection]
huoxito has quit [Ping timeout: 248 seconds]
Elhu has joined #rubygems
Elhu has quit [Client Quit]
huoxito has joined #rubygems
stoffus has joined #rubygems
workmad3 has quit [Ping timeout: 245 seconds]
baburdick has joined #rubygems
baburdick has quit [Ping timeout: 252 seconds]
<stoffus> I'm currently experiencing lots of timeouts when using https, is this a known issue?
kgrz has joined #rubygems
baburdick has joined #rubygems
kgrz has quit [Read error: Connection reset by peer]
baburdick has quit [Ping timeout: 256 seconds]
workmad3 has joined #rubygems
nezumi has quit [Ping timeout: 260 seconds]
workmad3 has quit [Ping timeout: 264 seconds]
newUser1234 has joined #rubygems
workmad3 has joined #rubygems
newUser1234 has quit [Remote host closed the connection]
tbuehlmann has quit [Remote host closed the connection]
charliesome has quit [Quit: Textual IRC Client: www.textualapp.com]
huoxito has quit [Quit: Leaving]
ddd has quit [Quit: restarting tmux]
ddd has joined #rubygems
yashshah has quit [Ping timeout: 245 seconds]
ddd has quit [Client Quit]
ddd has joined #rubygems
ddd has quit [Client Quit]
tekin has joined #rubygems
ddd has joined #rubygems
ddd has quit [Client Quit]
ddd has joined #rubygems
ddd has quit [Client Quit]
ddd has joined #rubygems
Elhu has joined #rubygems
ddd has quit [Quit: ok restarting with no tmux. just means I will disconnect when done]
ddd has joined #rubygems
tbuehlmann has joined #rubygems
workmad3 has quit [Ping timeout: 248 seconds]
Elhu has quit [Quit: Computer has gone to sleep.]
tekin has quit [Quit: ["Textual IRC Client: www.textualapp.com"]]
workmad3 has joined #rubygems
ddd has quit [Quit: Lost terminal]
Elhu has joined #rubygems
ZachBeta has joined #rubygems
workmad3 has quit [Ping timeout: 245 seconds]
ddd has joined #rubygems
yerhot has joined #rubygems
yerhot has quit [Remote host closed the connection]
nezumi has joined #rubygems
ZachBeta has quit [Quit: Computer has gone to sleep.]
dwradcliffe_ has joined #rubygems
dwradcliffe has quit [Ping timeout: 246 seconds]
kgrz has joined #rubygems
Elhu has quit [Quit: Computer has gone to sleep.]
tenderlove has quit [Remote host closed the connection]
tenderlove has joined #rubygems
tenderlove has quit [Remote host closed the connection]
yashshah has joined #rubygems
markalanevans has joined #rubygems
nezumi has quit [Ping timeout: 245 seconds]
fidothe_ has quit [Ping timeout: 246 seconds]
fidothe_ has joined #rubygems
kgrz has quit [Quit: Computer has gone to sleep.]
workmad3 has joined #rubygems
nezumi has joined #rubygems
workmad3 has quit [Ping timeout: 252 seconds]
stoffus has quit []
nezumi has quit [Ping timeout: 240 seconds]
Elhu has joined #rubygems
workmad3 has joined #rubygems
tbuehlmann has quit [Remote host closed the connection]
nomadic has quit [Ping timeout: 245 seconds]
nateberkopec has joined #rubygems
nomadic has joined #rubygems
Elhu has quit [Quit: Computer has gone to sleep.]
baphled has joined #rubygems
workmad3 has quit [Ping timeout: 246 seconds]
lsegal has joined #rubygems
Elhu has joined #rubygems
Elhu has quit [Quit: Computer has gone to sleep.]
kyzh has joined #rubygems
kyzh is now known as Guest63016
workmad3 has joined #rubygems
havenwood has joined #rubygems
Elhu has joined #rubygems
Elhu has quit [Quit: Computer has gone to sleep.]
workmad3 has quit [Ping timeout: 252 seconds]
baphled has quit [Ping timeout: 256 seconds]
baphled has joined #rubygems
adkron has joined #rubygems
Elhu has joined #rubygems
nateberkopec has quit [Quit: Leaving...]
Elhu has quit [Quit: Computer has gone to sleep.]
adkron has quit [Ping timeout: 264 seconds]
ckrailo has joined #rubygems
Guest63016 has quit [Read error: Operation timed out]
rhinux has joined #rubygems
ckrailo has quit [Ping timeout: 276 seconds]
gramos has joined #rubygems
adkron has joined #rubygems
gramos has quit [Ping timeout: 260 seconds]
nezumi has joined #rubygems
nateberkopec has joined #rubygems
mpapis has quit [Excess Flood]
mpapis has joined #rubygems
baphled has quit [Ping timeout: 252 seconds]
ZachBeta has joined #rubygems
yerhot has joined #rubygems
ZachBeta has quit [Quit: Computer has gone to sleep.]
ZachBeta has joined #rubygems
ZachBeta has quit [Client Quit]
yerhot has quit [Remote host closed the connection]
ZachBeta has joined #rubygems
calmyournerves is now known as Guest
nateberkopec has quit [Quit: Linkinus - http://linkinus.com]
havenwood has quit [Remote host closed the connection]
ZachBeta has quit [Quit: Computer has gone to sleep.]
ZachBeta has joined #rubygems
huoxito has joined #rubygems
ZachBeta has quit [Client Quit]
lsegal has quit [Quit: Quit: Quit: Quit: Stack Overflow.]
ZachBeta has joined #rubygems
huoxito has quit [Ping timeout: 264 seconds]
ZachBeta has quit [Quit: Computer has gone to sleep.]