purr changed the topic of #elliottcable to: this is CLEARLY a cult.YEP
Sgeo has joined #elliottcable
yorick has quit [Remote host closed the connection]
vil has quit [Ping timeout: 272 seconds]
<whitequark>
haha what the hell, why does this anime from 2009 mention submarine patents?!
<whitequark>
for real, that's one of the most obscure topics I know, and it was even before the whole samsung-apple thing
<alexgordon>
hahaha
<alexgordon>
brooker segment on ukraine is fucking hilarious
<alexgordon>
hahahahahah
<alexgordon>
cunk <3
alexgordon has quit [Quit: My iMac has gone to sleep. ZZZzzz…]
vil has joined #elliottcable
devyn has quit [Quit: leaving]
devyn has joined #elliottcable
devyn has quit [Changing host]
devyn has joined #elliottcable
<devyn>
joelteon: you reaaallly have to be careful hiring people to work on MySQL... being such a popular DB, a lot of shitty webdevs *think* they know how to use it properly (they don't)
<devyn>
and SQL in general
<joelteon>
it wasn't a sql issue, lol
<joelteon>
so i spent a week running this expensive conversion on our backup slaves
<devyn>
oh good, because that would have been totally unsurprising
<joelteon>
tl;dr someone fucked up utf-8 a long time before i joined the company
<joelteon>
so the data was corrupted
<joelteon>
so i unfucked it
<joelteon>
took a long time
<joelteon>
anyway, the backup slave has the right data
<joelteon>
so they rebuild the read slaves off the backup slaves so we have good data and redundancy
<devyn>
haha how'd you even figure out how it was fucked
<joelteon>
then tuesday night we did the failover
<joelteon>
oh, it was easy
<joelteon>
saw a bunch of entries like "Bill Me Laterî"
<joelteon>
in the database, not the webapp
<joelteon>
it's a great story
<joelteon>
but anyway, tuesday night we do the failover
<joelteon>
so our hostnames look like outaxdb01-m, outaxdb02-m, etc.
<joelteon>
so michael points the -m hostnames at the old backup slaves, which are now configured to be the new masters
<joelteon>
then we deploy the webapp, bring everything back up
<joelteon>
and it looks good
<joelteon>
so what michael managed to do this morning was point the -m hostnames back at the *old database*
<joelteon>
the one that has bad data on it
<joelteon>
which shouldn't be a big deal because once the DB is out of the rotation, it's read-only
<joelteon>
according to SOP
<joelteon>
of course, michael forgot to reload the mysql config after the failover
<joelteon>
so we had four or five companies and all their entries written to a database that was destroyed and rebuilt shortly afterwards
<joelteon>
it was
<joelteon>
absolutely amazing
<joelteon>
also devyn, i found out how the data was fucked when updating some unit tests when I was trying to upgrade to ruby 1.9
<joelteon>
i was like "why is this failing?"
<joelteon>
and ben says "oh jesus"
<joelteon>
and we go into a conference room and he draws on a whiteboard for 15 minutes
<joelteon>
that was great
<whitequark>
hahaha, SZS getting all angsty about UTC leap seconds
<gq>
jesus christ
<gq>
words, words all over my screen
<whitequark>
I really like it. "it's amazing" like this time, not "it's so horrible it's kinda good" like.
<devyn>
joelteon: haha
<devyn>
ffs no purr?
<devyn>
aww, ell.io isn't accepting my ssh key
<gq>
wow there really is no purr
<gq>
everyone enjoy saying lol without being echoed
<devyn>
nah, it's no fun without purr
eligrey has quit [Quit: Leaving]
<joelteon>
nope
sharkbot has quit [Remote host closed the connection]
sharkbot has joined #elliottcable
yorick has joined #elliottcable
alexgordon has joined #elliottcable
Sgeo has quit [Read error: Connection reset by peer]
<glowcoil>
devyn: yeah ec's server stuff got rebooted and he hasn't had time to restart all of it
eligrey has joined #elliottcable
<joelteon>
just automate it NERD
eligrey has quit [Ping timeout: 272 seconds]
<ELLIOTTCABLE>
That's why he's not back up. Got side-tracked into trying to systemd-ify minecraft, starbound, and purr, so I can restart with impunity.
<joelteon>
oh, so he's down because you're busy making it so he won't go down again
<joelteon>
you are a true programmer
<gq>
lol
eligrey has joined #elliottcable
<whitequark>
fucking systemd
<whitequark>
put it back where you took it.
<yorickpeterse>
systemd aint that bad
<yorickpeterse>
better than sysvinit
<whitequark>
marginally. does it permit you to touch your network interfaces yet?
<whitequark>
systemd has some nice ideas, but the implementation is so horribly butchered I can't see how anyone sane can use this crap
<whitequark>
upstart is nice, but only by comparison to systemd
<whitequark>
its event system doesn't really map very well to the problem domain
<whitequark>
the best init system I've seen is initng, but it died circa 2006. such a pity.
<joelteon>
poor thing
<joelteon>
upstart is the least objectionable one I've tried
<whitequark>
yea I really hope debian chooses upstart
<whitequark>
they'll add some polish and it would be a nice compromise for the time being.
<joelteon>
i got a lot of shit in #debian for asking when they were going to support it
<joelteon>
those people are pretty resistant to change
<joelteon>
they wondered why we should change an init system that's proven to work
<joelteon>
not sure what they meant by "proven to work"
<joelteon>
init scripts are fucking gross
<whitequark>
perl is everywhere in debian. you don't notice it, because it works