00:05 <kentonv> yeah they apparently don't consider this to be an outage, you're just expected to know that this can happen, even though they don't document it anywhere AFAICT

00:06 <kentonv> I mean I guess it's relatively obvious in retrospect?

00:08 <kentonv> but mostly they seem to say "shame on you for not making your app multi-homed, even though that's an enormous engineering effort"

00:08 <TimMc> Ugh.

00:08 <TimMc> So what happened here, you (or an automated process) tried to roll a cluster using deallocate/allocate and the allocate failed?

00:10 <TimMc> (also happy to wait for a postmorterm or whatever)

00:10 <kentonv> it's cool, I'm just waiting for disks to move...

00:10 <kentonv> the blackrock cluster master automatically recreates instances as needed. Presumably an instance died for one reason or another, and then the master couldn't recreate it

00:11 <TimMc> So, attrition.

00:12 <TimMc> things die and can't be replaced

00:12 <kentonv> the master aborts if gcloud gives it an unexpected error. It restarts immediately and tries to rebuild the cluster. Unfortunately, it always has to recreate all worker machines in this case, because bugs in nbd mean they tend to get into bad states easily.

00:13 <kentonv> so it shut down the worker machines and then couldn't start them back up

00:13 <kentonv> meanwhile the gateway machine (which receives all incoming traffic) was also destroyed at some point. Maybe it was the first machine to be destroyed, I dunno.

00:14 <kentonv> so it was stuck in a loop trying to recreate all these

00:15 <TimMc> and you've got a whole datacenter's worth of other customers circling your instances like vultures

00:15 <kentonv> probably