Google’s London cloud outage spread to the zones meant to absorb it

On 22 October, Google Cloud published a preliminary incident report on the outage that hit its europe-west2 region in London two days earlier. At 06:53 London time on 20 October, a power grid disruption hit the data center facility that hosts zone europe-west2-b. The load moved to UPS battery banks during the start-up sequence of the site’s diesel generators, but the programmable logic controllers that automate the generator start, “both primary and backup PLCs”, failed to initiate the transfer. From 07:00, as the batteries ran out, the servers, network switching and storage in the zone lost power.

Google puts the disruption at six hours and 10 minutes, ending at 13:10. Most services had recovered by 12:29, new virtual machine creation resumed at 12:46 and by 13:10 more than 99% of Compute Engine virtual machines were healthy, according to the report. It lists 40 affected products, from Compute Engine and Cloud SQL to BigQuery, Kubernetes Engine and Google’s SecOps security platform.

Google sells europe-west2 as three zones, a, b and c, and its documentation sets out what that is for: “Putting resources in different zones in a region reduces the risk of an infrastructure outage affecting all resources simultaneously.” Zones are the unit of redundancy in that design: a database with a standby in a second zone, a cluster spread across all three.

According to the preliminary report, zone b failed on 20 October for a physical reason that a primary and a backup controller were supposed to cover. Zones a and c then took on its work at once, and the failover traffic saturated their compute capacity.

Why zones a and c slowed down

While the outage was running, Google’s status page offered one workaround, posted at 10:24 London time: “If feasible, the impacted customer may try to migrate their workload impacted to another available location (zone or region).” The first update, at 09:35, had said there was no workaround. Two days later, the preliminary report shows that zones a and c were affected too.

Google names two causes for the impact outside zone b: “instantaneous demand surges on unimpacted compute capacity” and “synchronous control-plane validation dependencies across zonal boundaries”. In the first, redirected queries, database failovers and compute workloads saturated capacity in the surviving zones, which produced admission queues, delayed database failovers and intermittent 500 and 503 errors across the region. In the second, regional control planes kept calling metadata and catalog components in the offline zone and waited on answers that did not come.

The report says BigQuery, Looker, Compute Engine, Kubernetes Engine, regional persistent disk and Cloud SQL “experienced unexpected degradation and failover issues”. By Google’s own account, some customers running workloads in zones a or c, or on regional architectures spanning the region, were affected by a power outage that was “isolated strictly to datacenter facilities in europe-west2-b”.

Cloud SQL, AlloyDB, Memorystore for Redis and Cloud Bigtable failed over into the unaffected zones, Google says, although failover completions were delayed while capacity was short.

What the service level agreement counts

Google’s Compute Engine service level agreement promises at least 99.99% monthly uptime for “Instances in Multiple Zones” on its Premium network tier, and 99.9% for a single instance of most machine families. For a multi-zone deployment, it defines downtime as loss of external connectivity or persistent disk access for “all applicable running instances”. The preliminary report describes errors, queues, slow failovers and degraded regional persistent disk outside zone b. It does not say whether any multi-zone customer lost connectivity or disk access on every instance at once, which is the bar that definition sets.

A virtual machine in zone b that was down for the full six hours and 10 minutes ran at about 99.2% uptime for October. Under the agreement’s bands for most single instances, anything from 95% to below 99.9% qualifies for a credit of 10% of that instance’s monthly bill, applied to future bills, subject to the agreement’s exclusions and to a claim made within 60 days with log files showing the downtime. The agreement calls its credits the customer’s “sole and exclusive remedy” for a missed target.

Why a second region means leaving London

Google’s documentation pairs its zone guidance with a wider one: “Google recommends deploying applications across multiple zones and multiple regions.” For a workload hosted in London, that second region is in another country. The documentation lists europe-west2 as Google’s only Compute Engine region in London, and its nearest neighbors include St. Ghislain in Belgium, Eemshaven in the Netherlands and Paris.

Google says a final incident report with preventative actions will follow. Until it does, the preliminary report describes a London region whose three zones were less independent than its design implies: zones a and c carried zone b’s failover traffic and regional control planes waited on calls to the offline zone, while the multi-zone agreement counts downtime only when every applicable running instance loses external connectivity or disk access.