Google and Oracle Clouds Hit By Heatwave-Related Outages

July 2022 saw record temperatures in England. It wasn't just the humans that were suffering. Several public cloud providers were too.

Google Cloud services from an unnamed London data centre were disrupted for between 18 to 35 hours  as a result of multiple concurrent failures of the building's cooling system and an error in rerouting traffic.

Oracle Cloud services also suffered cooling failures at one of its London data centres.

Such outages shouldn't happen

Resilient cooling systems are designed to work even if one unit fails. Most of the time, they keep things cool until the original problem can be fixed.

However, on occasion, multiple units fail in close succession. Sometimes, after a unit fails, the backup system refuses to kick in. At other times, the backup unit takes over but aggregate demand overwhelms it.

Cooling failures aren't that common. Power supply issues are more likely. So what can you do to avoid them?

Stop this downtime happening to you

Use multiple data centres.

Have your workloads/data/databases replicated between data centres, with automatic 'failover' to a backup location.

Your public-facing IP addresses may also need to failover to the new location.

Don't assume that a single data centre is sufficient to keep you online - regardless of how 'resilient' it is supposed to be. 

If you are using big public cloud providers like Google or Oracle, you may need to re-configure your applications to use multiple availability zones within your chosen region.

Our own virtual private cloud hosting service can take care of that hosting your resrouces, so if there's a problem with one data centre, your hosting can restart in another automatically.

This protects you against individual data centre failures.

Server downtime due to climate change

The biggest danger to your server-based workloads isn't that high temperatures will cause air conditioning units to fail.

It's that unusually high or low temperatures cause a spike in electricity demand that leads to temporary local blackouts.

The former almost happened during the July 2022 heatwave. The problem was averted due to National Grid buying electricity from Belgium at 50 times the normal price.

The electricity supply crunch is likely to increase as the UK shifts towards net zero. More electricity will be needed to power vehicles, as new petrol & diesel cars are banned from sale from 2030. Electricity will also be needed for domestic heating as new homes built after 2025 won't be allowed to install gas boilers.

It remains to be seen whether electricity generation and distribution will be able to keep up with this surging demand. So far, the signs aren't encouraging. Recently, more nuclear generators have been decommissioned than commissioned. And the Greater London Authority is already saying that new housing developments requiring over 1MW of power "will have to wait several years to receive new electricity connections" because data centres have "created capacity constraints on both the distribution and transmission networks... absorbing remaining electricity capacity in SSEN's West London region for the remainder of the decade."

Given the supply problems and distribution problems, blackouts do look more likely than they have been in the past. It's important to protect your business-critical workloads from potential power issues by using uninterruptible power supplies and backup generators. The quickest and cheapest way to achieve this is to shift your server workloads to resilient cloud hosting or make use of colocation.