On June 2, 2019, Google Cloud experienced a significant network congestion issue primarily affecting the eastern USA, leading to elevated packet loss and service unavailability across multiple regions. The incident began at 11:45 US/Pacific and impacted various Google Cloud and G Suite services, as well as YouTube. Initial customer impact was observed between 11:47-11:49 US/Pacific.

The root cause was a combination of two normally-benign misconfigurations and a specific software bug in Google’s datacenter automation software. This allowed the system to deschedule multiple independent network control plane clusters simultaneously across different physical locations during a maintenance event, despite existing resilience mechanisms.

Customers experienced increased latency, intermittent errors, and connectivity loss to instances in regions like us-central1, us-east1, us-east4, us-west2, northamerica-northeast1, and southamerica-east1. Services such as Compute Engine, App Engine, Cloud Endpoints, Cloud Interconnect, Cloud VPN, Cloud Console, Stackdriver, Cloud Pub/Sub, BigQuery, Cloud Spanner, and Cloud Storage were affected, with some regions experiencing near-total unavailability for certain services.

Google engineers identified the root cause by 13:01 US/Pacific and halted the automation. Recovery involved re-enabling the network control plane and rebuilding lost configuration data, which was time-consuming due to the scale. Network capacity began to return at 15:19 US/Pacific, and full service was resumed by 16:10 US/Pacific.

To prevent recurrence, Google immediately halted the problematic datacenter automation software. Future actions include hardening cluster management software, reconfiguring the network control plane to correctly handle maintenance and persist configuration, and extending the network’s ‘fail static’ mode duration. Emergency response tooling and disaster recovery testing will also be enhanced.