In early 2023, Google operators deployed a Google Cloud VMware Engine (GCVE) Private Cloud for UniSuper using an internal tool. Due to an inadvertent misconfiguration where a parameter was left blank, the system defaulted the Private Cloud to a fixed one-year term with automatic deletion. After this one-year period, the customer’s GCVE Private Cloud was automatically deleted, leading to an incident that required several days of recovery efforts in May 2024.

The root cause was an internal tool used for capacity management, which was later deprecated in Q4 2023. When provisioning the GCVE Private Cloud, a Google operator left a specific input parameter blank. This blank parameter caused the system to assign an unknown default fixed one-year term, leading to the unintended automatic deletion of the Private Cloud after that period.

The incident impacted one customer, UniSuper, in Australia, specifically one of their GCVE Private Clouds across two zones. This deletion led to an outage for UniSuper. However, other Google Cloud services, other GCVE customers, and UniSuper’s other GCVE Private Clouds or data backups in Google Cloud Storage were not affected.

Google Cloud has taken several steps to prevent recurrence. The internal tool that caused the issue has been deprecated, and its functionality is now fully automated and customer-controlled. Google also reviewed all existing GCVE Private Clouds to ensure no other deployments were at risk and corrected the system behavior related to deletion workflows.