On April 4, 2025, from 00:16 to 01:49 UTC, CircleCI experienced a service disruption affecting its user interface and build capabilities. During this period, customers were unable to access the CircleCI UI or initiate new builds. The incident began when an inadvertently applied Web Application Firewall (WAF) rule started blocking legitimate traffic.

The issue manifested as degraded performance across multiple services, a drop in GitHub webhooks, and widespread connectivity issues between the frontend and backend services, including CORS errors. Initial investigations explored various causes like recent deployments, but the root cause remained unclear for some time.

The root cause was identified as an inadvertently applied WAF rule. A misconfiguration in IAM controls allowed an operator to manually modify WAF settings outside of the standard Terraform deployment process, believing they were performing read-only actions. This change blocked legitimate traffic to api.circleci.com and circleci.com CloudFront distributions.

Incident response was complicated by the assumption that all WAF changes would go through Terraform, leading responders to initially deprioritize WAF configuration as a suspect. The diverse symptoms and the incident’s proximity to another unrelated issue also led to fruitless paths of inquiry. The problem was eventually identified when automated Terraform drift detection surfaced the discrepancy.

To prevent recurrence, CircleCI implemented stricter IAM policies to prevent direct infrastructure modification outside of the infrastructure-as-code pipeline. Enhancements are being made to Terraform drift detection for faster alerts, and technical guardrails are being added for configuration management. Additionally, WAF-specific monitoring and security control policies (SCPs) are being introduced to further reduce the risk of accidental misconfigurations.