{"UUID":"24be1eec-6da2-4825-853c-4b2561bdd422","URL":"https://blog.cloudflare.com/18-november-2025-outage/","ArchiveURL":"","Title":"Cloudflare systemwide outage, November 18, 2025","StartTime":"2025-11-18T11:20:00Z","EndTime":"2025-11-18T17:06:00Z","Categories":["automation","config-change"],"Keywords":["cloudflare","outage","bot management","clickhouse","core proxy","5xx errors","cdn","access"],"Company":"Cloudflare","Product":"Bot Management","SourcePublishedAt":"0001-01-01T00:00:00Z","SourceFetchedAt":"2026-05-04T19:51:05.703118Z","Summary":"On 18 Nov. 2025, a change in permissions in a database in Cloudflare's bot-detection systems caused a file to be output that exceeded the limits of the software that runs that system. That file was propagated througout Cloudflare's network, causing a systemwide outage.","Description":"On November 18, 2025, Cloudflare's network began experiencing significant failures at 11:20 UTC, leading to widespread HTTP 5xx errors for customers. The main impact was resolved by 14:30 UTC, with all services fully restored by 17:06 UTC.\n\nThe incident was triggered by a change in database permissions management for a ClickHouse cluster at 11:05 UTC. This change caused a query, used by Cloudflare's Bot Management system, to return duplicate rows. Consequently, the Bot Management system's feature configuration file doubled in size.\n\nThis oversized feature file was then propagated across Cloudflare's network. The software running on the core proxy, responsible for routing traffic, had a preallocated memory limit for this feature file. When the doubled-size file exceeded this limit, the software panicked, resulting in 5xx errors for traffic dependent on the Bot Management module.\n\nThe outage caused widespread HTTP 5xx errors for customers utilizing Cloudflare's core CDN and security services. Other services like Workers KV, Cloudflare Access, and Turnstile were also significantly impacted. The Cloudflare Dashboard experienced reduced availability, and some customers using bot scores in their rules observed false positives.\n\nCloudflare's teams identified the issue, stopped the generation and propagation of the faulty feature file, and deployed a known good version. They also implemented bypasses for Workers KV and Cloudflare Access to reduce impact. Follow-up steps include hardening configuration file ingestion, enabling global kill switches, and reviewing failure modes across core proxy modules."}