On October 4, 2023, Cloudflare’s 1.1.1.1 DNS resolver experienced lookup failures, leading to an increase in SERVFAIL responses. The incident began at 07:00 UTC when DNSSEC signatures in a stale root zone file expired, and was resolved by 11:02 UTC. External reports started at 07:57 UTC, and an internal incident was declared at 08:03 UTC.

The core issue stemmed from Cloudflare’s static_zone WebAssembly app, which is responsible for parsing and serving the DNS root zone. On September 21, a new ZONEMD resource record type was introduced into the root zone. Cloudflare’s parser, designed to handle the root zone in presentation format, failed to correctly parse this new, unknown record type. This prevented the static_zone app from updating its copy of the root zone.

As a result of the parsing failure, the static_zone app continued to use an outdated version of the root zone from September 21. When the DNSSEC signatures in this stale root zone expired on October 4 at 07:00 UTC, some Cloudflare resolver systems could no longer validate DNSSEC signatures and began returning SERVFAIL responses. The rate of SERVFAILs peaked at 15% of total queries, significantly higher than the usual 3% baseline, with impact concentrated in major data centers.

Initial attempts to disable the static_zone app via override rules were unsuccessful because internal forwarding mechanisms did not propagate the necessary tags. The incident was ultimately resolved at 10:30 UTC when the static_zone app was entirely stopped, leading to responses returning to normal by 10:32 UTC. Cloudflare is implementing several follow-up actions, including improved monitoring for stale root zone files, enhancing resilience to new record types, increasing test coverage for parsing failures, and better managing the lifetime of cached root zone data.