{"UUID":"465ab32f-57ed-43b2-8430-c2ec691b0d1d","URL":"https://blog.cloudflare.com/1-1-1-1-lookup-failures-on-october-4th-2023/","ArchiveURL":"","Title":"Cloudflare 1.1.1.1 lookup failures on October 4, 2023","StartTime":"2023-10-04T07:00:00Z","EndTime":"2023-10-04T11:02:00Z","Categories":["automation","cloud"],"Keywords":["dns","1.1.1.1","cloudflare","servfail","dnssec","root zone","zonemd","parser error"],"Company":"Cloudflare","Product":"1.1.1.1 DNS resolver","SourcePublishedAt":"2023-10-04T20:40:34+01:00","SourceFetchedAt":"2026-05-04T19:51:41.522828Z","Summary":"On 4 October 2023, Cloudflare experienced DNS resolution problems starting at 07:00 UTC and ending at 11:00 UTC. Some users of 1.1.1.1 or products like WARP, Zero Trust, or third party DNS resolvers which use 1.1.1.1 may have received SERVFAIL DNS responses to valid queries. We’re very sorry for this outage. This outage was an internal software error and not the result of an attack. In this blog, we’re going to talk about what the failure was, why it occurred, and what we’re doing to make sure this doesn’t happen again.","Description":"On October 4, 2023, Cloudflare's 1.1.1.1 DNS resolver experienced lookup failures, leading to an increase in SERVFAIL responses. The incident began at 07:00 UTC when DNSSEC signatures in a stale root zone file expired, and was resolved by 11:02 UTC. External reports started at 07:57 UTC, and an internal incident was declared at 08:03 UTC.\n\nThe core issue stemmed from Cloudflare's `static_zone` WebAssembly app, which is responsible for parsing and serving the DNS root zone. On September 21, a new ZONEMD resource record type was introduced into the root zone. Cloudflare's parser, designed to handle the root zone in presentation format, failed to correctly parse this new, unknown record type. This prevented the `static_zone` app from updating its copy of the root zone.\n\nAs a result of the parsing failure, the `static_zone` app continued to use an outdated version of the root zone from September 21. When the DNSSEC signatures in this stale root zone expired on October 4 at 07:00 UTC, some Cloudflare resolver systems could no longer validate DNSSEC signatures and began returning SERVFAIL responses. The rate of SERVFAILs peaked at 15% of total queries, significantly higher than the usual 3% baseline, with impact concentrated in major data centers.\n\nInitial attempts to disable the `static_zone` app via override rules were unsuccessful because internal forwarding mechanisms did not propagate the necessary tags. The incident was ultimately resolved at 10:30 UTC when the `static_zone` app was entirely stopped, leading to responses returning to normal by 10:32 UTC. Cloudflare is implementing several follow-up actions, including improved monitoring for stale root zone files, enhancing resilience to new record types, increasing test coverage for parsing failures, and better managing the lifetime of cached root zone data."}