On January 13th, 2022, Firefox became unusable for nearly two hours for users globally. This incident began around 07:28 UTC when Google Cloud Platform (GCP) deployed an unannounced change, making HTTP/3 the default for load balancers set to “Automatic”. Shortly after, Firefox users experienced hangs and crashes, rendering the browser unresponsive and unable to load web content.

The problem stemmed from a specific code path in Firefox’s network stack, Necko, related to HTTP/3 upload requests. When connections were made to internal services hosted on GCP, Firefox began using HTTP/3. This interaction triggered a bug that caused the browser to hang.

The core issue was a mismatch in case sensitivity within Necko’s HTTP/3 implementation. Higher-level Necko APIs checked for the Content-Length header in a case-insensitive manner, but the lower-level HTTP/3 code was case-sensitive. The viaduct library, used by Rust components like Telemetry, lower-cased headers before passing them to Necko.

When Telemetry, the only Rust-based component adding a Content-Length header, made a request, viaduct lower-cased it. This meant the case-sensitive HTTP/3 code couldn’t find the Content-Length header, leading it to incorrectly determine the request was complete while the body remained unsent. This unexpected state caused an infinite loop in the single socket thread responsible for all network requests.

This infinite loop effectively blocked all further network communication, making Firefox unresponsive. To resolve the issue, HTTP/3 was explicitly disabled on the affected GCP load balancers at 09:12 UTC, unblocking users. Mozilla is now reviewing service configurations and improving continuous integration testing to prevent similar incidents, particularly for major changes like HTTP version choices and additional networking layers.