{"UUID":"4f96113f-c48f-4b51-9c18-a7f32e649ec2","URL":"https://aws.amazon.com/message/061323/","ArchiveURL":"","Title":"AWS Lambda Service Event in Northern Virginia (US-EAST-1) Region on June 13th, 2023","StartTime":"2023-06-13T18:49:00Z","EndTime":"2023-06-13T22:37:00Z","Categories":["cloud"],"Keywords":["lambda","us-east-1","scaling","software defect","sts","eks","eventbridge","management console"],"Company":"Amazon","Product":"AWS Lambda","SourcePublishedAt":"0001-01-01T00:00:00Z","SourceFetchedAt":"2026-05-04T19:51:42.046666Z","Summary":"As the Lambda Frontend fleet scaled in response to normal daily traffic growth, it crossed a capacity threshold within a single cell that had never been reached before, triggering a latent software defect that caused Execution Environments to be successfully allocated but never fully utilized. Lambda invocations failed in the affected cell, cascading into STS, EKS, EventBridge, Connect, and the AWS Management Console for several hours.","Description":"On June 13th, 2023, starting at 11:49 AM PDT, customers in the Northern Virginia (US-EAST-1) Region experienced increased error rates and latencies for AWS Lambda function invocations. Synchronous Lambda invocations began to recover by 1:45 PM PDT, and all affected services had fully recovered by 3:37 PM PDT.\n\nThe incident was triggered when the Lambda Frontend fleet, scaling in response to normal daily traffic growth, crossed a previously unreached capacity threshold within a single cell. This activated a latent software defect, causing Lambda Execution Environments to be successfully allocated but never fully utilized by the Frontend.\n\nThis degradation in Lambda function invocations led to increased error rates and latencies in several other AWS services. These included Amazon STS, AWS Management Console, Amazon EKS, Amazon Connect, Amazon EventBridge, and AWS Support Center. Customers experienced issues such as failed calls, chat/task initiation failures, sign-in problems, and console unavailability.\n\nAs an immediate mitigation, engineers identified the defect and scaled down the Lambda Frontend fleet to a level that no longer triggered the issue. To prevent recurrence, the scaling activities that caused the event were disabled, and the latent bug was resolved and deployed across all regions. Additionally, a gap in Lambda's cellular architecture for Frontend scaling was identified, with immediate actions taken and a larger effort planned to ensure cells are bounded to well-tested sizes."}