{"UUID":"e9248be4-2e55-481e-bde1-f53f60667e21","URL":"https://aws.amazon.com/message/74876-2/","ArchiveURL":"","Title":"Amazon EC2 DNS Resolution Issues in AP-NORTHEAST-2","StartTime":"2018-11-22T08:19:00+09:00","EndTime":"2018-11-22T09:43:00+09:00","Categories":["cloud"],"Keywords":["ec2","dns","seoul","ap-northeast-2","configuration","resolver","aws"],"Company":"Amazon","Product":"EC2 DNS","SourcePublishedAt":"0001-01-01T00:00:00Z","SourceFetchedAt":"2026-05-04T19:52:45.965472Z","Summary":"A configuration change in the Seoul region incorrectly removed the setting that specifies the minimum healthy hosts for the EC2 DNS resolver fleet, so the system fell back to a very low default. The fleet's healthy host count dropped and in-VPC DNS queries from EC2 instances failed for ~84 minutes until capacity was manually restored. AWS added semantic config validation and per-hour throttling on host removal as remediations.","Description":"On November 22, 2018, between 8:19 AM and 9:43 AM KST, Amazon EC2 instances in the Asia Pacific (Seoul) region (AP-NORTHEAST-2) experienced DNS resolution issues. AWS engineering was alerted at 8:21 AM KST and began working on a resolution, identifying the root cause by 8:48 AM KST. Full recovery for DNS queries from within EC2 instances was achieved by 9:43 AM KST.\n\nThe incident was caused by a reduction in the number of healthy hosts within the EC2 DNS resolver fleet, which provides recursive DNS service to EC2 instances. This reduction led to DNS queries from within EC2 instances failing. EC2 network connectivity and DNS resolution outside of EC2 instances were not affected.\n\nThe root cause was a configuration update that incorrectly removed the setting specifying the minimum healthy hosts for the EC2 DNS resolver fleet in the Seoul Region. This error caused the system to interpret the minimum healthy hosts configuration as a very low default value, resulting in fewer in-service healthy hosts.\n\nTo prevent recurrence, AWS immediately validated and ensured correct capacity settings for the EC2 DNS resolver service across all regions. They are implementing semantic configuration validation for all EC2 DNS resolver configuration updates to guarantee sufficient minimum healthy hosts. Additionally, throttling is being added to limit the amount of healthy host capacity that can be removed from service each hour, which will prevent downscaling of the EC2 DNS resolver fleet even if an invalid configuration parameter is introduced."}