‘It’s always DNS’: Single-point DNS bug in AWS DynamoDB sparks 16‑hour global outage

On October 26, a single software bug in Amazon’s DNS management machinery set off a chain reaction that left millions of users staring at error screens for more than 15 hours. Amazon engineers say the incident — which the company pegs at 15 hours and 32 minutes — began in a DynamoDB component used to update DNS configurations and rapidly cascaded until the database itself was effectively taken out of service. Network monitoring company Ookla reported that its DownDetector platform received more than 17 million reports tied to the event, spanning some 3,500 organizations. The US, UK and Germany generated the most reports. Snapchat, AWS-hosted services and Roblox were among the most frequently flagged outages; Ookla called the incident “among the largest internet outages on record” for its service. Amazon’s post‑mortem points to a race condition in the DNS Enactor — the piece of DynamoDB responsible for enacting updates to domain lookup tables across AWS endpoints to optimize load balancing. Engineers say the Enactor experienced “unusually high delays” on several endpoints and had to retry updates. While it retried, the DNS Planner continued to churn out new plans and a second Enactor began to implement them. The timing of those overlapping operations produced the race condition that ultimately overwhelmed DynamoDB. In plain terms, two parts of the same system tried to change the DNS state at the same time and the interaction — driven by timing and unexpected delays — produced behavior the software couldn’t safely handle. What began as an internal state-management problem rippled outward, affecting services and customers that depend on DynamoDB and AWS’s DNS infrastructure. The outage highlighted a painful truth for many operations teams: DNS and configuration systems are brittle, and when they fail they can take a lot with them. “It’s always DNS,” read early reactions among engineers on developer forums — a wry shorthand for the tendency of name resolution and routing problems to produce outsized outages. The event also reopened debates about architecture and redundancy. Some engineers and operators pointed to the risk of concentrated cloud dependency: when a large fraction of web services and apps sit on the same underlying provider and share control planes, a failure in a single subsystem can cascade across thousands of customers. The post‑mortem public discussion underscored another constraint: running multi‑region or multi‑cloud redundancy is expensive and complex, and for many companies it remains a tradeoff between cost and resilience. Commentators noted that the difficulty of scaling monolithic apps to multi‑region redundancy can leave even massive services exposed when an internal component becomes unexpectedly popular or heavily loaded. Popular orchestration platforms such as Kubernetes are often cited as a way to plan for scale and portability, but they are not a magic bullet and add operational overhead of their own. Amazon said its engineers have identified the software bug and provided the detailed timeline in the post‑mortem; the company is implementing fixes and mitigations to prevent the same class of failure. For the broader internet, the outage served as a reminder that critical infrastructure — from DNS to distributed databases — demands careful design, testing and, in many cases, diversity of providers or regions to limit blast radius. For now, engineers and business leaders are left to weigh the balance between cost and safety: invest in expensive multi‑region redundancy and multi‑cloud complexity, or accept the risk that a single, subtle bug in a core control plane can become a 16‑hour global story.