What can we learn from the recent AWS outage, and how can we apply those lessons to our own infrastructure?

What Happened?

On October 20, 2025, AWS experienced a major disruption that rippled across the internet (and social media), affecting widely used services such as Zoom, Microsoft Teams, Slack, and Atlassian. The issue originated not in a single data center or customer workload, but in the AWS control plane, the management layer that coordinates how resources like EC2 instances, DynamoDB tables, and IAM roles operate.

The initial trigger appears to be a DNS failure around the DynamoDB API endpoint in the US-EAST-1 region, compounded by a malfunction in the subsystem that monitors network load balancers. Because those health-monitoring services also run in US-EAST-1, AWS throttled new EC2 instance launches while restoring the subsystem.

Though AWS markets its regions as isolated clusters of data centers with independent power and cooling, this incident showed that core control-plane functions remain centralized, creating hidden dependencies that can cascade globally.

Root Cause: A Single-Region Control Plane

Analysts quickly identified that US-EAST-1 hosts AWS's common control plane that supports a host of global services. Workloads running in Europe or Asia actually rely on API calls that route back through or to US-EAST-1, so the failure there had global consequences.

When the region's DNS and health-check subsystems failed, those control-plane calls stalled worldwide. The end result was a global slowdown in EC2 launches, configuration updates, and authentication, despite the other regions being technically "healthy."

AWS's own design guidance encourages customers to spread workloads across availability zones for resiliency, but these customer-facing resiliency mechanisms ultimately depend on the same centralized control plane. In other words, data-plane isolation worked as designed, but control-plane isolation did not.

This pattern has surfaced before, not just at AWS. Cloudflare, Microsoft, and Google have all suffered outages triggered by control-plane or configuration failures that propagated globally. The lesson here is that in modern distributed systems, control-plane fragility can become a single point of failure.

The Broader Pattern

AWS may be in the spotlight now, but looking across the industry, nearly every major cloud or CDN provider, AWS, Cloudflare, Microsoft, Google, has experienced control-plane-related outages in the past five years. These are rarely caused by attacks; more often, they stem from routine operational changes, misconfigurations, or centralized service dependencies.

The October 2025 AWS outage simply demonstrates that no cloud provider is immune. The best defense is architectural: distribute risk, decouple dependencies, and design for graceful degradation.

/* ====== CONFIG ====== */ const MOUNT_ID = "cloud-outages-viz"; /* ====== DATA ====== */ const outages = [ {date:"2/28/2017", year:2017, provider:"AWS", service:"AWS – S3 (US-EAST-1)", cause:"Maintenance engineer mis-typed command; S3 index/placement subsystems deleted.", impact:"~4h S3 unavailability in US-EAST-1; ripple effects widely."}, {date:"6/2/2019", year:2019, provider:"Google", service:"GCP – network control plane", cause:"Misconfig + bug in maintenance automation; BGP withdrawn; severe packet loss.", impact:"~3–4h congestion/packet loss (11:45–~16:10 PT)."}, {date:"6/21/2019", year:2019, provider:"Cloudflare", service:"Cloudflare – CDN / internet routing", cause:"Verizon BGP misconfig routed huge traffic via small ISP; routes corrected later.", impact:"~3h global disruption for Cloudflare-protected sites."}, {date:"10/22/2019", year:2019, provider:"Google", service:"GCP – leader election / networking", cause:"Chubby leadership flaps + race; invalid network programming; 100% packet loss.", impact:"~2.5h; us-west-1b + others lost connectivity (16:30–18:51 PT)."}, {date:"1/25/2020", year:2020, provider:"Microsoft", service:"Azure – SQL / backend dependency", cause:"Backend dependency failure in Azure SQL; broad warnings across services.", impact:"~6h from 05:53."}, {date:"7/17/2020", year:2020, provider:"Cloudflare", service:"Cloudflare – backbone network", cause:"Config change diverted all backbone traffic to ATL; router overwhelmed.", impact:"~27m (21:12–21:39 UTC); normalized by 22:10 UTC."}, {date:"8/30/2020", year:2020, provider:"Cloudflare", service:"Cloudflare – transit connectivity", cause:"CenturyLink/Level 3 outage; failing paths increased 5xx; manual reroute.", impact:"Several hours degraded performance; external network failure."}, {date:"11/25/2020", year:2020, provider:"AWS", service:"AWS – Kinesis & dependent", cause:"Added capacity increased inter-host threads; hit OS thread limit; FE failed.", impact:"~17h; Kinesis + CloudWatch/Lambda/Cognito/Firehose etc."}, {date:"12/14/2020", year:2020, provider:"Google", service:"Google – OAuth/auth", cause:"Quota migration mis-reported zero; automated enforcement blocked writes.", impact:"~47m global auth failures + elevated API errors."}, {date:"2/12/2021", year:2021, provider:"Microsoft", service:"Azure – Cosmos DB & related", cause:"Cosmos DB connectivity failures in East/West US; cascaded to others.", impact:"~1h15m (00:30–01:43 UTC)."}, {date:"3/11/2021", year:2021, provider:"Oracle", service:"OCI – global network & console", cause:"Issues with dynamic DNS provider across many regions.", impact:"~2h outage."}, {date:"4/1/2021", year:2021, provider:"Microsoft", service:"Azure – DNS infrastructure", cause:"Surge exposed code defect; cache efficiency dropped; retries overwhelmed.", impact:"Recovered by 22:00–22:30 UTC (~2.5h)."}, {date:"6/8/2021", year:2021, provider:"Fastly", service:"Fastly – CDN", cause:"Customer config change hit undiscovered bug; 85% network errors.", impact:"95% recovered in 49m."}, {date:"7/22/2021", year:2021, provider:"Akamai", service:"Akamai – Edge DNS", cause:"Software update bug; DNS outage; rollback restored.", impact:"~1h DNS failures."}, {date:"11/16/2021", year:2021, provider:"Google", service:"GCP – network configuration", cause:"Latent bug triggered during leader election; LB misconfig; widespread 404.", impact:"~1h50m (12:39–~14:28 ET)."}, {date:"12/7/2021", year:2021, provider:"AWS", service:"AWS – US-EAST-1 network devices", cause:"Impairment of several network devices disrupted APIs and services.", impact:"Several hours until capacity restored."}, {date:"12/15/2021", year:2021, provider:"AWS", service:"AWS – US-West-1/2 congestion", cause:"Traffic-engineering change caused backbone/ISP congestion.", impact:"~1h degraded services (US-West-1/2)."}, {date:"12/22/2021", year:2021, provider:"AWS", service:"AWS – US-EAST-1 AZ power", cause:"Power failure in AZ-4 at 04:35 PST; instance/volume issues lingered.", impact:"~1h power loss + recovery tail."}, {date:"6/21/2022", year:2022, provider:"Cloudflare", service:"Cloudflare – CDN / data plane", cause:"New architecture rollout withdrew critical BGP prefixes; 19 DCs offline.", impact:"~1h15m (06:27–07:42 UTC)."}, {date:"6/29/2022", year:2022, provider:"Microsoft", service:"Azure – Firewall & Data Explorer", cause:"Multi-region connectivity issues; no disclosed root cause.", impact:"~24h degradation."}, {date:"7/19/2022", year:2022, provider:"Oracle", service:"Oracle & Google – UK cooling", cause:"Heatwave cooling failures; powered down to protect equipment.", impact:"Several hours unavailable until cooling/power restored."}, {date:"8/9/2022", year:2022, provider:"Google", service:"Google – Search/Maps/YouTube", cause:"Software update issue; 500/502 globally.", impact:"~30m until rollback."}, {date:"1/18/2023", year:2023, provider:"Microsoft", service:"Azure – West Europe storage", cause:"Power event in data hall; storage impacted; cascade.", impact:"Several hours; restored by 13:23 UTC."}, {date:"1/24/2023", year:2023, provider:"Cloudflare", service:"Cloudflare – Control plane / ZT", cause:"Release overwrote token metadata; tokens invalidated.", impact:"~121m services degraded/unavailable."}, {date:"1/25/2023", year:2023, provider:"Microsoft", service:"Azure – Outlook/Teams/Portal", cause:"WAN connectivity issue; rollback restored.", impact:"Several hours; multi-continent."}, {date:"2/13/2023", year:2023, provider:"Oracle", service:"OCI – Public DNS API", cause:"Back-end perf issue blocked DNS API.", impact:"~55h (Feb 13 17:30–Feb 15 22:30 GMT)."}, {date:"10/4/2023", year:2023, provider:"Cloudflare", service:"Cloudflare – 1.1.1.1", cause:"Root zone ZONEMD change; DNSSEC signatures expired.", impact:"~3h SERVFAIL spikes."}, {date:"11/2/2023", year:2023, provider:"Cloudflare", service:"Cloudflare – Control plane/analytics", cause:"Flexential PDX power feed lost; dependencies remained.", impact:"~40h (Nov 2 11:44–Nov 4 04:25 UTC)."}, {date:"1/16/2024", year:2024, provider:"Oracle", service:"OCI – network disruption", cause:"Network disruption across regions; coincided with security patch.", impact:"~40m; brief recurrence."}, {date:"3/14/2024", year:2024, provider:"Microsoft", service:"Azure – South Africa N/W", cause:"Multiple subsea cuts + router optic failure; capacity reduced.", impact:"Mar 14 10:33–Mar 15 11:00 UTC."}, {date:"3/26/2024", year:2024, provider:"Cloudflare", service:"Cloudflare – Control plane (PDX01)", cause:"Portland facility power loss; auto failover.", impact:"~7m API/dashboard; analytics degraded part of day."}, {date:"7/30/2024", year:2024, provider:"Microsoft", service:"Azure – M365/Azure", cause:"DDoS + faulty response amplified impact.", impact:"~8h."}, {date:"10/24/2024", year:2024, provider:"Google", service:"GCP – europe-west3", cause:"Power failure + electrical arc; cooling degraded; partial shutdown.", impact:"~7.5h (02:22–10:01 UTC)."}, {date:"11/13/2024", year:2024, provider:"Microsoft", service:"Azure – DNS/Databricks", cause:"DNS issues across regions; mitigated within hours.", impact:"~8.5h (01:13–~09:53 UTC)."}, {date:"12/26/2024", year:2024, provider:"Microsoft", service:"Azure – South Central US", cause:"Data hall failed to switch to alternate power; compute/net/storage loss.", impact:"Dec 26 18:40 UTC → Dec 27 varying times."}, {date:"1/8/2025", year:2025, provider:"Microsoft", service:"Azure – East US2", cause:"Network config issue; 3 storage partitions unhealthy; VM impact.", impact:"Jan 8 → Jan 10 16:39 UTC (~2 days)."}, {date:"3/21/2025", year:2025, provider:"Cloudflare", service:"Cloudflare – R2 & Workers", cause:"Prod creds deleted due to CLI run without --env production; writes failed.", impact:"~1h07m global R2/Workers outage."}, {date:"5/20/2025", year:2025, provider:"IBM", service:"IBM Cloud – control plane (IAM/login)", cause:"Cloud Logs update created traffic surge; IAM overwhelmed.", impact:"~2h10m; 14 services."}, {date:"5/20/2025", year:2025, provider:"Oracle", service:"OCI – identity (Frankfurt)", cause:"Identity outage in Germany Central; failover to NL failed.", impact:"~6h; EU customers couldn’t authenticate."}, {date:"6/2/2025", year:2025, provider:"IBM", service:"IBM Cloud – control plane collapse", cause:"Delayed logins triggered retries; global control overwhelmed; IAM/DNS cascade.", impact:"~14h (09:05–23:10 UTC); 41 services."}, {date:"6/5/2025", year:2025, provider:"IBM", service:"IBM Cloud – multi-service outage", cause:"Another IAM failure; at least 54 services impacted.", impact:">4h; console/API access blocked."}, {date:"6/12/2025", year:2025, provider:"Cloudflare", service:"Cloudflare – Workers KV/Zero Trust", cause:"Third-party storage provider failure; over-dependence acknowledged.", impact:"~2h28m; KV, WARP, Access, Gateway, Images degraded."}, {date:"6/12/2025", year:2025, provider:"Google", service:"GCP – Service Control & APIs", cause:"New quota-policy feature introduced null-pointer; corrupted policy propagated.", impact:"Started 10:51 AM PDT; most regions ~40m, us-central-1 ~2h40m."}, {date:"7/14/2025", year:2025, provider:"Cloudflare", service:"Cloudflare – 1.1.1.1 resolver", cause:"Dormant config error from Jun 6 triggered on deploy.", impact:"~62m global resolver failure."}, {date:"8/11/2025", year:2025, provider:"IBM", service:"IBM Cloud – authentication failures", cause:"IAM failures locked users out; 27 services across 10 regions.", impact:"~2h23m (12:59–14:09 UTC)."}, {date:"8/21/2025", year:2025, provider:"Cloudflare", service:"Cloudflare – Peering links w/ AWS", cause:"Traffic surge from single AWS us-east-1 customer saturated peering links.", impact:"~3h50m (16:27–20:18 UTC)."}, {date:"9/12/2025", year:2025, provider:"Cloudflare", service:"Cloudflare – Dashboard & APIs", cause:"Dashboard bug spammed Tenant Service API; overwhelmed.", impact:"~1h until revert."}, {date:"10/9/2025", year:2025, provider:"Microsoft", service:"Azure – Front Door/portals", cause:"Capacity loss across AFD in EU/AF; routing issues.", impact:"~5h (≈07:40 UTC–≈12:07 EDT)."}, {date:"10/20/2025", year:2025, provider:"AWS", service:"AWS – US-EAST-1 control plane", cause:"DynamoDB endpoint DNS resolution issues from internal health monitor.", impact:"Several hours; global impacts (Slack, Snapchat, Atlassian, etc.)."} ]; /* ====== UTIL ====== */ const parseUSDate = s => { const [m,d,y] = s.split("/").map(v=>parseInt(v,10)); return new Date(y, m-1, d); }; const groupBy = (arr, keyFn) => arr.reduce((m,o)=>((m[keyFn(o)]??=[]).push(o),m),{}); const uniq = arr => Array.from(new Set(arr)); /* ====== RENDER ====== */ (function render(){ const root = document.getElementById(MOUNT_ID); root.innerHTML = ""; root.style.fontFamily = "Inter, system-ui, -apple-system, Segoe UI, Roboto, Arial, sans-serif"; root.style.lineHeight = "1.35"; root.style.color = "#111"; root.style.maxWidth = "1100px"; root.style.margin = "24px auto"; root.style.padding = "8px 12px"; const card = ()=>{ const c=document.createElement("div"); c.style.border="1px solid #e5e7eb"; c.style.borderRadius="12px"; c.style.padding="12px"; c.style.boxShadow="0 1px 2px rgba(0,0,0,0.04)"; c.style.background="#fff"; c.style.marginBottom="16px"; return c; }; // Full-width timeline (top) const timelineCard = card(); root.appendChild(timelineCard); // Full-width incidents-by-year chart (second) const barchartCard = card(); root.appendChild(barchartCard); // Provider picker (AFTER incidents-by-year) const controlsCard = card(); root.appendChild(controlsCard); // Details table (last) const tableCard = card(); root.appendChild(tableCard); // Colors - using actual brand colors from logos const colorFor = (p)=>{ const map = { "AWS": "#FF9900", // AWS orange "Google": "#4285F4", // Google blue "Microsoft": "#0078d4", // Microsoft/Azure blue "Cloudflare": "#FF6633", // Cloudflare orange "Oracle": "#C74634", // Oracle red "Fastly": "#FF282B", // Fastly red "Akamai": "#002F6C", // Akamai blue "IBM": "#0F62FE" // IBM blue }; return map[p] || "#64748b"; }; // Providers/filters const providers = uniq(outages.map(o=>o.provider)).sort(); const active = new Set(providers); const renderControls = ()=>{ controlsCard.innerHTML = ""; const isMobile = window.innerWidth { const lab = document.createElement("label"); lab.style.display="inline-flex"; lab.style.alignItems="center"; lab.style.gap="6px"; lab.style.cursor="pointer"; const cb = document.createElement("input"); cb.type="checkbox"; cb.checked = active.has(p); cb.addEventListener("change", ()=>{ cb.checked?active.add(p):active.delete(p); update(); }); const txt = document.createElement("span"); txt.textContent=p; txt.style.fontSize="12px"; const swatch = document.createElement("span"); swatch.style.width="10px"; swatch.style.height="10px"; swatch.style.borderRadius="2px"; swatch.style.display="inline-block"; swatch.style.background=colorFor(p); lab.appendChild(cb); lab.appendChild(txt); lab.appendChild(swatch); row.appendChild(lab); }); }; // Timeline const makeTimeline = (rows)=>{ timelineCard.innerHTML=""; const h3 = document.createElement("h3"); h3.textContent="Timeline of major cloud outages"; h3.style.margin="0 0 8px 0"; h3.style.fontSize="16px"; timelineCard.appendChild(h3); const W = timelineCard.clientWidth - 24, H = 260, PADL=64, PADR=16, PADT=20, PADB=36; const svg = document.createElementNS("http://www.w3.org/2000/svg","svg"); svg.setAttribute("width","100%"); svg.setAttribute("height", H); svg.setAttribute("viewBox", `0 0 ${W} ${H}`); svg.style.display="block"; svg.style.userSelect="none"; svg.style.touchAction="none"; timelineCard.appendChild(svg); if(rows.length===0){ const empty=document.createElement("div"); empty.textContent="No incidents for selected providers."; empty.style.color="#6b7280"; empty.style.padding="8px 0"; timelineCard.appendChild(empty); return; } const xs = rows.map(r=>parseUSDate(r.date).getTime()); const xMin = Math.min(...xs), xMax = Math.max(...xs); const xScale = t => xMax===xMin ? PADL + (W-PADL-PADR)/2 : PADL + ((t-xMin)/(xMax-xMin))*(W-PADL-PADR); // axis/ticks (years) const axis = document.createElementNS(svg.namespaceURI,"g"); axis.setAttribute("transform",`translate(0, ${H-PADB})`); svg.appendChild(axis); const years = uniq(rows.map(r=>r.year)).sort((a,b)=>a-b); const isMobile = window.innerWidth { const t = new Date(y,0,1).getTime(); const x = xScale(t); const line = document.createElementNS(svg.namespaceURI,"line"); line.setAttribute("x1",x); line.setAttribute("x2",x); line.setAttribute("y1",0); line.setAttribute("y2",8); line.setAttribute("stroke","#94a3b8"); axis.appendChild(line); const txt = document.createElementNS(svg.namespaceURI,"text"); txt.setAttribute("x",x); txt.setAttribute("y",22); txt.setAttribute("text-anchor","middle"); txt.setAttribute("font-size","11"); txt.setAttribute("fill","#475569"); // Use short year format for mobile (except 2017), full year for desktop const yearText = (isMobile && y !== 2017) ? `'${String(y).slice(-2)}` : y; txt.textContent = yearText; axis.appendChild(txt); }); const base = document.createElementNS(svg.namespaceURI,"line"); base.setAttribute("x1",PADL); base.setAttribute("x2",W-PADR); base.setAttribute("y1",H-PADB); base.setAttribute("y2",H-PADB); base.setAttribute("stroke","#cbd5e1"); svg.appendChild(base); const provs = uniq(rows.map(r=>r.provider)).sort(); const yForProv = (p)=> { const i = provs.indexOf(p); const slotH = (H-PADT-PADB) / Math.max(1,provs.length); return PADT + slotH*(i+0.5); }; // provider rows + guides provs.forEach(p=>{ const y = yForProv(p); const t = document.createElementNS(svg.namespaceURI,"text"); t.setAttribute("x",12); t.setAttribute("y",y+4); t.setAttribute("font-size","11"); t.setAttribute("fill","#64748b"); t.textContent = p; svg.appendChild(t); const guide = document.createElementNS(svg.namespaceURI,"line"); guide.setAttribute("x1",PADL); guide.setAttribute("x2",W-PADR); guide.setAttribute("y1",y); guide.setAttribute("y2",y); guide.setAttribute("stroke","#f1f5f9"); svg.appendChild(guide); }); // tooltip const tip = document.createElement("div"); tip.style.position="absolute"; tip.style.pointerEvents="none"; tip.style.background="#111827"; tip.style.color="#fff"; tip.style.padding="8px 10px"; tip.style.borderRadius="8px"; tip.style.fontSize="12px"; tip.style.boxShadow="0 6px 20px rgba(0,0,0,0.2)"; tip.style.display="none"; tip.style.maxWidth="320px"; tip.style.zIndex="999"; timelineCard.style.position="relative"; timelineCard.appendChild(tip); const showTip = (evt, r)=>{ const rect = timelineCard.getBoundingClientRect(); tip.innerHTML = `${r.provider} — ${r.service}
${r.date} • ${r.year}
Root cause: ${r.cause}
Impact: ${r.impact}
`; tip.style.left = (evt.clientX - rect.left + 12) + "px"; tip.style.top = (evt.clientY - rect.top - 10) + "px"; tip.style.display="block"; }; const hideTip = ()=> tip.style.display="none"; // points rows.forEach(r=>{ const x = xScale(parseUSDate(r.date).getTime()); const y = yForProv(r.provider); const g = document.createElementNS(svg.namespaceURI,"g"); const dot = document.createElementNS(svg.namespaceURI,"circle"); dot.setAttribute("cx",x); dot.setAttribute("cy",y); dot.setAttribute("r",6); dot.setAttribute("fill", colorFor(r.provider)); dot.setAttribute("opacity","0.9"); const stroke = document.createElementNS(svg.namespaceURI,"circle"); stroke.setAttribute("cx",x); stroke.setAttribute("cy",y); stroke.setAttribute("r",9); stroke.setAttribute("fill","none"); stroke.setAttribute("stroke", colorFor(r.provider)); stroke.setAttribute("opacity","0.25"); g.appendChild(stroke); g.appendChild(dot); g.style.cursor="pointer"; g.addEventListener("mousemove",(e)=>showTip(e,r)); g.addEventListener("mouseleave", hideTip); svg.appendChild(g); }); }; // Full-width bar chart const makeBar = (rows)=>{ barchartCard.innerHTML=""; const h3 = document.createElement("h3"); h3.textContent="Incidents by year"; h3.style.margin="0 0 8px 0"; h3.style.fontSize="16px"; barchartCard.appendChild(h3); const W = barchartCard.clientWidth - 24, H = 260, PADL=44, PADR=12, PADT=12, PADB=32; const svg = document.createElementNS("http://www.w3.org/2000/svg","svg"); svg.setAttribute("width","100%"); svg.setAttribute("height", H); svg.setAttribute("viewBox", `0 0 ${W} ${H}`); svg.style.display="block"; barchartCard.appendChild(svg); if(rows.length===0){ const empty=document.createElement("div"); empty.textContent="No data."; empty.style.color="#6b7280"; empty.style.padding="8px 0"; barchartCard.appendChild(empty); return; } const byYear = groupBy(rows, r=>r.year); const years = Object.keys(byYear).map(Number).sort((a,b)=>a-b); const counts = years.map(y=>byYear[y].length); const xScale = i => PADL + i * ( (W-PADL-PADR) / years.length ) + 6; const bw = Math.max(10, (W-PADL-PADR) / years.length - 12); const maxC = Math.max(...counts); const yScale = v => H-PADB - ( (v/maxC) * (H-PADT-PADB) ); // axes const axisY = document.createElementNS(svg.namespaceURI,"line"); axisY.setAttribute("x1",PADL); axisY.setAttribute("x2",PADL); axisY.setAttribute("y1",PADT); axisY.setAttribute("y2",H-PADB); axisY.setAttribute("stroke","#cbd5e1"); svg.appendChild(axisY); const axisX = document.createElementNS(svg.namespaceURI,"line"); axisX.setAttribute("x1",PADL); axisX.setAttribute("x2",W-PADR); axisX.setAttribute("y1",H-PADB); axisX.setAttribute("y2",H-PADB); axisX.setAttribute("stroke","#cbd5e1"); svg.appendChild(axisX); // Tooltip for bar chart const tip = document.createElement("div"); tip.style.position="absolute"; tip.style.pointerEvents="none"; tip.style.background="#111827"; tip.style.color="#fff"; tip.style.padding="6px 8px"; tip.style.borderRadius="6px"; tip.style.fontSize="11px"; tip.style.boxShadow="0 4px 12px rgba(0,0,0,0.2)"; tip.style.display="none"; tip.style.zIndex="999"; barchartCard.style.position="relative"; barchartCard.appendChild(tip); const showBarTip = (evt, year, count)=>{ const rect = barchartCard.getBoundingClientRect(); tip.textContent = `${year}: ${count} incident${count !== 1 ? 's' : ''}`; tip.style.left = (evt.clientX - rect.left + 8) + "px"; tip.style.top = (evt.clientY - rect.top - 8) + "px"; tip.style.display="block"; }; const hideBarTip = ()=> tip.style.display="none"; // Y ticks const tickVals = Array.from({length: Math.min(5, maxC)}, (_,i)=>Math.round((i+1)*maxC/Math.min(5,maxC))); tickVals.forEach(v=>{ const y = yScale(v); const line = document.createElementNS(svg.namespaceURI,"line"); line.setAttribute("x1",PADL); line.setAttribute("x2",W-PADR); line.setAttribute("y1",y); line.setAttribute("y2",y); line.setAttribute("stroke","#eef2f7"); svg.appendChild(line); const t = document.createElementNS(svg.namespaceURI,"text"); t.setAttribute("x",PADL-8); t.setAttribute("y",y+4); t.setAttribute("text-anchor","end"); t.setAttribute("font-size","11"); t.setAttribute("fill","#64748b"); t.textContent=v; svg.appendChild(t); }); years.forEach((y,i)=>{ const v = byYear[y].length; const x = xScale(i); const yTop = yScale(v); const rect = document.createElementNS(svg.namespaceURI,"rect"); rect.setAttribute("x",x); rect.setAttribute("y",yTop); rect.setAttribute("width", bw); rect.setAttribute("height", H-PADB - yTop); // Color gradient based on incident count - darker for more incidents const intensity = v / maxC; // 0 to 1 const hue = 200; // Blue hue const saturation = 70 + (intensity * 30); // 70% to 100% const lightness = 80 - (intensity * 40); // 80% to 40% (darker for more incidents) rect.setAttribute("fill", `hsl(${hue}, ${saturation}%, ${lightness}%)`); rect.setAttribute("opacity","0.9"); rect.style.cursor="pointer"; rect.addEventListener("mousemove",(e)=>showBarTip(e,y,v)); rect.addEventListener("mouseleave", hideBarTip); svg.appendChild(rect); const label = document.createElementNS(svg.namespaceURI,"text"); label.setAttribute("x", x + bw/2); label.setAttribute("y", H-PADB+16); label.setAttribute("text-anchor","middle"); label.setAttribute("font-size","11"); label.setAttribute("fill","#475569"); label.textContent = y; svg.appendChild(label); }); }; // Compact + scrollable table const makeTable = (rows)=>{ tableCard.innerHTML=""; const h3 = document.createElement("h3"); h3.textContent="Details"; h3.style.margin="0 0 6px 0"; h3.style.fontSize="15px"; tableCard.appendChild(h3); const info = document.createElement("div"); info.style.fontSize="11px"; info.style.color="#475569"; const providerCounts = groupBy(rows, r=>r.provider); const summary = Object.keys(providerCounts).sort().map(p=>`${p}: ${providerCounts[p].length}`).join(" • "); info.textContent = rows.length + " incidents" + (summary? " — " + summary : ""); info.style.marginBottom = "4px"; tableCard.appendChild(info); const scroller = document.createElement("div"); scroller.style.overflow="auto"; scroller.style.maxHeight="360px"; scroller.style.border="1px solid #e5e7eb"; scroller.style.borderRadius="10px"; tableCard.appendChild(scroller); const table = document.createElement("table"); table.style.borderCollapse="separate"; table.style.borderSpacing="0"; table.style.width="100%"; table.style.fontSize="12px"; table.style.tableLayout="fixed"; scroller.appendChild(table); const th = (t,w)=>{ const e=document.createElement("th"); e.textContent=t; e.style.textAlign="left"; e.style.padding="8px 10px"; e.style.background="#f8fafc"; e.style.position="sticky"; e.style.top="0"; e.style.borderBottom="1px solid #e5e7eb"; e.style.whiteSpace="nowrap"; if(w) e.style.width=w; return e; }; const thead = document.createElement("thead"); const trh = document.createElement("tr"); const isMobile = window.innerWidth { const e=document.createElement("td"); e.textContent=t; e.style.padding="8px 10px"; e.style.borderBottom="1px solid #f1f5f9"; e.style.verticalAlign="top"; e.style.lineHeight="1.25"; e.style.wordBreak="break-word"; e.style.whiteSpace="normal"; if(isProv){ e.style.whiteSpace="nowrap"; e.style.fontWeight="600"; } return e; }; const tbody = document.createElement("tbody"); rows .sort((a,b)=>parseUSDate(b.date)-parseUSDate(a.date)) .forEach(r=>{ const tr=document.createElement("tr"); // Format date as "Month Year" (e.g., "October 2025") const date = parseUSDate(r.date); const formattedDate = date.toLocaleDateString('en-US', { month: 'long', year: 'numeric' }); tr.appendChild(td(formattedDate)); const ptd = td(r.provider, true); ptd.style.color = colorFor(r.provider); tr.appendChild(ptd); if (!isMobile) { tr.appendChild(td(r.service)); tr.appendChild(td(r.cause)); } tr.appendChild(td(r.impact)); tbody.appendChild(tr); }); table.appendChild(tbody); }; function currentRows(){ return outages.filter(o=>active.has(o.provider)); } function update(){ makeTimeline(currentRows()); makeBar(currentRows()); renderControls(); makeTable(currentRows()); } update(); // Responsive redraw let resizeTO=null; window.addEventListener("resize", ()=>{ clearTimeout(resizeTO); resizeTO=setTimeout(update,120); }); })();

Lessons Learned for Infrastructure Architects

Understanding and analyzing major outages like AWS's is essential for infrastructure engineers. Each incident reveals gaps between design assumptions and real-world complexity, exposing weak points that might otherwise go unnoticed, and ultimately impact service availability. By studying what failed, why it failed, and how recovery proceeds, architects can refine their infrastructure and systems to be more fault-tolerant and resilient. This mindset of continuous learning ensures that when the next disruption happens, the impact on users and business operations is minimized. So what are the key lessons here? 

1. Design for true multi-region, active-active operation. A standby region isn't enough if control traffic can't reach it. Run in an active-active configuration so that the loss of one resource or region doesn't disable the service. 

2. Avoid single-region control planes:   It seems obvious to say now, but configuration and metadata services should be either fully local or replicated globally. Any system that depends on a single region's DNS, load balancers, or other systems for coordination introduces a global risk.

3. Separate the control plane from the data plane:  The AWS incident began in the control plane but quickly cascaded to the data plane. Architect systems so runtime traffic can continue independently from configuration or provisioning systems.

4. Distribute DNS and caching layers:   Multi-provider DNS with long TTLs can reduce the impact of control-plane-related resolution failures. Cached or regional read replicas keep applications partially functional when the origin is unreachable.

5. Implement circuit breakers and bulkhead isolation:   Systems should fail fast and gracefully. Circuit breakers can reroute or degrade functionality instead of hammering a failing endpoint. Bulkhead isolation limits the spread of failures between components.

6. Continuously test failure scenarios:   Regular "chaos" testing validates that redundancy, DNS failover, and RTO/RPO objectives work in practice. AWS's own Resilience Hub encourages these tests, but the lesson applies to any cloud or hybrid deployment. Check out Chaos Monkey, introduced by Netflix in 2011, as an example. 

7. Plan for multi-cloud or hybrid resiliency:   Multi-availability-zone redundancy doesn't protect against control-plane issues. Deploying across multiple cloud providers or keeping a minimal on-prem footprint prevents total dependence on one provider's management systems.

8. Decouple capacity from failover logic:  AWS mitigated the outage by throttling new instance launches, buying time, not resilience. Reserve compute in secondary regions ahead of time, but ensure failover logic works autonomously from the control plane.

We’re proud that Wallarm's Security Edge demonstrates how these lessons can be applied proactively. By shifting API protection into a multi-cloud, active-active edge fabric, organizations gain resilience not just against attacks, but against the infrastructure failures that even the largest providers occasionally suffer.

Security Edge: Lessons Learned Applied

In designing the architecture for Security Edge, we wanted to address many of these specific challenges. Today, Wallarm's Security Edge embodies many of these principles by design. It was intentionally built  to avoid the single-provider fragility exposed by the AWS outage. Security Edge includes: 

  • Active-active, multi-cloud architecture:  Security Edge runs enforcement nodes across AWS, Azure and other providers in an active-active configuration. Traffic can be instantly rerouted if one provider experiences degradation. This multi-cloud distribution eliminates the single-region or single-provider risk that impacted AWS. 
  • Decoupled from customer infrastructure:  Security Edge operates as a managed, cloud-native security layer, not a component embedded in customer environments. Its filtering and enforcement nodes are positioned close to the API edge, but isolated from the customer's own data plane. In other words, even if a customer's cloud or provider fails, the Wallarm protection layer remains operational.
  • Always-on availability with automated failover:  Wallarm's architecture provides automatic global failover and high availability independent of any one cloud provider or CDN. Customers benefit from security continuity even when the underlying infrastructure is disrupted.
  • Low-latency, edge-proximate enforcement:  By placing filtering nodes near the customer's API endpoints, Security Edge maintains low latency while providing deep inspection and telemetry. The distributed footprint ensures that even during a provider outage, legitimate traffic continues to flow efficiently.
  • API-native and protocol-aware:  The platform supports REST, GraphQL, gRPC, Web Sockets and legacy protocols, ensuring that resilience doesn't come at the cost of compatibility. 
  • Secure by design. Mutual TLS (mTLS) provides secure, authenticated connections across all nodes, essential for regulated industries.
  • Simplified deployment and management:  Because Security Edge is provisioned via a DNS change, teams don't have to manage control-plane dependencies or infrastructure updates. The security layer remains operational even if customer environments or specific regions experience downtime.

What’s Next? 

Interested in hearing from Wallarm's Field CTO, Tim Ebbers  on lessons learned from the AWS outage and how to architect for resiliency. Watch the fireside chat on demand. Click here.