Progressive Collapse in Distributed Systems: Turning the Availability Budget into a Governance Decision
The decision hiding inside a conference talk
A postmortem ends with a shopping list: a circuit breaker here, a timeout there, a second region everywhere. Nobody asks who pays for the second region or which team owns the timeout.
On 19 August 2026, InfoQ published the transcript of a talk by Sam Newman, "Understanding Progressive Collapse: How To Avoid A Cascading Failure", recorded at QCon London on 18 March 2026. The argument comes from structural engineering, where progressive collapse describes a local failure that takes down a disproportionate share of a structure. Newman applies it to distributed systems and draws the uncomfortable conclusion: the mitigation catalogue is a list of expensive architectural choices, not practices you can paste into a service.
That matters where you approve spend and sign off on risk. The talk reduces resilience work to three levers, lifted from a NIST guide on reducing the potential for progressive collapse in buildings (NISTIR 7396, February 2007): reduce hazards, strengthen components, and reduce interconnection. Each has a bill attached. The useful question is not "did we implement the pattern" but "which of these did we buy, what did it cost, and who accepted the residual risk".
Newman also pushes back on the reflex to hunt for a single root cause. A cascade rarely has one origin, and resilience engineering deliberately differs from traditional safety management. Traditional safety tries to stop the bad event from ever happening; resilience accepts that the event happens and works to bound the damage. For an organisation, that reframes the budget: protection money is committed before the incident, not in the week after it.
Three failures, read in a different order
Start with the case that lands closest to home.
A second-hand vehicle marketplace ran on ten servers and normally handled 30 to 60 concurrent requests. A downstream site began accepting connections and then hanging on them. The client waited 30 seconds before timing out, far too lenient, and that wait was enough to exhaust the connection pool. With the pool empty, requests for unrelated product lines could not be dispatched and blocked in turn; the native threading model turned every in-flight request into an operating system thread; the CPU saturated; live traffic climbed past 800 concurrent requests. Meanwhile the humans on the other side of the connection were pressing reload, and every reload added pressure to a system that was already failing. The fix list that followed was unglamorous and effective: a separate connection pool per downstream server, circuit breakers, and bulkheads around each dependency. The measure that would actually have prevented the collapse was the cheapest one on the list and was missing: declaring a maximum acceptable load and dropping the excess.
Now scale the same shape up. At 23:48 PDT on 19 October 2025, the internal DNS management system for DynamoDB in the us-east-1 region left an empty DNS record for the regional endpoint, and the automation failed to repair it. The cause was a latent race condition between two components that exist precisely to provide availability. A planner watches the health and capacity of the load balancers and produces plans; several enactors then apply those plans endpoint by endpoint. One enactor hit unusual delays and kept retrying. The planner kept producing newer plans. Another enactor picked up one of the newer plans and raced through all the endpoints in seconds. When the slow enactor finally finished, it wiped what the fast one had written.
Because those lower-level services are consumed by other services in the same cloud, the cascade had a long reach: network load balancers went down, and compute, queues and EKS followed. The talk names the products that sat on top: Alexa, Ring, Slack, Snapchat, Zoom and Shopify, along with a surprising number of United Kingdom banks. Two AWS responses are worth copying. The remediation replaced the live plan with a new plan inside a Route53 transaction, so every endpoint was updated with a consistent plan even when several enactors competed. And the automated scaling activity that triggered the event was switched off until every remediation shipped.
The third case is the origin of the metaphor. Ronan Point, a tower block in Canning Town, London, partially collapsed on 16 May 1968. A gas leak produced an explosion in a kitchen at around quarter to six in the morning. The outer wall was load-bearing, the four floors above came down in a concertina effect, and four people died. The building used a prefabricated system designed for a workforce without specialist training, and the joints between panels had been filled with newspaper and cigarette packets instead of concrete. Dismantling it piece by piece produced a new building code, closer supervision, and short daily briefings for the crews.
The recovery window nobody budgets for
Restoring the DNS was not the same as restoring the platform. According to the ThousandEyes analysis, DynamoDB connectivity came back at 09:25 UTC, yet EC2 instance launches kept failing until 20:50 UTC, more than eleven hours later, and full restoration took until 11:05 UTC on 21 October, more than a day after the initial fault. The cause was fixed in hours; the system took more than a day to become healthy, because every layer holding inconsistent state drains its backlog at its own pace. A recovery objective is a budget for backlog, not for a fix.
Pricing the three levers
**Reduce hazards** is the cheap lever: switch off the automation that caused the incident, cap the depth of every queue, drop work above a declared capacity, add monitoring. It buys margin, and it does not change the architecture.
**Strengthen components** is the expensive lever, and it is where money gets decided. The disaster recovery ladder that AWS documents has four rungs: backup and restore, pilot light, warm standby, and multi-site active/active. For the loss of a single data centre, backup and restore may be enough; if your definition of a disaster reaches an entire region, or you carry regulatory obligations, you need to climb. Pilot light keeps assets and data flowing in the secondary region but cannot serve traffic until you turn on servers; warm standby keeps a scaled-down environment running and only needs scaling; active/active removes the downtime and the data loss window at the highest cost. The choice is driven by the recovery time objective and the recovery point objective, which Newman reminds us are business requirements to agree with the product owner, not parameters the platform team sets alone.
**Reduce interconnection** is the middle lever, and the easiest to apply badly. The bulkhead pattern from the Microsoft Azure architecture guidance is the canonical version: isolate the resources dedicated to each dependency so that a disruption in one service cannot reach the whole application, preserve whatever functionality still works, and optionally combine it with retry, circuit breaker and throttling. The pattern is not free: it trades resource efficiency for isolation, and a bulkhead that is too generous is a bulkhead nobody designed. Lower down the same list sit separate connection pools per dependency, a local fallback when the upstream provider fails, and eventually a completely independent implementation on a second provider. Monzo runs that last option in production: the stand-in platform is a separate set of systems on Google Cloud that can take over card spending, cash withdrawal, transfers and balances while the primary platform runs on AWS. Both are Kubernetes clusters, each approves or declines transactions on its own, and no service runs in both platforms, so there is no shared code to fail in the same way twice.
The paradox you have to accept
Every one of these levers adds machinery, and machinery adds failure modes. The redundancy that AWS built into the DNS enactor system, several copies so that the loss of an availability zone would not take the service down, was exactly what made the race condition possible: with one enactor the plans would have been applied one after another. The same trap sits behind circuit breakers. A breaker that trips on a partially broken dependency cuts off the healthy half of it. If every client trips at the same moment, an overloaded system becomes a system with no load, and the returning traffic arrives as a wave. Breakers configured around retries can produce the retry storm they were meant to prevent. Marc Brooker, who has written about both failure modes, notes that circuit breakers introduce modal behaviour, make the system harder to test and can extend recovery, and recommends limiting retries locally with a token bucket, or placing breakers only around retry paths, instead of trusting the cut-off blindly.
The conclusion Newman draws is worth carrying into a design review: sometimes the right decision is to not do the thing. Declining multi-cloud or active/active is legitimate, provided someone writes it down and accepts the residual risk. Resilience work done by rote adds failure modes faster than it removes them.
Turning it into an availability budget
Every critical service needs a declared service level objective, and that objective produces an error budget: the amount of failure the organisation has agreed to tolerate over a rolling window of 30 days. With a 99.9% objective the budget is 0.1% of requests. That 0.1% absorbs a full outage of roughly twenty minutes, and it also absorbs a week of two percent scattered errors. The difference is a risk decision, and it belongs to a person with a name on it.
Google's site reliability engineering guidance gives you the instrument. The burn rate measures how fast the budget is consumed. Against a 99.9% objective over 30 days, a rate of one equals 0.1% errors and exhausts the budget in 30 days; a rate of two in 15 days; a rate of ten in three days; a rate of one thousand takes the whole budget in 43 minutes at a 100% error rate. Spending 5% of a 30-day budget in a single hour takes a rate of 36. Those numbers give you two kinds of alert: one that pages when the budget burns fast, and one that opens a ticket for the slow burn that would quietly exhaust the month without waking anyone.
Recording the ratio once keeps the rules readable:
```yaml groups: - name: availability_budget_recording rules: - record: service:error_ratio:rate1h expr: sum(rate(http_requests_total{code=~"5.."}[1h])) / sum(rate(http_requests_total[1h])) - record: service:error_ratio:rate1d expr: sum(rate(http_requests_total{code=~"5.."}[1d])) / sum(rate(http_requests_total[1d])) - name: availability_budget_alerts rules: - alert: error_budget_burn_fast expr: service:error_ratio:rate1h > (36 * 0.001) for: 2m labels: severity: page - alert: error_budget_burn_slow expr: service:error_ratio:rate1d > (10 * 0.001) for: 30m labels: severity: ticket ```
The platform team then publishes the operating limits that today live in the head of each service owner. Envoy gives circuit breaking per cluster and per priority, enabled by default with a modest 1024 connections per cluster, live state in the proxy statistics, and an overloaded header stamped on requests once a breaker opens:
```yaml clusters: - name: payments connect_timeout: 0.5s circuit_breakers: thresholds: - priority: DEFAULT max_connections: 1024 max_pending_requests: 1024 max_requests: 1024 max_retries: 3 - priority: HIGH max_connections: 256 max_pending_requests: 128 max_requests: 256 max_retries: 3 ```
Give every outbound call a route-level retry policy with a per-try timeout and a hard retry count, so that retries stay bounded even when the dependency is slow rather than failing.
Timeouts at the edge must reflect the worst acceptable latency, and the connection budget must be smaller than the downstream can survive. Where that budget is enforced, in the proxy or in the client library, matters more than its size: a pool capped at the real capacity of the downstream is worth more than a breaker with a generous threshold.
Backpressure deserves its own query, because it is the one that fails first. If the consumer is slower than the producer, the queue grows until it eats the memory. A depth limit plus explicit rejection of the excess is the only defence:
``` curl -s 'http://localhost:9090/api/v1/query' --data-urlencode 'query=max(queue_depth) by (queue)' ```
Retries deserve one more limit, the cheapest on the list. Google recommends a retry budget per process, 60 retries per minute as the worked example: once the budget is spent, stop retrying and return the error. The gap between a capacity planning failure and a global collapse is exactly that ceiling.
Who signs what
Split the decision three ways. The business decides what level of availability it is buying, because that is a commercial choice expressed in money. The platform team turns that level into an error budget, a set of limits and an alert policy. Each product team decides which part of the cascade it will accept, with the cost on the table, and records the answer. The output should read like a contract clause: we accept that dependency X takes us down for four to eight minutes per quarter, which is why we are not paying for a second active region.
Four metrics keep that conversation honest: the burn rate of each service budget, actual recovery time against the agreed objective, the count of critical dependencies with no bulkhead and no queue depth limit, and the share of traffic that still completes in degraded mode. With those numbers the review stops being a list of engineering wishes and becomes a decision table with an owner, a cost and a date.
A ninety-day plan
Nothing here needs a new budget line to start.
**Days 1 to 30, measure.** Map the dependency topology of every critical service, declare one objective per service, derive the budget, and publish the burn rate on the operations dashboard. Expect the first finding to be uncomfortable: most services have no declared objective.
**Days 31 to 60, bound.** One declared timeout per dependency, a retry budget per process, a breaker on every outbound call, a maximum queue depth, and a load shedding path on the critical route. Every limit ships with the metric that watches it and the runbook page for when it trips.
**Days 61 to 90, rehearse.** Break a dependency for real in pre-production, measure the recovery time and the traffic lost, then write the decision: move one rung up the recovery ladder, accept the risk in writing, or take that dependency off the critical path.
Who this is for
This is for the person who approves the spend and the person who answers for the outcome: platform leads, chief technology officers and engineering managers who need to answer the question, what do we do so this never happens again, with a number, an owner and a date. It is not a tutorial on circuit breakers. It is an argument for treating resilience as a governed decision with a budget that can be audited every quarter.
The full transcript is on InfoQ, published on 19 August 2026, and Newman's book dedicates a chapter to progressive collapse and another to disaster recovery across sites. Read both, then take one uncomfortable question into your next architecture review: with the budget we have, which failures are we choosing to accept, and who signed for them?