For most of the last two decades, the default question in system design was "how fast can we make it?" Throughput, requests per second, p99 latency under peak load. These remain important. But in 2026 the more pressing question for production systems is "what happens when something breaks?" Resilience, the ability of a system to continue operating at an acceptable level when components fail, has moved from a nice-to-have quality to the primary design constraint. This shift connects directly to themes explored across these distributed systems notes: the reality of partial failure, the cost of coordination, and the consequences of queueing under saturation.

This article examines why resilience now takes precedence over raw throughput, how to measure it as an engineering property rather than a vague aspiration, the architectural patterns that support fault tolerance, how to validate those patterns through deliberate testing, and the organisational factors that determine whether any of it holds up in a real incident.

Why resilience overtakes throughput

The reasons are both technical and structural. On the technical side, modern systems are deeply composed. A single user request may traverse dozens of services, queues, caches, and external APIs. Each hop introduces a failure probability. When you multiply modest per-component failure rates across a long call chain, the aggregate probability of some partial failure on any given request becomes significant. A system optimised purely for throughput, with tight timeouts, aggressive parallelism, and minimal buffering, tends to be brittle under these conditions. Small disturbances amplify rather than dissipate.

On the structural side, regulatory expectations have tightened. Financial services regulators in the EU and UK now require firms to demonstrate operational resilience under specific disruption scenarios. DORA (the Digital Operational Resilience Act) imposes testing obligations that go well beyond "we have a disaster recovery plan." Healthcare, energy, and telecommunications sectors face analogous pressures. Customers, too, have recalibrated. The tolerance for outages has dropped. A fifteen-minute payment processing failure that might have been forgiven in 2018 now generates regulatory scrutiny, social media escalation, and measurable customer churn.

There is also the cascading failure problem. High-throughput systems that lack resilience mechanisms tend to fail in correlated ways. One overloaded service triggers timeouts in its callers, which retry, which increases load further. The observability to detect this early and the circuit-breaking to arrest it are resilience properties, not throughput properties. Building faster does not help if the system collapses under its own retry storms.

Measuring resilience

If resilience is a design goal, it must be measurable. Vague claims about "high availability" are not useful. There are several concrete properties worth tracking.

  • Mean time to recovery (MTTR)How quickly the system returns to normal operation after a failure. MTTR is often more important than mean time between failures (MTBF) because in complex systems, failures will happen. The question is how fast you recover.
  • Blast radiusThe proportion of users or functionality affected by a single component failure. A well-isolated system confines the impact of a database outage to the features that depend on that database, not the entire product.
  • Degradation modesWhat the system does under partial failure. Does it return stale data? Queue requests for later processing? Shed low-priority traffic? A system with defined degradation modes is more resilient than one that either works perfectly or fails completely.
  • Recovery point and recovery time objectives (RPO/RTO)How much data loss and how much downtime are acceptable. These are business decisions that constrain architectural choices. They should be explicit, tested, and reviewed periodically.

Measuring these properties requires investment in observability tooling that captures not just steady-state metrics but also the transitions: how long did it take to detect the problem, how long to mitigate, how long to fully recover. SLIs and SLOs, when defined honestly and reviewed regularly, provide a framework for this. The key word is honestly. An SLO that is never breached is probably set too loosely to drive useful engineering decisions.

Architectural patterns for fault tolerance

Several patterns have proven effective across different system types. None of them is novel. What has changed is the expectation that production systems employ them by default rather than as afterthoughts.

Bulkheads isolate components so that failure in one does not propagate to others. The metaphor comes from ship design: watertight compartments prevent a single hull breach from sinking the vessel. In software, bulkheads take the form of separate thread pools, connection pools, or process boundaries for different subsystems. If your recommendation engine exhausts its database connections, your checkout flow should be unaffected.

Circuit breakers prevent a failing downstream dependency from consuming resources in its callers. When a service detects that calls to a dependency are consistently failing or timing out, it opens the circuit, returning a fallback response immediately rather than waiting. This limits the blast radius and gives the failing component time to recover without being hammered by retries. The pattern interacts closely with retry semantics and idempotency, since retries without circuit-breaking create exactly the amplification you are trying to avoid.

Graceful degradation means designing the system to offer reduced functionality rather than total failure. A product listing page that cannot reach the personalisation service should still render, just without personalised recommendations. This requires thinking about feature dependencies at design time and building fallback paths for each optional dependency. It is more work than treating every dependency as critical, but it dramatically improves the user experience during partial outages.

Load shedding is the deliberate rejection of excess traffic to protect system stability. When a service approaches its capacity limit, it begins returning errors for a fraction of incoming requests rather than allowing all requests to slow down. This is counterintuitive for teams accustomed to optimising for throughput, but it is essential for resilience. A system that serves 80% of requests successfully during a traffic spike is far more useful than one that serves 0% because it collapsed under load. Queueing theory explains why: as utilisation approaches 100%, latency grows without bound, and the system becomes effectively unavailable for everyone.

Timeouts and deadlines propagate through the call chain. A top-level request with a 2-second deadline should pass remaining budget to each downstream call. Without deadline propagation, a slow downstream service can consume the entire budget, leaving no time for the caller to try alternatives or return a partial result.

Testing and validating resilience

Architectural patterns only matter if they work under real conditions. The gap between "we have circuit breakers" and "our circuit breakers are correctly configured and actually fire when needed" is large. Closing that gap requires deliberate testing.

Chaos engineering is the practice of injecting controlled failures into production or production-like environments to verify that resilience mechanisms behave as expected. The emphasis is on controlled. The goal is not to break things for fun, but to discover weaknesses before they surface during real incidents. Start small: kill a single instance and verify that the load balancer routes around it. Introduce latency on a dependency and confirm that circuit breakers trip. Simulate a network partition between availability zones and check that the system degrades gracefully.

Game days are scheduled exercises where a team simulates a major incident, including detection, communication, mitigation, and recovery. Game days test not just the technical systems but the organisational response. Does the on-call engineer know how to find the relevant runbook? Does the escalation path work? Can the team communicate effectively under pressure? Game days often reveal gaps that no amount of automated testing will find.

Fault injection in CI/CD brings resilience testing earlier in the development cycle. Some organisations run a baseline set of failure scenarios as part of their deployment pipeline: if the new version does not handle a killed dependency gracefully, it does not deploy. This is harder to set up than production chaos testing but catches regressions before they reach users.

The common thread is that resilience must be validated empirically. Reviewing architecture diagrams is necessary but not sufficient. Systems are full of assumptions that only surface under failure: retries that were never tested, failover paths that depend on stale configuration, timeouts that were set once and never revisited.

Organisational readiness

Technical resilience is necessary but not sufficient. Organisations need the structures and habits that allow humans to respond effectively when automated systems are not enough.

On-call readiness means that the person who gets paged at 3 AM can actually diagnose and mitigate the problem. This requires up-to-date runbooks, access to the right dashboards, and sufficient context about recent changes. It also requires sustainable on-call rotations. Fatigued engineers make worse decisions, and chronically stressful on-call loads drive attrition, which degrades institutional knowledge, which makes future incidents worse. This is a reinforcing loop that many organisations underestimate.

Runbooks should be living documents, tested regularly and updated after every incident that reveals a gap. A runbook that was last updated eighteen months ago is a liability. The best runbooks are short, action-oriented, and linked directly from alerting tools so that the responder does not have to search for them.

Incident retrospectives (often called post-mortems, though the less morbid terminology is gaining ground) are the mechanism for turning incidents into lasting improvements. Effective retrospectives focus on systemic causes rather than individual blame. They produce concrete action items with owners and deadlines. Critically, those action items must actually be completed. A backlog full of unaddressed post-mortem actions is a leading indicator of future incidents.

Resilience as a shared responsibility means that it is not delegated entirely to a platform or SRE team. Product engineers make design decisions that affect blast radius, degradation modes, and retry behaviour every day. If resilience is only considered during architecture reviews, it will be inconsistent across the system. Teams that treat resilience as a first-class property of every feature, discussed during design, tested during development, and validated in production, build systems that hold up under real-world conditions.

Evaluation checklist

When assessing whether a system meets its resilience requirements, the following questions provide a practical starting point.

  • Are failure modes documented?For each component and dependency, what happens when it is slow, unavailable, or returning errors? If nobody can answer this question, the system's behaviour under failure is undefined, which means it is whatever the code happens to do.
  • Are degradation modes explicit?When a non-critical dependency fails, does the system degrade gracefully or fail entirely? Are fallback paths tested?
  • Are SLOs defined and measured?Do they cover not just steady-state performance but also recovery time and degradation behaviour? Are error budgets used to prioritise reliability work?
  • Has resilience been tested empirically?Have chaos experiments or game days been conducted recently? Did they reveal any surprises? Were the resulting action items completed?
  • Are runbooks current and accessible?Can the on-call engineer find and follow them within minutes of being paged?
  • Is blast radius bounded?Does the architecture use bulkheads, circuit breakers, or other isolation mechanisms to prevent single-component failures from cascading?
  • Are timeouts and retries configured with care?Are deadlines propagated through the call chain? Are retries bounded, backed off, and idempotent?

None of these questions is difficult to ask. The difficulty is in answering them honestly and acting on the answers. Resilience is not a feature you ship once. It is a property you maintain through continuous investment in architecture, testing, and organisational discipline. Systems that treat it as a first-class design constraint, rather than something to bolt on after the throughput targets are met, are the ones that hold up when conditions deteriorate. And in production, conditions always deteriorate eventually.

Related notes

Back to topic • Back to Notes index