Architecture Resilience: Best Practices & Working Practices on AWS

Resiliency Series — Bridge Post. This post synthesises Parts 1–5 into a practitioner’s playbook: the AWS architecture patterns that hold under pressure, and the team working practices that make those patterns survive contact with real organisations. It also maps everything to the AWS Well-Architected Reliability Pillar so it doubles as a reference for architecture reviews and AWS Community talks.

A colleague’s question that made me write this post

Last week, someone I’ve worked with for years caught me after an architecture review and asked the question I’ve been asked in some form for most of my career:

“What are the real architecture resilience best practices — not the textbook stuff, the ones that actually hold up in production?”

He wasn’t asking about circuit breakers. He’d read the papers. He wanted to know what I’d actually reach for at 2 a.m. when something was burning, and — more importantly — what I’d done before 2 a.m. to make the fire smaller.

After seventeen years building infrastructure across financial services, retail, and platform teams, and five posts in this series, that question deserves a direct answer.

Here it is.

Where we are in this series

The Two-Layer Resilience Model

Here is the thing most teams miss: resilient systems require work at two distinct layers, and they are not interchangeable.

graph TD

    subgraph ARCH["Architecture Layer (Design-Time)"]

        A1[Multi-AZ Deployment]

        A2[Auto Scaling]

        A3[Route 53 Health-Check Failover]

        A4[Circuit Breakers & Bulkheads]

        A5[Cell-Based Isolation]

    end



    subgraph WORK["Working Practices Layer (Runtime / Team)"]

        W1[Game Days & Fire Drills]

        W2[Runbooks as Living Docs]

        W3[Blameless Post-Mortems]

        W4[SLO / SLI Discipline]

        W5[On-Call as Feedback Loop]

    end



    ARCH -->|"patterns only hold if teams know how to operate them"| WORK

    WORK -->|"feedback drives architecture decisions"| ARCH

Architecture patterns are what you build. They determine blast radius, failover speed, and theoretical maximum availability.

Working practices are how your team operates. They determine whether those patterns ever get exercised before an outage, whether the runbook exists and is accurate, and whether the post-mortem actually changes anything.

Most teams nail one layer and neglect the other. I’ve seen beautiful multi-AZ architectures with no game-day programme. I’ve seen mature SRE cultures running on architectures with single points of failure nobody had bothered to remove. Both combinations fail — just differently.

Architecture Best Practices — The AWS Reliability Foundation

Five core patterns, mapped to the AWS Well-Architected Reliability Pillar

The AWS Well-Architected Reliability Pillar organises its guidance into five areas. I’ll use that structure because it lines up cleanly with what I’ve actually deployed, and it gives you a framework that AWS reviewers and your own architecture boards will recognise.

Pattern 1: Multi-AZ as the minimum viable floor

Every production workload I build today starts with at least three AZs. Not two. With two, a single AZ loss means 50% capacity. With three, it means 33% — which your Auto Scaling Group can typically absorb without customer impact.

The oft-repeated advice is correct: treat each AZ as an independent failure domain. Do not let your load balancer, database, and cache all sit in the same AZ, even if that’s cheaper. The December 2021 us-east-1 event cost organisations in the millions of dollars per hour. Businesses that had genuine multi-AZ — not just the checkbox, but actual traffic distribution and tested failover — rode it out. Those that had ticked the box but concentrated their database primary in a single AZ did not.

Pattern 2: Auto Scaling Groups with meaningful health checks

An ASG with only EC2 health checks is not resilient. It’s just self-healing hardware. Configure ELB health checks so the ASG replaces instances that are running but serving errors. Set a minimum capacity that reflects your actual load at the trough, not your optimistic assumption about low traffic.

Target Tracking Policies on CPU or custom metrics (request count per target is usually better) handle gradual load ramps well. For sudden spikes, add Scheduled Scaling anchored to your known traffic patterns, and keep the scheduled scale-up 15 minutes early — traffic doesn’t wait for your autoscaler to warm up.

Pattern 3: Route 53 health-check failover

Route 53 health checks combined with DNS failover routing are one of the most underused reliability levers on AWS. A health check can target an endpoint, and when it fails, Route 53 switches DNS within 60 seconds to a failover record — whether that’s another region, a CloudFront distribution serving a static fallback, or an on-premise endpoint.

Keep your TTL at 60 seconds or lower on critical records. A five-minute TTL during a regional failover means five minutes of clients hitting a dead endpoint after the DNS switch, which is five minutes you can’t afford.

Pattern 4: Circuit breakers — no native AWS primitive, multiple implementation paths

AWS does not give you a circuit breaker as a managed service. You build it. Options I’ve used in production:

• API Gateway + Lambda authoriser: reject requests early when downstream health degrades
• AppConfig feature flags: turn off non-essential call paths at runtime without a deployment
• Application-level circuit breaker libraries (Resilience4j, AWS SDK retry configuration): essential for service-to-service calls inside a VPC

The pattern from Part 3 still applies: open on error-rate threshold, half-open on probe, close on sustained success. The implementation details change per language and per call path.

Pattern 5: Health-check-driven traffic shifting

Combine Route 53 weighted routing with CodeDeploy or ECS Blue/Green deployments. Start a new deployment at 10% traffic, watch your synthetic monitors and error rate, then shift the remaining 90%. If the error rate climbs, automated rollback triggers before 90% of your users notice.

This is not new advice. What is consistently missing in teams I review is the automated rollback trigger. Manual rollback at 2 a.m. is not a rollback plan — it is a prayer.

AWS Platform Toolkit — What to Actually Use

flowchart LR

    R53["Route 53\nHealth Checks + Failover\nWeighted / Latency Routing"]

    ALB["Application\nLoad Balancer\nPath routing\nSticky sessions"]

    ASG["Auto Scaling Group\nTarget Tracking\nLifecycle Hooks"]

    APP["Application\nTier\n(ECS / EKS / EC2)"]

    RDS["RDS Multi-AZ\nor Aurora\nGlobal Database"]

    CW["CloudWatch\nComposite Alarms\nSynthetics Canaries"]

    FIS["AWS FIS\nChaos Experiments"]

    RH["Resilience Hub\nRTO/RPO Assessments"]

    SSM["Systems Manager\nRunbooks\nOpsCenter"]



    R53 --> ALB --> ASG --> APP --> RDS

    CW -.->|"alarm → SNS → auto-remediation"| SSM

    FIS -.->|"injects failures"| APP

    FIS -.->|"injects failures"| RDS

    RH -.->|"assesses architecture"| APP

AWS Fault Injection Simulator (FIS) is the most overlooked tool in this stack. It runs structured chaos experiments — CPU stress, network latency injection, EC2 instance termination, RDS failover triggers — against your live or staging environment. If you are not running FIS experiments against your production architecture at least quarterly, you are guessing about your resilience, not measuring it.

AWS Resilience Hub gives you an automated RTO/RPO assessment against your deployed CloudFormation or Terraform-defined architecture. It flags gaps between your architecture’s actual recovery capability and the targets you’ve declared. I use it as a gate in our architecture review process: new workloads above a certain criticality level must pass a Resilience Hub assessment before going to production.

CloudWatch Composite Alarms let you combine multiple signals — high CPU and elevated error rate and decreased healthy host count — into a single alarm that represents “something is actually wrong” rather than “one metric crossed a threshold.” The false positive rate drops significantly. Use them to trigger your automated runbooks in Systems Manager OpsCenter.

Systems Manager Runbooks (the Automation documents) are where you encode your institutional knowledge. Not in a Confluence page that nobody will find at 3 a.m. — in executable, version-controlled Automation documents that run against your actual infrastructure. Restart an ECS service. Fail over an RDS cluster. Drain and replace an ASG instance. These should be tested monthly during game days, not discovered during incidents.

Working Practices — How Resilient Teams Actually Operate

Architecture buys you the capacity for resilience. Working practices are what you spend that capacity on.

Game days and fire drills

“Practice failure before it practices you.”

A game day is a structured exercise where you deliberately inject failures — a service going down, an AZ becoming unreachable, a key dependency returning errors — and watch how your team responds. Not your monitoring. Not your runbooks. Your people.

The discovery from almost every game day I’ve run: the architecture held. The processes did not. The person on call did not know the runbook existed. The runbook referenced an IAM role that had been renamed six months ago. The escalation path went to someone who was on holiday.

Run game days quarterly. Make them realistic. Debrief honestly.

Runbooks as living documentation

A runbook has one job: get a competent engineer to the right action in under five minutes at 3 a.m., having just been woken from a dead sleep.

That means: short, numbered steps. Specific commands, not general guidance. Links to dashboards, not descriptions of where dashboards live. A “what does success look like” check at the end of each step.

If your runbooks are longer than two pages and have not been touched in six months, they are shelfware. Automate the common path in Systems Manager, and use the human-readable runbook only for the judgment calls that cannot be automated.

Blameless post-mortems

Culture eats architecture for breakfast. I have seen architectures with redundant redundancy where incidents dragged on for hours because engineers were afraid to try fixes without approval, because the last engineer who tried something in production and broke it got publicly dressed down.

A blameless post-mortem has a simple rule: the system failed, not the person. Your analysis finds the conditions that made a human error possible and made it consequential. Then you fix the conditions.

The five-whys analysis in your post-mortem should surface: a missing alert, an absent circuit breaker, a test environment that did not match production, a deployment without a rollback plan. Those become tickets. The tickets close before the next architecture review.

SLO / SLI discipline — the hierarchy you cannot skip

Teams that skip SLOs and jump straight to SLAs always fight fires. Without an SLO, you don’t know whether your reliability is trending toward a breach until you’re already there. With an SLO and an error budget, you know weeks in advance. You can throttle risky deployments when the budget is low and move fast when it is high.

Instrument your SLIs in CloudWatch, set SLO-based alarms, and make error budget status a standing item in your sprint reviews.

Architecture review cadence

Run your internal architecture review plus an AWS Well-Architected Tool review for every workload that crosses a criticality threshold. The Well-Architected Tool forces you to answer questions you would otherwise not ask — specifically the “what happens if this component fails” questions that feel obvious but consistently get skipped under deadline pressure.

The cadence I use: Well-Architected Tool review at initial design, at first production deployment, and annually thereafter. Internal architecture review at each major change.

On-call rotation as a resilience feedback loop

On-call is not a punishment. It is the most accurate signal you have about the real state of your system. The person who gets paged at 2 a.m. for a noisy alert knows something that no architecture review will surface: that alert is miscalibrated and has been waking people up for months and nobody fixed it because it always resolved itself.

Make on-call feedback a first-class input to your engineering roadmap. Count the number of pages per sprint. Set a target. Treat a high page rate as technical debt with a concrete cost (engineer sleep, morale, attrition) and pay it down.

The Resilience Flywheel

Resilience is not a project with a completion date. It is a cycle.

flowchart LR

    D["Design\n---\nWell-Architected reviews\nRTO/RPO definition\nBlast radius mapping"]

    B["Build\n---\nMulti-AZ patterns\nASG + health checks\nRoute 53 failover"]

    O["Observe\n---\nCloudWatch Composite Alarms\nSynthetics Canaries\nSLI dashboards"]

    T["Test\n---\nAWS FIS experiments\nGame days\nRunbook drills"]

    I["Improve\n---\nBlameless post-mortems\nError budget reviews\nResilience Hub re-assessment"]



    D --> B --> O --> T --> I --> D

Each turn of the flywheel tightens the loop. The first FIS experiment surfaces three gaps you did not know about. You fix them. The next experiment finds one. The post-mortems get shorter. The game days get boring — which is exactly what you want.

The teams I’ve worked with that have this flywheel running have MTTR measured in minutes, not hours. The teams that treat resilience as a design-time activity and then move on have MTTR measured in war stories.

Resilience Maturity Model

Where is your team today, and where should it be?

graph LR

    L1["Level 1\nReactive\n---\nNo monitoring\nOutages discovered\nby customers\nFire-fighting culture"]

    L2["Level 2\nAware\n---\nBasic monitoring\nPost-mortems exist\nbut rarely acted on\nManual runbooks"]

    L3["Level 3\nProactive\n---\nSLOs defined\nGame days run\nRunbooks automated\nCircuit breakers in place"]

    L4["Level 4\nPredictive\n---\nError budgets tracked\nFIS experiments quarterly\nResilience Hub gates\nOn-call feedback loop"]

    L5["Level 5\nChaos-Ready\n---\nCell-based isolation\nContinuous chaos\nin production\nAutomated remediation\nfor known failure modes"]



    L1 --> L2 --> L3 --> L4 --> L5



    style L1 fill:#fecaca,stroke:#ef4444,color:#7f1d1d

    style L2 fill:#fed7aa,stroke:#f97316,color:#7c2d12

    style L3 fill:#fef08a,stroke:#eab308,color:#713f12

    style L4 fill:#bbf7d0,stroke:#22c55e,color:#14532d

    style L5 fill:#bfdbfe,stroke:#3b82f6,color:#1e3a8a

In my experience, the majority of enterprise engineering teams sit at Level 2 to Level 3. They have monitoring. They write post-mortems. They just don’t act on them fast enough, and they haven’t tested their runbooks in production conditions.

Moving from Level 3 to Level 4 is the highest-leverage jump: it’s where game days, FIS experiments, and error budget tracking compound on each other. The jump from Level 4 to Level 5 — cell-based architecture and continuous chaos in production — is the investment Part 4 and Part 5 of this series cover in depth.

RTO / RPO Decision Matrix

Before you decide which architecture pattern to implement, anchor the conversation here. Every pattern has a cost. Match the cost to the business requirement, not the engineering preference.

The most common mistake I see: teams with a warm-standby architecture and an active-active SLA. When the failure happens, the gap between what the business thinks will happen and what the architecture can actually deliver is where the real outage cost lives.

If your RTO/RPO conversation with the business has not happened, make it happen before the next sprint. The architecture review is the wrong place to discover that the business expected five minutes but the team built for two hours.

Bringing It All Together

Five posts. One journey.

• Part 1 — Embracing Failure: Failure is not a possibility — it is a certainty. Design for it from day one. Werner Vogels said it; seventeen years of production have confirmed it.

• Part 2 — Anatomy of Failures: 70% of outages are application-level. Most organisations spend 80% of their resilience budget on infrastructure. The gap between those two numbers is where most incidents live. Detect in under 30 seconds.

• Part 3 — Resilience Patterns: Circuit breakers stop cascades. Bulkheads contain noisy neighbours. Timeouts fail fast rather than hang forever. Retries need jitter or they create the thundering herd that finishes off whatever was already struggling. These patterns cover the application layer — roughly 70% of your failure surface.

• Part 4 — The Selection Matrix: Infrastructure failures — the remaining 30% — require infrastructure-level isolation. Cell-based architecture is the most complete answer: a cell per isolation boundary, blast radius contained to a slice of your user base. This post maps six isolation boundaries from Namespace to Multi-Platform and gives you the decision matrix to pick the right boundary for your actual requirements.

• Part 5 — Multi-Platform Cells: The only isolation boundary that survives losing an entire cloud provider. High complexity, high cost, and the right answer for a narrow set of workloads. If you need it, you know why. If you’re not sure, you probably don’t need it yet.

This post is the bridge: the practical playbook that connects the pattern catalog to how real teams operate — AWS tools, working practices, maturity levels, and the flywheel that keeps improving all of it.

The architecture resilience best practices that actually hold in production are not the ones in any single framework document. They are the ones your team has exercised, automated, and reflected on enough times that the 2 a.m. response is almost boring.

Read the full series