Multi-Region Reliability: Building Systems That Survive Regional Failures

*By Zak Hassan — Staff SRE | May 2026*

A cloud region going down is not a theoretical risk. AWS us-east-1 has had multi-hour outages. GCP us-central1 has taken down dependent services across the industry. When it happens, the systems that survive are not the ones with the best blog posts — they are the ones whose architects made hard decisions about data consistency, traffic routing, and failure boundaries long before any incident began. This post is about those decisions: how to think about multi-region architectures, where the traps are, and how to build and test the machinery before you actually need it.

Active-Active vs Active-Passive vs Active-Standby

These three terms get thrown around loosely, so let's be precise about what each one actually means operationally.

Active-active means every region accepts live writes and reads simultaneously. Traffic is distributed across regions, and each region is a full peer. Recovery from a regional failure is automatic and near-instant because other regions are already carrying production load. The cost: you must handle conflict resolution for writes that happened in different regions concurrently, and your data replication needs to move fast enough that reads in one region don't return stale data that was written in another millisecond ago.

Active-passive means one region handles all writes; the passive region is a warm replica that can be promoted. Failover is manual or semi-automated, typically taking minutes to tens of minutes. This is dramatically simpler from a data consistency standpoint — there's one writer, so there are no conflicts — but your RTO (recovery time objective) is measured in minutes, and during failover you're accepting downtime.

Active-standby sits between the two. The standby handles reads (or some subset of traffic), but writes still go to the primary. It's cheaper than full active-active because you don't need to solve multi-master writes, but it gives you faster failover than cold active-passive because the standby is already warm and serving traffic.

The decision framework boils down to three questions:

Decision Framework: Choosing a Multi-Region Model
--------------------------------------------------

Q1: What is your RTO requirement?
  - < 30 seconds → Active-Active (only option with automatic failover)
  - 1–10 minutes → Active-Standby with automated promotion
  - 10+ minutes  → Active-Passive is probably fine

Q2: Can your data model tolerate eventual consistency?
  - Yes (social feeds, analytics, event logs) → Active-Active is viable
  - No (financial ledgers, inventory, booking seats) → Active-Passive or
    Active-Standby with strong consistency replication (e.g., CockroachDB,
    Spanner) — and accept the latency cost

Q3: What's your budget?
  - Active-Active: ~2–3x the infra cost of a single region (full stack everywhere)
  - Active-Standby: ~1.5x (right-sized standby, not full peer capacity)
  - Active-Passive: ~1.2–1.3x (replica storage + minimal compute)

Most SaaS companies at mid-scale should be in active-standby. Full active-active is appropriate for consumer-facing products with global audiences and strict sub-second latency requirements, or for any service where minutes of downtime translate to millions in direct revenue loss.

The Data Consistency Problem in Active-Active

The biggest mistake teams make when going active-active is treating it as a replication problem rather than a consistency problem. You cannot simply point MySQL replication at two masters and call it done. Concurrent writes to the same row in two different regions will create conflicts, and MySQL's default behavior — last write wins based on timestamp — will silently drop data.

The actual options for conflict resolution in active-active are:

1. Last write wins (LWW) — simple, lossy. Fine for caches and session data. Never use for anything a user cares about.
2. CRDTs (Conflict-free Replicated Data Types) — data structures designed so any merge order produces the same result. Counters and sets are easy; arbitrary records are not.
3. Application-level conflict detection — write a version vector or logical clock into every record, detect conflicts at read time, and surface them to the application to resolve. Used by systems like DynamoDB's conditional writes and Cassandra's lightweight transactions.
4. Region ownership — partition your data such that each record is "owned" by one region. User 1's writes always go to us-east-1; user 2's always go to eu-west-1. This avoids multi-master conflicts entirely but adds routing complexity and hurts latency if a user travels.

For most teams, region ownership is the pragmatic path to active-active without the full consistency nightmare. CockroachDB and Google Spanner implement serializable distributed transactions using hybrid logical clocks — they're the right tools if you need genuine multi-master writes with strong consistency, but they come with higher operational overhead and non-trivial latency on cross-region transactions.

Global Load Balancing

Two mechanisms dominate here: DNS-based routing (AWS Route 53, Cloudflare) and anycast-based routing (Cloudflare's network, GCP Premium Tier).

Route 53 latency routing routes each user to the region with the lowest measured latency. It does not perform health checks by default — it just routes based on AWS's latency tables. If a region goes down, Route 53 will keep sending traffic there unless you combine latency routing with health checks.

Here's a Terraform configuration for a Route 53 latency policy with health-check failover:

resource "aws_route53_health_check" "us_east_1" {
  fqdn              = "api-us-east-1.internal.example.com"
  port              = 443
  type              = "HTTPS"
  resource_path     = "/healthz"
  failure_threshold = 3
  request_interval  = 10

  tags = {
    Name   = "api-us-east-1"
    Region = "us-east-1"
  }
}

resource "aws_route53_health_check" "eu_west_1" {
  fqdn              = "api-eu-west-1.internal.example.com"
  port              = 443
  type              = "HTTPS"
  resource_path     = "/healthz"
  failure_threshold = 3
  request_interval  = 10

  tags = {
    Name   = "api-eu-west-1"
    Region = "eu-west-1"
  }
}

resource "aws_route53_record" "api_us_east_1" {
  zone_id        = var.hosted_zone_id
  name           = "api.example.com"
  type           = "A"
  set_identifier = "us-east-1"

  latency_routing_policy {
    region = "us-east-1"
  }

  health_check_id = aws_route53_health_check.us_east_1.id
  ttl             = 30

  records = [aws_eip.us_east_1_nlb.public_ip]
}

resource "aws_route53_record" "api_eu_west_1" {
  zone_id        = var.hosted_zone_id
  name           = "api.example.com"
  type           = "A"
  set_identifier = "eu-west-1"

  latency_routing_policy {
    region = "eu-west-1"
  }

  health_check_id = aws_route53_health_check.eu_west_1.id
  ttl             = 30

  records = [aws_eip.eu_west_1_nlb.public_ip]
}

The critical configuration detail: set your TTL to 30–60 seconds. A 300-second TTL means five minutes of clients resolving to a dead region even after Route 53 has removed it. The trade-off is increased DNS query volume, which is negligible at most scales.

Cloudflare's Anycast routes users to the nearest Cloudflare point of presence at the network layer, not the DNS layer. This is fundamentally faster for failover — route withdrawals propagate in seconds, not the minutes it takes DNS TTLs to expire across resolvers. If you're running behind Cloudflare, you should be using their Health Checks and Load Balancing product in addition to (or instead of) Route 53 for the routing layer.

Stateless Service Replication

The question of what belongs in every region and what can be centralized is one of the most consequential architectural decisions in a multi-region system.

Must be in every region:

• API servers and application logic (stateless, replicate freely)
• CDN edge / static asset serving
• Read replicas of the database (for latency and availability)
• In-region caches (Redis clusters) — do not route cache reads cross-region
• Service discovery (Consul, AWS Cloud Map) — must be local to survive a partition

Can be centralized (with caveats):

• Log aggregation (acceptable to lose logs during a partition; fix after recovery)
• Long-term metrics storage (Thanos or Cortex can aggregate from multi-region Prometheus)
• Admin tooling and internal dashboards
• Write-primary database if you're in active-passive (by definition)

The anti-pattern to avoid: centralizing anything that is in the critical path of serving user traffic. Session validation, feature flags, rate limiting state — if any of these require a cross-region network call, you've introduced a dependency that a regional partition will sever, and your "multi-region" system will fail in a single-region way.

Regional Failover Testing

The only reliable way to know your failover works is to run it. Chaos engineering at the regional level means intentionally withdrawing traffic from a region and measuring what happens.

A practical traffic-shifting script using AWS CLI for a staged regional failover test:

import boto3
import time
import logging
from dataclasses import dataclass
from typing import Optional

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
logger = logging.getLogger(__name__)

@dataclass
class RegionHealthStatus:
    region: str
    healthy: bool
    latency_p99_ms: Optional[float]
    error_rate: Optional[float]

class RegionalFailoverController:
    def __init__(self, hosted_zone_id: str, record_name: str):
        self.route53 = boto3.client("route53")
        self.cloudwatch = boto3.client("cloudwatch", region_name="us-east-1")
        self.hosted_zone_id = hosted_zone_id
        self.record_name = record_name

    def get_region_health(self, region: str, namespace: str = "MyApp") -> RegionHealthStatus:
        """Pull error rate and p99 latency from CloudWatch for a region."""
        end = time.time()
        start = end - 300  # last 5 minutes

        def get_metric(metric_name: str, stat: str) -> Optional[float]:
            resp = self.cloudwatch.get_metric_statistics(
                Namespace=namespace,
                MetricName=metric_name,
                Dimensions=[{"Name": "Region", "Value": region}],
                StartTime=start,
                EndTime=end,
                Period=300,
                Statistics=[stat],
            )
            points = resp.get("Datapoints", [])
            return points[0][stat] if points else None

        error_rate = get_metric("ErrorRate", "Average")
        latency_p99 = get_metric("LatencyP99", "Average")
        healthy = (error_rate or 0) < 0.05 and (latency_p99 or 0) < 2000

        return RegionHealthStatus(
            region=region,
            healthy=healthy,
            latency_p99_ms=latency_p99,
            error_rate=error_rate,
        )

    def disable_region_in_dns(self, region: str, record_set_id: str) -> None:
        """Mark a latency record as unhealthy to drain traffic from a region."""
        logger.info(f"Disabling region {region} in Route 53 (record {record_set_id})")
        self.route53.change_resource_record_sets(
            HostedZoneId=self.hosted_zone_id,
            ChangeBatch={
                "Changes": [{
                    "Action": "UPSERT",
                    "ResourceRecordSet": {
                        "Name": self.record_name,
                        "Type": "A",
                        "SetIdentifier": region,
                        "LatencyRoutingPolicy": {"Region": region},
                        "HealthCheckId": record_set_id,
                        "TTL": 30,
                        "ResourceRecords": [{"Value": "192.0.2.1"}],  # blackhole IP
                    }
                }]
            }
        )

    def run_failover_test(self, target_region: str, health_check_id: str,
                          observe_seconds: int = 120) -> bool:
        """
        Drain traffic from target_region, observe for observe_seconds,
        then restore. Returns True if remaining regions stayed healthy.
        """
        logger.info(f"Starting failover test: draining {target_region}")
        self.disable_region_in_dns(target_region, health_check_id)

        logger.info(f"Observing for {observe_seconds}s — check your dashboards now")
        time.sleep(observe_seconds)

        # Check that surviving regions are still healthy
        surviving_regions = ["us-east-1", "eu-west-1"]  # adjust as needed
        surviving_regions = [r for r in surviving_regions if r != target_region]
        all_healthy = True

        for region in surviving_regions:
            status = self.get_region_health(region)
            logger.info(
                f"  {region}: healthy={status.healthy}, "
                f"error_rate={status.error_rate:.3f}, "
                f"p99={status.latency_p99_ms:.0f}ms"
            )
            if not status.healthy:
                logger.error(f"  FAIL: {region} is unhealthy during failover test")
                all_healthy = False

        logger.info(f"Restoring {target_region} to DNS")
        # Restore by re-pointing to real IP — omitted for brevity
        return all_healthy

Run this monthly, not annually. Each test teaches you something: which upstream limits you didn't account for, whether your auto-scaling groups in the surviving region can absorb the extra load, whether the on-call engineers know the playbook.

Cross-Region Observability

A Grafana dashboard scoped to us-east-1 will not tell you that eu-west-1 is silently degraded. Cross-region observability requires a deliberate aggregation layer.

The pattern that works at scale: deploy Prometheus in each region with a 15-day local retention. Run a Thanos sidecar alongside each Prometheus instance. All sidecar data flows into a central Thanos Query layer (or Cortex/Mimir if you need longer retention). The Grafana instance queries the global Thanos Query endpoint, which fans out to all regions and merges the results.

Key metrics that must appear on a global dashboard:

• Error rate by region (side-by-side, not averaged)
• P99 latency by region
• Database replication lag between primary and each replica region
• DNS health check state per region (pull via Route 53 API or CloudWatch)
• Active connection count per region (to detect lopsided traffic after DNS changes)

The replication lag metric is especially important in active-passive and active-standby setups. If your replica is 45 seconds behind and you fail over, you've just accepted a 45-second data loss. That needs to be visible before it's a crisis, not discovered during one.

The Split-Brain Problem

A network partition between regions is the hardest failure mode in distributed systems. Both regions are running, both are healthy from their own perspective, but they cannot communicate with each other. Without coordination, both will continue accepting writes, diverging silently.

The defense is fencing: a mechanism that prevents a partitioned node from accepting writes. In practice this means one of two things:

Quorum-based fencing — if you have three regions, a region that cannot reach a quorum of the others stops accepting writes. This is how etcd, ZooKeeper, and most Raft-based systems work. It requires an odd number of members (typically 3 or 5) and forces you to accept that the minority partition becomes read-only until the partition heals.

External arbiter fencing — a lightweight arbiter service (often hosted in a third region or an availability zone not shared by either primary region) acts as a tiebreaker. Each region checks in with the arbiter; if it can't reach the arbiter AND can't reach the other region, it demotes itself to read-only. This is how AWS RDS Multi-AZ and Aurora Global Database handle it.

To detect a split-brain condition in a bespoke system, instrument your inter-region heartbeat:

import time
import requests
from prometheus_client import Gauge, start_http_server

INTER_REGION_REACHABLE = Gauge(
    "inter_region_reachable",
    "Whether this region can reach a peer region (1=yes, 0=no)",
    ["peer_region"]
)

PEER_ENDPOINTS = {
    "us-east-1": "https://api-us-east-1.internal.example.com/healthz",
    "eu-west-1": "https://api-eu-west-1.internal.example.com/healthz",
}

CURRENT_REGION = "ap-southeast-1"  # set via env var in practice

def probe_peers():
    while True:
        for region, url in PEER_ENDPOINTS.items():
            if region == CURRENT_REGION:
                continue
            try:
                resp = requests.get(url, timeout=3)
                reachable = 1 if resp.status_code == 200 else 0
            except Exception:
                reachable = 0
            INTER_REGION_REACHABLE.labels(peer_region=region).set(reachable)
        time.sleep(10)

if __name__ == "__main__":
    start_http_server(9100)
    probe_peers()

Alert when inter_region_reachable drops to 0 for more than 30 seconds. At that point your blog post should trigger: verify the partition is real (not just the probe host), demote the minority region to read-only if you have quorum logic, page an engineer, and begin preparing for a controlled recovery. Do not wait for users to report problems.

Closing Thoughts

Multi-region reliability is not a feature you ship once — it is an ongoing operational discipline. The architecture decisions (active-active vs. passive, conflict resolution strategy, DNS TTL, quorum topology) are made once but constrain everything that follows. The testing (failover drills, chaos experiments, traffic-shift exercises) must happen continuously or the architecture rots as the system changes around it. And the observability (global dashboards, replication lag metrics, inter-region heartbeats) must be in place before an incident, not built during one.

The engineers who sleep well during regional outages are not the ones who wrote the best incident blog posts. They're the ones who already ran the test six weeks ago and know exactly what their system does when a region goes away.

*Zak Hassan is a Staff SRE specializing in distributed systems reliability, multi-region architecture, and large-scale incident management. Find him at zakhassan.com or on LinkedIn.*