Every SRE knows the reliability basics: define SLOs, eliminate toil, blameless postmortems, error budgets. This framework is well-established and it works. But when you apply it to payment infrastructure — the systems that move money, authorize transactions, and maintain financial records — you discover that some of the standard assumptions don't hold, some of the standard tools are insufficient, and some failure modes have consequences that "availability went down for 5 minutes" doesn't capture.
Payment systems are where reliability engineering gets interesting.
Why Payments Are Different
The differences aren't arbitrary. They follow from what money is and what financial systems must guarantee.
Transactions must be exactly-once. In a conventional web service, a request that times out can be safely retried by the client. In a payment system, a charge that times out cannot be naively retried — if the original request succeeded but the response was lost, retrying charges the customer twice. Every retry mechanism in a payment system must account for idempotency, and idempotency requires every request to carry a unique identifier that allows the server to detect and deduplicate repeated requests.
# The idempotency key pattern
POST /charges
Idempotency-Key: a3f7d8e2-b4c1-4f2a-9e3d-1a2b3c4d5e6f
{
"amount": 4999,
"currency": "usd",
"customer": "cus_abc123"
}
# If this request times out and is retried with the same key,
# the server returns the original response — not a new charge.Partial failures have asymmetric consequences. In a web service, a partial failure (some requests fail, some succeed) is bad but recoverable. In a payment system, a partial failure might mean some authorizations succeeded but funds weren't captured, or some ledger entries were written but their compensating entries were not. The recovery path for partial financial failures requires reconciliation — a process of comparing records across systems to identify inconsistencies and correct them. This is engineering work that most non-financial systems never need.
Consistency requirements exceed what "eventual" provides. Eventual consistency is an acceptable model for many distributed systems — your social feed being slightly stale is a minor UX issue. For financial records, eventual consistency can mean a customer's balance shows funds that have already been spent, or a merchant sees a charge that will later be reversed. Payment systems require strong consistency guarantees, which imposes architectural constraints that eventual-consistency systems don't have.
Compliance is not optional. PCI DSS (Payment Card Industry Data Security Standard) mandates specific security controls for any system that touches cardholder data. These aren't engineering preferences — they're contractual and legal obligations. SRE work in payment infrastructure happens within a compliance framework that constrains tooling choices, access patterns, and operational procedures.
SLOs for Payment Systems
Standard SLO definitions cover availability and latency. For payment systems, you need additional dimensions.
Authorization success rate. The percentage of payment authorization attempts that succeed, excluding declines from the card network (which are not infrastructure failures). A drop in authorization success rate — your system failing to process charges that should succeed — is a revenue impact that maps to real dollars per minute. This is a higher-urgency signal than generic availability.
Transaction consistency rate. The percentage of completed transactions where all financial records are internally consistent — the charge amount, the ledger debit, the merchant payout record all agree. Inconsistencies here require manual reconciliation and potentially refunds, and they create compliance risk. This SLO is harder to measure but critical.
Settlement timing SLA. Payment processors have strict deadlines for submitting captured charges for settlement. Missing settlement windows means delayed merchant payouts and potential financial penalties. This is an SLA with external consequences, not just internal targets.
Dispute resolution latency. When a cardholder disputes a charge, there are regulatory deadlines for responding with evidence. The reliability of the dispute management system — including all the tooling that gathers transaction evidence — affects the organization's ability to win disputes it should win.
Failure Modes Unique to Payment Infrastructure
Split-brain in the authorization path. If your authorization system has a network partition during a transaction, you can end up with one side having authorized the charge and the other side having no record of it. Recovery requires cross-system reconciliation, not just automatic failover. This is why payment systems prefer to reject a transaction during a partition (conservative) rather than approve it (optimistic).
Clock skew and idempotency. Idempotency windows — the period during which a duplicate request with the same idempotency key is deduplicated — depend on accurate timestamps. Clock skew between distributed nodes can cause requests that appear to fall within the idempotency window to be processed as duplicates or, worse, processed twice if the window is incorrectly computed.
Cascading failures through payment networks. Your payment processor, the card networks (Visa, Mastercard), card-issuing banks, and fraud detection systems are all in the authorization path. Degradation at any point can cascade into your system in unexpected ways. An upstream timeout that your code doesn't handle correctly becomes a customer-facing failure. Defensive coding in the payment path — with explicit timeout handling, fallback paths, and circuit breakers — is essential.
The soft decline problem. Card networks return a large number of distinct response codes, including "soft declines" — temporary rejections that may succeed if retried. The retry logic for soft declines needs to be implemented carefully: which decline codes warrant a retry, at what interval, how many times, and whether retries should use a different processing path. Getting this wrong costs authorization revenue on one side (not retrying codes that would succeed) and causes customer problems on the other (retrying declined cards that the customer expects to be declined).
Observability in Payment Systems
Payment observability has additional constraints: cardholder data (card numbers, billing details) cannot appear in logs or traces. This sounds obvious but requires active work — log scrubbing, field redaction at the SDK level, and audit processes to catch PII that surfaces in unexpected places.
Beyond compliance constraints, payment observability needs to surface financial metrics alongside infrastructure metrics. A monitoring dashboard that shows latency and error rates without showing authorization success rate and captured volume is incomplete for a payments context.
# Payment-specific metrics alongside infrastructure metrics
class PaymentMetricsEmitter:
def record_authorization_attempt(
self,
result: str, # "approved", "declined_insufficient_funds",
# "declined_network_error", "timed_out", "error"
processor: str,
card_network: str,
amount_cents: int,
latency_ms: float
):
tags = {
"result": result,
"processor": processor,
"network": card_network,
"amount_bucket": self._bucket_amount(amount_cents)
}
self.metrics.increment("payments.authorization.attempts", tags=tags)
self.metrics.histogram("payments.authorization.latency_ms",
latency_ms, tags=tags)
if result == "approved":
self.metrics.increment("payments.authorization.approved", tags=tags)
self.metrics.gauge("payments.authorization.volume_cents",
amount_cents, tags=tags)
elif result.startswith("declined"):
self.metrics.increment("payments.authorization.declined", tags=tags)
elif result in ("timed_out", "error"):
# Infrastructure failure — higher urgency alert
self.metrics.increment("payments.authorization.infra_failure", tags=tags)Disaster Recovery for Financial Systems
The disaster recovery requirements for payment systems go beyond "restore from backup."
Point-in-time recovery. If a database corruption event occurs, you need to restore to a specific moment in time — before the corruption, but after the last clean state. This requires continuous transaction log backup (not just daily snapshots), tested recovery procedures that can target a specific timestamp, and verification processes that confirm financial records are internally consistent after recovery.
The reconciliation process. After any recovery event, you need to reconcile your internal records against external records from payment processors and banks. The reconciliation identifies gaps — transactions that appeared in one system but not the other — and resolves them through the appropriate financial process (refunds, captures, or void records). This process needs to be automated, not manual, because the window for reconciliation after recovery is short.
Active-active vs. active-passive. Active-active architectures (where multiple regions process transactions simultaneously) provide the best availability but require careful handling of global state — your ledger cannot have two independent writers without a coordination mechanism. Most payment systems use active-passive with fast failover for the critical transaction path, and accept the brief unavailability of failover rather than the consistency risks of active-active.
The AI Angle: Fraud and Reliability
AI is increasingly used in payment systems for fraud detection, and fraud detection has reliability implications. A model that incorrectly declines legitimate transactions (false positives) reduces authorization success rate as surely as an infrastructure failure. A model that's slow to score transactions adds latency to the authorization path.
The reliability of fraud models — their accuracy, their latency, and their failure behavior — is an SRE concern, not just a data science concern. Fraud model latency should be in your authorization latency SLO. Model fallback behavior (what happens when the model is unavailable?) should be defined and tested. Model degradation (when the model is slow or producing anomalous scores) should trigger alerts.
The convergence of ML reliability and infrastructure reliability is where the interesting SRE work is in payment systems. The teams doing it well are applying SRE disciplines — SLOs, error budgets, on-call rotations, blameless postmortems — to ML systems as first-class production services.
*Zak Hassan is a Staff SRE specializing in distributed systems reliability and AI-powered infrastructure automation. Find him at zakhassan.com or on LinkedIn.*
Topic Paths