Failure Is the Default, Availability Is the Exception

This is not pessimism. It is physics. Networks drop packets. Disks develop bad sectors.
Application servers exhaust heap. Maintenance windows get extended. Third-party APIs
impose rate limits. Every enterprise platform I have built or operated in production has
experienced all of these — often on the same day, occasionally in the same hour.

The question at design time is not “what if this service is unavailable?” The question
is “what does this service do when the thing it depends on is unavailable — and how
quickly does that failure become visible to the right person?”

The IBM Maximo Toolkit — Failure as a Design Input

The operations automation toolkit I built at IBM for IBM Maximo on Azure had a specific
design constraint: every automated operation had to be safe to run in an unknown state.
You could not assume the application server was healthy before running a patch. You could
not assume the database was reachable before starting a deployment. The toolkit had to
discover the current state before taking any action — and handle every discovery result
gracefully, including the ones that should not exist.

The startup orchestration script does not start the application server. It checks
whether WebSphere Application Server is already running, confirms the DB2 connection is
healthy, verifies the MQ queue manager is available, and only then triggers the startup
sequence — with each step logged and each failure producing a human-readable diagnostic
rather than a stack trace.

Azure DR: RTO and RPO as Design Constraints

At T-Systems, the enterprise Azure DR architecture used ASR for VM replication to a
secondary region, Traffic Manager for DNS-level failover, and an Azure SQL geo-replica
in read-only mode at the secondary site — promoted to primary on failover. The key
discipline: failover procedures were documented and tested quarterly under realistic
conditions, not rehearsed once at project delivery and never touched again.

Untested DR is not DR. It is a plan that may or may not work when the people most
familiar with the system are not available, at a time when the business cannot afford
ambiguity.

The Five Patterns That Compose

These are not alternatives. They are layers applied to the same failure scenario:

• Retry with backoff and jitter. Transient faults — rate limits, brief
  network interruptions, pod restarts during rolling deployments — resolve within seconds.
  Retry three times with exponential intervals (1s, 2s, 4s) and ±20% jitter. Jitter
  prevents retry storms when many callers fail simultaneously and retry in lockstep.
• Circuit breaker. When retries are not enough, stop trying. After a
  threshold of failures in a time window, the circuit opens and subsequent calls fail fast
  without touching the downstream service. A 30-second recovery window, then a probe. The
  circuit state is a dashboard metric — it fires an alert before the service desk call.
• Timeouts. A slow service is worse than a failed one. Set explicit
  timeouts on every external call. When a timeout fires, execute a fallback — return cached
  data, a degraded response, or enqueue for async completion. Never leave a thread waiting
  indefinitely.
• Dead-letter queues. In messaging architectures, a message that cannot
  be processed is not discarded. It moves to the DLQ with the original payload and failure
  reason. A non-zero DLQ depth triggers an alert. The engineer sees the exact message that
  failed, the exact error, and can fix and replay without any data loss.
• Bulkheads. Resource quotas per service prevent one misbehaving
  component from consuming resources allocated to others. A memory leak in one service
  stays local. The rest of the platform continues operating.

← Back to Architecture Decisions