choice — one that trades the complexity of exactly-once semantics for the simplicity and
reliability of durable delivery. The contract is clear: messages may arrive more than once.
Your consumer must be ready for that.
The Failure Sequence You Will See in Production
A payment instruction arrives. The IIB message flow processes it — validates the format,
enriches the payload from the reference database, and writes the resulting credit transfer
record. The write completes. Before the acknowledgement can be sent back to the queue
manager, the application server is taken offline for an emergency OS patch.
IBM MQ sees no acknowledgement. After the syncpoint timeout, the message is redelivered
to the queue. A new IIB flow instance picks it up and begins processing. It validates the
format (passes), enriches the payload (passes), and attempts to write the credit transfer
record — which now conflicts with the record already created by the previous instance.
Without an idempotency mechanism, this is a duplicate. With one, it is a no-op.
The failure sequence is not hypothetical. Node failures, rolling deployments, and
network interruptions produce this scenario routinely in long-running enterprise systems.
The question is not whether you will see it. It is whether your system handles it
correctly when you do.
The Pattern
Every message consumer checks a processed_events table for the producer’s correlation ID before processing. If found, the consumer acknowledges and exits. If not found, the consumer processes the event and writes the correlation ID in the same database transaction as the business state change. The correlation ID is the producer’s business identifier — not the broker’s message ID.
The implementation is deliberately minimal. The table stores the correlation ID, the
event type, and the timestamp of processing. It does not need to be large or complex —
it needs to be transactionally consistent with the data it protects.
-- processed_events table: minimal and correct CREATE TABLE processed_events ( correlation_id VARCHAR(36) PRIMARY KEY, event_type VARCHAR(128) NOT NULL, processed_at DATETIME NOT NULL DEFAULT GETUTCDATE(), consumer_version VARCHAR(20) NOT NULL ); -- consumer logic (pseudocode order matters) BEGIN TRANSACTION IF EXISTS (SELECT 1 FROM processed_events WHERE correlation_id = @msgId) ROLLBACK; ACKNOWLEDGE; RETURN -- already processed -- do business work here INSERT INTO work_orders (...) VALUES (...) INSERT INTO processed_events (correlation_id, event_type, processed_at, consumer_version) VALUES (@msgId, @eventType, GETUTCDATE(), @version) COMMIT; ACKNOWLEDGE
Why the Correlation ID Must Be the Producer’s, Not the Broker’s
The IBM MQ message ID, the Azure Service Bus message ID, and the Kafka offset all
identify the message transport, not the business event. When you replay a message from
a DLQ, forward it through a routing table, or archive and restore it, the broker-assigned
ID may change. The producer-assigned correlation ID — tied to the business event being
communicated — does not change. It represents the event itself, not its envelope.
Use the broker message ID for deduplication and you will have duplicate records the
first time you replay from an archive. Use the producer correlation ID and replay is
safe regardless of how the message arrived.
The ordering trap: idempotency handles the same message arriving twice. It does not handle messages arriving out of order. A StatusUpdated:degraded event that arrives after a StatusUpdated:failed event — due to a brief network delay — will revert the status backward if you process it without considering sequence. Idempotency and ordering are complementary patterns, not interchangeable ones.
The API Layer: Idempotency Keys
The same principle applies to synchronous APIs that perform write operations. A client
that times out and retries does not know whether the original request succeeded. Without
an idempotency mechanism, the retry creates a duplicate. With one, it returns the same
result as the original.
The pattern: clients send an Idempotency-Key header on write requests.
The API checks a cache (Azure Cache for Redis works well) before processing. If the key
is found, the cached response is returned immediately without reprocessing. If not, the
request is processed and the response is cached with a TTL that covers the realistic retry
window — 24 hours is sufficient for most enterprise scenarios.