Module 03
Reliable asynchronous processing
- At-least-once delivery & idempotent consumers
Crash a consumer after its side effect but before its ack, see the redelivery double-charge, then enable a dedup table and replay.
- Idempotency keys (API level)
A client retries a timed-out payment request, and an Idempotency-Key returns the original response instead of charging again.
- Transactional outbox
Crash the service between "DB commit" and "publish" to strand work, then switch to an outbox and watch the relay recover it.
- Retries with backoff & jitter
Visualize a hundred clients retrying against a recovering service, with and without exponential backoff and jitter, and watch synchronized retry waves smooth out.
- Dead-letter queues & poison messages
A malformed message fails forever and blocks its partition until bounded retries move it to a DLQ, where you replay it, possibly causing a second incident.
- Reconciliation jobs
Silently corrupt derived state, then run a reconciler that diffs it against the source of truth and repairs the gaps.
- Progress tracking & cancellation
Cancel a long job mid-flight and see which steps finished, which were rolled back, and which kept running because the worker never checked for cancellation.
- Queues vs pub/sub, partition keys, slow consumers
Route ride-lifecycle events through Kafka-style partitions, pick the partition key, and slow one consumer group to see that the others are unaffected while ordering holds per key.
- Event-driven architecture
Convert a synchronous checkout call chain into events and see new failure modes (partial completion, out-of-order events) appear.
- Two-phase commit
Kill the 2PC coordinator after the prepare phase and watch participants stuck holding locks.
- Sagas (orchestration vs choreography)
Fail any step in a flight + hotel + car booking, watch compensations unwind backward, then make a compensation fail too.
- CQRS & event sourcing
Append events to a log, rebuild the read model by replay, and add snapshots to see replay cost drop.