Case Study: Async Notification Fanout Engine

01 //

The Engineering Challenge & Constraints

In multi-channel alert systems (monitoring, financial anomalies, verification OTPs), standard synchronous HTTP calls fail when recipient services (SMTP servers, SMS gateways, APNS) experience transient latency spikes or throttling. A blocking request thread can exhaust backend connection pools within seconds.

The objective was to design a completely decoupled event ingestion and worker fanout pipeline that guarantees:

Strict Sub-Millisecond Ingestion: Ingest API endpoints must respond in under 12ms without waiting on downstream delivery.
Absolute Idempotency: Duplicate event emissions from upstream clients must never result in duplicate emails or SMS messages.
Dead-Letter Queue (DLQ) & Graceful Retry: Transient provider failures must back off exponentially before dropping to an inspection queue.

02 //

System Architecture & Data Pipeline Flow

The architecture decouples request ingestion from worker dispatch using partitioned Apache Kafka topics, backed by an atomic Redis sliding-window idempotency layer.

FIGURE 1: ASYNCHRONOUS FANOUT TOPOLOGY PIPELINE VELOCITY: 10K+ EVENTS/DAY

STAGE 01

Client Ingestion

FastAPI • HTTP/2

STAGE 02

Deduplication Gate

Redis Key TTL • SHA-256

STAGE 03

Event Bus

Kafka Partitions

STAGE 04

Worker Pool

Go Goroutines

STAGE 05

Audit & Dispatch

PostgreSQL • Multi-Channel

03 //

Critical Architectural Decisions & Trade-offs

DECISION // MESSAGE BROKER

Kafka Partitioning vs RabbitMQ

Selected Apache Kafka over RabbitMQ to leverage consumer group partition offsets. Partitioning by user tenant ID guarantees sequential ordering per recipient while enabling horizontal worker pool scaling without race conditions.

DECISION // IDEMPOTENCY

Redis Distributed Locks with Sliding TTL

Employed an atomic Redis SETNX with an automated 24-hour sliding TTL keyed to the SHA-256 hash of (recipient_id + template_id + payload_digest). If a duplicate event arrives, the gateway immediately returns HTTP 200 with an IDEMPOTENT_IGNORED status.

DECISION // RUNTIME

Go Goroutine Worker Pool vs Python Celery

Replaced legacy Celery workers with custom Go goroutine pools. Memory footprint decreased from 1.2GB per container to 68MB, and channel dispatch throughput increased by 4.2x with zero garbage collection pause impact.

DECISION // FAULT ISOLATION

Channel-Specific Dead-Letter Queues

If an external email SMTP provider triggers rate limits (HTTP 429), SMS and Push channels remain completely unaffected. Failing payloads are diverted to a dedicated retry topic with exponential jitter backoff.

04 //

Telemetry & Production Execution Log

Live simulated stream of asynchronous worker pool event consumption and channel fanout execution:

FANOUT WORKER DAEMON // CLUSTER NODE-03 // GO v1.22 STREAM: LIVE

[23:14:02.102] INFO broker/kafka.go:142 • Partition [02] offset 49102 consumed by worker-pool-04

[23:14:02.104] EXEC idempotency/redis.go:88 • Checking lock: alert:user_8921:digest_8f3a91 → LOCK_ACQUIRED

[23:14:02.108] DISP channels/push.go:61 • APNS payload dispatched (latency: 3.4ms, status: 200 OK)

[23:14:02.112] DISP channels/sms.go:94 • Twilio SMS carrier handshake acknowledged (status: DELIVERED)

[23:14:02.118] AUDT db/postgres.go:210 • INSERT INTO alert_audit_log (id, status, p99) VALUES (...) → COMMITTED

[23:14:02.119] INFO fanout.go:305 • EVENT COMPLETED IN 11.2ms • ZERO PACKET LOSS