How do you handle asynchronism, message queues and backpressure at scale?

Question

Accepted Answer

## Step 1: Use Cases, Constraints, Assumptions, Back-of-Envelope

### Use cases
We'll scope to handle only the core flows that expose the bottlenecks; everything else is deferred.

- **User** triggers async work — POST returns 202, job enqueued (email, thumbnail, analytics), polled or webhook on done.
- **Service** consumes — workers pull with prefetch, ack on success, DLQ on poison, retry with backoff.
- **Service** handles backpressure — bounded queues, shed/slow producers, scale consumers on lag.
- **Service** guarantees — at-least-once vs exactly-once via idempotency keys and deduplication.

### Out of scope
- Specific broker tuning (Kafka partition math beyond overview), exactly-once transactions deep dive, stream processing (Flink) beyond queue.

### State assumptions
- Queue depth 1M messages, each 1 KB → 1 GB buffered; throughput 10k msg/s.
- Consumer lag target <5s, scale trigger lag >1000.
- Backpressure: producer 20k/s vs consumer 10k/s → queue fills 10k/s, hits bounded 100k

How do you handle asynchronism, message queues and backpressure at scale?

Understand the problem

Attempt it yourself

Study the solution

Step 1: Use Cases, Constraints, Assumptions, Back-of-Envelope

Use cases

Solution ready — 2 min read

Join the discussion

Discussion (0)

KEEP THE
STREAK ALIVE.

Understand the problem

Attempt it yourself

Study the solution

Step 1: Use Cases, Constraints, Assumptions, Back-of-Envelope

Use cases

Solution ready — 2 min read

Join the discussion

Discussion (0)

KEEP THESTREAK ALIVE.

KEEP THE
STREAK ALIVE.