How would you design AI inference infra with GPU queue and KV-cache?

Question

Accepted Answer

## Step 1: Outline use cases and constraints

> Gather requirements and scope the problem. Ask questions to clarify use cases and constraints. Discuss assumptions.

### Use cases

#### We'll scope the problem to handle only the following use cases

* **User** performs core action described in How would you design AI inference infra with GPU queue and KV-cache?
* **User** reads/views result with low latency (<200ms P99)
* **User** expects durability and availability across AZs
* **Service** processes requests via stateless API + queue
* **Service** stores data durably and serves via cache + DB
* **Service** provides analytics and retention

#### Out of scope

* Advanced ML ranking
* Billing/payments detail
* Full admin console

### Constraints and assumptions

#### State assumptions

* Traffic not evenly distributed (Pareto 80/20)
* High availability 99.99% (52m downtime/yr), durability 99.999999%
* Estimate 10M DAU, 100M requests/day
* Read-heavy or write-heavy per use case, 5x peak ov

How would you design AI inference infra with GPU queue and KV-cache?

Understand the problem

Attempt it yourself

Study the solution

Step 1: Outline use cases and constraints

Use cases

We'll scope the problem to handle only the following use cases

Solution ready — 2 min read

Join the discussion

Discussion (0)

KEEP THE
STREAK ALIVE.

Understand the problem

Attempt it yourself

Study the solution

Step 1: Outline use cases and constraints

Use cases

We'll scope the problem to handle only the following use cases

Solution ready — 2 min read

Join the discussion

Discussion (0)

KEEP THESTREAK ALIVE.

KEEP THE
STREAK ALIVE.