Design an Inference API

The question

Design the API in front of a large language model. Users send a prompt and wait for the answer, some streaming, some not. Behind the API is a pool of GPUs running model replicas. Batch requests so the GPUs stay busy, keep tail latency bounded, tier free and paid users, survive a GPU dying mid-batch, and say how many GPUs you need for ten thousand requests a second.

This question comes in two costumes. The open one is the whole serving stack: continuous batching, KV cache, prefill against decode, autoscaling. The narrow one hands you a GPU function you cannot change and asks you to design only the dispatch layer around it:

def batchstring(inputs: list[str]) -> list[str]: ...
# You CANNOT change any of these:
# - 1 <= len(inputs) <= 100 strings per batch
# - exactly one output per input, same order
# - ~100 ms per batch, FIXED, does not depend on batch size
# - each GPU runs exactly ONE batch at a time

Client calls are synchronous HTTP with a 500 ms P95 budget end to end. Be ready for 100 requests a second and for 10,000. And there is a follow-up that shows up in both costumes: the pool is fixed at eight GPUs, one model needs a single GPU per batch, the other needs all eight at once. Write the scheduler.

Explain it to a ten-year-old

There is a ferry across a river. The crossing takes ten minutes whether one person is aboard or a hundred, and the boat holds a hundred. You are the person at the dock deciding when it leaves.

If you wait until it is full, the first person aboard on a quiet morning waits an hour. If you send it the moment anyone arrives, you run a hundred crossings to carry a hundred people and the queue on the bank grows forever. So you make a rule: leave when the boat is full, or when the first person aboard has waited one minute, whichever comes first. On a quiet morning the minute rule fires and a few people cross. At rush hour the boat fills in seconds and the full rule fires. That rule is the whole design, and everything else, how many ferries, who boards first, what happens if a ferry breaks down mid-river, is built on top of it.

flowchart TB
  c[Clients] --> gw[API gateway<br/>auth, idempotency key,<br/>rate limit per tier]
  gw --> pq[Priority queues<br/>paid ahead of free]
  pq --> b[Batcher<br/>flush at 100 or at 50 ms]
  b --> d[Dispatcher<br/>least-loaded GPU]
  d --> g1[GPU 1]
  d --> g2[GPU 2]
  d --> gn[GPU n]
  b --> m[(Request → batch<br/>mapping table)]
  g1 --> r[Response fan-out<br/>back to waiting clients]
  a[Autoscaler + queue depth monitor] -.-> gw
  a -.-> d
  style b fill:#fed7aa,stroke:#ea580c

Here is the dock rule as a picture. Six slots stand in for a hundred. Watch the timer bar against the queue:

The trick

Everything the GPU function fixes, you stop arguing about. What is left is the fifty milliseconds in front of it, and the arithmetic.

Batch on size or time, whichever comes first. Then do the capacity sum out loud before anyone asks. One GPU runs one batch of up to 100 in 100 ms, so its ceiling is 1,000 requests a second at perfect fill. Ten thousand a second therefore needs ten GPUs as a floor, and a floor is not an answer: at 100 percent utilisation every queue grows without bound, so you run at 60 to 70 percent and add spares, which lands somewhere between 15 and 20. The candidates who stumble on this round stumble because they never wrote that formula down.

In the open costume the trick is the sentence “decode is memory bound”, and the rest is the continuous batching and KV cache lessons. Name paged attention and in-flight batching together in the first five minutes and the interviewer will follow you into whichever one you know best.

The steps

One thing about this round cuts against my own 45-minute plan: do not walk the generic template. The interviewer skims requirements and API and spends the round on one or two depth areas, usually batching, capacity or rate limiting. So compress steps one and two to three minutes and get to the batcher.

1. Find out which costume. Ask what you may change. If the batch function is fixed, continuous batching, autoscaling and even rate limiting may be ruled out, and the entire round lives in the dispatch layer. If it is the open question, ask for the model size, the context length and whether the traffic is chat or batch. Ask whether the client waits or polls.
2. Write the budget. 500 ms P95. Spend it: up to 50 ms waiting in the batcher, 100 ms on the GPU, a few tens of milliseconds in queues and network, and keep about 200 ms back. That reserve is not slack. It is exactly one retry after a GPU dies mid-batch, and saying so is what turns a number into a design.
3. Do the capacity sum. Per GPU: batch size divided by batch latency, 100 / 0.1 = 1,000 requests a second. Floor for 10,000: ten GPUs. Then say why you will never see it. Batches do not fill perfectly, requests arrive in bursts, one GPU is always being replaced, and a latency-bounded online service runs well below an offline benchmark’s throughput. Target 65 percent and N+2: about 18 GPUs. For 100 requests a second the sum flips. In a 50 ms window you collect five requests, so batches are tiny, one or two GPUs are plenty, and the timer, not the size, is the lever you tune. If the interviewer gives you B achieved requests per batch and C independent batch slots per server, the servers needed are at least 10,000 / (B × C / 0.1). Ask for B and C. Do not assume the eight-GPU model has eight slots. It has one.
4. Build the batcher. One in-memory pending list per model. Flush when it reaches 100 or when the oldest entry is 50 ms old. Each request carries a future the HTTP handler awaits. On flush, write the request-to-batch mapping to a table, hand the batch to a GPU, and on return resolve each future in order. Say why 50 and not 100: the timer plus the GPU time must fit under the budget with the retry reserve intact.
5. Dispatch load-aware. Round robin and random are the first answer and the interviewer will attack it: two GPUs with unequal queue depth, or one GPU slower than the rest. Route to the GPU with the shortest queue, or use a pull model where an idle GPU takes the next ready batch. The pull model also makes a dead GPU harmless, because it simply stops pulling.
6. Tier the traffic. Two priority queues, paid and free, feeding the same batcher. Fill from paid first, top up from free. Under overload the free queue is the shock absorber: it is shed first, its rate limit tightens first, and its timeout is longer.
7. Survive a GPU dying mid-batch. The mapping table tells you which requests were in the lost batch. Re-enqueue them at the head of the queue on another GPU. The client’s idempotency key, on the user-facing request ID, means a retry from their side does not double-charge or double-generate. Partial batches on flush are fine. A partial batch on failure is the case to walk through.
8. Handle overload honestly. GPUs take minutes to start, so autoscaling does not save you in the next thirty seconds. What does is backpressure at the front door: watch queue depth, and when it crosses a line tighten rate limits dynamically, shed free tier, return 429 with a retry-after. The answer I want is throttling tied to observed queue depth, not a static token bucket. A token bucket is a policy. Queue depth is the truth.
9. Cold start, and bring GPUs up faster anyway. A request arrives for a model version with no live replica. The scheduler takes hardware from a warm pool (GPUs up, memory empty), streams that version’s weights from a fast weights store on NVMe or object storage, marks the replica ready, and only then does the router send traffic. That is minutes, so say so, and say what you do meanwhile: queue with a longer timeout, or answer with a retry-after. Never pre-load the weights of inactive models as insurance; the warm pool is hardware, not weights. To make the load itself faster: local NVMe, a regional mirror rather than a cross-country pull, and peer-to-peer distribution when many replicas need the same version.
10. Decide about a cache. A response cache sounds free and usually is not. Estimate the hit rate: identical prompts are rare in chat and common in classification. If it is under a few percent, say so and leave it out. A defended omission beats a reflex inclusion. A prefix cache of computed KV blocks is a different thing and usually worth it for shared system prompts.
11. Isolate tenants. One customer sending a 10,000-document job must not stall everyone else’s chat. Per-tenant queues with quotas, a cap on tokens per request, and fair fill into the batch (paid first, then round-robin across tenants) are the answer; say “noisy neighbour” and the interviewer knows you have run a multi-tenant system.
12. Place the guardrails. Safety classifiers pre-model on the prompt, post-model on the output, or both. Both costs latency twice. Say where in the budget it lives and whether it runs on the GPU pool or on cheaper hardware beside it.
13. The eight-GPU follow-up. Small batches reserve one GPU each. A large batch reserves all eight atomically or not at all. When a large batch is waiting, stop admitting small ones and let the running ones drain, then launch. Bound the wait with aging or fixed turns so neither queue starves. Then say the cost out loud: during the drain, GPUs sit idle, so this policy trades utilisation for fairness and is not throughput-optimal.

Here is step thirteen running. Teal is the one-GPU model, purple is the eight-GPU model, dashed is idle:

The board

This is the picture I want on the screen by minute forty. Front door on the left, the router that turns model and version into a replica pool, per-tenant queues feeding a per-replica batcher, the replicas along the third row, and the two things candidates forget on the bottom row: a warm pool of hardware with no weights loaded, and the weights store the scheduler streams from on a cold start. Solid arrows are the request, dashed ones are control, cold start and failure.

Download the one-page cheat sheet (A4, two sides: front is what to ask and what to design in the 55-minute order, back is this board).

Three words on that board trip people up, so define them out loud in the first ten minutes:

• Model is a blueprint: a name, a version, and a set of weights sitting on storage. Weights for a frontier model are on the order of a terabyte of numbers.
• Replica is hardware: a fixed group of GPUs with one model version’s weights loaded into their memory, a queue in front of it, and a state (warming, ready, draining). A replica does not care which model it runs until weights are loaded. Every model version has a minimum GPU count, the way software has minimum requirements on the box.
• Batch is not an entity. It is what the replica’s queue does: a group of requests processed together so the GPUs stay busy. Say it as a property of the replica and the interviewer relaxes.

Because weights live in GPU memory, serving is memory-bound, and that decides the cold-start design: keep hardware warm, never the weights of a model nobody is using. A GPU pinned with idle weights cannot serve anyone else.

The template

The narrow costume is a shape you will meet again: a dispatch layer in front of a fixed-cost batch backend. Database bulk writers, log shippers, and payment settlement files all have the same loop.

on request(r):
    r.future = new Future
    QUEUE[r.TIER].append(r)
    if len(pending for MODEL) >= MAX_BATCH: flush(MODEL)
    elif no timer running: start timer(MAX_WAIT) -> flush(MODEL)
    return await r.future            # HTTP handler blocks here

flush(MODEL):
    batch = take up to MAX_BATCH, PAID first, then FREE
    MAPPING[batch.id] = [r.id for r in batch]
    gpu = LEAST_LOADED(pool for MODEL)    # or let an idle GPU pull
    gpu.submit(batch)

on result(batch, outputs):
    for r, out in zip(MAPPING[batch.id], outputs): r.future.set(out)

on gpu_failure(gpu):
    for batch in gpu.inflight: requeue(MAPPING[batch.id], head=True)

on queue_depth > HIGH_WATER:
    tighten RATE_LIMIT[FREE], then RATE_LIMIT[PAID], answer 429 + retry-after

What changes per problem is the four capitals at the top: MAX_BATCH and MAX_WAIT come from the backend’s contract and the latency budget, TIER comes from the product, and LEAST_LOADED is whatever signal you can actually observe. What must be understood, not memorised, is why MAX_WAIT is the lever at low load and MAX_BATCH at high load, and why the retry reserve in the budget decides MAX_WAIT rather than the other way round. Memorise the loop, then be able to derive every constant in it from the numbers you were given.

In GPU infrastructure

This is the one lesson where I do not have to translate. The serving pool is a slice of the same fleet I burn in and health-check, and every box in the diagram maps to something that pages someone. The dispatcher’s “least loaded” signal comes from the same telemetry that catches stragglers in training, because a GPU with a degraded NVLink or a throttling clock finishes its batch late and, under round robin, quietly drags the P95 of one eighth of all traffic. The “bring GPUs up faster” follow-up is a real project, not an interview flourish: baked images, weights staged on local NVMe from a regional mirror, and a warm pool sized to the slowest replacement time we have measured. And the eight-GPU scheduler is exactly the tension between a job that wants a whole node with its NVLink fabric and a stream of small jobs that would happily fragment it. Reserve atomically, drain, and know what the idle minutes cost.

What I am listening for

• Whether you ask which parts of the design may change before you design any of it.
• Whether the capacity formula appears in the first ten minutes without prompting, and whether you then say why the floor is not the answer.
• Whether “size or time, whichever first” is your batcher, and whether you can derive the timer from the latency budget.
• Whether you defend round robin when I attack it, or move to a load-aware or pull-based dispatch.
• Whether your overload answer is a queue depth, not a token bucket.
• Whether you volunteer the utilisation cost of draining for the eight-GPU model.
• The follow-ups, which are where the round is decided:
  • Half the GPUs just died. Protect the SLA. Backpressure at the gateway, tighten limits by tier, shed free first, re-route in-flight batches from the mapping table.
  • A GPU crashes mid-batch. Which requests, where do they go, and does the client see a duplicate?
  • How do you bring machines up faster? Images, local weights, mirrors, a warm pool.
  • Why is production throughput below the benchmark? Batch fill under a latency bound, bursts, and spares.
  • Do you need a cache? Estimate the hit rate first. Be willing to say no.
  • Two models, eight GPUs. Atomic reservation, drain, aging, and the cost.

How the round is delivered

This question turns up in a few formats, and knowing which one you are in saves ten minutes.

• A design-doc review. You are given an existing inference-server design with planted weaknesses and asked to critique it. Do not annotate every box. Go straight to the batching strategy and the KV cache trade-offs, because that is what the interviewer is grading, and leave time for the follow-ups.
• A Google Doc, not a whiteboard. Often a three-box diagram, client to API to GPU pool, and you type your reasoning rather than draw. Practise writing the capacity sum in plain text.
• A phone screen. A few minutes to read the doc, then entirely verbal. Load balancing and batching first, then the eight-GPU follow-up.
• Say “technical challenges”, not “non-functional requirements”. Full GPU utilisation, bounded tail latency, cold start, isolation, availability over consistency, and metrics are challenges the design must solve; listing them by that name keeps the interviewer and you on the same page, and four or five of them is the right number.
• A pressure round. Some interviewers give few hints and only open the next follow-up once you have landed the expected answer. Keep proposing concrete mechanisms. Silence is not a hint.

Go deeper

• Orca, the paper that introduced iteration-level, in-flight batching, and PagedAttention / vLLM, the memory side. Together they are the baseline every serving stack converged on.
• DistServe, on running prefill and decode on separate GPUs, which is the answer when the interviewer asks about long prompts stalling everyone else’s tokens.
• The design docs for vLLM, TensorRT-LLM and Triton Inference Server. Read at least one properly. The interviewer will follow you into whichever you name.
• Continuous Batching for Inference, Size the KV Cache and Design a Topology-Aware GPU Scheduler are the three lessons here that the open costume leans on. Design a Rate Limiter and Design a Load Balancer cover the front door.

With AI on the table. An assistant will produce the batcher loop and the vLLM vocabulary in one go. So I give it the fixed-batch contract and a P95 of 500 ms and ask it for the timer value. It will say 50 ms. Then I ask you to defend 50 against 20 and against 80 using only the numbers in the contract. The tool knows the loop. You have to know where each constant comes from.