Latency & throughput

Once a model is running, two numbers decide whether the deployment is any good — and they pull in opposite directions. Latency is how fast one user gets served. Throughput is how many users you can serve at once. Serving AI well is mostly the art of balancing them.

Scroll to feel both.

The numbers that matter

For an interactive model, three metrics come up constantly:

• Time to first token (TTFT) — the pause before anything appears. This is the prefill (Chapter 3) plus queueing. It's what makes an assistant feel responsive or laggy.
• Tokens per second — how fast the answer streams once it starts. Below reading speed feels slow; well above it feels instant.
• Throughput — total tokens per second the server produces across everyone. This is what determines your cost per request.

Batching: the central trick

Here's the tension. The GPU is happiest doing lots of math at once, but a single user's decode step is a small job that leaves it mostly idle. The fix is batching: gather many users' requests and run them through the model together, in one pass. The GPU stays busy, and total throughput soars.

Why this shapes everything

Every product decision downstream — the price of an API, whether streaming is on, how big a model you can afford to serve, whether it runs in the cloud or on a phone — traces back to this latency/throughput balance. And the way you win both at once is to make the model itself cheaper to run. That's next: faster, smaller, cheaper.