One token at a time

A chatbot doesn't produce its whole answer in one shot. Inference for a language model is a loop: run a forward pass to get one token, stick it onto the end of the text, and run again to get the next. That's why ChatGPT's replies stream in word by word — you're literally watching the inference loop tick.

Scroll to run the loop.

Prefill, then decode

LLM inference has two distinct phases, and they have very different costs:

• Prefill — the model reads your entire prompt at once, in parallel, to build up its internal state. This is fast per token (it's one big batched pass) but scales with prompt length.
• Decode — it then generates the answer one token at a time, each requiring its own forward pass. This is the slow, sequential part: you can't compute word 10 until you've committed word 9.

This is also why the context window costs what it does — a longer prompt is more to prefill, and every generated token re-reads the whole thing.

The KV cache: don't redo work

Naively, generating token 100 would mean re-processing all 99 previous tokens from scratch — every step. That would be brutally slow. The fix is the KV cache: the model stores the intermediate results ("keys" and "values") for every token it has already seen, so each new token only computes its own contribution and reuses the rest.

Where we're headed

Because decoding is sequential, an LLM's speed is felt in two very human ways: how long until the first word appears, and how fast the rest stream. Those are the numbers that make a deployment feel snappy or sluggish. Next: latency & throughput.