Chapter 4 · Part 2

Cache what repeats

Look at any production AI app's requests side by side and you'll notice something absurd: they're almost identical. The same 5,000-token system prompt. The same instructions, the same examples, the same reference documents. The only thing that changes is the little question at the end — yet Chapter 2 taught us the model re-reads (and you re-buy) the whole thing every call.

Providers noticed too, and built the single best discount in AI: prompt caching. If the beginning of your request is byte-for-byte identical to a recent one, the provider reuses its internal computation over that prefix instead of redoing it — and charges you roughly a tenth of the input price for those tokens.

Scroll to watch the discount kick in.

Call 1: the big prefix is processed at full price, plus a small fee to write it into the cache.

scroll↓

Why it works — and what it costs

When a model reads your prompt, it builds up internal state for every token (the attention keys and values, if you've taken the attention course). Caching simply saves that state keyed on the exact prefix, and reloads it next time instead of recomputing. That's why reads are so cheap — the expensive work is skipped entirely.

The economics: cache writes cost a small premium over normal input (~1.25×), and reads cost ~0.1×. So caching pays for itself by the second request — and a hot system prompt might be read millions of times.

The one rule: byte-identical, stable-first

The cache matches the prompt prefix, exactly. One changed character invalidates everything after it. Which leads to the design rule that separates teams who get the discount from teams who don't:

  • Stable content first — system prompt, instructions, examples, documents.
  • Changing content last — the user's question, this turn's data.
  • Never put volatility up front. A timestamp, a request ID, or a "Today is Tuesday" at the top of the system prompt silently breaks caching on every single call. (The bill won't warn you — it just stays big.)

Multi-turn chats get the same gift: each turn extends the previous one, so the whole history can be a cache read, defusing much of Chapter 2's context tax.

Caching makes repeated tokens cheap. The next lever is blunter: send and generate fewer tokens in the first place.