Faster, smaller, cheaper

Because inference runs billions of times, shaving a little off each run is worth enormous effort. A whole field exists just to make the same model cheaper and faster to serve — often trading a sliver of accuracy for a huge cut in size, latency and cost. Here are the big four.

Scroll to stack the optimizations.

The optimization toolkit

Each technique attacks a different cost:

• Quantization — represent the weights (and sometimes activations) with fewer bits: 32-bit floats → 8-bit or even 4-bit integers. The model shrinks ~4–8× and runs faster on hardware that loves low-precision math, with usually minor accuracy loss.
• Distillation — train a small "student" network to mimic a big "teacher." You get a fraction of the size at a fraction of the quality cost — often surprisingly little.
• KV caching — the "don't recompute the past" trick from Chapter 3; essential for LLM decode speed.
• Batching — serve many requests per GPU pass (Chapter 4) to slash cost per request.

The tradeoff, stated plainly

None of these are free — each trades a little accuracy or added complexity for speed and cost. The skill is knowing how much you can give up before users notice. Done well, a model that cost millions to train can serve a single answer for a fraction of a cent.

We've now seen inference end to end: the two phases, the forward pass, the token loop, the metrics, and the cost-cutting. To finish, let's spot it everywhere. Next: where inference runs.