Where inference runs

Here's the punchline: almost every time you touch AI, you're triggering inference. Training happened once, in a lab, months ago. Inference is happening right now, on your phone and in data centers, for you and millions of others. This closing tour points it out in the wild.

Cloud vs edge — where does the forward pass happen?

A big practical question is where the model runs, and it's a tradeoff:

• Cloud inference — the model lives on powerful servers. Best for large models (LLMs, image generation) that can't fit on a device. Costs a round-trip of latency and network, and someone pays per request.
• On-device / edge inference — the model runs locally on your phone, laptop, camera or car. Fast, private, works offline — but limited to models small enough to fit, which is exactly what the optimizations from the last chapter make possible.

The same trained weights can be served either way; the deployment choice is all about latency, privacy, cost and model size.

That's the course

Zoom out and inference is one simple idea seen from every angle:

• It's the second phase — using a model, not training it — and it's where the real-world compute goes.
• Mechanically it's a forward pass: input in, answer out, weights frozen.
• For a chatbot that pass runs in a token-by-token loop.
• Its quality is measured in latency and throughput.
• And it's made faster, smaller, cheaper because it runs a billion times.

So next time an app answers you in an instant, you'll know what just happened: a frozen model ran a forward pass — one of billions — and that, not the training, is what AI mostly is in daily life.

If you enjoyed this, the other courses cover what's being run — how ChatGPT works, how images are generated, and the chips that make inference fast.