Layer 2: Chips

The second layer turns electricity into math. It's also the layer NVIDIA is best known for, so it's worth being clear about what makes an "AI chip" different from the processor in your laptop.

AI is a mountain of simple math

Under the hood, a model producing one token mostly does one thing, over and over: multiply two numbers and add the result to a running total. A large model does this billions of times per token.

Each of those calculations is simple, and most of them don't depend on each other. That changes what a good chip looks like. A laptop's CPU has a handful of very fast, very clever cores, built to race through a sequence of steps one after another. An AI chip has thousands of simpler cores that all do multiply-adds at the same time.

Huang's essay describes chips at this layer as built for exactly two things: massive parallelism and high-bandwidth memory. The animation shows both.

• Parallelism. Split the math into millions of independent pieces and do them together. That's what GPUs were originally built for, since drawing a video game frame is also millions of independent calculations. It turned out to fit AI almost perfectly.
• Memory bandwidth. A model's numbers (its weights) can take up hundreds of gigabytes. For every token, the chip has to read a large share of them. If memory can't deliver them fast enough, the cores sit idle, however fast they are. That's why AI chips come packaged with special high-bandwidth memory (HBM), and why memory makers have become part of the AI story.

Why efficiency is the whole game

Go back to the constraint from the last chapter. If a data center can only get so much power, then how much AI it can produce depends on one number: how many tokens each chip makes per watt. Huang makes this point directly: chip efficiency decides how fast AI can scale, and how affordable intelligence becomes.

That's why each new chip generation is sold on tokens-per-watt and cost-per-token as much as raw speed. A chip that does the same work on half the energy doubles what a power-limited data center can do. It also halves the electricity cost of each answer.

Where chips come from

An AI chip isn't made by one company. Designers like NVIDIA, AMD, Google and Amazon design them. Almost all of the most advanced ones are manufactured by a single company, TSMC, mostly in Taiwan. Memory comes from a few specialist makers, and the finished chips are assembled into servers by yet more companies. At Davos, Huang pointed to TSMC planning around 20 new chip plants, and to server builders adding around 30 new computer factories, as signs of how fast this layer is being expanded.

A single chip, however good, can't run a frontier model on its own. The next layer is about wiring tens of thousands of them together.