Setup & text as bytes

Every large language model reads text through a tokenizer — the thing that chops your prompt into "tokens" before the model ever sees it. In this hands-on course you'll build one from scratch: byte pair encoding (BPE), the exact algorithm behind GPT and most modern LLMs. If you took How ChatGPT Actually Works, this is how those tokens are actually made.

Setup is the easiest of any course here: there's nothing to install. Pure Python, not even NumPy. Just make sure you have Python 3:

python3 --version

A tokenizer's raw material: bytes

Before a tokenizer can do anything clever, it needs text as numbers. The trick that makes a tokenizer work for every language — English, Japanese, emoji, code — is to start from UTF-8 bytes. Any string encodes to a sequence of bytes, each a number from 0 to 255:

text = "tokens"
raw = text.encode("utf-8")     # a bytes object
ids = list(raw)                # turn it into a list of ints

print(ids)   # [116, 111, 107, 101, 110, 115]

Each number is one byte: 116 is t, 111 is o, and so on. For plain English, one byte is one character. But the magic is what happens with anything else:

print(list("hi 🚀".encode("utf-8")))
# [104, 105, 32, 240, 159, 154, 128]

The rocket emoji became four bytes (240 159 154 128). That's fine — our tokenizer will never need to know what an emoji is; it just sees bytes, and bytes cover literally everything. Starting from 0–255 means our vocabulary begins with exactly 256 possible tokens, one per byte value.

The plan

That 256-token vocabulary is where every BPE tokenizer starts — and it's tiny. BPE grows it by repeatedly finding the most common pair of tokens and fusing them into a new one. Do that a few thousand times and you get the vocabulary a real model uses. We'll build it in three moves: count pairs, merge the top pair, and repeat. First, counting.