Compression & real tokenizers

Your tokenizer works. Let's measure how well — and then see how the exact algorithm you wrote scales into the tokenizers running inside GPT.

The compression ratio

A tokenizer's whole point is to represent text in fewer units than raw bytes. The ratio of bytes to tokens is its compression:

ids = list(text.encode("utf-8"))
tokens = encode(text)
print(f"{len(ids)} bytes -> {len(tokens)} tokens")
print(f"compression: {len(ids) / len(tokens):.2f}x")

247 bytes -> 147 tokens
compression: 1.68x

Just 20 merges already pack the text 1.68× tighter. Train more merges and the ratio climbs — real tokenizers use tens of thousands and hit roughly 4 bytes per token on English. Fewer tokens means cheaper, faster models: it's literally why you're billed per token.

It handles anything — for free

Because we started from bytes, your tokenizer already works on text it never trained on, in any script, including emoji:

s = "any text works 🚀"
print(len(s.encode("utf-8")), "bytes ->", len(encode(s)), "tokens")
print("round-trips:", decode(encode(s)) == s)

19 bytes -> 14 tokens
round-trips: True

The rocket emoji is just four bytes to us; we tokenize and perfectly reconstruct it without a single line of emoji-specific code. That robustness is exactly why real tokenizers are byte-level too.

How the real ones differ

What you built is the genuine core of GPT's tokenizer. Production tokenizers like OpenAI's tiktoken add a few things on top:

• Way more merges. GPT-4's vocabulary is ~100,000 tokens, not 276 — so common words are single tokens and even rare ones are a handful.
• A regex pre-split. Before merging, they split text on a pattern so merges never cross word/punctuation boundaries (keeping "dog" and "dog." sensible).
• Special tokens. Markers like <|endoftext|> are reserved ids the model treats specially.

The count-pairs → merge → encode/decode loop, though? That's exactly what you wrote.

Where to go next

• Feed it a bigger corpus — a book, your codebase — and bump vocab_size to a few thousand. Watch whole words become single tokens.
• Compare with the real thing: pip install tiktoken, then tiktoken.get_encoding("cl100k_base").encode("hello world") to see GPT-4's tokenizer on the same text.
• See what happens next: tokens are the input to the model — How ChatGPT Actually Works and How AI Pays Attention pick up exactly where your token ids leave off.

You didn't just learn what a token is — you built the machine that makes them.