(Byte Pair Encoding (BPE))
Build the byte-pair encoding tokenizer behind GPT — in pure Python, in about half an hour.
A free, hands-on tutorial: build a byte pair encoding (BPE) tokenizer from scratch in pure Python — the exact algorithm behind GPT and most modern LLMs. Train it to learn merges from text, then encode and decode, and see how words become tokens. Zero dependencies, runs entirely on your laptop.
Every LLM reads text through a tokenizer like this one — it decides what a 'token' is, why you're billed per token, and why models sometimes stumble on spelling and rare words.
Invent a brand-new token by replacing every occurrence of the top pair with a single new id.
Loop count-and-merge to learn a vocabulary — and watch it build ' to' then ' toke' from smaller pieces.
Apply the learned merges to turn new text into tokens — and rebuild the exact original bytes back.