Counting pairs

From here we build one program. By the end of Chapter 5 the code stacks into a single bpe.py. It starts with the one insight the whole algorithm rests on: some pairs of tokens show up far more often than others. In English, t is followed by h constantly; e is followed by a space all the time. If a pair is that common, it's worth its own token.

So step one is simply to count every adjacent pair.

Start your bpe.py

We'll train on a chunk of text. Create bpe.py with a sample corpus and convert it to token ids (bytes, from last chapter):

text = ("the cat sat on the mat. the cat sat on the hat. "
      "tokenization turns text into tokens. a token is a chunk of text. "
      "the tokenizer learns the most common pairs of characters and merges "
      "them into new tokens. this is how large language models read text.")

ids = list(text.encode("utf-8"))
print("length in bytes:", len(ids))

Count every adjacent pair

get_stats walks the sequence and tallies each neighboring pair into a dictionary. zip(ids, ids[1:]) is the classic trick for "every consecutive pair":

def get_stats(ids):
  counts = {}
  for pair in zip(ids, ids[1:]):     # (ids[0],ids[1]), (ids[1],ids[2]), ...
      counts[pair] = counts.get(pair, 0) + 1
  return counts

Find the most common one

max with key=counts.get returns the pair with the highest count. Let's see what wins on our corpus — and decode the two bytes so it's readable:

stats = get_stats(ids)
top = max(stats, key=stats.get)
print(top, "->", stats[top], "times")
print("that's:", bytes(top).decode("utf-8"))

(32, 116) -> 16 times
that's: ' t'

The winner is (32, 116) — a space followed by t — appearing 16 times (all those the, text, tokens, to…). That " t" pair is the first thing our tokenizer will decide deserves its own token. Next chapter: actually fusing it.