Encode & decode

A trained tokenizer has one job in each direction: encode turns new text into token ids, and decode turns ids back into text. Both use the merges and vocab you just built.

Encode: apply the merges in order

To encode text, start from its bytes and keep merging — but there's a subtlety. You must apply merges in the order they were learned, because later merges were built assuming the earlier ones already happened. Each round, of all the pairs present that we know a merge for, we apply the one with the lowest id (the earliest-learned):

def encode(text):
  ids = list(text.encode("utf-8"))
  while len(ids) >= 2:
      stats = get_stats(ids)
      # pick the present pair whose merge was learned earliest (lowest new id)
      pair = min(stats, key=lambda p: merges.get(p, float("inf")))
      if pair not in merges:
          break                      # no learnable pair left — done
      ids = merge(ids, pair, merges[pair])
  return ids

Decode: look up and concatenate

Decoding is simpler — every id is in vocab, so we glue the bytes together and turn them back into a string:

def decode(ids):
  data = b"".join(vocab[i] for i in ids)
  return data.decode("utf-8", errors="replace")

Try it out

Encode a phrase the tokenizer never saw during training and check the round-trip:

sample = "the tokens"
enc = encode(sample)
print("text: ", sample)
print("ids:  ", enc)
print("count:", len(sample.encode("utf-8")), "bytes ->", len(enc), "tokens")
print("decode:", decode(enc))
print("lossless:", decode(enc) == sample)

text:  the tokens
ids:   [116, 257, 265, 115]
count: 10 bytes -> 4 tokens
decode: the tokens
lossless: True

Ten bytes became four tokens: 116 (t), 257 (he), 265 (" token"), 115 (s). The whole word " token" collapsed into a single id — because the tokenizer learned it in training. Decoding rebuilds the exact original string, character-for-character. That lossless round-trip is the contract every real tokenizer keeps: whatever the model reads, it can always be turned back into the exact text. Last stop: how good is our tokenizer, really?