Step 3 — Understand tokenizers

Step 3 — Understand tokenizers 🔪¶

Goal: write Byte Pair Encoding from a blank file, understand every line of it, and finish with a measurement script that tells you — in one number — whether any tokenizer is fit for your language.

Why this step matters¶

Think of tokenization as cutting vegetables before you cook. If you cut them badly, everything after that is harder, slower, and worse. Nothing you do later fixes a bad cut.

A token is one piece of text. It might be a whole word, part of a word, or a single letter. A tokenizer is the tool that does the cutting.

The character-level tokenizer from Step 1 cuts too finely. One sentence becomes hundreds of tokens, and attention cost grows with the square of the number of tokens. That gets expensive very fast.

But here is the part people miss, and it is the reason this chapter exists at all. The damage to Indic scripts does not happen inside the BPE algorithm. It happens in the regular expression that runs before it. You cannot see that from outside a library. You can see it very clearly from inside forty lines of your own code.

The number that matters: fertility 📏¶

Fertility is simply this: how many tokens does one average word become?

• English, with a normal tokenizer: about 1.3 tokens per word.

• Sanskrit, Tamil, and Telugu with the same tokenizer: often 3 to 4, and sometimes worse.

Sit with that number for a moment. If your tokenizer needs 3 tokens where English needs 1.3, then:

• Your context window holds less than half as much Sanskrit.

• Training costs more than twice as much for the same amount of meaning.

• Your model wastes capacity gluing word pieces back together.

• Every user query costs you more to serve, forever.

Purpose-built Indic tokenizers get this down to roughly 1.4 to 2.1.

That improvement is free performance, and it is available to you in one afternoon. This is the highest-leverage chapter in Part 2.

The shape of this chapter 🗺️¶

A tokenizer is four stages in a pipe. Three of them are boring and one of them is where your language gets hurt:

We build them in that order, then measure.

1 · Normalize first, always 🧼¶

Two Devanagari strings can look completely identical on screen and be stored as different sequences of code points. To you they are the same word. To your tokenizer they are two different words, taking two vocabulary slots and splitting your training signal in half.

import unicodedata

def normalize(text: str) -> str:
    return unicodedata.normalize("NFC", text)

That is the whole fix. It is one line, it is not optional, and it is not a detail. Every other file in this book starts with it.

2 · Pre-tokenization — where the damage happens ⚠️¶

Before BPE ever runs, a regular expression splits the text into rough chunks, usually at spaces and punctuation. BPE can never merge across those boundaries. So a bad cut here can never be repaired later, no matter how large your vocabulary is.

Here is the pattern most tokenizers use, near enough — the GPT-2 shape:

GPT2_ISH = regex.compile(
    r"'s|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+"
)

Read it charitably. It is not hostile to Devanagari — \p{L} means “any Unicode letter”, so Devanagari consonants match fine. The problem is what is missing: \p{M}, the category for combining marks. In Devanagari that category holds every vowel sign, the virama, the nukta, anusvara, visarga — the marks that turn a bare consonant into a syllable. They are not letters, so they fall through to the punctuation branch, and a syllable gets cut in half.

Now the Indic-aware version. It is the same pattern with two changes:

INDIC = regex.compile(
    r" ?[\p{L}\p{M}]+| ?\p{N}+| ?[\u0964\u0965]| ?[^\s\p{L}\p{M}\p{N}]+|\s+"
)

• \p{M} joins \p{L} inside the letter run, so a consonant and its vowel sign are never separated.

• The danda । and double danda ॥ get their own branch, because they are Sanskrit’s full stops, not stray symbols.

3 · BPE, in full — train_bpe 🧩¶

Now the algorithm itself, which is genuinely simple:

1. Start with single characters.

2. Find the most common pair of neighbouring pieces.

3. Glue that pair into one new token.

4. Repeat until you have the vocabulary size you want.

There is no step five. Here it is as code:

def train_bpe(text, vocab_size, pattern=INDIC):
    # count identical pre-tokens once, with a frequency — this is what makes
    # a readable implementation fast enough to actually run
    word_freqs = Counter(pre_tokenize(normalize(text), pattern))
    words = {tuple(w): f for w, f in word_freqs.items()}

    vocab = sorted({ch for w in words for ch in w})   # start: the alphabet
    merges = []

    while len(vocab) < vocab_size:
        pairs = Counter()
        for word, freq in words.items():
            for a, b in zip(word, word[1:]):          # every adjacent pair
                pairs[(a, b)] += freq                 # weighted by word frequency
        if not pairs:
            break

        best, count = pairs.most_common(1)[0]
        if count < 2:
            break                                     # nothing left worth merging

        merged = best[0] + best[1]
        words = {_merge_word(w, best, merged): f for w, f in words.items()}
        merges.append(best)
        vocab.append(merged)

    return vocab, merges

The words dictionary is the trick that makes this readable and fast: instead of walking the whole corpus on every iteration, we walk the set of distinct pre-tokens, weighted by how often each one appeared.

Watch the first merges it learns on Sanskrit — they are not arbitrary:

  merge   1: 'र' + '्' -> 'र्'  (12x)
  merge   2: 'त' + '्' -> 'त्'  (12x)
  merge   3: ' ' + 'स' -> ' स'  (12x)
  merge   5: '्' + 'य' -> '्य'  (8x)
  merge   8: 'ध' + 'र्' -> 'धर्'  (5x)
  merge   9: 'धर्' + 'म' -> 'धर्म'  (5x)

Merges 1, 2 and 5 are the model discovering the virama — that a consonant plus ् behaves as one unit. Merges 8 and 9 are it building the word धर्म out of pieces it learned two steps earlier. Nobody told it about Devanagari. It found the structure by counting, which is exactly what a language model does one level up.

4 · Encoding with the merge table 🔁¶

def _encode_chunk(self, chunk):
    parts = list(chunk)
    while len(parts) > 1:
        candidates = [(self.ranks[p], i)
                      for i, p in enumerate(zip(parts, parts[1:]))
                      if p in self.ranks]
        if not candidates:
            break
        _, i = min(candidates)                       # earliest-learned merge wins
        parts[i:i + 2] = [parts[i] + parts[i + 1]]
    return parts

Split the chunk into characters, then repeatedly apply the lowest-rank merge available until none applies. That is decoding the training process forwards.

5 · The measurement — your real deliverable 📐¶

def fertility(tokenizer, text):
    words = normalize(text).split()
    return len(tokenizer.tokenize(text)) / len(words)


def compression(tokenizer, text):
    tokens = tokenizer.tokenize(text)
    return len(normalize(text)) / len(tokens)

Measure both. Fertility is the number people quote, but it depends on whitespace, and “one word” is a slippery idea in a language with compounds that run to twenty syllables. Characters-per-token does not care about spaces, so when the two disagree, that disagreement is telling you something about your language.

What you should see ▶️¶

python bpe_from_scratch.py

corpus: 516 chars, 74 words

held-out text: धर्मस्य तत्त्वं ज्ञानेन प्राप्यते श्रीगणेशाय नमः

tokenizer      vocab   tokens  fertility  chars/token
-----------------------------------------------------
codepoint         46       48       8.00         1.00
grapheme         111       23       3.83         2.09
bpe              109       33       5.50         1.45

how each one cuts the first two words:
  codepoint    ध | र | ् | म | स | ् | य |   | त | त | ् | त | ् | व | ं
  grapheme     ध | र्म | स्य |   | त | त्त्वं
  bpe          धर्म | स्य |  त | त् | त् | व | ं

BPE round-trip (encode -> decode) is lossless. OK

Then the experiment that makes the point:

=== pre-tokenizer A/B (same BPE, same vocab size, only the regex differs) ===
pre-tokenizer     pre-tokens  fertility  chars/token
----------------------------------------------------
gpt2-ish                 369       7.17         1.12
indic-aware               87       5.50         1.45

Same algorithm. Different cut. That gap is free performance.

Same corpus. Same algorithm. Same vocabulary size. The English pattern produced 369 pre-token chunks where the Indic one produced 87, and fertility fell from 7.17 to 5.50 as a direct result. Nothing about BPE changed. Only the regex did.

6 · The two families, in one paragraph each 👨‍👩‍👧¶

You now know BPE from the inside, so the alternatives take thirty seconds.

• BPE glues pieces together, from small to large. It is what you just wrote, and it is what most current models use.

• Unigram (used by SentencePiece) works the other way: it starts with a large candidate list and repeatedly throws away the pieces that hurt the corpus likelihood least — from large to small.

Both work. Unigram tends to produce slightly more linguistically sensible pieces on morphologically rich languages, which makes it worth an experiment for Sanskrit. BPE is more common, better supported, and what Step 4 uses. Try both; report both.

Where people usually get stuck¶

Believing that a big multilingual tokenizer “supports” your language because the letters do not turn into question marks. Displaying correctly and tokenizing well are completely different things. A tokenizer can handle every Devanagari character perfectly and still be terrible at Sanskrit. Always measure fertility yourself. Never trust a claim of support.

Comparing tokenizers with different vocabulary sizes. A 64k tokenizer will beat a 32k one on fertility almost regardless of quality. Hold vocabulary size fixed, or the comparison means nothing.

Skipping the round-trip assertion. A tokenizer that cannot rebuild its own input is a silent data-loss bug that will not surface until your model is generating truncated text and you have no idea why. Two lines:

assert tok.decode(tok.encode(text)) == normalize(text)

Tuning on the training corpus. You will pick the tokenizer that memorised best, not the one that generalises best.

You are ready to move on when¶

You can measure the fertility of any tokenizer on any text file, using a script you wrote, and you have a table of results for Sanskrit against at least three tokenizers from different model families.

A good test: someone hands you a tokenizer and claims it is good at Telugu. You can either confirm or refute that claim in five minutes, with a number.