Step 4 — Build a Sanskrit tokenizer

Step 4 — Build a Sanskrit tokenizer 🕉️¶

Goal: train a tokenizer for Sanskrit, prove with numbers that it beats the general-purpose ones, and publish it.

Why this step matters¶

This is your first real, original contribution. It is small, it is measurable, and it is genuinely useful to other people. A tokenizer file and a fertility table are a publishable artifact that took you an afternoon.

It is also the step where Sanskrit stops being an example and starts being an interesting technical problem — because the thing that makes Sanskrit hard to tokenize is the same thing that makes it interesting: the written form hides the word boundaries.

The shape of this chapter 🗺️¶

1 · Normalize everything first 🧼¶

Apply NFC normalization to your whole corpus. Then look for the common Devanagari problems:

• The virama (halant, ्) marks a consonant with no vowel, and is how conjunct letters are formed. Different sources use it inconsistently.

• The nukta modifies a consonant and is sometimes present as a separate character and sometimes baked into a single code point.

• Vowel signs are occasionally written in different orders that render identically.

Count your unique characters before and after normalizing. The drop tells you how much silent duplication you had:

import unicodedata

raw = open("corpus.txt", encoding="utf-8").read()
nfc = unicodedata.normalize("NFC", raw)
print(f"unique code points: {len(set(raw))} -> {len(set(nfc))}")

2 · Fix the unit: a “character” is a lie 🔤¶

Before subwords, get the atom right.

🔴 The naive way, kept on purpose to measure the damage¶

The classic one-liner chars = sorted(set(text)) splits on code points. It is perfect for English and shreds Sanskrit into orthographic atoms.

class CharTokenizer:
    """Code-point level tokenizer. Correct for English, wrong for Devanagari."""
    name = "codepoint"
    def __init__(self, text):
        self.units = sorted(set(text))                 # each unit = one code point
        self.stoi = {u: i for i, u in enumerate(self.units)}
        self.itos = {i: u for u, i in self.stoi.items()}
    @property
    def vocab_size(self): return len(self.units)
    def encode(self, s):  return [self.stoi[c] for c in s if c in self.stoi]
    def decode(self, ids): return "".join(self.itos[int(i)] for i in ids)

🟢 The right way — grapheme clusters (≈ aksharas)¶

The regex module’s \X matches a full grapheme cluster — the chunk a human perceives as one character — in a single line.

import regex

class GraphemeTokenizer:
    """Grapheme-cluster tokenizer — the right default for Devanagari."""
    name = "grapheme"
    def __init__(self, text):
        self.units = sorted(set(self._split(text)))
        self.stoi = {u: i for i, u in enumerate(self.units)}
        self.itos = {i: u for u, i in self.stoi.items()}
    @staticmethod
    def _split(text):
        return regex.findall(r"\X", text)              # \X = one grapheme cluster
    @property
    def vocab_size(self): return len(self.units)
    def encode(self, s):
        return [self.stoi[g] for g in self._split(s) if g in self.stoi]
    def decode(self, ids): return "".join(self.itos[int(i)] for i in ids)

Running both on संस्कृतम् ज्ञानम् श्रीगणेशाय नमः:

Code-point tokens (32 of them):  स | ं | स | ् | क | ृ | त | म | ् | ...
Grapheme  tokens  (17 of them):  सं | स्कृ | त | म् | ज्ञा | न | म् | श्री | ...

The grapheme tokenizer produces 44% shorter sequences and a bigger-but-more-meaningful vocabulary.

3 · Train real BPE on top of the right unit 🧩¶

Grapheme clusters fix the atom. They do not fix fertility — you still get one token per syllable, and Sanskrit has a lot of syllables. Now we put Step 3’s BPE on top, using a production library this time.

The whole configuration is four lines, and the interesting one is the third:

from tokenizers import Tokenizer, Regex, models, pre_tokenizers, normalizers, decoders

INDIC_SPLIT = r"[\p{L}\p{M}]+|[\u0964\u0965]|\p{N}+|[^\s\p{L}\p{M}\p{N}]+"
LATIN_SPLIT = r"\p{L}+|\p{N}+|[^\s\p{L}\p{N}]+"          # the English assumption

def build_tokenizer(pattern="indic"):
    tok = Tokenizer(models.BPE(unk_token="<unk>"))
    tok.normalizer = normalizers.NFC()                    # ← never skip this
    rx = INDIC_SPLIT if pattern == "indic" else LATIN_SPLIT
    tok.pre_tokenizer = pre_tokenizers.Sequence([
        pre_tokenizers.Split(pattern=Regex(rx), behavior="isolated"),
        pre_tokenizers.ByteLevel(add_prefix_space=False, use_regex=False),
    ])
    tok.decoder = decoders.ByteLevel()
    return tok

\p{M} sitting inside the letter run is the entire Indic fix, carried over from Step 3. use_regex=False on the ByteLevel stage matters more than it looks: leave it on the default and ByteLevel re-splits with its own English-shaped pattern, quietly undoing the work you just did.

Training is then unremarkable, which is the point:

trainer = trainers.BpeTrainer(
    vocab_size=vocab_size,
    special_tokens=["<unk>", "<s>", "</s>", "<pad>"],
    initial_alphabet=pre_tokenizers.ByteLevel.alphabet(),
    min_frequency=2,
)
tok.train_from_iterator(lines, trainer)

4 · Pick a vocabulary size by sweeping, not guessing 📊¶

For a single language, 32,000 to 64,000 tokens is the usual starting range. But bigger is not automatically better, and the reason is a trade you can compute:

An embedding table is vocab_size × n_embd parameters, and it is duplicated at the output head. At n_embd = 256, going from 32k to 64k vocabulary costs you about 16 million parameters — which may be more than the rest of your model. You pay for vocabulary with model capacity.

So sweep it. --sweep trains at several sizes and prints fertility for each. Published work on Indic tokenizers finds returns flattening well before 200,000 tokens, and corpus size mattering less than you would expect past about 10 GB — but your corpus is not their corpus, so find your own flattening point.

5 · Decide what to do about sandhi ✂️¶

This is the interesting Sanskrit-specific question, and it has no settled answer.

Sandhi is the set of rules by which sounds change where words meet. When words join, the join changes both of them:

तत् + अपि  →  तदपि

So the “word” sitting in your text may be three joined words wearing a disguise. A tokenizer trained on raw text learns the disguise, not the words.

Do both. Measure both. The --sandhi flag runs the A/B for you, using a deliberately crude splitter — a handful of the most common vowel-join rules and nothing else:

SANDHI_RULES = [
    (regex.compile(r"ा(?=\p{L})"), "ा "),      # long-a often marks a join
    (regex.compile(r"ो(?=\s*\p{L})"), "ः "),   # visarga -> o before voiced sounds
    (regex.compile(r"श्च"), "ः च"),             # -aḥ + ca
]

The crudeness is deliberate. If a bad splitter already moves the number, a good one is worth the dependency. If it does not move the number at all, you just saved yourself a component that can be wrong at inference time. That is a real result either way, and it is exactly the kind of small honest experiment a large generalist team never bothers to run on Sanskrit.

Sanskrit compounds (samasa) raise the same question one level up: an entire descriptive phrase can be written as one word. Decide whether to split them at their natural joins or keep them whole — and again, measure rather than guess.

What you should see ▶️¶

python train_sanskrit_bpe.py --sweep --sandhi

corpus: 587 chars, 84 words, 14 lines
held-out: धर्मस्य तत्त्वं ज्ञानेन प्राप्यते श्रीगणेशाय नमः सत्यमेव जयते

=== experiment 1 — pre-tokenizer, vocab_size=800 ===
tokenizer                  vocab  tokens  fertility  chars/token
----------------------------------------------------------------
codepoint                     45      61       7.62         1.00
grapheme (Step 1)            119      32       4.00         1.91
BPE latin-split              316      55       6.88         1.11
BPE indic-split              359      43       5.38         1.42

=== experiment 2 — vocabulary size sweep ===
BPE indic v=300              300      57       7.12         1.07
BPE indic v=600              359      43       5.38         1.42
BPE indic v=1200             359      43       5.38         1.42
BPE indic v=2400             359      43       5.38         1.42

=== experiment 3 — sandhi splitting ===
BPE on raw text              359      43       5.38         1.42
BPE on sandhi-split          361      43       5.38         1.42

round-trip (encode -> decode) is lossless. OK
saved -> sanskrit_bpe.json

Three readings, and all three are the point of the chapter:

Experiment 1 works exactly as advertised. Same algorithm, same vocabulary budget — the Indic pre-tokenizer cuts fertility from 6.88 to 5.38 against the Latin one. That is the regex fix from Step 3, paying out.

Experiment 2 shows saturation, not improvement. Vocabulary 600, 1200 and 2400 all land on the same 359 real tokens. The trainer ran out of pairs seen twice and stopped. On this corpus, asking for a bigger vocabulary changes nothing except the size of your embedding table — which is the exact failure the sweep exists to catch.

Experiment 3 says the crude splitter did not help. 5.38 either way. On this corpus, on this splitter, sandhi splitting is not worth a dependency. Report that. A clean negative result is publishable and useful.

6 · Publish the numbers 📐¶

A simple table of tokenizer versus fertility, on real Sanskrit text, with the test set included, is something people will actually use and cite. Publish four things together:

• sanskrit_bpe.json — the tokenizer file itself

• the held-out test text you measured on

• the table, including the baselines you lost to

• the exact command that reproduces it

The baselines you lost to are the part that makes it credible.

Where people usually get stuck¶

Training the tokenizer on text that was never normalized. Covered above, and still the most common mistake in this chapter.

Measuring on the training corpus. Every tokenizer looks excellent on the text it was fitted to. Hold out text your tokenizer never saw, and hold it out before you start.

Comparing across different vocabulary sizes. A 64k tokenizer beats a 32k one on fertility almost regardless of quality. Hold the budget fixed.

Leaving use_regex=True on the ByteLevel pre-tokenizer. It silently re-splits with an English-shaped pattern and undoes your Indic fix. Your fertility will barely move and you will blame BPE.

Chasing fertility off a cliff. Fertility down and model quality down at the same time is possible: a huge vocabulary gives you great fertility and terrible per-token statistics, because most tokens are now rare. Fertility is a diagnostic, not the goal.

You are ready to move on when¶

Your tokenizer beats the general-purpose ones on fertility on held-out Sanskrit, you have the table to prove it, and you can explain in one sentence which of the three experiments moved the number and which did not.