Step 5 — Build an Urdu tokenizer

Step 5 — Build an Urdu tokenizer ☪️¶

Goal: repeat Step 4 for a script that behaves completely differently — and find out that the hard part is not the tokenizer at all.

Why this step matters¶

Sanskrit taught you about morphology — how words are built and joined.

Urdu will teach you about script and encoding — a separate skill, and one that catches almost everyone the first time.

Here is why it catches people. In Sanskrit, the problem was visible: you could print the tokens and watch स्कृ shatter. In Urdu, the problem is invisible. Two strings render identically on your screen, pass every eyeball check you can think of, and are different code points underneath. Your tokenizer sees two different words. You see one. Nobody is wrong and everything is broken.

The shape of this chapter 🗺️¶

Four decisions and one measurement. Three of the four have no universally right answer — but all four have the same universally wrong answer, which is letting different files make different choices by accident.

1 · Separate rendering problems from encoding problems 🎭¶

Urdu letters change shape depending on where they sit in a word: one form at the start, another in the middle, another at the end, another alone.

This is rendering. Unicode stores the same base letter regardless of shape, and your font does the shaping at display time. Your tokenizer never sees the shapes.

Beginners lose days here, writing code to handle something that is their font’s job. Do not be one of them. There is no shaping code anywhere in this chapter, and there should be none in yours.

2 · Fix the Arabic-versus-Urdu confusion ⚠️¶

This is the real encoding problem, and it is everywhere in scraped text.

Urdu and Arabic share a script but not a character inventory. Web-scraped Urdu is full of Arabic-form letters — from Arabic keyboards, from copy-paste, from software that assumed “Arabic script” means “Arabic”, from bad OCR. Often within the same document, sometimes within the same sentence.

To your model, کرو and كرو are two unrelated words.

ARABIC_TO_URDU = {
    "\u064A": "\u06CC",  # ي ARABIC YEH        -> ی FARSI YEH   (the most common by far)
    "\u0649": "\u06CC",  # ى ALEF MAKSURA      -> ی FARSI YEH
    "\u0643": "\u06A9",  # ك ARABIC KAF        -> ک KEHEH
    "\u0647": "\u06C1",  # ه ARABIC HEH        -> ہ HEH GOAL
    "\u0629": "\u06C3",  # ة TEH MARBUTA       -> ۃ TEH MARBUTA GOAL
    "\u06BE": "\u06BE",  # ھ DOACHASHMEE HEH   -> itself (listed so you don't "fix" it)
    "\u0660": "\u06F0", "\u0661": "\u06F1", "\u0662": "\u06F2",   # Arabic-Indic digits
    "\u0663": "\u06F3", "\u0664": "\u06F4", "\u0665": "\u06F5",   # -> Extended (Urdu)
    "\u0666": "\u06F6", "\u0667": "\u06F7", "\u0668": "\u06F8",
    "\u0669": "\u06F9",
}

3 · Deal with the zero-width characters 👻¶

The zero-width non-joiner (ZWNJ, U+200C) is an invisible character that stops two letters from joining. Urdu uses it a lot, and uses it inconsistently — the same word appears with and without it in the same corpus.

Two siblings travel with it, and they are not the same thing:

ZWNJ = "\u200C"   # stops letters joining — sometimes meaningful in Urdu
ZWJ  = "\u200D"   # forces letters to join — a rendering hint, not content
ZWSP = "\u200B"   # zero-width space — almost always accidental

ZWSP and ZWJ you can delete without thinking. ZWNJ is a real decision, which is why it is a flag rather than a hard-coded choice:

text = text.replace(ZWSP, "")                       # never intentional
text = text.replace(ZWJ, "")                        # rendering hint, not content
text = text.replace(ZWNJ, ZWNJ if keep_zwnj else "")   # ← your call

Decide on one rule. Apply it everywhere. Write down why.

4 · Decide about short vowels 🔉¶

Urdu usually does not write its short vowels. Sometimes they appear as diacritics — zer, zabar, pesh — especially in poetry, religious text, and teaching material.

DIACRITICS = "".join([
    "\u064B", "\u064C", "\u064D",   # tanween
    "\u064E", "\u064F", "\u0650",   # zabar, pesh, zer
    "\u0651", "\u0652",             # tashdeed, jazm/sukun
    "\u0653", "\u0654", "\u0655",   # maddah, hamza above/below
    "\u0670",                       # superscript alef
])

This means the same written word can be read several ways, and a human reader resolves it from context. Your model will have to do the same.

Stripping them makes your vocabulary smaller and your corpus more consistent. Keeping them preserves information that matters if you care about poetry or pronunciation. Both are defensible; only one can be true of your corpus. Decide, and be consistent.

5 · Make a deliberate decision about Roman Urdu 🔤¶

A very large share of Urdu written online uses English letters. This is called Roman Urdu, and there is no standard spelling for it.

Ask yourself directly: is that your language or not?

• If your users type in Roman Urdu, you must support it.

• If you only want Urdu script, you must filter it out on purpose.

The classifier is eight lines, and its job is to turn an accident into a decision:

def script_profile(line):
    arabic = len(ARABIC_SCRIPT.findall(line))
    latin  = len(LATIN.findall(line))
    total  = arabic + latin
    if total == 0:
        return "other"
    r = arabic / total
    if r > 0.9:  return "urdu-script"
    if r < 0.1:  return "roman/latin"
    return "mixed"

The mixed bucket is the one to look at hardest. Urdu online is heavily code-mixed with English, and those lines are often your most natural, most-representative user text.

What you should see ▶️¶

python urdu_normalize.py

=== script profile (before anything) ===
  urdu-script    9 lines
  roman/latin    2 lines

=== what normalization actually bought you ===
                         chars  unique chars  unique words
----------------------------------------------------------
before                     255            33            42
after                      252            28            27

15 'words' disappeared. They were never different words —
they were the same words spelled with Arabic-form letters.

=== why you must not trust your eyes ===
  کرو  ->  ['0x6a9', '0x631', '0x648']
  كرو  ->  ['0x643', '0x631', '0x648']
  equal on screen? yes.   equal to Python? False
  after normalize_urdu?   True

=== fertility on held-out Urdu ===
  grapheme tokenizer: 4.50 tokens/word (vocab 28)

Read the middle table carefully. The character count barely moved — 255 to 252 — so nothing was really deleted. But the vocabulary fell from 42 distinct words to 27, and unique characters from 33 to 28.

Those 15 vanished words were never words. They were the same eight sentences, spelled twice, and your model would have learned two separate half-strength representations of each one and never connected them.

6 · Measure fertility, and report an honest result 📐¶

Run the same fertility script from Step 3 on held-out Urdu.

You may be surprised. Urdu often has lower fertility than other South Asian languages under general-purpose tokenizers — it is better represented in the crawls those tokenizers were fitted on, and its orthography stacks less than Devanagari. So your gains may be much smaller than they were for Sanskrit.

That is a useful and honest result. Report it. Not every language has the same amount of headroom, and knowing where the headroom is not is valuable information — it tells you and everyone reading you where to spend effort.

Where people usually get stuck¶

Right-to-left text confusing them in the terminal. Text is stored in logical order — the order you would say it — not in the order it appears on screen. Your editor reverses it for display, and a mixed Urdu/English line will appear to have its parts in an order that no data structure agrees with.

Trust the code point sequence, never the rendering:

print([hex(ord(c)) for c in text])

Writing code to handle letter shaping. Covered in section 1, and worth repeating because it eats whole days. That is your font’s job.

Normalizing the doachashmee heh away. ھ is a real letter that changes meaning. Folding it into ہ merges genuinely different words. When in doubt, fold fewer characters and check what your vocabulary count does.

Applying the Sanskrit recipe unchanged. NFC still matters, but NFC alone does nothing about the Arabic/Urdu confusion — those are separate code points, not composition variants, so Unicode considers them correctly distinct. The mapping table is yours to maintain; no library will do it for you.

You are ready to move on when¶

You have working tokenizers for both Sanskrit and Urdu, with measured fertility numbers for both, and you can explain in plain words why the two languages needed completely different work.

You should also have four written-down decisions for Urdu — diacritics, ZWNJ, Roman Urdu, and your character mapping table — with a reason next to each.