Step 10 — Test it honestly

Step 10 — Test it honestly 🔬¶

Goal: find out whether your model is actually any good — with a test suite you built, that a stranger could re-run, that does not flatter you.

Why this step matters¶

A falling loss curve does not mean a good model.

This step separates a real project from a demo. It is also, for Sanskrit and Urdu, mostly unexplored ground — which means the work you do here may be more valuable to other people than your model.

The characteristic failure of a small model on a rich language is fluent nonsense: output that has the right shapes, the right rhythm, and no meaning. Your loss curve cannot see it. Perplexity cannot see it. A human sees it in about four minutes, and a rule-based checker sees it instantly.

The shape of this chapter 🗺️¶

Cheap and weak at the top; expensive and decisive at the bottom. Use all five. Trust the bottom two.

1 · Perplexity, and the trap 📐¶

Perplexity roughly means “how surprised is the model by this text”. It is exp(average cross-entropy), so it is your training loss wearing a different hat. Lower is better. It is easy to compute and a genuinely useful sanity check.

It is also, used carelessly, worse than useless:

The fix is to normalize by something the tokenizer cannot change:

def perplexity(loss_per_token):
    return math.exp(loss_per_token)


def bits_per_char(loss_per_token, n_tokens, n_chars):
    """Cross-entropy comes out in nats, so divide by ln 2 for bits. Then
    rescale from 'per token' to 'per character' by the compression ratio.
    Nothing here depends on the vocabulary, which is the point."""
    return (loss_per_token / math.log(2)) * (n_tokens / n_chars)

perplexity.py demonstrates the trap by training two identical models — same seed, same architecture, same steps — on the same text with two different tokenizers:

tokenizer      vocab   tokens     loss   perplexity   bits/char
--------------------------------------------------------------
codepoint         39     4128   2.6454        14.09      3.8166
grapheme          79     2220   2.7891        16.27      2.1640

lowest perplexity: codepoint
lowest bits/char:  grapheme

The two metrics disagree, and bits/char is the one telling the truth.

The code-point model “wins” on perplexity because it only ever chooses between 39 options — an easier question, asked more times. On the measure that counts, it is worse by nearly a factor of two.

If you report only perplexity, you will pick the wrong model and never know.

2 · Accept that the benchmarks you need do not exist 🕳️¶

Most multilingual benchmarks for these languages are machine translations of English benchmarks. They test translation quality as much as language ability. For classical Sanskrit specifically, there is very little that tests what you actually care about.

That is not a reason to skip evaluation. It is the reason this chapter is mostly about building your own.

3 · Build tests around the parts of your language that have rules ✅¶

This is the highest-value idea in the chapter, and it is the Panini advantage cashed out as a test set instead of a model.

Metre is fully rule-based, which means it is fully checkable — no human in the loop, no LLM judge, no argument about what counts as correct. A line either scans or it does not.

The weight rules¶

A syllable is GURU (heavy) if
   - its vowel is long (आ ई ऊ ॠ ए ऐ ओ औ and their matras), OR
   - it is followed by anusvara (ं) or visarga (ः), OR
   - two or more consonants follow before the next vowel
Otherwise it is LAGHU (light).

The one implementation detail that matters¶

for n, s in enumerate(syls):
    stop = syls[n + 1]["after"] if n + 1 < len(syls) else len(text)
    between = text[s["after"]:stop]
    s["nasal"] = any(c in (ANUSVARA, VISARGA, CANDRABINDU) for c in between)
    # Two or more consonants before the next vowel = a conjunct = heavy.
    s["conjunct"] = sum(1 for c in between if c in CONSONANTS) >= 2

Note that we count consonants between two vowels, not within an orthographic syllable. In धर्म the र् is written attached to म, but for metre it closes the previous syllable — dhar-ma, heavy then light. Get this wrong and every conjunct in your corpus scans incorrectly.

The anustubh check¶

An anustubh (shloka) has four padas of eight syllables. Within each pada: syllable 5 is laghu, syllable 6 is guru, and syllable 7 is guru in odd padas and laghu in even ones. That is the whole metre, and it covers most of the Ramayana, most of the Mahabharata, and the Bhagavad Gita.

What you should see ▶️¶

python sanskrit_evals.py

Gita 1.1a-b:  धर्मक्षेत्रे कुरुक्षेत्रे समवेता युयुत्सवः
   धG  र्मG  क्षेG  त्रेG  कुL  रुG  क्षेG  त्रेG  सL  मL  वेG  ताG  युL  युG  त्सL  वG
   pada 1  GGGGLGGG  OK
   pada 2  LLGGLGLG  OK
   valid anustubh: True

deliberately broken:  विद्या विद्या विद्या विद्या
   विG  द्याG  विG  द्याG  विG  द्याG  विG  द्याG
   pada 1  GGGGGGGG  BAD syllable 5 should be laghu
   valid anustubh: False

Check one syllable by hand and you will trust the rest. कु in कुरुक्षेत्रे is short u followed by a single र — laghu. The next syllable रु is short u followed by क्ष, two consonants — guru. That is exactly what the scanner printed, on the opening line of the Gita, with no model involved.

Grading a model’s answers¶

python sanskrit_evals.py --grade predictions.jsonl

  MISS [metre] विद्या विद्या विद्या विद्या  truth=False said=True
  MISS [sandhi] विद्यार्थी  want='विद्या अर्थी' got='विद्य अर्थी'
  MISS [completion] विद्या ददाति विनयं  exact=False scans=False

task              n   accuracy
------------------------------
completion        2       50%
metre             3       67%
sandhi            3       67%

Every miss is printed with its detail, because an aggregate you cannot drill into is a number you cannot act on. Note the third one: the completion was both wrong and unmetrical, which tells you something an exact-match score alone would not.

4 · More tasks worth building 🛠️¶

Good Sanskrit tasks, all mechanically checkable:

• Sandhi splitting. Give a joined form, ask for the parts.

• Compound splitting. Break a samasa into its members.

• Grammatical agreement. Which form is correct in this sentence?

• Metre. Implemented above.

• Verse completion. Complete a verse from a known text, check against the real one — and check that it scans.

• Faithful translation. Into English or Hindi, scored by someone who knows both. Not mechanical, and worth it anyway.

Good Urdu tasks:

• Correct reading of a word with unwritten short vowels, given context

• Roman Urdu to Urdu script conversion

• Formal versus informal register

• Poetry metre, which Urdu also has strict rules for

5 · Use an LLM as a judge carefully ⚖️¶

Having a large model score your outputs is fast and useful. It also has known biases:

• It prefers longer answers

• It prefers whichever answer it saw first

• It prefers text that sounds like its own writing

For Sanskrit there is a fourth, and it is worse: the judge is also bad at Sanskrit. You are asking a model that produces grammatically wrong Sanskrit to grade grammatically wrong Sanskrit. Never use it as your only measure, and calibrate it against your rule-based tasks before you trust it on anything else.

6 · Do human evaluation 👤¶

Even 30 examples reviewed by one person who genuinely knows the language will teach you more than 3,000 automatic scores.

For Sanskrit this usually means finding a scholar. It is worth the effort. In philosophical and religious material, a confident wrong answer is much more costly than an uncertain one, and only a human will catch the difference — a rule checker will happily pass a metrically perfect statement that is theologically absurd.

7 · Check you did not break the base model 🧠¶

If you came from Step 11, also re-run general benchmarks. A model that got much better at Sanskrit and much worse at everything else has not been improved; it has been damaged in a direction you were not measuring. See the note on catastrophic forgetting there.

And re-read your Step 7 contamination log before you believe any number on this page. If your test verses were in the training data, every score here is fiction.

Where people usually get stuck¶

Reporting only perplexity, because it is easy, and never finding out that the model produces fluent nonsense. Perplexity will not show it to you. A human will, in about four minutes.

Ranking models by perplexity across different tokenizers. Demonstrated above. Report bits per character.

Building an eval and not stating its limits. Your pathya-only checker will be used by someone as a general metre validator unless you say otherwise, loudly, in the output.

Evaluating on data that was in training. Step 7 exists for this. Check the log, do not assume.

Reporting accuracy without n. 67% of three items is not 67%. Print n next to every number, always.

Waiting until the model is finished to build the eval. Build it in Step 6, before you have anything to grade. An eval built after the fact tends to test what your model happens to do well.

You are ready to move on when¶

You have a small evaluation suite you built yourself, honest numbers from it with n reported next to each, bits per character rather than raw perplexity, and at least one human review from someone who knows the language.

A good test: hand your eval set and grading script to a stranger. If they can run it and get your numbers, you have an evaluation. If they cannot, you have a demo.