Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Step 7 — Clean your data 🧹

Goal: turn a pile of scraped text into a training corpus you trust — and a recipe someone else could run and get the same corpus.

Why this step matters

Small models are extremely sensitive to bad data. A large model trained on trillions of tokens can absorb some noise. Yours cannot. Every bad document is a larger fraction of what your model believes.

Digitised ancient texts have a particular problem: OCR errors. A scanner reading a hundred-year-old Devanagari print will make mistakes, and those mistakes become things your model confidently learns.

But the failure that actually ruins projects is quieter than either of those. The same verse appears across dozens of sources, with small differences in punctuation, transliteration, or attached commentary. Exact-duplicate removal catches none of it. Your model memorises instead of learning, your validation loss looks fine because the duplicates are on both sides of the split, and you find out in Step 10 — or worse, you don’t.

You already know what that looks like. It is experiment 5 from Step 2.

The shape of this chapter 🗺️

Six stages in a pipe. The order is not decorative — get it wrong and each stage quietly undoes the one before it:

Two ordering rules to internalise:

1 · Normalize, and detect the language 🧼

Same NFC rule as everywhere else in this book, plus whitespace collapsing:

def normalize(text):
    text = unicodedata.normalize("NFC", text)
    text = regex.sub(r"[ \t]+", " ", text)
    return "\n".join(ln.strip() for ln in text.splitlines()).strip()

Then language detection, which deserves a warning. Off-the-shelf detectors are poor at telling apart closely related South Asian languages, and they fall apart completely on code-mixed text.

2 · Filter for quality, and always say why 🚦

Start crude. Crude filters catch most of the damage:

def quality_report(text, min_chars=100):
    """Return (keep?, reason). Always return the reason."""
    if len(text) < min_chars:
        return False, "too short"
    if devanagari_ratio(text) < 0.60:
        return False, f"devanagari only {devanagari_ratio(text):.0%} (OCR failure?)"
    if repetition_ratio(text) > 0.30:
        return False, f"{repetition_ratio(text):.0%} repeated lines (headers/footers?)"
    return True, "ok"

Two of those three are catching real, specific, common failures:

def repetition_ratio(text):
    lines = [ln for ln in text.splitlines() if ln.strip()]
    return 1 - (len(set(lines)) / len(lines))

You can go further later — filtering by perplexity from a reference model, where anything the reference finds wildly surprising is usually broken text rather than interesting text. Do not start there. Start with the three rules above and read what they dropped.

3 · Exact duplicates — cheap, so do it first 🔁

def content_hash(text):
    return hashlib.sha256(text.encode("utf-8")).hexdigest()

Hash every document, drop repeats. Fast, exact, and it shrinks the input to the expensive stage that follows.

4 · Near-duplicates — the stage that actually matters 🔑

This is the one people skip, because exact dedup already ran and felt like the job was done. They are not the same thing, and the second one matters far more for Sanskrit.

MinHash is implemented in the file from scratch, in about thirty lines, for the same reason BPE was in Step 3: it is short enough to read, and once you have read it you can reason about the threshold instead of guessing at it.

a) Shingles — the unit of comparison

def shingles(text, k=5):
    """Character k-grams."""
    s = regex.sub(r"\s+", " ", text)
    return {s[i:i + k] for i in range(max(len(s) - k + 1, 1))}

Character shingles, not word shingles. Word shingles are the usual default and they are wrong here: Sanskrit word boundaries move around under sandhi, so the same passage can have completely different “words” and nearly identical characters. Compare the characters.

b) MinHash — a signature instead of a set

def minhash(text, num_perm=64, k=5):
    sh = shingles(text, k)
    sig = []
    for i in range(num_perm):
        salt = str(i).encode()
        sig.append(min(int(hashlib.blake2b(salt + g.encode(), digest_size=8).hexdigest(), 16)
                       for g in sh))
    return sig


def jaccard_estimate(a, b):
    return sum(1 for x, y in zip(a, b) if x == y) / len(a)

The trick is worth stating plainly: for each of 64 “permutations” we keep only the minimum hash. Two documents with similar shingle sets are likely to have the same minimum, so the fraction of matching signature slots estimates the Jaccard similarity of the original sets — an O(64) comparison instead of comparing two full sets of thousands of shingles.

c) LSH banding — so it finishes this century

Even at 64 numbers per document, comparing all pairs is quadratic. Banding fixes that:

rows = num_perm // bands
buckets = defaultdict(list)
for idx, d in enumerate(docs):
    for b in range(bands):
        band = tuple(d["_sig"][b * rows:(b + 1) * rows])
        buckets[(b, band)].append(idx)     # only compare docs sharing a band

Split each signature into 16 bands of 4, and only compare documents that share an identical band. Similar documents almost always collide in at least one band; dissimilar ones almost never do.

d) Which copy do you keep?

# Keep the longer document — it usually has the commentary attached.
keep, drop = (i, j) if len(docs[i]["text"]) >= len(docs[j]["text"]) else (j, i)

A one-line policy, and worth thinking about for your corpus. For classical texts the longer version usually carries commentary you want. For scraped news, the longer version is usually the one with the navigation menu attached, and you want the shorter one. Decide deliberately.

5 · Contamination — and the metric change 🕵️

Remove anything that appears in the test sets you plan to evaluate on. Do this now, before training. If you do it after, your Step 10 numbers are a lie and you will not know.

Here is the subtlety that catches almost everyone:

def contamination(docs, eval_texts, threshold=0.8):
    hits = []
    for n, ev in enumerate(eval_texts):
        ev_sh = shingles(ev)
        for d in docs:
            overlap = len(ev_sh & shingles(d["text"])) / len(ev_sh)   # containment
            if overlap >= threshold:
                hits.append((d["id"], n, overlap))
    return hits

Note the denominator: the size of the eval shingle set, not the union. That single change is the difference between finding your leaks and not.

6 · Split by source, never by line ✂️

def split_by_source(docs, ratios=(0.9, 0.05, 0.05)):
    by_source = defaultdict(list)
    for d in docs:
        by_source[d["source"]].append(d)

    out = {"train": [], "val": [], "test": []}
    for source, group in sorted(by_source.items()):
        r = int(hashlib.sha256(source.encode()).hexdigest(), 16) % 1000 / 1000
        name = "train" if r < ratios[0] else ("val" if r < ratios[0] + ratios[1] else "test")
        out[name].extend(group)
    return out

Split by line and the same verse appears on both sides — your validation loss then measures memorisation and calls it generalisation. Assign whole sources to splits, and hash the source name so the assignment is deterministic: re-running the pipeline gives you the same split, which is what makes results comparable across runs.

This is also the payoff for the one-folder-per-source layout from Step 6.

What you should see ▶️

python clean_corpus.py
loaded 7 documents

=== quality filter ===
  KEEP  gita-1           ok
  KEEP  gita-2           ok
  KEEP  gita-3           ok
  KEEP  subhashita-1     ok
  DROP  manuscript-ocr   devanagari only 0% (OCR failure or wrong language?)
  DROP  loop-scrape      95% repeated lines (headers/footers?)
  DROP  fragment         too short
  -> 4/7 survive

=== exact dedup ===
  DROP  gita-2           byte-identical to gita-1
  -> 3 remain

=== near dedup (MinHash, threshold 0.7) ===
  DROP  gita-1           ~77% similar to gita-3
  -> 2 remain

=== contamination check against the eval set ===
  LEAK  subhashita-1     ~100% similar to eval item 0
  -> 1 remain after removing leaks

=== split by source ===
  train   1 docs  sources=['sacred-texts']
  val     0 docs  sources=[]
  test    0 docs  sources=[]

wrote corpus/train.txt, val.txt, test.txt and manifest.json  (187 chars kept)

Seven documents in. One document out. Read the stages:

7 · Two data sources worth considering 🔀

Transliteration — nearly free extra data

Sanskrit is written in several scripts: Devanagari, Grantha, Telugu, Kannada, and Roman (IAST). Converting between them is mechanical and reliable.

This can meaningfully increase your usable data, and it also teaches your model that the same text can wear different clothes. The same trick works between Urdu script and Hindi Devanagari, since the spoken languages are very close.

Synthetic data — carefully

You can generate text with a large model, or translate text into Sanskrit. Two warnings:

8 · Fix OCR errors, and stop early ✏️

For scanned text, expect:

Build a list of the most common errors in your sources and fix them with rules. Perfect is not the goal — reducing the top ten error patterns gets you most of the benefit, and the eleventh through hundredth will cost you a month.

The same applies to personal information: names, phone numbers, addresses, and identifiers. Less of an issue for classical texts, a real issue for scraped Urdu web data.

Where people usually get stuck

Skipping near-duplicate removal because exact-duplicate removal already ran. They are not the same thing, and the second one matters far more for Sanskrit.

Trusting a MinHash threshold they never calibrated. 0.7 is a starting guess, not an answer. Run with --near-threshold 0.5 and 0.9 on your own corpus and read the pairs it reports. If it is dropping things you wanted, raise it. If the “nothing found” line appears, be suspicious rather than pleased.

Using Jaccard for contamination. Covered in stage 5, and it is the single most expensive mistake in this chapter because the failure is silent and only surfaces as inexplicably good Step 10 numbers.

Splitting by line. Or by document without grouping sources. Either way the same verse ends up on both sides.

Cleaning without a manifest. Six months from now someone will ask “where did this line come from?” and you will have no answer, and no legal basis for the release you were planning in Step 25.

You are ready to move on when

You have a clean corpus, a documented recipe someone else could run to reproduce it exactly, a manifest recording every surviving document’s source, and a train/validation/test split made at the source level with more than one source on each side.

A good test: delete your output folder and rebuild it from the raw data with one command. If you cannot, you have a corpus but not a recipe.

Part 2 - Feed the machine
6. Collect data
Part 3 - Train it properly
8. Modern architecture