Step 6 — Collect your data

Step 6 — Collect your data 📚¶

Goal: build the biggest, cleanest Sanskrit corpus you can, count it with your own tokenizer, and find out honestly how big that actually is.

Why this step matters¶

This is not a preparation step. For a low-resource language, this is the project. Everything else in this book is easier than this.

It is also the step where you discover the single most important fact about your project — a number that decides your architecture, your training plan, and whether Step 9 or Step 11 is your real destination. Do not skip it, and do not guess the answer.

Most people guess. They assume more Sanskrit exists and that they simply have not found it yet, and they spend three weeks building a training pipeline for a corpus that will never arrive.

The shape of this chapter 🗺️¶

Steps 1 through 4 take a week. Step 5 takes ten seconds and decides the rest of the book.

1 · Gather from every source you can find 🔍¶

• Sanskrit Wikipedia and Wikisource dumps

• Digital Sanskrit archives — GRETIL, SARIT, the Digital Corpus of Sanskrit

• Public-domain printed books

• The language splits of large web crawls

• Scanned books put through OCR, with all the noise that brings

• Your own collection, if you have one — this is your moat

See the corpora appendix for a working list.

2 · Record where every single file came from 🧾¶

Source, date, licence, and how you got it. Do this while you collect, not at release time.

data/
├── gretil/
│   ├── SOURCE.md          # url, date pulled, licence, how
│   └── *.txt
├── wikisource/
│   ├── SOURCE.md
│   └── *.txt
└── my-scans/
    ├── SOURCE.md
    └── *.txt

Reconstructing this later is miserable, and without it you cannot legally release anything. The folder structure above also does a second job for free: token_budget.py and the Step 7 splitter both group by the parent folder, so “one folder per source” is what makes source-level splitting work later.

3 · Count your tokens — with your tokenizer 🔢¶

Use your Step 4 tokenizer, not a general-purpose one. The numbers differ by two to three times, and yours is the one that governs your training run.

def load_tokenizer(path):
    """Your Step 4 BPE if you have it; the grapheme fallback if you don't."""
    if path:
        from tokenizers import Tokenizer
        tok = Tokenizer.from_file(path)
        return ("step-4 BPE", lambda t: len(tok.encode(t).ids))
    return ("grapheme fallback", lambda t: len(regex.findall(r"\X", t)))

And count deduplicated tokens, not raw ones:

h = hashlib.sha256(text.encode()).hexdigest()
if h not in seen:
    seen.add(h)
    uniq_toks += n                # only unique documents count

4 · Compare against what you need 📊¶

A rough rule of thumb: a useful model wants somewhere between 5 and 20 tokens of training data for every parameter it has. The low end is “you can get away with it”; the high end is roughly where results stop improving.

BUDGET_LOW, BUDGET_HIGH = 5, 20

for label, params in SIZES:
    lo, hi = params * BUDGET_LOW, params * BUDGET_HIGH
    shortfall = lo / uniq_toks
    ...

What you should see ▶️¶

python token_budget.py --data ./data --tokenizer ../step-04-sanskrit-tokenizer/sanskrit_bpe.json

Here is the tool run on a small sample, so you can see the shape of the output:

tokenizer: grapheme fallback
documents: 3  (2 unique after exact dedup)
characters: 9,720
tokens (raw):        5,165
tokens (deduped):    3,405   <- the honest number

what different model sizes want:
   model        tokens wanted     you have    shortfall
--------------------------------------------------------
    100M        500.0M - 2.0B         3.4K     146,843x
    500M         2.5B - 10.0B         3.4K     734,214x
      1B         5.0B - 20.0B         3.4K   1,468,429x
      7B       35.0B - 140.0B         3.4K  10,279,001x

Note the first two lines already doing useful work: three documents went in, two came out, because the Gita was collected twice from two different sites. That is a third of the corpus gone before any real cleaning, on a corpus of three files.

Now run it on everything you have gathered. The shortfall column will still be a number with commas in it. That is the whole point of this chapter.

5 · Face the number 😮¶

You will almost certainly find that all the clean Sanskrit text in the world adds up to far less than the smallest row in that table. Perhaps a few hundred million tokens, and much of that repeated across sources.

Sanskrit lands firmly in the right-hand branch, and knowing that on day seven instead of day seventy is worth more than any other single thing in Part 2.

6 · Do the same for Urdu ☪️¶

You will find much more text, and much of it much dirtier. A different problem needing different solutions.

Run the same two scripts. Expect the raw number to look encouraging and the deduplicated number to fall hard — Urdu web text is heavily syndicated, with the same news articles republished across dozens of sites. Then expect Step 7 to take another large bite for quality.

The useful comparison is not “which language has more text” but which language has more text per unit of cleaning effort. Write both numbers down.

7 · Write down your answer to one question ✍️¶

What do you actually want this model to do?

Not “understand Sanskrit”. Something you could test:

• Translate classical texts into English faithfully?

• Split sandhi and parse grammar?

• Answer questions about a specific body of texts?

• Generate text in a particular classical style?

• Check and correct metre?

Your answer changes your data mix, your evaluation, and your architecture. A vague answer here produces a vague model later.

Where people usually get stuck¶

Assuming more data exists and that they just have not found it yet. Do the count. Trust the count. The count is the plan.

Counting raw tokens. Your corpus is smaller than it looks, and the gap between the raw and deduplicated numbers is the single best early predictor of how much Step 7 work is ahead of you.

Counting with a general-purpose tokenizer. Off by two to three times, in the direction that flatters you.

Collecting for a month before counting once. Count on day two, with whatever you have. The number will already tell you which branch of the diagram you are on, and everything after that is refinement.

Leaving provenance for later. There is no later. There is only a folder of files you cannot legally publish.

You are ready to move on when¶

You have a deduplicated token count for both languages, produced with your own tokenizer, you believe it, and you have written one clear sentence saying what your model is for.

You should also be able to say which branch of the decision diagram you are on, and why.