Goal: let the model work with a whole text instead of a paragraph.
Why this step matters¶
Sanskrit texts are long, highly structured, and full of cross-references. A commentary refers to a verse, which refers to an earlier chapter. A model that only sees 2,000 tokens cannot do this work.
There are two different tools here, and people confuse them:
Long context — make the model itself able to read more at once.
RAG — retrieve the relevant passages and put only those in the context.
You usually want both.
Part A — Long context¶
Understand why a short-trained model fails on long text. It has never seen those position values before, so it is being asked to do something genuinely unfamiliar.
Learn position interpolation and its improved versions (NTK-aware scaling, YaRN). The idea is to squeeze the position values into the range the model already knows, so nothing looks unfamiliar.
Do a short continued-pretraining run on long documents after extending the context. Interpolation alone gets you part of the way.
Be sceptical of needle-in-a-haystack tests. Finding one planted fact in a long document is much easier than understanding the document. Build tests that need information from several places at once.
Watch the KV cache grow. Long context is mostly a memory problem, not a modelling problem.
Part B — RAG¶
RAG stands for Retrieval-Augmented Generation. Instead of making the model read everything, you search your corpus for the relevant passages and put only those in the prompt.
This is unusually effective for classical texts. Your entire corpus may be small enough that retrieval is fast and accurate, and some individual works fit in context whole.
Chunk by structure, not by character count. Split at verse and chapter boundaries, not every 512 characters. Structure is meaningful here in a way it is not for web text.
You need a Sanskrit embedding model. General multilingual embedding models have the same weaknesses as general tokenizers. Test before trusting.
Retrieval quality usually matters more than model size. A good retriever plus a small model beats a large model with bad retrieval, and costs far less.
Always cite the source passage in the output. For scholarly use this is not optional. A claim without a citation is not useful to a researcher.
Part C — Cross-lingual search and RAG in practice 🔎¶
India’s documents and its users rarely share a script. The policy is in English, the verse is in Devanagari, and the question arrives in Tamil, or in Hinglish typed in Latin letters. Episode 8 builds search that works across that gap, and measures it.
1. Start with a letter-based baseline, and watch it fail across scripts.
embedders.py’s hash embedder turns text into hashed character pieces: no
model, no download. On the episode’s 35 answerable test questions it finds the
right passage in the top five 92% of the time when question and passage share
a script, and 9% when they don’t. A Tamil question shares no letters with a
Devanagari verse. That gap is the whole reason multilingual embeddings exist.
2. Use a multilingual embedding model. An embedding model turns a whole
text into one vector, trained so that the same meaning in different languages
lands in the same region. Good starting points: intfloat/multilingual-e5-small
(small and fast; it expects query: and passage: prefixes, which
embedders.py adds), BAAI/bge-m3 (stronger), and LaBSE. Test them on your own
questions before trusting any of them (point 3 above).
3. An index is just a matrix. Every passage becomes one row of unit-length vectors; a search is one matrix multiplication (cosine similarity) and a sort. That is fine up to about a million passages; beyond that, FAISS, pgvector or Qdrant do the same with less work.
4. Grade retrieval like everything else. eval_retrieval.py reports
recall@1, recall@5 and mean reciprocal rank, per question language, and split
into same-script and cross-script questions. The cross-script column is the one
that tells you whether the embedding model is doing its job.
5. Hybrid search: meaning plus letters. Meaning-based search finds
paraphrases and other languages but can miss exact codes such as 56767 or
BAL 8080; letter-based search is the opposite. Reciprocal rank fusion adds
1 / (60 + rank) from each ranking, needs no score calibration, and usually
beats either alone.
6. RAG with citations and refusals. rag.py pastes the top passages into
the prompt, each labelled with its id, and instructs the model to answer only
from them, cite ids in square brackets, and reply [NOT_FOUND] when they don’t
contain the answer. --eval grades four things on every question:
| Check | Question it answers |
|---|---|
retrieved | did the right passage reach the model at all? |
cited_right | does the answer cite the passage that contains it? |
cites_valid | was every cited id really in the prompt, so no invented sources? |
not_found | does it refuse exactly when the documents don’t answer? |
This is how Episode 2’s Gita explainer grows from 18 verses to all 700, and how Episode 6’s gateway can attach only the relevant policy paragraphs instead of the whole policy.
You are ready to move on when¶
Your system can answer a question that requires information from two different parts of a long text, and it shows you where it got each part.