Goal: connect a vision encoder to your language model, and use it to read manuscripts.
Why this step matters¶
The killer application here is OCR, and it feeds directly back into Step 6.
There are millions of unscanned or badly-scanned pages of Sanskrit and Urdu. The open tools for reading them are genuinely weak. Better OCR means more data, which means a better model, which means better OCR.
Large digitisation projects are working on exactly this problem. Their output is often openly published, which means you can build on it rather than duplicating it.
What to cover¶
The basic pattern¶
It is simpler than people expect and has three parts:
A vision encoder turns an image into a sequence of vectors.
A small projector — usually just a two-layer network — maps those vectors into your language model’s space.
Your language model reads those vectors exactly as if they were text tokens.
That is the whole architecture. The projector is the only new part.
Training, in two stages¶
Freeze both large models. Train only the projector, so the two learn to speak a common language. Cheap.
Unfreeze and instruction-tune together.
For manuscripts specifically¶
Handwritten Devanagari, and the many regional and historical letter forms
Nastaliq script for Urdu, which is genuinely hard because letters overlap and cascade
Palm leaf manuscripts, damage, staining, and faded ink
Layout: marginal notes, interlinear commentary, multiple hands on one page
Output should be structured, not a flat wall of text
Realistic scope¶
Building a small vision-language model on top of a small language model plus an off-the-shelf vision encoder is now a realistic weekend project. Building a production OCR system for damaged palm leaf manuscripts is not. Scope accordingly.
🧑💻 The same recipe reads manuscripts and X-rays¶
Whether it’s a Sanskrit manuscript page for OCR or a medical X-ray, the pattern is identical: encoder → head, using transfer learning.
For manuscript OCR, the transfer-learning spectrum is your friend: start from a pretrained vision (or vision-language) model and fine-tune on your script — you need far less labelled data than training from scratch. A modern route is to fine-tune a small vision-language OCR model (TrOCR-style: a ViT encoder + a text decoder) on (manuscript image → transliteration) pairs, which is the same encoder→decoder shape as the medical VLM in Step 24.