Step 18 — Make it small and serve it

Goal: get your model running fast enough and cheap enough for real use.

Why this step matters¶

A model that only runs on a rented data-centre GPU is a demo. A model that runs on a laptop is a product.

For a Sanskrit model this matters more than usual. A lot of the value is in running locally and privately — sacred texts, personal study, a scholar’s unpublished work. If it needs a cloud, you have lost part of the point.

What you do¶

1. Understand quantization¶

Model weights are normally 16-bit numbers. Quantization stores them in 8, 4, or even fewer bits.

It is like keeping fewer decimal places. Much smaller, slightly less accurate.

2. Use post-training quantization¶

GPTQ and AWQ are the common methods. They need no retraining — just a small calibration dataset that they use to work out which weights matter most.

Use Sanskrit text for calibration, not English. The calibration data decides what the compression protects. If you calibrate on English, you protect English.

3. Test the quality after quantizing, in Sanskrit¶

4. Use GGUF and llama.cpp for local running¶

This is what gets your model running on a normal laptop, or on Apple Silicon. It is the format most local tools expect.

5. Understand the KV cache¶

When generating text, the model saves its keys and values for earlier tokens so it does not have to recompute them for every new token.

This cache is often the biggest memory user during serving, and it grows with context length.

This is why grouped-query attention from Step 8 mattered: fewer key and value sets means a much smaller cache.

6. Serve with vLLM or SGLang¶

They handle continuous batching (starting new requests as old ones finish, instead of waiting for the whole batch) and paged attention (managing the KV cache like an operating system manages memory).

Together these give a very large throughput improvement over a naive generation loop.

7. Measure three numbers, not one¶

• Throughput — total tokens per second across all users

• Time to first token — how long before anything appears

• Time between tokens — how fast the text flows once it starts

Users feel the second and third. Your dashboard usually shows only the first.

Where people usually get stuck¶

Quantizing, checking an English benchmark, seeing a small drop, and shipping — without ever testing the actual target language.

You are ready to move on when¶

Your model runs on hardware a normal person owns, and you have measured its quality after compression in Sanskrit.

🖥️ Serving privately on your own NVIDIA hardware¶

For healthcare especially, the data often cannot leave the building — so you serve on your own GPUs. Know what each layer of the NVIDIA stack is for:

# A NIM exposes an OpenAI-compatible endpoint — your app code barely changes.
docker run --gpus all -p 8000:8000 -e NGC_API_KEY=$NGC_API_KEY \
  nvcr.io/nim/meta/llama-3.1-8b-instruct:latest

💰 Is self-hosting worth it? Do the arithmetic first¶

Before you quantize and serve anything for a client, answer the CFO’s question: why not just call a big API? Sometimes the honest answer is “you should”. code/season-2/ep01-small-models-win/ turns the question into arithmetic:

• API cost = requests × tokens × price per token. Tokens = words × tokens-per-word for that language, plus the system prompt on every call.

• Self-host cost = hardware spread over its useful life + electricity + the hours someone spends looking after it + the one-off fine-tuning and data work.

• Capacity. Each request keeps a box busy for reading time (prompt tokens ÷ prefill speed) plus writing time (answer tokens ÷ decode speed). Reading (“prefill”) processes the whole prompt at once and is fast; writing (“decode”) makes one token at a time and is slow. Multiply by the busy-hour request rate; above 100% of one box, you need a second box.

• Break-even = the monthly volume where self-hosting becomes cheaper.

Two numbers in that arithmetic must be measured, not guessed: tokens-per-word for the client’s language (fertility.py, using Step 3’s idea), and reading and writing speed on your serving stack at realistic concurrency (sections 6 and 7 above).

With example prices, the ten business cases in the folder split like this:

cd code/season-2/ep01-small-models-win
python compare_scenarios.py             # the ten-case table
python cost_calculator.py --sweep 1000,5000000   # cost vs. volume, for a chart

💻 From adapter to laptop (Season 2, Episode 5)¶

Sections 1–4 above are the theory. This is the concrete path for a model you fine-tuned with LoRA in Episodes 2–4:

LoRA adapter → merge → bf16 GGUF → (importance matrix) → Q8_0 / Q5_K_M / Q4_K_M → Ollama

1. See quantization with your own eyes first. quantize_by_hand.py quantizes one weight matrix that has a few large outliers, as real LLM layers do. With one scale for the whole matrix, 4-bit rounding wipes out almost every small weight: the outliers set the scale. A scale per row, or per small group of weights, cuts the damage sharply for a fraction of a bit more. GGUF’s Q4_K_M, GPTQ and AWQ are refinements of that group-wise idea.

2. Merge, then check the merge. Laptop tools want one ordinary model, not base + adapter. merge_adapter.py adds the adapter’s matrices into the base weights (W + (α/r)·B·A) and then compares the merged model with base + adapter on a sample prompt. Same next-token choices, differences around 10⁻⁷: ship it. Different: don’t.

3. Calibrate on your languages. llama.cpp’s importance matrix is the GGUF version of section 2’s calibration data. export_gguf.sh builds it from the episode’s own training text with make_calibration.py, so the weights that matter for Urdu or Tamil are the ones protected.

4. Package the rules with the model. make_modelfile.py writes an Ollama Modelfile with the GGUF file, the client’s system prompt from prompts.json and temperature 0. Whoever runs the model gets the behaviour you tested.

5. Re-test every level, per language. compare_quants.sh scores each GGUF file through Ollama with the same evaluate.py and test set as Episodes 2–4; quant_table.py shows each score’s change from the bf16 reference, language by language. Choose the smallest file that stays within a point or two of the reference in every language: section 3’s warning in practice.

🏭 One box, many clients (Season 2, Episode 6)¶

Multi-LoRA serving. Because every client’s adapter shares the same frozen base model, one server can hold the base once and switch adapters per request. vLLM’s --enable-lora --lora-modules insuranceplus=… hamrah=… bazaarkart=… turns each adapter into a model name; a request with "model": "hamrah" runs the base plus HamRah’s adapter. Three clients, one GPU, one copy of the base weights. This is the business reason to keep adapters unmerged on a server, the opposite of the laptop case above.

A note on mixture-of-experts models (Step 16). A model like Sarvam-30B stores about 30 billion parameters but uses only a few billion for each token. It needs the memory of a 30B model and runs at roughly the speed of a small one: a good trade on a 128 GB DGX Spark, a bad one on a laptop.

Measure under load, with numbers a customer feels. load_test.py replays an episode’s real test messages at 1, 4, 16 or more simultaneous users and reports:

Choose the highest concurrency whose p95 you would still promise, and let --write-scenario put the measured requests per second into the Episode 1 cost calculator: the estimate becomes a measurement.

Put a gateway in front of the model. A model server answers prompts; a business needs more. gateway.py (standard-library Python, readable end to end) does six things for every message:

1. clean it with the same normalizer used in training;

2. prompt it with the client’s rules and policy text;

3. ask the right adapter;

4. check JSON answers: retry once with a stricter reminder, then fall back to a safe reply and a human;

5. route [ESCALATE], fraud intents and high-priority tickets to people;

6. log every exchange with phone, card, bank-account (IBAN), Emirates ID, UK National Insurance, Aadhaar and email numbers masked.

Authentication, HTTPS and rate limiting still belong in front of it; any reverse proxy or API gateway provides them.