Step 24 — Medical images and heart sounds

Goal: see how much of this book transfers to a completely different domain.

Why this step matters¶

This chapter exists to prove a claim made in the introduction: almost everything you learned here transfers.

Medical imaging and biosignals are classic low-resource domains, with the same shape of problem as Sanskrit: scarce high-quality labelled data, expensive expert annotation, no good general benchmarks, and domain shift between sources.

If you can build a Sanskrit model, you can build a chest X-ray classifier. The thinking is roughly 70 to 80 percent the same.

What transfers directly¶

The data-centric mindset. Cleaning, deduplication, careful splits, handling imbalance, expert validation. Medical data has the same problems as low-resource text, often worse.

Adaptation strategy. Start from a strong pretrained backbone. Do continued pretraining on your domain. Use LoRA or QLoRA for the fine-tuning. This transfers almost exactly.

Training fundamentals. Optimizers, learning rate schedules, reading loss curves, spotting overfitting. Identical.

Efficiency work. Mixed precision, gradient checkpointing, quantization, local serving. Identical.

Evaluation discipline. Held-out sets, careful metric choice, expert review. Even more important here, because a confident wrong answer has consequences.

What changes¶

The biggest conceptual difference: there is no direct equivalent of tokenization. But you still make critical preprocessing choices — patch size and resolution for images, spectrogram parameters for audio — and those choices have the same kind of downstream importance that tokenization does. The lesson of Step 3 transfers even though the mechanism does not.

A practical starting path¶

1. Start from existing strong backbones. Do not build from scratch again unless you want the learning experience a second time.

   • Images: MONAI is the standard medical imaging library, plus vision transformer backbones or a medical foundation model

   • Audio: torchaudio, plus an audio spectrogram transformer or a self-supervised speech backbone

2. Use the same efficient fine-tuning toolkit you used in Step 11.

3. Spend most of your time on data preparation and expert validation. This will feel very familiar.

4. For multimodal work — X-ray plus radiology report — the pattern is the same as Step 21.

🧑‍💻 The universal recipe: encoder → head¶

Here is the idea that makes ECG, X-ray, and video feel like one subject instead of three: almost every model is an encoder that turns raw input into vectors, then a head (or decoder) that produces the output. Fine-tuning is just “adapt part of that stack to my data” — the exact same transfer-learning spectrum you met with LoRA in Step 11.

❤️ Heartbeat → arrhythmia (a 1D signal)¶

An ECG is a time series of voltages, usually 12 leads. Real datasets: PTB-XL (~21k clinical 12-lead recordings) and MIT-BIH. The encoder is a 1D CNN — Conv1d slides over time instead of space. Full runnable file (with synthetic data so it runs today): code/step-24-medical-ecg/ecg_arrhythmia.py.

🧱 A 1D ResNet block¶

import torch.nn as nn

class ResBlock1D(nn.Module):
    def __init__(self, c_in, c_out, stride=1):
        super().__init__()
        self.conv1 = nn.Conv1d(c_in, c_out, 7, stride, padding=3, bias=False)
        self.bn1 = nn.BatchNorm1d(c_out)
        self.conv2 = nn.Conv1d(c_out, c_out, 7, 1, padding=3, bias=False)
        self.bn2 = nn.BatchNorm1d(c_out)
        self.down = (nn.Sequential(nn.Conv1d(c_in, c_out, 1, stride, bias=False),
                                   nn.BatchNorm1d(c_out))
                     if (stride != 1 or c_in != c_out) else nn.Identity())
        self.act = nn.ReLU(inplace=True)
    def forward(self, x):
        r = self.down(x)
        x = self.act(self.bn1(self.conv1(x)))
        x = self.bn2(self.conv2(x))
        return self.act(x + r)      # residual — same trick as the transformer block

⚖️ The clinical move that matters most: imbalance¶

Dangerous arrhythmias are rare. A model that always says “normal” can score 98% accuracy and catch zero arrhythmias. So we weight the loss by inverse class frequency and evaluate with per-class recall / AUPRC, never accuracy:

import torch, torch.nn as nn
counts  = torch.tensor([4000., 250., 600., 150., 500.])       # very imbalanced
weights = counts.sum() / (len(counts) * counts)               # up-weight rare classes
criterion = nn.CrossEntropyLoss(weight=weights)               # evaluate with recall, not accuracy

🦴 X-ray → fracture / findings (a 2D image)¶

Datasets: ChestX-ray14, CheXpert, MURA. The recipe is transfer learning: take a pretrained backbone (or a medical one like MedSigLIP), replace the head, fine-tune. Full runnable file: code/step-24-medical-xray/xray_finetune.py.

import torch, torch.nn as nn, torchvision

def build_model(n_classes=2, pretrained=True, freeze_backbone=False):
    m = torchvision.models.resnet50(weights="IMAGENET1K_V2" if pretrained else None)
    if freeze_backbone:                        # "linear probing": train only the head
        for p in m.parameters():
            p.requires_grad = False
    m.fc = nn.Linear(m.fc.in_features, n_classes)   # new head: fracture / normal
    return m

criterion = nn.CrossEntropyLoss(weight=torch.tensor([1.0, 3.0]))  # up-weight rare "fracture"

🖼️➡️📝 Generative: fine-tuning a medical VLM (MedGemma)¶

A vision-language model reads an image and a question and writes text — a radiology report, a visual answer. It’s the encoder→projector→LLM pattern: MedGemma is Gemma 3 with MedSigLIP as its eyes. You fine-tune it on (image, prompt, target-text) triples — mechanically it’s LoRA SFT (Step 11) with the image passed through the processor.

import torch
from transformers import AutoProcessor, AutoModelForImageTextToText
from peft import LoraConfig, get_peft_model

proc  = AutoProcessor.from_pretrained("google/medgemma-4b-it")
model = AutoModelForImageTextToText.from_pretrained(
    "google/medgemma-4b-it", dtype=torch.bfloat16, device_map="auto")
model = get_peft_model(model, LoraConfig(     # keep the heavy vision tower frozen
    r=16, lora_alpha=32, task_type="CAUSAL_LM",
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"]))