Model Distillation 2026: How to Train Small Models From Big Ones, With Real Tools
A 70B teacher can produce a 7B student that keeps most of the quality - that is distillation, and it is how cheap models ship. Here is the full loop with open tools: data, hyperparameters, and the failure modes that waste a week.
📜 Table of Contents
Model Distillation 2026: Training Small Models From Big Ones
A 70B teacher can produce a 7B student that keeps most of the quality.
What Is Actually Learned
| Type | Student learns | Data | Cost |
|---|---|---|---|
| Response | final answers | prompt + answer | low |
| Rationale | intermediate reasoning | prompt + reasoning + answer | medium |
| Distribution | teacher token probabilities | logits | high, needs local teacher |
Most teams need the first two. Distillation compresses behaviour on your task distribution - the student cannot learn what the teacher does not know.
Offline vs Online
| Mode | How | Pro | Con |
|---|---|---|---|
| Offline | teacher generates a dataset once | cheap, reproducible, filterable | no feedback on new samples |
| Online | teacher scores during training | tighter alignment | inference cost scales with steps |
Start offline.
Six Steps
- Collect prompts from real logs, 1k-10k covering your task types.
- Generate with the strongest model, asking for reasoning before the answer, save as jsonl messages format.
- Filter - rule filters (length, format, refusals, duplicates) plus a model-based 1-5 score keeping only high scores. This is the highest-leverage step.
- SFT with LoRA:
from datasets import load_dataset
from trl import SFTTrainer, SFTConfig
ds = load_dataset("json", data_files="distill.jsonl", split="train")
trainer = SFTTrainer(model="Qwen/Qwen2.5-7B-Instruct", train_dataset=ds,
args=SFTConfig(output_dir="student", num_train_epochs=2,
learning_rate=2e-4, per_device_train_batch_size=2,
gradient_accumulation_steps=8, max_length=4096))
trainer.train()
APIs shift between versions; follow current TRL docs.
- Evaluate on 200-500 held-out samples: teacher vs student vs base. Check accuracy, format compliance, and output length (students learn to ramble).
- Iterate on data, not epochs. A class of failures usually means 200 more teacher samples for that class.
Hyperparameters
| Param | Start | Note |
|---|---|---|
| Dataset size | 1k-10k | quality over quantity |
| Epochs | 1-3 | overfitting is common |
| LR (LoRA) | 1e-4 to 2e-4 | full FT an order lower |
| LoRA rank | 16-64 | higher for further domains |
| max_length | covers 95% of samples | truncation silently drops answers |
Failure Modes
- Training without filtering - students inherit hallucinations and sloppy format.
- Capacity mismatch - multi-step tasks need more than 1.5B.
- Learning answers but not reasoning - breaks on rephrasing.
- Eval leakage - split by source, not randomly.
FAQ
Distillation vs fine-tuning? Distillation is where data comes from; fine-tuning is how you train.
Can the student beat the teacher? On a narrow task, yes. In general capability, no.
VRAM? 7B LoRA at 4-bit fits a large consumer card. Start there.
Closed teachers? Technically possible, but check the provider's terms on training competing models.
Note: interfaces and VRAM needs change; follow TRL, Unsloth and PEFT docs.
Related
Written by our editorial team; tools listed here are tested or verified against public sources. Links point to official sites or GitHub repos for reference only — no paid placements.