Model Distillation 2026: How to Train Small Models From Big Ones, With Real Tools

A 70B teacher produces a 7B student that keeps most of the quality - that is distillation, and it is how companies ship cheap models. How does it actually work with open tools?

💡 What You Will Learn

A 70B teacher produces a 7B student that keeps most of the quality - that is distillation, and it is how companies ship cheap models. How does it actually work with open tools?

📜 Table of Contents

• The Distillation Idea in One Sentence
• Why It Matters in 2026
• Two Flavors
• The Practical Open-Source Stack
• The Standard Recipe
• The Failure Modes
• The Honest Expectation

The Distillation Idea in One Sentence

Train a small student model to imitate a large teacher model, using the teacher's outputs (including its confidence distribution) as training signal. The student learns the teacher's reasoning shortcuts without the teacher's size.

Why It Matters in 2026

Distillation is the economics layer of AI: DeepSeek distilled R1's reasoning into 1.5B-70B variants that run on commodity hardware, and every major lab ships distilled models alongside flagships. For a team, a distilled 7B beats a raw 7B trained from scratch on most tasks - and costs a fraction of running the 70B.

Two Flavors

1. Offline (static) distillation - run the teacher once on a dataset, save its outputs (logits or answers), train the student offline. Simple, one-time teacher cost.
2. Online distillation - the teacher scores the student during training (e.g. via preference feedback or reward). Better alignment, more expensive.

The Practical Open-Source Stack

The Standard Recipe

1. Take 10k-100k high-quality prompts.
2. Run the teacher, collect answers (with reasoning traces if the teacher shows them).
3. Filter: keep only outputs that pass a quality check (an LLM judge or rules).
4. Train the student with SFT on those (prompt, answer) pairs.
5. Evaluate the student against the teacher on a held-out set; iterate.

The Failure Modes

• Imitation of garbage - if the teacher outputs are unfiltered, the student learns the teacher's errors. Filtering is the highest-leverage step.
• Wrong student size - a 1B student cannot absorb a 70B teacher's reasoning; match capacity to task complexity.
• Benchmark chasing - distilling toward a leaderboard instead of your use case produces a model that is great at the benchmark and wrong at your job.

The Honest Expectation

A well-distilled student typically keeps 85-95% of the teacher's quality on the target task at 5-20x less inference cost. That trade is why distillation is the standard 2026 move for shipping models at scale.