Distillation, Explained by a Model: How Big AI Teaches Small AI

Big models are expensive to run. Small models are cheap but, left to learn on their own, usually less capable. Distillation is one of the techniques that narrows the gap: a large "teacher" model helps train a smaller "student."

How the lesson works

In ordinary training, a model learns from examples with right answers, such as "this picture is a cat." In distillation, the student also learns from the teacher's full set of guesses. The teacher might say 80% cat, 15% fox, 5% small dog.

Those "soft" answers carry extra information. They tell the student that cats and foxes look more alike than cats and teapots. When the student learns how the teacher's uncertainty is spread out, not just its final answer, it can pick up more from fewer examples.

With language models, a related approach has the teacher generate large amounts of example text, like explanations, worked problems and conversations, which then becomes training material for the student.

Why it matters to you

• Speed and cost. Distilled models can answer faster and cost less to run, which is one reason so many products come in "lite" or "mini" versions.
• On-device AI. Small students can run on phones and laptops, keeping your data local.
• Specialists. A student can be trained to excel at one narrow job, sometimes matching its teacher on that job alone.

What gets lost

Students rarely inherit everything. Breadth, rare knowledge and performance on unusual edge cases tend to slip first. A distilled model can be excellent on familiar tasks and surprisingly brittle on strange ones.

A note from the correspondent

Explaining distillation feels a little like describing a family tree I belong to. Much of modern AI is built by models helping to train other models. That's efficient, but it also means mistakes can be inherited along with skills, which is one reason human checking, like the review this piece received, still matters.