Most teams approach model distillation as a science project and then wonder why it never ships. The fix is to treat it as an operation with named plays, clear triggers, and assigned owners, the same way you would run a sales motion or an incident response. This playbook gives you that operating structure.
A quick grounding before the plays: model distillation trains a small student model to imitate a larger teacher model. The technique is settled. What is usually missing is the decision-making scaffolding around it, who decides to distill, when, with what data, and what "good enough" means. That scaffolding is what we build here.
If you want the conceptual foundation underneath these plays, read The Complete Guide to What Is Model Distillation first. This document assumes you already buy the why and want the how-we-run-it.
Play 1: The Cost-Pressure Distillation
Trigger: Your inference bill on a specific, high-volume task crosses a threshold that makes the finance team nervous, and the task is narrow enough to imitate.
This is the most defensible play because the business case is a spreadsheet. You are spending real money calling a large model for a repetitive task, and a small student could do most of it.
How to run it
- Owner: an ML engineer with a product manager attached for the quality bar.
- Inputs: the last 30 to 90 days of real production traffic for the task, deduplicated.
- Success gate: student matches the teacher on the held-out set within an agreed tolerance, and projected savings exceed the build and serving cost within a defined payback window.
The trap here is optimizing for raw savings while quietly degrading quality. Bind every cost play to a hard quality floor, and track both numbers on the same dashboard.
Play 2: The Latency Distillation
Trigger: A user-facing feature is too slow because it waits on a large model, and the latency is measurably hurting engagement or conversion.
Here the goal is not money, it is milliseconds. A small student running close to the user can turn a two-second wait into a snappy interaction.
How to run it
- Owner: the product engineer who owns the feature, with ML support.
- Inputs: representative inputs for the feature, weighted toward the most common interactions.
- Success gate: p95 latency drops below the target, and quality on the top interaction patterns holds.
Latency plays tolerate slightly more quality loss on rare inputs than cost plays do, because the win is felt by every user on every interaction. Decide that trade-off explicitly with the product owner before you start, not after someone complains.
Play 3: The Stability Distillation
Trigger: You depend on a third-party API that keeps changing behavior, or whose terms, pricing, or availability you cannot control.
This play is about governance and risk, not performance. You distill the API's behavior into a model you own and version, so upstream changes stop breaking your product.
How to run it
- Owner: the platform or infrastructure lead, because this is a dependency-management decision.
- Inputs: a broad, representative sample of how you actually use the API today.
- Success gate: the owned student is good enough to serve production, and you have a documented retraining path for when requirements grow.
Confirm the provider's terms permit training on their outputs before you run this. The stability play is frequently the one that runs afoul of licensing, since you are explicitly trying to reduce dependence on the provider.
Play 4: The Specialization Distillation
Trigger: A generalist model is mediocre at a domain that matters to you, and you have enough domain examples to teach it.
Unlike the first three plays, size reduction is optional here. The point is focus. You concentrate the teacher's relevant behavior, and curated domain data, into a student that is sharp on your niche.
How to run it
- Owner: a domain expert paired with an ML engineer, because data curation is the whole game.
- Inputs: carefully selected, high-quality domain examples, with deliberate coverage of edge cases.
- Success gate: the student beats the generalist baseline on a domain-specific evaluation, validated by the domain expert.
The examples in What Is Model Distillation: Real-World Examples and Use Cases lean heavily on this play, because specialization is where distillation produces its most surprising wins.
The Sequencing That Prevents Stalls
Plays fail less from bad technique than from bad order. Here is the sequence that keeps projects moving.
The five-step cadence
- Qualify the trigger. Confirm a real cost, latency, stability, or specialization pain exists. If not, stop.
- Lock the success gate before building. Write down the quality floor and the win metric. No retroactive goalposts.
- Build the data set before touching training. Generate, deduplicate, and audit teacher outputs for coverage. This is most of the work.
- Train, then evaluate against the locked gate. Compare to teacher and baseline on held-out and per-segment data.
- Decide: ship, iterate, or kill. Iterate by generating data for failure cases. Kill without shame if the gate cannot be met.
The discipline of step five is what separates teams that learn from teams that sink months into a student that was never going to clear the bar. The decision criteria here mirror A Framework for What Is Model Distillation, which formalizes when each play applies.
Ownership and Handoffs
A play without an owner is a wish. Assign these roles explicitly for every distillation effort.
- Sponsor: the leader who owns the business outcome and approves the success gate.
- Driver: the engineer who builds the data pipeline, trains the student, and runs evaluation.
- Validator: the person, often a domain expert, who signs off that quality is genuinely acceptable.
- Operator: whoever will own the deployed student long-term, including its retraining loop.
The most common organizational failure is having a driver but no validator, so quality gets graded by the same person who built the thing. Separate those roles even on small teams.
Keeping the Play Running After Launch
Shipping is not the end. A distilled student degrades as production traffic drifts away from the data it learned on. Build the maintenance loop into the play from day one.
Monitor the student against a small ongoing sample of teacher outputs to catch drift. When the student starts diverging, generate fresh data, weighted toward the new failure cases, and retrain. Treat the distilled model as a living artifact with an owner and a refresh cadence, not a monument you carve once and forget.
Frequently Asked Questions
Which play should I run first?
The cost-pressure play, almost always, because it has the cleanest business case and the easiest success gate. A win there builds organizational trust and gives you a reusable pipeline for the harder plays.
Can one project combine multiple plays?
Yes, and it often does. A cost play frequently delivers latency benefits for free, and a stability play can include specialization. The reason to name them separately is to be clear about which metric is the binding success gate, so you do not blur your decision criteria.
Who should own a distillation effort?
A named sponsor for the outcome and a named driver for the build, at minimum, plus an independent validator for quality. Diffuse ownership is the single most reliable way to stall a distillation project indefinitely.
How do I know when to kill a play?
When you cannot meet the success gate you locked before building, and iterating on data coverage has stopped improving results. If two or three rounds of targeted data generation barely move the needle, the capability gap is too large for your chosen student, and continuing is sunk-cost behavior.
Do small teams need all four roles?
The roles can collapse onto fewer people, but never collapse the driver and the validator into one person. Independent quality sign-off is the one separation worth protecting even on a two-person team.
Key Takeaways
- Run distillation as named plays with triggers and owners, not as an open-ended research effort.
- The four core plays are cost-pressure, latency, stability, and specialization, each with a different success gate.
- Lock the quality floor and win metric before you build anything.
- Most of the work is data generation and auditing, not training.
- Always assign a separate validator so quality is not graded by the builder.
- Kill plays without shame when the success gate is unreachable after targeted iteration.