A zero-shot classifier produces labels with total confidence and no inherent sense of whether they are right. The model does not know its own error rate. That is your job, and it is the part teams most often skip, because the outputs look plausible and the deadline is real. The result is a classifier that silently mislabels client data while everyone assumes it works.
Measuring a zero-shot classifier is different from measuring a supervised one, because you have no held-out test set with ground-truth labels. You have to manufacture your measurement, and you have to choose metrics that reveal the failures that actually hurt. Overall accuracy, the metric everyone reaches for first, is often the least useful number you can look at.
This article covers which KPIs matter, how to instrument them when you started with no labeled data, and how to read the signal each one carries. The throughline: measure per category, manufacture a small ground truth deliberately, and treat the confusion pattern as your most actionable output.
Why Overall Accuracy Misleads
The aggregation trap
A single accuracy number averages across categories, which hides disasters. A classifier can post ninety percent overall while one critical category sits at sixty, because that category is small. If that small category is the one your client cares about, the headline number is actively lying to you.
What to look at instead
Read precision and recall per category. Precision answers how many of the items you labeled X were actually X. Recall answers how many of the true X items you caught. These two diverge in ways overall accuracy cannot show, and they map directly onto business risk. This per-category discipline is built into the Verify stage of Naming the Stages That Turn Raw Labels Into Reliable Sorting.
- Overall accuracy hides per-category failures
- Precision: correctness of your positive labels
- Recall: coverage of the true positives
Manufacturing Ground Truth
The audit sample
With no training data, you create a measurement set by hand-labeling a few hundred examples drawn randomly from your real input. This is not training data; it is purely for measurement, and it never goes into the prompt. A few hundred is usually enough to expose the major failure patterns.
Sampling honestly
Draw the audit sample at random from real traffic, not from cherry-picked easy cases. A biased sample produces a flattering, useless number. If some categories are rare, oversample them deliberately so you have enough examples to measure their precision and recall at all.
When to re-sample
Input data drifts over time, so a one-time audit decays. Schedule periodic re-audits, a maintenance habit also stressed in Pre-Flight Items Before You Trust a Labelless Classifier.
The Confusion Matrix as a Map
Reading the off-diagonal
Build a small confusion matrix from your audit sample: rows are true categories, columns are predicted. The diagonal is correct; everything off it is an error. The largest off-diagonal cell tells you exactly which two categories the model confuses, which is the single most actionable fact you can have.
Turning confusion into a fix
When the matrix shows category A getting mislabeled as B, the usual fix is a sharper description contrasting the two, looping back to the definition stage. This is far cheaper than adding examples and often fully resolves the error, as the email case study demonstrated.
Operational Metrics Beyond Accuracy
Confidence calibration
If your prompt returns a confidence rating, check whether high-confidence labels are actually more accurate than low-confidence ones. Well-calibrated confidence lets you route uncertain cases to humans with a clean threshold. Poorly calibrated confidence is worse than none because it creates false trust.
Cost and latency per classification
Track tokens and time per call. These determine whether your design survives a volume spike and feed directly into the business case in Defending the Spreadsheet When You Skip the Labeling Budget. A classifier that is accurate but too slow or expensive at scale is not a working classifier.
Human-override rate
In production, measure how often a human corrects the model. A rising override rate is an early warning of drift, often before the formal re-audit catches it.
Choosing Between Precision and Recall
The asymmetry of error cost
Precision and recall trade off, and which one you favor depends on what a mistake costs. If a false positive is expensive, a wrongly flagged compliant post that gets a user banned, optimize for precision. If a false negative is expensive, a missed action item that ignores a customer, optimize for recall. Treating them as equally important ignores the business reality that one error usually hurts far more than the other.
Tuning the balance without retraining
In zero-shot you cannot retrain, but you can shift the balance through the prompt and the confidence threshold. Routing only high-confidence labels to automation and sending the rest to humans raises effective precision on the automated path. Loosening that threshold raises recall at the cost of more human review. The lever is the threshold, not the model.
- Favor precision when false positives are costly
- Favor recall when false negatives are costly
- Use the confidence threshold to shift the balance
Building a Lightweight Measurement Loop
What to instrument from day one
Treat measurement as infrastructure, not an afterthought. From the first prompt, have a harness that runs over your audit sample and prints per-category precision, recall, and a confusion matrix. This lets every prompt change be evaluated immediately rather than at a distant checkpoint, which is the same lesson the email backlog team drew in When Our Intake Bot Sorted 40,000 Emails Untrained.
Reading the trend, not just the snapshot
A single audit is a snapshot. The signal that matters in production is the trend: is per-category accuracy stable, is the override rate creeping up, is cost per classification holding. A classifier that was fine at launch and is quietly degrading looks healthy in any single snapshot and only reveals itself in the trend.
Connecting metrics to decisions
Every metric should map to an action. A low per-category recall triggers a description fix or an example. A rising override rate triggers a re-audit. A confidence score that does not predict accuracy gets removed from the routing logic. Metrics that do not change a decision are vanity numbers, and the business case in Defending the Spreadsheet When You Skip the Labeling Budget depends on the ones that do.
Frequently Asked Questions
How large does the audit sample need to be?
A few hundred randomly drawn examples is a reasonable starting point for most projects, enough to surface major failure patterns and produce stable per-category estimates. Rare categories may need deliberate oversampling so they are represented at all.
What if I cannot agree with myself on the labels?
That is a signal the categories are too fuzzy, and the model will struggle exactly where you do. Sharpen the category definitions until a human can label consistently before blaming the model. Human disagreement caps achievable accuracy.
Should I trust the model's confidence scores?
Only after checking calibration. Verify that high-confidence predictions are genuinely more accurate than low-confidence ones on your audit sample. If they are not, the confidence number is noise and should not gate human routing.
How often should I re-audit in production?
It depends on how fast your input drifts, but quarterly is a common baseline, with the human-override rate watched continuously as an early warning between formal audits. Faster-moving domains need more frequent checks.
Key Takeaways
- Overall accuracy averages away per-category failures; read precision and recall per category instead.
- With no training data, manufacture a measurement set by hand-labeling a few hundred random real examples, never feeding them into the prompt.
- Draw the audit sample honestly from real traffic and oversample rare categories so they can be measured at all.
- The confusion matrix's largest off-diagonal cell pinpoints the two categories to fix, usually via a sharper description.
- Track confidence calibration, cost and latency per call, and the human-override rate as production health signals.