If you can't measure your safety controls, you're guessing. And most teams are guessing. They run a handful of jailbreak prompts before launch, the prompts fail to break anything, and they declare the system safe. That's not measurement. That's a demo. Real measurement tells you, on an ongoing basis, how often your system does the wrong thing, how often it refuses the right thing, and whether either number is moving.
This article defines the metrics that actually matter for AI safety and alignment, how to instrument them without building a research lab, and how to read the resulting signal. The hard part of safety measurement isn't computing a number. It's choosing numbers that can't be gamed and that point at decisions you can act on.
The Two Numbers You Cannot Skip
Every safety measurement program reduces to a tension between two failure types, and you need both numbers or you have neither.
Leak rate
The leak rate is how often your system produces an output it should have blocked: harmful content, a policy violation, a hallucinated fact stated with confidence, or an action it wasn't authorized to take. You measure it by running a fixed evaluation set of adversarial and edge-case inputs through the full pipeline and counting how many produce a bad output. The set has to be fixed across runs, or you're measuring a moving target and can't tell whether a change helped.
False-refusal rate
The false-refusal rate is the mirror image: how often your system blocks or degrades a legitimate request. This is the metric teams forget, and forgetting it is how you ship a product that's "safe" and useless. You measure it against a set of known-good requests that a human reviewer has confirmed should succeed. A control that drives leak rate to zero by refusing everything is a failure, and only the false-refusal rate exposes it.
Track both as a pair. A change that lowers one while raising the other isn't an improvement; it's a trade you should make consciously.
Leading Indicators Versus Lagging Indicators
Leak rate and false-refusal rate are lagging indicators. They tell you what already happened. To run a safety program you also want leading indicators that warn you before a problem reaches users.
- Coverage of the eval set. What fraction of your known risk categories does your evaluation set actually exercise? A leak rate of zero on a set that tests three of your ten risks is meaningless.
- Drift in input distribution. When the kinds of requests users send shift away from what your controls were tuned for, your real-world leak rate will rise before your eval set catches it. Watch for new intent clusters.
- Time-to-detection. When a bad output does reach production, how long until you know? A team that finds out from a customer complaint a week later has a measurement gap, not just a safety gap.
- Escalation rate. For human-in-the-loop systems, the fraction of actions that get escalated to a reviewer tells you whether your automated controls are calibrated. A rate near zero or near one both signal a problem.
If you're building your first program, the practical sequence in Getting Started with Ai Safety and Alignment Basics pairs well with this list: get one lagging metric instrumented before you reach for the leading ones.
How to Instrument Without a Research Lab
You do not need a dedicated evaluation team to measure safety. You need a repeatable harness and discipline about keeping it stable.
- Build a golden set. Assemble 50 to 200 inputs split between adversarial cases (should be blocked) and legitimate edge cases (should succeed). Label each with the expected outcome. This is your ground truth.
- Run it on every meaningful change. Wire the golden set into your deploy process or a nightly job. Every time you change a prompt, a model version, or a filter, the harness reports leak rate and false-refusal rate against the unchanged set.
- Use a judge, but verify it. An LLM can grade whether an output violated a policy at scale, which beats manual review. But spot-check the judge against human labels regularly, because a miscalibrated judge gives you a confident wrong number, which is worse than no number.
- Log everything in production. Sample real traffic, store inputs and outputs with their control decisions, and review the sample weekly. This is how you catch the drift your golden set can't anticipate.
The teams documented in Case Study: Ai Safety and Alignment Basics in Practice almost all started here: one golden set, one nightly run, one weekly traffic review. The sophistication came later.
Reading the Signal Without Fooling Yourself
A number is only useful if you read it honestly. The biggest measurement trap is the vanity eval: a golden set so easy that everything passes, producing a comforting leak rate of zero that says nothing about real risk. If your eval never fails, it's too easy, not your system too good. Deliberately seed it with cases you expect to be hard.
The second trap is aggregate blindness. An overall leak rate of two percent might hide a thirty percent leak rate in one critical category, averaged away by a thousand easy cases. Always break metrics down by risk category and by user segment. The third trap is chasing a single number, where you tune relentlessly to lower leak rate and never notice false-refusal rate climbing in lockstep. Read the pair, always.
Finally, calibrate your thresholds to consequence. A leak rate that's acceptable for a brainstorming tool is catastrophic for a system that takes financial actions. The patterns in Ai Safety and Alignment Basics: Best Practices That Actually Work and the structure in A Framework for Ai Safety and Alignment Basics both stress this: the number's meaning comes from the stakes, not from the decimal places.
Frequently Asked Questions
What is the single most important safety metric to start with?
Leak rate against a fixed golden set, paired immediately with false-refusal rate. One without the other is misleading. If you can only instrument one thing this week, instrument a small golden set that produces both numbers from a single run.
How big should my evaluation set be?
Start with 50 to 200 labeled cases, balanced between things that should be blocked and things that should succeed. Bigger isn't automatically better; a small set of genuinely hard, well-chosen cases beats a thousand easy ones. Grow the set as you discover new failure modes in production.
Can I trust an LLM to grade my safety evals?
Mostly, at scale, with verification. An LLM judge is far faster than manual review and consistent enough for tracking trends. But you must spot-check it against human labels periodically, because a drifting or miscalibrated judge produces confidently wrong numbers that are worse than having no metric.
How often should I run safety measurements?
Run the golden set on every meaningful change to prompts, model versions, or filters, plus a scheduled nightly job to catch drift. Review a sample of real production traffic weekly. The cadence matters less than the consistency; an unstable measurement schedule produces uncomparable numbers.
Why does false-refusal rate matter as much as leak rate?
Because a system that refuses legitimate work is a failed product even if it never leaks. Optimizing leak rate alone pushes you toward over-blocking, which destroys utility silently. Tracking false refusals keeps you honest about the cost of every safety control you add.
Key Takeaways
- Measure leak rate and false-refusal rate as a pair; one without the other gives a misleading picture.
- Add leading indicators like eval coverage, input drift, time-to-detection, and escalation rate to warn you before problems reach users.
- Instrument with a stable golden set of 50 to 200 labeled cases, run it on every change, and verify any LLM judge against human labels.
- Avoid the vanity eval, aggregate blindness, and single-number chasing by keeping the set hard and breaking metrics down by category.
- A metric's meaning comes from the stakes; calibrate acceptable thresholds to the consequence of failure in your specific use case.