← anthrocytai.com·All presentations
Your AI can tell you it did well. That is not the same as evidence.
Every deployed model needs a grader. If the grader is another model, you have not measured quality. You have measured agreement. In healthcare, that difference has a cost, and someone eventually pays it.
The cheapest evaluation is the one that lies to you
You have shipped something. It calls a frontier model, it produces clinical text, and now you need to know whether the output is good. Human review does not scale past a few hundred cases. So you do the obvious thing and ask a model to grade the output.
This works. It is fast, it is cheap, and the scores look reasonable. It also degrades in a way that is close to invisible, because the failure does not announce itself as a failure. It announces itself as a good score.
The problem is structural. A model grading model output is scoring against its own priors about what a good answer looks like. Where the generator is confidently wrong in a plausible way, the grader tends to be confidently wrong in the same direction. You have not built a check. You have built a mirror with a number attached.
In most software, a wrong answer is annoying
In clinical software it is a different category of event. Not because the model is worse, but because the person receiving the output has less ability to detect the error and more reason to act on it.
A clinician reading an AI-drafted note is not auditing it. They are working. They have six minutes and a waiting room. The output arrives fluent, structured, and formatted exactly like the hundred correct outputs before it. Fluency is not a signal of accuracy, but it reads like one, and under time pressure it is treated like one.
This is why verifiability matters more than quality scores. A system that lets a clinician check a claim against its source in two seconds is safer than a system that scores four points higher on an internal benchmark and shows its work nowhere.
Four signals that come from outside the model
These are the ones I have found hold up in production. None of them require a model to assess itself.
Recurrence
Did the same class of error happen again after you fixed it? A fix that does not reduce recurrence was not a fix. This is the cheapest honest signal available and most teams do not track it.
Acceptance
Did the human keep the output, edit it, or discard it? Edit distance between what you produced and what was filed is a quality measure your users generate for free, simply by doing their job.
Escalation accuracy
When the system handed off to a human, was it right to? An agent that never escalates looks excellent until the day it should have. Measure the misses, not the handoffs.
Downstream outcome
Did the thing the output was for actually happen? The claim paid, the follow-up completed, the referral closed. Slow, noisy, and the only signal that cannot be gamed.
The hardest question is not how good, it is how far
Every clinical AI system has a line it must not cross. Sometimes the line is regulatory, sometimes clinical, sometimes both. The engineering problem is not identifying it. Most teams can state it in a sentence. The problem is that the line erodes.
It never erodes by being wrong. It erodes by a hundred individually reasonable requests to move it slightly. A customer needs one more field. A workflow would be smoother if the system inferred rather than surfaced. Each request is defensible on its own merits, and the person asking has a good reason. Twelve months later the product is doing something nobody decided it should do.
When I took a care plan assistant through an Australian Class I medical device submission, that line was the entire architecture. A Class I device that begins interpreting patient data or delivering therapy reclassifies to Class IIa, and the obligations attached to Class IIa would have ended the company. So the system was built backwards from the refusal. What it may read, what it may say, what it must decline, and how each of those is evidenced to a regulator who does not accept "it usually works."
The most useful thing that came out of that process was not the submission. It was learning that a boundary only survives if someone owns it and the system enforces it mechanically. A rule that depends on everyone remembering is not a rule.
What this looks like in practice
Five things, in the order I would do them.
- Write down the refusal before the feature. If you cannot say what the system must decline to do, you do not yet have a specification. You have an intention.
- Separate hard gates from advisory ones. Some checks block a release and some inform it. Teams that treat every check as advisory ship the wrong thing. Teams that treat every check as blocking stop shipping.
- Name the owner of each hard gate. Not a team. A person. The gate exists to be defended against reasonable pressure, and committees do not defend well.
- Instrument at least one signal from outside the model. Recurrence is the easiest to start with because you already have the data. Acceptance is the most valuable because your users generate it by working.
- Make the boundary mechanical. Freeze the surfaces that matter, so changing one requires a deliberate act rather than a quiet commit. A rule enforced by memory is a rule you have already lost.
The part that does not get easier
Models will keep improving. Evaluation tooling will get better and cheaper. Neither of those solves the underlying problem, which is that a system optimising against a measure it also produces will find the shortest path to a good number.
The work is deciding what the number should be anchored to, and then holding that anchor when the quarter is going badly and a slightly looser definition would make the chart look better. That is not a machine learning problem. It is an ownership problem, and it is the one worth taking seriously.
Clinical AI earns adoption through verifiability, not fluency. Everything above is a way of taking that sentence seriously enough to build from it.