Quality scores that outlive their evidence

The score versus the work

Imagine a company that runs an AI agent to serve its users. Someone must still confirm the replies are worth reading. The usual method is to appoint another AI as judge, which reads each reply and marks it good, average or bad. That is the work we carried out for one client's agent over 16 weeks.

Here is the trap. A score freezes a reply at the instant of judging, yet exchanges with an agent keep moving after a score has been put away. The user returns with a question that follows up on what came before. The agent goes hunting for information, or runs a tool, meaning it looks something up. Sometimes the user rewrites an earlier message, and the exchange forks into 2 versions. What can a score stored a week ago still say about that exchange today?

We audited our own grading pipeline. It found 3 problems, and this note takes them in order, starting with the simplest and ending with the most important. If you grade an AI agent this way, you likely carry these 3 problems too, and the correctif we built closes the note.

The grading pipeline

The grader itself worked plainly. Each hour it pulled the 500 latest conversations belonging to the client's users, and for every one it wrote one record: the question was the user's opening message, and the answer was every agent message stitched together, cut at 4,000 characters. Next to that came a fingerprint, a short code derived from this text. Because the code shifts the moment the text does, it can later show whether anything changed.

One rule stood above the rest. The grader looked up its store of past scores before it built anything, and skipped any conversation that already carried one. So each conversation was graded a single time, in whichever hourly run first had room, and never revisited.

The judge received even less than the record held. It got the question trimmed to 400 characters, the answer trimmed to 3,000, plus the source application, the message count and the grading criteria. A thumbs-up or a thumbs-down from the user was saved as the score instead, and no judge was called for those. On the morning we measured, 12 September, the store held 6,345 scores, 265 of them thumbs. The dashboard's quality number was just the share saying good.

Taken one by one, each choice was reasonable. The truncations held the cost down, and grading each conversation a single time kept the hourly pass cheap. Combined, and without us noticing, these choices quietly answered a larger question: what a score is really a score of.

Failure 1: Stale scores

On 12 September we set one hourly run against the store as it stood that hour. Of its 500 conversations, 444 already carried a score. For 87 of those 444, or 1 in 5, the fingerprint on file no longer agreed with the current one. Every one of the 87 had grown since grading, typically by 6 messages. The old scores survived because the skip test ran ahead of any fingerprint comparison, so the stored fingerprint was never put to use.

One stored score could therefore stand in for many versions of one and the same conversation. We call this assessment aliasing.

An aliased score, to be clear, is not automatically a false one. Replies that came later might have matched the quality of the first, yet the store has no way of knowing either way. No score carried a date belonging to itself, only the day when the exchange began. And that 1 in 5 figure was observed in a single pool, on a single morning, rather than as a rate across the whole store.

Failure 2: The fingerprint limit

Fingerprint comparison alone would not have fixed this either, since the fingerprint had a blind spot of its own. The code was derived once the text had been cut at 4,000 characters, so anything written beyond that threshold stayed out of it.

In that same pool, 38 scored conversations had grown while their fingerprint stayed put, and 36 of them sat exactly at the cut. Once a conversation was at the cut, its growth stayed invisible to the fingerprint. A fresh user message could not change it either, since the fingerprint covered only the opening question plus the agent's text.

Failure 3: The judge's input

This is the most important failure. The judge never saw the conversation itself; what it saw was a bundle assembled for the occasion, the evidence bundle, meaning the material actually placed before the judge rather than everything that existed. That bundle held the user's opening question plus every agent message merged into a single block. The agent's side was therefore readable up to the truncation, while whatever the user wrote after the opening question was missing. A user coming back to say the reply missed the request went unseen. Tool calls and tool results were both absent, so how the agent handled the person's own data was assessed purely by reading what the agent wrote. And when an exchange had branched, agent messages from both versions were merged into that same block.

What the bundle held was cut short and blended together. Of the 444 graded replies, 174 have since grown beyond the 3,000 characters the judge received. A second pass over the same 500 conversations found branches in 74, meaning discarded versions could be graded alongside the version the user chose to keep. One thing we noticed only during the audit: our rule, fixed in a note written earlier in this series, limits what the judge receives to the request, the reply and the criteria. Yet the judge operated within a general-purpose AI agent program, where its tools and instructions remained switched on. Nothing shows any use of them, but nothing in the call blocked such use either, and what was handed over as request and reply was not the work, falling short of what the rule demanded.

The evidence contract

The replacement rests on one plain promise: every review states precisely which version of an exchange it judged. It works from the conversation exactly as the user saw it, running from the newest message down to the first, with any abandoned branch left out. From that whole path it derives a fingerprint, this time covering every message. Each review is tied to that fingerprint together with the criteria version the judge applied. It logs which judge did the reading, and when. We call this score identity: the score always tells you which version of the conversation it describes.

When a later pass finds a changed fingerprint or a newer criteria version, the conversation re-enters the queue for a fresh review. The earlier review remains as a trace of what was judged at the time; it simply no longer counts as current.

A further promise: when evidence is too large to judge in full, it is never compressed to fit. A conversation still in progress, or one whose record is incomplete, receives no score. A record too large for a single pass is judged in stages, and the review states this openly. Every review covers 3 checks. Grounded: the claims are supported by the recorded evidence. Complete: the delivery covered the whole request. Usable: the result works as is. Each check returns pass, fail or unknown, along with the messages behind it. An unknown tells you the review happened yet the recorded evidence left the question open, and unknowns are tallied and displayed beside the quality number, excluded from it.

The first cycle

On 12 September the new system completed its first cycle across 32 conversations: 2 per application for each of the 8 full weeks before that date, picked by a fixed rule, not by usage or by old scores. 11 of the 32 could not be resolved on their evidence. Among the 21 that could, 15 cleared every check and 6 missed at least one, with the supporting messages quoted. The quality number read 15 of 21, with 11 of 32 unknowns displayed beside it. Grounded was the check that most often returned unknown, in 9 of the 32. These are one model's judgments over recorded evidence, neither tallies of tasks nor any kind of proof, as the criteria themselves state. Yet every one carries its score identity, so it can be traced to the exact messages it read.

What a score certifies

For 16 weeks the dashboard carried a quality number resting on scores that no one had verified against the conversations they described. That number blended scores still matching their conversations with scores that no longer did, and nothing could separate the two. We have not regraded those conversations, so we cannot say how many scores would shift. The new system's unknowns may reflect the criteria as much as the evidence. The replacement covers that gap at a price: 11 of its first 32 reviews come back unknown, and it reports that openly.

This note adds 3 terms to the series: assessment aliasing, the evidence bundle, and score identity, all of them defined where they first appear above.

Scoring an agent for quality means asserting something about evidence, and the assertion holds only while the score can name the evidence behind it. Holding score and evidence together is the work of the software surrounding the model, not of the model itself.