Your AI got worse and you didn't notice

A company scored 2,611 job interviews handled by an AI agent over six months. Every single one went through the same nine-part rubric. Then a stricter reviewer joined the panel, and the improvement everyone had been celebrating disappeared from the chart.

The agent had not regressed. The scoring had. That is the uncomfortable part of using AI in a business you cannot staff with specialists: the tool changes under you, the people judging it change too, and the number on the dashboard quietly stops meaning what you think it means.

This piece is about the smallest version of that problem, the one a company of three to fifty people can actually act on. Two researchers published the evidence on 4 October 2026, and the fix they describe costs an hour a month.

What the study actually measured

The paper is "Reliability of AI Agents: Rater Effects, Drift, and the Return to an Evaluation Program" by Liu Zhang and Mark Esposito, posted to arXiv on 4 October 2026. The subject was a production voice-and-video agent that runs live job interviews for one firm.

The data is unusually concrete for this kind of claim: 2,611 scored interview recordings, sorted into 14 batches between late March and late August 2026, overlapping human reviewers so that the same batch was graded by more than one person, the agent's own change log across three production deploys, and from May onward the firm's daily support tickets.

That combination is what makes it useful outside the company it came from. Most AI quality debates stall on opinions, because almost nobody keeps a record of what the system did, who judged it and what else changed that week. This firm did, and the record shows three separate things going wrong at once.

Two people can score the same AI very differently

The firm's reviewers overlapped on purpose: the same batches were graded by more than one person, which is the only way to find out how much the grader matters. In the study, two reviewers assessing the same batches differed by 0.79 standard deviations on the composite quality index.

That is not a rounding error. It is bigger than the improvement the company was measuring.

Then a stricter reviewer joined, the raw trend line went flat, and anyone reading the dashboard would have concluded the agent had stopped improving. After adjusting for who was scoring, the underlying quality had actually risen by 0.53 standard deviations between March and August 2026. The apparent decline and the real improvement were the same data read two ways.

If you have ever asked a colleague to "have a look at what the chatbot wrote this week", you have run this experiment without noticing. Your colleague's patience, mood and standards are variables in the measurement, and nothing in your dashboard records them.

Your quality evidence expires in about five weeks

The second result is the one that changes how often you check. The system's reliability moved between logged deploys, and the researchers estimated the rate of unexplained movement to work out how fast an old evaluation stops describing the live system: their forecast uncertainty reaches the size of the largest observed change after about five weeks. They flag the estimate as imprecise, resting on a handful of step changes rather than a steady slide. As orders of magnitude go, that is what you get.

Inside that window things also moved in a non-obvious shape. While the composite index climbed, critical failures did something else: they peaked at 6.3% around mid-May and fell steadily to 2.5% by mid-August. A single trend number would have hidden the hump, and the hump is the part a customer notices.

One more thing about how the evidence was checked. The team did not lean on its own scores alone. Interview-related support tickets fell by about 10% per month relative to technical-issue tickets — an operational number that does not come from the review process at all, moving in the same direction as the quality ratings. When a number you control and a number your customers generate agree, you have something. When only the number you control moves, you have a spreadsheet.

The three checks that keep a small team honest

The paper's closing recommendation is a three-part routine for anyone judging a deployed AI system over time. You can lift it straight into a small business, and you can do it without statistics.

• Comparability: whoever scores the output has to be the same person, using the same written criteria as last time, or the two scores cannot be compared at all. If somebody else reviews this month, write that down next to the score.
• Currency: put a date on the check and treat it as good for a few weeks, not forever. Redo it after any change to the tool, the prompts, the data it reads, or the person using it.
• Corroboration: pair the quality score with one number from your operation that the AI does not produce — complaints, refunds, repeat calls, time-to-reply, orders cancelled.

The same logic explains why AI projects die quietly. RAND's interview study of AI practitioners, published in August 2024, found that 84% of interviewees traced project failures to decisions and expectations set by leadership, well above data or infrastructure problems. Nobody agrees on what the system is supposed to do, nobody writes down what "working" means, and the project drifts into an argument about the model.

A 20-minute check you can actually keep

Skip the dashboard. The routine below is deliberately manual, because printing the output takes the work out of the tool that is being judged, and because a sheet of paper in a folder is easier to compare month to month than a screen full of charts.

1. Pull 15 to 20 real outputs from the last month — actual replies, summaries, quotes or transcripts, not test cases you invented.
2. Print them and score each one against three or four written criteria: correct on the facts, right tone, complete, nothing invented. Write the criteria down once and reuse the same wording next month.
3. Circle what is wrong on the page instead of fixing it. The paper trail matters more than a tidy document.
4. Write the total, the date and your own name on the top sheet. If someone else scores next time, their name goes there too.
5. Pick one operational number in advance and write that on the same sheet: complaints per 100 orders, calls that had to be escalated, time to first reply, or refunds. Take it from your invoicing, your helpdesk or your phone bill, not from the AI tool's own report.

Twenty minutes a month, one folder. Redo it after you change the prompt, the model, the tools the agent can touch, or the person who runs it — a change mid-month is exactly when an old score starts lying to you.

Reading the result without overthinking it

When the score drops, the useful question is not "is the model worse". It is which of three things moved.

Open the flagged pages from this month and the same pages from last month and compare the kind of mistake. If the failures are new in character — invented figures where there were none, a tone your customers would read as rude, a field filled with the wrong value — that is the system, and it is worth fixing before it reaches a customer. If the failures are the same as last month but more of them, it is usually volume or a new use case, not decay.

If the low score comes from a reviewer who is not the usual one, you have not measured the AI, you have measured the reviewer. Rescore ten of the same outputs with the regular person before you touch anything. This is the single most common false alarm, and it is cheap to rule out.

If the quality score is flat but the operational number is getting worse, believe the operational number. Your customers do not read your rubric.

And a decision it is worth making on purpose: for some tasks, spot-checking output is the wrong job. If a person rewrites everything the assistant produces anyway, you do not need to score the drafts — you need to track how long the rewriting takes, because that is where your money is going.

Frequently asked questions

How many outputs do I need to check?

Fifteen to twenty from the last month is enough to see the pattern you are looking for, and few enough to finish over a coffee. The value comes from doing it the same way every month, not from the sample size. If your volume is tiny, check everything you have.

Can I just ask the AI to grade its own output?

No. A grader that shares the assumptions of the system being graded will not tell you that those assumptions stopped matching your business. It also cannot see the second problem in the study, which was that the people doing the scoring had drifted. Use a person, and keep it the same person where you can.

Our AI only drafts and a human always edits it. Do we need this?

You need a version of it, with a different number. Track how long the edits take and how often the draft is thrown away. If rewriting time creeps up month over month, the assistant is quietly getting worse even though nothing has visibly broken and no one has complained.

What if we cannot manage a monthly review?

Then do the minimum that still protects you: put a date on the last check you ran, and re-run it after every change you make to the tool. That single habit catches the failures that come from your own edits, which are more common than the failures that appear on their own.