Ordinary software has two states. It works, or it throws an error and somebody fixes it. The failure announces itself.
An AI system has a third state, and it is the one that costs money. Still running. Still producing confident, well-formatted, plausible output. Quietly worse than it was six months ago. Nothing broke. Nobody changed anything. The extraction that was right on roughly nineteen invoices in twenty in March is right on about seventeen in twenty by September, and the only signal is that a few more things than usual have needed sorting out lately.
Monitoring gets listed as a requirement constantly and defined almost never. The orchestration piece gives it one clause and moves on. The word for what it is watching for is drift, and this is what drift actually is.
The three things that actually drift
Drift is not one phenomenon. It is three, with different causes and different fixes, and telling them apart is most of the diagnosis.
The inputs change. By a distance the most common, and the most mundane. A supplier redesigns their invoice template in April. A form gains a field. A new person joins and phrases enquiries differently from the person who trained the system’s expectations. The seasonal mix shifts, so the work arriving in November genuinely differs from the work the system was tuned on in June. Nothing about the system changed. What it is being handed did.
The world changes. A pattern learned from last year’s data stops holding. Prices moved, so the range that used to flag an anomaly now flags a third of legitimate transactions. A regulation changed and the document that was correct is now incomplete. A competitor entered the market and the enquiries arriving are different enquiries.
The model changes. The provider updates the version behind the system, or retires the one it was built on. This is the least frequent and the most disorienting, because it can move behaviour on your specific work even when the new version scores better on general benchmarks. Your prompts were tuned against particular behaviour, and tuning is exactly what a new version disturbs. It is also the strongest practical argument for keeping the model swappable instead of assuming the one you started with will be there.
The reason all three go unnoticed is the same. A degraded AI system does not fail loudly. It produces the same fluent output at the same rate, with the same confident tone, and a small number of those outputs are now wrong.
What to watch
Waiting for errors to surface is the default and it is far too late, because by the time an error surfaces someone has acted on it. Relying on the reviewers to notice is little better: the literature on automation bias is consistent that people supervising a mostly-correct system stop scrutinising it, which is exactly the condition a slowly degrading system creates. Useful monitoring watches the shape of what the system produces, on the reasoning that the shape changes before the damage does.
Four figures carry most of the signal, and none requires special tooling.
The share deferred to a person. If the system normally hands back eight percent and is now handing back fourteen, something upstream changed. This is usually the first indicator to move and it is often the only one you need. It is a strong argument for building the exception queue with a visible count instead of as a folder.
The override rate. How often a reviewer corrects what the system produced. Rising overrides mean accuracy is falling on exactly the cases a person sees, which is a direct measurement rather than a proxy. This is the figure that most directly says the system is getting things wrong, and it is free if the corrections are being logged.
The distribution of outputs. If a classifier normally sends 60% to category A and is now sending 35%, that is worth understanding even if every individual output looks defensible. In most of the processes worth automating the mix is fairly stable week to week, so a sudden shift is either a real change in the work or a change in the system, and both are worth knowing about.
Volume and cost per item. A sudden rise in cost per item usually means the system is doing more work per case: more retries, longer inputs, more steps. That is often the earliest sign that something upstream is malformed, and it is the same mechanism that turns a runaway loop into an unexpected bill.
Alongside those four, the fixed test set gets re-run on a schedule. Cases with known answers, run quarterly and after every change, measure accuracy directly instead of inferring it. Size it honestly: thirty cases catches a system that has fallen over, and detecting a gentle slide of a few percentage points needs a hundred or more. For the gentle slides, the override rate above is the sharper instrument, because it measures every case a person saw instead of a sample.
The distinction between the two halves matters. The four figures watch the live traffic and catch changes in what is arriving. The test set watches the system against fixed input and catches changes in the system itself. Both are needed, and they fail to substitute for each other in exactly the case that matters, which is a slow decline.
Thresholds and who looks
Monitoring that produces a dashboard nobody opens is theatre. Three things make it real.
Thresholds set in advance. Deferral rate above twelve percent, override rate above five percent, cost per item up by a third. The specific numbers matter less than the fact that they were decided before the situation arose, because a threshold set while looking at a worrying chart is a threshold argued down.
A named person and a real cadence. For most businesses this is fifteen minutes a month looking at four numbers, plus an alert if a threshold is crossed. It should be somebody’s job rather than everybody’s.
A defined response. What happens when a threshold trips: pull twenty recent cases and check them by hand, re-run the test set, look at what changed upstream in the last fortnight. Decided in advance, so the response is a procedure instead of an improvisation on a busy day.
Reading what you find
When a figure moves, the useful discipline is to establish what changed before deciding what to do, because the instinct is to blame the model and the model is usually innocent.
Work outward in order of likelihood. Did anything upstream change: a supplier format, a form, a new person, a seasonal shift. Did the document set behind the system change, which for anything using retrieval is a common and invisible cause. Did anyone edit a prompt or a rule. Did the provider announce a version change. And only then, has the system genuinely got worse at unchanged work.
Most of the time the answer is in the first question, and the fix is a small change to handle a new format instead of anything to do with AI. That is worth internalising, because the alternative is a business that responds to every wobble by considering a rebuild.
The cost of not doing it
This is a recurring cost, and it belongs in the business case at the start rather than arriving as a surprise in year two.
The monitoring itself is small: some instrumentation at build time, then a quarter hour a month. The consequence of skipping it is not small. A system degrading unnoticed produces wrong output that people act on, and by the time it surfaces the wrong outputs have flowed into invoices, records and customer commitments. The unwinding costs more than the monitoring would have, and it costs something harder to replace, which is the willingness of the staff to trust the system at all. A team that has once found the system had been quietly wrong for four months does not go back to trusting it, and that is the point at which the old spreadsheet comes back.
There is also a straightforward reason to keep the record. When someone asks whether a system has been performing as expected, a business with six months of deferral rates and override counts can answer. A business without one is offering an opinion.
Month nine, not day one
Working on the day it ships is the easy half of an AI system. Still working in month nine, on work that has quietly changed, is the half that decides whether it was worth building.
Watch four things: what share gets handed back, how often a person overrides it, whether the shape of the output has shifted, and what it costs per item. Re-run a fixed set of known cases quarterly and after any change. Set the thresholds before you need them, give the numbers to a named person, and decide in advance what happens when one trips. When something moves, look upstream first, because upstream is usually where it came from.
The question worth putting to anyone building this for you: how will I find out if this stops working properly, and who sees that. An answer naming specific figures, a threshold and a person has thought about the whole life of the thing. An answer about monitoring and support has thought about the launch.