Every business owner has heard the old rule. Garbage in, garbage out. It was easy to follow, because garbage announced itself. Bad numbers looked like bad numbers. A typo looked like a typo. You could see it and fix it.
That rule no longer protects you, and the reason is worth sitting with. Feed a modern AI system a flawed record of how your business has decided things for the last ten years, and what comes out the other side is not visibly rubbish. It is articulate. It is consistent. It is delivered in the same even, unbothered voice as everything else, and it will hold its position under mild questioning. It looks like a considered judgment from something that has read everything.
Garbage in, gospel out. That is the actual risk, and it is the one nobody prepares for, because the failure does not look like failure. It looks like an answer.
Your records are a log of decisions, not a record of facts
Here is the reframe that makes the whole problem visible.
You probably think of your business data as a description of what happened. The quotes you sent, the jobs you won, the people you hired, the invoices you wrote, the customers you fired. Facts.
It isn’t. It is a record of what your business decided, over years, under pressure, with imperfect information, by people who were tired on Thursdays. Every one of those rows is a choice somebody made. Which quotes got the discount. Which customers got chased and which got let go. Who got the interview. Which supplier got the benefit of the doubt when the delivery was late.
Most of those choices were fine. Some were made for reasons that no longer apply. And some, if you are honest about a decade of trading, were made for reasons you would not write down and would not defend if somebody asked you to justify them at a tribunal.
An AI trained or grounded on that record does not know which is which. It has no way to distinguish the decisions you are proud of from the ones you would take back. It sees a pattern, at scale, held consistently over time, and it does the only thing it is built to do. It continues the pattern. Then it applies it uniformly, tirelessly, to every case from now on, without the Thursday-afternoon inconsistency that used to be the only thing stopping a bad habit from becoming a policy.
That is the part people miss. The machine does not introduce the bias. It removes the randomness that was hiding it, and it scales what is left.
Three cases worth knowing
These are the canonical ones, and they are canonical because in each the mechanism is unusually clear.
Amazon’s recruiting tool. Reuters reported in 2018 that Amazon had built an experimental system to screen résumés and eventually scrapped it after finding it discriminated against women. The mechanism is the instructive part. It was trained on ten years of the company’s own hiring, in an industry where most applicants and most hires had been men. It learned that pattern. It reportedly penalised résumés containing the word “women’s,” as in “women’s chess club captain.” Nobody instructed it to. It read the record and drew the obvious inference: this company hires men. Which was true. That was precisely the problem.
The healthcare risk algorithm. In 2019, a study published in Science by Obermeyer and colleagues examined a widely used algorithm that identified which patients should get extra care, and found it systematically underestimated the needs of Black patients. Again, the mechanism matters more than the headline. The system did not use race. It used prior healthcare spending as a stand-in for how sick someone was, which sounds sensible and is not, because less money had historically been spent on Black patients at the same level of illness. The algorithm measured a real thing accurately. It was just the wrong real thing. It answered “who has cost the most” and everybody read the answer as “who is sickest.”
COMPAS, and the argument about it. ProPublica reported in 2016 that a widely used criminal risk-scoring tool produced different error patterns for Black and white defendants. It is the most cited example in this field and I want to handle it honestly, because the rebuttal is legitimate: the tool’s developers and independent statisticians pointed out that the system was equally accurate across groups in the sense of a given score meaning the same likelihood of reoffending, and that it is mathematically impossible to satisfy every reasonable definition of fairness at once when base rates differ. Both sides were correct about different definitions. That is the actual lesson, and it is more useful than the headline: fairness is not one thing, and somebody has to choose which definition the system is optimising for. If nobody chooses deliberately, the choice gets made by default, by whoever wrote the code, and nobody in your business will know it was made.
Notice what none of these are. None is a story about prejudiced engineers. Each is a story about a system faithfully reproducing a pattern that was really there in the data, and about the pattern meaning something other than what everyone assumed.
Deleting the column does not fix it
The instinct, on hearing all this, is to strip the sensitive fields. Take out gender, take out age, take out anything that looks like ethnicity, and the problem is solved.
It is not, and this is the single most expensive misunderstanding in the area.
Information does not live only in the field named after it. It is smeared across everything else. Postal code carries income and often ethnicity. The school someone attended carries class. A gap in employment history carries parenthood, illness, or caregiving. The vocabulary in a written application carries first language. Which supplier a customer used carries neighbourhood. A model with enough of these fields does not need the one you deleted. It can reconstruct it, implicitly, from the rest, and it will, because those proxies genuinely do predict the outcome you trained it on.
So deleting the column achieves something worse than nothing. The bias remains, operating through proxies, and you have destroyed your ability to measure it, because you can no longer check whether outcomes differ across the group you stopped recording. You have made the system blind, not fair, and you have made yourself blind too.
The uncomfortable implication is that testing for a disparity usually requires holding the very data you were tempted to throw away, and handling it carefully, for the specific purpose of checking.
It happens without training, too
Most small businesses will never train a model. That does not exempt you, because the common pattern now is retrieval: you point an AI at your own documents and it answers from them, with citations.
That system inherits your filing cabinet exactly as it is.
If three versions of a procedure are sitting in the folder and only one is current, the answer depends on which one the retrieval happens to surface, and the citation makes the wrong one look authoritative. If your written policies reflect how things were done in 2019, the AI will confidently tell your newest employee to do it the 2019 way. If past case notes recorded some kinds of customers more sceptically than others, the system will reproduce that tone and present it as house style. And because it comes back with a source attached, it carries more authority than a colleague’s offhand answer would, which is exactly backwards. The citation makes it more persuasive without making it more correct.
This is why what you collect and how you structure it sets the ceiling on everything built later. A well-kept record produces a system worth trusting. A pile of contradictory documents produces a machine that answers contradictions in a confident voice.
Why the confidence is the actual danger
Step back and the real hazard is not that these systems are sometimes wrong. Every tool is sometimes wrong, and people are wrong constantly.
The hazard is the mismatch between how wrong they are and how they sound. A colleague who is unsure hedges. Their voice changes, they say “I think,” they look uncomfortable, and you calibrate accordingly without noticing you are doing it. A model has no such tell. It produces the fluent, evenly-worded answer whether it is on firm ground or reproducing a bad habit from 2017. The delivery is identical.
That is why this connects to the deeper limit on what these systems can reason about. A pattern-matching machine cannot tell you why the pattern exists. It cannot distinguish “we hired more men because more men applied,” from “we hired more men because our screening was skewed,” from “this correlation is a coincidence.” Those three have completely different implications and identical data. The machine sees one shape and continues it, and the confident voice makes the continuation sound like a conclusion.
What to actually do about it
None of this requires a data science team. Four things, all doable by a business with fifteen staff.
Build a test set before you go live. Take thirty or fifty real cases from last year, and write down what the right answer should have been for each. Run the system against them. This single afternoon of work is the difference between a system you can trust and a system you hope about, and almost nobody does it because it is dull.
Check outcomes across the groups you actually care about. Not as a philosophical exercise, as arithmetic. Are approvals, prices, priorities or flags landing differently across the categories that matter in your business and in law? You cannot answer that without measuring it, and you will not measure it by accident.
Watch the overrides. This is the cheapest signal available and it costs nothing to collect. When staff disagree with the system and do something else, log it. If they override it for one kind of case far more often than others, they have found something the test set missed. Your people are already doing quality assurance on the machine. Write down what they find.
Keep the decision with a person where it affects someone’s life. Hiring, credit, clinical judgment, who gets served. Assistance from a machine is fine. The decision, and the accountability for it, stays with a human being who can be asked to explain it. Canada’s privacy commissioners put this plainly in their joint guidance on generative AI: accountability for decisions rests with the organisation, not with any automated system used to support the decision. “The system flagged it” is not a defence, and a BC tribunal has already made short work of a company that tried a version of that argument.
The part worth carrying out the door
Your data is not a neutral description of the world. It is the accumulated residue of every choice your business has made, and some of those choices were shaped by things you would not choose again.
Point an AI at that record and it will find every pattern in it, including the ones you never articulated and would not defend. Then it will apply them evenly, at speed, to every case going forward, and it will describe what it is doing in calm, competent, unbothered prose that sounds like the house position, because in a real sense it is: it is what your house has actually been doing, said out loud for the first time.
That is genuinely uncomfortable, and it is also the opportunity. A system built on your history will show you your history, at a resolution you have probably never seen. Most businesses find one or two things in there they did not know were true. That is worth knowing on its own, before you automate anything.
Garbage in, garbage out was a warning about your inputs. Garbage in, gospel out is a warning about your ears. The machine will always sound certain. Whether it has earned that certainty is a question only you can ask, and you have to remember to ask it, every time, because it will never volunteer the doubt itself.