We trust in AI too much until it makes a mistake, then we judge that mistake more harshly than a person’s. As agents move into everyday work, the fix is the same, checks on every step, whether an agent or a person takes it.
Wharton researchers found in 2015 that people abandon a forecasting algorithm after watching it make a mistake, even though that same algorithm had performed better than a human forecaster. Confidence in a human forecaster fell more slowly than confidence in the algorithm after the same mistake. In the authors’ words, “the errors that we tolerate in humans become less tolerable when machines make them.”
Before they see an AI fail, lay people lean on it more than they would on a person. Researchers call this algorithm appreciation, a close relative of automation bias. Once the AI fails, we punish it more than if a human had made the same mistake.
Each reaction skips the step that matters, which is checking the work itself.
Why it matters now
Gartner expects AI agents to make 15% of day-to-day work decisions on their own by 2028, up from zero in 2024. It also expects more than 40% of agentic AI projects to be canceled by the end of 2027, partly because of weak risk controls. Agents are coming into real workflows, and the ones that last will have controls that hold up.
How we over-trust
In a 2023 study published in Radiology, 27 radiologists read mammograms alongside diagnostic categories they believed came from an AI. A wrong suggestion pulled even the most experienced readers down.
Medicine has measured this effect for years. A review of 74 studies pooled four of them and found that wrong advice from a decision-support system raised the risk of a wrong clinical decision by 26%. In government, Ben Green looked at 41 policies that require human oversight of algorithms and found that people “are unable to perform the desired oversight functions” the policies count on. Those rules buy a false sense of security. The reviewer tends to approve what the machine already decided.
How we punish agents
People often hold AI to a harder standard, and a visible failure makes it worse. In experiments on hypothetical semi-autonomous car crashes, people blamed the automation more than a human driver for the same crash and rated it as more severe. The pattern has limits. When a person and an automated car both make a mistake, people blame the person more. In one city survey, Tianjin residents said they would accept self-driving cars only if they were about four to five times as safe as human drivers. In one of two experiments in a 2025 study, one AI’s wrongdoing lowered opinions of other AIs, while one person’s wrongdoing didn’t change how people judged other people.
Picture an agent that runs for months with light review, makes one public mistake, and gets switched off. The manual process it replaced goes back to making the same mistakes, and nobody counts them because most of the time they are not even looking.
People and agents fail in similar ways
Many AI failure modes have a human twin.
OpenAI researchers argue that language models hallucinate partly because training and benchmarks reward a confident guess over admitting uncertainty, which is overconfidence by another route. When researchers ran GPT models through 18 classic tests of human decision bias, the models showed the human bias in about half of them, including overconfidence and confirmation bias. A study of multi-agent systems found that many failures came from how the agents were organized and coordinated, and not only from the model itself.
Two incidents eight years apart, one caused by a person and one by an agent, by its user’s account, ended with the same kind of fix.
Both companies responded by changing what the tool allows.
The same checks on every step
Safety engineering worked this out decades ago. James Reason’s Swiss cheese model treats error as expected and builds safety from several imperfect layers of defense. As he wrote, “though we cannot change the human condition, we can change the conditions under which humans work.”
An agentic workflow deserves the same treatment. Each step passes the same checks whether a staffer or an agent takes it, and no single check is trusted alone.
What happens next
Within three years, agencies will stop writing “a human reviews it” into their AI policies as the safeguard. They will write down the checks every step must pass instead, and the agencies that keep their agents running will be the ones that made that switch first.
Do it yourself
Pick one workflow where agents and staff share the work. List each step and who or what takes it.
Give every step the same checks: sources traced, a dry run before anything is deleted or sent, a stated confidence level, and a review at each hand-off.
Put destructive actions behind a limit in the tool itself. A reviewer’s attention is the weakest place to put it.
Test your human reviewers. Plant a known error in a test copy of an agent’s output and see whether it gets through.
Measure the manual process’s error rate before an agent takes over, so a visible mistake has something to be compared with.
The companion worksheet walks through one workflow, step by step.
