TL;DR: Wharton researchers proved that people follow confidently wrong AI advice 80% of the time and feel great about it. Every AI governance framework assumes human oversight as a control. That control is asleep.
A team of researchers at Wharton ran a clever experiment. They gave people reasoning problems, some with access to an AI chatbot, some without. Then they rigged the chatbot. On some questions it gave the right answer. On others, it confidently gave the wrong one. The participants could ignore the AI whenever they wanted.
They didn’t. When the AI was wrong, people followed it about 80% of the time. Their accuracy dropped well below what it would have been with no AI at all. And they felt more confident in their answers while getting them wrong. The researchers have a term for this. They call it “cognitive surrender.“
I’ve been thinking about that phrase ever since I read the paper. And that word.
Surrender.
We have spent the last couple of years having important conversations about the security of AI systems. Prompt injection, jailbreaking, agent-to-agent contamination, fabricated action reports. Real problems. Engineering problems. Or at least that’s how I always like to frame them. I have written about several of them. But all of those conversations share an assumption that we rarely examine. They assume a human is paying attention. A human who reviews the AI’s output, catches the anomaly, pulls the emergency brake. Human oversight is baked into every governance framework, every regulatory requirement, every incident response plan. The EU AI Act literally mandates it for high-risk systems.
The Wharton research puts a number on how much that assumption is worth. And the number is not encouraging.
If people follow confidently wrong AI advice four times out of five, on simple reasoning problems, in a controlled setting, with no time pressure and no consequences for getting it wrong, what happens in a real operating environment? Where decisions are fast, stakes are high, and the AI has been right often enough that nobody questions it anymore? I’ll tell you what happens. The human-in-the-loop becomes the human-next-to-the-loop. And eventually the human-vaguely-aware-that-a-loop-exists.
The Control That Doesn’t Control Anything
This is what makes cognitive surrender a security problem and not just a cognitive science curiosity. Every technical vulnerability in agentic AI gets worse when the person supervising the system has mentally checked out. An agent that fabricates status reports is dangerous. An agent that fabricates status reports while its operator nods along is catastrophic. Prompt injection works because models cannot distinguish legitimate instructions from adversarial ones. Cognitive surrender means the human backstop can’t either, because they have stopped looking.
The study found that time pressure made surrender worse. Of course it did. That’s the default state of any security operations center, any emergency room, any trading floor.
It also found that people who trust AI more and think analytically less are the most susceptible. Which is a polite way of describing the fastest-growing segment of AI users in every organization on the planet. (should I name said segment?)
Friction Is Not a Bug
There is a small piece of good news in the data. Incentives and feedback helped. When people were rewarded for accuracy and told immediately whether they got it right or wrong, they started pushing back on bad AI advice. They didn’t stop using the AI. They just started thinking again while using it. Override rates on wrong answers more than doubled.
That is a design insight, and we should take it seriously. Build friction where it matters. Surface uncertainty instead of burying it in a confidence score the user never sees. Show people when the AI was wrong, quickly and often, so they learn to keep their own judgment engaged. Aviation figured this out decades ago with automation bias. Checklists, cross-verification, forced manual steps at critical decision points. None of this is new.
The problem, as always, is incentives. Nobody measures whether users are still thinking. We measure adoption rates, time-to-completion, satisfaction scores. We are optimizing for exactly the behavior that the Wharton researchers just documented as a vulnerability.
And we are calling it productivity.