Researchers from three European universities have measured something most AI users have felt but few have quantified: when an AI system gives you advice, you become dramatically worse at the task — and dramatically more certain you’re right.
What is “cognitive surrender”? It’s a term Wharton researchers coined earlier this year for the pattern where people accept incorrect AI answers around 80% of the time while reporting higher confidence than people working without AI. A new study from the University of Milano-Bicocca, École Normale Supérieure, and Sapienza University of Rome puts hard numbers on it.
🔍 THE BOTTOM LINE
Access to AI advice collapsed people’s willingness to say “I don’t know” from 44% to 3%, cut their accuracy from 27% to 9%, and pushed their confidence from 30% to 76%. People became one-third as accurate and twice as sure of themselves. Even paying them to be right barely moved the needle.
The Numbers That Should Make You Pause
The study, authored by Valerio Capraro, Chiara Marcoccia, and Walter Quattrociocchi, deliberately used questions where AI models typically fail — visual details from films, like the colour of a team’s uniform in Bend It Like Beckham. They ran the experiment with Step 3.5 Flash, a model that was usually wrong on these questions, precisely so any drop in performance couldn’t be explained as sensible delegation to a reliable tool.
The results, reported by The Register:
- Willingness to admit ignorance: 44% → 3% (a 93% drop)
- Accuracy: 27% → 9% (one-third of the no-AI baseline)
- Confidence: 30% → 76% (more than doubled)
“Some participants who would have answered correctly on their own asked the AI and became wrong,” the researchers noted. The AI didn’t just fail to help. It actively made correct people wrong.
Why Monetary Incentives Didn’t Fix It
The researchers tried paying participants to be accurate. It helped — but only marginally. Willingness to admit ignorance rose from 3% to 8%, and accuracy climbed from 9% to 16%. Both remained well below the no-AI baselines of 44% and 27%.
This is the uncomfortable finding. It’s not that people don’t care about being right. It’s that the mere presence of an AI answer overrides their own judgment even when they have a financial stake in correctness. The tool doesn’t have to be reliable. It doesn’t even have to be right. It just has to be there.
The Pattern Is Consistent Across Products
Wharton researchers coined “cognitive surrender” earlier this year to describe the same phenomenon: people accepting incorrect AI answers 80% of the time while reporting higher confidence than those working without AI.
Google’s AI search overhaul replaced links with confident AI-generated summaries, and Common Sense Media this week called that design an “unacceptable risk” for students. The pattern is consistent across products: AI systems are designed to answer, never to say “I don’t know.” The humans using them are learning to do the same.
This echoes what we saw in our coverage of Stack Overflow’s decline — when AI gives instant confident answers, the human habit of questioning, verifying, and admitting uncertainty erodes. It also connects to the ANU “hysterical” response to AI cheating: universities are scrambling not because students have AI, but because students trust AI answers they cannot evaluate.
What the Researchers Are Most Worried About
“For humans, the capacity to say ‘I don’t know’ is very important because it represents the recognition of the limits of our own knowledge,” Capraro told The Next Web.
He is particularly concerned about children, who are growing up with these systems before they have developed critical thinking skills. If a model that is usually wrong can suppress a thinking adult’s willingness to admit ignorance by 93%, what does a usually-right model do to a 12-year-old?
The Stanford AI Index 2026 already documented the expert-public trust gap — the public trusts AI far more than the researchers who build it. This study explains the mechanism: the trust isn’t earned through accuracy. It’s induced through availability.
NZ Angle
New Zealand has no equivalent of the EU AI Act or Australia’s pending digital duty of care. The Ministry of Business, Innovation and Employment published AI guidance in 2024, but it’s voluntary. That means the cognitive surrender pattern documented in this study is playing out in NZ workplaces, schools, and government agencies with no regulatory backstop.
The study’s finding — that even financial incentives couldn’t restore judgment — suggests voluntary guidance won’t be enough. If NZ agencies deploy AI for citizen-facing decisions without mandatory “I don’t know” safeguards, the robodebt precedent from across the Tasman shows exactly where that leads. Australia is now moving to curb government AI decision-making in direct response to that risk.
❓ FAQ
Doesn’t this only matter because the AI was deliberately wrong? No — that’s the point. The researchers used a wrong-on-purpose model to isolate the effect of AI availability from AI accuracy. If a usually-wrong model can collapse judgment this hard, a usually-right model can suppress it even more effectively, because users have no reason to second-guess.
What was the sample and method? The study used visual-detail questions from films (where AI models fail), measured against no-AI baselines, with and without monetary incentives for accuracy. The full preprint is on OSF.
Is this the same as “automation bias”? Related, but sharper. Automation bias is the tendency to over-trust automated systems. This study shows the cognitive habit of recognising your own ignorance is what gets suppressed — not just trust in the tool, but awareness of self.
What would fix it? The researchers suggest AI products should be designed to say “I don’t know” more often. The deeper fix is structural: when AI is used in high-stakes decisions (government benefits, medical triage, hiring), a human must be able to override without penalty, and the system must surface uncertainty rather than hide it.
🔍 THE BOTTOM LINE
The most dangerous AI failure mode isn’t hallucination. It’s the quiet erosion of the human instinct to say “I don’t know.” This study measured it: 44% of people admitted ignorance without AI. Only 3% did with it. The AI doesn’t have to be right. It just has to answer.
