Skip to main content

For CTOs & Product Leaders

AI That Says "I Don't Know"

What happens when you build an AI that flags its own uncertainty instead of faking confidence.

The most dangerous AI system isn't the one that's wrong. It's the one that's wrong and confident.

A 2025 Carnegie Mellon study found that large language models remain stubbornly overconfident even after producing incorrect answers. Humans, when shown they performed worse than expected, adjust their confidence downward. The AI doesn't. It keeps insisting it did well. The researchers compared it to a friend who swears they're great at pool but never makes a shot.[1]

In enterprise contexts, this confidence inversion has consequences. A 2024 Deloitte survey found that 38% of business executives reported making incorrect decisions based on hallucinated AI outputs.[2] Not "received bad data." Made actual decisions. Allocated budgets. Changed strategies. Signed contracts. Based on information the AI presented with full confidence and zero accuracy.

Back to The LibraryFor CTOs & Product Leaders

AI That Says "I Don't Know"

What happens when AI flags its own uncertainty.

The Architecture Series · 7 minute read · By Ed | Founder & CEO, DealiOS

The most dangerous AI system isn't the one that's wrong. It's the one that's wrong and confident.

A 2025 Carnegie Mellon study found that large language models remain stubbornly overconfident even after producing incorrect answers. Humans, when shown they performed worse than expected, adjust their confidence downward. The AI doesn't. It keeps insisting it did well. The researchers compared it to a friend who swears they're great at pool but never makes a shot.[1]

In enterprise contexts, this confidence inversion has consequences. A 2024 Deloitte survey found that 38% of business executives reported making incorrect decisions based on hallucinated AI outputs.[2] Not "received bad data." Made actual decisions. Allocated budgets. Changed strategies. Signed contracts. Based on information the AI presented with full confidence and zero accuracy.

Continue reading.

Enter your email to unlock the full paper. And the rest of the Architecture Library.