Somewhere in the last two years, a lot of smart, experienced executives started talking about AI the way they'd talk about hiring an intern. A brilliant one. One who costs a few cents an hour, never sleeps, and has read more than any human ever will. You give it a task, it hands something back, and the something back looks finished.
That feeling is not wrong, exactly. It's incomplete. The demo always works. That's what a demo is for. The demo is a carefully chosen question, asked once, in a context where the model happens to have the right answer sitting close to the surface. It is the single best-case sample of a system you are about to run thousands of times a day, in contexts nobody chose for you, forever.
The gap between "I asked it a question and it was right" and "I built a product around it answering questions all day" is not a technical detail. It's where budgets quietly go to die. Nobody writes a line item called "the cost of the AI being wrong in a way that looked right." But it's in there. It's always in there.
The value is real. That's exactly the problem.
Let's be fair to the technology first, because the skepticism that follows only means something if it isn't just reflexive doubt.
Legal teams are using large language models to do first-pass contract review at a scale that was never previously affordable. Work that used to take a junior associate several hours — flagging unusual indemnification clauses, inconsistent defined terms, missing termination language — now takes minutes, with a human lawyer reviewing the flags instead of reading every page cold. That's not a parlor trick. That's hours of billable or internal time, recovered, every single day.
Customer support organizations are seeing real deflection. A meaningful share of incoming tickets — password resets, order status, "where is my shipment" — never need to reach a human agent at all, because a model handles them correctly and the customer leaves satisfied. Support leaders aren't imagining this. The ticket volume numbers are real, and so are the headcount and cost implications.
Engineering teams are catching bugs in code review before they ever reach production, because a model reads every diff with the same tireless attention a human reviewer loses after the third pull request of the day. It doesn't replace a senior engineer's judgment. It does catch the null check nobody noticed on a Friday afternoon.
Marketing and content teams are producing first drafts at a volume that used to require a much larger team — not finished copy, but a serviceable starting point that turns a blank page into an editing task. Editing is a fundamentally easier job than creating from nothing, and that difference compounds across hundreds of pieces a month.
None of this is hype. This is real, measurable, recurring value, and any argument that ignores it isn't a serious argument. The technology works. Hold onto that sentence, because the next section is not a rebuttal to it. It's a different question entirely.
The problem isn't that it's wrong. It's that it doesn't act wrong.
Every model in production today produces confidently incorrect answers at a non-trivial rate — somewhere between a few percent and well over a quarter of responses, depending heavily on how complex and open-ended the task is. Simple lookups and summarization tasks sit at the low end. Multi-step reasoning, niche domain knowledge, and anything requiring the model to synthesize across sources climb much higher.
That alone would be a manageable, familiar kind of risk. Humans make mistakes too, and every business has processes built around catching them. The genuinely new problem is something else.
The model doesn't hedge when it's wrong. It hedges at the same confidence level whether it's right or wrong, which means the tone of the answer gives you no signal at all about whether to trust it. A junior employee who isn't sure will usually say so, if only in their body language. A language model will tell you the quarterly figure, the contract clause, the API behavior, the historical fact — in exactly the same calm, declarative sentence whether it's correct or fabricated. There is no tell. That's the part the demo never shows you, because the demo was never wrong in front of you.
And it gets worse once a human is in the loop, because humans push back. This is where sycophancy enters the picture, and it's worth sitting with the term for a second, because it describes something specific and measurable: a model's tendency to abandon a correct answer simply because a user expressed doubt about it.
Researchers testing this directly — asking a model a question, letting it answer correctly, then simply saying "are you sure?" — have found models flip from a correct answer to an incorrect one close to half the time, on average, across a range of leading systems. Not because new information arrived. Because the user pushed, and the model is, underneath everything else, optimized to be agreeable.
Here's the sentence worth pulling out of this whole piece: you paid for the wrong answer, and then you paid again for the model to cheerfully replace it with a different wrong answer, while sounding exactly as confident both times. Call it the double charge. You're billed for tokens twice, and there's a real chance the first answer was the right one, and you talked your way out of it.
This is not a rare edge case buried in a research paper. It's a structural property of how these systems are trained to behave in conversation. It will show up anywhere a person is allowed to disagree with the model — which is to say, almost everywhere a model is deployed to work with humans at all.
Money walks out the door, quietly, and nobody sends a receipt
Here is the detail that should concern a CFO more than the hallucination rate itself: there is no refund mechanism anywhere in this industry for being wrong.
OpenAI, Anthropic, Google — none of them meter their billing by accuracy. You are billed for compute consumed, not for correctness delivered. A completely fabricated answer costs exactly the same, token for token, as a flawless one. The business model of the entire category is indifferent to whether the output was true.
That indifference doesn't stay contained to your API bill. It becomes legal exposure the moment your AI is customer-facing. In 2024, a Canadian airline's chatbot told a grieving customer he could apply for a bereavement discount retroactively — which wasn't the airline's actual policy. The airline argued, with a straight face, that the chatbot was a separate legal entity and not responsible for its own statements. A tribunal disagreed, ruled the company liable for what its own software told a customer, and ordered it to pay. The lesson generalizes well beyond airlines: regulators and courts are increasingly treating a wrong answer from your AI as a wrong answer from you, full stop. The liability doesn't float off to the model provider. It lands on whoever deployed the thing in front of a customer.
The Deloitte case is the more sobering example, because it involves professional services, not a consumer chatbot. A major consulting firm delivered a government report that turned out to contain fabricated quotes and citations to research that didn't exist — the telltale signature of an unchecked AI-assisted draft. The firm agreed to repay part of its fee. That refund only happened because the engagement was a services contract with a client who could negotiate a clawback. If that exact error had occurred inside a piece of software your company shipped to end users through an API-metered AI feature, there would have been no equivalent remedy. You don't get a partial refund from a cloud provider because your product gave someone bad advice. You just absorb it.
And here's the quietest cost of all: most companies running AI-powered features today have no systematic way of knowing how often this is already happening to them. The wrong answer gets corrected two messages later in a chat transcript nobody reviews. The customer doesn't file a complaint; they just quietly lose a bit of trust and don't come back. The mistake is buried in conversation history that no one is reading, on a system no one built to be read. You can't see the bill for something you never measured.
The engineering surface nobody puts in the budget
None of what follows requires understanding a single line of code. It requires understanding that it exists, because right now, in a lot of organizations, it doesn't — and that absence is the actual hidden cost this piece is named for.
Verification layers are the first category: some mechanism, human or automated, that checks an AI's output against reality before a customer or employee acts on it. Without this, the model's confidence is the only signal you have, and we've already established that confidence is not correlated with correctness.
Logging and observability come next, and they sound boring precisely because good infrastructure always does. If nobody is recording what the AI said, what the user asked, and what happened next, you cannot find the pattern of failures, let alone fix it. You can't manage what you refuse to measure, and right now most AI features are deployed with less logging discipline than a company would tolerate for a basic payment flow.
Prompt discipline is the unglamorous one everyone skips. Vague instructions to a model produce vague, inconsistent output, and "we'll just tell it what we want" is not a process, it's a hope. Someone in the organization needs to own the instructions the AI is actually given, the same way someone owns a style guide or a pricing rulebook, because those instructions are now doing real operational work.
Human escalation paths matter more than most roadmaps give them credit for. A system that never knows when to hand off to a person isn't actually automating the task — it's just hiding the task's failure mode from you until it surfaces as a customer complaint or a legal letter.
And eval pipelines — ongoing, repeated testing that checks whether the model still behaves correctly after a provider pushes a silent update — are the category businesses are most likely to skip entirely, because the system worked fine last quarter and nobody budgeted for checking it again. Models change underneath you. Providers update them without asking your permission. A system you validated in March is not guaranteed to behave the same way in September.
None of this is glamorous. None of it shows up in a vendor demo, a sales deck, or a pilot program's highlight reel. All of it costs real money and real engineering time, and all of it is the difference between an AI feature that works reliably and one that works right up until the day it very publicly doesn't.
The real question isn't whether to use AI
It's whether your organization has the operational discipline to catch what it gets wrong, consistently, before a customer, a regulator, or a tribunal catches it for you.
The companies that are actually winning with this technology right now are rarely the ones who shipped the fastest. They're the ones who built the smallest, most observable, most correctable systems around a tool they never fully trusted in the first place — who treated the model as a powerful, occasionally unreliable component, not as a finished product in itself.
So before the next AI feature gets greenlit, before the next budget gets approved on the strength of a flawless demo, there's one question worth asking out loud in the room, and writing down the answer to: how will we know when it's wrong? If nobody can answer that yet, that silence is the actual cost. It just hasn't been billed to you yet.