Why AI systems that pass testing fail in production — and what to demand before you deploy.
Most AI deployment failures aren't dramatic. The system doesn't crash. The output looks confident and coherent. A number in a slide deck said 99% accuracy. Someone signed off. And it's still wrong — in ways that are hard to catch and expensive to correct.
This is the failure mode that doesn't make headlines. It shows up quietly, in production, after go-live, when the people the system was meant to free up are correcting its outputs by hand anyway.
It's not a technology problem. It's a gap between what the system appears to be doing and what it's actually doing in your environment, under your conditions, with your data. Closing that gap is the real work of AI transformation. And it starts before you deploy.
Most enterprise AI systems today are built on RAG — Retrieval Augmented Generation. The idea is straightforward: give the model access to your documents, your data, your institutional knowledge, and it will generate answers grounded in what you actually know rather than what it learned during training.
In principle, this solves the hallucination problem. In practice, it introduces three failure modes that most teams don't anticipate until they're already in production.
The retrieval works. The right document is found and passed to the model. But generation doesn't stay inside it. The model mixes retrieved facts with its parametric knowledge — the beliefs baked in during training — in ways that are invisible to the user. The output reads as confident and sourced. Parts of it are. Parts aren't. Your users can't tell the difference, and neither can your QA process unless it's specifically designed to catch this.
This failure mode is particularly dangerous because the system looks like it's grounded. The retrieval step completed successfully. Something was found. That creates a false sense of reliability that's harder to challenge than an obvious hallucination.
Require explicit attribution. Prompt the model to cite which retrieved chunk supports each claim — if it can't cite it, it shouldn't say it. Build evaluation pipelines that check generation against retrieved context directly, not just against human judgment of the final output.
The retrieval returns the right document. But the answer lives in a specific table on page 4, and the chunk that was retrieved is the executive summary on page 1. The model fills the gap with inference. The inference is plausible. It's also wrong.
This is a chunking and indexing problem as much as a model problem. How you break documents into retrievable pieces determines what the model can actually access. Most teams treat this as solved once retrieval accuracy looks good on a benchmark. It rarely is, because benchmarks test whether the right document comes back — not whether the right passage comes back.
Test retrieval at the granularity your questions actually require. Ask your team to show you failure cases where the right document was retrieved but the right answer wasn't found. Invest in chunking strategy early. It is unglamorous work with outsized impact on output quality.
This is the most underappreciated failure mode in agentic AI systems, and the one that scales worst as deployments get more ambitious. An agent retrieves context at step one. By step three or four of the reasoning loop, that context has been summarised, passed through intermediate steps, and effectively forgotten. The model is now reasoning from its own prior outputs rather than from the original source. The drift is gradual and invisible. There is no error message. The output still looks coherent.
The instinct here is often to limit agent scope — fewer steps, narrower tasks. That helps, but it treats the symptom. The more precise fixes are:
Re-retrieval at key reasoning steps, not just at the start of the loop. Anchor back to source material before each consequential decision.
Explicit grounding instructions that prompt the model to reference retrieved chunks at each step, making the dependency visible and checkable.
Shorter chains with verification gates between steps — human or automated — rather than long autonomous loops that accumulate error silently and surface it only at the end.
Limiting scope is a consequence of good agent design, not the prescription itself. Design for grounding. Scope will follow.
Orchestration — how you coordinate multiple agents, tools, and retrieval systems into a coherent workflow — is where each of the failure modes above amplifies. Each handoff between components is a point where context can be lost, instructions can drift, and errors can propagate without triggering any obvious failure signal. A mistake in step two doesn't announce itself. It shapes step five, and you discover it when the downstream output is wrong and the audit trail is opaque.
Most teams treat orchestration as a technical plumbing problem: which framework, which infrastructure, which deployment pattern. It isn't. It's a decision architecture problem. The questions that matter aren't engineering questions. They're questions a business leader should be asking before the build starts:
At which points does a human need to verify before the system proceeds?
How does the system signal low confidence rather than generating confidently anyway?
What is the rollback path when a multi-step agent produces a bad output downstream?
Who absorbs the error when it gets it wrong?
If your team can't answer those questions before deployment, you're not ready to deploy.
That number is almost always measured under conditions that don't reflect your production environment — a clean dataset, controlled conditions, a curated benchmark. It tells you what the system can do at its best. It tells you almost nothing about what it does at the edges, under load, with your data, in your environment under real operating conditions.
Ask instead: "Show me how it fails."
The answer to that question tells you more about production readiness than any benchmark. If the team can't answer it — or hasn't looked — that is itself the answer.
Test the edges yourself. Find the conditions under which it breaks. Decide whether the failure mode is acceptable before you scale, not after. The boring and repetitive tasks are the hardest to automate reliably. And the 1% gap — the failure cases, the edge conditions — is precisely where your customers, your patients, your staff notice.
The benchmark tells you what the system can do under ideal conditions. Production is not ideal conditions.
Closing the gap between benchmark performance and production reliability is not a technical afterthought. It is the work. And it starts with the question every team should be able to answer before go-live: show me how it fails.