Across 157 enterprises, half have shipped an AI agent that passed their internal evaluations and then failed a customer in production, according to VentureBeat Pulse Research. A quarter have seen it happen more than once. Only 5% fully trust the automated evaluations meant to gate that deployment. Yet two-thirds of organisations are actively removing the human in the loop.
The finding defines what the report calls the “evaluation gap” — the distance between the autonomy enterprises are granting their AI agents and the trust they place in the tests that are supposed to catch the failures. The autonomy is arriving faster than the assurance.
🔍 THE BOTTOM LINE
The numbers are a flashing red light for anyone deploying agentic AI in production. Half of enterprises are shipping agents their own tests cleared, only to watch them fail real customers — and instead of pausing, two-thirds are engineering toward removing the human check entirely. This is the structural risk behind every agentic AI governance framework now being drafted: the evaluation infrastructure does not yet match the deployment ambition.
The Defining Number
The report’s central finding is precise and uncomfortable. Of organisations that run pre-deployment evaluations, 50% have deployed an agent or LLM feature that passed those evaluations and then caused a customer-facing failure — an incorrect output, a broken workflow, or a quality incident. Twenty-six percent said it happened once. Twenty-four percent said it happened more than once, a recurring gap between evaluation and reality. Only 36% reported no such failure. The remainder either run no pre-deployment evaluations at all (8%) or don’t track closely enough to know (6%).
The failure is not theoretical. The evaluation said the agent was ready, and it was not. Everything else in the report — how enterprises trust their evals, what they monitor, how much autonomy they grant — is shaped by this experience.
Almost No One Trusts the Tests
Trust in automated evaluation is scarce and specific. Only 5% of organisations say they fully trust automated evaluation as it stands. The remaining 95% name a limitation. The most common, at 29%, directly explains the production-failure rate: evaluations align poorly with real-world outcomes — they pass agents that later fail. Bias or inconsistency (21%) and lack of explainability (18%) follow. Seventeen percent cite data-leakage or privacy concerns in the evaluation process itself.
The tests meant to certify agents are not yet trusted to certify them. That is precisely what makes the autonomy trajectory so striking.
The Autonomy Ceiling Is Rising Anyway
Here is the paradox at the heart of the report. Even though almost no one fully trusts automated evaluation, two-thirds of organisations (66%) either already allow zero-human-in-the-loop deployment for low-risk agents (34%) or are actively engineering their pipelines to permit it within twelve months (33%). Only 22% rule it out for the foreseeable future.
The direction is unambiguous: enterprises are moving to let evaluations gate production autonomously, removing the human check, at the same moment they say those evaluations do not reliably match reality. The autonomy ceiling is rising faster than the assurance beneath it. That is the mechanism by which the false-confidence failures scale rather than shrink.
The assumption that large, regulated organisations hold the human in the loop longest is, in this sample, backwards. Larger enterprises are slightly further down the path toward zero human review than smaller companies (70% versus 64%) and slightly more likely to have shipped an evaluation-passing agent that then failed (54% versus 48%). Scale does not buy caution — it buys confidence.
The Evaluation Stack Is Fragmented
The evaluation layer is early and unconsolidated. Provider-native tooling leads — OpenAI’s native evals and traces (17%) and Anthropic’s Claude Console evals (13%) together outweigh any independent platform. But it is tied at the top by a striking answer: 17% of enterprises use no dedicated agent-evaluation tooling at all. The specialist vendors — DeepEval (12%), Braintrust (8%), LangSmith, Weave, Promptfoo, Langfuse, Arize — are scattered across single to low double digits. Eleven percent have built their own.
No independent platform has yet become the category standard. Most enterprises are evaluating agents with provider-native tools, home-grown scripts, or nothing. For organisations deploying agents to customers, that is a significant structural gap — the equivalent of a bank running its own stress tests with no external validation and then being surprised when the portfolio blows up.
The Monitoring Blind Spot
Production monitoring for an AI agent can watch two very different things: whether the system is functioning (is it up, responding, fast, error-free) or whether the output is correct (did it give the right answer, take the right action, stay within policy). The distinction matters because a confidently wrong answer is invisible to the first kind of monitoring. The request completes, the response is fast, no error is thrown, and every functioning-metric reads healthy.
Only a quarter of enterprises run real-time quality checks on live production traffic. The rest monitor infrastructure spans, token counts, and raw I/O for post-hoc debugging. The agent could be confidently telling customers the wrong thing, and the dashboard would show green.
NZ Angle
For New Zealand organisations deploying agentic AI — and the agentic era layoffs and restructuring wave suggests many are — the report’s findings are directly applicable. The evaluation gap is not a Silicon Valley problem. It is a deployment problem. Any NZ company shipping an AI agent to customers without production-quality monitoring is running the same risk the report quantifies. The Singapore agentic AI governance framework and similar regulatory efforts are attempting to close this gap from the policy side, but the operational gap — the distance between “the eval passed” and “the customer is served correctly” — is one every deploying organisation has to close itself.
❓ FAQ
What is the evaluation gap? The distance between how much autonomy enterprises grant their AI agents and how much they trust the tests meant to catch failures. The report finds the autonomy is arriving faster than the assurance — 66% are removing the human check, but only 5% fully trust the automated evaluations that would replace it.
How common is the production failure? Half of organisations that run evaluations have shipped an agent that passed those evaluations and then failed a customer. A quarter have seen it happen more than once.
Why don’t enterprises just fix their evaluations? The most-cited limitation (29%) is that evaluations don’t align with real-world outcomes. The tests pass agents that fail in production because the test environment doesn’t reproduce the conditions that cause failure — the same problem that has plagued software testing for decades, now amplified by the unpredictability of LLM outputs.
Which evaluation tools do enterprises use? Provider-native tools lead (OpenAI 17%, Anthropic 13%), tied with “no dedicated tooling at all” (17%). The independent specialist vendors are fragmented across single digits. No category standard exists.
What is the monitoring blind spot? Only 25% of enterprises run real-time quality checks on live production traffic. The rest monitor infrastructure metrics — uptime, speed, error rates — which cannot detect a confidently wrong answer. The dashboard shows green while the customer gets bad output.
🔍 THE BOTTOM LINE
The report’s defining paradox is the real story: enterprises know their evaluations do not work, know the agents they certify fail in production, and are removing the human safety net anyway. That is not a technology problem — it is an organisational pressure problem. The deployment cycle is being driven by competitive urgency, not by evaluation maturity, and the result is a widening gap between what AI agents are allowed to do and what anyone can prove they do safely. Until the evaluation stack consolidates and production monitoring catches up, every autonomous deployment is a bet that the failure you haven’t seen yet is the one you won’t hit.
