Eighty-five percent of enterprises are piloting AI agents right now. Five percent have shipped them to production. That gap calls for a closer look at how companies frame the work. And the clearest explanation of what is actually going wrong came from Amazon’s AGI lab this week.
Bryan Silverthorn, Director of AGI Autonomy at Amazon, told VB Transform 2026 on July 15 that managing reliability is central to closing the gap between piloting and shipping. Models can impress in demos and pass internal evaluations, then break in the real world. Half of surveyed enterprises have shipped an agent that passed every internal test and still failed a customer. That pattern puts the team’s assumptions about the agent under scrutiny.
Inside Amazon’s AGI lab, researchers call their agents “interns,” an operating philosophy that sets expectations for supervision. “I’ll have my intern talk to your intern” is real language inside the team. The framing carries a specific set of expectations. Interns are powerful. They can do real work. They also do things wrong in ways you would not predict. They need supervision, clear boundaries, backup plans, and someone paying attention to what they are doing before the work goes out the door.
That is the management layer most companies skip entirely.
The evaluation trap
Silverthorn described a customer that deployed an agent for software QA. It extracted serial numbers from screens. Worked perfectly for two months. Then it started reading wrong numbers intermittently. The cause was invisible to anyone looking at the agent’s outputs in testing. A software update shifted where serial numbers appeared on screen. The underlying system processed those locations differently. No benchmark caught it because no benchmark tested for that kind of drift.
His point was that tests need to measure reliability over time. Completing a task once establishes a narrower result than completing it correctly a thousand times in a row, across conditions that shift without warning.
VentureBeat’s own research backs this up. Most enterprises track whether their agents are online. Almost none track whether the answers are right. They are checking the pulse without checking the diagnosis.
Silverthorn breaks reliability into four dimensions, drawing from Princeton research: consistency, robustness, predictability, and safety. Most evaluations collapse all four into a single pass/fail gate. A team that separates them can spot exactly where an agent starts to drift before customers notice. A team that does not ends up with an agent that passed evaluation and failed production, which is exactly what half the industry is experiencing right now.
The frame determines the outcome
Here is where the insight gets practical. If you think of an AI agent as software, you test it like software. You run it through a QA cycle, it passes, you ship it, and you move on. When it breaks in production, you treat it as a bug and file a ticket. That is the cycle most companies are stuck in. It is why 85% pilot and 5% ship. The testing model assumes static behavior, while the agent’s behavior can vary as conditions change.
If you think of an AI agent as a collaborator, you manage it like a person. You ask what could go wrong. You build in undo capability. You decide in advance what level of error is acceptable for this particular workflow. You check its work at deployment and regularly afterward. You give it clear instructions and constraints before it starts so the team is prepared for failures.
Amazon’s lab has one agent running experiments around the clock on its own research plan. They accept that it will occasionally run the wrong experiment. The team consciously accepts that tradeoff for velocity and assigns responsibility before the agent starts working. Testing then has to account for that agreed tolerance.
Getting agents into production requires treating them as collaborators to manage throughout their working life, with ongoing responsibility after shipping.
What this means for your team
The practical takeaway reaches into organizational posture as well as testing. When your team deploys an agent, do they hand it off like a software release, or do they assign someone to manage it like a new hire?
The companies stuck at 85% have a testing problem they think is a capability problem. Alongside model capability and benchmarks, teams need management infrastructure around the agent: who supervises it, how errors get caught before customers see them, what the acceptable failure rate is for each specific workflow, and who updates the agent’s constraints when the environment shifts.
Silverthorn’s argument puts management quality at the center of moving beyond the 85% still piloting agents.
That makes supervision an operations decision as well as a technology investment. And every week a company runs another pilot without making it, the gap between experimenting and actually using agents grows wider. The intern is ready to work. The question is whether anyone on your team knows how to manage one.