A July 2026 Healthcare IT News account describes Johns Hopkins testing frontier AI agents on healthcare tasks. Only 28 percent of complex tasks were completed on a first attempt. Fewer than 8 percent of agents remained consistently successful across repeated runs (Healthcare IT News, July 2026). The tests cover clinical workflows, documentation chains, and operational tasks that real staff perform every day.
Those numbers make task scope worth testing alongside model choice.
Test the assignment as well as the model
Vendor benchmarks can help shortlist agents. A local trial still needs to cover the systems, permissions, and task complexity the agent will encounter. If it fails, inspect the assignment and the model before deciding what to change.
In a hypothetical twelve-step process, the twelfth step might depend on a system the agent cannot access. It might also need information lost from the context window after step four. Test those dependencies explicitly; reasoning performance in a demonstration leaves them unresolved.
With twenty-eight percent first-attempt success in the reported test, narrowing the assignment is one option for a follow-up experiment.
Compare bounded assignments
A useful follow-up test is to break a complex workflow into bounded assignments: processing a form, checking a set of records against a standard, or preparing a handoff. Measure each assignment under repeated runs before linking the steps.
Consider an illustrative comparison: an agent that completes a simple, bounded task at 94 percent reliability across hundreds of cases a day, and one that attempts a complex multi-step process and fails 72 percent of the time. Compare the usable work and review effort before choosing the broader assignment.
The deployment-to-value gap also appears outside healthcare. Writer reported that 97 percent of executives had deployed agents but only 29 percent were seeing ROI (July 2026). That makes the fit between agent capability and assigned work worth examining.
Benchmark under the conditions of use
A benchmark for deployment should include the task complexity, data quality, and failure modes the team expects in daily work. Request that evidence alongside the vendor’s controlled demonstration.
Teams that go straight from vendor demo to pilot deployment can miss that step. If a pilot fails, examine the match between the agent’s capability and the task’s complexity before deciding what to change.
If you run an organization deploying or evaluating AI agents, test what the agent can complete at production quality under your actual conditions. Use the gap between the demo and those results to adjust the scope and decide which capabilities need improvement.
Apply the method cautiously outside healthcare
Johns Hopkins operates in a specialized healthcare setting. The 28 percent result applies to the reported test. Other organizations need to measure reliability under their own workflow and data conditions.
For example, compare an agent pulling data from three sources for a daily report with one attempting an entire quarterly analysis. The broader assignment may be valuable, but its review burden and failure recovery need measurement too. Choose the scope using completed work, reliability, and staff effort under the same test conditions.