- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a new 10-task clinical EHR benchmark and uses it to show that model performance varies as a function of the way the task is presented to the model. Computer use performs the worst, with FHIR a bit better, and then plain text is better than that.
Critical strengths
- The core argument, that modality is important, is sound
Critical weaknesses
- The proposed 10-task benchmark is not scientifically valid. First, it's proprietary, so we reviewers can't audit it, and neither can the eventual readers of this work. We have no idea if it's set up correctly, or even many of the details of how it is set up. Second, it's tiny. 10 tasks are barely anything. The confidence intervals in this work are massive.
- The novelty is unclear. Prior work already establishes that observation and action interfaces substantially affect agent performance. Examples include HealthAdminBench’s (accessibility-tree vs screenshot observations), Beyond Browsing’s (API-based vs browsing) and MCPWorld (API, GUI, vs hybrid).
Questions for authors
None
2 comments
- You question our novelty against HealthAdminBench, Beyond Browsing and MCPWorld. “Prior work already establishes that observation and action interfaces substantially affect agent performance. Examples include HealthAdminBench’s (accessibility-tree vs screenshot observations), Beyond Browsing’s (API-based vs browsing) and MCPWorld (API, GUI, vs hybrid).”
We agree that these studies show the interface changes agent scores, and we do not claim that finding as new, but they answer a different question from ours. In each, the compared agents differ in more than the route to the data. These are reasonable choices for their goals, but they mean the route itself is not isolated:
- HealthAdminBench studies administrative workflows (prior authorization, appeals, equipment orders) rather than clinical care. Both arms of its ablation use the graphical interface with the same actions: one observes screenshots, the other an accessibility tree with element identifiers that, as its authors note, real EHR and payer systems typically do not expose. It has no API or full-record arm.
- Beyond Browsing compares a browsing agent that reads the accessibility tree, not screenshots, with an agent that uses each website's API. API coverage varies by site, from 988 endpoints on one to 31 on another, for which the authors added APIs, and some features have no API. The two agents therefore have different sets of possible actions.
- MCPWorld evaluates one model, and its GUI-only agent can also run shell commands. Its API-only agent scored lowest (53% against 71% for GUI-only), and its authors attribute 48% of the API-only failures to insufficient API coverage.
- OSWorld-MCP (Jia et al., ICLR 2026) adds MCP tools to graphical agents on OSWorld tasks. The tools generally raise success, but even the strongest models invoke them in only about a third of cases. Its agents keep the screen and gain actions, so the route and the action set change together.
- GUI vs. CLI (Zhou et al., 2026) gives screen-only and command-line agents the same 440 desktop tasks, starting states and verifiers. The command-line agent acts through skills rather than the screen's operations, so the action set changes with the route, and none of its tasks involves a clinical record. Its strongest command-line agent trailed its strongest screen-only agent, a different model (48.2% against 59.1%). Verifier-guided skill patching raised the command-line agent to 69.3%, which the authors present as a diagnostic upper bound, not a baseline.
The direction differs across these studies: the API agent leads in Beyond Browsing and trails in MCPWorld, adding tools helps in OSWorld-MCP, and in GUI vs. CLI the command-line agent trails unless its skills are patched. In each, the result depends on how much of the task the non-graphical route covers or how often agents use it, and that is what our design holds fixed. The tasks, records, episode clock, actions, executor, pre-save validation and grader are the same under every modality, for five models from three providers. Only the route changes, and one route needs no retrieval at all. Section 2 (pages 2–3) and Table 22 (Appendix J) now set out these differences, and Section 2 now cites OSWorld-MCP and GUI vs. CLI. This paper adds three things.
- It gives a controlled comparison on clinical tasks. Our tasks include orders, medication changes, coded allergies and signed notes, graded on the chart state after the episode and on the record of each write, not on a final answer alone.
- It gives evidence that parity decides such comparisons. One answer-bearing schema took a structured pilot task from 8/8 passes to 0/8 (Table 6). After the reviews, we found that our own packet on five tasks was task-filtered, which favoured Packet+Action (Appendix B.1). Section 3 states the parity requirements that comparisons across access modalities need, and Table 17 gives the direction of every remaining asymmetry.
- It shows where the gap arises. Process checks and a first-failed-stage decomposition identify failures to reach the evidence and to finish and persist writes as major contributors to the Pixel deficit. On nine of the ten original tasks, among trials that had seen the evidence and persisted their writes, 57.6% passed the decision gates and the judge under Pixel against 62.5% under FHIR, a difference of −4.9 points (95% CI −27.8 to 22.0; Section 7), under extracted-answer scoring as in the revised Section 7. This analysis is exploratory: the process checks were defined after the run, and the stage decomposition is post hoc.
We would welcome a pointer to any other study that holds the clinical task, record, actions and grader fixed while varying the route.
Response to Reviewer RurE (1/2)
We answer both points in order. Section and table numbers refer to the revised PDF; Tables 1 to 22 keep their submitted numbers. The general comment describes the corrections we made after the reviews. The numbers below are the corrected ones.
In short, we agree that interfaces change agent scores and do not claim that finding as ours. What we add is a comparison on clinical tasks with chart writes in which the actions, executor and grader are held fixed and only the route to the record changes. The closest prior comparisons also varied what agents could observe or do, and their results point in different directions. On size, we are adding about twenty harder tasks. On access, reviewers will be able to inspect the tasks, rubrics, records and every trial's trajectory through an anonymized link.
On size and intervals, we agree. The study has ten tasks, and ten tasks are its effective sample size: the task is the unit of inference, every interval resamples tasks, and the primary tests are exact sign-flip tests over the ten per-task differences (Section 5). The 1,500 trials narrow each cell's estimate but add no independent tasks. The intervals are correspondingly wide (Table 3).
Section 6 therefore draws no conclusion about any single model's deficit. Section 8 states that ten tasks and ten trials per cell give wide per-model intervals, and Section 1 says that ten tasks cannot support claims about clinical competence. The variance comparison now says only that modality and model account for comparable shares, with overlapping intervals (Table D.3).
We are adding about twenty harder tasks and will report their results (general comment); none exist yet.
As for the proprietary materials, through an anonymized link that we will share in a comment visible only to them, the reviewers and the AC will be able to inspect the task specifications, grading contracts and rubrics, the ten synthetic records, the first requests sent to each model, the per-trial outcome tables and the full trajectory of every trial (general comment, 2/2). The harness, the simulated EHR application, the chart generator and the task-building tooling are not included and remain proprietary. We do not plan a public release, so eventual readers will not have these materials. The public paper gives the successes in all 150 cells (Table 11), from which the pooled rates, the paired differences and the exact sign-flip tests can be recomputed, but readers cannot inspect the tasks, rubrics or trajectories themselves.