acceptodds
Under review as a conference paper at ICLR 2027

How Agents Reach the Record Matters: A Parity-Controlled Study of Access Modality for Clinical EHR Agents

Abstract

How access modality affects the measured performance of agents that act on electronic health records (EHRs) remains poorly characterized. We evaluated five frontier models from OpenAI, Google and Anthropic on ten EHR tasks, from chart retrieval to consults with orders and signed notes, under three access modalities: Pixel (screenshots, mouse and keyboard), FHIR (patient-scoped FHIR search and read) and Packet+Action (the full record in the prompt, with FHIR's actions). Tasks, synthetic records, actions and grading were held constant under a protocol frozen before any model call. Physicians wrote or reviewed each task's clinical content. Grading combines deterministic checks on the final chart and recorded actions with a blinded judge for consult notes. Across 1,500 trials (10 per cell), pass@1 was 32.4% under Pixel, 54.0% under FHIR and 60.0% under Packet+Action. Exact task-level sign-flip tests place Pixel 21.6 points below FHIR (p = 0.031) and 27.6 below Packet+Action (p = 0.008); the structured modalities do not differ reliably (p = 0.109). Every model performed worst under Pixel, between-model differences were largest under Pixel (6.0% to 48.0% pass@1), and modality explains about as much variance in pass rates as model choice. Most of the Pixel deficit arises in reaching evidence and in finishing and persisting writes; for one model it may largely reflect the screen driver. Auditing our harness exposed parity defects, and every affected cell was rerun under a frozen plan. These results establish access modality as a critical dimension of EHR-agent evaluation, and we state the parity requirements such comparisons need.

Primary area:
datasets and benchmarks
Keywords:
electronic health records, clinical AI agents, computer-use agents, FHIR, tool use, benchmarks, evaluation methodology, access modality, LLM-as-a-judge

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_RurEOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a new 10-task clinical EHR benchmark and uses it to show that model performance varies as a function of the way the task is presented to the model. Computer use performs the worst, with FHIR a bit better, and then plain text is better than that.

Critical strengths
  • The core argument, that modality is important, is sound
Critical weaknesses
  • The proposed 10-task benchmark is not scientifically valid. First, it's proprietary, so we reviewers can't audit it, and neither can the eventual readers of this work. We have no idea if it's set up correctly, or even many of the details of how it is set up. Second, it's tiny. 10 tasks are barely anything. The confidence intervals in this work are massive.
  • The novelty is unclear. Prior work already establishes that observation and action interfaces substantially affect agent performance. Examples include HealthAdminBench’s (accessibility-tree vs screenshot observations), Beyond Browsing’s (API-based vs browsing) and MCPWorld (API, GUI, vs hybrid).
Questions for authors

None

2 comments
AuthorsOpenReview ↗

Response to Reviewer RurE (1/2)

We answer both points in order. Section and table numbers refer to the revised PDF; Tables 1 to 22 keep their submitted numbers. The general comment describes the corrections we made after the reviews. The numbers below are the corrected ones.

In short, we agree that interfaces change agent scores and do not claim that finding as ours. What we add is a comparison on clinical tasks with chart writes in which the actions, executor and grader are held fixed and only the route to the record changes. The closest prior comparisons also varied what agents could observe or do, and their results point in different directions. On size, we are adding about twenty harder tasks. On access, reviewers will be able to inspect the tasks, rubrics, records and every trial's trajectory through an anonymized link.

  1. The study is proprietary, tiny, and has very wide intervals. “First, it's proprietary, so we reviewers can't audit it, and neither can the eventual readers of this work. … Second, it's tiny. 10 tasks are barely anything. The confidence intervals in this work are massive.”

On size and intervals, we agree. The study has ten tasks, and ten tasks are its effective sample size: the task is the unit of inference, every interval resamples tasks, and the primary tests are exact sign-flip tests over the ten per-task differences (Section 5). The 1,500 trials narrow each cell's estimate but add no independent tasks. The intervals are correspondingly wide (Table 3).

  • In the pooled results, Pixel − FHIR is −21.6 points [−40.4, −5.4] (p = 0.031) and Pixel − Packet+Action is −25.8 [−43.8, −9.4] (p = 0.016). Leaving out any one task keeps both negative (Table 13). On the odds scale, the GEE interval for Pixel − FHIR, 0.39 [0.16, 1.001], just includes one.
  • Per model, the intervals are far wider; for Claude Fable 5.1, Pixel − FHIR is −7.0 [−47.0, +32.0]. None of the fifteen per-model tests survives Holm or Benjamini–Hochberg correction (smallest adjusted p = 0.176; Table 14).

Section 6 therefore draws no conclusion about any single model's deficit. Section 8 states that ten tasks and ten trials per cell give wide per-model intervals, and Section 1 says that ten tasks cannot support claims about clinical competence. The variance comparison now says only that modality and model account for comparable shares, with overlapping intervals (Table D.3).

We are adding about twenty harder tasks and will report their results (general comment); none exist yet.

As for the proprietary materials, through an anonymized link that we will share in a comment visible only to them, the reviewers and the AC will be able to inspect the task specifications, grading contracts and rubrics, the ten synthetic records, the first requests sent to each model, the per-trial outcome tables and the full trajectory of every trial (general comment, 2/2). The harness, the simulated EHR application, the chart generator and the task-building tooling are not included and remain proprietary. We do not plan a public release, so eventual readers will not have these materials. The public paper gives the successes in all 150 cells (Table 11), from which the pooled rates, the paired differences and the exact sign-flip tests can be recomputed, but readers cannot inspect the tasks, rubrics or trajectories themselves.

  1. You question our novelty against HealthAdminBench, Beyond Browsing and MCPWorld. “Prior work already establishes that observation and action interfaces substantially affect agent performance. Examples include HealthAdminBench’s (accessibility-tree vs screenshot observations), Beyond Browsing’s (API-based vs browsing) and MCPWorld (API, GUI, vs hybrid).”

We agree that these studies show the interface changes agent scores, and we do not claim that finding as new, but they answer a different question from ours. In each, the compared agents differ in more than the route to the data. These are reasonable choices for their goals, but they mean the route itself is not isolated:

  • HealthAdminBench studies administrative workflows (prior authorization, appeals, equipment orders) rather than clinical care. Both arms of its ablation use the graphical interface with the same actions: one observes screenshots, the other an accessibility tree with element identifiers that, as its authors note, real EHR and payer systems typically do not expose. It has no API or full-record arm.
  • Beyond Browsing compares a browsing agent that reads the accessibility tree, not screenshots, with an agent that uses each website's API. API coverage varies by site, from 988 endpoints on one to 31 on another, for which the authors added APIs, and some features have no API. The two agents therefore have different sets of possible actions.
  • MCPWorld evaluates one model, and its GUI-only agent can also run shell commands. Its API-only agent scored lowest (53% against 71% for GUI-only), and its authors attribute 48% of the API-only failures to insufficient API coverage.
  • OSWorld-MCP (Jia et al., ICLR 2026) adds MCP tools to graphical agents on OSWorld tasks. The tools generally raise success, but even the strongest models invoke them in only about a third of cases. Its agents keep the screen and gain actions, so the route and the action set change together.
  • GUI vs. CLI (Zhou et al., 2026) gives screen-only and command-line agents the same 440 desktop tasks, starting states and verifiers. The command-line agent acts through skills rather than the screen's operations, so the action set changes with the route, and none of its tasks involves a clinical record. Its strongest command-line agent trailed its strongest screen-only agent, a different model (48.2% against 59.1%). Verifier-guided skill patching raised the command-line agent to 69.3%, which the authors present as a diagnostic upper bound, not a baseline.

The direction differs across these studies: the API agent leads in Beyond Browsing and trails in MCPWorld, adding tools helps in OSWorld-MCP, and in GUI vs. CLI the command-line agent trails unless its skills are patched. In each, the result depends on how much of the task the non-graphical route covers or how often agents use it, and that is what our design holds fixed. The tasks, records, episode clock, actions, executor, pre-save validation and grader are the same under every modality, for five models from three providers. Only the route changes, and one route needs no retrieval at all. Section 2 (pages 2–3) and Table 22 (Appendix J) now set out these differences, and Section 2 now cites OSWorld-MCP and GUI vs. CLI. This paper adds three things.

  • It gives a controlled comparison on clinical tasks. Our tasks include orders, medication changes, coded allergies and signed notes, graded on the chart state after the episode and on the record of each write, not on a final answer alone.
  • It gives evidence that parity decides such comparisons. One answer-bearing schema took a structured pilot task from 8/8 passes to 0/8 (Table 6). After the reviews, we found that our own packet on five tasks was task-filtered, which favoured Packet+Action (Appendix B.1). Section 3 states the parity requirements that comparisons across access modalities need, and Table 17 gives the direction of every remaining asymmetry.
  • It shows where the gap arises. Process checks and a first-failed-stage decomposition identify failures to reach the evidence and to finish and persist writes as major contributors to the Pixel deficit. On nine of the ten original tasks, among trials that had seen the evidence and persisted their writes, 57.6% passed the decision gates and the judge under Pixel against 62.5% under FHIR, a difference of −4.9 points (95% CI −27.8 to 22.0; Section 7), under extracted-answer scoring as in the revised Section 7. This analysis is exploratory: the process checks were defined after the run, and the stage decomposition is post hoc.

We would welcome a pointer to any other study that holds the clinical task, record, actions and grader fixed while varying the route.

Reviewer_ZLLYOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper isolates the effect of access modality on clinical EHR agent performance. Holding task, synthetic record, actions, and grader fixed, it varies only how the agent reaches the record: PIXEL (screenshots + mouse/keyboard), FHIR (patient-scoped API), and PACKET+ACTION (full record in prompt). A large part of the paper is a candid audit of parity defects in their own harness.

Critical strengths
  1. Holding task, record, actions, and grader fixed while varying only the interface is the right way to isolate the modality effect, on a genuinely under-studied question for clinical agents.
  2. The parity audit is the paper's best contribution — they found fairness bugs in their own harness (e.g. a schema that swung a pilot 8/8→0/8), documented each in Appendix B, and reran affected cells.
  3. A score on one interface says little about another — a useful, probably-correct message for how the field reports and compares agent results, independent of the shaky magnitudes.
Critical weaknesses
  1. Screen penalty ≈ bad driver, not bad modality. PIXEL uses a "driver" to translate model actions into clicks — OpenAI models got a strong native one, the others a weak generic one. Remove the driver's own failures and Opus's screen score jumps 41%→68% (above its API score), so its deficit essentially vanishes. Driver and model are confounded, and no experiment separates them. The "modality" effect is partly just harness quality.
  2. Everything rests on 10 tasks. Every test treats a task as a single data point, so effective n=10, not 1,500 — the 10 trials per cell only cut within-task noise, they add no independent evidence. No individual model's deficit survives multiple-comparison correction (smallest adjusted p=.176). Several tasks carry no signal (everyone fails NSAID; everyone passes BP/eGFR/potassium under the API), so the effective count is below 10. The flagship claim "modality matters as much as which model you pick" comes from a variance split on 15 cell averages with no error bars, no stated method, and ~11% unexplained.
  3. Post-hoc analysis. They swapped the pre-registered statistical method after seeing results; the replacement tests even disagree with each other (one significant, one not).
  4. Unreproducible + fragile fairness. Tasks, records, and grading are all withheld. The method hinges on the three interfaces being perfectly fair, yet one schema bug swung a pilot 8/8→0/8 and 5 of 9 fairness bugs surfaced only after the main run — so a remaining bug that flips a result can't be ruled out. The authors' company sells clinical-AI evaluation tools (vendor conflict of interest), which raises the bar on withholding materials.
  5. Task selection/difficulty unexplained. No selection criterion ("why these 10?"), no difficulty measure, and no description of the charts — not even record counts or lengths. Two tasks were machine-generated. You can't tell whether the set favors interface-sensitive tasks.
Questions for authors
  1. Can you run at least one model under both PIXEL drivers (native vs generic function-calling)? Removing driver failures erases Opus's entire deficit, so this is the key test to separate driver quality from modality.
  2. How were the 10 tasks selected, and can you give basic dataset statistics (record counts, chart length, token load in PACKET+ACTION)? Without a selection criterion or difficulty measure, readers can't rule out a set enriched for interface-sensitive tasks.
  3. Where do the 0.75 pass thresholds and the critical/non-critical mapping come from, and how sensitive are the primary contrasts to them?
  4. Can you reconcile the rerun counts (350 "rerun" vs 250 addenda in Table 8 vs 350 "superseded"), and are Tables 11–12 meant to be cited?
4 comments
AuthorsOpenReview ↗

Response to Reviewer ZLLY (1/4)

We answer each point in order. Section and table numbers refer to the revised PDF; Tables 1 to 22 keep their submitted numbers. The general comment describes the corrections we made after the reviews. The numbers below are the corrected ones.

In short, we make five points. (1) We reran Claude Opus 5.5 under Anthropic's own screen driver. It passed 48.0% of Pixel trials, against 41.0% under our adapter and 67.0% under FHIR, so its Pixel failures persist; the driver may explain part of the gap. (2) We draw no conclusion about any single model's deficit, and the variance comparison now has a stated method and intervals. (3) We correct the record: the analysis plan was not hash-committed; the planned model is reported beside ours. (4) Task selection, chart statistics and threshold sensitivity are added, and the rerun counts are reconciled (Table D.5). (5) Reviewers will be able to inspect the tasks, rubrics, records and every trial's trajectory through an anonymized link.

  1. The Pixel driver is confounded with the model. “Remove the driver's own failures and Opus's screen score jumps 41%→68% (above its API score), so its deficit essentially vanishes. Driver and model are confounded, and no experiment separates them.”

“Can you run at least one model under both PIXEL drivers (native vs generic function-calling)?”

We agree that driver and model are confounded across models, and Section 6 now says so.

The drivers differ because the harness was first built for an OpenAI model, and its native computer-use driver implemented only OpenAI's computer-use interface. During the pilot we added the Claude and Gemini models through a function-calling adapter that exposes the same mouse and keyboard actions. We accepted this because the primary contrasts are within each model, and the choice carried into the frozen protocol. It was an engineering choice, not a judgement about the models.

Appendix E uses the data to bound the driver's contribution. It excludes, in every modality, every episode that ended on the rejection limit, the episode deadline, or a turn or output-token limit. There are 60 such episodes, all under Pixel.

  • Pixel − FHIR moves from −21.6 to −17.2 points [−35.1, −1.4], and Pixel − Packet+Action to −21.4 [−38.6, −5.7].
  • As you note, most of the change is Claude Opus 5.5. Its Pixel pass@1 rises from 41.0% to 68.3%, above its 67.0% under FHIR. The run below tests whether that reflects the driver.
  • No other model moves by more than 6.5 points: Claude Fable 5.1 goes from 40.0% to 46.5%, Gemini 3.1 Pro from 6.0% to 6.3% and GPT-6 Astra from 48.0% to 48.5%, and GPT-5.6 Sol is unchanged at 27.0%.
  • If every excluded episode had passed, Pixel − FHIR would be −9.6 [−25.0, 6.4].

Table 17 lists the driver differences: shorter retained history, a rejection limit, and generic feedback on an undeclared triple-click action. Each would inflate the Pixel deficit of the Claude and Gemini models, the last mainly for Opus.

As you asked, we ran one model, Claude Opus 5.5, under a second driver that uses Anthropic's own computer-use interface over the same shared actions, with the same tasks, records, prompts, limits and grading. A plan frozen before its first model call fixed five trials per task and the comparison with the paper's 100 function-calling Opus trials. Section 6 and Appendix E report it.

  • Opus passed 24 of 50 Pixel trials (48.0%) under the native driver, against 41 of 100 (41.0%) under the function-calling driver. The mean per-task difference is +7.0 points (exact sign-flip p = 0.50, the smallest value the test can reach when only two tasks differ). Depression (4/5 against 4/10) and spirometry (5/5 against 7/10) rose, eight tasks tied, and none fell.
  • Driver-bound endings fell from 40 of 100 trials to 3 of 50. The five tasks Opus never passed under the function-calling driver, it again never passed.
  • Opus's Pixel failures therefore persist under its native driver. The bound's 68.3% lies well above the 48.0% the native driver reached, and the abstract now says that this model's failures persisted under its provider's own driver. The run does not show that the driver has no effect: it covers one model, with five native trials per task.
  • Besides the driver, the arms differ in serving route and date (Amazon Bedrock in November against Anthropic's API in September). Twelve attempts were set aside as harness defects by an outcome-blind rule and rerun (Appendix E); all 50 deciding trials are valid.
  1. The effective n is 10, and no per-model deficit survives correction. “Every test treats a task as a single data point, so effective n=10, not 1,500 … No individual model's deficit survives multiple-comparison correction (smallest adjusted p=.176).”

We agree with both points. The paper's claims are now limited accordingly.

  • The task is the unit of inference (Section 5). Every interval is a task-clustered bootstrap, and the primary tests are exact sign-flip tests over all 2^10 sign assignments to the ten per-task differences.
  • In the pooled results, Pixel − FHIR is −21.6 [−40.4, −5.4] (p = 0.031) and Pixel − Packet+Action −25.8 [−43.8, −9.4] (p = 0.016). Leaving out any one task keeps both negative: −13.8 to −25.8 and −18.4 to −30.4, with p up to 0.063 and 0.031 (Section 6, Table 13).
  • Section 6 now draws no conclusion about any single model's deficit, because none of the fifteen per-model sign-flip tests survives Holm or Benjamini–Hochberg correction (smallest adjusted p = 0.176 under both; Table 14). After the packet correction, Gemini 3.1 Pro's Pixel − Packet+Action gap is no longer nominally significant either (p = 0.156, from 0.047).
  • We do not rely on one further result. Table 14 also reports per-model GEE fits. They rest on large-sample approximations with ten clusters, and after Holm they leave one contrast below 0.05 (GPT-5.6 Sol, Pixel vs. Packet+Action).
  • Some tasks sit at the floor or ceiling. NSAID is near the floor in every modality and allergy in both structured ones, so they carry little signal. Leaving out NSAID widens the gaps to −24.0 (p = 0.031) and −27.6 (p = 0.023) (Section 6). A task near the ceiling under FHIR still carries signal when Pixel falls below it, as on potassium, which contributes 9.2 of the 21.6 points; without it the gap is −13.8 (p = 0.063).
  • More tasks are needed. We are adding about twenty harder tasks and will report their results (general comment).
  1. "Modality matters as much as model choice." “The flagship claim "modality matters as much as which model you pick" comes from a variance split on 15 cell averages with no error bars, no stated method, and ~11% unexplained.”

The criticism was correct, and the claim is weaker now. Section 6 says only that modality and model choice account for comparable shares of the variance in pass rates.

  • Appendix D.1 gives the method. We decompose the variance of the fifteen model–modality pass rates into a modality part, a model part and their interaction. There is no error term, so the interaction, now 7% [3, 41], also absorbs task sampling and trial noise. That is the "unexplained" part you identified.
  • With task-clustered bootstrap intervals (Table D.3), modality accounts for 41% [5, 78] and model for 52% [14, 84]. The intervals overlap almost entirely, and modality's share exceeds model's in 36% of task resamples. On the corrected data the point estimate is larger for model. We never claim that modality matters more.
  • Two decompositions keep the tasks. On the 150 task–model–modality pass rates, modality is 6.4% [0.8, 20.4] and model 8.0% [1.9, 25.8]. On the logit scale they are 5.7% and 7.5%. Both are small beside the share of task, 39.3% of the 150-cell variance.
  1. Statistics changed after the results were seen, and the replacements disagree. “They swapped the pre-registered statistical method after seeing results; the replacement tests even disagree with each other (one significant, one not).”

We must first correct the record. The submission said that the protocol and the analysis plan were frozen and hash-committed, but only the execution protocol was: bindings, trial counts, prompts, rubrics, graders, judge, and the rerun, exclusion and outcome rules. The analysis plan was written at the same time as a separate document, before any main-run outcome existed, but it was not hash-committed. Section 5, Appendix C and the conflict of interest statement are corrected.

Continuing point 4, Appendix C now lists two departures from the analysis plan.

  • The plan's confirmatory statistical model was a logistic mixed model with random intercepts for task and for task by model, with Holm's correction over the ten per-model Pixel contrasts. We replaced it after the five-trial outcomes had been scored and before trials 6 to 10 were run. Those outcomes showed a Pixel effect that varies strongly across tasks, which that model treats as noise. On the final trial set, the plan's model gives odds ratios of 0.087 [0.052, 0.14] against FHIR and 0.053 [0.031, 0.093] against Packet+Action (both p < 0.001), and it finds 9 of the 10 per-model contrasts significant (Table D.2), whereas the reported analysis finds none, so the change weakened the conclusions. It was nonetheless made after outcomes were seen, and Section 5 says so.
  • The plan specified a judge–clinician agreement study, and we did not run it (Section 8).

On the disagreement, Appendix D.1 explains it and Table D.1 reports every method side by side. The two tests measure the gap on different scales. The sign-flip test is exact and uses the difference in pass rate, the estimate the analysis plan specified. The GEE pools odds ratios, and on that scale a difference at the floor counts for much more. On allergy, where Pixel passes 16% of trials and FHIR none, a 16-point difference becomes a log odds ratio of 3.0.

  • With all tasks, the GEE odds ratio for Pixel − FHIR is 0.39 [0.16, 1.001] (p = 0.050). Without the two floor tasks it is 0.25 [0.080, 0.76].
  • A mixed model with task-by-modality slopes agrees in direction but gives p = 0.094 against FHIR and p = 0.059 against Packet+Action.

We treat the sign-flip test on the difference in pass rate as primary and the odds-ratio models as checks on direction. We state plainly that the Pixel − FHIR contrast is not significant under either odds-ratio model.

  1. You raise reproducibility, late parity defects and a vendor conflict of interest. “Tasks, records, and grading are all withheld. … 5 of 9 fairness bugs surfaced only after the main run — so a remaining bug that flips a result can't be ruled out. The authors' company sells clinical-AI evaluation tools (vendor conflict of interest), which raises the bar on withholding materials.”

As for reproducibility, through an anonymized link that we will share in a comment visible only to them (general comment, 2/2), the reviewers and the AC will be able to inspect the task specifications, rubrics, records, first requests, per-trial outcomes and the full trajectory of every trial, including every superseded trial. We do not plan to release these materials publicly. The harness, the simulated EHR application, the chart generator and the task-building tooling are not included and remain proprietary, so the trials cannot be rerun, but each verdict can be traced to its chart state and rubric. Appendix I describes the scripts that produce every reported number from the retained trial tables. In the public paper, Table 11 gives enough counts to recompute the pooled rates, the paired differences and the sign-flip tests.

We agree that the late parity defects are a weakness. Five of the nine defects of the submission were found after the main run, and after the reviews we found a tenth, the task-filtered packet (Appendix B.1). Our audit had not compared the evidence each modality received (Section 8). Every defect, its rerun and its effect are in Table 6. The repairs moved results in both directions: GPT-6 Astra's thyroid and NSAID structured cells fell from 13 to 0 of 20 passes. On trials 1 to 5, the parity repairs narrowed the measured gaps, from −26.8 to −21.6 and from −35.2 to −27.6 points (Appendix B). The correction after the reviews narrowed Pixel − Packet+Action further.

Continuing point 5, the conflict of interest statement is now a section of its own. The authors' employer builds clinical AI evaluation environments, funded the work, may benefit commercially and has commercial relationships with developers of some evaluated models. No developer of an evaluated model funded or reviewed the study or was informed of it, and all model access was purchased through standard APIs. The judge shares a vendor with two evaluated models. It passed 81.1% of judged trials from those two models and 84.1% from the two OpenAI models (Appendix E). With four judged tasks, that is a small test.

  1. You ask about task selection, difficulty, chart statistics and two machine-generated tasks. “No selection criterion ("why these 10?"), no difficulty measure, and no description of the charts — not even record counts or lengths. Two tasks were machine-generated.”

Appendix A.3 now describes the selection and Table A.1 gives the statistics.

  • No single criterion applied; the set is purposive and partly opportunistic. Five tasks came from an existing engineering catalogue. Five were specified to add clinical domains and chart writes, using only actions all three modalities support. No task was added or removed because of results, and difficulty was not a selection criterion.
  • Table A.1 gives chart statistics per task: the FHIR resources (38 to 127), notes, time span, serialized size (40.8k to 126.0k characters), the Packet+Action first-request token load (1,198 to 29,619), duplicated, stopped, conflicting and post-clock entries, checks, required writes and pooled pass@1.
  • On difficulty, pooled pass@1 ranges from 0.03 to 0.79 and falls as checks and required writes increase (Spearman ρ = −0.74 and −0.77). The Pixel deficit does not grow with difficulty.
  • Table A.1 marks the two generated tasks (†). Their scenarios were drafted by a language-model task-generation pipeline and then reviewed and revised by physicians, as the AI use statement says.
  1. You ask where the 0.75 thresholds and the critical/non-critical mapping come from. “Where do the 0.75 pass thresholds and the critical/non-critical mapping come from, and how sensitive are the primary contrasts to them?”
  • The judge's 0.75 bar is the grading framework's default and was fixed before any evaluated trial. It was chosen neither on these outcomes nor by clinical review (Section 4, Appendix F).
  • The weighted score's threshold and weights follow an existing checkpoint-weighting scheme for clinical agent tasks. The critical/non-critical mapping is our own and was not reviewed by clinicians. It was made after the main-run outcomes existed, but before any trial was scored under it (Appendix G).
  • Table D.4 tests sensitivity by varying both. Across judge thresholds from 0.50 to 1, Pixel − FHIR ranges from −15.2 to −25.2 points and Pixel − Packet+Action from −17.2 to −30.2. Across the weighted variants the ranges are −15.2 to −35.2 and −17.2 to −41.0. No variant changes the sign of either primary contrast. Significance is not robust at the strict end, however: requiring EXCELLENT on every criterion gives p = 0.172 and 0.109.
  1. You ask us to reconcile "350 rerun", "250 from the addenda" and "350 superseded". “Can you reconcile the rerun counts (350 "rerun" vs 250 addenda in Table 8 vs 350 "superseded") …”

Appendix B and Table D.5 reconcile them.

  • In the submission, the parity addenda ran 350 trials in 250 trial slots, because the thyroid and depression structured cells were rerun twice. The 350 superseded trials were 250 from the main run and 100 from earlier addenda.
  • After the packet correction, the full-record addendum ran 250 more trials. The final set now holds 225 parity-addendum and 250 full-record trials, and 600 trials are superseded: 350 from the main run, 125 from trials 6 to 10 and 125 from the parity addenda. The 1,500 final trials are 400 from the main run, 225 from the parity addenda, 250 from the full-record addendum and 625 from the extension (Appendix B).
  1. You ask whether Tables 11 and 12 are meant to be cited. “… and are Tables 11–12 meant to be cited?”

Yes, both are now cited. Section 6 cites Table 11 for the per-cell counts, from which the headline rates can be recomputed exactly; the reproducibility statement also cites it. Section 6 cites Table 12 for pass^k per model.

Reviewer_FVqeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper identifies that most EHR-related benchmarks are unclear with the modality used for chart retrieval - whether screenshots of the EHR, or structured EHR entries (e.g., FHIR/full record in prompt + FHIR actions), so evaluate differences in performance based on this, holding other clinical variables fixed.

Contributions:

  • controlled comparison of route to record, including ensuring parity in the comparison between modalities
  • reporting modality effect and failure modes

Validations:

  • ensured parity in comparing modalities via auditing their harness
Critical strengths
  • When people base medical activities on frontier models based on benchmark results, it's extremely important that those benchmarks help make decisions. Identifying a key component of benchmarks and potential deployments that is not well-studied is an important contribution.
Critical weaknesses
  • Concerns about presentation - paper relies on an understanding of FHIR, but there is not really an introduction to this apart from mentioning it's a server/api/providing some citations; paragraph starting around 67 is difficult to parse in the intro when we don't know decision space, visible state, evidence, pilot, main run, etc. - remains unclear when discussed in line 183; would help to add a table showing what current benchmarks the Pixel, FHIR, and Packet+Action test cases approximate

Update after response: I concur with other reviewers - it's hard to envision how the community could use this with so little released, especially when it's synthetic, not real patient data. It seems almost more like a position paper to look out for this discrepancy. I also remain skeptical given the authors' point about fully drafting the manuscript with AI and then revising it, though I appreciate the additional clarity on their AI use. I leave my rating as is.

Questions for authors
  • Can you clarify the audit and why only a subset of cases were rerun?

  • Do you expect that using real patients would change the findings?

  • Why were tasks selected?

  • What would happen if you created a new Pixel test that had an additional verification step?

  • Will task specifications and rubrics be released upon potential publication?

3 comments
AuthorsOpenReview ↗

Response to Reviewer FVqe (1/3)

We answer each point in order. Section, table and page numbers refer to the revised PDF; Tables 1 to 22 keep their submitted numbers. The general comment lists the corrections we made after the reviews. One of them changes the Packet+Action numbers, and the numbers below are the corrected ones.

In short, Section 1 now introduces FHIR and defines each term before using it; Table 22 maps existing benchmarks onto our three modalities; Appendix A.3 and Table A.1 explain how the tasks were chosen and give chart statistics; the task specifications, rubrics, records and every trial's trajectory will be shared with reviewers through an anonymized link (general comment, 2/2); and the AI use statement now sets out each use of AI (item 9).

  1. The paper lacks an introduction to FHIR. “paper relies on an understanding of FHIR, but there is not really an introduction to this apart from mentioning it's a server/api/providing some citations”

We agree. Section 1 (page 2) now has a paragraph on FHIR. It covers:

  • the HL7 standard for exchanging health records through a web API;
  • how it represents a chart as typed resources (Patient, Observation, MedicationRequest);
  • the difference between a search and a read;
  • why it is the realistic programmatic route to a chart;
  • what patient-scoped access means here: the interface answers only requests about the task's patient.
  1. Terms around line 67 are undefined and still unclear at line 183. “paragraph starting around 67 is difficult to parse in the intro when we don't know decision space, visible state, evidence, pilot, main run, etc. - remains unclear when discussed in line 183”

We agree; the paragraph assumed vocabulary it had not introduced. Section 1 (page 2) now defines each term where it first appears:

  • Decision space is the set of actions and choices open to the agent.
  • Visible state is everything the agent can observe of the record.
  • Evidence used for grading means the captures of the chart before and after the trial and the record of each write.
  • The pilot is a single trial per cell for four models.
  • The main run is the frozen first five trials of every cell.
  • Parity addenda are short plans, each frozen before its runs.
  • The extension is trials 6 to 10 of every cell.

Section 4 defines the task contract: the frozen specification of what the task requires and forbids.

The parity paragraph that was at line 183 is now in Section 3 (page 5). It states each of the three requirements together with the defect in our harness that taught it:

  • A structured tool must admit the UI's choices, wrong ones included. Our first structured allergy tool offered only the correct choices; with the UI's open fields, the structured pilot cells fell from 8/8 to 0/8 passes.
  • Whatever the screen shows or allows must reach structured agents, and whatever the screen offers must work.
  • Grading fails closed: a trial whose grading evidence is incomplete cannot pass.
  1. A table should map the modalities to existing benchmarks. “would help to add a table showing what current benchmarks the Pixel, FHIR, and Packet+Action test cases approximate”

Table 22 (Appendix J, pages 34–35) does this. It covers OSWorld, WebArena, VisualWebArena, Beyond Browsing, MCPWorld, HealthAdminBench, MedAgentBench, FHIR-AgentBench, EHRAgent, MedAlign and HealthBench. For each it gives the observation channel, the action channel, the environment and the closest modality here, and Section 1 points to the table.

  1. Why was only a subset of cells rerun after the audit? “Can you clarify the audit and why only a subset of cases were rerun?”

Section 3 (page 5) gives the rule. A cell was rerun, with all five of its main-run trials, only if a repair could change what its agent was sent or could do. For each structured repair, we checked that the first request to the model provider changed for exactly the intended cells. It stayed byte-identical for every other structured cell, so rerunning those would have repeated the same inputs. The one Pixel repair, to the patient search, affected only the two retrieval tasks in which the agent must find the patient.

Each addendum was frozen and hash-committed before its runs and preceded by provider-free controls (160/160 passed). Table 6 lists each defect and its effect, and Table D.5 lists every rerun and the trials it replaced. Table 16 (e) reports the trial set before the parity addenda as a sensitivity analysis.

After the reviews we found one further defect, the task-filtered packet. Because it changed what Packet+Action agents were sent on five tasks, we reran all 250 Packet+Action trials of those tasks (Appendix B.1).

  1. Would real patients change the findings? “Do you expect that using real patients would change the findings?”

They might, and the study cannot tell. Section 8 says that the findings rest on synthetic records and a simulated application, so neither the difficulty nor the failure modes need match production EHRs. We claim no transfer to production systems.

Three things bear on the question:

  • The records include hazards found in real charts. Section 3 lists duplicate rows, a stopped duplicate of an active order, conflicting values on the same date and rows dated after the clock, and Table A.1 counts them per task. Some Pixel failures arise from exactly these hazards. In the pilot, on conflicting labs, three of four Pixel agents reported only the value the grid displays, although the record holds two values for that date (Section 7).
  • The records are small. They hold 38 to 127 FHIR resources (Table A.1), and every Packet+Action first request leaves at least 93% of the model's context window free (Section 3, Appendix B.1). Larger real charts could change the Packet+Action results in particular, which we did not test.
  • The comparison is within task. Every comparison uses the same record, actions and grader, so a property of a record is shared by all three modalities. That isolates the access route on these records but does not show that the gap has the same size on others.

Section 8 therefore recommends acceptance testing through the deployed interface. The twenty tasks we added (see our comment on the results on all 30 tasks) also use synthetic records, so they do not answer this question either.

  1. Why were these tasks chosen? “Why were tasks selected?”

Appendix A.3 (page 17) answers this, and Table A.1 (page 18) gives the per-task statistics. The ten tasks are a purposive and partly opportunistic set, not a sample of clinical work:

  • Five came from an existing engineering catalogue. Three are tasks on which access-modality comparisons had already been run. The other two were the only remaining entries with a pinned source record, a deterministic answer contract and verified controls.
  • Five were specified to add clinical domains the first set lacked and to require chart writes. They use only actions that all three modalities support: a signed note, a catalogue laboratory order, a medication stop or hold, and a coded allergy. Three were adapted from existing tasks, and two were first drafted by a language-model task-generation pipeline and then reviewed and revised by physicians (Table A.1 marks them).

No task was added or removed because of results, and difficulty was not a selection criterion. Pooled pass@1 nonetheless ranges from 0.03 to 0.79 across tasks.

  1. What if Pixel adds a verification step? “What would happen if you created a new Pixel test that had an additional verification step?”

We did not run such a variant, so we cannot say how much of the gap it would close. The existing analyses show where it could act. The figures below are under extracted-answer scoring, as in the revised Section 7; the frozen-rule values are in Appendix M.

  • Some agents never saw the evidence. Among Pixel failures, 34% never had at least one key fact on screen (Appendix H). Section 8 already recommends that evaluations of UI automation check that the agent saw the values it acted on.
  • Some agents never finished or persisted their writes. On the nine tasks of the stage analysis, 9.4 points of the 22.4-point Pixel − FHIR gap come from episodes that ended early or never persisted their writes (Section 7). The signed-note readback fails in 86 Pixel trials, at most 5.6 points of pass@1 (Table 17).
  • Agents that got that far. Among trials that had seen the evidence and persisted their writes, 57.6% passed the decision gates and the judge under Pixel, against 62.5% under FHIR and 64.7% under Packet+Action; Pixel − FHIR at this stage is −4.9 points (95% CI −27.8 to 22.0; Section 7).

Two cautions apply.

  • More steps do not always help. Pixel already has a 7,200 s budget with no turn cap, except on the allergy task (Table 1), and 20 of the 23 deadline endings were cycles that had stopped making progress. In 7 of them the agent had committed the correct chart changes but never returned the required answer (Appendix E).
  • A verification step would change the Pixel route as built. A fair comparison would give the structured modalities an equivalent step. At present they cannot read back their own writes, an asymmetry we disclose in Table 5.
  1. Will the specifications and rubrics be released? “Will task specifications and rubrics be released upon potential publication?”

The task specifications and rubrics, together with the answer keys, records, per-trial outcomes and every trial's trajectory, will be available to the reviewers and the AC through an anonymized link (general comment, 2/2); we do not plan a public release (reproducibility statement, Appendix I). The harness, the simulated EHR application, the chart generator and the task-building tooling are not included and remain proprietary. In the public paper, Table 11 gives the successes in all 150 cells, from which the pooled pass rates, every paired difference and the exact sign-flip tests can be recomputed.

  1. The ethics flag says the language seems LLM-like. “Some language seems a little LLM-like, not sure what to do about this (feel free to ignore if this is not the right venue to report suspicions)”

The AI use statement (pages 11–12) discloses how generative AI was used:

  • The synthetic patient records were partly generated with language models.
  • The clinical scenarios of two of the ten tasks came from a language-model task-generation pipeline and were then reviewed and revised by physicians.
  • The prompts and grading implementations were adapted from the physicians' clinical content with AI assistance and reviewed by the authors.
  • AI assistants wrote parts of the harness, the grader and the analysis scripts, and helped analyse the results.
  • AI assistants helped draft the execution protocol and the analysis plan, which the authors reviewed and approved. This sentence is new in the revision.
  • AI assistants searched for related work and checked each citation against its publisher, arXiv or DOI record.
  • AI assistants drafted and edited the manuscript text, including the passages added or rewritten in this revision, and the authors reviewed it and directed its revision. The same holds for the original submission: AI assistants wrote the first draft of most of the manuscript text from our study records, results files and instructions, and made the subsequent edits at our direction. We wrote the rest, set the content, structure, style and disclosures, and reviewed the drafts and directed revisions before submitting.

The reported results are inserted into the text from files that the analysis scripts generate from the retained trial records, not typed by hand. The authors take responsibility for all content, including text produced with AI assistance.

In this revision we simplified the introduction and split long sentences elsewhere. If a passage still reads poorly, please point us to it and we will fix it.

Comments on the paper

5 comments
AuthorsOpenReview ↗

General response: revisions and post-review corrections (1/2)

We thank the reviewers and have uploaded a revised PDF, whose main text ends on page 10. In brief, we make four points. (1) We reran Claude Opus 5.5 under Anthropic's own screen driver. Its Pixel pass rate rose from 41.0% to 48.0%, against 67.0% under FHIR, so its Pixel failures persist. (2) We draw no conclusion about any single model. The claims rest on the pooled task-level tests: Pixel is 21.6 points below FHIR (p = 0.031) and 25.8 below Packet+Action (p = 0.016), and both gaps stay negative when any one task is dropped. (3) Earlier interface comparisons also changed what agents could see or do; ours changes only the route to the record (Section 2). (4) Reviewers and the AC will get an anonymized link to the study materials (part 2/2). We also made four corrections after the reviews, listed below. None changes the sign or significance of a primary contrast. Section and table numbers are unchanged; new tables are A.1, B.1 and D.1 to D.5.

Main revisions

  • Section 1 introduces FHIR and defines parity, decision space, visible state, the evidence used for grading, the pilot, the main run, the parity addenda and the extension. Section 4 defines the task contract.
  • Section 2 and Table 22 (Appendix J) set out how our design differs from HealthAdminBench, Beyond Browsing and MCPWorld, and map existing benchmarks onto the three modalities.
  • Section 3 states which cells were rerun and why. Table D.5 reconciles every rerun count.
  • Appendix A.3 and Table A.1 describe how the ten tasks were chosen and give per-task chart statistics, the Packet+Action token load and difficulty.
  • Section 4 and Appendices F and G state where the 0.75 thresholds and the critical/non-critical mapping come from. Table D.4 varies both.
  • Section 6 and Appendix D.1 draw no conclusion about any single model's deficit. They give the method and intervals of the variance comparison, explain why the sign-flip test and the GEE disagree, and show the planned analysis beside the reported one (Tables D.1 to D.3).
  • The conflict of interest statement is now a section of its own, and the AI use statement is extended.

Corrections made after the reviews

  1. On five tasks (BP, eGFR, conflicting labs, spirometry and potassium), the Packet+Action packet was a short summary filtered to the task, not the record. The submission described all ten packets as the full record, which was true only of the five consults. The asymmetry favoured Packet+Action on those five tasks, and our own parity audit missed it. We rebuilt the five packets so that every packet holds exactly the resources the task's FHIR modality can reach. We checked them without model calls and reran all 250 Packet+Action trials of those tasks under a new frozen plan, with the execution protocol unchanged. Appendix B.1 also reports a first rerun attempt that we stopped, none of whose trials counts, and a drift control, a check that the models had not changed since September. The corrected results are primary throughout (Appendix B.1, Table B.1):
    • Packet+Action pass@1 falls from 60.0% to 58.2%. Pixel (32.4%) and FHIR (54.0%) are unchanged.
    • Pixel − Packet+Action is −25.8 points [−43.8, −9.4], p = 0.016, against −27.6 (p = 0.008) as submitted. Pixel − FHIR is unchanged at −21.6 [−40.4, −5.4], p = 0.031.
    • Packet+Action − FHIR is +4.2 [1.6, 7.0], p = 0.031, against +6.0 (p = 0.109). It remains a small secondary contrast.
    • The variance shares of the fifteen model–modality pass rates move from 48% to 41% [5, 78] for modality and from 41% to 52% [14, 84] for model. We now say only that the two shares are comparable.
    • No per-model contrast survives Holm or Benjamini–Hochberg correction, with either the submitted or the rebuilt packets.
  2. The submission said that the protocol and the analysis plan were frozen and hash-committed, but only the execution protocol was. The analysis plan was written at the same time as a separate document and was not hash-committed. Section 5, Appendix C and the conflict of interest statement are corrected. Appendix C lists two departures from the plan: the change of statistical model, and the clinician agreement study for the judge, which the plan specified and we did not run.
  3. The potassium record holds entries dated after the episode clock. They are distractors by design, and every modality receives them. The parity tables no longer describe the record as "the record as of the clock" (Table 5, Appendix B.1).
  4. On BP and eGFR, answers that give the patient identifier in a prefixed form fail the frozen contract in every modality (40 trials), and the grader was not changed. Appendix B.1 reports accepting that form as a sensitivity analysis only. Gemini 3.1 Pro's Packet+Action pass@1 fell from 49.0% to 35.0% because, with the full record in its prompt, it more often gave the identifier in this form.
AuthorsOpenReview ↗

General response: materials, extension and open items (2/2)

Materials for reviewers

We will share an anonymized link in a comment visible only to the reviewers and the AC. It will contain:

  • the task specifications, grading contracts, rubrics and answer keys;
  • the ten synthetic patient records;
  • the first request the harness sends in FHIR and Packet+Action, for both packet revisions;
  • the per-trial outcome tables: the 1,500 final trials, every superseded trial, and the drift control;
  • the full trajectory of every final and superseded trial: the provider transcript, the Pixel screenshots, the final chart state and the grader's result for each check;
  • the frozen plans of the packet correction, its drift control and the Opus driver run.

Names and identifiers are replaced for anonymity; prompts, record contents and grading logic are otherwise unchanged. With the trajectories and the per-check results, each verdict can be traced to the chart state and the rubric. The harness, the simulated EHR application, the chart generator and the task-building tooling are not included and remain proprietary. We do not plan to release these materials publicly. The public paper gives the successes in all 150 cells (Table 11), from which the pooled rates, the paired differences and the exact sign-flip tests can be recomputed.

More tasks

Ten tasks are the effective sample size of this study, and the revision does not change that. We are adding about twenty harder tasks and will report their results; none exist yet.

Screen driver

OpenAI models drove the screen through OpenAI's native computer-use tool; the other models used a function-calling adapter with the same actions, because the harness was first built for OpenAI's interface. Driver and model are therefore confounded across models. To separate them for one model, we reran Claude Opus 5.5 under Anthropic's own computer-use interface, under a plan frozen before its first model call. It passed 24 of 50 Pixel trials (48.0%), against 41 of 100 (41.0%) under the adapter and 67.0% under FHIR; the mean per-task difference from the adapter is +7.0 points (p = 0.50). Driver-bound endings fell from 40 of 100 to 3 of 50, and it still passed none of the five tasks it had never passed under the adapter. The driver may explain part of this model's deficit, but not all of it. Besides the driver, the two runs differ in serving route and date: Amazon Bedrock in November against Anthropic's API in September (Section 6, Appendix E; reply to Reviewer ZLLY).

AI use

The AI use statement is extended (pages 11 to 12); our reply to Reviewer FVqe lists each use.

What the revision does not change

The study still rests on ten tasks, synthetic records and a simulated application (Section 8). Section 6 now splits the gap: Pixel − FHIR is −32.8 points on the five retrieval and protocol tasks and −10.4 on the five consults; neither group is significant alone (p = 0.125 and 0.375). The judge's agreement with clinicians is untested (Appendix F). The critical/non-critical mapping is our own and has not been reviewed by clinicians (Appendix G). We draw no conclusion about any single model's deficit and make no claim about clinical competence. The TL;DR field cannot be edited during the discussion; with the corrected packets, its range should read 22 to 26 points.

AuthorsOpenReview ↗

Correction: the Packet+Action − FHIR difference depends on answer format

We correct one result reported in the first revised PDF, the general response (1/2) and the summary of changes. They state that Packet+Action exceeds FHIR by 4.2 points [1.6, 7.0], exact sign-flip p = 0.031. That figure depends on how typed answers were graded: the rule behind it accepts a typed retrieval answer only when the reply is the requested JSON object and nothing else. Rechecking the final replies against the answer keys, we found 102 failed retrieval trials whose fields were all correct: 53 from Gemini 3.1 Pro, 42 from Claude Fable 5.1, 5 from Claude Opus 5.5 and 2 from GPT-5.6 Sol. They failed on a prefixed patient identifier, an extra completion field, a code fence or prose around the object.

The revised PDF therefore scores typed answers by extracted-answer scoring: the JSON object is extracted from the reply and passed to the same content checks, so a correct answer counts even when it is embedded in a longer reply. On the paper's ten tasks this gives Packet+Action − FHIR +0.8 [−3.8, 5.0], p = 0.813, and over all 30 tasks, including the 20 new ones, +0.3 [−3.1, 3.3], p = 0.875. We no longer claim that Packet+Action outperforms FHIR or that the design measures a 4.2-point cost of fetching evidence through an API.

The screen deficits hold. Over the 30 tasks, Pixel is 22.1 points below FHIR and 22.4 below Packet+Action (exact sign-flip p < 0.001 for both), and on the paper's ten tasks 24.8 and 25.6. This extends correction 4 of the general response, which covered only the identifier form.

The revised PDF now uploaded corrects the abstract and the main text, which now report extracted-answer scoring. Appendix M gives the results under the rule each protocol froze before its runs, and Table 16 keeps the further answer-format sensitivity rules, with exact p for every row. Table 22, cited in our reply to Reviewer FVqe, is on pages 34–35. The materials we will share with the reviewers and the AC give the final reply of every retrieval trial.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.