acceptodds
Under review as a conference paper at ICLR 2027

PhysicianUIBench: Benchmarking Computer-Use Agents through a High-Fidelity EHR Interface

Abstract

Computer-use agents offer a way to automate clinical workflows through the electronic health record (EHR) interfaces physicians use in practice. Yet existing evaluations provide limited coverage of end-to-end workflows that require both clinical reasoning and multi-step EHR interaction. We introduce PhysicianUIBench, a benchmark for physician-annotated, real-world clinical workflows in a high-fidelity, interactive EHR simulation with an Epic-inspired interface. The benchmark comprises 100 long-horizon tasks and 772 checkpoints. Each task requires an agent, observing only screenshots and acting through mouse and keyboard, to locate evidence across a longitudinal patient chart, reach a clinical decision, and carry it through the interface by placing orders and writing the note, so that chart grounding, clinical reasoning, and EHR execution are exercised within a single workflow. Evaluation combines deterministic checks of application state and interaction logs with rubric-based assessment of clinical documentation. Among all evaluated agents, the highest task success rate is only 35%. These results reveal substantial limitations in current computer-use agents' ability to complete end-to-end clinical workflows reliably. PhysicianUIBench provides a testbed for evaluating how agents integrate clinical reasoning with EHR interface execution.

Primary area:
datasets and benchmarks
Keywords:
LMM agents, computer use agents, Healthcare AI, Electronic Health Records

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_thTZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PhysicianUIBench, a benchmark of 100 clinical workflow tasks in OpenEpic, an interactive EHR simulation. It evaluates chart grounding, clinical reasoning, and EHR execution across eleven agent configurations, with a highest task success rate of 35%.

Critical strengths
  • The task is valuable.
  • The paper uses real-world medical records and an interactive EHR simulation to evaluate chart grounding, clinical reasoning, and EHR execution.
Critical weaknesses
  • The contributions are somewhat limited. The tasks and clinical objectives are primarily derived from PhysicianBench. The new contributions focus mainly on the interface environment and the adaptation of evaluation criteria.
  • The description of the interface and workflow setup lacks clarity, which could affect the practical validity of the tasks.
Questions for authors
  • What issues does GUI-based evaluation reveal that API-based evaluation does not? Given that the tasks and clinical objectives are largely drawn from PhysicianBench, with new contributions centered on the interface environment and evaluation adaptation, how do the processes and outcomes differ when completing the same tasks through FHIR APIs versus the GUI, with the available information kept as consistent as possible? The paper currently provides no direct comparison of this kind.
  • The reliability of the automated scoring system is not sufficiently established to support such strict task success criteria. The appendix reports a sample of 100 LLM-judged checkpoints, each independently evaluated by three physicians. Agreement between the LLM judge and the physician majority is 82%. Which types of checkpoints are most prone to scoring disagreements? There is also a gap between checkpoint-level agreement and the reliability of the overall task success rate. Since a single failed checkpoint causes the entire task to fail, a disagreement on even one checkpoint could change the final task outcome. Reporting only checkpoint-level agreement is insufficient to determine whether the overall task metrics are robust.
  • The difficulty an agent faces in completing a task may stem from the interface and workflow design. The paper does not sufficiently detail which aspects of real-world clinical workflows were retained or simplified. It also does not report physicians’ task success rates or completion times in the simulator. While the interface is described as Epic-inspired, the evidence that it represents actual clinical practice remains limited. When a task fails, it is important to distinguish agent limitations from difficulties caused by the simulator’s data organization, missing fields, or interaction design.
  • The failure analysis describes where a failure occurred but does not adequately explain why it happened. The paper categorizes failures as grounding, reasoning, and execution, but these categories do not necessarily correspond to distinct, independent root causes. For example, in Figure 5, the model treated an existing abdominal ultrasound as a substitute for the required renal ultrasound and removed an order it had previously added. The resulting missing order was classified as an execution failure, although the underlying issue involved evidence interpretation and clinical judgment.
Appendix C.1 distinguishes EHR execution from operating the interface. The reported 32.7% share of execution failures therefore does not quantify how many failures stem from limitations in interface interaction. The share of failures assigned to each category also depends on the number of checkpoints in that category. If reasoning checkpoints are more common, they may account for a larger share of failures. To support the analysis of the main bottlenecks, the paper should report both the number of checkpoints and the pass rate for each category.
  • Task coverage and the stability of the results need clearer reporting. Were the experiments for each configuration repeated, and what was the variability across runs?
4 comments
AuthorsOpenReview ↗

Response to Reviewer thTZ [Q1-2]

We thank the reviewer for finding the task valuable and for the detailed questions, which we address in turn.

  • Q1. What GUI-based evaluation reveals that API-based evaluation does not

We ran the comparison the reviewer asks for: Claude Fable 5.1 and GPT-6 Sol completed the same 100 tasks through the source benchmark's FHIR harness, with the same chart contents available in both settings, and both settings were graded by the same judge and rubric text (pass rates in G2). The outcomes differ in how tasks fail. Each failed checkpoint receives one label, at the earliest stage that failed. GUI-specific labels are assigned only where the episode's screenshots show the mechanism.

Failed checkpointsFable 5.1 GUIAPIGPT-6 Sol GUIAPI
GUI-specific interaction27NA31NA
Navigation: required view never opened2NA5NA
Exploration: evidence in an opened view never brought on screen17NA16NA
Memory: evidence seen, but out of the agent's context when it wrote the note8NA9NA
Incomplete step: order left unsigned or note never saved0NA1NA
Clinical reasoning7465117119
Reasoning or plan element absent35257144
Wrong clinical conclusion33341621
Declined to commit to a required decision663054
Execution28834061
Evidence omitted or misstated in the note, other than above21383540
Required order set incomplete, or stated in the note but never entered71159
Tool loop: evidence missing from a response, order call with wrong argumentsNA34NA12
Others1041
Failed checkpoints130148192181
Failed tasks64648380
Failed tasks with a GUI-specific failure21NA23NA
Failed tasks whose only failures are GUI-specific6NA3NA

NA: the category does not exist in that setting.

What the interface adds are outcomes that an API-based evaluation cannot observe, because a query hands the agent the requested records as text: a required view never opened, evidence in an opened view never brought on screen, and evidence seen but no longer in context when the note is written. These failures reach a quarter to a third of each model's failed tasks, and on some tasks they are the only failures, so the task fails for a reason an API evaluation cannot record.

In the clearest case, GPT-6 Sol opened the results view in which ALT, AST, and alkaline phosphatase sat below the fold, never scrolled to them, and called the enzymes normal in its note. Through the API, one lab query returned the values, and the note cited them. The tool loop, in turn, has failures that the interface never produces. The two settings thus expose different failures, and the GUI benchmark adds those that exist only where the agent operates the record, measured by what reaches the signed note and orders. The revision adds this comparison to the appendix.

  • Q2. Reliability of the automated scoring, and the gap between checkpoint and task reliability

Checkpoints that ask for specific clinical evidence are the most prone to disagreement: physicians overturned 7 of the 19 sampled, all judge FAILs on notes that reported the required values but were failed for an omission, such as a date or a diagnosis code, or for an extra statement the judge read as unsupported (G3).

For the task-level metric, we carried the panel's error rates through every judged checkpoint of every configuration, assuming independent errors within a task (G3). Pooled over the eleven configurations, task success under physician grading would be 26.7. Reported success is therefore likely conservative, and the ranking is preserved (rank correlation with the judge's ranking: median 0.96).

The strict task criterion is a design choice for high-stakes clinical decision-making: every step of a decision has to be correct, because a correct recommendation that rests on false or hallucinated evidence is not one a clinician can trust.

AuthorsOpenReview ↗

Response to Reviewer thTZ [Q3-5]

  • Q3. The interface and workflow setup

Physicians took part in building the environment, for both interface fidelity and clinical workflow alignment. Three physicians reviewed every surface side by side with a development instance of the reference system, checking that each view presents the chart as the production system does and that each workflow, from chart review through order entry, signing, and the note, proceeds as it does in their practice.

Retained: the chart organization, order entry from a catalog with signing, and note writing. Simplified: Medications and Orders are persistent tabs rather than encounter-gated activities, and there is no medication administration records.

A physician baseline is an important direction for future work. Studies with physicians require IRB approval, which we have just received. We have started the initial feasibility evaluation via a small-scale pilot study involving three independent physicians from different specialties. We will incorporate a comprehensive human study in future work.

To separate agent limitations from simulator causes, we audited every failed checkpoint of Claude Fable 5.1 and GPT-6 Sol. None of the 315 failures traces to the interface, such as an unreachable affordance, and only 3 trace to the chart data, all from one medication status converted incorrectly from the source record.

  • Q4. Why failures happen, the Figure 5 case, and counts and pass rates per category

Why failures happen is analyzed in Q1 and Q3 for every failed checkpoint of Claude Fable 5.1 and GPT-6 Sol, from its trajectory and screenshots: Q1 labels the earliest stage that failed and counts the failures that stem from interface interaction, and Q3 assigns a primary cause. The Figure 5 case is labeled execution because the signed order list lacked the renal ultrasound after the agent had staged it. The interpretive error that led there is recorded in the case study, and the two are not independent. The revision describes the categories as failure loci and states that they are not independent causes.

The number of checkpoints and the pass rate of each category (Section 2.3), pooled over the eleven configurations (per configuration in G1):

Checkpoint categoryCheckpointsPass rate, pooled
Grounding20677.7
Reasoning35369.7
Execution21368.6

Reasoning checkpoints are the most numerous by design: the benchmark supervises the process, so the rationale behind each decision is graded as well as the orders that carry it out. The revision reports these counts in Section 2.3 and the pass rates in the appendix.

  • Q5. Repeated experiments and variability

To measure the variability, we added two runs of GPT-6 Sol on the same environment build and grader (G4). The totals are stable across runs while individual tasks vary. Due to the time limit, we will add entire repeated experiments for all models in the revision.

Reviewer_thTZOpenReview ↗

Follow-up Question

Thank you for the detailed response and additional experiments. I still have some follow-up questions.

(1) Q1 table: why does the API setting produce more failures in "Evidence omitted or misstated in the note" (for Fable 5.1) and "Declined to commit to a required decision" (for GPT-6 Sol) than the GUI setting?

(2) Q1 vs. Q3: GUI failed checkpoints in Q1 sum to 322 (130 and 192), whereas Q3 shows 315 failures were audited.

(3) Q2: assuming independent judge errors across checkpoints within the same task is strong. There may be error propagation from earlier checkpoints to later ones within the same task.

AuthorsOpenReview ↗

Response to Reviewer thTZ's Follow-up Question

Thank you for the follow-up.

(1) Why the API setting shows more failures in two rows

The evidence row records a value that goes missing or wrong when the note is written. Through the API, the records arrive as text and stay in context, so Claude Fable 5.1's value failures almost all happen at that moment. Through the interface, many happen earlier, when a value is never found or drops out of view, and are recorded at that earlier step instead. GPT-6 Sol's API notes, written with the full record in context, take up decisions that its interface notes leave out, often conditionally, so more decisions are hedged through the API and more are omitted through the interface.

(2) 322 and 315

The 315 failures audited in our Q3 come from the grading behind the paper's Table 1, and the 322 in our Q1 from the new matched API and GUI experiment, which graded the same GUI notes again with the same judge. Rerunning the judge changed some of the judged verdicts (95.1 percent agreement), which accounts for the difference.

(3) Dependence between the checkpoints of a task

Each checkpoint is graded in a separate call that sees only the note and its own rubric, without the verdicts on other checkpoints. Errors on checkpoints of the same task can still be correlated through the shared note, but an earlier verdict is not carried into a later one.

Reviewer_s2GfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PhysicianUIBench evaluates computer-use agents on chart review, clinical reasoning, order entry, and documentation through OpenEpic, an Epic-inspired EHR interface. It comprises 100 tasks and 772 checkpoints adapted from physician-authored tasks and real longitudinal records. Evaluation combines deterministic checks with LLM-based grading. Across 11 agent configurations, the highest complete-task success rate is 35%. Three physicians independently assessed a subset of checkpoints to validate automated grading.

Critical strengths
  • Meaningful evaluation setting. Integrating evidence retrieval, clinical decisions, and committed EHR actions captures failures that question-answering or navigation-only benchmarks miss. Checking signed orders and saved documentation is particularly valuable.

  • Useful infrastructure and analysis. Real longitudinal records, a resettable environment, and recorded trajectories support reproducible evaluation. Broad model comparisons and concrete failure cases provide useful directions for improvement.

Critical weaknesses
  • Insufficient validation of scoring reliability. Appendix B.1 reports 82% agreement between the LLM judge and physician majority on 100 checkpoints. Because task success requires every checkpoint to pass, the absence of error-direction analysis and task-level human verification leaves the robustness of success rates and model rankings unclear.

  • Limited direct validation of interface fidelity. Although physicians contributed to iterative development, the paper reports neither clinician task-completion rates in OpenEpic nor quantitative fidelity assessments. This limits the evidence that low agent success reflects difficulties representative of real EHR workflows.

Questions for authors

The following are clarification requests that could affect the assessment:

  1. Were judge–physician disagreements primarily false passes or false failures? Is task-level human verification available to establish whether success rates and model rankings remain stable?

  2. How many physicians completed benchmark tasks in OpenEpic? Were their completion rates and any interface or chart-reconstruction discrepancies recorded?

2 comments
AuthorsOpenReview ↗

Response to Reviewer s2Gf

We thank the reviewer for recognizing the evaluation setting, the value of checking signed orders and saved documentation, and the infrastructure for reproducible evaluation. Please find our point-to-point response below.

  • Q1. Direction of judge disagreements, and task-level human verification

Disagreements are almost all false failures: physicians overturned 1 of 56 judge PASSes and 17 of 44 judge FAILs (G3). This is the safer direction in a clinical setting, where a false pass is the costly error, and it makes reported task success more likely an underestimate than an overstatement. Physician validation targets the LLM judge because it is the only part of task grading that involves judgment: the 210 deterministic checkpoints are verified against application state (the orders signed and the views opened). Agent with similar grading likewise validate only their judged checkpoints (e.g., [1][2]). Carrying the panel's error rates through every judged checkpoint preserves the ranking (median rank correlation with the judge's ranking 0.96, G3).

[1] Bedi, Suhana, et al. "HealthAdminBench: Evaluating Computer-Use Agents on Healthcare Administration Tasks." arXiv preprint arXiv:2604.09937 (2026).

[2] Bandi, Chaithanya, et al. "MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers." arXiv preprint arXiv:2602.00933 (2026).

  • Q2. Physicians completing tasks, and interface or chart discrepancies

A physician baseline is an important direction for future work. Studies with physicians require IRB approval, which we have just received. We have started the initial feasibility evaluation via a small-scale pilot study involving three independent physicians from different specialties. We will incorporate a comprehensive human study in future work.

For discrepancies, three checks exist now. During development, three physicians reviewed each surface against a development instance of the reference system. The chart reconstruction passes an automated audit against the source records on all 100 tasks: every lab row and diagnosis code is present or its exclusion documented, and every parsed value traces to source text. And in an audit of every failed checkpoint of two configurations, none of 315 failures traced to the interface and 3 traced to the chart data, all from one medication status in one chart.

  • Q3. Ethics flag

We will add the following to the ethics statement. The charts derive from the source benchmark's de-identified and perturbed records, which were obtained under institutional review board approval and are distributed under a research data-use agreement. Evaluation runs only inside the simulation. No live system is read or written.

Reviewer_s2GfOpenReview ↗

Response to Authors

Thank you for the detailed rebuttal.

The additional verification and audit results effectively address most of my concerns regarding judge–physician disagreements, scoring reliability, and ethical considerations. However, direct validation of interface fidelity remains limited without comprehensive physician task-completion results.

Overall, the revisions strengthen the paper, though further real-world human evaluation would better support its claims.

Reviewer_HmrnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PHYSICIANUIBENCH, a benchmark of 100 clinical workflows (772 checkpoints) adapted from PhysicianBench and executed within OPENEPIC, an Epic-inspired web EHR simulator for screenshot-based computer-use agents. Evaluating eleven frontier models under a 400-turn budget combines deterministic database state and audit-log checks with GPT-5.5 rubric grading of saved clinical notes, finding a top task completion rate of 35%.

Critical strengths
  • Developing a stateful, Epic-inspired graphical EHR simulator (OPENEPIC) that integrates chart navigation, order entry, and clinical documentation enables realistic closed-loop evaluation of medical computer-use agents without patient-safety risks.
  • Combining deterministic application-state and trace verification with fine-grained failure analysis across grounding, reasoning, and execution surfaces concrete operational failure modes such as premature order withdrawal and ungrounded clinical claims.
  • Benchmarking eleven frontier models from four model families under a unified 400-turn interaction budget provides a clear baseline for long-horizon graphical workflow execution in healthcare.
Critical weaknesses
  • Adapting all 100 clinical workflows, patient records, and grading rubrics from PhysicianBench (Section 2.2) without comparing the same models on the original text/FHIR API setting makes it difficult to separate GUI-specific interaction bottlenecks from underlying clinical reasoning limitations (which represent 44.9% of pooled failures in Figure 3).
  • Reporting only binary all-or-nothing task success rates over 100 workflows from a single rollout without confidence intervals or checkpoint-level pass rates (Table 1) makes it hard to interpret narrow differences between models (such as 19.0% vs. 18.0%, corresponding to a single task) or partial workflow progress.
  • Allowing some model adapters to capture intermediate screenshots within an action batch while others observe only the final screen (Section 2.1, p. 4), together with a fixed five-screenshot history window (Section 3.1), may introduce differences in visual observability across model families during multi-tab chart review.
Questions for authors
  • How do the evaluated models perform on the same 100 workflows in the original PhysicianBench API setting compared to OPENEPIC, and what are the checkpoint-level pass rates broken down by deterministic state/trace checks versus GPT-5.5 judged checks?
  • Which model adapters capture intermediate screenshots during batched actions versus only the final screen (Section 2.1), and how sensitive is agent performance to expanding the five-screenshot context window?
  • Could the authors provide summary statistics on the number of unique patients and clinical specialties across the 100 workflows, and clarify how the 93 checkpoints where GPT-6 Sol fails and Fable 5.1 passes in Appendix D (pp. 13–14) relate to the 52 checkpoints summarized in Table A2?
1 comment
AuthorsOpenReview ↗

Response to Reviewer Hmrn

We thank the reviewer for recognizing the stateful EHR simulator, the combination of deterministic verification with fine-grained failure analysis, and the breadth of the model comparison. Please find our point-to-point response below.

  • Q1. Performance in the original API setting, and checkpoint-level pass rates by check type

We ran this comparison: Claude Fable 5.1 and GPT-6 Sol completed the same 100 workflows through PhysicianBench's FHIR harness, with the same chart contents, and both settings were graded by the same judge and rubric text. To separate GUI-specific interaction from clinical reasoning, each failed checkpoint receives one label, at the earliest stage that failed.

Failed checkpointsFable 5.1 GUIAPIGPT-6 Sol GUIAPI
GUI-specific interaction27NA31NA
Navigation: required view never opened2NA5NA
Exploration: evidence in an opened view never brought on screen17NA16NA
Memory: evidence seen, but out of the agent's context when it wrote the note8NA9NA
Incomplete step: order left unsigned or note never saved0NA1NA
Clinical reasoning7465117119
Reasoning or plan element absent35257144
Wrong clinical conclusion33341621
Declined to commit to a required decision663054
Execution28834061
Evidence omitted or misstated in the note, other than above21383540
Required order set incomplete, or stated in the note but never entered71159
Tool loop: evidence missing from a response, order call with wrong argumentsNA34NA12
Others1041
Failed checkpoints130148192181

NA: the category does not exist in that setting.

The interface adds failures that an API benchmark cannot observe, because a query returns the requested records as text. Through the interface the agent has to find the evidence, bring it on screen, and keep it until the note is written. The most common GUI-specific failure is required evidence that the agent never brought on screen in a view it had opened. What distinguishes our benchmark is that it evaluates clinical reasoning and EHR operation together, on real longitudinal records.

  • Q2. Single rollout, confidence intervals, and narrow differences

We added Wilson 95 percent intervals for task success and checkpoint pass rates by check type, which show partial progress (G1). We also added two further runs of GPT-6 Sol on the same environment build and grader (G4). Task success varies by a few tasks between runs, so a one-task difference such as 19.0 versus 18.0 lies within run-to-run variation, while larger gaps, such as Claude Fable 5.1 over GPT-6 Sol, hold in every run.

  • Q3. Intermediate screenshots, and sensitivity to the context window

The native computer-use adapter returns intermediate screenshots, one after every action. The function-calling adapter returns one screenshot per turn, but these models take 1.00 to 1.05 actions per turn, so both families see the screen after nearly every action. Both adapters keep the screens of the last 5 to 14 turns, and the native adapter keeps up to three screenshots per turn.

The adapters therefore differ in how much visual history they retain, not in what the agent observes. Expanding the window would recover few of the omissions we traced. Of the 84 required values that Claude Fable 5.1 and GPT-6 Sol saw but left out of the note, a window twice as large would leave at least 71 omitted. What matters is whether the agent carries a finding forward in its own text, which the harness keeps in full. The revision corrects Section 3.1 accordingly.

  • Q4. Patients and specialties, and the 93 versus 52 checkpoints

The 100 workflows use 100 distinct patients, one per task, across 21 specialties: Family Medicine 11, Endocrinology 10, Gastroenterology 9, Hematology 9, Infectious Disease 8, Primary Care 7, Internal Medicine 7, Cardiology 5, Psychiatry 5, Gynecology 4, Hepatology 4, Pulmonology 3, Nephrology 3, Geriatric Medicine 3, Otolaryngology 3, Neurology 3, Allergy and Immunology 2, Dermatology 1, Rheumatology 1, Oncology 1, Urology 1. We will add this to the appendix.

Table A2 classifies a subset of the 93 checkpoints that GPT-6 Sol fails and Claude Fable 5.1 passes: the 52 that Claude Opus 5 also passes. Requiring a second passing note under the same judge reduces the chance that a difference reflects how the judge read a single note rather than what GPT-6 Sol wrote. The revision states this selection.

Reviewer_yoXwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces PhysicianUIBench, a benchmark for evaluating computer-use agents on end-to-end clinical workflows within OpenEpic, an interactive EHR simulation inspired by Epic. The benchmark adapts physician-authored tasks from PhysicianBench and comprises 100 tasks with 772 checkpoints. Agents interact exclusively through screenshots, mouse actions, and keyboard inputs to review longitudinal patient records, make clinical decisions, place orders, and document their assessments. Evaluation combines application-state checks, interaction logs, and LLM-based assessment of clinical documentation. Across eleven agent configurations, the highest task success rate is 35%.

Critical strengths
  1. Clinically relevant motivation. Evaluating whether agents can translate clinical decisions into completed EHR actions addresses an important limitation of medical question-answering and API-based evaluations.
  2. Useful evaluation infrastructure. The environment integrates longitudinal chart review, order entry, and documentation within a stateful interface. Separating historical records from episode-specific outputs and recording audit logs supports reproducible evaluation and detailed trajectory analysis.
  3. Execution-grounded assessment. Checking whether orders are actually entered and signed, rather than merely recommended, provides a meaningful measure of workflow completion. The combination of state, trace, and documentation checks captures complementary aspects of performance.
Critical weaknesses
  1. The incremental contribution over closely related benchmarks requires stronger empirical justification. The clinical tasks and objectives are inherited from PhysicianBench, while screenshot-based clinical GUI evaluation has already been explored by MedCUA-Bench. The principal contribution therefore appears to be the integration of longitudinal clinical tasks into an Epic-inspired interface and the adaptation of their evaluation criteria. This is potentially valuable, but the paper should demonstrate what distinctive capabilities or failure modes this setting reveals beyond existing benchmarks.
  2. The experiments do not adequately disentangle clinical reasoning from interface execution. Low end-to-end success may arise from evidence retrieval, clinical decision-making, GUI manipulation, documentation omissions, or their interaction. The current failure taxonomy describes observed deficiencies but does not isolate these factors. Matched API-versus-GUI experiments, evidence-provided conditions, or execution-only tasks with a supplied clinical plan would substantially strengthen the analysis.
  3. The high-fidelity claim lacks quantitative validation. Physician involvement in interface development supports plausibility, but the manuscript does not provide a sufficiently detailed assessment of workflow fidelity or a measured physician completion baseline. Evidence concerning task completion, interaction burden, and preservation of task-critical information would strengthen the claimed connection to real clinical workflows.
  4. Statistical uncertainty is underreported. The main results lack confidence intervals, and the number of independent runs per task is unclear. With 100 tasks, several reported differences correspond to only one or a few successful workflows. Repeated evaluations of representative models and paired task-level comparisons are needed to establish the stability of the rankings.
  5. Failure categories and the completion metric require more careful interpretation. Missing reasoning in a note does not necessarily indicate failed clinical reasoning, and an absent order can reflect either a clinical decision or an interface failure. The label “root cause” is therefore stronger than the evidence supports. Additionally, strict task completion should be complemented by checkpoint-level performance and clinically meaningful error-severity measures.
Questions for authors
  1. How were the original PhysicianBench checkpoints transformed into the 772 checkpoints used here? Please distinguish inherited, modified, and newly introduced criteria.
  2. Can you provide a matched comparison between GUI and API execution on the same clinical tasks, using aligned clinical evaluation criteria?
  3. How would performance change if agents received either the relevant chart evidence or a correct clinical plan? These conditions could help isolate retrieval, reasoning, and execution bottlenecks.
  4. Among the checkpoints reviewed by physicians, what were the judge’s false-pass and false-fail rates? Does physician adjudication of complete tasks preserve the reported success rates and model rankings?
  5. How many physicians validated the reconstructed environment and adapted tasks, and what procedures were used? Have physicians completed representative tasks within OpenEpic under the same grading criteria?
  6. How do action batching, screenshot frequency, and context management differ across adapters? Are the principal conclusions stable under a standardized configuration or alternative action budgets?
  7. Were tasks evaluated more than once per model? Please provide uncertainty estimates and clarify whether the leading models differ reliably under paired evaluation.
4 comments
AuthorsOpenReview ↗

Response to Reviewer yoXw [Q1-2]

We thank the reviewer for recognizing the clinically relevant motivation, the evaluation infrastructure, and the execution-grounded assessment. Please find our point-to-point response below.

  • Q1. Contribution beyond PhysicianBench and MedCUA-Bench

Section 4 and Table 2 compare our benchmark with existing ones. PhysicianBench provides the agent with FHIR API tools only, so it never operates a GUI. Benchmarks that run in clinical interfaces, such as MedCUA-Bench, use synthetic records and rule-based checks, which do not grade the clinical assessment a physician would document. Our benchmark places real longitudinal records in the EHR interface physicians use, where the agent must find evidence across the views of a long record and its decisions take effect through signed orders and notes. We rebuilt each source task for the interface, reconstructing its FHIR record as the linked views of a longitudinal chart, rewriting its instruction as a clinical handoff, and replacing its checks on FHIR reads and writes with checks on the views opened, the orders signed and note saved.

To show what the interface reveals, we ran the matched comparison the reviewer suggests in G2. The GUI-specific labels follow failure analyses of GUI agents [1][2] and are assigned only where the episode's screenshots show the mechanism. Clinical reasoning and execution follow the categories of Section 3.3.

Failed checkpointsFable 5.1 GUIAPIGPT-6 Sol GUIAPI
GUI-specific interaction27NA31NA
Navigation: required view never opened2NA5NA
Exploration: evidence in an opened view never brought on screen17NA16NA
Memory: evidence seen, but out of the agent's context when it wrote the note8NA9NA
Incomplete step: order left unsigned or note never saved0NA1NA
Clinical reasoning7465117119
Reasoning or plan element absent35257144
Wrong clinical conclusion33341621
Declined to commit to a required decision663054
Execution28834061
Evidence omitted or misstated in the note21383540
Required order set incomplete, or stated in the note but never entered71159
Tool loop: evidence missing from a response, order call with wrong argumentsNA34NA12
Others1041
Failed checkpoints130148192181

NA: the category does not exist in that setting.

The interface adds failures that an API benchmark cannot observe, because a query returns the requested records as text. Through the interface the agent has to find the evidence, bring it on screen, and keep it until the note is written. The most common GUI-specific failure is required evidence that the agent never brought on screen in a view it had opened. What distinguishes our benchmark is that it evaluates clinical reasoning and EHR operation together, on real longitudinal records. We will add this comparison to the appendix and refer to it in the contribution statement.

[1] Xue, Tianci, et al. "An Illusion of Progress? Assessing the Current State of Web Agents." arXiv preprint arXiv:2504.01382 (2025).

[2] Boisvert, Léo, et al. "WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks." NeurIPS 2024 Datasets and Benchmarks Track.

  • Q2. Separating clinical reasoning from interface execution

Our checkpoint design, rubrics, and failure analysis are built to separate the two. Deterministic checkpoints grade operation of the interface (the views opened and the orders signed), judged checkpoints grade the saved note (the clinical values it reports, its clinical reasoning, and its documentation) against rubrics, and the failure analysis assigns each failed checkpoint to grounding, reasoning, or execution from its type and the judge's explanation (Section 3.3).

To examine this further, we followed the reviewer's suggestion and ran the matched API-versus-GUI comparison (Q1). For the same model, the interface adds failures of its own, mostly in evidence retrieval. They occur in a quarter to a third of each model's failed tasks, typically as evidence missing or wrong in the signed note. About a fifth of failed checkpoints involve two or more of the factors the reviewer lists, which is why the analysis assigns each to the earliest stage that failed.

AuthorsOpenReview ↗

Response to Reviewer yoXw [Q3-8]

  • Q3. Fidelity and a physician baseline

We adopt following steps to ensure fidelity. First, three physicians reviewed every surface of the interface side by side with a development instance of the reference system. Second, task information is preserved by construction: every lab row and diagnosis code of the source record is present in the chart, every parsed value traces to source text. Third, an audit of every failed checkpoint of two configurations traced none of 315 failures to the interface and 3 to the chart data, all from one medication status in one chart.

A physician baseline is an important direction for future work. Studies with physicians require IRB approval, which we have just received. We have started the initial feasibility evaluation via a small-scale pilot study involving three independent physicians from different specialties. We will conduct a comprehensive human study in future work.

  • Q4. Statistical uncertainty and repeated runs

We add Wilson 95 percent intervals for every configuration (G1) and two further runs of GPT-6 Sol on the same environment build and grader (G4). We will add the intervals and the all model runs with their mean and range in the revision.

  • Q5. Interpretation of the failure categories and the completion metric

We add three analyses that interpret the failure categories and complement task completion. First, checkpoint pass rates by evaluation category and check type for every configuration (G1). Second, every failed checkpoint of Claude Fable 5.1 and GPT-6 Sol in Table 1 that traces to the agent is annotated with its clinical consequence, which separates an element missing from the note from a wrong or withheld decision:

Clinical consequence of the failed checkpointClaude Fable 5.1GPT-6 Sol
Wrong, hedged, or withheld conclusion, or a statement the chart contradicts6769
Wrong or missing order75
Required element missing from the note4687
Navigation checkpoint only03
Total120164

Most failed tasks contain a wrong or withheld conclusion or a wrong or missing order. Third, the analysis separates the two readings of an absent order that the reviewer raises.

  • Q6. How the 772 checkpoints relate to the source checkpoints

Of the source benchmark's 670 checkpoints, 102 combine two conditions, mostly a trajectory condition with a judged one, and each of these became two checkpoints, which gives 772. The 210 deterministic checkpoints are retargeted: 101 data-retrieval conditions that required FHIR reads now require opening the corresponding chart views, and 109 order conditions that required FHIR writes now require a signed order. The 562 judged checkpoints keep the source rubric text.

  • Q7. Judge false-pass and false-fail rates, and adjudication of complete tasks

Of 56 judge PASSes, physicians overturned 1, and of 44 judge FAILs, they overturned 17. Reported task success is therefore likely conservative. Physician validation targets the LLM judge because it is the only part of task grading that involves judgment: the 210 deterministic checkpoints are verified against application state (the orders signed and the views opened). Agent benchmarks with similar grading likewise validate only their judged checkpoints [3]. The ranking is preserved (median rank correlation with the judge's ranking 0.96, G3).

[3] Bandi, Chaithanya, et al. "MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers." arXiv preprint arXiv:2602.00933 (2026).

  • Q8. Differences between adapters

The adapters differ in action batching, screenshot frequency and retained context:

AdapterActions per turnScreenshot returnedScreenshots in context
Native computer use3.4 to 4.0after every actionup to 3 per turn, last 5 to 14 turns
Function calling1.00 to 1.05after each turn1 per turn, last 5 to 14 turns

The principal conclusions do not depend on the action budget: the median interaction turns are far below the upper bound (Figure 2(a)). We will add this table to the appendix.

Reviewer_yoXwOpenReview ↗

Remaining Concerns After the Rebuttal

Thank you for the detailed rebuttal and the additional experiments. However, the numbering in the rebuttal (Q1–Q8) does not align with my original questions (Q1–Q7), making it difficult to track which concerns have been addressed. After reviewing the responses, I still have four concerns:

  1. The failure analysis relies on post-hoc categorization, and around 20% of failed checkpoints involve multiple factors. How is the earliest failed stage reliably determined? The API–GUI comparison does not fully isolate reasoning, retrieval, and execution. The evidence-provided or plan-provided experiments suggested in my review remain unaddressed.

  2. How many independent runs were conducted for each model? The response only mentions two additional runs for GPT-6 Sol. Are the differences between leading models statistically reliable under paired task-level comparisons?

  3. Physicians overturned 17 of 44 judge FAIL decisions. How does this affect the reported task-level success rates under the strict completion metric? A rank correlation of 0.96 does not establish the reliability of absolute success rates.

  4. The response lists differences in action batching, screenshot frequency, and context retention, but does not show whether the results are robust to these differences. Being below the action budget does not rule out adapter effects.

These concerns remain important to my assessment.

AuthorsOpenReview ↗

Response to Reviewer yoXw's remaining concerns

Thank you for the follow-up. Our rebuttal answered your weaknesses as Q1 to Q5 and the questions as Q6 to Q8. Please find our point-to-point response to remaining concerns below.

1. Failure attribution, and evidence- or plan-provided conditions

In a checkpoint with several failed elements, each element is placed at a stage from the record. The audit log shows whether its view was opened, the screenshots whether the value came on screen and stayed in context, and the note and signed orders what was decided and done. The stages follow the order the work requires (finding, reading, deciding, ordering, documenting), so the earliest failed stage is the one the others depend on, for example a wrong decision made without the value it needed.

To separate execution from the clinical decisions, we ran GPT-6 Sol on 50 random tasks with a clinical plan in the instruction, written from the rubric without chart values or view names. It signed all 53 required orders (47 to 49 in standard runs), while its value checkpoints passed 37 of 51 times (32 to 36 in standard runs), so supplying the decisions removed the order failures but changed value retrieval little. We did not run an evidence-provided condition, since it would hand the agent the content the benchmark asks it to find.

2. Runs and paired comparisons

Table 1 reports one full run per configuration. Within the discussion period we added two runs of GPT-6 Sol, and the revision will report repeated runs for every configuration. Exact McNemar tests on paired task outcomes, which need only one run, support three tiers, with Claude Fable 5.1, Claude Opus 5, Claude Fable 5 and Muse Spark 1.3 in the first, GPT-6 Sol, GPT-6 Astra, GPT-5.6 Sol and DeepSeek V4.1 Flash in the second, and Qwen 3.8 Max, GLM-5.3 Flash and Kimi K3 in the third.

3. Effect of physician overturns on task success

The absolute rates in Table 1 are therefore relatively conservative. The G3 simulation puts the bias at about seven tasks per configuration, raising the highest rate from 35 to about 41 percent. We will include this clarification in the revision.

4. Adapter effects

A more direct check than the action budget is the source benchmark's FHIR harness, where both models call the same tools without any GUI adapter, and there Claude Fable 5.1 still beats GPT-6 Sol 22 to 6 (p = 0.004). Given the time limit of the discussion period, we will add each of the ablation experiments in the revision.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank all reviewers for their careful reading and constructive suggestions. Below we report the new experiments suggested by most reviewers.

G1. Checkpoint-level results. Task success with Wilson 95 percent intervals, and checkpoint pass rates in the three evaluation categories. Each check is deterministic (D) or judged (J).

ConfigurationTask success (95% CI)Grounding: views opened (D, n=101)Grounding: clinical evidence cited (J, n=105)Reasoning (J, n=353)Execution: orders signed (D, n=109)Execution: documentation (J, n=104)
Claude Fable 5.135 (26 to 45)99.078.181.391.766.3
Claude Opus 529 (21 to 39)96.071.480.290.857.7
Claude Fable 529 (21 to 39)96.077.176.287.254.8
Muse Spark 1.326 (18 to 35)97.078.174.590.856.7
GPT-6 Sol19 (13 to 28)97.074.371.793.657.7
GPT-6 Astra18 (12 to 27)99.066.772.889.954.8
GPT-5.6 Sol18 (12 to 27)96.066.773.787.253.8
DeepSeek V4.1 Flash15 (9 to 23)98.053.366.393.640.4
Qwen 3.8 Max9 (5 to 16)79.245.756.978.935.6
GLM-5.3 Flash8 (4 to 15)86.141.060.379.836.5
Kimi K38 (4 to 15)82.237.153.373.427.9

G2. Matched API and GUI runs. Claude Fable 5.1 and GPT-6 Sol completed the same 100 tasks through the source benchmark's FHIR tool harness. Both settings were graded in one session by the same judge and rubric text on the checkpoints they share.

Pass rate, percentClaude Fable 5.1 GUIAPIGPT-6 Sol GUIAPI
Grounding: clinical evidence cited (J, n=105)81.078.172.485.7
Reasoning (J, n=353)81.679.069.770.0
Execution: documentation (J, n=104)66.367.355.858.7
Execution: orders signed (D, n=107)91.684.193.584.1
Task success, of 10036361720

G3. Judge reliability and its effect on task success. Three physicians graded 100 judged checkpoints, blinded to the judge and to one another (Fleiss kappa 0.746).

Judged checkpoint categorynFalse passesFalse failures
Grounding1907
Reasoning6206
Execution1914
All100117

False failures are most frequent on grounding checkpoints, mainly on notes that reported the required values but were failed for an omission, such as a date or a diagnosis code. To estimate the effect on task success, we simulated physician verdicts at the panel's error rates on every judged checkpoint of the eleven configurations, assuming independent errors within a task. Pooled over the configurations, task success would be 26.7, so reported success is likely conservative but remains low. The ranking is preserved (rank correlation with the judge's ranking: median 0.96, 2.5th percentile 0.82).

G4. Run-to-run variation. We add two runs of GPT-6 Sol on the same environment build and grader.

GPT-6 Sol runTask successCheckpoint pass rate
Table 11976.6
Repeat 12175.3
Repeat 21975.5

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.