- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces PhysicianUIBench, a benchmark of 100 clinical workflow tasks in OpenEpic, an interactive EHR simulation. It evaluates chart grounding, clinical reasoning, and EHR execution across eleven agent configurations, with a highest task success rate of 35%.
Critical strengths
- The task is valuable.
- The paper uses real-world medical records and an interactive EHR simulation to evaluate chart grounding, clinical reasoning, and EHR execution.
Critical weaknesses
- The contributions are somewhat limited. The tasks and clinical objectives are primarily derived from PhysicianBench. The new contributions focus mainly on the interface environment and the adaptation of evaluation criteria.
- The description of the interface and workflow setup lacks clarity, which could affect the practical validity of the tasks.
Questions for authors
- What issues does GUI-based evaluation reveal that API-based evaluation does not? Given that the tasks and clinical objectives are largely drawn from PhysicianBench, with new contributions centered on the interface environment and evaluation adaptation, how do the processes and outcomes differ when completing the same tasks through FHIR APIs versus the GUI, with the available information kept as consistent as possible? The paper currently provides no direct comparison of this kind.
- The reliability of the automated scoring system is not sufficiently established to support such strict task success criteria. The appendix reports a sample of 100 LLM-judged checkpoints, each independently evaluated by three physicians. Agreement between the LLM judge and the physician majority is 82%. Which types of checkpoints are most prone to scoring disagreements? There is also a gap between checkpoint-level agreement and the reliability of the overall task success rate. Since a single failed checkpoint causes the entire task to fail, a disagreement on even one checkpoint could change the final task outcome. Reporting only checkpoint-level agreement is insufficient to determine whether the overall task metrics are robust.
- The difficulty an agent faces in completing a task may stem from the interface and workflow design. The paper does not sufficiently detail which aspects of real-world clinical workflows were retained or simplified. It also does not report physicians’ task success rates or completion times in the simulator. While the interface is described as Epic-inspired, the evidence that it represents actual clinical practice remains limited. When a task fails, it is important to distinguish agent limitations from difficulties caused by the simulator’s data organization, missing fields, or interaction design.
- The failure analysis describes where a failure occurred but does not adequately explain why it happened. The paper categorizes failures as grounding, reasoning, and execution, but these categories do not necessarily correspond to distinct, independent root causes. For example, in Figure 5, the model treated an existing abdominal ultrasound as a substitute for the required renal ultrasound and removed an order it had previously added. The resulting missing order was classified as an execution failure, although the underlying issue involved evidence interpretation and clinical judgment. Appendix C.1 distinguishes EHR execution from operating the interface. The reported 32.7% share of execution failures therefore does not quantify how many failures stem from limitations in interface interaction. The share of failures assigned to each category also depends on the number of checkpoints in that category. If reasoning checkpoints are more common, they may account for a larger share of failures. To support the analysis of the main bottlenecks, the paper should report both the number of checkpoints and the pass rate for each category.
- Task coverage and the stability of the results need clearer reporting. Were the experiments for each configuration repeated, and what was the variability across runs?
4 comments
Response to Reviewer thTZ [Q3-5]
- Q3. The interface and workflow setup
Physicians took part in building the environment, for both interface fidelity and clinical workflow alignment. Three physicians reviewed every surface side by side with a development instance of the reference system, checking that each view presents the chart as the production system does and that each workflow, from chart review through order entry, signing, and the note, proceeds as it does in their practice.
Retained: the chart organization, order entry from a catalog with signing, and note writing. Simplified: Medications and Orders are persistent tabs rather than encounter-gated activities, and there is no medication administration records.
A physician baseline is an important direction for future work. Studies with physicians require IRB approval, which we have just received. We have started the initial feasibility evaluation via a small-scale pilot study involving three independent physicians from different specialties. We will incorporate a comprehensive human study in future work.
To separate agent limitations from simulator causes, we audited every failed checkpoint of Claude Fable 5.1 and GPT-6 Sol. None of the 315 failures traces to the interface, such as an unreachable affordance, and only 3 trace to the chart data, all from one medication status converted incorrectly from the source record.
- Q4. Why failures happen, the Figure 5 case, and counts and pass rates per category
Why failures happen is analyzed in Q1 and Q3 for every failed checkpoint of Claude Fable 5.1 and GPT-6 Sol, from its trajectory and screenshots: Q1 labels the earliest stage that failed and counts the failures that stem from interface interaction, and Q3 assigns a primary cause. The Figure 5 case is labeled execution because the signed order list lacked the renal ultrasound after the agent had staged it. The interpretive error that led there is recorded in the case study, and the two are not independent. The revision describes the categories as failure loci and states that they are not independent causes.
The number of checkpoints and the pass rate of each category (Section 2.3), pooled over the eleven configurations (per configuration in G1):
| Checkpoint category | Checkpoints | Pass rate, pooled |
|---|---|---|
| Grounding | 206 | 77.7 |
| Reasoning | 353 | 69.7 |
| Execution | 213 | 68.6 |
Reasoning checkpoints are the most numerous by design: the benchmark supervises the process, so the rationale behind each decision is graded as well as the orders that carry it out. The revision reports these counts in Section 2.3 and the pass rates in the appendix.
- Q5. Repeated experiments and variability
To measure the variability, we added two runs of GPT-6 Sol on the same environment build and grader (G4). The totals are stable across runs while individual tasks vary. Due to the time limit, we will add entire repeated experiments for all models in the revision.
Follow-up Question
Thank you for the detailed response and additional experiments. I still have some follow-up questions.
(1) Q1 table: why does the API setting produce more failures in "Evidence omitted or misstated in the note" (for Fable 5.1) and "Declined to commit to a required decision" (for GPT-6 Sol) than the GUI setting?
(2) Q1 vs. Q3: GUI failed checkpoints in Q1 sum to 322 (130 and 192), whereas Q3 shows 315 failures were audited.
(3) Q2: assuming independent judge errors across checkpoints within the same task is strong. There may be error propagation from earlier checkpoints to later ones within the same task.
Response to Reviewer thTZ's Follow-up Question
Thank you for the follow-up.
(1) Why the API setting shows more failures in two rows
The evidence row records a value that goes missing or wrong when the note is written. Through the API, the records arrive as text and stay in context, so Claude Fable 5.1's value failures almost all happen at that moment. Through the interface, many happen earlier, when a value is never found or drops out of view, and are recorded at that earlier step instead. GPT-6 Sol's API notes, written with the full record in context, take up decisions that its interface notes leave out, often conditionally, so more decisions are hedged through the API and more are omitted through the interface.
(2) 322 and 315
The 315 failures audited in our Q3 come from the grading behind the paper's Table 1, and the 322 in our Q1 from the new matched API and GUI experiment, which graded the same GUI notes again with the same judge. Rerunning the judge changed some of the judged verdicts (95.1 percent agreement), which accounts for the difference.
(3) Dependence between the checkpoints of a task
Each checkpoint is graded in a separate call that sees only the note and its own rubric, without the verdicts on other checkpoints. Errors on checkpoints of the same task can still be correlated through the shared note, but an earlier verdict is not carried into a later one.
Response to Reviewer thTZ [Q1-2]
We thank the reviewer for finding the task valuable and for the detailed questions, which we address in turn.
We ran the comparison the reviewer asks for: Claude Fable 5.1 and GPT-6 Sol completed the same 100 tasks through the source benchmark's FHIR harness, with the same chart contents available in both settings, and both settings were graded by the same judge and rubric text (pass rates in G2). The outcomes differ in how tasks fail. Each failed checkpoint receives one label, at the earliest stage that failed. GUI-specific labels are assigned only where the episode's screenshots show the mechanism.
NA: the category does not exist in that setting.
What the interface adds are outcomes that an API-based evaluation cannot observe, because a query hands the agent the requested records as text: a required view never opened, evidence in an opened view never brought on screen, and evidence seen but no longer in context when the note is written. These failures reach a quarter to a third of each model's failed tasks, and on some tasks they are the only failures, so the task fails for a reason an API evaluation cannot record.
In the clearest case, GPT-6 Sol opened the results view in which ALT, AST, and alkaline phosphatase sat below the fold, never scrolled to them, and called the enzymes normal in its note. Through the API, one lab query returned the values, and the note cited them. The tool loop, in turn, has failures that the interface never produces. The two settings thus expose different failures, and the GUI benchmark adds those that exist only where the agent operates the record, measured by what reaches the signed note and orders. The revision adds this comparison to the appendix.
Checkpoints that ask for specific clinical evidence are the most prone to disagreement: physicians overturned 7 of the 19 sampled, all judge FAILs on notes that reported the required values but were failed for an omission, such as a date or a diagnosis code, or for an extra statement the judge read as unsupported (G3).
For the task-level metric, we carried the panel's error rates through every judged checkpoint of every configuration, assuming independent errors within a task (G3). Pooled over the eleven configurations, task success under physician grading would be 26.7. Reported success is therefore likely conservative, and the ranking is preserved (rank correlation with the judge's ranking: median 0.96).
The strict task criterion is a design choice for high-stakes clinical decision-making: every step of a decision has to be correct, because a correct recommendation that rests on false or hallucinated evidence is not one a clinician can trust.