acceptodds
Withdrawn from ICLR 2027

SHIFT: Benchmarking Clinical Computer-Use Agents over Full Patient Trajectories

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Computer-use agents operate software through its graphical interface and could work inside the electronic health record (EHR) systems clinicians use. Yet most medical AI benchmarks still rely on pre-curated text, images, or structured data, and evaluate isolated tasks rather than a continuous course of care. It therefore remains unclear whether agents can navigate clinical interfaces, maintain a patient's evolving state, and carry care from initial assessment through follow-up. Here we introduce SHIFT (Stateful Hospital Interaction for Full Trajectories), a benchmark for clinical computer-use agents over complete patient trajectories. SHIFT builds on a real EHR (OpenMRS) and image viewer (OHIF), connecting interviews, charting, orders, results, image interpretation, diagnosis, treatment, and follow-up to a shared patient state. We score agents along seven clinical capabilities, using case-specific dependency graphs to check the evidence supporting each scoring item while accepting clinically valid alternatives. Across nine models, GPT-6-Astra achieves the highest mean score of 53.82 out of 100. Five models score below 30. Common errors include incorrect patient information, misinterpreted test results, and neglected earlier findings. Scores are lower in completed six-patient sessions. By exposing where agents lose track of patient state and evidence, SHIFT quantifies the gap that must be closed before computer-use agents can safely operate real clinical software.

Primary area:
datasets and benchmarks
Keywords:
Benchmarking and Evaluation, Computer Use, AI Agents, Medical Artificial Intelligence

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CPJyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces SHIFT, a benchmark in which computer-use agents manage oncology patients through a real EHR (OpenMRS 3) and an image viewer (OHIF/Orthanc). Agents see only browser screenshots and act through mouse and keyboard input. Each episode covers the full course of care: interviewing the patient, charting, ordering tests, reviewing results, reading images, diagnosis, staging, treatment, and follow-up. The benchmark contains 600 cases, 100 for each of six pathways (BC, CRLM, GBM, HCC, HNSCC, and NSCLC).

Scoring is built around seven fixed capabilities whose points add up to 100. Each case has its own rubric items, scored 0/0.25/0.5/0.75/1, and a "dependency graph" (DAG) records which evidence each judgment needs. Reference trajectories were produced by three clinicians acting as author, checker, and adjudicator.

Critical strengths
  1. Timely and important target. Evaluating agents inside real clinical software, rather than through text or curated APIs, fills a real gap. The design principle that "writing that an order was submitted does not prove it happened" is a good one.
  2. Full-trajectory evaluation unit. Following one patient from first contact to follow-up tests whether agents carry evidence forward, which single-step benchmarks cannot do.
Critical weaknesses

W1. The paper never states who or what does the scoring, and the grader is not validated.

  • Section 4.2 states that each scoring decision "includes an explanation and references to the supporting records", but it never says whether scoring is done by clinicians, an LLM judge, rule-based checks, or a combination of these.
  • The scale makes full human grading very unlikely. There are 600 cases × 9 models × 3 runs = 16,200 single-patient episodes, plus the multi-patient sessions, and each of them has many items scored with graded partial credit. If an LLM judge is used, the paper must name the judge model, provide its prompts, and report its agreement with clinicians, e.g., Cohen's or weighted κ on the 0–1 item scores.
  • Three of the top four models are GPT-family models (GPT-6-Astra, GPT-5.6 Terra, GPT-5.6 Luna). If the judge is also a GPT model, self-preference bias is a real risk that must be ruled out.
  • The authors acknowledge that "scoring consistency requires further validation" (Sec. 6). For a benchmark paper, however, grader validity is a core contribution, not future work.

W2. The paper does not describe where the cases come from, how they were built, or how the simulated patient works.

  • The case IDs "ISPY1 1206/1064" and "LUNG1-272" match public TCIA collections: the I-SPY1 (ACRIN 6657) breast MRI collection and NSCLC-Radiomics "LUNG1" (Aerts et al., Nat. Commun. 2014). These collections contain images and a limited set of clinical variables (e.g., pCR and survival), but no interview transcripts, full charts, or laboratory panels.
  • The paper does not say how the missing content (history, interview answers, test results, reports, and follow-up) was created, whether by clinicians, by an LLM, or from templates. This matters for realism, bias, and possible leakage.
  • The paper also does not describe how the "patient" answers interview questions (scripted answers or an LLM simulator) or what happens when the agent asks something off-script.
  • None of the source datasets is cited, and neither their licenses (some TCIA collections are restricted to non-commercial use) nor the release plan for SHIFT is stated.

W3. The reference standard and its reliability are not described well enough.

  • Three clinicians in fixed roles produced the references for 600 cases across six oncology pathways in five subspecialties (breast, hepatobiliary, neuro-oncology, head-and-neck, and thoracic). The paper reports neither their specialties and experience nor how the workload was divided (did one clinician complete all 600 trajectories in the GUI?).
  • No agreement statistics are given, e.g., how often the checker disagreed with the author or how many items the adjudicator changed. The paper also does not say how the "acceptable alternatives" were listed or how complete they are.
  • No human baseline. No clinician or resident completes even a subset of the cases through the same interface under the same rubric, so a score of 53.82/100 cannot be interpreted. A strict or ambiguous rubric could make even experts score low, especially with partial credit in steps of 0.25.

W4. Details of the agent setup and benchmark statistics are missing, which limits reproducibility.

  • The paper does not describe the agent scaffold (each vendor's native computer-use API or a single shared harness), the system prompt, the action space, the screenshot resolution, how history is kept over 200–470 tool calls, the step/time/token limits, the "conditions for ending the task", the decoding settings, the exact model versions and API dates, or the cost per case.
  • Case and rubric statistics are missing: the number of items per case, the distribution of item weights, the DAG size (nodes and edges), the number of stages, the number of images per case, and the tests available.
  • No full example rubric or DAG is shown (Fig. 2 is "illustrative").
  • It is unclear whether the environment, cases, and rubrics will be released, and there is no reproducibility or code/data availability statement.
  • All nine models are accessed through closed APIs, so the results will drift as the APIs change. No open-weight model is tested, whether a general-purpose VLM or a GUI-specialized agent model.

W5. Safety results are promised but not reported.

  • Section 4.4 states that "safety events, including actions on the wrong patient, dangerous clinical decisions, and missed critical results, are recorded separately and reported alongside the clinical scores".
  • However, only wrong-patient action rates in multi-patient sessions are reported. Dangerous decisions and missed critical results are never shown, even though the abstract claims that SHIFT "quantifies the gap that must be closed before computer-use agents can safely operate real clinical software".
  • The NSCLC example, in which an agent schedules a biopsy and staging for a patient whose death has been recorded, shows that such events do occur. A per-model safety table is needed.

W6. The contribution beyond existing benchmarks is mainly the combination of existing elements, and the comparison table is hard to interpret.

  • Each ingredient already exists: GUI-based clinical workflows (CareFlow/CarePilot), EHR agent environments (MedAgentBench, PhysicianBench), staged clinical pathways (MedChain, CP-Env), interactive diagnosis with test ordering (AgentClinic; also SDBench/MAI-DxO, Nori et al., 2025, not cited), and rubric-based process scoring (PaperBench; HealthBench, Arora et al., 2025, not cited). The novelty is in combining them. This is useful, but the paper should present its contribution more modestly.
  • Table 1 marks most cells for other benchmarks as "N/A" ("outside the task setting") rather than ✗. This blurs the line between "not supported" and "not applicable" and makes the comparison look more favorable than it is. Some entries are also debatable. For example, MedAgentBench runs on a FHIR EHR with persistent patient records, yet it is marked N/A for "persistent patient state".

W7. Much essential content is missing, and the paper has no appendix.

  • The paper is short (about nine pages) and has no appendix, so the rubric design, case construction, patient simulator, agent harness, and grader are not described anywhere.
  • There is no ethics statement (covering the use of patient-derived data, licensing, and privacy) and no analysis of the grader's limitations.
  • The introduction and abstract are clear and well motivated. The conclusion, however, restates the results without discussing what the benchmark can and cannot measure, e.g., GUI skill vs. reasoning and grader validity.
Questions for authors

Please see Weaknesses above.

2 comments
AuthorsOpenReview ↗

Response to CPJy: Grader validity, clinical references, safety, and reproducibility

Thank you for the thorough review. M###/A### denote revised main-paper and appendix lines.

W1. Grader validity (M213–239; App. B, A217–468)

Audit-log checks verify actions, while an LLM judge (gpt-5.6-terra) scores clinical content item by item with model identity hidden (prompt: App. B.2, A257). Two board-certified oncologists independent of the rubric panel scored 144 episodes, blinded to judge scores. Judge–clinician weighted κ is 0.84/0.82, versus 0.88 between clinicians (Table B2, A293).

Because the judge is itself one of the evaluated GPT models, we tested self-preference directly (Tables B5–B6, A347–380):

  • Against clinicians, its offset is +0.29 points larger on GPT-family than on non-GPT episodes (95% CI [−0.48, +1.06], n = 48), and +0.24 larger on its own model's episodes ([−0.98, +1.46], n = 16).
  • DeepSeek-V4-Pro, outside all evaluated families, re-scored every single-patient GUI episode. Its offsets from the primary judge are similar for GPT (−2.96) and non-GPT (−2.80) models. The overall ranking is unchanged (τ = 1.000; M333–335).

Because the audit intervals are too wide to rule out a small bias, we added the cross-family re-scoring.

W2. Sources, construction, simulated patient (M153–156, M174–179; App. D, A1220–1377)

All six TCIA collections are cited with DOIs, licenses, and selection rules (Table D1, A1260). NSCLC-Radiomics is CC BY-NC 3.0. Raw data are not redistributed; users re-acquire them from TCIA under each license. Clinicians, not LLMs, wrote the missing reports and chart notes from stage-locked templates, and a second clinician checked every field.

Patient replies are scripted: matched questions return clinician-written fact blocks verbatim, while others get a neutral cannot-answer reply.

W3. Reference standard and human baseline (M180–187; App. D, A1416–1485)

Thirteen clinicians across five oncology subspecialties and radiology fill three roles per case (Table D2, A1444). No single clinician built all 600 references: each author wrote 50–67 cases, and a same-subspecialty clinician cross-checked every case. The checker agreed on 89.3% of items (10.7% flagged), and the adjudicator modified 6.2%. This is sequential review agreement, not independent inter-rater reliability. Author and checker listed acceptable alternatives independently, and the adjudicator merged them.

Three oncologists outside the panel each completed the same 60 cases in the same GUI, blind to the rubrics. They average 91.40, versus 54.14 for GPT-6-Astra on those cases (M335–337; Tables H1–H2, A2484). An audit attributes 0.62 of their 8.60 lost points to valid alternatives missing from the rubric.

W4. Setup, case statistics, reproducibility

All models share one OSWorld-adapted harness with pinned model snapshots (M312–318; Table A3, A114). App. A (A003–216) gives the prompt, action space, screenshot size, history handling, limits, decoding settings, and costs. Cases have 22–27 items and 17–26 prerequisite edges (Table D3, A1379), and App. C.6 (A864–1215) gives a full 24-item rubric with scores.

Two open-weight models, UI-TARS-72B and Qwen2.5-VL-72B, score 20.61 and 18.01 on one run each (Table 3). Our code repository is at https://anonymous.4open.science/r/SHIFT-C84D/ (M542–546; App. G, A2395). It includes the scoring code with unit tests, worked scoring examples, and agent-run recordings. Researchers can apply for the full environment, all case packages and rubrics, and the grading code under the data use agreement in the repository (Table G1, A2400).

W5. Safety (M419–423; App. E.1, A1489–1635)

Table E2 (A1538) reports per-model counts for each safety event class. GPT-6-Astra has 3.4 dangerous-management events per 100 episodes, versus 16.4–73.4 for the other eight. Missed-material-evidence events number 674–1,550 per model across 1,800 episodes. This class counts material information that a note omits or contradicts in records that were released and reviewed, rather than every result left unopened (A1633–1635). Clinician–judge agreement on safety events is κ = 0.86 pooled and 0.79–0.88 by class; dangerous management has κ = 0.88 (Table L2, A3169). The abstract now states what SHIFT measures (M027–029).

W6. Novelty and Table 1

The individual components have precedents; our contribution is integrating them into one continuous episode (M069–070). Table 1 now uses × instead of N/A and marks MedAgentBench as having persistent state (M108–125). SDBench/MAI-DxO and HealthBench are also cited (M080, M093).

W7. Appendix, ethics, scope

We added an Ethics Statement (M527–535), limitations covering grader validity (M500–515), and a conclusion stating what SHIFT does and does not measure (M519–525). In a direct-access condition with chart/report text and DICOM tools (M415–419; Table H3, A2592), overall scores rise by 8.76–15.82 points, yet the best imaging score is 9.04/17. Because inputs and tools both change, this measures interface sensitivity.

Dear Reviewer CPJy,

Thank you again for the time and care you put into reviewing our paper. We appreciate your recognition of the importance of evaluating agents within real clinical software, verifying actions against system records rather than agent claims, and using full trajectories as the unit of evaluation.

Our rebuttal covers grader validation against blinded oncologists (including tests for self-preference bias); case sources, construction, and the scripted patient; the reference panel and a clinician baseline; and the agent harness, case statistics, open-weight models, and code release. We also present the per-model safety table, the revised Table 1 and novelty positioning, and the new appendix, ethics statement, and limitations.

Could you let us know whether our rebuttal and revisions address your concerns, and if any questions remain that we can help clarify?

Best regards, Paper 24504 Authors

Reviewer_7ganOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SHIFT, a benchmark evaluating clinical computer-use agents over complete oncology patient trajectories. Agents interact with OpenMRS and OHIF through screenshots and GUI actions, performing tasks from initial assessment through diagnosis, treatment, and follow-up. The benchmark contains 600 cases across six oncology pathways and uses case-specific evidence-dependency graphs to score seven clinical capabilities.

Across nine models, the highest score is 53.82/100, with imaging interpretation and disease-extent assessment showing the weakest performance. In completed six-patient sessions, all models score below their single-patient baselines.

Critical strengths
  • Evaluating longitudinal clinical care through real software addresses a meaningful gap between clinical reasoning benchmarks and isolated EHR interaction tasks.
  • Screenshot-only interaction and verification against persistent system records prevent agents from receiving credit for actions they merely claim to have performed.
  • Promising evaluation design. Evidence-dependent scoring, acceptance of clinically valid alternative pathways, and multi-patient management are valuable contributions.
Critical weaknesses
  1. Dataset provenance and clinical annotation are insufficiently described (Section 3.3). The paper does not identify the underlying data sources. Case IDs such as ISPY1_1206 and LUNG1-272 suggest public oncology cohorts. Please clarify the source and selection criteria for each pathway, and how charts, interviews, and follow-up records were constructed. In particular, are longitudinal outcomes predetermined or affected by agent decisions? Is the patient interview scripted or LLM-simulated? The three-clinician reference process also needs clarification regarding clinical expertise, annotation agreement before adjudication, and rubric/DAG construction. A fully worked example with its reference trajectory, rubric, dependency graph, and scored agent output would help establish the benchmark's clinical validity.
  2. Scoring reliability and the contribution of dependency graphs are not demonstrated (Section 4). The grading mechanism is central to SHIFT, but it is unclear whether item scores are assigned by clinicians, automated rules, or LLM judges. Please specify the grading implementation and report agreement with independent clinician assessments on a representative subset. The added value of evidence-dependency scoring also remains untested. A useful ablation would compare full scoring against scoring without evidence-dependency checks, reporting changes in item scores and model rankings (e.g., Kendall's τ). A quantitative analysis of downstream errors conditional on upstream evidence failures would further support the claimed error-propagation findings, which currently rely on illustrative cases.
  3. Statistical reporting and multi-patient evaluation require clarification (Section 5). Multi-patient scores include only completed six-patient sessions, but completion rates and session counts are not reported. This creates potential selection bias, particularly when compared with single-patient evaluations. Please report completion rates, clarify how run-level standard deviations are computed, and provide uncertainty estimates for the performance differences. Ideally, single- and multi-patient comparisons should use matched cases. Section 4.4 also states that dangerous decisions and missed critical results are recorded as safety events, yet the results primarily report wrong-patient action rates. Please provide the remaining safety statistics and absolute wrong-patient action counts.
  4. Novelty positioning and reproducibility need strengthening. SHIFT integrates established components, but its distinctive contributions relative to closely related benchmarks should be clearer. In particular:
  • MedCUA-Bench (arXiv:2606.03203): screenshot-based clinical computer use with EHR and imaging interfaces.
  • ABRA (arXiv:2605.11224): OHIF/Orthanc radiology interaction and longitudinal comparison.
  • LongMedBench (arXiv:2607.09322): longitudinal EHR reasoning.
  • MTBBench (arXiv:2511.20490): multimodal, longitudinal oncology decision-making.

The authors should clarify what SHIFT uniquely measures beyond these settings. An oracle-evidence or text-based condition could also help distinguish GUI execution failures from clinical reasoning errors, particularly for imaging. Finally, please clarify the agent configurations, execution budgets, and plans to release the environment, cases, rubrics, dependency graphs, and grading implementation.

Questions for authors
  1. Data provenance. What is the source dataset and license for each pathway? How were the chart narratives, interview content, and follow-up records constructed, particularly for cohorts with limited follow-up imaging? Is the patient interview driven by an LLM simulator or by scripted responses?
  2. Reference standard. What are the three clinicians’ specialties relative to the six pathways? What was inter-annotator agreement before adjudication, how many rubric items does each case have, and were LLMs used in drafting the charts, rubrics, or DAGs?
  3. Variance. Given that each run averages 600 cases, what explains run-level SDs of 1.06–2.31 points? Are there correlated run-level effects such as environment failures, grader variability, or version changes?
  4. Dependency graph. When evidence checks are disabled and items are scored on action completion and final correctness alone, how many item scores change, and do model rankings change?
  5. GUI versus reasoning. Can you report an oracle-navigation or text-mode condition to separate interface competence from clinical interpretation, particularly for the imaging capability?
2 comments
AuthorsOpenReview ↗

Response to 7gan: Provenance, grading validity, dependency ablations, and run variability

Thank you for the constructive review. M###/A### denote revised main-paper and appendix lines.

1. Data provenance (M153–156, M167–179; A1220–1377)

Cases come from six public TCIA collections (Table D1, A1260). Source data are kept verbatim. Clinicians write the missing reports, and a second clinician checks every field. Outcomes are fixed in advance. Follow-up uses paired imaging in 241 cases and clinical records in the rest. Interviews are scripted, not LLM-simulated.

2. Reference standard (M180–187, M202–206; A1416–1485)

Thirteen clinicians fill three roles per case (Table D2, A1444). In sequential review, author and checker agreed on 89.3% of items (10.7% flagged). The adjudicator modified 6.2%.

Each case has 22–27 rubric items (mean 24.10). No LLM wrote any case content, rubric, or DAG (M179). App. C.6 (A864–1215) gives a complete scored 24-item rubric (LUNG1-010) with prerequisites and judge rationales.

3. Grading and clinician agreement (M213–239; A217–468)

Audit-log checks verify actions. An LLM judge (gpt-5.6-terra) scores clinical content item by item. Two independent, blinded oncologists scored 144 episodes: judge–clinician weighted κ is 0.84/0.82, versus 0.88 between clinicians (Table B2, A293). DeepSeek-V4-Pro, a cross-family judge, re-scored every single-patient GUI episode with the same overall ranking (τ = 1.000; Table B6, A380).

All reported scores come from one scoring pipeline: item-level judging with three calls, then the prerequisite bound of Eq. 1 (M248–268; A3078–3085).

4. Dependency graph (M405–412; A1914–2137)

Evidence checks change absolute scores substantially without changing model order:

  • Graph bound disabled: 11.4% of item scores change. Means rise by 5.18–9.80 points (τ = 1.000).
  • Evidence checks removed from both judge and graph (100-case subset): 23.8% of item scores change. Means rise by 11.40–18.65 points (τ = 0.944).

The bound follows the weakest-link convention of GRADE and AJCC staging (A504–509). Across all runs, 76.4% of the lowered items have a zero-credit prerequisite. Applying the bound only at zero credit changes means by 0.64–2.86 points and keeps the ranking (Table L1, A3147).

Downstream items score zero in 80.4% of pairs whose prerequisite scored zero, versus 33.8% otherwise (Table F2, A1998).

5. Run variance (M339–345; A2217–2392)

We run each model three times to check that scores are stable, not to separate close models. The SDs of 1.06–2.31 points are across three run means, each over the same 600 cases. Snapshots are pinned and failed container attempts are never scored. Judge-replicate variability adds at most 0.018 points to a run mean's SD under the independence approximation. Run shifts differ in direction across models but agree across all six pathways within a model (Table L3, A3240). A shared environment or judge shift would move all models the same way.

Scores show performance levels and trends, not a strict order of adjacent models. In Table F6 (A2343), only the top gap (+12.46) excludes zero under both ranges (M514–515). One full 600-case run costs about $3,090 in API fees for GPT-6-Astra (Table A3, A114). Three runs match PaperBench (ICML 2025); OSWorld, WebArena, and SWE-bench report single runs.

6. Multi-patient and safety (M419–455; A1487–1909)

Models complete 38–60 of 60 six-patient sessions. In completed sessions, every model scores 2.13–9.84 points below the same cases run singly. Counting all 60 attempted sessions lowers means by at most 3.21 points, still below single-patient baselines (Table E3, A1680).

GPT-6-Astra has 3.4 dangerous-management events per 100 episodes, versus 16.4–73.4 for the other eight models (Table E2, A1538). Class-level judge–clinician κ is 0.79–0.88 (Table L2, A3169). In completed sessions, per-model wrong-patient counts range from 0 to 2,487, with 55,586 to 161,492 logged actions (Table E6(b), A1809).

7. Novelty, setup, interface, code

None of the benchmarks you raised combines screenshot-only execution, full-episode care, evidence-dependent scoring, and multi-patient switching (Table 1; A2646).

All models share one screenshot-action harness (Table A2, A056). Single-patient episodes have a 6-hour budget with no step cap, while six-patient sessions share a separate 36-hour budget.

A direct-access condition replaces the GUI with chart/report text and DICOM tools (Table H3, A2592). Scores rise by 8.76–15.82 points overall and 2.46–3.74 in imaging, yet the best imaging score is 9.04/17. Clinicians score 14.60/17 in the GUI on a separate 60-case subset, not a matched comparison. Both inputs and tools change, so this measures interface sensitivity.

Our code repository is at https://anonymous.4open.science/r/SHIFT-C84D/ (A2395). It includes the scoring code, tests, worked examples, and recordings. Researchers can apply for the full environment, all case packages, and the grading code under the repository's data use agreement (Table G1, A2400).

Dear Reviewer 7gan,

Thank you again for your time and effort in reviewing our paper. We appreciate your recognition of our evaluation of longitudinal care in real clinical software, our verification of actions against persistent system records, our evidence-dependent scoring that accepts clinically valid alternatives, and our inclusion of multi-patient management.

Our rebuttal addresses data provenance and scripted patient interviews, the reference annotation process and its agreement statistics, and our grading implementation and agreement with independent clinicians. We also cover the dependency-graph ablations you suggested, the sources of run-level variance, multi-patient completion rates and full safety statistics, a direct-access condition separating interface effects from clinical reasoning, and our code and data release.

Please let us know whether our responses and revisions have resolved your concerns, or if there is anything further we could clarify.

Best regards, Paper 24504 Authors

Reviewer_y7xKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces SHIFT, benchmark for evaluating computer-use agents over full patient trajectories (interviews, charting, orders, results, image interpretation, diagnosis, treatment, and follow-up) instead of isolated tasks. It builds an environment on top of a real EHR system (OpenMRS 3) and a DICOM image viewer (OHIF), where agents must interact purely through browser screenshots and mouse/keyboard actions from initial assessment to follow-up. With 600 cases across six oncology pathways and scores agents across seven clinical capabilities using case-specific evidence dependency graphs (DAGs), it experimented across nine models, and revealed that even the strongest model (GPT-6-Astra) scores 53.82 out of 100 and all models struggle heavily on image interpretation and multi-patient management.

Critical strengths

The researchers evaluated computer-use agents under a clinical environment that is much closer to reality than previous work, by combining a real EHR and medical imaging viewer, through only screenshots across full patient trajectory. It is also critical that their process evaluation checks whether decisions are backed by correct evidence, rather than only looking at the final diagnosis or treatment recommendation, which makes the final benchmarking more convincing. Finally, it is also practical to consider the multi-patient scenario, which is mostly neglected by previous work.

Critical weaknesses

The Major weakness is in the Evaluation Method section (Section 4):

  • First, the authors never provide a complete case rubric (or even a full single-case example) despite repeatedly referring to it to explain key rules. For example, L202 ("The items and their point values are specified in the case rubric before evaluation") and L266 ("Other items are scored according to their own requirements").

  • Second, for the maximum points of the seven clinical capabilities in Table 2 (15, 13, 17, 15, 10, 20, 10), the paper never explains how these specific weights were defined (e.g., why 13 and 17 points, or why longitudinal follow-up is only worth 10 points) or whether any sensitivity analysis was conducted on them.

  • Third, the paper never states who or what actually assigns the item and capability scores across the 600 cases, 9 models, and 3 independent runs (16,200 trajectories in total). Given the workload, this scoring is almost certainly done by an automated grader or LLM-as-a-judge, which raises another major concern: without knowing which model is used as the judge or seeing human-agreement validation, the autograder could easily be biased (especially toward models from the same family).

Another concern I have is survivorship bias over the multi-patient evaluation, as clinical scores are only calculated on the six-patient sessions that successfully completed the full workflow. As the total number of sessions is already very small, the models evaluation can be very inaccurate and unfair.

Questions for authors

1- What does a complete case rubric look like, and how are its specific items and point values decided?

2- How were the maximum points for the seven clinical capabilities in Table 2 determined, and how sensitive are the model rankings and overall scores to different weighting choices?

3- Who or what actually assigns the capability scores across the 16,200 trajectories? If an autograder or LLM-as-a-judge is used, which model serves as the judge, and how well does it agree with the human clinicians?

3 comments
AuthorsOpenReview ↗

Response to y7xK: Rubric, weighting, grader validation, and completion bias

Thank you for the careful review. M###/A### denote revised main-paper and appendix lines.

Q1. Complete rubric example; how items and points are set

App. C.6 (A864–1215) gives the full rubric for NSCLC case LUNG1-010. It lists all 24 items with weights, prerequisites, and verbatim criteria for five credit levels (0 to 1.00 in steps of 0.25). It also details GPT-5.6 Luna's credit and the judge's rationale. The ledger in Table C5(b) (A1080) allows the 22.00/100 total to be reproduced manually. A second ledger covers BC case ISPY1_1206 (23 items, 41.50/100; App. C.5, A648–838). Both totals can be recomputed offline using our code repository (https://anonymous.4open.science/r/SHIFT-C84D/). It includes the scoring code, unit tests, and both ledgers.

Clinicians define items, weights, prerequisites, and credit criteria before any model runs. Rubrics are then frozen and shared by all models (M202–205; A226–231). Each case has 22–27 items (mean 24.10). Item weights are integers from 1 to 6, summing to the capability maximum. Core clinical decisions, safety verification, and imaging interpretation receive higher weights (App. C.1, A472–487).

Q2. Capability maxima and sensitivity

The clinician panel fixed the maxima before model evaluation, following the stages of an oncology episode. These maxima are: initial assessment 15, diagnostic workup 45 (evidence strategy 13, imaging 17, synthesis 15), extent/risk 10, treatment 20, and follow-up 10. The 13 and 17 are parts of one 45-point workup block. We state that these are design choices rather than uniquely optimal weights (M198–201).

The ranking remains stable under alternative capability weights (M412–414; Table F3, A2160–2215):

  • Equal weighting of the seven capabilities preserves the full ranking (τ = 1.000).
  • Removing one capability at a time preserves the ranking in six of seven cases. Removing initial assessment swaps one adjacent pair (τ = 0.944).
  • Across the nine listed schemes, GPT-6-Astra scores 50.46–56.92.
  • Across 10,000 capability-weight vectors sampled uniformly from the simplex, GPT-6-Astra ranks first in 99.9% of draws. The top-five order holds in 99.8%.

Q3. Who scores, and agreement with clinicians

Deterministic audit-log checks verify actions. An LLM judge (gpt-5.6-terra) scores clinical content item by item against frozen criteria at temperature 0. It takes the median item credit over three calls per episode, with model identity and run index hidden. It never returns a total. Prerequisite bounds and totals are computed by fixed formulas (M213–215, M232–233; App. B.1–B.2, A218–273).

Two board-certified oncologists, independent of the rubric panel, scored 144 episodes (3,472 items). They were blinded to the judge's scores and to each other. Weighted κ between the judge and each clinician is 0.84 and 0.82, versus 0.88 between the clinicians (M234–236; Table B2, A293).

The primary judge is itself an evaluated model. We tested same-family preference in two ways:

  • Clinician audit (Table B5, A347): the judge's offset from clinicians is +0.29 points larger on GPT-family episodes than on non-GPT episodes (95% CI [−0.48, +1.06], n = 48). It is +0.24 larger on its own model's episodes ([−0.98, +1.46], n = 16). Neither result is significant, but these intervals cannot exclude bias (M236–239).
  • Cross-family re-scoring (M333–335; Table B6, A380): DeepSeek-V4-Pro, outside every evaluated family, re-scored all 16,200 closed-API single-patient GUI episodes and the 1,200 open-weight episodes. It scores the closed-API models 2.68–3.12 points lower, with similar mean offsets for GPT (−2.96) and non-GPT (−2.80) models. The overall ranking remains the same (τ = 1.000).

Multi-patient evaluation and survivorship bias

We report completion and score every attempted session (M427–430, M445–455; App. E.2, Table E3, A1680). Models complete 38–60 of 60 six-patient sessions (63.3%–100%). In all-session scoring, unstarted patients receive 0. Partly finished patients keep the credit earned on the full 100-point rubric, without rescaling. All-session means are no higher than completed-session means, and all models remain below their single-patient baselines (M447–448).

Completed sessions are also compared with the same cases run singly. Every model scores lower by 5.97–8.58 points (same-disease) and 2.13–9.84 (cross-disease). Session-bootstrap 95% interval half-widths are 0.94–1.68. With 3–6 completed same-disease sessions per pathway per model, these comparisons remain descriptive (M507–510).

Is the 100-point scale attainable?

Three oncologists outside the reference panel each completed the same 60 cases in the same GUI. They were blind to rubrics and reference trajectories. They average 91.40, versus 54.14 for GPT-6-Astra on those cases (M335–337; App. H.2, A2444–2478).

Dear Reviewer y7xK,

Thank you again for the time and effort you dedicated to reviewing our paper. We appreciate your recognition of our realistic clinical environment—combining a real EHR and image viewer via screenshot-only interaction—along with our process evaluation verifying that decisions are supported by correct evidence, and our inclusion of multi-patient scenarios.

In our rebuttal, we provide a complete scored case rubric (App. C.6), clarify how capability weights were set and how sensitive the rankings are to them, and report the LLM judge's identity and agreement with blinded clinicians (including tests for same-family bias). We also include completion rates and all-session scoring for the multi-patient evaluation, as well as a clinician baseline on the same cases.

Could you please let us know whether our rebuttal and revisions have addressed your concerns, or if there are any remaining points we can clarify?

Best regards, Paper 24504 Authors

I appreciate the authors' engagement and for providing details to address my concerns. However, considering the overall quality and contribution of the paper, I'll maintain my original score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

We thank all reviewers. The revision adds the requested evidence.

  • Grading (Sec. 4.2). Audit-log checks verify actions; an LLM judge (gpt-5.6-terra) scores clinical content. Weighted κ against two blinded oncologists is 0.84/0.82 (0.88 between clinicians). A cross-family judge re-scores all single-patient GUI episodes with the same overall ranking. All reported scores come from one scoring pipeline fixed before the formal runs (App. K).
  • Provenance and references (Sec. 3). Cases derive from six cited TCIA collections. Clinicians write and cross-check all reconstructed records, rubrics, and dependency graphs; interviews are scripted. A 13-clinician panel builds the references.
  • Human baseline (Sec. 5.2). Independent clinicians score 91.40 versus 54.14 for the best model on the same 60 cases.
  • Ablations (Sec. 5.3). Disabling the dependency bound raises scores by 5.18–9.80 points with unchanged ranking; 76.4% of lowered items have a zero-credit prerequisite. We report weight sensitivity and direct access.
  • Run variability (Sec. 5.2). Three runs check stability; scores show levels and trends, not a strict order of adjacent models.
  • Multi-patient and safety (Sec. 5.3–5.4). Completion rates, all-session scores, matched comparisons, and safety counts.
  • Scope. Two open-weight models, a revised Table 1, and Ethics and Reproducibility Statements.

The appendix adds protocols, a complete scored rubric (App. C.6), and screenshots (App. J).

Code repository: https://anonymous.4open.science/r/SHIFT-C84D/. It includes scoring code, tests, worked examples, and recordings. Researchers can apply for the full environment, all case packages, and the grading code under the data use agreement in the repository (Table G1). Raw data come from TCIA under each license.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.