- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces SHIFT, a benchmark in which computer-use agents manage oncology patients through a real EHR (OpenMRS 3) and an image viewer (OHIF/Orthanc). Agents see only browser screenshots and act through mouse and keyboard input. Each episode covers the full course of care: interviewing the patient, charting, ordering tests, reviewing results, reading images, diagnosis, staging, treatment, and follow-up. The benchmark contains 600 cases, 100 for each of six pathways (BC, CRLM, GBM, HCC, HNSCC, and NSCLC).
Scoring is built around seven fixed capabilities whose points add up to 100. Each case has its own rubric items, scored 0/0.25/0.5/0.75/1, and a "dependency graph" (DAG) records which evidence each judgment needs. Reference trajectories were produced by three clinicians acting as author, checker, and adjudicator.
Critical strengths
- Timely and important target. Evaluating agents inside real clinical software, rather than through text or curated APIs, fills a real gap. The design principle that "writing that an order was submitted does not prove it happened" is a good one.
- Full-trajectory evaluation unit. Following one patient from first contact to follow-up tests whether agents carry evidence forward, which single-step benchmarks cannot do.
Critical weaknesses
W1. The paper never states who or what does the scoring, and the grader is not validated.
- Section 4.2 states that each scoring decision "includes an explanation and references to the supporting records", but it never says whether scoring is done by clinicians, an LLM judge, rule-based checks, or a combination of these.
- The scale makes full human grading very unlikely. There are 600 cases × 9 models × 3 runs = 16,200 single-patient episodes, plus the multi-patient sessions, and each of them has many items scored with graded partial credit. If an LLM judge is used, the paper must name the judge model, provide its prompts, and report its agreement with clinicians, e.g., Cohen's or weighted κ on the 0–1 item scores.
- Three of the top four models are GPT-family models (GPT-6-Astra, GPT-5.6 Terra, GPT-5.6 Luna). If the judge is also a GPT model, self-preference bias is a real risk that must be ruled out.
- The authors acknowledge that "scoring consistency requires further validation" (Sec. 6). For a benchmark paper, however, grader validity is a core contribution, not future work.
W2. The paper does not describe where the cases come from, how they were built, or how the simulated patient works.
- The case IDs "ISPY1 1206/1064" and "LUNG1-272" match public TCIA collections: the I-SPY1 (ACRIN 6657) breast MRI collection and NSCLC-Radiomics "LUNG1" (Aerts et al., Nat. Commun. 2014). These collections contain images and a limited set of clinical variables (e.g., pCR and survival), but no interview transcripts, full charts, or laboratory panels.
- The paper does not say how the missing content (history, interview answers, test results, reports, and follow-up) was created, whether by clinicians, by an LLM, or from templates. This matters for realism, bias, and possible leakage.
- The paper also does not describe how the "patient" answers interview questions (scripted answers or an LLM simulator) or what happens when the agent asks something off-script.
- None of the source datasets is cited, and neither their licenses (some TCIA collections are restricted to non-commercial use) nor the release plan for SHIFT is stated.
W3. The reference standard and its reliability are not described well enough.
- Three clinicians in fixed roles produced the references for 600 cases across six oncology pathways in five subspecialties (breast, hepatobiliary, neuro-oncology, head-and-neck, and thoracic). The paper reports neither their specialties and experience nor how the workload was divided (did one clinician complete all 600 trajectories in the GUI?).
- No agreement statistics are given, e.g., how often the checker disagreed with the author or how many items the adjudicator changed. The paper also does not say how the "acceptable alternatives" were listed or how complete they are.
- No human baseline. No clinician or resident completes even a subset of the cases through the same interface under the same rubric, so a score of 53.82/100 cannot be interpreted. A strict or ambiguous rubric could make even experts score low, especially with partial credit in steps of 0.25.
W4. Details of the agent setup and benchmark statistics are missing, which limits reproducibility.
- The paper does not describe the agent scaffold (each vendor's native computer-use API or a single shared harness), the system prompt, the action space, the screenshot resolution, how history is kept over 200–470 tool calls, the step/time/token limits, the "conditions for ending the task", the decoding settings, the exact model versions and API dates, or the cost per case.
- Case and rubric statistics are missing: the number of items per case, the distribution of item weights, the DAG size (nodes and edges), the number of stages, the number of images per case, and the tests available.
- No full example rubric or DAG is shown (Fig. 2 is "illustrative").
- It is unclear whether the environment, cases, and rubrics will be released, and there is no reproducibility or code/data availability statement.
- All nine models are accessed through closed APIs, so the results will drift as the APIs change. No open-weight model is tested, whether a general-purpose VLM or a GUI-specialized agent model.
W5. Safety results are promised but not reported.
- Section 4.4 states that "safety events, including actions on the wrong patient, dangerous clinical decisions, and missed critical results, are recorded separately and reported alongside the clinical scores".
- However, only wrong-patient action rates in multi-patient sessions are reported. Dangerous decisions and missed critical results are never shown, even though the abstract claims that SHIFT "quantifies the gap that must be closed before computer-use agents can safely operate real clinical software".
- The NSCLC example, in which an agent schedules a biopsy and staging for a patient whose death has been recorded, shows that such events do occur. A per-model safety table is needed.
W6. The contribution beyond existing benchmarks is mainly the combination of existing elements, and the comparison table is hard to interpret.
- Each ingredient already exists: GUI-based clinical workflows (CareFlow/CarePilot), EHR agent environments (MedAgentBench, PhysicianBench), staged clinical pathways (MedChain, CP-Env), interactive diagnosis with test ordering (AgentClinic; also SDBench/MAI-DxO, Nori et al., 2025, not cited), and rubric-based process scoring (PaperBench; HealthBench, Arora et al., 2025, not cited). The novelty is in combining them. This is useful, but the paper should present its contribution more modestly.
- Table 1 marks most cells for other benchmarks as "N/A" ("outside the task setting") rather than ✗. This blurs the line between "not supported" and "not applicable" and makes the comparison look more favorable than it is. Some entries are also debatable. For example, MedAgentBench runs on a FHIR EHR with persistent patient records, yet it is marked N/A for "persistent patient state".
W7. Much essential content is missing, and the paper has no appendix.
- The paper is short (about nine pages) and has no appendix, so the rubric design, case construction, patient simulator, agent harness, and grader are not described anywhere.
- There is no ethics statement (covering the use of patient-derived data, licensing, and privacy) and no analysis of the grader's limitations.
- The introduction and abstract are clear and well motivated. The conclusion, however, restates the results without discussing what the benchmark can and cannot measure, e.g., GUI skill vs. reasoning and grader validity.
Questions for authors
Please see Weaknesses above.
2 comments
Dear Reviewer CPJy,
Thank you again for the time and care you put into reviewing our paper. We appreciate your recognition of the importance of evaluating agents within real clinical software, verifying actions against system records rather than agent claims, and using full trajectories as the unit of evaluation.
Our rebuttal covers grader validation against blinded oncologists (including tests for self-preference bias); case sources, construction, and the scripted patient; the reference panel and a clinician baseline; and the agent harness, case statistics, open-weight models, and code release. We also present the per-model safety table, the revised Table 1 and novelty positioning, and the new appendix, ethics statement, and limitations.
Could you let us know whether our rebuttal and revisions address your concerns, and if any questions remain that we can help clarify?
Best regards, Paper 24504 Authors
Response to CPJy: Grader validity, clinical references, safety, and reproducibility
Thank you for the thorough review. M###/A### denote revised main-paper and appendix lines.
W1. Grader validity (M213–239; App. B, A217–468)
Audit-log checks verify actions, while an LLM judge (gpt-5.6-terra) scores clinical content item by item with model identity hidden (prompt: App. B.2, A257). Two board-certified oncologists independent of the rubric panel scored 144 episodes, blinded to judge scores. Judge–clinician weighted κ is 0.84/0.82, versus 0.88 between clinicians (Table B2, A293).
Because the judge is itself one of the evaluated GPT models, we tested self-preference directly (Tables B5–B6, A347–380):
Because the audit intervals are too wide to rule out a small bias, we added the cross-family re-scoring.
W2. Sources, construction, simulated patient (M153–156, M174–179; App. D, A1220–1377)
All six TCIA collections are cited with DOIs, licenses, and selection rules (Table D1, A1260). NSCLC-Radiomics is CC BY-NC 3.0. Raw data are not redistributed; users re-acquire them from TCIA under each license. Clinicians, not LLMs, wrote the missing reports and chart notes from stage-locked templates, and a second clinician checked every field.
Patient replies are scripted: matched questions return clinician-written fact blocks verbatim, while others get a neutral cannot-answer reply.
W3. Reference standard and human baseline (M180–187; App. D, A1416–1485)
Thirteen clinicians across five oncology subspecialties and radiology fill three roles per case (Table D2, A1444). No single clinician built all 600 references: each author wrote 50–67 cases, and a same-subspecialty clinician cross-checked every case. The checker agreed on 89.3% of items (10.7% flagged), and the adjudicator modified 6.2%. This is sequential review agreement, not independent inter-rater reliability. Author and checker listed acceptable alternatives independently, and the adjudicator merged them.
Three oncologists outside the panel each completed the same 60 cases in the same GUI, blind to the rubrics. They average 91.40, versus 54.14 for GPT-6-Astra on those cases (M335–337; Tables H1–H2, A2484). An audit attributes 0.62 of their 8.60 lost points to valid alternatives missing from the rubric.
W4. Setup, case statistics, reproducibility
All models share one OSWorld-adapted harness with pinned model snapshots (M312–318; Table A3, A114). App. A (A003–216) gives the prompt, action space, screenshot size, history handling, limits, decoding settings, and costs. Cases have 22–27 items and 17–26 prerequisite edges (Table D3, A1379), and App. C.6 (A864–1215) gives a full 24-item rubric with scores.
Two open-weight models, UI-TARS-72B and Qwen2.5-VL-72B, score 20.61 and 18.01 on one run each (Table 3). Our code repository is at https://anonymous.4open.science/r/SHIFT-C84D/ (M542–546; App. G, A2395). It includes the scoring code with unit tests, worked scoring examples, and agent-run recordings. Researchers can apply for the full environment, all case packages and rubrics, and the grading code under the data use agreement in the repository (Table G1, A2400).
W5. Safety (M419–423; App. E.1, A1489–1635)
Table E2 (A1538) reports per-model counts for each safety event class. GPT-6-Astra has 3.4 dangerous-management events per 100 episodes, versus 16.4–73.4 for the other eight. Missed-material-evidence events number 674–1,550 per model across 1,800 episodes. This class counts material information that a note omits or contradicts in records that were released and reviewed, rather than every result left unopened (A1633–1635). Clinician–judge agreement on safety events is κ = 0.86 pooled and 0.79–0.88 by class; dangerous management has κ = 0.88 (Table L2, A3169). The abstract now states what SHIFT measures (M027–029).
W6. Novelty and Table 1
The individual components have precedents; our contribution is integrating them into one continuous episode (M069–070). Table 1 now uses × instead of N/A and marks MedAgentBench as having persistent state (M108–125). SDBench/MAI-DxO and HealthBench are also cited (M080, M093).
W7. Appendix, ethics, scope
We added an Ethics Statement (M527–535), limitations covering grader validity (M500–515), and a conclusion stating what SHIFT does and does not measure (M519–525). In a direct-access condition with chart/report text and DICOM tools (M415–419; Table H3, A2592), overall scores rise by 8.76–15.82 points, yet the best imaging score is 9.04/17. Because inputs and tools both change, this measures interface sensitivity.