acceptodds
Under review as a conference paper at ICLR 2027

MedImageOSWorld: Benchmarking GUI Agents for Medical Image Consoles

Abstract

Graphical consoles offer a practical interface for medical acquisition assistance, allowing agents to work through the controls and visual feedback used by human operators. Reliable assistance requires linking on-screen anatomy to acquisition decisions that determine what image evidence becomes available next. We introduce MedImageOSWorld, a benchmark for evaluating this capability in simulated CT, MR, and ultrasound consoles. Using screenshots and mouse-and-keyboard actions, agents configure protocols, plan acquisitions, inspect the resulting images, and make corrective adjustments across seven capability levels, from console operation to feedback-driven control. Evaluation combines task-specific workflow checks with hidden anatomical ground truth to assess procedural completion and acquisition outcomes separately. A common evaluation protocol specifies episode conditions and interaction budgets, while recorded trajectories support analysis of how agents observe, act, and respond to acquisition feedback. Across eleven open-weight agents, success rates range from 3.0 to 25.0 on a 0–100 scale while workflow-progress rates reach 21.5–74.5: agents complete much of the console workflow but rarely acquire the intended anatomy. Success collapses between perception and planning, from 79–90% at the lowest three levels for the best agent to at most 12% for millimetre-level planning and 0% for closed-loop control. Two proprietary agents reach success rates of about 34 and exceed the best open-weight agent mainly in acquisition quality (61 versus 42). MedImageOSWorld provides a controlled setting for studying whether general-purpose GUI agents can translate visual observations into effective medical acquisition decisions.

Primary area:
datasets and benchmarks
Keywords:
GUI agents, computer-use agents, medical image acquisition, vision-language models, physically grounded evaluation, benchmarks

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bURSOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MedImageOSWorld, a benchmark for evaluating GUI agents on medical image acquisition across simulated CT, MR, and ultrasound consoles. It evaluates many agents and analyzes their failure modes using workflow checks and hidden anatomical ground truth.

Critical strengths
  1. The novelty is strong, addressing the underexplored problem of action-conditioned medical image acquisition.

  2. The evaluation protocol is comprehensive, separating workflow completion from acquisition quality.

  3. The in-depth failure analysis provides useful insights into the limitations of current GUI agents.

Critical weaknesses
  1. The simulated environment has limited physical fidelity, and its validity for real-world medical image acquisition remains unclear.

  2. The comparison between proprietary and open-weight models is not fully fair.

Questions for authors
  1. How do proprietary and open-weight models compare when evaluated on the same set of test episodes?

  2. Are there results from human baseline that can serve as a reference for evaluating agent performance?

2 comments
AuthorsOpenReview ↗

Official Comment by Authors

We thank the reviewer for the constructive questions. Both concern the reference points against which agent performance is read: whether proprietary and open-weight models are compared on the same test episodes, and whether a human baseline is available.

1. The comparison between proprietary and open-weight models (Q1 & W2)

The original evaluation uses the same task definitions for all models, but the proprietary models are evaluated on only the five anatomies per task due to limited resources. We therefore restricted the results of the open-weight models to the same test episodes, matching the task, asset, condition, seed and reset, and recomputed the scores with the same scoring and aggregation procedures as in Figure 3.

As Table 1 shows, GPT-6 Sol and Claude Opus 5.5 achieve SRs of 35.6 and 34.1, respectively, compared with 24.4 for the strongest open-weight model, Qwen3.8-27B. On the matched episodes, both proprietary models remain ahead of all open-weight models on SR, PR and AQ, and the ranking of the strongest open-weight model is unchanged. The smaller proprietary-model cohort reflects the cost of these evaluations. Evaluating the two proprietary models on five subjects already cost approximately USD 3,000–4,000; extending the same evaluation to 30 subjects per task would cost roughly USD 18,000–24,000. We therefore used the fixed five-subject subset and provided the matched-episode comparison above to assess whether the main comparison holds on the same test instances.

Table 1. Performance on the proprietary models' test set, consisting of five subject cases per task. Open-weight models are evaluated on the same cases using the scoring rules in Figure 3. SR and PR are calculated across 49 tasks, while AQ is calculated across 28 applicable tasks, following Eq. 1. Results for proprietary models are taken from Figure 3. All values are reported as percentages.

-------------------------------------------------------
Model                           SR       PR       AQ
-------------------------------------------------------
GPT-6 Sol                      35.6     85.5     61.1
Claude Opus 5.5                34.1     83.4     61.3
-------------------------------------------------------
Qwen3.8-27B                    24.4     74.1     42.7
Muse-Glimmer-30B               21.1     72.7     28.5
Gemma-4-31B                    14.7     63.6     23.0
UI-Mate-27B                     9.8     51.5     19.9
Qwen3-VL-30B-A3B-Thinking       8.0     43.6     24.6
GUI-Owl-1.5-32B-Thinking        6.5     33.0     15.1
Qwen2.5-VL-72B-Instruct         5.5     38.5     19.8
GUI-Owl-1.5-32B-Instruct        5.2     32.1     19.9
OpenCUA-32B                     4.1     36.4     22.3
Qwen3-VL-30B-A3B-Instruct       3.8     21.3      0.8
UI-TARS-1.5-7B                  3.4     22.1      5.3
-------------------------------------------------------

2. Human baseline as a reference (Q2)

Three medical imaging experts reviewed the design of all benchmark tasks. During development, all tasks were completed manually within the specified interaction budgets on a separate digital subject that is excluded from the benchmark test cohorts. This check established that the tasks are feasible; it was not a systematic measurement of human performance across the test subjects. We have clarified this distinction in Section 4.


3. Physical fidelity of the simulated environment (W1)

Thanks for pointing out this insightful concern. We agree that the simulator’s physical fidelity is limited, and we have added this in the Limitation section more explicitly. Simulation offers three properties that are essential for benchmarking and that real scans cannot provide. First, the ground truth is known and every condition is controllable: the anatomy is specified exactly, so each episode can be scored deterministically against the hidden ground truth, without human or model judgment. Second, the evaluation is reproducible: every agent faces exactly the same cases, and any run can be repeated under identical conditions, which is not possible in real scan. Third, the cost is far lower than that of real scanning which makes it feasible to evaluate agents over a large number of episodes.

On validity for real-world acquisition, we have begun running the same agent setup on a real cardiac MR scanner. As in the benchmark, the agent observes the scanner console through screenshots and operates it with a mouse and keyboard, here to perform the scan autonomously. We have already observed successful autonomous scans. This evaluation is ongoing and is not part of this paper.


We sincerely thank the reviewer again for your thoughtful and constructive feedback. Your comments have helped us reflect more carefully on our work and improve the manuscript. We hope our responses have addressed your concerns, and we would be happy to provide any further clarification or discussion if needed.

I appreciate the author's rebuttal. Despite some shortcomings, I acknowledge that this work is meaningful. I will maintain my rating which is positive.

Reviewer_ib83OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MedImageOSWorld, a benchmark for screenshot-based GUI agents operating simulated CT, MR, and ultrasound acquisition consoles, where actions change the physical acquisition state and subsequently revealed image evidence. The benchmark separates workflow completion from acquisition quality through hidden subject-frame anatomical ground truth, and organizes 49 tasks into a seven-level capability ladder covering console operation, perception, geometric planning, parameter control, feedback, and recovery.

Critical strengths
  1. The distinction between workflow progress and hidden acquisition quality is well motivated and clinically meaningful, because a completed console workflow does not necessarily imply that the acquired volume, plane, or ultrasound view is adequate.

  2. The use of a shared subject coordinate frame enables automatic evaluation of geometric quantities such as coverage, endpoint error, centre error, orientation error, and field of view without relying solely on textual reports or action-trace matching.

Critical weaknesses
  1. Could the authors clarify the extent to which the simulated CT, MR, and ultrasound consoles faithfully represent clinically relevant acquisition physics and real console interaction constraints? The paper states that the environment uses “task-relevant physical abstractions rather than a full scanner model,” but the conclusions are framed around medical acquisition capability. Please specify which scanner effects are modeled or omitted, including patient motion, noise, reconstruction artifacts, coil/probe constraints, CT dose-image-quality tradeoffs, MR sequence timing, ultrasound speckle, shadowing, and deformable probe pressure.

  2. How is the benchmark protected against simulator-specific shortcut learning? Since all consoles are rendered from XCAT-based digital phantoms with fixed layouts, fixed UI designs, and a limited set of anatomy configurations, agents may exploit pixel-level correlations among UI widgets, scout appearances, task wording, phantom identities, or reset seeds rather than learn transferable acquisition reasoning. Please provide cross-phantom, cross-renderer, cross-layout, and out-of-distribution anatomical-variation evaluations.

  3. Why are the primary results dominated by binary PASS-based SR when many tasks involve continuous geometric outcomes, such as coverage fraction, isocentre offset, endpoint error, plane angle, and field-of-view adequacy? Small changes near thresholds can convert an episode from PASS to FAIL despite nearly equivalent acquisition quality. Please report threshold-sweep curves, continuous utility scores, calibration around clinical tolerances, and rankings under alternative clinically justified threshold profiles.

Questions for authors
  1. Could the authors clarify how acquisition quality AQAQ is normalized and aggregated across heterogeneous CT, MR, and ultrasound tasks? Figure 3 states that AQ is computed only on 28 eligible geometry and parameter tasks, but it is compared with SR and PR over all 49 tasks. Please provide the exact task-level functions, normalization ranges, treatment of tasks without AQ, and whether a high AQ can result from partial success despite missing required workflow stages.

  2. Could the authors provide complete, model-specific agent harness details in the main paper rather than only referring to a supplementary configuration table? Since prompts, action codecs, deterministic normalization, repair policies, additional feedback, rejected-action handling, wall-time limits, and provider-specific tool-calling formats may materially affect GUI-agent performance, “frozen task-side protocol” alone does not guarantee comparability.

  3. How are action codecs and output repairs prevented from becoming hidden agent-specific policies? Section 3.1 permits a “declared, version-fixed, deterministic codec” to normalize model output into legal actions, and permits model-specific repair logic and additional prompting. Please quantify, for each agent, the fraction of actions requiring normalization, repair, rejection, or corrective feedback, and report scores with repairs disabled or replaced by a uniform shared codec.

3 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/3)

We thank the reviewer for the detailed and constructive comments. The three questions concern how the acquisition-quality metric is defined and aggregated, and whether the agent harness and the per-model action codecs allow a fair comparison between agents. The detailed response follow.

1. AQ definition, normalization, and aggregation (Q1 & W3)

Thank you for identifying this ambiguity.

Eligible tasks and aggregation. AQ applies to 28 tasks (8 CT, 18 MR and 2 ultrasound). These are the geometry and parameter tasks, whose outcome is an acquired image or a prescription that can be compared quantitatively with the hidden ground truth. The other 21 tasks test workflow and console operation; their outcome is a discrete state that is either correct or not, which SR and PR already capture, so no continuous quality score is defined for them. These tasks are excluded from AQ rather than assigned zero. Within each eligible task, we average the retained episode-level quality scores, and we then aggregate these task means using capability-level weights:

\qquad w _ t = L _ t + 1 .$$ Here $\overline{\mathrm{AQ}}_t$ is the task mean and $L_t$ is the assigned capability level of task $t$. The weights sum to 132 over the 28 eligible tasks. SR and PR aggregate all 49 tasks with the same weighting, with a weight sum of 181. We have clarified these distinct aggregates in Section 5 and in the caption of Figure 3. **Task-level functions and normalization.** The supplementary AQ table (Table 7) now lists the quality function, level and weight of each of the 28 tasks. Every episode-level AQ lies in $[0,1]$ and is built from three elements: the coverage fraction of the target anatomy; a tolerance-normalized error score $q(e,\tau)=\max(0, 1-|e|/(2\tau))$, which is 1 for zero error, 0.5 at the tolerance $\tau$ and 0 at $2\tau$ or beyond; and, for the cardiac tasks, the fraction of the selected quality gates that are passed. Tasks with several components combine them by a geometric mean. **Can AQ be high when workflow stages are missing?** Yes. AQ measures the specified quality of the acquired or prescribed result under task-specific prerequisites, so a high AQ can coexist with unmet workflow or terminal requirements. This is why SR remains the primary metric: it records whether an episode satisfies the complete task requirements, including all applicable workflow and acquisition criteria. PR and AQ complement the binary SR. **Figure 3(a)** now lists the PR and AQ values alongside SR, so that workflow progress and acquisition quality are easier to compare, and we have clarified these distinct roles in the revised metric description. **Sensitivity of SR to the tolerances.** As an additional analysis, we report a tolerance sweep on all cardiac MR tasks reported(Table 1). The benchmark setup are unchanged; the sweep shows the success rate that results when every angle, distance and coverage tolerance is scaled by a common factor $k$. We rescored the cardiac MR episodes with every angle, distance and coverage tolerance scaled by a common factor $k$. The ranking of the 11 open-weight agents is identical at every scale from $0.5\times$ to $1.5\times$. **Table 1.** Cardiac MR success rate (%, mean over all cardiac MR tasks) when every angle, distance and coverage tolerance is scaled by $k$. ```text ------------------------------------------------------------------------- Tolerance scale k --------------------------------------------- Agent 0.5x 0.75x 1x 1.25x 1.5x ------------------------------------------------------------------------- Qwen3.8-27B 53.1 54.8 57.4 58.2 58.5 Muse-Glimmer-30B 40.7 42.6 45.3 46.1 46.1 Gemma-4-31B 39.5 39.5 39.5 39.5 39.5 UI-Mate-27B 27.7 28.5 30.7 33.7 33.7 Qwen2.5-VL-72B-Instruct 15.6 15.8 15.9 15.9 15.9 Qwen3-VL-30B-A3B-Thinking 14.4 14.5 14.6 14.7 14.7 GUI-Owl-1.5-32B-Instruct 12.6 12.6 12.6 12.6 12.6 GUI-Owl-1.5-32B-Thinking 11.8 11.9 12.0 12.0 12.0 OpenCUA-32B 9.3 9.3 9.3 9.3 9.3 Qwen3-VL-30B-A3B-Instruct 8.8 8.8 8.8 8.8 8.8 UI-TARS-1.5-7B 5.6 5.6 5.6 5.6 5.6 ------------------------------------------------------------------------- ```
AuthorsOpenReview ↗

Official Comment by Authors (2/3)

2. Harness detail and short-cut learning (Q2 & W2)

Thanks for pointing this out. We agree that the harness belongs in the main text, and we have added the detailed description in manuscript's Section 6.1. The open-weight agents share common harnesses, which fix the prompt template, the history policy and the feedback text for every agent. The proprietary agents run in their vendors' CLIs (Codex CLI and Claude Code), each restricted to a single GUI-action tool. The per-agent normalization, repair, rejection and corrective-feedback rates are reported in our answer to Q3.

We evaluate existing pretrained agents without benchmark-specific training, fine-tuning, or test-time weight updates. The evaluation covers multiple parent anatomies and their derived variants, as described in the supplement. These results characterize performance across the represented anatomical variations; they do not establish transfer across rendering systems, interface layouts, or out-of-distribution anatomies. We have clarified the fixed-weight setting in Section 6.1 and the evaluation scope in the limitations.


3. Action codec and repair inspection (Q3)

The codecs adapt model-specific output formats; they do not select task actions on the agent’s behalf. Each codec is a deterministic function of the model's raw reply and the task's static action allowlist. Because these are its only inputs, a codec cannot make decisions that depend on the image or on the state of the console. A rejected reply consumes one action from the budget. The corrective feedback is the next prompt sent to the agent and contains only the codec's error code and a description of the error.

Table R2 reports, for the cardiac MR tasks as an example, the share of model calls that were normalized, repaired, rejected or followed by corrective feedback. Of these four categories, only normalization and repair change the model's raw output. Normalization is below 1% of calls for every agent, and repair is at most 5.1% for ten of the 13 agents, including both proprietary agents and the four strongest open-weight agents. Repair is substantial for three agents only: Qwen2.5-VL-72B-Instruct (86.2%), Qwen3-VL-30B-A3B-Thinking (39.7%) and OpenCUA-32B (18.1%).

Disabling repair or normalization would therefore mainly affect these three agents, all of which rank seventh or lower among the 13 agents; the comparison among the leading agents does not depend on the repair logic. Forcing all models into a single format would measure format compliance, rather than the abilities we evaluate: GUI operation and geometric understanding.

Table 2. Codec statistics on the cardiac MR tasks: share of model calls (%).

------------------------------------------------------------------------
                                          % of model calls
                            --------------------------------------------
Agent                          Norm.     Repaired   Rejected   Feedback
------------------------------------------------------------------------
Qwen3.8-27B                     0.9         1.3       12.7       12.6
UI-Mate-27B                     0.6         5.1       75.1       74.0
GUI-Owl-1.5-32B-Instruct        0.0         1.1        0.1        0.1
GUI-Owl-1.5-32B-Thinking        0.3         1.8        1.8        1.7
Qwen3-VL-30B-A3B-Instruct       0.0         0.2        0.0        0.0
Qwen3-VL-30B-A3B-Thinking       0.0        39.7        2.7        2.6
UI-TARS-1.5-7B                  0.0         0.4        0.9        0.9
Qwen2.5-VL-72B-Instruct         0.0        86.2        0.2        0.2
OpenCUA-32B                     0.1        18.1        0.5        0.5
Gemma-4-31B                     0.0         0.1       11.7       11.7
Muse-Glimmer-30B                0.0         0.0        5.0        5.0
------------------------------------------------------------------------
Claude Opus 5.5                 0.0         0.0        1.8        1.8
GPT-6 Sol                       0.0         0.0        0.0        0.0
------------------------------------------------------------------------
AuthorsOpenReview ↗

Official Comment by Authors (3/3)

4. Clarification of simulated consoles (W1)

Thank you for raising this concern. The consoles are designed to evaluate acquisition decisions. Each forward model therefore reproduces the visible consequences of the decisions we score, and omits scanner effects that do not bear on those decisions. We added Table 9 in supplementary listing the effects as you suggested. In brief, anatomy outside an acquired CT range or MR slab is absent from the series. MR contrast follows a bSSFP signal model with heartbeat-resolved T1T _ 1/T2T _ 2 mapping. The ultrasound image is rendered from the probe pose and skin contact, with speckle and shadowing. CT and MR noise, reconstruction artifacts, respiratory motion in CT and MR, MR coil effects, and probe-pressure deformation are not modeled.

The benchmarked abilities are essential for real clinical scans. The benchmark tests three abilities: recognizing anatomy on the console, planning the acquisition geometry, and setting acquisition parameters so that the intended anatomy is acquired. At a real scanner, an operator must do exactly this before any image is acquired, and an examination that misses the target anatomy cannot be rescued by image quality. An agent that fails at these steps in simulation cannot be expected to succeed at a clinical scanner, so these abilities are a necessary condition for real-world acquisition.

Omitted effects and future work. CT and MR noise, reconstruction artifacts, respiratory motion in CT and MR, MR coil effects and probe-pressure deformation are not yet modeled. We deliberately begin under relatively idealized imaging conditions, so that failures can be attributed to anatomical understanding and planning rather than to image degradation. Even under these conditions, many current open-weight agents rarely acquire the intended anatomy. Future work will add noise, reconstruction artifacts, the dose–noise trade-off, respiratory motion and probe-pressure deformation, so that the benchmark also covers decisions that depend on these effects. We have described this scope in Sections 3 and 7.

Reviewer_AeNGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces MedImageOSWorld, a benchmark for GUI agents operating simulated CT, MRI, and ultrasound consoles. Its 49 tasks evaluate console operation, acquisition planning, and feedback-driven correction. By separating workflow progress from acquisition quality measured against hidden anatomical ground truth, the benchmark shows that current agents often complete procedural steps without acquiring the intended anatomy.

Critical strengths
  1. Addresses an important and relatively underexplored problem: connecting GUI interaction with medical image acquisition.
  2. Separates workflow completion from anatomical correctness, revealing failures that conventional GUI success measures may overlook.
Critical weaknesses
  1. No systematic expert performance baseline is reported under comparable interaction conditions, making it difficult to assess task feasibility and the appropriateness of interaction budgets and scoring thresholds.
  2. The benchmark jointly tests GUI interaction, medical image understanding, and acquisition knowledge without controlled experiments isolating their contributions. Moreover, some capability labels exceed the demonstrated task coverage: L5 contains only three MRI mapping workflows, limiting broader conclusions about dynamic closed-loop control.
  3. The methods describe ground-truth-based scoring, while the AI-use statement reports AI-assigned final scores. The authors should specify where model judgment enters evaluation and report its agreement with independently verified scores.
Questions for authors
  1. Could the authors report an expert baseline under comparable interaction conditions to contextualize task difficulty and assess the appropriateness of interaction budgets and scoring thresholds?
  2. Could controlled experiments disentangle GUI interaction from medical image understanding? For example, providing kidney locations when asking agents to identify or correct incomplete renal coverage could help isolate the perceptual bottleneck.
  3. Could the authors clarify the scoring pipeline, particularly how ground-truth-based scoring relates to the AI-assigned final scores mentioned in the AI-use statement, and which stages involve model judgment?
4 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/3)

Thanks for the careful and constructive assessment. The three concerns address how the benchmark's difficulty is calibrated, what its end-to-end score actually measures, and how the scores are produced. Detailed responses follow.


1. Expert baseline, interaction budgets and scoring thresholds (Q1&W1)

Three medical imaging experts helped design the tasks and performed them during development. We used their runs to verify that every task can be completed and to set the step budgets. We have not yet conducted a separate human-baseline study under the same evaluation protocol as the agents, and we will state this distinction explicitly in the paper (Section 4 Task Sourcing).

Interaction budgets. We gave agents more steps than the experts needed, leaving room for exploration and mistakes. In successful cardiac MR runs, the strongest open-weight agent used only 25% of the available steps at the median. Agents that exhausted the budget were usually repeating actions or submitting actions that the interface rejected, rather than steadily approaching a solution.

Scoring thresholds. The imaging experts set the tolerances based on their clinical experience, and the same tolerances apply to every model. To test whether these choices drive the results, we relaxed the tolerances on all recorded cardiac MR episodes and rescored them. Even with tolerances relaxed by 10x, for example, the success rate of Qwen3.8-27B rose only from 79.1% to 84.3%, and that of UI-Mate-27B from 36.7% to 43.0%, also that the ranking between all agents preserved.

AuthorsOpenReview ↗

Official Comment by Authors (2/3)

2. GUI interaction, image understanding and acquisition knowledge (Q2 & W2)

We agree that the predefined tasks combine the three abilities. This is intentional, because a medical imaging scan requires the agent to understand the anatomy shown on screen and operate the scanner console. We also agree that evaluating each ability separately is important for understanding where the agents struggle. Therefore, we conducted an exploratory experiment on a single task following the reviewer's suggestion.

Experiment Design. On the L4 renal coverage-correction task (30 cases: 10 anatomies ×\times 3 variations; 23 with a coverage defect and 7 clean controls that need no change), all 13 agents ran a paired 2 ×\times 2 design.

  • Kidney contours (C) draw the left and right kidney outlines from the hidden masks on every displayed image, as the reviewer suggested.
  • Simplified numeric input (N) replaces clicking and typing with a single action. The agent still chooses the field and the value.

We kept the scanner console, image-only interface, task budgets, and scoring method unchanged. In order to isolate the medical image understanding, the contours can only help if the agent successfully operates the console, adjusts the scan prescription, and acquires a new image series. We therefore first checked whether each agent could complete these steps.

Table 1. Renal MR L4 coverage correction under the four conditions. Orig.: original task; C: kidney contours; N: simplified numeric input; C+N: both. The seven agents not shown are blocked earlier by GUI operation, with detailed analysis in findings.

----------------------------------------------------------------------------
                             SR (%)                        AQ (%)
                  ----------------------------  ----------------------------
Model              Orig.    C      N     C+N     Orig.    C      N     C+N
----------------------------------------------------------------------------
Claude Opus 5.5      67     87     57     90       97    100     95    100
GPT-6 Sol            47     45     32     50       77     96     93     98
Qwen3.8-27B          41     60     45     67       88     93     82     93
Gemma-4-31B           3      3      0      7       88     86     89     84
Muse-Glimmer-30B      7      0     10     10       43     33     67     65
UI-Mate-27B           0      0      0      3       48     33     38     53
----------------------------------------------------------------------------

Findings. Different agents have different bottlenecks, and neither providing anatomical locations nor simplifying input generally solves the end-to-end task.

  • Perception limits the strongest agents. Kidney contours improved Claude Opus 5.5's SR and helped it achieve full kidney coverage. Qwen3.8-27B showed similar improvements. GPT-6 Sol improved its coverage, where the failure cases ran out of steps before completing the task.
  • Acquisition knowledge limits the mid-tier agents. Gemma-4-31B, Muse-Glimmer-30B, and UI-Mate-27B can operate the console but often adjust the wrong parameters, even with contours. For example, Gemma-4-31B selected the wrong extent field in 51 of 92 defect cases.
  • GUI operation limits the other seven agents, which are therefore not shown in Table 1. They are blocked before image understanding: they rarely complete a re-acquisition with a changed prescription (0–8 of 92 defect episodes), so neither aid can change their outcome. Their failures are GUI failures: repeated clicks on the same spot, stopping without acquiring, typing into read-only fields, and outputs that do not follow the action format. None of them passed a defect case in any condition, and their AQ is 0 or reflects a re-acquisition without the needed change.
  • Simplified numeric input is not the missing piece. It did not consistently raise SR for the agents that complete the GUI loop (even for agents that could operate the console.).

Even with both aids, 10 of the 13 agents stay at or below 23% SR in every condition, and the ranking of agents changes little under the interventions (Kendall τb\tau_b 0.77–0.92 against the original condition). The controlled experiment therefore show that improving perception or input alone helps only agents that already operate the console and choose the right correction. An agent intended to assist with real clinical acquisition needs this end-to-end ability and in the future work we plan to extend the benchmark with diagnostic conditions of this kind, measuring each ability separately.

AuthorsOpenReview ↗

Official Comment by Authors (3/3)

3. Ground-truth-based scoring clarification (W3 & Q3)

We apologize for the unclear wording in the AI-use statement. By "AI-assigned final scores" we meant that AI coding agents helped execute our deterministic scoring and aggregation scripts, not that an AI model judged episodes or assigned scores. We have revised the statement accordingly.

Scoring involves no model judgment. The scoring programs apply predefined rules to the recorded episode outcomes. SR checks the complete task requirements, PR measures progress through predefined workflow stages, and AQ measures task-specific acquisition or prescription quality against hidden ground truth. Scripts then aggregate these values. Model assistance occurred only outside episode scoring: in drafting tasks together with the experts, in writing code that the authors reviewed and tested, and in monitoring the progress of the runs.

The pipeline was validated before testing and then frozen. We developed and validated the full pipeline on a held-out digital subject (XCAT vfemale50) that is not part of any test cohort. On this subject, the human experts checked that the scorer's measurements and outputs agreed with the rendered images, the logged actions and the hidden ground truth. The rendering, task definitions, thresholds, and the scoring and aggregation code were then frozen, and every scorer version is fixed for benchmarking. All test anatomies were run with this frozen pipeline, with no per-subject or per-model adjustment. Because no score is assigned by an AI model, there is no model-versus-human agreement to report; the relevant check is this episode-by-episode verification of the scorer against ground truth.


We sincerely thank the reviewer again for your thoughtful and constructive feedback. Your comments have helped us reflect more carefully on our work and improve the manuscript. We hope our responses have addressed your concerns, and we would be happy to provide any further clarification or discussion if needed.

Thank the authors for the additional experiments. They have clarified my questions and I would like to maintain my score towards acceptance.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comment by Authors

Thank you all for the thoughtful feedback and constructive discussion.

Medical image acquisition requires professionals to connect anatomical understanding with precise console actions. As visual and GUI agents advance, MedImageOSWorld examines whether they can perform this work and where they fall short. Our central finding is that completing a console workflow does not ensure acquisition of the intended anatomy - a gap that workflow metrics alone can miss.

Our additional experiments help further explain this gap and address the reviewers’ concerns: controlled interventions probe perceptual and interaction bottlenecks, threshold sweeps preserve the main performance differences, and matched-episode comparisons retain the proprietary models’ lead. We also clarified the scoring procedures, agent interfaces, and the scope of the simulated environments.

By making these capability gaps measurable, we hope to support progress toward agents that ease the workload of imaging professionals and make high-quality imaging more accessible.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.