- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces MedImageOSWorld, a benchmark for evaluating GUI agents on medical image acquisition across simulated CT, MR, and ultrasound consoles. It evaluates many agents and analyzes their failure modes using workflow checks and hidden anatomical ground truth.
Critical strengths
-
The novelty is strong, addressing the underexplored problem of action-conditioned medical image acquisition.
-
The evaluation protocol is comprehensive, separating workflow completion from acquisition quality.
-
The in-depth failure analysis provides useful insights into the limitations of current GUI agents.
Critical weaknesses
-
The simulated environment has limited physical fidelity, and its validity for real-world medical image acquisition remains unclear.
-
The comparison between proprietary and open-weight models is not fully fair.
Questions for authors
-
How do proprietary and open-weight models compare when evaluated on the same set of test episodes?
-
Are there results from human baseline that can serve as a reference for evaluating agent performance?
2 comments
I appreciate the author's rebuttal. Despite some shortcomings, I acknowledge that this work is meaningful. I will maintain my rating which is positive.
Official Comment by Authors
We thank the reviewer for the constructive questions. Both concern the reference points against which agent performance is read: whether proprietary and open-weight models are compared on the same test episodes, and whether a human baseline is available.
1. The comparison between proprietary and open-weight models (Q1 & W2)
The original evaluation uses the same task definitions for all models, but the proprietary models are evaluated on only the five anatomies per task due to limited resources. We therefore restricted the results of the open-weight models to the same test episodes, matching the task, asset, condition, seed and reset, and recomputed the scores with the same scoring and aggregation procedures as in Figure 3.
As Table 1 shows, GPT-6 Sol and Claude Opus 5.5 achieve SRs of 35.6 and 34.1, respectively, compared with 24.4 for the strongest open-weight model, Qwen3.8-27B. On the matched episodes, both proprietary models remain ahead of all open-weight models on SR, PR and AQ, and the ranking of the strongest open-weight model is unchanged. The smaller proprietary-model cohort reflects the cost of these evaluations. Evaluating the two proprietary models on five subjects already cost approximately USD 3,000–4,000; extending the same evaluation to 30 subjects per task would cost roughly USD 18,000–24,000. We therefore used the fixed five-subject subset and provided the matched-episode comparison above to assess whether the main comparison holds on the same test instances.
Table 1. Performance on the proprietary models' test set, consisting of five subject cases per task. Open-weight models are evaluated on the same cases using the scoring rules in Figure 3. SR and PR are calculated across 49 tasks, while AQ is calculated across 28 applicable tasks, following Eq. 1. Results for proprietary models are taken from Figure 3. All values are reported as percentages.
2. Human baseline as a reference (Q2)
Three medical imaging experts reviewed the design of all benchmark tasks. During development, all tasks were completed manually within the specified interaction budgets on a separate digital subject that is excluded from the benchmark test cohorts. This check established that the tasks are feasible; it was not a systematic measurement of human performance across the test subjects. We have clarified this distinction in Section 4.
3. Physical fidelity of the simulated environment (W1)
Thanks for pointing out this insightful concern. We agree that the simulator’s physical fidelity is limited, and we have added this in the Limitation section more explicitly. Simulation offers three properties that are essential for benchmarking and that real scans cannot provide. First, the ground truth is known and every condition is controllable: the anatomy is specified exactly, so each episode can be scored deterministically against the hidden ground truth, without human or model judgment. Second, the evaluation is reproducible: every agent faces exactly the same cases, and any run can be repeated under identical conditions, which is not possible in real scan. Third, the cost is far lower than that of real scanning which makes it feasible to evaluate agents over a large number of episodes.
On validity for real-world acquisition, we have begun running the same agent setup on a real cardiac MR scanner. As in the benchmark, the agent observes the scanner console through screenshots and operates it with a mouse and keyboard, here to perform the scan autonomously. We have already observed successful autonomous scans. This evaluation is ongoing and is not part of this paper.
We sincerely thank the reviewer again for your thoughtful and constructive feedback. Your comments have helped us reflect more carefully on our work and improve the manuscript. We hope our responses have addressed your concerns, and we would be happy to provide any further clarification or discussion if needed.