- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper formulates ultrasound examination as a POMDP in which an agent sequentially chooses acquisition actions, observes action-dependent evidence, decides when to stop, and generates structured and free-text reports. It introduces SONOARENA, a symbolic interactive benchmark derived from structured ultrasound reports plus simulated acoustic-access and observation events, provides analyses of when open-loop versus adaptive acquisition policies are favorable, and evaluates programmatic, general-purpose LLM, and task-trained LLM policies.
Critical strengths
- The paper identifies an important distinction that is often flattened in medical-AI evaluation: interpreting a fixed acquired image set is not equivalent to deciding what information should be acquired before reporting. The formulation explicitly separates latent pathology from window adequacy , making the key issue of “absence of evidence versus evidence of absence” operational rather than rhetorical.
- The benchmark design separates trajectory quality, structured verdict quality, and free-text report faithfulness.
- The theoretical analysis is more substantive than a generic invocation of POMDPs. Proposition 1 establishes a clear sufficient regime in which an open-loop protocol can be near-optimal, while Proposition 2 formalizes a branch-point setting where observation-conditioned acquisition has value.
- The empirical results broadly support the benchmark’s intended diagnosis.
Critical weaknesses
- The paper’s empirical conclusions concern a highly constructed symbolic environment, not ultrasound acquisition in the usual visual or physical sense.
- Equation 1 defines , but is not defined as carefully as , and the concrete utility matrix, action costs, reporting thresholds, family-specific finding sets, and budget choices are not available in the main text. Since SonoScore normalizes each episode using an optimistic analytical clairvoyant relaxation, its absolute scale is not directly interpretable as clinical utility or as proximity to the achievable POMDP optimum. This matters because the gap between the executable clairvoyant and the 100-point ceiling can be substantial. The main paper should include the actual default utility table, , budgets, key observation probabilities, and a concise definition of the per-case normalization. It should also report unnormalized returns and action costs alongside SonoScore.
- Section 2.3 states that an LLM attributor parses free-text reports into claims before deterministic checking. Thus, unsupported-negative, contradiction, and undeclared-claim metrics in Table 2 depend on a frozen LLM attribution model, even if the downstream rule checking is deterministic. This is especially important because Figure 3(b) and several headline conclusions rely on those report-level metrics. The paper should report the attributor’s precision and recall on real agent-generated reports with clinician annotation, quantify sensitivity to the choice of attributor, and avoid characterizing the full report scorer as having “no LLM-as-judge.”
Questions for authors
- What exact distribution is used in the expectation in HYDRA’s EVOI expression, and how is the cost of an -look plan computed when a sign is observed before look ? Please provide pseudocode for belief updating, plan evaluation, and report generation.
- Can the authors quantify the gap between HYDRA and a stronger approximate POMDP planner on small instances where exact belief-space planning is feasible? This would clarify whether HYDRA is a useful teacher because it is close to optimal, or primarily because it supplies a structured heuristic.
- Do the authors intend SONOARENA to evaluate examination-level planning only, or to support claims about autonomous ultrasound examination more broadly? If the latter, what interfaces are envisioned for coupling this benchmark to image perception, image-quality assessment, and low-level robotic probe control?
5 comments
(2/4)
2. Evaluation validity: SonoScore and report metrics (W2, W3)
W2: Thank you for these constructive suggestions. As you highlighted in Strength 2, SonoArena separates trajectory quality, structured verdict quality, and free-text report faithfulness. Within this framework, SonoScore evaluates sequential examination and diagnostic decisions under explicit assumptions about diagnostic outcomes and acquisition costs. It is a configurable, decision-theoretic evaluation metric rather than an absolute measure of clinical utility. We agree that its underlying assumptions and normalization procedure must be clearly specified to ensure transparent and reproducible evaluation.
Evaluation assumptions and transparency. To address your specific concerns, we have clarified the evaluation definitions, parameters, and normalization procedure in the revised manuscript. The table below summarizes the corresponding changes and their locations.
| Your concern | Revision | Location |
|---|---|---|
| in Eq. 1 | Defined and clarified how it and enter the utility function . | §2.1, lines 167–173 |
| Utility matrix | Added default values in new Table 1 and the complete matrix in new Table 6. | §2.1, Table 1 (lines 169–178); App. B.2 |
| , budgets, observation probabilities | Specified all three in new Table 1, including per-family settings. | §2.1, Table 1 (lines 173–179); App. B.6 |
| SonoScore normalization | Clarified SonoScore as computed from family-mean returns rather than a per-episode normalization. | §4, lines 376–382 |
| Attainable optimum | Added exact-planning analysis on 300 small instances. | new App. G.2 |
| Clairvoyant bound | Clarified restricted to policies that report PRESENT only after a sign. | App. B.10 |
| Unnormalised returns and action costs | Added unnormalised returns and action costs for all 28 policies. | new App. G.1 |
Unnormalised returns and action costs. To make the underlying policy performance transparent, we now report unnormalised returns and action costs for all 28 policies in Appendix G.1. The table below provides representative results, with mean per-episode returns and action costs in parentheses.
| thyroid | breast | liver | focused emergency | |
|---|---|---|---|---|
| analytical bound | 39.6 | 30.6 | 22.2 | 26.0 |
| executable clairvoyant | 27.2 (3.5) | 23.2 (2.8) | 21.0 (2.7) | 16.4 (3.5) |
| HYDRA-Teacher | 17.9 (3.8) | 13.5 (3.2) | 19.8 (3.0) | 9.3 (4.8) |
| SFT→GRPO | 17.9 (3.8) | 14.5 (3.1) | 14.3 (2.9) | −0.4 (4.6) |
| re-look + abstain | 14.9 (3.8) | 14.9 (2.9) | 11.1 (2.4) | −7.9 (2.9) |
| fixed protocol | −2.9 (2.6) | 2.0 (2.0) | 9.3 (2.1) | −10.3 (2.4) |
We can see that the return scale differs across families, while the main comparative conclusions hold in general. In our original design of SonoScore, we therefore normalized the returns to enable comparisons across families, so that averaging the scores would reflect overall performance across families. SonoScore should be understood as a comparison score between policies rather than an absolute clinical utility rating.
Gaps between the ceiling and attainable optimum. The 100-point ceiling is the analytical bound , a theoretical maximum rather than an attainable target. It assumes that the hidden state is known and that the first look at each chosen window gives a usable plane and reveals every finding (Appendix B.10). The executable clairvoyant knows the hidden state but still acts through the noisy gateway. It must re-scan when scanning fails, and it also spends time on measurement, detailed inspection and image capture. It is a heuristic reference rather than the optimal policy given the hidden state, so it stays below the theoretical bound.
(3/4)
W3: Thank you for pointing this out. We agree that the role of the LLM attributor and its impact on report-based metrics should be made explicit.
Role of the LLM attributor and metric dependence.
We use an LLM attributor because free-text reports must first be parsed into window-level claims before deterministic checking can be applied.
The attributor is fixed across all agents so that every report is processed consistently.
We agree that this means the complete report-scoring pipeline is not free of LLM dependence.
We have therefore removed the “no LLM-as-judge” characterization and revised the AI Use and Reproducibility Statements and Sec.4 to clearly distinguish the attributor-dependent metrics from those who don't (lines 390–392, 544, and 558).
Sensitivity to the choice of attributor.
To quantify sensitivity to the attributor, we compared four attributors on synthetic reports with known answers, using the calibration protocol of Appendix B.7 (details in new Appendix G.7).
The table below gives their precision and recall (%):
| attributor | precision | recall | thyroid P / R | breast P / R | liver P / R | focused emergency P / R |
|---|---|---|---|---|---|---|
| gpt-4o-mini | 98.3 | 73.0 | 94.4 / 65.6 | 98.9 / 48.6 | 99.7 / 79.4 | 99.4 / 95.7 |
| gpt-6-luna | 97.6 | 87.3 | 95.5 / 79.0 | 96.4 / 73.1 | 98.5 / 93.1 | 99.3 / 99.8 |
| gemini-3.8-flash | 97.5 | 87.0 | 95.4 / 79.0 | 96.3 / 76.1 | 98.5 / 91.0 | 99.1 / 99.8 |
| claude-sonnet-4-6 | 97.2 | 85.7 | 95.2 / 77.2 | 95.5 / 72.2 | 98.5 / 90.5 | 98.5 / 99.8 |
We can see that precision is at least 97% for every attributor, and the three newer models have higher recall than gpt-4o-mini. We also re-attributed the generated reports with two of these models and compared policy rankings: individual middle ranks move, but the strongest and weakest groups remain broadly consistent (Appendix G.7). Thus, the conclusions drawn from Table 3 hold under every attributor tested. To facilitate future comparisons, we also updated Table 3 and Figure 3(b) using gpt-6-luna as the attributor (line 544).
(1/4)
Thank you for recognising the distinction between interpreting acquired images and deciding what evidence to acquire, and for your positive assessment of the theoretical analysis. We organise our response around paper positioning (W1, Q3), addressed through further scope clarification, and evaluation validity (W2–W3), addressed through explicit metric definitions, unnormalised returns, utility sensitivity and attributor comparisons. We then answer Q1–Q2 with HYDRA pseudocode and comparisons against exact planning and POMCP. Please look for the revisions highlighted in
purpleandbluein the manuscript where we updated to address your concerns.1. Paper positioning: Benchmark scope (W1, Q3)
W1, Q3: Thank you for raising this important question. As you highlighted in Strength 1, “interpreting a fixed acquired image set is not equivalent to deciding what information should be acquired before reporting.” This distinction is precisely the motivation for our work. While conventional medical imaging evaluation primarily focuses on interpreting already acquired images, sequential diagnostic decision-making requires an agent to determine what evidence to acquire, how to distinguish “absence of evidence” from “evidence of absence”, and when sufficient evidence has been collected to support a diagnostic report. We believe this is a critical gap and fundamental problem in medical AI research that deserves independent and systematic evaluation.
To evaluate examination-level planning independently of visual perception and physical control, we provide every policy with the same standardized interpretation of each acquired image. This design serves exactly the purpose you highlighted in Strength 1: making the key issue of "absence of evidence versus evidence of absence" operational rather than rhetorical. Full autonomous examination also requires image perception, image-quality assessment and probe guidance, together with their interaction with decisions; these are beyond the scope of the current study. We have clarified this scope in the manuscript (lines 104–105 and 517–518).
To assess the robustness of this abstraction, we additionally simulated fallible observation from vision systems and alternative obstacle mechanisms as sensitivity analysis (Appendices G.4 and G.5). Among these tests, policy orderings and regime findings were largely stable. Under the default utility, the top four policies retain their order with the tested fallible detectors. Across the alternative obstacle mechanisms, the adaptive advantage remains small on thyroid and breast and large on liver and focused emergency.
To address how SonoArena could be integrated into a complete autonomous ultrasound system, we identify potential interfaces between its symbolic actions and observations and lower-level ultrasound components. Specifically, symbolic observations could be supplied by image perception and quality-assessment models, scan(w) could be mapped to robotic probe-control commands, and mode(μ) and capture(w) could interface with ultrasound-device controls. Developing and validating these interfaces, including their interactions with sequential decision-making, remains an important direction for future work and is beyond the scope of current study. We have updated the manuscript to further clarify this (lines 104–105 and 517–518, and Table 5 in Appendix A).