acceptodds
Under review as a conference paper at ICLR 2027

SonoArena: Formulating and Benchmarking Agentic Decision-Making in Ultrasound Examination

Abstract

Ultrasound examination is a sequential decision-making process: a sonographer interprets the evidence acquired so far, decides where to scan next, and stops once the evidence is sufficient. Previous work on ultrasound AI has largely overlooked this decision loop. In this work, we formulate ultrasound examination as a partially observable Markov decision process (POMDP), in which diagnostic evidence is revealed only when the corresponding acquisition action is taken. Building on this formulation, we introduce SonoArena, an interactive benchmark in which ultrasound agents sequentially acquire diagnostic evidence and decide what to report. We theoretically characterize the operating regimes of different policy types, which are then supported by the empirical results. We benchmark 24 large language model (LLM)-based policy configurations, spanning general-purpose and task-trained policies. The strongest task-trained policy outperforms the evaluated baselines overall but remains below the reference, leaving room for more targeted training methods. These results demonstrate the necessity and feasibility of evaluating and training ultrasound agents not only by how well they interpret acquired observations, but also by whether they make the right acquisition decisions to support their conclusions. Code is provided in the supplementary material.

Primary area:
datasets and benchmarks
Keywords:
ultrasound imaging, agentic medical AI, LLM agents, interactive benchmark

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_h98wOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper formulates ultrasound examination as a POMDP in which an agent sequentially chooses acquisition actions, observes action-dependent evidence, decides when to stop, and generates structured and free-text reports. It introduces SONOARENA, a symbolic interactive benchmark derived from structured ultrasound reports plus simulated acoustic-access and observation events, provides analyses of when open-loop versus adaptive acquisition policies are favorable, and evaluates programmatic, general-purpose LLM, and task-trained LLM policies.

Critical strengths
  1. The paper identifies an important distinction that is often flattened in medical-AI evaluation: interpreting a fixed acquired image set is not equivalent to deciding what information should be acquired before reporting. The formulation explicitly separates latent pathology yy from window adequacy Ω\Omega, making the key issue of “absence of evidence versus evidence of absence” operational rather than rhetorical.
  2. The benchmark design separates trajectory quality, structured verdict quality, and free-text report faithfulness.
  3. The theoretical analysis is more substantive than a generic invocation of POMDPs. Proposition 1 establishes a clear sufficient regime in which an open-loop protocol can be near-optimal, while Proposition 2 formalizes a branch-point setting where observation-conditioned acquisition has value.
  4. The empirical results broadly support the benchmark’s intended diagnosis.
Critical weaknesses
  1. The paper’s empirical conclusions concern a highly constructed symbolic environment, not ultrasound acquisition in the usual visual or physical sense.
  2. Equation 1 defines R(s,τ,r^)=∑iu(r^i,yi,Ωi,hT)−λ∑a∈τc(a)R(s,\tau,\hat r)=\sum_i u(\hat r_i,y_i,\Omega_i,h_T)-\lambda\sum_{a\in\tau}c(a), but Ωi\Omega_i is not defined as carefully as Ωwi\Omega_{wi}, and the concrete utility matrix, action costs, reporting thresholds, family-specific finding sets, and budget choices are not available in the main text. Since SonoScore normalizes each episode using an optimistic analytical clairvoyant relaxation, its absolute scale is not directly interpretable as clinical utility or as proximity to the achievable POMDP optimum. This matters because the gap between the executable clairvoyant and the 100-point ceiling can be substantial. The main paper should include the actual default utility table, λ\lambda, budgets, key observation probabilities, and a concise definition of the per-case normalization. It should also report unnormalized returns and action costs alongside SonoScore.
  3. Section 2.3 states that an LLM attributor parses free-text reports into claims before deterministic checking. Thus, unsupported-negative, contradiction, and undeclared-claim metrics in Table 2 depend on a frozen LLM attribution model, even if the downstream rule checking is deterministic. This is especially important because Figure 3(b) and several headline conclusions rely on those report-level metrics. The paper should report the attributor’s precision and recall on real agent-generated reports with clinician annotation, quantify sensitivity to the choice of attributor, and avoid characterizing the full report scorer as having “no LLM-as-judge.”
Questions for authors
  1. What exact distribution is used in the expectation in HYDRA’s EVOI expression, and how is the cost of an mm-look plan computed when a sign is observed before look mm? Please provide pseudocode for belief updating, plan evaluation, and report generation.
  2. Can the authors quantify the gap between HYDRA and a stronger approximate POMDP planner on small instances where exact belief-space planning is feasible? This would clarify whether HYDRA is a useful teacher because it is close to optimal, or primarily because it supplies a structured heuristic.
  3. Do the authors intend SONOARENA to evaluate examination-level planning only, or to support claims about autonomous ultrasound examination more broadly? If the latter, what interfaces are envisioned for coupling this benchmark to image perception, image-quality assessment, and low-level robotic probe control?
5 comments

Thank you for recognising the distinction between interpreting acquired images and deciding what evidence to acquire, and for your positive assessment of the theoretical analysis. We organise our response around paper positioning (W1, Q3), addressed through further scope clarification, and evaluation validity (W2–W3), addressed through explicit metric definitions, unnormalised returns, utility sensitivity and attributor comparisons. We then answer Q1–Q2 with HYDRA pseudocode and comparisons against exact planning and POMCP. Please look for the revisions highlighted in purple and blue in the manuscript where we updated to address your concerns.


1. Paper positioning: Benchmark scope (W1, Q3)

W1, Q3: Thank you for raising this important question. As you highlighted in Strength 1, “interpreting a fixed acquired image set is not equivalent to deciding what information should be acquired before reporting.” This distinction is precisely the motivation for our work. While conventional medical imaging evaluation primarily focuses on interpreting already acquired images, sequential diagnostic decision-making requires an agent to determine what evidence to acquire, how to distinguish “absence of evidence” from “evidence of absence”, and when sufficient evidence has been collected to support a diagnostic report. We believe this is a critical gap and fundamental problem in medical AI research that deserves independent and systematic evaluation.

To evaluate examination-level planning independently of visual perception and physical control, we provide every policy with the same standardized interpretation of each acquired image. This design serves exactly the purpose you highlighted in Strength 1: making the key issue of "absence of evidence versus evidence of absence" operational rather than rhetorical. Full autonomous examination also requires image perception, image-quality assessment and probe guidance, together with their interaction with decisions; these are beyond the scope of the current study. We have clarified this scope in the manuscript (lines 104–105 and 517–518).

To assess the robustness of this abstraction, we additionally simulated fallible observation from vision systems and alternative obstacle mechanisms as sensitivity analysis (Appendices G.4 and G.5). Among these tests, policy orderings and regime findings were largely stable. Under the default utility, the top four policies retain their order with the tested fallible detectors. Across the alternative obstacle mechanisms, the adaptive advantage remains small on thyroid and breast and large on liver and focused emergency.

To address how SonoArena could be integrated into a complete autonomous ultrasound system, we identify potential interfaces between its symbolic actions and observations and lower-level ultrasound components. Specifically, symbolic observations could be supplied by image perception and quality-assessment models, scan(w) could be mapped to robotic probe-control commands, and mode(μ) and capture(w) could interface with ultrasound-device controls. Developing and validating these interfaces, including their interactions with sequential decision-making, remains an important direction for future work and is beyond the scope of current study. We have updated the manuscript to further clarify this (lines 104–105 and 517–518, and Table 5 in Appendix A).

2. Evaluation validity: SonoScore and report metrics (W2, W3)

W2: Thank you for these constructive suggestions. As you highlighted in Strength 2, SonoArena separates trajectory quality, structured verdict quality, and free-text report faithfulness. Within this framework, SonoScore evaluates sequential examination and diagnostic decisions under explicit assumptions about diagnostic outcomes and acquisition costs. It is a configurable, decision-theoretic evaluation metric rather than an absolute measure of clinical utility. We agree that its underlying assumptions and normalization procedure must be clearly specified to ensure transparent and reproducible evaluation.

Evaluation assumptions and transparency. To address your specific concerns, we have clarified the evaluation definitions, parameters, and normalization procedure in the revised manuscript. The table below summarizes the corresponding changes and their locations.

Your concernRevisionLocation
Ωi\Omega_i in Eq. 1Defined Ω⋅i=(Ωwi)w∈W\Omega_{\cdot i}=(\Omega_{wi})_{w\in W} and clarified how it and hTh_T enter the utility function uu.§2.1, lines 167–173
Utility matrixAdded default values in new Table 1 and the complete matrix in new Table 6.§2.1, Table 1 (lines 169–178); App. B.2
λ\lambda, budgets, observation probabilitiesSpecified all three in new Table 1, including per-family settings.§2.1, Table 1 (lines 173–179); App. B.6
SonoScore normalizationClarified SonoScore as computed from family-mean returns rather than a per-episode normalization.§4, lines 376–382
Attainable optimumAdded exact-planning analysis on 300 small instances.new App. G.2
Clairvoyant boundClarified Vπ≤V⋆≤VˉV^\pi\le V^\star\le\bar V restricted to policies that report PRESENT only after a sign.App. B.10
Unnormalised returns and action costsAdded unnormalised returns and action costs for all 28 policies.new App. G.1

Unnormalised returns and action costs. To make the underlying policy performance transparent, we now report unnormalised returns and action costs for all 28 policies in Appendix G.1. The table below provides representative results, with mean per-episode returns and action costs in parentheses.

thyroidbreastliverfocused emergency
analytical bound Vˉ\bar V39.630.622.226.0
executable clairvoyant27.2 (3.5)23.2 (2.8)21.0 (2.7)16.4 (3.5)
HYDRA-Teacher17.9 (3.8)13.5 (3.2)19.8 (3.0)9.3 (4.8)
SFT→GRPO17.9 (3.8)14.5 (3.1)14.3 (2.9)−0.4 (4.6)
re-look + abstain14.9 (3.8)14.9 (2.9)11.1 (2.4)−7.9 (2.9)
fixed protocol−2.9 (2.6)2.0 (2.0)9.3 (2.1)−10.3 (2.4)

We can see that the return scale differs across families, while the main comparative conclusions hold in general. In our original design of SonoScore, we therefore normalized the returns to enable comparisons across families, so that averaging the scores would reflect overall performance across families. SonoScore should be understood as a comparison score between policies rather than an absolute clinical utility rating.

Gaps between the ceiling and attainable optimum. The 100-point ceiling is the analytical bound Vˉ\bar V, a theoretical maximum rather than an attainable target. It assumes that the hidden state is known and that the first look at each chosen window gives a usable plane and reveals every finding (Appendix B.10). The executable clairvoyant knows the hidden state but still acts through the noisy gateway. It must re-scan when scanning fails, and it also spends time on measurement, detailed inspection and image capture. It is a heuristic reference rather than the optimal policy given the hidden state, so it stays below the theoretical bound.

W3: Thank you for pointing this out. We agree that the role of the LLM attributor and its impact on report-based metrics should be made explicit.

Role of the LLM attributor and metric dependence.
We use an LLM attributor because free-text reports must first be parsed into window-level claims before deterministic checking can be applied. The attributor is fixed across all agents so that every report is processed consistently. We agree that this means the complete report-scoring pipeline is not free of LLM dependence. We have therefore removed the “no LLM-as-judge” characterization and revised the AI Use and Reproducibility Statements and Sec.4 to clearly distinguish the attributor-dependent metrics from those who don't (lines 390–392, 544, and 558).

Sensitivity to the choice of attributor.
To quantify sensitivity to the attributor, we compared four attributors on synthetic reports with known answers, using the calibration protocol of Appendix B.7 (details in new Appendix G.7). The table below gives their precision and recall (%):

attributorprecisionrecallthyroid P / Rbreast P / Rliver P / Rfocused emergency P / R
gpt-4o-mini98.373.094.4 / 65.698.9 / 48.699.7 / 79.499.4 / 95.7
gpt-6-luna97.687.395.5 / 79.096.4 / 73.198.5 / 93.199.3 / 99.8
gemini-3.8-flash97.587.095.4 / 79.096.3 / 76.198.5 / 91.099.1 / 99.8
claude-sonnet-4-697.285.795.2 / 77.295.5 / 72.298.5 / 90.598.5 / 99.8

We can see that precision is at least 97% for every attributor, and the three newer models have higher recall than gpt-4o-mini. We also re-attributed the generated reports with two of these models and compared policy rankings: individual middle ranks move, but the strongest and weakest groups remain broadly consistent (Appendix G.7). Thus, the conclusions drawn from Table 3 hold under every attributor tested. To facilitate future comparisons, we also updated Table 3 and Figure 3(b) using gpt-6-luna as the attributor (line 544).

Reviewer_rfUNOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

SonoArena models ultrasound examination as a POMDP and evaluates how agents select acquisition windows, repeat acquisitions, decide when to stop, and produce evidence-supported findings and reports. The paper combines theoretical analysis with comparisons of programmatic and LLM-based policies to study when fixed protocols and adaptive acquisition are effective.

Critical strengths
  1. The benchmark takes real examination conditions into account and extends the evaluation to the whole examination process. It separates pathology from acoustic access, and “no abnormality seen” from “evidence that there is none.”

  2. The paper compares fixed and adaptive acquisition. Reacquisition with abstention works well for thyroid and breast exams, but on focused emergency tasks adaptive policies do better (Table 1). Report faithfulness and documentation quality are scored separately, so models with a correct diagnosis but an unsupported or incomplete report can be identified.

Critical weaknesses
  1. Appendix B.1 (p. 15) assumes no false positives, so one positive observation settles the diagnosis and confirming it adds almost nothing. Measurements, detailed inspection, and image capture still have a cost, but they only show up in the separate QC metrics. A higher main score doesn’t necessarily mean a more complete exam. Once an abnormality is found, the main score actually penalizes agents that keep going and finish the exam.

  2. It’s unclear how well the acquisition scenarios generalize. Thyroid and breast pathology comes from clinical reports, but acoustic obstacles and counterfactual acquisition outcomes are mostly set by the environment, and focused emergency is fully synthetic (pp. 21–22). Which policy comes out ahead, fixed or adaptive, depends on these settings. With a different set of scenarios, the conclusions could change.

Questions for authors
  1. What do the reported trajectories and QC results show about measurements, detailed inspection, and image capture after a positive finding? How do these behaviors relate to expert assessments of examination completeness?

  2. Which conclusions have been checked under alternative window relationships or acoustic-obstacle mechanisms?

  3. For Tables 1 and 2, what are the evaluated case counts and results on common case subsets, and what quantitative precision and recall were obtained in the report-attributor calibration?

5 comments

Thank you for recognising both the distinction between pathology and acoustic access and the value of assessing acquisition, report faithfulness and documentation separately. We address your comments from three perspectives: examination completeness (W1, Q1), clarifying what SonoScore measures and analysing post-positive work-up and confirmation; design considerations and robustness (W2, Q2), testing alternative acquisition mechanisms; and implementation details (Q3), clarifying case counts and report-attributor calibration. Please look for the revisions highlighted in green and blue in the manuscript where we updated to address your concerns.


1. Examination completeness: what SonoScore measures (W1, Q1)

W1, Q1: Thank you for raising this important point. As you highlighted in Strength 2, evaluating report faithfulness and documentation quality separately reveals differences that diagnostic performance alone would miss. This distinction is precisely the motivation for our evaluation design. A complete ultrasound examination involves several distinct questions: whether sufficient evidence was acquired to support the diagnostic verdicts, whether the required work-up and documentation were completed, and whether the final report remained faithful to the acquired evidence. Within this framework, SonoScore evaluates evidence acquisition and diagnostic decisions under explicit probe-time costs. It is a decision-theoretic measure of examination planning rather than a complete measure of clinical examination quality. The QC metric separately evaluates measurement, detailed inspection and image capture against the clinical quality-control checklist (Appendices B.1 and C), while unsupported-claim metrics assess report faithfulness. We believe keeping these aspects separate is important, as it makes the trade-offs between diagnostic utility, examination completeness and report faithfulness visible.

Post-positive work-up and documentation (Q1).
To assess examination completeness after a positive finding, we analysed the policies' use of measurement, detailed inspection and image storage (new Appendix G.6). The table below gives representative results (%):

policymeasureinspectimage capture (focal lesion)complete examsQCSonoScore
fixed protocol58.083.299.571.263.525.8
gpt-5.591.790.257.447.473.755.1
qwen3.8-max94.392.497.289.880.751.6
SFT→GRPO48.567.651.120.354.756.0
HYDRA57.084.415.212.363.266.8

We can see that general-purpose LLMs often perform the offered work-up but may omit image documentation, whereas return-optimised policies tend to omit work-up actions that are not rewarded by SonoScore. We further tested reserving probe time for image captures, which makes most HYDRA examinations complete at a modest SonoScore cost (Appendix G.6). Thus, QC and SonoScore act as complementary evaluation metrics: the former captures examination completeness, while the latter captures evidence acquisition and verdict utility under probe-time costs.

1. Examination completeness (continued)

Confirmation under false-positive signs (W1). Under the original α = 0 assumption, an observed positive sign is conclusive, so the model does not require confirmation. Following your concern, we further tested false-positive signs at two rates (0.05 and 0.15) and false-positive verdict costs of −1 and −10, where a re-look at the same window can serve as confirmation (new Appendix G.4). We also introduced an α-aware HYDRA that maintains a posterior after observing a sign and can explicitly plan confirmation looks.

The table below gives representative results when false-positive verdicts cost −10, including SonoScore and the proportion of false signs reported as PRESENT (Appendix G.4, Table 28):

Policyα = 0α = 0.05α = 0.15False signs reported PRESENT
α-aware HYDRA69.232.740.156%
gpt-5.539.3 (2)22.1 (1)30.4 (1)76%
gemini-3.8-flash42.7 (1)16.0 (2)23.0 (2)74%
claude-opus-529.5 (3)9.9 (3)15.4 (3)71%
qwen3.8-max18.6 (4)6.3 (4)0.3 (5)87%
gpt-5.4-mini9.7 (5)2.6 (5)1.4 (4)95%

We can see that α-aware HYDRA improves over the original HYDRA by 2.8 and 12.7 SonoScore points at α = 0.05 and 0.15, respectively (Appendix G.4, Table 26), showing that confirmation becomes valuable when false-positive evidence is costly. The relative ranking of LLMs changes little, while most LLMs still report false signs as PRESENT. LLMs may require specific training to handle uncertain observations.

Expert assessment of completeness (Q1). To answer your question, we further analyzed the relationship between expert ratings and our examination metrics. During the rebuttal, we added a second sonographer, who rated the same 40 episodes as the first (Appendix D.3). The table below gives the Spearman ρ between the two sonographers, and the Spearman ρ between each metric and the ratings, averaged over the two sonographers (Appendices D.3 and G.6):

Spearman ρAcquisitionOverall
Sonographer 1 vs sonographer 20.620.67
Regret−0.51−0.57
QC0.360.50
Coverage0.440.31
Work-up rate−0.060.00

We can see that the two sonographers' ratings agree moderately. Both give higher ratings to episodes with lower regret. QC tracks the overall rating more closely than coverage does, but the acquisition rating less closely. Work-up rate shows no clear association with either rating. However, this is intended as a preliminary human alignment check rather than clinical validation, which would be beyond the scope of the current work.

2. Design considerations and robustness: acquisition scenarios (W2, Q2)

W2, Q2: Thank you for raising this important question. As you highlighted in Strength 1, SonoArena distinguishes underlying pathology from acoustic access. This distinction is also why these two components are specified differently. Pathological findings are grounded in clinical reports, whereas acoustic access represents a latent property of the patient state that determines which evidence can be acquired and where the agent should scan next. In our benchmark, the acquisition environment specifies these contingencies so that different policies can be evaluated under the same conditions. Both the contingency-generation setup and the focused-emergency family were constructed with reference to official clinical guidelines, including those from the American Institute of Ultrasound in Medicine (§4, lines 354–356). To further assess how much our conclusions depend on the particular acquisition assumptions, we conducted additional experiments with alternative acquisition mechanisms.

Robustness to alternative acquisition mechanisms (Q2). To test whether the main findings depend on the simulated acquisition conditions, we varied the environment's acoustic-access and counterfactual-outcome assumptions. We evaluated three alternative mechanisms: correlated closure, transient obstacles, and changed window–finding relationships, covering eight shifts, four examination families, and five seeds (details in new Appendix G.5). The table below reports HYDRA's SonoScore minus that of re-look + abstain:

SettingThyroidBreastLiverFocused emergency
Control4.6−3.131.151.5
Extra closure 15% / 30%, independent or correlated1.0 to 2.6−6.0 to −2.830.6 to 32.550.0 to 52.8
Transient obstacles 15% / 30%4.9 / 3.6−3.5 / −2.025.2 / 18.544.7 / 38.5
One window dropped / added (focused emergency only)= control= control= control49.4 / 36.6

Across the tested mechanisms, the gap remains small on thyroid and breast and larger on liver and focused-emergency examinations, whose acquisition is more adaptive. Our main findings are also robust under the acquisition conditions variation.

Reviewer_sG2kOpenReview ↗
Rating
3(+1)
Confidence
1(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SonoArena, an interactive benchmark that formulates ultrasound examination as a partially observable Markov decision process (POMDP). Agents sequentially select acquisition actions, decide when to stop, and generate evidence-grounded findings and reports. The benchmark covers thyroid, breast, liver, and focused-emergency examinations, and evaluates general-purpose and task-trained LLM policies alongside programmatic references. Results highlight the value of adaptive acquisition and SFT followed by GRPO, but clinical realism and evaluation validity remain insufficiently established.

Critical strengths

1.The research questions have clear innovative value, breaking through the paradigm of static medical imaging evaluation, while also possessing a certain depth of theoretical analysis, rather than merely proposing simulators and running LLMs.

  1. The experiment covers diverse strategy types and yields valuable negative findings. The paper not only compares various general-purpose LLMs but also employs programmatic strategies as a baseline, investigating three training approaches: SFT, GRPO, and SFT→GRPO..
Critical weaknesses
  1. The benchmark relies on simulated textual observations rather than actual ultrasound images. Since acoustic-access events are generated and emergency cases are fully synthetic, its applicability to real-world ultrasound examinations remains unclear.

2.The primary SonoScore relies on a manually specified utility matrix and an optimistic case-specific normalization. For example, the default false-negative penalty is linked to an illustrative 5% biopsy threshold rather than estimated across the four clinical families. The one-sonographer review of 40 episodes shows only moderate agreement with the scorer, limiting evidence for clinical validity.

Questions for authors

1.Do the relative policy rankings persist when observations come from actual ultrasound images or a fallible visual model, rather than the simulator's standardized textual interpretations?

2.How do the results change when false-positive signs and confirmation actions are introduced into the observation model?

3.Can the authors provide clinically calibrated or task-specific utility settings, and validate SonoScore with additional independent sonographers? How stable are rankings across these settings?

5 comments

Thank you for recognising the value of the research question and theoretical analysis. We address your comments from three perspectives: paper positioning (W1), clarifying the benchmark scope; design considerations and robustness (W1, Q1–Q2), testing the simulated acquisition and observation assumptions; and evaluation validity (W2, Q3), addressed through utility sensitivity, unnormalised returns and a second sonographer's ratings. Please look for the revisions highlighted in orange and blue in the manuscript where we updated to address your concerns.


1. Paper positioning: benchmark scope (W1)

W1: Thank you for raising this important question. As you highlighted in Strength 1, our work moves beyond static medical-imaging evaluation to examine what evidence should be acquired before reporting. This distinction is precisely the motivation for our work.

While conventional medical-imaging evaluation primarily focuses on interpreting already acquired images, sequential diagnostic decision-making requires an agent to select the next acquisition based on previous findings and access constraints, distinguish “absence of evidence” from “evidence of absence”, and decide when sufficient evidence supports a diagnostic report. We believe this is a critical gap and fundamental problem in medical AI research that deserves independent and systematic evaluation.

To evaluate examination-level planning independently of visual perception and physical control, we provide every policy with the same standardized interpretation of each acquired image. This controlled abstraction allows us to compare evidence-acquisition and reporting policies under the same observation conditions.

SonoArena's current scope is therefore not intended for end-to-end ultrasound examination. A complete such system would additionally require image perception, image-quality assessment and probe guidance, together with their interaction with the agentic decisions. These components and validation on real ultrasound examinations are beyond the scope of the current study. We believe that isolating and systematically evaluating examination-level planning provides an important foundation for building and assessing such more complicated systems. We have clarified this scope in §1, §2.2 and §7 (lines 104–105, 206–207 and 517–518).

2. Design considerations and robustness (W1, Q1, Q2)

To examine whether our conclusions depend on the acquisition and observation assumptions used, we conducted further analyses using alternative acquisition conditions, fallible observation channels, and false-positive signs with confirmation actions.

Robustness to simulated acquisition conditions (W1).

For focused-emergency cases, we used clinical guidelines from AIUM as seed material (§4, lines 354–356). To test whether the main findings depend on the simulated acquisition conditions, we varied the environment's acoustic-access and counterfactual-outcome assumptions. We evaluated three alternative mechanisms: correlated closure, transient obstacles, and changed window–finding relationships, covering eight shifts, four examination families, and five seeds (details in new Appendix G.5). The table below reports HYDRA's SonoScore minus that of re-look + abstain:

SettingThyroidBreastLiverFocused emergency
Control4.6−3.131.151.5
Extra closure 15% / 30%, independent or correlated1.0 to 2.6−6.0 to −2.830.6 to 32.550.0 to 52.8
Transient obstacles 15% / 30%4.9 / 3.6−3.5 / −2.025.2 / 18.544.7 / 38.5
One window dropped / added (focused emergency only)= control= control= control49.4 / 36.6

Across the tested mechanisms, the gap remains small on thyroid and breast and larger on liver and focused-emergency examinations, whose acquisition is more adaptive. Our main findings are also robust under the acquisition conditions variation.

2. Design considerations and robustness (continued)

Fallible observation robustness (Q1).
The default benchmark uses standardized observations so that sequential decision-making can be evaluated without the additional errors of a particular visual model. To test the robustness when this assumption is relaxed, we introduced and tested two simulated fallible detectors with lesion-reveal / false-positive rates of 0.9 / 0.1 and 0.8 / 0.2 respectively in our benchmark (details in new Appendix G.4). This experiment specifically evaluates robustness to errors in the observation channel. The table below shows some examples of the result, giving SonoScore and rank:

PolicyPaper modelDetector 0.9 / 0.1Detector 0.8 / 0.2
Clairvoyant81.9 (1)83.9 (1)84.1 (1)
HYDRA62.4 (2)62.5 (2)55.0 (2)
gpt-5.554.8 (3)54.6 (3)50.6 (3)
gemini-3.8-flash48.2 (4)51.7 (4)48.8 (4)
re-look + abstain40.2 (5)41.8 (8)41.0 (6)
claude-opus-539.2 (6)47.8 (6)40.0 (7)
fixed protocol36.6 (7)46.9 (7)44.7 (5)
qwen3.8-max36.3 (8)50.6 (5)38.7 (8)
gpt-5.4-mini25.2 (9)27.3 (9)19.4 (9)

The top four policies retain their order under the default utility, although several middle ranks change. The main relative-ranking conclusion is relatively robust to the observation errors tested. Extending the evaluation to actual ultrasound images and learned perception models is an important extension towards a complete autonomous ultrasound system, but is beyond the scope of the current study.

False-positive signs and confirmation (Q2).
Under the original α = 0 assumption, a positive sign is conclusive, so the model does not require confirmation. Following your suggestion, we further tested false-positive signs at two rates (0.05 and 0.15) and false-positive costs of −1 and −10 (details in new Appendix G.4, with a pointer in Appendix B.1), where a re-look at the same window can serve as confirmation. We also introduced an α-aware HYDRA that retains a posterior after observing a sign and can explicitly plan confirmation looks. The table below gives some examples of each policy's SonoScore, rank, and the proportion of false signs reported as PRESENT:

Policy−1, α = 0−1, α = 0.05−1, α = 0.15−10, α = 0−10, α = 0.05−10, α = 0.15False signs reported PRESENT
α-aware HYDRA62.455.151.369.232.740.185% / 56% (−1 / −10)
gpt-5.554.9 (1)49.5 (1)57.0 (1)39.3 (2)22.1 (1)30.4 (1)76%
gemini-3.8-flash48.2 (2)41.1 (2)46.4 (2)42.7 (1)16.0 (2)23.0 (2)74%
claude-opus-539.2 (3)38.7 (3)41.1 (3)29.5 (3)9.9 (3)15.4 (3)71%
qwen3.8-max36.3 (4)36.4 (4)37.5 (4)18.6 (4)6.3 (4)0.3 (5)87%
gpt-5.4-mini15.5 (5)21.4 (5)17.5 (5)9.7 (5)2.6 (5)1.4 (4)95%

When false-positive verdicts cost −10, the α-aware HYDRA improves over the original HYDRA by 2.8 and 12.7 SonoScore points at α = 0.05 and 0.15, respectively, showing that confirmation becomes valuable when false-positive evidence is costly. The relative ranking of the LLMs changes little, and many LLMs still report most false signs as PRESENT. LLMs do not generally understand false-positive signs without specific training. Together, these results show that the extended benchmark can evaluate confirmation behavior and provide a basis for developing agents that can reason more reliably under uncertain observations.

3. Evaluation validity: assumptions, scale and expert alignment (W2, Q3)

Thank you for raising these important concerns. SonoScore is intended as a configurable, decision-theoretic comparison metric under explicit utility assumptions, rather than a clinically calibrated measure of patient outcomes. We therefore address evaluation validity from three complementary perspectives: sensitivity to utility choices, interpretation of the normalized score, and agreement with independent expert judgement.

Utility assumptions and sensitivity (W2, Q3).
We used 5% as a default threshold since this is a commonly used value in clinical practice. To further examine whether the reported policy comparisons depend on this choice, we examined all 28 policies under ten alternative utility matrices informed by clinical guidelines (details in new Appendix G.1). The resulting sensitivity analysis allows us to assess whether the main policy comparisons remain stable across a range of alternative utility specifications. The default utility values are now explicitly reported in §2.1, Table 1 (lines 170–177).

The table below summarizes ranking stability relative to the default utility matrix. Kendall's τ measures agreement between the policy ranking under each alternative matrix and the default ranking, with 1 indicating identical orderings:

SettingKendall's τ with default rankingBest non-privileged policy
8 moderate changes0.85–0.98SFT→GRPO in 4, glm-5.3 in 4
Family-specific0.85SFT→GRPO
Miss priced at −49 instead of −190.67re-look + abstain
False positive priced at −10 instead of −10.79claude-opus-5

The rankings remain closely aligned under moderate and family-specific changes, while larger shifts occur under more extreme variations. These results suggest that the main utility-based comparison is relatively robust under utility value changes.

Normalization and score interpretation (W2).
SonoScore is defined as 100(Rˉπ−Rˉfloor)/(Rˉceiling−Rˉfloor)100(\bar R_\pi-\bar R_{\rm floor})/(\bar R_{\rm ceiling}-\bar R_{\rm floor}), where the policy, floor and ceiling returns are family means over the same cases (§4, lines 376–382). Normalizing the SonoScore enables comparison and aggregation across different examination families with different return scales. It should therefore be interpreted as a normalized comparison across families rather than as a percentage of the attainable clinical optimum.

To make this distinction more explicit, we further clarified the gap between the analytical bound and the attainable optimum in §3.2 and §4 (lines 272–277 and 376–382). We also further report all unnormalized returns and action costs in new Appendix G.1. The table below summarizes mean per-episode returns, with action costs in parentheses, for some representative policies under the default utility matrix:

Policythyroidbreastliverfocused emergency
Analytical bound Vˉ\bar V39.630.622.226.0
Executable clairvoyant27.2 (3.5)23.2 (2.8)21.0 (2.7)16.4 (3.5)
HYDRA-Teacher17.9 (3.8)13.5 (3.2)19.8 (3.0)9.3 (4.8)
SFT→GRPO17.9 (3.8)14.5 (3.1)14.3 (2.9)−0.4 (4.6)
re-look + abstain14.9 (3.8)14.9 (2.9)11.1 (2.4)−7.9 (2.9)
Fixed protocol−2.9 (2.6)2.0 (2.0)9.3 (2.1)−10.3 (2.4)

From this table, we can see the differences in return scale and gaps to the analytical bound across examination families. Normalization was therefore needed to ensure a fair comparison across families.

Independent expert ratings (W2, Q3).
To strengthen the human-alignment analysis, a second independent sonographer rated the same 40 episodes used in the original evaluation (revised Appendix D.3). The table below summarizes the second rater's agreement with regret and the agreement between the two sonographers:

Spearman's ρAcquisitionReportOverall
Second rater vs regret−0.64−0.69−0.71
First vs second rater0.620.720.67

The magnitude of the second rater's agreement with regret is comparable to the inter-rater agreement. This provides additional evidence of agreement between our evaluation setting and expert judgement. However, this is intended as human alignment rather than actual clinical validation, which is beyond the scope of this work.


We hope these additions address your concerns and would be glad to provide any further clarification if needed. In light of these revisions, we would be grateful if you would consider revisiting your assessment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Discussion Summary

Dear ACs and Reviewers,

Thank you again for the careful reviews and constructive discussion. We appreciate the reviewers’ recognition of several key strengths of our work. First, the reviewers valued SonoArena’s formulation of ultrasound examination as sequential decision-making, which addresses a key gap in ultrasound-AI evaluation (sG2k, rfUN, h98w). Second, our theoretical analysis goes beyond a generic POMDP formulation by characterising when fixed protocols suffice and when adaptive acquisition is needed (sG2k, h98w). Third, the reviewers found our comparisons of a wide spectrum of policies extensive and informative (sG2k, rfUN, h98w). The sensitivity analyses added during the discussion further support the robustness of our evaluation (sG2k, rfUN, h98w).

Following the discussion, all three reviewers recommend acceptance. To address the questions raised during the discussion, we have provided further details on the scoring settings, the HYDRA planner and the evaluated case counts. We have also conducted robustness analyses under alternative utilities, fallible observations, false-positive signs, alternative acoustic obstacles and different report attributors, and obtained ratings from a second sonographer. We have updated the manuscript accordingly and included the new analyses in Appendix G.

We thank the AC for considering these revisions and the reviewers’ assessments in reaching a final decision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.