acceptodds
Withdrawn from ICLR 2027

UltraG-Bench: A Multi-task Benchmark for assessing Large Vision-Language Models on Pixel-level Evidence Grounding in Ultrasound

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Ultrasound is one of the most widely used medical imaging modalities, and recent large vision-language models(VLMs) have shown increasing capabilities in ultrasound image understanding. However, these models fail to provide pixel-level visual evidence aligned with their semantic predictions, and their fine-grained grounding capability in ultrasound remains largely unclear. We introduce UltraG-Bench, a large-scale multi-task benchmark for evaluating pixel-level evidence grounding in ultrasound. UltraG-Bench is built by annotating 40 public ultrasound segmentation datasets spanning 13 anatomical categories, and comprises three progressive tasks: instruction-guided segmentation, evidence-grounded VQA, and evidence-grounded report generation, with 331125, 666779, and 138832 annotations, respectively. Comprehensive evaluation of 14 state-of-the-art models reveals a substantial gap between semantic understanding and fine-grained pixel-level localization. We further propose UltraG-Agent, which combines the semantic reasoning capabilities of a VLM with the ultrasound-specific segmentation capability of UltraSAM3. Experiments show that UltraG-Agent substantially improves both semantic prediction and pixel-level visual grounding.

Primary area:
datasets and benchmarks
Keywords:
Ultrasound, Pixel-level Evidence Grounding, Medical Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wUUeOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces UltraG-Bench, consolidating 40 public ultrasound segmentation datasets into three tasks: instruction-guided segmentation, evidence-grounded VQA, and evidence-grounded report generation. It also proposes UltraG-Agent, which combines a multimodal large language model with UltraSAM3. The research problem is relevant, and the dataset consolidation has potential value, but some core evaluation metrics require further validation of their definitions and implementation.

Critical strengths

1.Meaningful evaluation objective. Evaluating both semantic responses and spatial localization is important for ultrasound image understanding. Distinguishing answer accuracy from grounded accuracy is a useful design choice.

2.Substantial dataset consolidation. Standardizing heterogeneous ultrasound datasets and explicitly linking language annotations to segmentation masks could facilitate subsequent research.

3.Controlled annotation scope. Restricting generated descriptions to information supported by available annotations is a reasonable design choice, provided that these constraints are consistently enforced and label provenance is clearly documented.

4.Useful failure analysis. The appendix presents segmentation failures, target-selection errors, and difficulties involving multiple structures, helping characterize the limitations of the proposed pipeline.

Critical weaknesses

1. The report metrics assess spatial attributes but do not directly validate the predicted pixel-level evidence.

SemAcc in Appendix A.6 evaluates the location and size described in the report and therefore captures spatial-attribute correctness. However, these attributes are extracted from the report text; the metric does not directly compare predicted evidence masks with ground-truth masks. Consequently, replacing the accompanying masks while keeping the text unchanged would not affect the report-generation metrics in Table 2.

Thus, Table 2 alone is insufficient to establish improved sentence-level pixel evidence. The released implementation already includes report-mask IoU and grounding recall; these results should be reported, preferably alongside a joint semantic-and-evidence correctness metric.

2. The released VQA scoring function admits clear counterexamples, although its impact on the reported results remains unconfirmed.

The inspected code revision uses a reference-token coverage threshold for some answer types. For example, the following incorrect predictions are accepted:

Reference answerPredicted answerEvaluator output
“There are 2 targets.”“There are 3 targets.”Correct
“left ventricle”“right ventricle”Correct
“77 x 45 pixels”“100 x 100 pixels”Correct

This shows that the implementation cannot reliably distinguish some counting errors, although it does not mean that all counting errors are accepted.

3. The stated SemAcc definition and released implementation are inconsistent, complicating reconstruction of the results.

The paper defines SemAcc as a fixed average of four attributes, but the values in Tables 4 and 2 do not reconcile under this definition.

ModelMean of the four attributes in Table 4SemAcc in Table 2
UltraG-Agent Q+US30.6128250.5766
UltraG-Agent I+US30.6956000.6687
Questions for authors
  1. Evaluation version and results: Which scoring-code revision produced Table 2? Do the main conclusions remain unchanged after correcting the VQA scoring errors and reconciling the SemAcc definition?

  2. Source of the Agent’s gains: Under matched input information, how well does “UltraSAM3 + geometric computation + fixed text templates” perform? This control would help distinguish the contribution of the specialized segmentation model from that of the additional planning and generation modules.

  3. Annotation validity: When no mask is available, how do you distinguish confirmed target absence from missing annotation or a not-applicable case? How many errors were identified and corrected during expert review? The paper states that two experts independently reviewed 10% of the samples from a corpus of 138,832 images. Please clarify the total review time and the average time spent reviewing each sample.

  4. Experimental configuration: Please explain the model-name differences between Figure 4 and Table 2, describe how native segmentation outputs were handled, and provide the final evaluation splits together with their relationship to UltraSAM3’s training data.

Reviewer_uHfROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The manuscript introduces UltraG-Bench, a benchmark built from 40 public ultrasound segmentation datasets spanning 13 anatomical categories. It constructs instruction-guided segmentation, evidence-grounded VQA, and evidence-grounded report generation by associating generated text with existing masks. The authors also propose UltraG-Agent, which uses an MLLM to select tasks and targets, invokes UltraSAM3 for segmentation, and conditions response generation on mask-derived facts. Experiments compare general-purpose, medical, and grounded MLLMs with segmentation models and agent variants. The authors report substantial gains from UltraG-Agent, particularly in grounding and spatial description.

Critical strengths
  1. Integrating 40 ultrasound datasets across 13 anatomical categories could provide a useful resource for evaluating models across diverse organs and targets. This breadth gives the benchmark potential community value beyond an individual dataset.

  2. Requiring pixel-level evidence alongside textual predictions addresses a limitation of conventional answer-only evaluation. Explicit text-mask associations could help distinguish plausible language output from correctly localized evidence, provided the evaluation is reliable.

Critical weaknesses
  1. Limited novelty. Although UltraG-Bench introduces pixel-level evidence associations, its tasks largely rely on existing segmentation annotations, leaving its added value over U2-BENCH unclear. The methodological contribution of UltraG-Agent beyond UltraSAM3 and prior grounded reporting approaches also remains unclear.

  2. Limited clinical relevance. The generated reports resemble structured mask descriptions rather than clinical ultrasound reports.

  3. Concerns about evaluation reliability. Table 4 reports unexpectedly low location and size accuracy for many MLLMs despite relatively coarse evaluation criteria, raising concerns about potential evaluation artifacts.

Questions for authors
  1. Contribution and novelty. What additional capabilities does UltraG-Bench evaluate compared with existing benchmarks such as U2-BENCH? What are the substantive differences between UltraG-Agent and existing segmentation-based agents or grounded reporting approaches?

  2. Spatial evaluation reliability. The near-zero location and size accuracy of several MLLMs in Table 4 is unexpected given the relatively coarse evaluation criteria. Could the authors clarify how parsing failures, coordinate conversion, and missing attributes are handled, and verify whether these factors contribute to the low accuracy?

  3. Scoring consistency. Could the authors clarify the differences in SemAcc between Tables 2 and 4 (e.g., 0.6956 versus 0.6687 under equal weighting), including sample sets and aggregation procedures? Were the reported results obtained using these implementations, and would a unified scoring procedure affect the conclusions?

Reviewer_3iaeOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper provides pixel level evidence for model prediction, improves traceability, and reduces unsupported spatial declarations. It also integrates 40 ultrasound datasets into a unified multitasking benchmark, providing clear resource value for future basic research. However, benchmark annotations are largely synthetic, expert reviews are limited, and reporting tasks are closer to mask condition captions than clinical reports. The proposed agent is a pure inference orchestration of a pre trained model, without training modules or isolated algorithm contributions. Mask errors are considered authoritative errors and will not be iteratively corrected.

Critical strengths
  1. The model must not only output text answers or reports but also provide pixel-level masks supporting the conclusions. The facts in the text, such as presence, location, and size, align with the masks, transforming the output from "free generation" to "generation under evidence constraints.".
  2. The paper unifies 40 publicly available ultrasound segmentation datasets into 13 anatomical categories and establishes three progressive tasks: instruction-guided segmentation, evidence-grounded VQA, and evidence-grounded report generation. This provides a standardized data and evaluation framework for ultrasound grounding research.
Critical weaknesses
  1. UltraG-Agent did not propose new learning objectives, network structures, or training algorithms. MLLM and UltraSAM3 are both pre trained models, and the entire system is inference only.
  2. Instructions, VQA, and reports are mainly generated automatically by GPT-5.5 based on mask geometry. The so-called report generation is closer to mask conditioned captioning than clinical ultrasound reports.
  3. Manual review only covers 10% of the samples for each anatomical structure, and there are no inter rater agreements, error rates, or expert judgment statistics.
  4. The paper clearly lacks consistency among experts, which is a key gap in benchmark data.
  5. There are significant differences in annotation protocols, target definitions, and imaging conditions among the 40 datasets, but there is a lack of unified ontology, cross dataset deduplication, and patient level split explanations.
  6. The system only selects the highest score mask from UltraSAM3 candidates, without low confidence retry re-prompt、re-segment、 Multiple rounds of verification or manual handover.
Questions for authors

Please refer to 'Weaknesses'.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.