- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces UltraG-Bench, consolidating 40 public ultrasound segmentation datasets into three tasks: instruction-guided segmentation, evidence-grounded VQA, and evidence-grounded report generation. It also proposes UltraG-Agent, which combines a multimodal large language model with UltraSAM3. The research problem is relevant, and the dataset consolidation has potential value, but some core evaluation metrics require further validation of their definitions and implementation.
Critical strengths
1.Meaningful evaluation objective. Evaluating both semantic responses and spatial localization is important for ultrasound image understanding. Distinguishing answer accuracy from grounded accuracy is a useful design choice.
2.Substantial dataset consolidation. Standardizing heterogeneous ultrasound datasets and explicitly linking language annotations to segmentation masks could facilitate subsequent research.
3.Controlled annotation scope. Restricting generated descriptions to information supported by available annotations is a reasonable design choice, provided that these constraints are consistently enforced and label provenance is clearly documented.
4.Useful failure analysis. The appendix presents segmentation failures, target-selection errors, and difficulties involving multiple structures, helping characterize the limitations of the proposed pipeline.
Critical weaknesses
1. The report metrics assess spatial attributes but do not directly validate the predicted pixel-level evidence.
SemAcc in Appendix A.6 evaluates the location and size described in the report and therefore captures spatial-attribute correctness. However, these attributes are extracted from the report text; the metric does not directly compare predicted evidence masks with ground-truth masks. Consequently, replacing the accompanying masks while keeping the text unchanged would not affect the report-generation metrics in Table 2.
Thus, Table 2 alone is insufficient to establish improved sentence-level pixel evidence. The released implementation already includes report-mask IoU and grounding recall; these results should be reported, preferably alongside a joint semantic-and-evidence correctness metric.
2. The released VQA scoring function admits clear counterexamples, although its impact on the reported results remains unconfirmed.
The inspected code revision uses a reference-token coverage threshold for some answer types. For example, the following incorrect predictions are accepted:
| Reference answer | Predicted answer | Evaluator output |
|---|---|---|
| “There are 2 targets.” | “There are 3 targets.” | Correct |
| “left ventricle” | “right ventricle” | Correct |
| “77 x 45 pixels” | “100 x 100 pixels” | Correct |
This shows that the implementation cannot reliably distinguish some counting errors, although it does not mean that all counting errors are accepted.
3. The stated SemAcc definition and released implementation are inconsistent, complicating reconstruction of the results.
The paper defines SemAcc as a fixed average of four attributes, but the values in Tables 4 and 2 do not reconcile under this definition.
| Model | Mean of the four attributes in Table 4 | SemAcc in Table 2 |
|---|---|---|
| UltraG-Agent Q+US3 | 0.612825 | 0.5766 |
| UltraG-Agent I+US3 | 0.695600 | 0.6687 |
Questions for authors
-
Evaluation version and results: Which scoring-code revision produced Table 2? Do the main conclusions remain unchanged after correcting the VQA scoring errors and reconciling the SemAcc definition?
-
Source of the Agent’s gains: Under matched input information, how well does “UltraSAM3 + geometric computation + fixed text templates” perform? This control would help distinguish the contribution of the specialized segmentation model from that of the additional planning and generation modules.
-
Annotation validity: When no mask is available, how do you distinguish confirmed target absence from missing annotation or a not-applicable case? How many errors were identified and corrected during expert review? The paper states that two experts independently reviewed 10% of the samples from a corpus of 138,832 images. Please clarify the total review time and the average time spent reviewing each sample.
-
Experimental configuration: Please explain the model-name differences between Figure 4 and Table 2, describe how native segmentation outputs were handled, and provide the final evaluation splits together with their relationship to UltraSAM3’s training data.