- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether free-text medical segmentation models select the intended clinical concept when related targets coexist in the same image. It introduces Clinical Contrast Sets, which changes the text prompt while holding the image fixed and measures whether predictions switch toward the requested target. Experiments with Medical-SAM3 and VoxTell show that prompts requesting non-MS white-matter hyperintensities often select MS lesions instead. The study includes wording and size controls, anatomical and tumor contrasts, confidence analysis, and confirmation on a held-out 20-patient cohort.
Critical strengths
-
Keeping the image fixed while changing the requested concept helps distinguish target-selection errors from generic segmentation inaccuracies. The wording, volume, and component-size controls strengthen the central observation.
-
Independent confirmation of the observed failure. The official-test protocol was frozen before evaluation, and the 20-patient cohort reproduces the development-set pattern. Reporting unsuccessful mitigation and limitations of the failure detector also helps establish the boundaries of the findings.
Critical weaknesses
-
Limited novelty and scientific insight. The paper carefully documents a concept-selection failure, but the resulting insight appears incremental relative to known limitations in semantic grounding, prompt sensitivity, and confident incorrect predictions. The controls strengthen the observation without establishing a generalizable mechanism or explaining which conditions determine successful concept selection. The contrast-set protocol is useful, but its methodological advance and the depth of the resulting discoveries remain limited.
-
Narrow empirical scope and unresolved interpretation. The strongest failure evidence concerns one annotation-defined distinction in MS3SEG. The kidney and tumor experiments mainly establish successful control cases rather than independently replicating the failure across other clinical distinctions. Furthermore, the presence of “white matter hyperintensities” in VoxTell’s training inventory does not establish alignment with MS3SEG’s specific nWMH annotation convention. The current evidence therefore does not adequately distinguish a broader semantic-grounding limitation from a dataset-specific mismatch in concept definitions.
Questions for authors
See weakness
Thank you for asking what is learned beyond "category errors occur" and which conditions determine selection. Revised text is blue and tagged [YyGW] (Section 5.7, p. 9; Table 16; Appendix P; Section 6; Limitations); Table 10 marks every new analysis as post hoc or exploratory.
W1: Insight, and which conditions determine success
1. A mask can respond strongly without selecting the requested target. VoxTell v1.1 changes 41% of the canonical masks' union when the concept changes (repeat-inference Dice 0.9997), yet the paired preference shift S is -0.033 [-0.050, -0.016] and no official patient has a positive nWMH direction. v1.0 shifts positively in 18/20 cases but still has 0/20 pairs. The protocol separates responsiveness, progress toward the target and preferential selection.
2. What separates success from failure. Kidney laterality: 40/40 pairs, S +1.90 (40/40 positive). BraTS: S +0.74 [0.59, 0.89] (38/40 positive), 19/40 pairs; 18 of 21 failures are enhancing-prompt masks that segment the enhancing tumor well (median Dice 0.97) but also cover the small embedded core (median 72 voxels): high enhancing-tumor Dice can conceal coverage of the competing non-enhancing core, which our pair criterion exposes. MS3SEG: S -0.033, 0/20. Six co-labeled CT contrasts give 233/240, broadening competence checks (Appendix P).
3. A matched intervention identifies a measured contributor. A rank-16 text adapter on frozen VoxTell v1.1 is trained on development patients; all arms use the same patients, seeds 0-2, exactly 1,200 updates and the same patch rule.
These are exploratory results on previously opened patients, not a universal rule.
4. Target-dataset recovery can conceal collateral failure. The first adapter recovers MS3SEG (57/60) but reduces unrelated BraTS pair success to 7/120. An embedding-retention anchor then gives 58/60 MS3SEG, 120/120 kidney and 64/120 BraTS pairs (19/40 frozen). The protocol shows where a repair damages another distinction. The anchor followed BraTS feedback, so this retention evaluation is exploratory (Tables 4 and 15).
W2: Definition mismatch and empirical scope
We agree that generic WMH in a training inventory does not establish MS3SEG's nWMH convention, and withdraw "not a missing label." The documented WMH source labels presumed vascular WMH; equivalence to MS3SEG's category is unknown, and v1.1 has unlisted additions. Absent distinction-specific supervision and a convention mismatch both remain viable; adaptation shows learnability but does not separate them.
The observation remains an operationally meaningful failure: the model accepts the clinical request and returns a confident mask covering its competitor. It persists on the frozen 20-patient confirmation, across both VoxTell versions, and within jointly size/intensity/ventricular-distance-matched components (-0.211 [-0.335, -0.092]). The strongest evidence is VoxTell on MS3SEG, with Medical-SAM3 corroborating. An independently specified second clinical distinction is the natural next test of breadth (Section 7).
Novelty relative to known limitations
We hope these experiments address your concern. We would be glad to address any remaining questions.