- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces MedCounterSeg-Bench, a benchmark comprising 6,018 supported–unsupported query pairs across six medical imaging modalities and five mismatch levels. The benchmark distinguishes concept recognition, target-image support decisions, and segmentation quality through paired queries and positive concept references. The authors further propose MedCounterSeg-RL, which learns segmentation and deferral jointly through reference-based visual reasoning and verification rewards. Experiments show improved overall paired accuracy over existing rejection-enabled methods, while revealing substantial challenges in fine-grained medical concept discrimination.
Critical strengths
Clear problem decomposition and meaningful benchmark design. The distinction between recognizing a medical concept and determining its presence in a particular image is well motivated. The paired-query protocol, positive concept references, and graded mismatch taxonomy provide a more informative evaluation framework than conventional rejection accuracy.
Dedicated method design aligned with the problem. MedCounterSeg-RL explicitly connects concept verification with segmentation and deferral. The verification and negative-mask rewards are well motivated, and the ablations provide useful evidence for their respective contributions.
Critical weaknesses
Severe benchmark imbalance and insufficient fine-grained evaluation. Levels 1–2 account for 75% of the benchmark, while Levels 4–5 contain only 181 and 201 pairs across all splits, with approximately 17 and 19 pairs in the in-distribution test set. The proposed method achieves 96.1%/95.2% paired accuracy on Levels 1–2 but only 23.5%/5.3% on Levels 4–5, underperforming SESAME and MedVeriSeg on the latter. Consequently, the overall improvement is dominated by coarse mismatches, while the most challenging distinctions remain insufficiently evaluated.
Performance claims and methodological contributions need stronger validation. The reported segmentation quality is close to ConceptSeg-R1 on accepted positive queries, but accounting for false deferrals reveals Dice reductions of 5.66 points on Test and 15.09 points on OOD. OOD paired accuracy also varies substantially across backbones (52.50% vs. 7.50%), while verification IoU remains relatively low (25.25%). Moreover, the proposed method uses additional reference images unavailable to rejection-enabled baselines, and the current ablation removes references and structured reasoning simultaneously. These factors make it difficult to isolate the contribution of the proposed reasoning mechanism.
Questions for authors
-
Benchmark imbalance and fine-grained evaluation. What motivates the severe imbalance across mismatch levels? Given the limited L4–L5 test samples and the underperformance against existing baselines, how well does the benchmark support conclusions about fine-grained concept discrimination? Macro-averaged results and confidence intervals would provide a more balanced assessment.
-
Performance and generalization. How should the substantial segmentation performance degradation after accounting for false deferrals be interpreted, particularly on OOD data? What explains the large OOD performance gap between backbones? Furthermore, does the relatively low verification IoU (25.25%) provide sufficient evidence of concept recognition during deferral?
-
Reference images and reasoning. A reference-augmented comparison with SAM3-Agent/MedSAM3-Agent would help assess the contribution of additional visual information. More importantly, what is the independent contribution of explicit structured reasoning? An ablation retaining reference images and verification supervision while removing the reasoning fields would help clarify this.
-
Training details. Please clarify the baseline training protocols, particularly the lack of target-domain adaptation for SESAME, and the composition of the 7,709 Stage I positive samples.