- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The manuscript focuses on independent mask quality assessment for language-guided segmentation. It fine-tunes Qwen2.5-VL via LoRA to predict the ground-truth IoU from the original image, the referring expression, and a mask-overlay image, and introduces the proportion of mask-internal attention from selected attention heads as a prior. The model is trained on SAM3-I predictions over RefCOCO together with synthetic errors, and its quality estimation and accept/reject performance are validated on SAM3-I, X-SAM, and ReasonSeg, with additional text-substitution experiments examining language dependence.
Critical strengths
The research clearly distinguishes geometric mask quality from linguistic referring consistency. The cross-generator evaluation and the text-substitution experiments with fixed images and masks provide empirical support for the auditor's use of linguistic information.
Critical weaknesses
-
Section 2 uses a 0.5 threshold on predicted IoU to decide acceptance or rejection, which has clear implications for an auditor trained under IoU supervision. However, Appendix A.2.2 directly uses the generator's native confidence, CLIP cosine similarity, and PMass without stating that they were IoU-calibrated. In Table 1, SAM3-I native confidence accepts all samples, while CLIP rejects all samples. This primarily exposes a mismatch between score scale and threshold; it cannot directly demonstrate that these signals lack validation ability. Table 4 also shows that SAM3-I native confidence achieves an AUROC of 0.809, indicating considerable ranking ability. A fair comparison should add in-training-set calibration, threshold selection on an independent validation set, and the false-acceptance rate at an equal rate of correct-mask retention. Table 4 does support that the auditor provides an improvement, but it cannot eliminate the risk that Tables 1–3 exaggerate the magnitude of the improvement; moreover, CLIP similarity does not appear in the AUROC table.
-
The manuscript already notes that MQA-RefAVS has proposed candidate-mask quality assessment without ground-truth masks at inference time, IoU prediction, and training an MLLM auditor through supervised instruction fine-tuning. Therefore, "separating segmentation generation from quality auditing" and "the MLLM predicting the IoU corresponding to a ground-truth-invisible mask" are not in themselves novel contributions of this paper. The differences of this paper lie mainly in the image domain, cross-generator/cross-dataset evaluation, and the PMass prior; however, the related work only briefly states the difference in domain without adequately explaining the substantive methodological differences, and it provides neither an adapted direct comparison nor an explanation of why such a comparison cannot be made.
-
Table 5 simultaneously changes the number of correct samples, the class ratio, the number of synthetic errors, and the overall training scale. Therefore, it cannot separately answer whether the benefit comes from class rebalancing or from specific synthetic errors. Moreover, the filtered training set yields decreased Macro F1 in both RefCOCO settings; the authors do acknowledge this trade-off, which is worth retaining — but it indicates that the strategy mainly shifts acceptance/rejection preference and cannot simply be regarded as an overall performance improvement. The supervised CLIP regressor uses the original training set while the auditor uses the filtered training set, which also adds confounding to the comparison. The attention prior lacks controls such as random scalars, mask area, or random attention heads, so the possible role of additional features or area correlations cannot be ruled out. The appendix's judgments that "aggregating content words is best" and "adding attention heads brings no benefit" are not accompanied by corresponding quantitative results.
-
The cross-generator experiments cover only one unseen generator, X-SAM; the training source is only SAM3-I, and the auditor backbone is only Qwen2.5-VL. The evidence supports that "transfer occurs in these test settings," which is not yet sufficient to generalize to broad generator independence or robust deployment capability. In particular, Table 5 shows that the base auditor's Recall on the ReasonSeg test set is only 0.608, meaning it rejects 39.2% of correct masks. Combined with the bad-mask proportion of 46.47% and Specificity of 0.818 in Table 3, and assuming the same evaluation set and global statistics, roughly 20.6% of its accepted samples would still be bad masks. The 18.2% in Table 3 is instead "the proportion of bad masks that are accepted," and the two meanings differ. The paper does not provide a risk–coverage curve, the residual error rate of the accepted set, or actual downstream-task experiments; therefore, the deployment value of the "reliability layer" still lacks direct verification. IoU ≥ 0.5 is also merely the operational standard adopted in this paper, not a guarantee of the safety of downstream actions.
Questions for authors
Details can be found in the Critical Weaknesses section.