- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes iFAN to address two limitations: the mismatch between query prediction scores and actual mask quality, and the potential loss of high-quality intermediate predictions in the final layer. To address these issues, the authors introduce APMR and CLSD. Extensive experiments across multiple segmentation tasks, datasets, and backbone scales demonstrate consistent performance improvements with negligible additional inference overhead. While the empirical results are promising and the proposed framework is practically useful, my primary concern is the limited methodological novelty, as the core ideas and motivations behind both components have substantial overlap with prior works such as Rank-DETR and MP-Former.
Critical strengths
-
The paper identifies an important issue in mask transformers: the queries with the highest prediction scores do not necessarily produce the most accurate masks. However, this observation is not entirely new, as similar issues have been discussed in the object detection literature, such as Rank-DETR. In my view, transformer-based detectors and segmenters share largely similar query-based prediction mechanisms, although their output representations differ. As a result, the novelty of the relevant contributions has been compromised.
-
The experimental evaluation is comprehensive. The authors validate their approach on two recent mask transformer architectures, EoMT and PMT, across multiple datasets and segmentation tasks. They also investigate different visual backbone scales and provide extensive ablation studies to examine the contributions of individual components and the sensitivity to key hyperparameters. These experiments provide solid empirical support for the effectiveness and generalizability of the proposed method.
Critical weaknesses
-
The proposed APMR component has limited novelty, as its core idea closely resembles existing quality-aware ranking methods such as Rank-DETR. Its main contribution lies in adapting this idea to the pixel-wise query competition mechanism of mask transformers, rather than introducing a fundamentally new strategy.
-
In Eq. (6) of the manuscript, the distillation loss is multiplied by the GT mask (), meaning that background regions do not contribute to the distillation gradients. While this encourages the student to produce stronger responses within the target region, it does not explicitly penalize incorrect responses outside the target. Moreover, improving the student's adjusted probability-mask score does not necessarily lead to better mask quality (e.g., mask IoU), since the student could reduce the loss by increasing its classification confidence or predicted quality score without improving the mask itself. The authors should clarify these concerns and provide more ablations to verify the proposed distillation objective is the best.
-
The observation that the final decoder layer does not necessarily produce the highest-quality masks has also been discussed in prior work, such as MP-Former [1]. Therefore, the motivation behind the proposed CLSD is not entirely new, which further limits the novelty of this work.
[1] Mp-former: Mask-piloted transformer for image segmentation. CVPR, 2023.
- Although the authors emphasize that iFAN introduces negligible additional inference overhead, its training cost is not sufficiently discussed. It would be helpful to report the additional training time and peak GPU memory usage relative to the baseline.
Questions for authors
See the Weakness (2) and (4).