acceptodds
Withdrawn from ICLR 2027

iFAN: Inference-Aware Learning for Plain Mask Transformers

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Query-based mask transformers assemble segmentation outputs through pixel-wise competition among query predictions of the final layer, yet this inference process is not explicitly optimized during training. We identify two key mismatches: the query with the highest probability–mask score does not necessarily produce the most accurate mask, and final-layer decoding may discard superior predictions from intermediate layers. To address these issues, we propose Inference-Aware Learning (iFAN), a general training framework for plain mask transformers. iFAN introduces Adjusted Probability-Mask Ranking (APMR), which aligns query competition with predicted mask quality and suppresses high-confidence but inaccurate competitors. We further employ Cross-Layer Self-Distillation (CLSD) to transfer stronger intermediate predictions to the final layer. The ranking and distillation objectives are training-only, while inference retains efficient final-layer decoding. Experiments on COCO, ADE20K, and Cityscapes demonstrate consistent improvements across panoptic, instance, and semantic segmentation, as well as across different architectures, backbone scales, and input resolutions. Averaged equally over the configurations in the main result tables, iFAN improves performance by 1.16 PQ, 1.38 AP, and 0.63 mIoU, with negligible additional parameters, FLOPs and inference latency.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Image Segmentation, Mask Transformers, Inference-Aware Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_J6qaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes iFAN to address two limitations: the mismatch between query prediction scores and actual mask quality, and the potential loss of high-quality intermediate predictions in the final layer. To address these issues, the authors introduce APMR and CLSD. Extensive experiments across multiple segmentation tasks, datasets, and backbone scales demonstrate consistent performance improvements with negligible additional inference overhead. While the empirical results are promising and the proposed framework is practically useful, my primary concern is the limited methodological novelty, as the core ideas and motivations behind both components have substantial overlap with prior works such as Rank-DETR and MP-Former.

Critical strengths
  1. The paper identifies an important issue in mask transformers: the queries with the highest prediction scores do not necessarily produce the most accurate masks. However, this observation is not entirely new, as similar issues have been discussed in the object detection literature, such as Rank-DETR. In my view, transformer-based detectors and segmenters share largely similar query-based prediction mechanisms, although their output representations differ. As a result, the novelty of the relevant contributions has been compromised.

  2. The experimental evaluation is comprehensive. The authors validate their approach on two recent mask transformer architectures, EoMT and PMT, across multiple datasets and segmentation tasks. They also investigate different visual backbone scales and provide extensive ablation studies to examine the contributions of individual components and the sensitivity to key hyperparameters. These experiments provide solid empirical support for the effectiveness and generalizability of the proposed method.

Critical weaknesses
  1. The proposed APMR component has limited novelty, as its core idea closely resembles existing quality-aware ranking methods such as Rank-DETR. Its main contribution lies in adapting this idea to the pixel-wise query competition mechanism of mask transformers, rather than introducing a fundamentally new strategy.

  2. In Eq. (6) of the manuscript, the distillation loss is multiplied by the GT mask (MkM_k), meaning that background regions do not contribute to the distillation gradients. While this encourages the student to produce stronger responses within the target region, it does not explicitly penalize incorrect responses outside the target. Moreover, improving the student's adjusted probability-mask score does not necessarily lead to better mask quality (e.g., mask IoU), since the student could reduce the loss by increasing its classification confidence or predicted quality score without improving the mask itself. The authors should clarify these concerns and provide more ablations to verify the proposed distillation objective is the best.

  3. The observation that the final decoder layer does not necessarily produce the highest-quality masks has also been discussed in prior work, such as MP-Former [1]. Therefore, the motivation behind the proposed CLSD is not entirely new, which further limits the novelty of this work.

[1] Mp-former: Mask-piloted transformer for image segmentation. CVPR, 2023.

  1. Although the authors emphasize that iFAN introduces negligible additional inference overhead, its training cost is not sufficiently discussed. It would be helpful to report the additional training time and peak GPU memory usage relative to the baseline.
Questions for authors

See the Weakness (2) and (4).

Reviewer_z93AOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents iFAN, a training framework for universal image segmentation with mask transformer models. In mask transformers, learnable queries are used to make segmentation predictions after interacting with image features. The paper argues that there is a mismatch between how these models are trained and how they are used during inference: (i) the inference procedure selects the query with the highest score, but this query does not necessarily produce the most accurate mask, and (ii) outputting predictions using only the final layer’s queries may result in discarding better predictions from earlier layers. To address the first limitation, iFAN introduces APMR, a ranking loss that guides a query matched to a ground-truth object to produce higher scores than unmatched queries. For the second limitation, it employs Cross-Layer Self-Distillation (CLSD) to distill accurate predictions from earlier layers to the final layers, to improve the final layer’s predictions. Experiments show that, when iFAN is applied on top of existing efficient mask transformer models, it improves performance by modest margins (1.16 PQ and 0.63 mIoU on average).

Critical strengths
  1. The proposed method is extensively evaluated, with experiments on three image segmentation tasks (instance, semantic, and panoptic), across multiple backbone sizes (ViT-S to ViT-G), using multiple image resolutions, and in combination with two plain mask transformer models (PMT and EoMT). Across these experiments, iFAN consistently outperforms the baseline mask transformer models by modest margins, i.e., 1.16 PQ and 0.63 mIoU on average. This extensive experimentation gives insights into the effectiveness of the proposed approach.
Critical weaknesses
  1. The motivation of the paper is not sufficiently substantiated and is not sufficiently connected to the proposed method.

    1. The main premise of the paper is that mask transformers are not trained to optimize for the inference process (L012-L013), but this inference process is not explained in sufficient detail. L036 mentions that the final output is obtained by competition among query-wise prediction maps (and the paper frequently keeps referring to this ‘competition’), and L044 mentions that the class distribution and foreground probability masks are combined to determine pixel-wise segments, but the paper never explains how this ‘competition’ works or how the class and mask predictions are combined to determine pixel-wise segments. As a result, it is not clear how the inference process works exactly (for the different segmentation tasks), why the training approach of existing methods doesn’t optimize for this inference process, and why iFAN does address this. This leaves a large gap in the motivation.
    2. L046 states that the class predicted for each query by existing models may be inconsistent with the predicted mask, because the mask prediction is isolated from the class prediction. As the text subsequently argues that this leads to segmentation errors, this makes it seem like this problem will be addressed by the proposed framework. However, iFAN does not specifically address this, as the classification prediction is still decoupled from the mask prediction, leaving this unaddressed.
    3. L070-L074 states that unmatched queries may produce plausible masks for objects they were previously matched to, and that this may cause strong competition during inference, which is something the paper wants to avoid. To address this, the paper introduces a loss that guides a matched query to produce higher scores than hard negatives. However, these hard negatives are unmatched queries with high class probability and low mask overlap (L077-L078), instead of the queries with low class probability and high mask overlap that the motivation wanted to avoid. As a result, it is unclear how the method solves the identified limitation and how it connects to the motivation.
  2. Several aspects of the proposed method’s operation are not explained sufficiently clearly, and the explanations contain inconsistencies. This makes it unclear how the method works and why it solves the limitations of existing works.

    1. The paper repeatedly refers to the “probability-mask score” or “probability-mask score map” (e.g., L059, L081, L118), and mentions that iFAN uses an adjusted score map. However, the paper never explains how this score (map) is obtained for the baseline mask transformers, and only explains how the “adjusted” score map is calculated (Eq. (3)). As a result, it is not clear how it differs from the baseline, and why this is better. Moreover, related to what I stated before, it is not clear how this score map is used during inference.
    2. The explanation of the selection of hard negatives is inconsistent. L077 states that the hard negatives are unmatched queries with high class probability and low mask overlap, and L210 mentions only that they are unmatched queries with high class probability, while L233 states that they are selected according to the order of probability-mask score map S^j\hat{S}_j, which also includes the mask and quality scores. It is unclear which of these explanations is correct.
    3. The paper frequently uses technical terms without properly defining or explaining them first. A non-exhaustive list of examples: (i) L053: ““max-IoU” strategy”, (ii) L125&L142: “soft-IoU”, (iii) L142: “one-sided objective”, (iv) L209: “augmented segmentation loss”, (v) L216: “object-oriented ranking”, (vi) L654: “shared-configuration accounting”.
    4. The reported value/schedule used for λdistill\lambda_\textrm{distill} is inconsistent. L297 mentions a value of 0.4 and L377 mentions a fixed value of 0.40 as well, whereas L297 later mentions that it is epoch-dependent and has a warmup and annealing stage, and L320 doesn’t mention the warmup but a start at epoch 4, followed by linear annealing.
  3. The efficiency/compute scores seem to have been copied from the original EoMT and PMT papers, as the FPS and GFLOPs numbers are identical. This implies that the authors have not evaluated these statistics on their own hardware and have just kept the efficiency numbers the same as the baseline. This makes it unclear whether the proposed method truly preserves the efficiency of EoMT and PMT, as claimed in the L355. It is critical that these numbers are measured (under fair conditions) rather than assumed.

  4. There is a discrepancy between the batch size used for iFAN (4, per L316), and for EoMT and PMT (16). As a result, it is not clear whether the differences in performance between the models are due to the different training hyperparameters or the iFAN framework. The methods should use the same setup, or the paper should motivate why the discrepancies still allow for a fair comparison.

Because there are only modest strengths but substantial weaknesses, particularly related to the motivation and explanation but also regarding the reported numbers, I give a ‘clear rejection’ rating.

Questions for authors

To change my assessment, all 4 of the identified critical weaknesses would need to be addressed. To summarize:

  • Improve the motivation of the paper by (i) explaining in detail how inference works for mask transformers, why existing models do not optimize it, and why iFAN addresses this, and (ii) resolving all inconsistencies between the motivation and the method.
  • Improve the explanation of the method and remove inconsistencies.
  • Compute the efficiency statistics for iFAN and the baselines on the authors’ own hardware to allow for a fair assessment of the relative efficiency.
  • Compare iFAN to EoMT and PMT under fair conditions or convincingly motivate why the current comparison is fair.
Reviewer_PjW1OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes iFAN, a training framework for plain mask transformers. It addresses two issues: inference scores may favor inaccurate masks, and intermediate layers may produce better predictions than the final layer. They author implemented Adjusted Probability-Mask Ranking (APMR), which introduces quality-aware scoring and a ranking loss against unmatched competitors. Cross-Layer Self-Distillation (CLSD) selectively distills stronger intermediate predictions into the final layer. Experiments report consistent improvements across segmentation tasks while retaining final-layer inference.

Critical strengths

Consistent improvements across settings. Gains across three segmentation tasks, three datasets, and two model families support the method’s practical usefulness.

Critical weaknesses
  1. Limited methodological novelty. The framework builds on established quality prediction, pairwise ranking, and self-distillation techniques. Its novelty lies primarily in adapting and combining these ideas for plain mask transformers. The ablations support the individual components, but do not establish whether the proposed ranking formulation offers advantages over simpler alternatives such as training-free mask rescoring or quality-aware classification losses.

  2. Generally modest empirical improvements. Although improvements are consistent, several are relatively small, including 0.4–0.5 PQ on some COCO configurations and 0.3 mIoU on Cityscapes. Without repeated-run statistics, the reliability of these smaller gains remains uncertain.

Questions for authors

Questions for the Authors (iFAN)

  1. Ablation reference point. S^\hat{S}-based ranking and distillation both require the quality head q^\hat{q}. Are Tables 4(b) and 5(a) run with the quality head enabled? If so, the proper reference is the quality-only model (56.43 PQ), not the 56.00 baseline, which would make CLSD contribute only +0.11+0.11 PQ and APMR +0.25+0.25 PQ. Please specify whether quality supervision and quality-adjusted inference are enabled in every row, and include a quality-only reference wherever applicable.

  2. Diagnostic protocol (A.3). The introduction states that 64.45% of segmentation results selected by max-prob-mask differ from their max-IoU counterparts, but A.3 computes "top-query conflict" per image, comparing the single highest-scoring query with the single best-quality query, which may belong to different objects. Could you report a per-object version of this statistic?

  3. Variance and model selection. Could you report mean ±\pm std over multiple seeds for the main settings? How were checkpoints and hyperparameters selected, given that all sweeps are on COCO val, the same split used for reporting?

  4. Contribution beyond quality rescoring. To further isolate this contribution on panoptic segmentation, how does iFAN compare with (a) training-free mask-confidence rescoring and, (b) an IoU-aware classification objective adapted from Align-DETR [1] without a separate quality head? Please also clarify how the quality head differs from existing mask-quality predictors such as the IoU head in SAM [2].

  5. Scope of the architectural claim. The abstract claims improvements “across different architectures,” while the evaluated baselines are both plain-ViT, encoder-only mask transformers. Is the intended claim restricted to this model family? If broader applicability is intended, is there evidence that iFAN also benefits a decoder-based mask transformer? Otherwise, please qualify the claim to match the evaluated scope.


References

[1] Cai et al. Align-DETR: Enhancing End-to-end Object Detection with Aligned Loss. BMVC 2024.
[2] Kirillov et al. Segment Anything. ICCV 2023.
[3] Carion et al. SAM 3: Segment Anything with Concepts. ICLR 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.