- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes SFE-Guard, which combines pseudo-label weighting and selection based on frozen zero-shot evidence with constrained update correction to limit false foreground expansion during test-time adaptation for open-vocabulary segmentation.
Critical strengths
- The paper reports improvements under its evaluation protocol. The component ablations in Table 2 appear controlled and provide evidence that CDI and HRC contribute to the reported Present Dice gains.
- The paper addresses a useful problem: test-time self-training can amplify false foreground predictions when queried classes are absent. SFE-Guard combines frozen zero-shot evidence for weighting and selecting pseudo-labels (CDI) with constrained correction of proposed updates (HRC).
Critical weaknesses
- The primary metric does not directly penalize a central failure the paper aims to address. “Present Dice” averages Dice only over images in which a class is present. Consequently, false predictions of that class on absent images do not reduce the headline score. For example, perfect segmentation on one present image and an equally large false mask on an absent image would yield Present Dice of 100%, while dataset-level class IoU would be 50%. Reporting absent foreground area separately is informative, but does not establish superior overall semantic segmentation performance. Standard mIoU results are needed to determine whether the reported improvements hold under established evaluation.
- The spatial evaluation protocol departs materially from the cited methods. The submission resizes entire images to 224×224. In contrast, NACLIP uses a shorter side of 336, or 560 for Cityscapes, followed by sliding-window inference; MLMP reconstructs full-resolution Cityscapes predictions from overlapping patches. Compressing an entire street scene into a small square changes object scale and aspect ratio, potentially affecting small objects and boundaries. Although applying the same preprocessing across methods supports internal comparisons, it does not establish comparability with published segmentation benchmarks. The authors should justify this departure and evaluate the method under an established protocol.
- The task and benchmark coverage are insufficient to support the broad open-vocabulary segmentation claims. One-query binary segmentation is a reasonable diagnostic for absent-query behavior, but it merges all nonqueried categories into background and removes much of the semantic competition required for scene parsing. The paper also includes joint multi-query evaluation, but its COCO setting covers only 80 object categories. It does not evaluate full COCO-Stuff with 171 categories or ADE20K with 150 categories, despite these benchmarks being used by the underlying NACLIP framework. Since the formulation supports arbitrary class counts, the omission requires justification. Joint evaluation on these benchmarks with standard mIoU would directly test whether the method generalizes beyond the reported diagnostic settings.
- The choice of BAPA as the principal methodological foundation needs stronger justification. The concern is the narrow scope of its binary adaptation formulation. The paper should clarify why this formulation provides an appropriate foundation for broader open-vocabulary semantic segmentation, and distinguish category-conditioned binary segmentation from referring expression segmentation. It should also establish whether the observed improvements require the proposed mechanisms or can be obtained through simpler supervision weighting and conservative updates.
- The following issues should be corrected, although they are secondary to the evaluation concerns above:
- Appendix E expands CMS as “counterfactual mask score.” The metric introduced in the cited Counterfactual Segmentation Reasoning paper is Confusion Mask Score.
- Tables C.2–C.3 introduce “Senior background” and “Account,” whereas the method defines background and aggregate risks without explaining this additional terminology.
- “Credit-Denominated Issuance” uses a banking metaphor for confidence-dependent weighting and anchor selection. A more direct description would make the methodological contribution easier to assess.
- Page 24 ends with the isolated sentence, “prediction artifacts will accompany the submission,” which appears to be an unfinished editorial fragment.
Questions for authors
I think this paper requires major revisions. With solid experiments on widely used benchmarks and settings provided, I would consider changing my ratings.
3 comments
Thanks for providing the requested experiments. They support this paper better in showing the ablation effects. However, at this moment, I still have some major concerns:
- The overall presentation is extremely difficult to follow. Many figures and visualizations are not self-contained. The notation system is chaos and hard to follow without a table of notations. The motivations are poorly explained while a bunch of terms and theories are thrown at readers. The qualitative results are limited in that no SOTA works' performance is demonstrated.
- Concerns toward reimplementations: the provided supplementary material contains some code but does not cover the full pipeline. There are many details missing. For example, NaCLIP's official GitHub repo reports mIoU of 64.07 on VOC21. But the rebuttal shows "zero-shot 29.707". It is unclear to me what makes this difference or where "last-12-layer mean features" comes from.
- The contributions still remain incremental. The paper does not sufficiently establish what substantial conceptual advance or broadly useful capability this combination provides.
Thank you for acknowledging the additional experimental evidence and explaining your remaining concerns. We appreciate the opportunity to clarify these points.
1. Presentation and readability.
We apologize for the difficulty in following the manuscript. Given the number of equations, omitting a notation table was an oversight on our part. We believe the motivation is meaningful: self-training can turn prediction errors into later supervision and progressively amplify false foreground. The reported results and component analyses support the value of addressing this feedback process, but we recognize that its explanation needs to be clearer. We will improve the language and organization, add a notation table, clarify figure labels and captions, and supplement qualitative comparisons with official-configuration OVS predictions.
2. Implementation settings and official NACLIP comparison.
Last-12-layer mean features. Our implementation builds on MLMP's multilayer feature code to establish a consistent adaptation comparison setup. We average the outputs of the last 12 visual layers to obtain dense features, using ViT-L/14 and three templates per prototype. We should have distinguished this more clearly from the complete official NACLIP configuration. The original NACLIP paper reports 57.9% VOC21 mIoU with ViT-L/14, the same backbone used in our study; the repository's 64.07% corresponds to ViT-B/16.
The performance gap primarily reflects our simplified prompt configuration. We use three rather than 80 templates, replace predefined background concepts with a generic background prototype, and omit PAMR postprocessing. For example, the official background vocabulary includes sky, grass, wall, and road, whereas we average "a background scene," "a background photo," and "an image background." Our intention was to use a unified configuration across all test datasets without enumerating background categories separately for each one, since deployment backgrounds may extend beyond a predefined vocabulary. We intended this setting to reflect such practical conditions. Using shared experimental conditions provides a fair basis for comparing the methods and assessing their effectiveness within this setting.
To address your concern, we reevaluated CDI+HRC under the official settings. Scores are percentages.
| Method | mIoU | Present Dice | Absent area | Pixel accuracy |
|---|---|---|---|---|
| Official-configuration zero-shot | 57.851 | 67.098 | 0.187 | 85.556 |
| + CDI+HRC | 60.243 | 69.265 | 0.181 | 88.765 |
Compared with zero-shot, CDI+HRC improves mIoU by 2.392 points, Present Dice by 2.167 points, and pixel accuracy by 3.209 points, while reducing absent area from 0.187% to 0.181%. These results support the method's effectiveness under the official configuration as well as our simplified setting.
3. Conceptual contribution.
We appreciate the concern about the extent of the contribution. To our knowledge, this is the first framework to explicitly investigate and jointly address the distinct adaptation failures across both single-query and multi-query OVS: foreground suppression under confidence sharpening and spurious foreground expansion under anchor self-training. Although the method builds on existing techniques, we believe its contribution goes beyond combining components: it identifies this feedback problem across query settings and introduces coordinated controls at its two stages.
Thank you for the evaluation suggestions. We address standard metrics, spatial protocols, and full scene-parsing datasets below. Scores are percentages.
W1–W3 / Questions for Authors: Standard evaluation.
Standard mIoU. The additional results below include standard mIoU, computed from the dataset-level confusion matrix. The NACLIP table also reports Present Dice, absent area, and pixel accuracy.
Spatial protocol and comparison with published OVS models. Beyond the original 224x224 evaluation, we added aspect-preserving resizing and original-resolution scoring. On the full VOC validation set, SCLIP retains its official 80-template ensemble, synonyms, sliding-window inference, temperature, and postprocessing. Adaptation uses one class residual shared across synonyms and normalized class scores; zero-shot predictions exactly reproduce the official inference path. Official SCLIP zero-shot obtains mIoU/Present Dice of 59.691/62.873. Multi-BAPA obtains 27.706/49.780, CDI-only 48.333/61.170, and CDI+HRC 62.384/64.241. Compared with zero-shot, CDI+HRC improves mIoU and Present Dice by 2.693 and 1.368 points, respectively.
Original-resolution NACLIP evaluation. On all 1,449 VOC validation images, NACLIP ViT-L/14 uses aspect-preserving resizing, original-resolution scoring, our three templates and last-12-layer mean features.
NACLIP on the full VOC validation set
CDI+HRC improves all four metrics over Multi-BAPA, CDI-only, and zero-shot. Compared with zero-shot, mIoU and Present Dice increase by 1.817 and 6.982 points, respectively, while absent area decreases from 2.305% to 1.959% and pixel accuracy increases from 32.347% to 56.752%. The full model improves segmentation quality while reducing false foreground.
ADE20K and COCO-Stuff. We evaluated the full validation sets: 2,000 images/150 classes for ADE20K and 5,000 images/171 classes for COCO-Stuff. The table reports mIoU.
CDI+HRC achieves the highest mIoU among the compared methods on both datasets, exceeding Multi-BAPA by 11.651/15.548 points and zero-shot by 1.901/0.714 points. Adding HRC improves CDI-only by 4.277/2.490 points, demonstrating complementary benefits of supervision control and update correction. These results show that the full model improves self-training and outperforms the frozen model in scene parsing with larger class sets.
W4: Choice of anchor self-training and component roles. We chose BAPA for its observation that severe class imbalance can drive existing adaptation methods to collapse, particularly in binary segmentation. We consider its class-balanced anchor strategy a sound response to this problem. Although multi-query segmentation has broader applications, we view the binary setting under severe class imbalance as a useful stress test of whether adaptation can preserve a small foreground against a dominant background. We adopt anchor self-training because class-balanced anchor supervision alleviates foreground suppression. Its dynamic refresh also exposes the problem we address: erroneous predictions become later supervision and drive false foreground expansion. CDI limits each class's total supervision using fixed semantic evidence, while HRC corrects candidate updates to reduce the risk of reinforcing prediction errors.
On full VOC, we compared simpler confidence weighting and its combination with KL consistency. Confidence weighting obtains Present Dice of 43.048%; combining it with KL consistency gives a best result of 56.233% across coefficients 0.1, 1, and 10, versus 58.669% for CDI+HRC. This gain over simpler alternatives demonstrates the value of class-total supervision control and candidate-update correction.
W5: We apologize for overlooking this stray sentence during the appendix checks. AI assistance in this editing step was limited to wording; the authors rigorously reviewed the contributions and results. The sentence does not affect our core claims and will be removed in the revision. We will expand CMS as Confusion Mask Score, replace "Senior background"/"Account" with background/aggregate risk, and describe CDI as limiting class supervision using semantic evidence.