- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Align-R addresses the trade-off between segmentation accuracy and category recognition recall in OVSS by preserving and realigning pretrained residual features rather than discarding them. It performs spatial alignment through semantic-center-guided exploration and concept alignment through reliable-anchor distillation. Experiments across eight benchmarks demonstrate consistent improvements in both training-based and training-free pipelines.
Critical strengths
Align-R provides a useful analysis of the semantic information preserved in CLIP residuals, supported by a controlled comparison with attention features. Its lightweight alignment module keeps CLIP frozen during training and adds little measured inference overhead.
Critical weaknesses
- Limited gains and validation on training-based methods. Align-R introduces task-specific supervision, so its large improvements over originally training-free pipelines are less surprising and should not, by themselves, be taken as strong evidence of its methodological advantage. More informative are its incremental gains on already-trained methods, which are relatively modest, +1.9, +1.2, and +0.5 mIoU on SAN, CAT-Seg, and DeCLIP, respectively. These comparisons also cover only four or five of the eight benchmark settings, without explaining the omissions. More comprehensive evaluations on recent training-based methods are needed to establish the added value of Align-R beyond improving originally training-free pipelines.
- The incremental benefit of the specialized spatial alignment (SA) architecture is modest. Simple supervised adapters already achieve most of the improvement, reaching 43.0 mIoU with an MLP and 43.4 with a ViT block, compared with 44.5 for full SA. Moreover, the global attention branch alone achieves 43.3 mIoU, while the remaining components jointly add only 1.2 points. These results support supervised residual adaptation but provide limited evidence that the more complex SA design is necessary.
Questions for authors
- Why are PC60, COCO Object, and Cityscapes omitted for all training-based frameworks? Could you extend the evaluation to these settings, or explain why they are unsuitable for evaluating Align-R?
- For SAN, CAT-Seg, and DeCLIP, is Align-R trained separately using each host’s CLIP checkpoint, or is a module trained on the original CLIP directly inserted? In the latter case, is any further adaptation performed for hosts with fine-tuned CLIP encoders?
1. Additional Training-Based Benchmarks
We thank the reviewer for this suggestion and apologize for the incomplete evaluation. PC60, COCO Object, and Cityscapes were initially omitted because the official implementations of the evaluated training-based frameworks do not provide evaluation pipelines for these datasets. Following OPMapper, we originally reported only the benchmarks commonly supported by both training-based and training-free hosts. The omission therefore reflected implementation availability rather than the suitability of these datasets for evaluating Align-R.
We therefore extend the evaluation to all three datasets. Based on the VOC21 evaluation protocol in CAT-Seg, we implement evaluation pipelines for PC60 and COCO Object that include their background categories, as well as the corresponding Cityscapes pipeline. The results below complete the evaluation across all eight benchmarks and further support the consistent improvements provided by Align-R.
2. Host-Specific Integration Protocols
We thank the reviewer for pointing out that the integration protocols were not explained clearly. The protocol depends on whether the host modifies the CLIP visual encoder. For SAN, the CLIP backbone remains frozen, so we use Align-R trained on the original frozen CLIP. We then freeze Align-R, insert it into SAN, and follow the official SAN training protocol while optimizing only SAN's original trainable components.
CAT-Seg and DeCLIP adapt CLIP through parameter-efficient tuning or distillation. We therefore first obtain their adapted backbone and decoder checkpoints using the official training procedures. We then freeze each adapted backbone and train Align-R on its residual representations rather than directly transferring the module trained on the original CLIP. After insertion, we keep the backbone and Align-R frozen and lightly fine-tune only the decoder for 10K iterations. For fairness, we also apply the same decoder-only fine-tuning to the corresponding baselines without Align-R and observe no performance gain. We have clarified these distinct integration protocols in the revised implementation details.