acceptodds
Withdrawn from ICLR 2027

Align-R: Rethinking Residuals in Open-Vocabulary Semantic Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In open‑vocabulary semantic segmentation (OVSS), the community commonly discards the final residual connections to bridge the adaptation gap between pretrained vision-language models and dense prediction, yielding better segmentation mIoU. We rethink this widely adopted heuristic and uncover an overlooked trade‑off: improved segmentation performance comes at the cost of recognition recall for categories present in the image. Pretrained residuals therefore carry meaningful pre-learned signals rather than pure noise, but align poorly with dense prediction objectives. From this observation, we identify two core failure modes: , where semantic responses are displaced from target regions; and , where residual features produce false activations for categories absent from the image. To address both misalignments, we present , which preserves residual information via instead of naive removal. Align-R performs spatial alignment through semantic-center-guided random exploration to recover fine-grained spatial cues, and concept alignment via reliable-anchor distillation to preserve correct category concepts and intra-class relationships. Evaluated across 8 OVSS benchmarks, Align-R yields consistent gains for both training-based and training‑free pipelines. Our results show that Align-R effectively mitigates residual misalignment and achieves a more favorable trade-off between recognition recall and segmentation performance. The code will be made publicly available.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Open-Vocabulary Semantic Segmentation, Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9X4JOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Align-R addresses the trade-off between segmentation accuracy and category recognition recall in OVSS by preserving and realigning pretrained residual features rather than discarding them. It performs spatial alignment through semantic-center-guided exploration and concept alignment through reliable-anchor distillation. Experiments across eight benchmarks demonstrate consistent improvements in both training-based and training-free pipelines.

Critical strengths

Align-R provides a useful analysis of the semantic information preserved in CLIP residuals, supported by a controlled comparison with attention features. Its lightweight alignment module keeps CLIP frozen during training and adds little measured inference overhead.

Critical weaknesses
  1. Limited gains and validation on training-based methods. Align-R introduces task-specific supervision, so its large improvements over originally training-free pipelines are less surprising and should not, by themselves, be taken as strong evidence of its methodological advantage. More informative are its incremental gains on already-trained methods, which are relatively modest, +1.9, +1.2, and +0.5 mIoU on SAN, CAT-Seg, and DeCLIP, respectively. These comparisons also cover only four or five of the eight benchmark settings, without explaining the omissions. More comprehensive evaluations on recent training-based methods are needed to establish the added value of Align-R beyond improving originally training-free pipelines.
  2. The incremental benefit of the specialized spatial alignment (SA) architecture is modest. Simple supervised adapters already achieve most of the improvement, reaching 43.0 mIoU with an MLP and 43.4 with a ViT block, compared with 44.5 for full SA. Moreover, the global attention branch alone achieves 43.3 mIoU, while the remaining components jointly add only 1.2 points. These results support supervised residual adaptation but provide limited evidence that the more complex SA design is necessary.
Questions for authors
  1. Why are PC60, COCO Object, and Cityscapes omitted for all training-based frameworks? Could you extend the evaluation to these settings, or explain why they are unsuitable for evaluating Align-R?
  2. For SAN, CAT-Seg, and DeCLIP, is Align-R trained separately using each host’s CLIP checkpoint, or is a module trained on the original CLIP directly inserted? In the latter case, is any further adaptation performed for hosts with fine-tuned CLIP encoders?
1 comment

1. Additional Training-Based Benchmarks

We thank the reviewer for this suggestion and apologize for the incomplete evaluation. PC60, COCO Object, and Cityscapes were initially omitted because the official implementations of the evaluated training-based frameworks do not provide evaluation pipelines for these datasets. Following OPMapper, we originally reported only the benchmarks commonly supported by both training-based and training-free hosts. The omission therefore reflected implementation availability rather than the suitability of these datasets for evaluating Align-R.

We therefore extend the evaluation to all three datasets. Based on the VOC21 evaluation protocol in CAT-Seg, we implement evaluation pipelines for PC60 and COCO Object that include their background categories, as well as the corresponding Cityscapes pipeline. The results below complete the evaluation across all eight benchmarks and further support the consistent improvements provided by Align-R.

MethodBackboneVOC21PC60ObjectVOC20PC59StuffCityADEAvg.
SANViT-B/1673.746.052.993.152.840.544.527.153.8
+ Align-RViT-B/1676.9 (+3.2)47.5 (+1.5)57.6 (+4.7)93.9 (+0.8)55.1 (+2.3)44.1 (+3.6)45.7 (+1.2)28.0 (+0.9)56.1 (+2.3)
CAT-SegViT-B/1675.849.058.894.556.445.750.131.057.7
+ Align-RViT-B/1677.7 (+1.9)50.6 (+1.6)61.2 (+2.4)94.9 (+0.4)57.7 (+1.3)47.4 (+1.7)51.0 (+0.9)31.7 (+0.7)59.0 (+1.3)
DeCLIPViT-B/1681.052.162.996.559.848.651.234.960.9
+ Align-RViT-B/1681.4 (+0.4)52.7 (+0.6)64.3 (+1.4)96.8 (+0.3)60.3 (+0.5)49.5 (+0.9)52.0 (+0.8)35.3 (+0.4)61.5 (+0.6)

2. Host-Specific Integration Protocols

We thank the reviewer for pointing out that the integration protocols were not explained clearly. The protocol depends on whether the host modifies the CLIP visual encoder. For SAN, the CLIP backbone remains frozen, so we use Align-R trained on the original frozen CLIP. We then freeze Align-R, insert it into SAN, and follow the official SAN training protocol while optimizing only SAN's original trainable components.

CAT-Seg and DeCLIP adapt CLIP through parameter-efficient tuning or distillation. We therefore first obtain their adapted backbone and decoder checkpoints using the official training procedures. We then freeze each adapted backbone and train Align-R on its residual representations rather than directly transferring the module trained on the original CLIP. After insertion, we keep the backbone and Align-R frozen and lightly fine-tune only the decoder for 10K iterations. For fairness, we also apply the same decoder-only fine-tuning to the corresponding baselines without Align-R and observe no performance gain. We have clarified these distinct integration protocols in the revised implementation details.

Reviewer_fuhROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates an overlooked issue in Vision-Language Models (VLMs) applied to Open-Vocabulary Semantic Segmentation (OVSS). The authors observe that the common heuristic of discarding the final-layer residual connection improves spatial coherence (segmentation mIoU) but sacrifices recognition recall for categories present in the image. To address this spatial and conceptual misalignment, the authors propose "Align-R", a lightweight, plug-and-play module. Align-R retains the residual connection but applies Spatial Alignment and Concept Alignment to explicitly align the pre-trained semantics for dense prediction.

Critical strengths
  1. Insightful Motivation and Solid Analysis: The statistical analysis revealing the trade-off between recognition recall and segmentation precision when removing residual connections is highly compelling. It clearly uncovers "why" the common heuristic works and what it loses, providing a strong theoretical and empirical foundation for the proposed Align-R.

  2. Broad Generalizability and Extensive Evaluation: The proposed Align-R is a lightweight, plug-and-play module that does not require fine-tuning the massive VLM. The experimental validation is remarkably thorough, demonstrating consistent improvements across 8 diverse datasets, multiple backbone scales (ViT-B, ViT-L, ViT-H), different pre-training sources, and both training-based and training-free pipelines.

Critical weaknesses
  1. Architectural Limitation: As acknowledged by the authors, the proposed method is heavily tailored to the structural properties of Vision Transformers. Its applicability to CNN-based VLMs or other non-transformer architectures remains unverified, which slightly limits the universality of the claim.

  2. Fairness in "Training-free" Comparisons: While Align-R acts as a plug-and-play module for "training-free" pipelines, the Align-R module itself requires a training phase on a source dataset to learn the alignment. Comparing a pipeline equipped with a module trained on dense annotations against strictly training-free methods might be seen as a slightly unfair comparison, as it introduces an inductive bias from the training set.

Questions for authors
  1. Regarding the training of the Align-R module: How sensitive is the module to the choice of the training dataset? You used COCO-Stuff; how would the generalization performance change if a smaller or domain-specific dataset was used to train Align-R?

  2. For the comparisons with "training-free" methods in Table 1, could you clarify your perspective on the fairness of the comparison? Since Align-R introduces parameters learned from dense segmentation masks, does it violate the strict premise of a zero-shot/training-free setup?

  3. The current design restricts alignment to the final visual block to maintain a lightweight design. Have you conducted preliminary experiments applying Align-R to intermediate layers? If so, does the performance-efficiency trade-off collapse, or are there marginal gains?

1 comment

1. Training-Dataset Sensitivity

We thank the reviewer for this valuable suggestion. Following common OVSS practice, our main experiments train Align-R on COCO-Stuff. To assess its sensitivity to the training data, we additionally train Align-R on ADE20K using the same architecture, optimization schedule, and preprocessing. As shown below, ADE20K training achieves an average mIoU of 38.0, compared with 45.9 for COCO-Stuff, while still substantially outperforming the original CLIP baseline of 11.6. Compared with COCO-Stuff training, ADE20K training improves the in-domain ADE result from 22.8 to 28.5, while reducing the average over the other seven benchmarks from 49.2 to 39.3. This pattern suggests stronger in-domain adaptation but weaker cross-dataset transfer.

The difference likely reflects dataset scale and semantic distribution. COCO-Stuff contains approximately 118K training images with diverse foreground-background compositions, providing broad supervision for transferable residual alignment. ADE20K contains approximately 20K images and is more scene-centric, with frequent structural categories and many sparsely represented classes. Following CAT-Seg, images are proportionally resized to a 384-pixel short edge and randomly cropped to 384×384384\times384. Although this preserves aspect ratios, wide or densely composed ADE20K images may experience stronger cropping, potentially reducing global context and increasing variation in object visibility and scale. Overall, Align-R remains effective when trained on a smaller, domain-specific dataset, while broader training data yield stronger cross-dataset generalization. We have added this experiment and discussion to the revised manuscript.

Training datasetVOC21PC60ObjectVOC20PC59StuffCityADEAvg.
CLIP16.27.75.541.89.24.45.52.111.6
Align-R (COCO-Stuff)41.243.442.091.350.040.136.622.845.9
Align-R (ADE20K)36.134.629.082.939.223.530.128.538.0

2. Fairness of Training-Free Comparisons

We thank the reviewer for raising this important point. We agree that Align-R is a supervised plug-in trained with dense masks on COCO-Stuff; therefore, the resulting systems are not strictly training-free. In Table 1, "training-free" refers only to the host frameworks. The experiment evaluates the transferability and plug-in effectiveness of Align-R: for each host, we preserve its pretrained weights and inference pipeline, insert the corresponding frozen Align-R, and perform no host retraining or target-benchmark adaptation. Thus, each paired comparison isolates the contribution of residual alignment under otherwise identical settings. This protocol follows OPMapper, which similarly evaluates a learned plug-in across training-free hosts.

These hosts provide a direct test of transferability because they operate in the original CLIP representation space without downstream feature adaptation. Align-R also provides consistent gains on training-based hosts under their corresponding integration protocols. Together, the improvements across training-based and training-free hosts, backbones, and label settings support the generalization of Align-R beyond its training pipeline. Accordingly, we interpret these results as evidence of plug-in transferability rather than strict training-free performance.

3. Intermediate-Layer Alignment

We thank the reviewer for this question. We have not evaluated simple multi-layer stacking because intermediate insertion is not a direct extension of the current design. Many recent OVSS methods remove the final-block residual and spatially calibrate its attention branch. Align-R therefore targets the same block, recovering the discarded residual while preserving the calibrated attention features and compatibility with existing pipelines.

Visual blocks are sequentially dependent, so modifying an intermediate residual changes all subsequent representations, including the final attention features used by existing methods. Multiple Align-R modules would consequently interact rather than operate independently, making the performance-efficiency trade-off dependent on a coordinated multi-layer design. We therefore focus on the final block as a lightweight and minimally invasive intervention point. Dependency-aware multi-layer alignment may provide further gains and represents an important direction for future work. We sincerely appreciate this valuable suggestion and have added this discussion to the future-work section of the revised manuscript.

Reviewer_52yKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper revisits the common practice in CLIP-based open-vocabulary semantic segmentation (OVSS) of discarding the final-layer residual connection (ClearCLIP). It argues that the residual carries useful pretrained semantics: removing it raises mIoU but lowers class-level recall. The authors attribute the problem to "spatial" and "conceptual" misalignment. They propose Align-R, a ~2M-parameter module that replaces the last-block residual of a frozen CLIP. It has two parts:

  • Spatial Alignment: a global attention branch, a deformable-attention local branch around the CLS token, and a spherical fusion.
  • Concept Alignment: two distillation losses on "reliable anchors" selected with ground-truth labels.

Align-R is trained once on COCO-Stuff with pixel-level cross-entropy and then plugged into training-free (ClearCLIP, ProxyCLIP, CASS, PEARL, GLA-CLIP) and training-based (SAN, CAT-Seg, DeCLIP) pipelines. The paper reports ~10 mIoU average gains for the former and +0.5 to +1.9 for the latter.

Critical strengths
  • The question of whether the discarded residual contains useful information is worth asking. The residual-vs-attention input control (Table 8) is a useful experiment.
  • The plug-in is evaluated broadly: five training-free pipelines, three CLIP pretraining sources and several backbone sizes. It is cheap at inference (Table 7).
  • The pixel-level error decomposition (App. A.3) is a reasonable diagnostic.
Critical weaknesses
  1. The headline gains come from dense supervision, not from residual alignment. Align-R is trained for 80k iterations on COCO-Stuff with full pixel-level labels (Sec. 5.1). Yet Table 1 places "training-free method + Align-R" next to the training-free originals, under a caption stating that "all methods are evaluated under the same setting". Once Align-R is added, these pipelines are no longer training-free. The ~10-point gains therefore mostly measure the effect of supervised adaptation. The paper's own Table 3 confirms this. Under the same supervision:

    • a plain MLP reaches 43.0 average mIoU;
    • a standard ViT block reaches 43.4;
    • the global branch alone (F_g) reaches 43.3;
    • the full Spatial Alignment design reaches 44.5;
    • the frozen-CLIP baseline is 12.5.

    About 31 of the 32 points come from training any adapter with labels. The specific design (SGRE, ASF) contributes about 1 point.

  2. In the like-for-like comparison, the gains largely disappear. For training-based frameworks, which use the same kind of supervision, the reported gains are small and measured against the authors' own reproductions. Those reproductions fall below the published results by about as much as the gains. All published numbers below are taken from the original papers, ViT-B/16:

    • CAT-Seg (Cho et al., CVPR 2024, Table 1) reports 31.8 / 57.5 / 94.6 on ADE-150 / PC-59 / VOC-20. CAT-Seg + Align-R reaches 31.8 / 57.7 / 94.9, i.e., +0.0 / +0.2 / +0.3 over the published numbers.
    • DeCLIP (Wang et al., CVPR 2025, Table 4) reports 36.3 / 60.6 / 96.6 / 81.3 on ADE-150 / PC-59 / VOC-20 / VOC-21. DeCLIP + Align-R reaches 35.3 / 60.2 / 96.7 / 81.3: below the published DeCLIP on ADE-150 and PC-59, and equal to it on VOC-21.
    • SAN (Xu et al., CVPR 2023, Table 2) reports 27.5±0.34 / 53.8±0.57 / 94.0±0.21 (mean over 5 runs). SAN + Align-R reaches 27.7 / 55.5 / 93.7, within one standard deviation on ADE-150 and lower on VOC-20.

    The paper reports a single run without variance. Its claim of "consistent gains" for training-based methods is therefore not established.

  3. The relevant baselines and benchmarks are missing. Because Align-R requires dense supervision, the appropriate comparison class is training-based OVSS and other trained plug-in modules. No 2026 training-based method is compared. DenseRC ("Reconstructing CLIP for Open-Vocabulary Dense Perception", CVPR 2026), which the paper itself cites as the source of its distillation teacher, reports 37.6 / 61.3 on ADE-150 / PC-59 with ViT-B/16. PCA-Seg (CVPR 2026) reports 38.1 / 62.2 / 97.3 on ADE-150 / PC-59 / VOC-20. Both exceed every configuration in Table 1. OPMapper, the most closely related trained plug-in, appears only in the appendix. In addition:

    • the standard large-vocabulary benchmarks for the training-based setting, ADE-847 and PC-459, are not reported;
    • the training-based rows report only a subset of datasets, which differs across methods.

    I am not asking for state-of-the-art results. Without these comparisons, the paper does not show that Align-R adds value in the setting it actually operates in.

  4. The motivating evidence is weak.

    • Higher class-level recall when the residual is retained (Fig. 1, App. A.2) is expected whenever predictions become noisier and cover more classes. Table 4 shows the cost: precision falls from 57.6% to 20.3%. Align-R also has lower precision (45.4%) than residual removal.
    • Fig. 1 relies on a single image.
    • In Fig. 3a, residual-derived class tokens are closer to the vanilla output. This follows directly from the residual stream carrying most of the representation. It does not show that the residual holds semantics useful for dense prediction.
  5. Model selection on the test benchmarks and unverified small effects. All ablations are run on the evaluation benchmarks, and hyperparameters are picked by their average there: λ2, the expansion size and the temperature τ (Tables 11–12, Fig. 5b). Fig. 5a tracks test performance during training. Many component effects are 0.1–0.3 mIoU (Tables 10–12), and in Table 10 Slerp outperforms the proposed ASF-Dir. All of these come from single runs, so they cannot be distinguished from noise.

  6. The integration procedure for training-based frameworks is unclear. Align-R is trained on a frozen, original CLIP with a cosine decoder (App. A.4). CAT-Seg, SAN and DeCLIP fine-tune or adapt CLIP. The paper should state whether these frameworks were retrained with Align-R inserted, or whether a module trained on the original CLIP was inserted into a fine-tuned encoder.

Questions for authors
  1. Please compare "training-free method + Align-R" with the same pipelines given a generic trained adapter (MLP or ViT block, as in Table 3) under identical supervision. Please also report the five non-COCO benchmarks separately.
  2. Can you report CAT-Seg, SAN and DeCLIP with and without Align-R using matched seeds (mean ± std), and explain why your reproductions fall below the published numbers?
  3. Were the training-based frameworks retrained with Align-R inserted, or was the module trained on frozen original CLIP inserted into their fine-tuned encoders?
  4. How does Align-R compare with DenseRC, PCA-Seg and OPMapper on ADE-847, PC-459, ADE-150, PC-59 and VOC-20?
  5. Were any hyperparameters or checkpoints selected using the reported test benchmarks?
1 comment

1. Generic Adapter Comparison

We thank the reviewer for this suggestion. We compare SA with generic MLP and ViT-block adapters on ClearCLIP under identical supervision, optimization, and inputs. MLP and ViT Block achieve 45.2 and 45.7 average mIoU, whereas SA reaches 47.4. SA outperforms them by 2.2 and 1.7 points using only 1.98M parameters, compared with 4.72M and 7.09M. These results confirm that the gains arise from the proposed residual alignment rather than generic feature adaptation or increased model capacity.

MethodBackboneΔ\Delta Params(M)VOC21PC60ObjectVOC20PC59StuffCityADEAvg.
ClearCLIPViT-B/16--51.832.633.080.935.923.930.016.738.1
+ MLPViT-B/164.7254.541.244.888.546.838.430.216.945.2 (+7.1)
+ ViT BlockViT-B/167.0955.042.242.191.548.638.830.317.145.7 (+7.6)
+ SA (Ours)ViT-B/161.9858.843.343.991.750.339.332.718.947.4 (+9.3)

2. Matched-Seed Evaluation

We thank the reviewer for this suggestion. We evaluate SAN, CAT-Seg, and DeCLIP with and without Align-R using the same three seeds and report the mean and standard deviation across runs. Following CAT-Seg's inference strategy for datasets with a background category, we also extend the evaluation to the previously missing benchmarks. Given our available resources, all methods are reproduced on two NVIDIA RTX 3090 GPUs. This differs from the larger multi-GPU setups used in the original implementations and requires adjustments to the batch size and learning rate, which may affect absolute performance. Nevertheless, each baseline and its Align-R variant use identical hardware, seeds, and host-specific training settings, ensuring fair paired comparisons.

MethodBackboneVOC21PC60ObjectVOC20PC59StuffCityADEAvg.Std.
SANViT-B/1673.746.052.993.152.840.544.527.153.80.33
+ Align-RViT-B/1676.947.557.693.955.144.145.728.056.1 (+2.3)0.21
CAT-SegViT-B/1675.849.058.894.556.445.750.131.057.70.08
+ Align-RViT-B/1677.750.661.294.957.747.451.031.759.0 (+1.3)0.10
DeCLIPViT-B/1681.052.162.996.559.848.651.234.960.90.11
+ Align-RViT-B/1681.452.764.396.860.349.552.035.361.5 (+0.6)0.07

3. Integration with Training-Based Frameworks

We thank the reviewer for requesting clarification. The protocol depends on whether the host modifies the CLIP backbone. For SAN, we train Align-R on the original frozen CLIP, freeze it, and insert it while optimizing only SAN's original trainable components under the official protocol. CAT-Seg and DeCLIP adapt CLIP through parameter-efficient tuning or distillation. We therefore obtain their adapted backbone and decoder checkpoints through official training, freeze the backbone, and train Align-R on its residual representations. We then insert Align-R and fine-tune only the decoder for 10K iterations, keeping the backbone and Align-R frozen. For fairness, we apply the same decoder-only fine-tuning to the baselines without Align-R and observe no gain. These protocols are clarified in the revised manuscript.

4. Comparison with Related Methods

We thank the reviewer for this question. Align-R is an approximately 2M-parameter residual-alignment plug-in rather than an independent OVSS system, so we assess it primarily through paired host comparisons. DenseRC and PCA-Seg use different framework designs and adaptation strategies, so direct absolute-score comparisons would not isolate the contribution of residual alignment. OPMapper is more closely related due to its lightweight plug-in design. Since its official implementation is unavailable, we compare against its reported results under common hosts, backbones, and benchmarks in the appendix, where Align-R achieves larger and more consistent gains. The official DenseRC implementation is also unavailable.

Our aim is to investigate the residual trade-off and introduce a lightweight, plug-and-play improvement for existing OVSS frameworks, rather than an independent OVSS system. We have clarified this scope in the revised manuscript.

5. Hyperparameter and Checkpoint Selection

We thank the reviewer for raising this point. We perform no benchmark-specific tuning or checkpoint selection. Hyperparameters are fixed across datasets, and the final COCO-Stuff checkpoint is evaluated on every benchmark.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.