acceptodds
Under review as a conference paper at ICLR 2027

ReGDec: Decoupling Region Cues from Global Semantics for VLM-Grounded Open-Vocabulary Dense Perception

Abstract

Vision-Language Pretrained Models (VLMs) achieve strong open-vocabulary perception by aligning visual representations with natural language semantics, but their global image-text pre-training objectives often limit local discriminability for dense perception. Existing approaches improve dense perception through feature refinement, external spatial guidance, or task-specific adaptation. In this paper, we analyze how spatial visual cues evolve across VLM depths and observe that preferred feature depths vary across tasks and regions. These observations motivate region-adaptive fusion of semantic and region cues across layers. To address this, we propose ReGDec (Region-Global Decoupling), a region-adaptive framework that decomposes region features into CLS-aligned components and orthogonal residuals. A compact selector learns separate region-conditioned weights to combine the two components across layers, while keeping the VLM encoders frozen. Experiments across multiple VLM backbones and dense perception benchmarks demonstrate the effectiveness of ReGDec, with gains varying across backbones and tasks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Open-Vocabulary Perception, Decoupling Region Cues, Local–Global Representation Decoupling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_vBWWOpenReview ↗
Rating
3(+1)
Confidence
3(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates the variation of region cues across feature depths and their preferences across different tasks and regions. To address this, the authors propose ReGDec, which decomposes region features into CLS-aligned components and orthogonal residuals, and adaptively fuses them across layers. Extensive experiments on CLIP-family backbones and dense prediction benchmarks demonstrate the effectiveness and transferability of the proposed method.

Critical strengths
  1. This paper includes a dedicated motivation section that clearly illustrates the motivation behind the proposed method through experimental results.

  2. Both quantitative and qualitative analyses thoroughly demonstrate the simplicity and efficiency of the proposed method.

Critical weaknesses
  1. The state-of-the-art performance of the proposed method is questionable. The methods used for comparison in the experiments are relatively outdated, with the most recent baselines published in 2023. Why are methods published in 2025 and 2026 not included in the comparisons? In particular, I noticed that the paper cites recent methods such as DeCLIP (2025), but does not include them in the experimental comparisons.

  2. The proposed method involves numerous hyperparameters, but the underlying mechanisms governing their joint effects are not sufficiently investigated. For instance, slight changes in these hyperparameters may lead to substantial fluctuations in model performance.

  3. The related work section is not appropriately presented. The core discussion of related work spans fewer than 24 lines. In particular, Section 5.2, "Open-Vocabulary Dense Perception," lacks detailed explanations of how previous methods work, making it difficult to distinguish the proposed method from existing approaches. Furthermore, why is the related work section placed after the experiments rather than immediately following the introduction?

Questions for authors

Please refer to 'Critical Weaknesses'.

5 comments
AuthorsOpenReview ↗

Response to Reviewer vBWW

We thank the reviewer for the careful review and constructive feedback. To address the concerns raised, we provide our responses below.

1. Comparison with SOTA

Thank you for pointing out that we cited DeCLIP (CVPR 2025) but did not include it in our detection comparison. Our original experiments focused on measuring the gain from adding ReGDec to the same detector, using EVA-CLIP and CLIPSelf to cover both standard and dense-adapted backbones. We evaluated the OV-COCO detector checkpoint released with the TPAMI 2026 extension of DeCLIP, kept the detector weights fixed and trained only the ReGDec selector, then used it to fuse features from multiple DeCLIP layers for each proposal in the detector’s VLM scoring branch. Adding ReGDec increased novel AP50 from 49.7 to 49.9 (+0.2 AP). We will include this paired comparison in the main paper.

Our paper examines how combining features across layers for each region can improve recognition with frozen CLIP-family encoders, and whether these gains carry over to open-vocabulary detection and segmentation. Our detection experiments therefore compare the same detector with and without ReGDec. CLIPSelf is particularly relevant because, like DeCLIP, it uses COCO training images to adapt pretrained CLIP visual encoders for dense perception in the OV-COCO setting. With the backbone and detector weights held fixed, adding ReGDec to the released F-ViT detector with CLIPSelf ViT-L/14 improved novel AP50 from 44.3 to 44.8 (+0.5). The gain on EVA-CLIP ViT-L/14 was larger, from 24.2 to 33.4 (+9.2). One plausible explanation is that CLIPSelf’s region-level self-distillation and DeCLIP's region-level distillation from DINOv2 already make their layer features more discriminative for region recognition, leaving less room for improvement from cross-layer fusion.

2. Joint sensitivity to hyperparameters

An exhaustive evaluation of all hyperparameter combinations would require more time than the rebuttal period allowed. We therefore conducted two important joint sensitivity analyses. The candidate window size ∣W∣|W| and sparsity KK jointly determine how many layers are available for fusion and how many are retained. The teacher temperature τu\tau_u and utility-loss weight λsel\lambda_{\mathrm{sel}} determine how strongly the teacher favors high-utility layers and how closely the selector is encouraged to follow that preference. We evaluated all nine combinations in each grid. Both tables report mean COCO Top-1 accuracy (%) across boxes, thing masks, and stuff masks.

Table R1. Candidate window × residual sparsity.

Candidate layersK=4K=4K=5K=5K=6K=6
Last 657.5057.6457.31
Last 958.4258.9758.53
Last 1258.2858.2357.94

Table R2. Utility-loss weight × teacher temperature.

Utility-loss weightτu=1\tau_u=1τu=3\tau_u=3τu=5\tau_u=5
0.556.8357.3057.28
1.056.3656.7656.92
2.056.4656.5856.55

Increasing KK from 4 to 5 helps more with nine candidate layers than with six and gives no gain with twelve, showing that the effect of residual sparsity depends on the candidate window. In the loss grid, increasing τu\tau_u from 1 to 5 changes the mean by +0.45, +0.56 and +0.09 at the three respective loss weights. The range of mean Top-1 accuracy is 1.66 percentage points in Table R1 and 0.94 in Table R2. For stuff masks alone, the corresponding ranges are 2.61 and 2.44 percentage points. In the revised main paper, we will expand the joint sensitivity analysis to cover additional hyperparameter combinations.

3. Positioning relative to prior work

We introduced the closest methods in the Introduction and placed the full Related Work section after the experiments so readers could follow the layer-wise observation directly into the motivation and the method. We agree that readers currently reach the main comparison with prior work too late. We will bring the distinction from related methods into the Introduction, expand Section 5.2 with concrete examples from detection and segmentation, and use the appendix for the fuller survey.

For detection, F-VLM uses a frozen VLM to score proposed regions and combines those scores with the detector’s predictions. For segmentation, CAT-Seg aggregates pixel–text matching costs. ResCLIP is particularly relevant because it uses intermediate layers to refine patch attention. ReGDec pools the CLS-aligned components and orthogonal residuals within each supplied region and learns separate cross-layer weights for the two components. We will make this distinction explicit in the main paper.

Thank you for the authors' detailed response. However, I would like to see actual revisions to the manuscript rather than simply receiving explanations in the response. Until substantive revisions are made and presented, I will maintain my current score.

AuthorsOpenReview ↗

Response to the Reviewer’s Suggestion for Manuscript Revisions

Thank you for your valuable feedback. Following your suggestions, we have revised the manuscript, with the revised content highlighted in blue. Specifically, we have expanded the related-work discussion in the Introduction (lines 43–53 and 73–75), Section 5.2 (lines 512–528), and Appendix A (lines 866–928); added the DeCLIP comparison to Table 4(a) (lines 344–345); and included joint hyperparameter sensitivity analyses in Appendix B.2 (lines 1155–1224).

Reviewer_soFyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ReGDec, a lightweight plug-in for frozen CLIP-style VLMs that improves open-vocabulary dense perception. It decomposes region features into CLS-aligned semantic and orthogonal residual components, then uses a region-conditioned selector to fuse them across layers. Only the selector is trained, using labeled source regions and classification plus utility-guided losses. Across dense recognition, open-vocabulary detection, and semantic segmentation, ReGDec shows strong dense-classification gains and notable improvements in some downstream tasks.

Critical strengths
  1. The motivation of the paper that region cues are present but depth-dependent in VLMs is intuitive and relevant to open-vocabulary dense prediction.
  2. The decomposition of patch/region features into a CLS-aligned component and an orthogonal residual is simple, interpretable, and technically appealing.
  3. The evaluation is broad, including dense recognition, OVOD, OVSS, multiple CLIP-family variants, larger backbones, additional datasets, and transfer settings.
Critical weaknesses
  1. The paper compare itself with MaskCLIP, CLIP Surgery, GEM, NaCLIP, CLIPSelf, but its novelty may seem incremental: improving dense features from frozen CLIP via intermediate-layer extraction and learned fusion. While the CLS-aligned/residual decomposition is promising, the paper should better explain how it differs from generic multi-layer feature fusion.
  2. Although the backbone is frozen, ReGDec still requires supervised training on labeled COCO regions, making it less lightweight than training-free CLIP dense-readout methods, and can also weaken its generalizability on domains significantly different from COCO.
  3. Improvements are substantial for some vanilla VLM settings but modest against stronger dense-adapted baselines like CLIPSelf and NaCLIP. For instance, with CLIPSelf ViT-B/16 dense classification, Mask Stuff Top-1 drops from 47.2 to 46.2, and some Top-5 metrics decline. This suggests the method may primarily recover information absent from weaker dense readouts rather than deliver broadly consistent gains.
Questions for authors
  1. Because each layer’s residual is defined relative to its own CLS direction, the residual subspaces are layer-dependent. It is unclear whether residuals from different layers are geometrically compatible for direct weighted summation.
  2. Although the proposed geometric statistics correlate with oracle utility, their ablation gains are small. This raises whether the selector learns meaningful region-conditioned layer preferences or mainly exploits a learned global layer prior.
  3. Selector preferences appear concentrated around a few layers, making it unclear how much improvement comes from genuine region-adaptive selection versus a backbone-specific global layer prior. A comparison with fixed learned layer weights shared across all regions would help clarify this.
  4. How should one choose the hyperparameters when the downstream task or dataset is unknown? Were these hyperparameters tuned only on the COCO source recognition split, or were any of them selected based on downstream OVOD/OVSS validation performance?
3 comments
AuthorsOpenReview ↗

Response to Reviewer soFy

Thank you for the careful review and constructive feedback.

1. Why can residuals from different layers be combined? Each residual is orthogonal to its own layer's CLS direction. ReGDec applies the same frozen visual projection to each layer's patch and CLS features before decomposition, so all residuals share the text-embedding coordinates. For a region RR and a category text embedding tt, the residual contribution to the score before final normalization is

t⊤∑ℓ∈KRw~RℓrRℓ=∑ℓ∈KRw~Rℓ(t⊤rRℓ).t^\top\sum_{\ell\in\mathcal K_R}\tilde w_R^\ell r_R^\ell=\sum_{\ell\in\mathcal K_R}\tilde w_R^\ell(t^\top r_R^\ell).

Each term contributes to the score along the same category direction, even though the layer-specific residual subspaces differ. We compared projecting before decomposition with projecting after fusion. ReGDec normalizes, decomposes, pools and fuses in text-embedding space. The alternative performs these operations in visual hidden space and projects the fused feature before text scoring. Projecting first improves Top-1 accuracy by 2.8 points on boxes and 2.5 on thing masks.

Table R1. Effect of projection order on COCO Top-1 accuracy (%).

Order of operationsBoxesThing masks
Project, then decompose and fuse in text-embedding space (ReGDec)70.872.3
Decompose and fuse in visual hidden space, then project68.069.8

2. Do the weights carry information about the particular region? We compared two ReGDec variants with the same feature decomposition. One learns a single set of layer weights shared across all regions. The other predicts weights for each region.

Table R2. COCO Top-1 accuracy (%) with shared and region-conditioned layer weights.

Layer weightsBoxesThing masksStuff masks
Learned weights shared across all regions67.369.426.2
Weights predicted for each region70.872.329.8

Predicting weights for each region improves recognition on all three region types. This also clarifies the modest gains in Table 7. Even without the three geometric statistics, the selector uses the pooled feature of each region to predict its layer weights. Table 7 measures the incremental benefit of adding those statistics to an already region-conditioned selector.

3. Comparison with learned layer weights shared across regions.

Table R2 gives the requested comparison with a learned global layer prior. Predicting weights for each region improves Top-1 accuracy by 3.5 percentage points on boxes, 2.9 on thing masks, and 3.6 on stuff masks. Thus, even when the selector tends to favor a few layers, adapting their weights to each region improves recognition.

We also compared two fusion strategies that both predict weights for each region over the same six candidate layers. Weighting CLS-aligned components and orthogonal residuals separately improves on merging whole region features by 2.1, 1.5, and 0.8 percentage points on boxes, thing masks, and stuff masks, respectively.

4. How should the hyperparameters be chosen?

We selected the ReGDec selector hyperparameters using COCO source recognition only. We did not use downstream OVOD or OVSS validation results to choose hyperparameters. For a new dataset with the same backbone and pipeline, we reuse the COCO-trained selector and its configuration. For a new backbone, we recommend freezing it, training the selector on labeled COCO regions, and jointly choosing the candidate window WW and the number of retained residual layers KK on a held-out COCO recognition split. When the downstream task is unknown, mean Top-1 accuracy across boxes, thing masks, and stuff masks provides a common selection criterion. If labeled target training data are available, a held-out subset of that training split can alternatively be used for selection.

5. Supervision cost and cross-dataset transfer.

We train only the selector using existing region annotations from 5,000 COCO training images and keep the VLM encoders frozen. In our label-efficiency ablation, 1,000 images recovered 94.6% of the mean Top-1 gain obtained with 5,000. For each backbone, we applied the same COCO-trained selector to all seven region-recognition benchmarks without target-specific retraining (Table 3). Mean Top-1 accuracy improved over the corresponding frozen baselines by 8.2 percentage points for CLIP ViT-L/14 and 6.0 for EVA-CLIP ViT-L/14. Gains are smaller and can be negative with strong dense baselines. On CLIPSelf ViT-B/16, stuff-mask Top-1 drops from 47.2% to 46.2%. With NaCLIP, mIoU stays at 19.0 on ADE-150 and rises from 83.0 to 84.7 on VOC.

I thank the authors for their detailed rebuttal. Most of my previous concerns have been addressed, so I will maintain my score and lean toward acceptance.

AuthorsOpenReview ↗

Response to the Reviewer’s Feedback and Summary of Manuscript Revisions

Thank you for your thoughtful feedback and positive assessment. We have revised the manuscript and added the corresponding experimental comparisons to Appendix B.1 (lines 963–975). The revised content is highlighted in blue.

Reviewer_n9LXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a new method, ReGDec (Region-Global Decoupling), for VLM-Grounded open-vocabulary dense perception. Specifically, the proposed method decomposes region features into CLS-aligned components and orthogonal residuals and then trains region-conditioned weights to combine the two components across layers. The experiments are extensive and demonstrate the effectiveness and generalizability of the proposed method.

Critical strengths
  1. The paper is well motivated and technically sound.
  2. The proposed method is simple and effective and can be extended to many tasks in the same area.
  3. The experiments are extensive and demonstrate the effectiveness of the proposed method. The authors conduct many experiments across baselines, tasks, and settings; it is encouraging that the proposed method consistently improves performance across these settings.
Critical weaknesses
  1. As the feature decomposition and region-conditioned combination are the main contributions of this paper, I believe that the comparison with region feature selection and merging is important and should be added to the main paper to demonstrate the advantages of the proposed method.
  2. The results in Table 7 show that the performance differences among different cue feature designs are relatively small. The proposed design only achieves improvements of 0.2%-0.3% across different tasks. It seems that the contribution of cue feature design is limited, and the authors should provide further justification.
  3. The visualization (Figure 10) in the appendix is interesting and should also be added to the main paper.
Questions for authors

Please refer to the Critical Weaknesses section.

4 comments
AuthorsOpenReview ↗

Response to Reviewer n9LX

We thank the reviewer for the careful review and constructive feedback. To address the concerns raised, we provide our responses below.

1. Comparison with Region Feature Selection and Merging

Following the reviewer’s suggestion, we compared ReGDec with region feature selection and region feature merging to verify whether decomposing region features and combining the two components separately improves region recognition. We will include this comparison in the main paper.

All three methods use the same fixed six layers of frozen EVA-CLIP ViT-L/14 and are evaluated on the same COCO regions. For each box or mask, the two baselines pool patch features within the region at each layer. Region feature selection uses the region feature from the layer with the highest selector weight. Region feature merging takes the weighted sum of the region features from the five highest-weighted layers. ReGDec decomposes each patch feature into a CLS-aligned component and an orthogonal residual, then pools each component within the region. For each region, the selector predicts one set of cross-layer weights for the CLS-aligned components and another for the orthogonal residuals.

Table R1. COCO region recognition.

MethodBoxesThing masksStuff masks
Region feature selection68.170.028.6
Region feature merging68.770.829.0
ReGDec70.872.329.8

ReGDec improves Top-1 accuracy over region feature selection by 2.7, 2.3, and 1.2 on boxes, thing masks, and stuff masks, respectively. Compared with region feature merging, the corresponding gains are 2.1, 1.5, and 0.8. Across all three tasks, learning separate cross-layer weights for the CLS-aligned components and orthogonal residuals yields higher accuracy than directly selecting or merging whole region features.

2. Contribution of the cue features in Table 7

We agree that the gains in Table 7 are small. One possible explanation is that the baseline selector already uses the pooled region feature zRLz_R^L to predict how much weight to assign to each layer’s residual. Even without the three statistics, different regions can therefore receive different layer weights. Table 7 therefore measures the incremental benefit of these statistics for region recognition: adding them improves Top-1 accuracy by 0.2, 0.3, and 0.2 percentage points on boxes, thing masks, and stuff masks, respectively.

We chose the three statistics to describe properties of each residual that are relevant to fusion. Norm measures its magnitude, Dot its alignment with the final-layer anchor, and Coh its directional agreement with the preceding layer’s residual. Figure 2(c) motivates this choice: with the anchor held fixed, these statistics correlate with the classification margin obtained when each candidate residual is added. They thus give the selector direct, label-free cues for refining its residual weights.

3. Placement of Figure 10 We will move Figure 10 alongside the region-recognition results in the main paper to show how cross-layer fusion changes patch similarity to the same query point.

AuthorsOpenReview ↗

Follow-up: Summary of Manuscript Revisions

We have revised the manuscript, with the revised content highlighted in blue. Specifically, we have moved the visualization previously shown in Figure 10 into the main paper as Figure 6 in Section 4.5 (lines 462–492), and added the comparison with region feature selection and merging to Appendix B.1 (lines 942–962).

Reviewer_n9LXOpenReview ↗

Feedback to the response

Thanks for the response, and most of my concerns have been addressed. I hope the author can revise the final version as promised. I will keep the original rating.

AuthorsOpenReview ↗

Update on the Promised Manuscript Revisions

Thank you for your response. We have further revised the manuscript to include the comparison with region feature selection and merging as Table 8 in Section 4.4 and the visualization as Figure 6 in Section 4.5, both in the main paper. Thanks again for your constructive suggestions, which have helped us improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.