acceptodds
Under review as a conference paper at ICLR 2027

Composing Complementary Dense Representation for Cost Aggregation in Open-Vocabulary Semantic Segmentation

Abstract

Open-vocabulary semantic segmentation enables dense scene understanding beyond a fixed training vocabulary. Existing cost-aggregation methods, however, typically reuse a single dense visual feature for both cost map construction and spatial guidance. Coupling these objectives within a shared dense feature space induces competition between semantic discrimination and spatial coherence, and simultaneously constrains the diversity of visual representations available across open-vocabulary scenarios. To address these limitations, we introduce Complementary Dense Representations (CoDR), a novel framework for constructing and composing dense representations for cost aggregation. Specifically, we first introduce Cost Spatial Decoupling, which assigns cost map construction and spatial guidance to distinct representation spaces, freeing both from the constraints of a shared feature. Building on this decoupling, Representation Diversification constructs multiple complementary representations within each space to preserve distinct semantic and spatial cues. Furthermore, we design Scene-Adaptive Representation Composition to hierarchically compose structurally distinct representations for each input scene, replacing a fixed global preference with scene-specific composition. Extensive experiments on five standard benchmarks demonstrate the strong performance of CoDR, which achieves 19.8 mIoU on ADE847. We will release all code and models upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Open-vocabulary Segmentation, Dense Representation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dJgQOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CoDR, a complementary-representation framework for cost aggregation, which employs CSD to establish separate cost and spatial representation spaces through asymmetric token flow, thereby alleviating the competition caused by a shared dense feature. Furthermore, they develop RD and SARC to preserve complementary cost and spatial representations and compose them for each input scene using lower-loss preference supervision.

Critical strengths
  1. The motivation and design of this paper are clear and well-structured. It addresses the problem of open-vocabulary semantic segmentation from a novel perspective, demonstrating meaningful research significance.

  2. The figures are visually appealing, and the writing is concise and precise.

Critical weaknesses
  1. I notice that other OVS methods are often evaluated on the VOC20-b benchmark in addition to the benchmarks used in this paper. However, this paper lacks experimental results on VOC20-b. Furthermore, I have two concerns regarding the claimed state-of-the-art performance. First, the proposed method is not compared with the latest methods published in 2026. Second, the performance improvements on some datasets, such as VOC20, are relatively marginal.

  2. Introducing the DINO encoder into open-vocabulary semantic segmentation to provide spatial guidance is reasonable. However, the methods compared in Table 1 lack similar settings, making it difficult to determine whether the performance improvements stem from the robust feature representations of DINO or from the proposed method itself.

Questions for authors

Please refer to Critical Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer dJgQ

Dear Reviewer dJgQ,

Thank you for your specific suggestions on evaluation coverage, comparisons with recent methods, and the contribution of DINO. Our responses are as follows.

Q1: Missing evaluation on VOC20-b.

Thank you for identifying this omission. We additionally report the results for CoDR-B and CoDR-L on VOC20-b and list the available background-inclusive evaluation results for methods in Table 1:

MethodBackbone/sizeVOC20-b
CAT-SegViT-B/1677.3
DeCLIPViT-B/1681.3
CoDRViT-B/1681.5
FC-CLIPConvNeXt-L81.8
CAT-SegViT-L/1482.5
DeCLIPViT-L/1483.9
H-CLIPViT-L/1483.2
CoDRViT-L/1484.1

Q2: Missing comparisons with 2026 methods and the small improvement on VOC20.

Thank you for raising this point. The submitted paper does not sufficiently compare our method with recent work, and we will expand these comparisons in the revision. Below, we list published results from relevant 2026 methods alongside CoDR's submitted results:

MethodVenueSizeADE847ADE150PC459PC59VOC20
InfoCLIP [1]AAAI 2026ViT-L16.638.524.663.597.5
LoGoSeg [2]AAAI 2026ViT-L16.438.224.563.697.4
PCA-Seg [3]CVPR 2026ViT-L18.341.526.764.798.4
DenseRC [4]CVPR 2026ViT-L18.441.526.764.0--
HyperCLIP++ [5]TPAMI 2026ViT-L17.038.725.064.498.5
CoDRThis workViT-L19.842.127.364.297.9

CoDR improves over DeCLIP by only 0.2 mIoU on VOC20, so the gain is indeed limited. Our method's main advantage is in large-vocabulary settings with more categories and more challenging semantic distinctions: compared with the best recent result on each dataset in the table, CoDR improves ADE847, ADE150, and PC459 by 1.4, 0.6, and 0.6 mIoU, respectively. This is consistent with our motivation to improve large-vocabulary generalization through complementary representations.

References

[1] InfoCLIP: Bridging Vision-Language Pretraining and Open-Vocabulary Semantic Segmentation via Information-Theoretic Alignment Transfer. AAAI 2026.

[2] LoGoSeg: Integrating Local and Global Features for Open-Vocabulary Semantic Segmentation. AAAI 2026.

[3] PCA-Seg: Revisiting Cost Aggregation for Open-Vocabulary Semantic and Part Segmentation. CVPR 2026.

[4] Reconstructing CLIP for Open-Vocabulary Dense Perception. CVPR 2026.

[5] HyperCLIP++: Fine-tuning CLIP for Open-vocabulary Semantic Segmentation in Hyperbolic Space. TPAMI 2026.

Q3: Do the improvements come from DINO or from CoDR's design?

The original Table 2 includes relevant controls, but we did not clearly explain the sources of the gains. We now add OpenAI CLIP-L controls without DINO; the remaining rows use EVA02-CLIP-L:

ConfigurationAdditional encoderADE847ADE150PC459PC59VOC20
OpenAI CLIP-L, CAT-SegNone16.037.923.863.397.0
OpenAI CLIP-L, CoDRNone16.438.124.263.597.1
Shared baseline (Table 2 A)None17.940.325.863.597.8
A + CSD (Table 2 C)None18.441.626.663.797.9
C + Cost RD (Table 2 D)None19.041.426.663.398.1
Shared EVA02-CLIP + DINOv3 (Table 2 B)DINOv3-L/1618.541.526.763.897.8
Full CoDR, DINOv2 control (new)DINOv2-L/1419.642.127.164.198.0
Full CoDR, submitted configuration (Table 2 F)DINOv3-L/1619.842.127.364.297.9

With OpenAI CLIP-L and no DINO, CoDR improves all five metrics over CAT-Seg, including +0.4 on both ADE847 and PC459. This provides direct evidence of gains without DINO.

With EVA02-CLIP-L and no DINO, adding CSD raises ADE847 from 17.9 to 18.4, and further adding Cost RD raises it to 19.0. The shared EVA+DINOv3 reference scores 18.5, compared with 19.8 for full CoDR, leaving a gap of 1.3. The gains therefore cannot be attributed entirely to DINO.

The added DINOv2 control replaces only DINOv3, using the same DINOv2 version as DeCLIP while keeping all other settings unchanged. It achieves 19.6 on ADE847, showing that the result does not depend on using the newer DINOv3.

Thank you very much for the authors' thoughtful responses. After carefully reviewing their clarifications, I fully recognize the significance of this work in open-vocabulary semantic segmentation. Therefore, I maintain my positive rating and recommend acceptance of the paper.

AuthorsOpenReview ↗

Thank You for Your Feedback

Thank you for carefully considering our responses and for recommending our paper for acceptance. We appreciate your thoughtful feedback, which helped us clarify our evaluation and explain our findings more clearly.

Reviewer_1VdpOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Complementary Dense Representations (CoDR) for training-based open-vocabulary semantic segmentation. They point out the limitation of existing cost aggregation module in CAT-Seg that one dense visual representation is simultaneously utilized into semantic matching and spatial guidance and it can be collided with each other. The authors tried to separate these representations and generate several complementary representations for cost aggregation. Finally, the proposed method achieves the state-of-the-art performance on several benchmark datasets.

Critical strengths
  1. The paper points out the issues between semantic matching and spatial guidance. These issues can easily be overlooked, but the authors focus on the cost aggregation process in detail.

  2. The experimental analysis is well-designed, and the authors demonstrate good performance on several benchmark datasets compared to existing OVSS methods.

Critical weaknesses
  1. The proposed complementary representations in Cost Representation Diversification. Are they really provide different information or just ensemble effect?? It is difficult to understand the design of complementary representations. It needs more robust and proper reasons and analysis about them.

  2. The performance of the proposed method is based on Dino-v3 and similar baseline, DeCLIP, uses Dino-v2. The gain of CoDR can be considered from ability of new vision foundation model. Fair comparison is required.

Questions for authors

As I already mentioned, I am wondering two points of the paper, 1) reasonable purpose of complementary representations in representation diversification and 2) gain from new and more powerful vision foundation model.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 1Vdp

Dear Reviewer 1Vdp,

Thank you for recognizing our research motivation and experimental analysis. We address your concerns about representation complementarity and the DINO version separately below.

Q1: Do the representations provide complementary information, or are the gains simply an ensemble effect?

CoDR constructs different dense representations from the same CLIP backbone, using local content, self-similarity relations, CLS residuals, and FFN refinement to change the information used for image–text matching:

RepresentationConstructionPreference rate on ADE847 (original Table 4)
r₁Local value projection + CLS + FFN12.8%
r₂Q–Q/K–K self-similarity over the complete CLS–patch sequence59.0%
r₃r₂ + CLS16.8%
r₄Further FFN refinement of r₂ + CLS11.5%

These representations share the backbone, aggregator, and decoder, rather than being complete segmentation models trained independently. After excluding ties, each representation performs best on some samples, supporting their suitability for different inputs. Appendix Table 11 also shows that CLS and FFN affect text alignment and regional coherence differently:

RepresentationText similarity marginWithin-region patch cosine similarity
r₂0.160.71
r₃ (+CLS)0.170.74
r₄ (+CLS+FFN)0.150.75

Appendix Figure 8 also illustrates complementarity in the final segmentation masks. With the spatial representations, aggregator, and decoder fixed in the same model, switching only the cost representation makes r₁ or r₃ perform best on different windowpane examples, and r₂ or r₄ on different floor examples. These examples illustrate scene-dependent complementarity in the final predictions.

In the same jointly trained model, retaining only one cost representation while keeping the other modules unchanged gives 18.3–18.8 on ADE847, below the 19.8 obtained by joint composition. With the representations fixed, mean fusion in both the cost and spatial representation spaces achieves 19.4, whereas SARC achieves 19.8. These results support representation nonredundancy and the role of scene-adaptive composition.

Q2: Are the gains over DeCLIP mainly due to DINOv3?

We added a direct replacement control: we replace only DINOv3-L/16 in CoDR-L with DINOv2-L/14, the same version used as DeCLIP's teacher, while keeping the other settings unchanged:

ViT-L settingAdditional modelADE847ADE150PC459PC59VOC20
DeCLIPDINOv2-L/14†17.640.725.963.997.7
CoDRDINOv2-L/1419.642.127.164.198.0
CoDRDINOv3-L/1619.842.127.364.297.9

† Denotes a distillation teacher.

After switching to DINOv2, all five CoDR scores differ from the DINOv3 version by at most 0.2 and remain higher than DeCLIP. This indicates that the gains cannot be attributed entirely to the newer DINOv3. We additionally provide OpenAI CLIP controls without DINO and separately report the shared EVA02-CLIP baseline with DINO:

CLIP and settingAdditional encoderADE847ADE150PC459PC59VOC20
OpenAI CLIP-L, CAT-SegNone16.037.923.863.397.0
OpenAI CLIP-L, CoDRNone16.438.124.263.597.1
EVA02-CLIP-L, shared baseline (Table 2 A)None17.940.325.863.597.8
EVA02-CLIP-L, shared baseline + DINO (Table 2 B)DINOv3-L/1618.541.526.763.897.8

With OpenAI CLIP, CoDR without DINO improves all five metrics. Adding DINO to the shared EVA02-CLIP baseline gives 18.5 on ADE847, compared with 19.8 for complete CoDR. Together with the no-DINO representation controls above, these results show that the representation design itself also contributes to the gains.

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns. In particular, the DINOv2 replacement experiment substantially alleviates my original concern regarding DINOv3.

However, two methodological concerns remain.

Q1. Complementarity vs. conventional feature fusion

In response to my original Q1, the authors demonstrate that different representations exhibit distinct behaviors and that combining them improves segmentation performance. However, such improvements can also arise from conventional feature diversification or ensembling.

It remains unclear why the specific constructions involving self-similarity, CLS residuals, and FFN refinement are necessary, rather than simpler feature combinations. Could the authors further clarify what distinguishes the proposed representation diversification from conventional multi-feature fusion?

Q2. Further clarification on representation composition

In response to my original Q1, the authors also highlight the effectiveness of SARC. However, its empirical gains appear relatively limited despite being one of the paper's central contributions.

Table 5 reports only a +0.4 mIoU improvement over mean fusion on ADE847 (19.4 → 19.8), without equivalent comparisons on the other four benchmarks. Furthermore, Table 2 shows marginal improvements on ADE150 (+0.1), PC459 (+0.2), and PC59 (+0.1), with a slight degradation on VOC20 (−0.1).

As acknowledged in the authors' response to Reviewer TUbh's Q4, the additional random-seed analysis covers only configurations A and B, excluding the E/F comparison needed to assess SARC's contribution. Therefore, it remains unclear whether these small improvements are consistent across random seeds and sufficiently justify the proposed composition mechanism.

Additionally, the paper identifies category-dependent representation preferences as a motivation for RD (Secs. 1 and 3.3), whereas SARC employs image-level scalar weights shared across categories and spatial locations. Could the authors clarify how such scene-level weighting effectively exploits category-dependent complementarity when multiple categories favoring different representations coexist within an image?

Overall assessment

Although the rebuttal substantially addresses my concern regarding DINOv3, I remain unconvinced that the proposed representation diversification and scene-adaptive composition provide sufficiently justified methodological contributions beyond conventional feature fusion.

Therefore, I maintain my original rating of Weak rejection (2).

AuthorsOpenReview ↗

Response to Reviewer 1Vdp Q1(a)

Thank you for reviewing our work again and for recognizing that the DINOv2 replacement experiment helped address your original concern. Your follow-up questions helped us clarify our design and motivated the additional experiments below.

Q1(a). Why these dense representations, and how are they complementary?

Open-vocabulary segmentation requires each patch to retain visual evidence while also matching class text. CLIP is pretrained to align whole images with text. MaskCLIP [1], SCLIP [2], CorrCLIP [3] and ClearCLIP [4] explored how to construct dense representations from its final layer. Their findings motivated us to examine how patch relations, CLS and the feed-forward network (FFN) affect the patch-to-text alignment already present in CLIP.

Self-correlation determines where a patch draws visual evidence from. MaskCLIP [1] showed that a patch's value projection already contains local semantics that can be matched to text. SCLIP [2] found that weighting values across positions by query–query and key–key similarity improves localization. The value projection preserves the response at the current position, while self-correlation lets visually similar patches reinforce one another across parts of an object.

CLS adds image-level semantics to local evidence. CLIP aggregates image information in CLS and aligns it with text. CorrCLIP [3] found that adding a region-specific class token back to the corresponding patches helps incorporate regional semantics. In our ADE150 experiment, explicitly adding the image-level CLS to the self-correlation representation increased the text-score margin between classes present and absent in the image from 0.16 to 0.17 (Appendix Table 11).

The FFN changes how each patch represents the same evidence. The pretrained FFN applies a nonlinear channel transformation at each position, which can change similarity to class text. In ClearCLIP's ablation [4], removing the FFN reduced average segmentation performance when the final-layer residual was retained, but improved it when the residual was removed. We also observed a trade-off. Applying the FFN after adding CLS increased mean within-region patch cosine similarity from 0.74 to 0.75, while the text-score margin fell from 0.17 to 0.15 (Appendix Table 11).

This raised a question: Do these changes benefit all zero-shot classes, or do different classes favor different representations? We explored two paths, local value projection (V) and self-correlation. On each, we tested the base representation, then added CLS, then the FFN, giving six candidates. Fig. 1(c) shows that different classes favor different representations. Appendix Fig. 9 illustrates their effects in patch feature space. We therefore construct complementary representations of the same patch that emphasize local visual evidence, relations between patches, or image-level semantics. When matching a patch to class text, the model can draw on these complementary cues to make fuller use of CLIP's existing image–text alignment.

In addition to Table 11, we compared the initial six candidates, which include r₁–r₄, with the combinations below. Some representations provide redundant information, and adding more does not consistently improve performance. This motivated our choice of r₁–r₄. To test your suggestion that conventional feature diversification could explain the gains, we added three learnable expert projections to r₁ and fused their outputs with r₁ (first row). This setting scored lower on ADE847, ADE150, PC459 and PC59, even falling below baseline C in Table 2 on those four benchmarks. This suggests that simply adding learned features may encourage overfitting at the expense of zero-shot performance.

RepresentationsADE847ADE150PC459PC59VOC20
r₁ + 3 expert projections18.141.226.363.198.1
Initial six19.041.726.763.697.9
r₁–r₄ + Q–Q/K–K18.641.326.763.597.9
r₁–r₄19.241.426.663.398.2

References

[1] Zhou, Loy and Dai. Extract Free Dense Labels from CLIP. ECCV 2022.

[2] Wang, Mei and Yuille. SCLIP: Rethinking Self-Attention for Dense Vision-Language Inference. ECCV 2024.

[3] Zhang, Liu and Tang. CorrCLIP: Reconstructing Patch Correlations in CLIP for Open-Vocabulary Semantic Segmentation. ICCV 2025.

[4] Lan et al. ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inference. ECCV 2024.

AuthorsOpenReview ↗

Response to Reviewer 1Vdp Q1(b)&Q2

Q1(b). How does our approach differ from conventional feature fusion?

Conventional fusion learns weights from the segmentation loss on training classes. Those weights may transfer poorly to unseen classes and scenes. This motivated Scene-Adaptive Representation Composition (SARC) for Representation Diversification (RD). RD constructs multiple patch representations for text alignment and multiple spatial representations. For zero-shot transfer, SARC learns weights that favor the representations best suited to each input scene. Fig. 6 shows how these weights vary across images.

To test this design, we extended Table 5 to compare uniform averaging (Mean), global learnable weights (Learnable), and SARC. All three perform similarly on COCO, the training domain. On the zero-shot benchmarks, SARC gains over both alternatives on ADE847, ADE150, PC459 and PC59.

Extended Table 5. Fusion methods compared by mIoU

SettingFusionCOCOADE847ADE150PC459PC59VOC20
Cost onlyMean19.041.426.663.398.1
Cost onlyLearnable19.041.526.463.498.1
Cost onlySARC19.241.826.963.798.0
Spatial onlyMean19.141.826.863.798.0
Spatial onlyLearnable19.041.826.763.898.1
Spatial onlySARC19.442.027.064.198.0
BothMean52.719.442.027.064.198.0
BothLearnable52.819.341.927.064.097.9
BothSARC52.619.842.127.364.297.9

We also split ADE847 by whether a class was seen during training.

SettingAllSeenUnseen
Both + Learnable19.335.916.8
Both + SARC19.836.017.5

Relative to Learnable, SARC gains about 0.1 mIoU on seen classes and 0.7 on unseen classes within the same benchmark.

Q2(a). SARC's contribution and consistency across random seeds

Thank you for noting that Table 5 covered only ADE847. Q1(b) now reports all five benchmarks. As the A/B seed analysis did not directly test SARC, we compared E and F from Table 2 on the same three seeds. SD is the sample standard deviation.

SettingSeedADE847ADE150PC459PC59VOC20
E5851659519.2641.9827.0864.1597.97
E5851659619.2042.0226.9864.1797.84
E5851659719.3141.8627.1164.0698.03
EMean±std19.26±0.0641.95±0.0827.06±0.0764.13±0.0697.95±0.10
F5851659519.7842.0927.2664.2497.86
F5851659619.7142.3827.4364.2097.98
F5851659719.8442.2027.1564.2498.14
FMean±std19.78±0.0742.22±0.1527.28±0.1464.23±0.0297.99±0.14
F−EMean±std+0.52±0.01+0.27±0.14+0.22±0.21+0.10±0.08+0.05±0.14

Adding SARC improved the first four benchmarks for all three seeds. The ADE847 gains (+0.52, +0.51 and +0.53) are consistent across runs. Extended Table 5 also shows that SARC benefits the cost and spatial branches individually. On ADE847, using SARC in both gives a larger gain over Mean (+0.4) than using it in either branch alone (+0.2 for cost and +0.3 for spatial). Learnable performs similarly to Mean, suggesting that learning global fusion weights from the training classes alone does not yield the same zero-shot gains. The advantage of SARC is clearest on difficult, large-vocabulary benchmarks such as ADE847 and PC459.

VOC20 distinguishes only 20 foreground classes, including people, cars and dogs, for which COCO-Stuff already provides mask supervision. This leaves less room to improve class matching by adjusting semantic representations. Some remaining errors may concern mask boundaries, which may help explain why SARC shows no consistent gain on VOC20.

Q2(b). How can scene-level weights exploit class-dependent complementarity?

You raise an important limitation. When classes in the same image favor different representations, a shared scene-level weight cannot select the best mixture for each class individually. We use scene-level weights because the classes likely to occur together vary with the scene, as with cabinets and countertops in kitchens, or vehicles and roads on streets. The image thus provides a cue to which representation may work better. During training, SARC learns this overall preference from the segmentation losses of two candidates on each image. Even when classes disagree, it can favor the representation that helps the image as a whole.

Class identity alone also does not fully determine preference. In the two examples of “floor” in Appendix Fig. 8, r₂ gives the highest IoU in one and r₄ in the other. This illustrates the value of conditioning on the image. Appendix Table 10 explores predicting weights from class text alone. Jointly conditioning on image context and class text could allow each class in the image to receive its own weights when preferences conflict.

We hope these clarifications and additional experiments address your concerns, and we welcome your further feedback.

Reviewer_TUbhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that cost-aggregation methods for open-vocabulary segmentation (OVS), such as CAT-Seg, use one dense visual feature both to build the cost map and to provide spatial guidance, which creates competition between semantic discrimination and spatial coherence. CoDR addresses this in three steps:

  • Cost Spatial Decoupling (CSD) separates the cost and spatial representations through an asymmetric CLS–patch attention mask.
  • Representation Diversification (RD) builds four cost constructions and combines CLIP spatial features with DINOv3 features.
  • Scene-Adaptive Representation Composition (SARC) predicts per-image fusion weights, supervised by which branch yields the lower loss.

The paper reports 19.8 mIoU on ADE-847 with a ViT-L/14 backbone and claims state-of-the-art results on five benchmarks.

Critical strengths
  1. The gradient-conflict diagnostics (App. A.2) and the representation-preference analyses (Table 4, App. A.5) make the motivation concrete.
  2. The component ablation (Table 2) includes a shared EVA02-CLIP + DINOv3 reference (row B), which helps separate the effect of DINO from that of the proposed modules.
  3. Inference overhead is reported (Table 13).
Critical weaknesses
  1. The contribution is incremental over PCA-Seg (CVPR 2026), which the paper cites but does not discuss. PCA-Seg starts from the same diagnosis: serial cost aggregation "induces knowledge interference between class-level semantics and spatial structural information". Its remedy follows the same three-step recipe as CoDR:

    • Decouple the semantic and spatial pathways. PCA-Seg runs spatial and class aggregation in parallel and adds an orthogonality loss between the two streams. CoDR separates cost and spatial representations (CSD).
    • Produce multiple complementary branches. PCA-Seg uses four expert blocks. CoDR uses four cost constructions and two spatial sources (RD).
    • Fuse them with learned, input-adaptive weights. PCA-Seg's coefficient mapper predicts pixel-specific softmax weights over the experts. CoDR's SARC predicts scene-level weights.

    The main difference is where decoupling is applied: at the encoder representations rather than in the aggregation stage. The paper even evaluates a patch-wise softmax composition (App. A.4, Table 10), which closely resembles PCA-Seg's coefficient mapper, without mentioning PCA-Seg. PCA-Seg receives one sentence in the related work, and the paper does not explain how CoDR differs in substance. The individual ingredients are also adapted from prior work: Q–Q/K–K self-similarity for r2 from SCLIP, DINO-based spatial knowledge from DeCLIP, and CAT-Seg's aggregation unchanged. Overall, CoDR reads as a recombination of known components within an existing framework.

  2. The main comparison does not disclose the backbones, and the headline claims are overstated.

    • Table 1 lists CoDR simply as "ViT-B/16" and "ViT-L/14", and the main text states only that "ablations pair EVA02-CLIP-L/14@336 with DINOv3-L". Appendix Table 6 shows that the main results use EVA02-CLIP-B/16 and EVA02-CLIP-L/14@336, plus a DINOv3-B/16 or DINOv3-L/16 encoder at inference.
    • None of the CAT-Seg-family baselines in Table 1 uses an additional encoder at inference. CAT-Seg itself uses OpenAI CLIP.
    • The paper's own ablation shows that the backbone accounts for the advantage over prior work. In Table 2, the shared EVA02-CLIP reference (row A) has no CoDR component and already reaches 17.9 / 40.3 / 25.8 / 63.5 / 97.8. That is above CAT-Seg (16.0 / 37.9 / 23.8 / 63.3 / 97.0) and H-CLIP (16.5 / 38.4 / 24.2 / 64.1 / 97.7), and above DeCLIP's 17.6 on ADE-847.

    The claimed "+2.2 mIoU over the strongest prior result" therefore cannot be attributed to CoDR. The method's own contribution over its baseline is +1.9 on ADE-847, of which adding DINOv3 alone accounts for +0.6 (row B). No matched-backbone ViT-B reference is reported at all. App. A.8 reports a "CAT-Seg+DINO" configuration only for timing, not for accuracy. Statements such as "achieves state-of-the-art performance across five standard OVS benchmarks", "the best results among the compared methods on all five benchmarks" (ViT-B/16), and "the larger gains on ADE847, ADE150, and PC459 suggest that complementary representations are particularly beneficial" are not supported once the backbone and the extra encoder are accounted for.

  3. The most directly related method is omitted from the comparison and outperforms CoDR. PCA-Seg (CVPR 2026) uses the same CAT-Seg framework, training data and benchmarks, with no auxiliary encoder at inference. It reports, for A-847 / PC-459 / A-150 / PC-59 / VOC-20:

    • ViT-B/16: 16.1 / 22.3 / 38.1 / 62.2 / 97.3, against CoDR's 15.8 / 22.6 / 36.8 / 61.5 / 97.1. PCA-Seg is higher on four of five benchmarks.
    • ViT-L/14: PCA-Seg is also higher on PC-59 (64.7 vs 64.2) and VOC-20 (98.4 vs 97.9).

    Since the paper cites PCA-Seg, its omission from Table 1 is hard to justify. With it included, the claim of best results on all five benchmarks at ViT-B/16 and on all four non-ADE-847 benchmarks at ViT-L/14 no longer holds.

  4. The baselines stop at 2025. Every method in Table 1 was published in 2025 or earlier (ZSSeg, ODISE, FC-CLIP, EBSeg, CAT-Seg, DeCLIP, Mask-Adapter, H-CLIP). Besides PCA-Seg, recent CAT-Seg-based work under the same protocol includes DenseRC ("Reconstructing CLIP for Open-Vocabulary Dense Perception", CVPR 2026). DenseRC uses DINOv2 only as a training-time teacher and reports 15.9 / 22.7 / 37.6 / 61.3 on A-847 / PC-459 / A-150 / PC-59 at ViT-B/16, above CoDR on three of the four. I am not asking for state-of-the-art results. However, the paper's central claims are comparative, and they cannot be assessed without the relevant recent methods, especially ones that reach similar or higher accuracy without an extra encoder at inference.

  5. Component gains are small and come from single runs. Several steps in Table 2 change results by 0.1–0.5 mIoU, some negatively. For example, Cost RD lowers A-150 and PC-59, and SARC changes VOC-20 by −0.1. The composition strategies in Table 10 differ by at most 0.4 on ADE-847. No variance is reported, so it is unclear which components are reliably beneficial.

Questions for authors
  1. Please state the CLIP weights, input resolution and any auxiliary encoder for every row of Table 1. Please also report CoDR with OpenAI CLIP and without DINOv3, matching CAT-Seg.
  2. What does CAT-Seg (OpenAI CLIP and EVA02-CLIP) + DINOv3 spatial guidance achieve in accuracy? App. A.8 reports this configuration only for timing.
  3. How does CoDR differ conceptually from PCA-Seg, beyond decoupling at the representation level rather than the aggregation level? How does it compare under matched backbones?
  4. Can you report the matched-backbone shared baseline for ViT-B/16, and the variance over seeds for the Table 2 ablation?
4 comments
AuthorsOpenReview ↗

Response to Reviewer TUbh(Q1&Q2)

Dear Reviewer TUbh,

Thank you for your careful review and specific suggestions. We respond to the four questions in turn below.

Q1: CLIP weights, input sizes, additional encoders, and matched controls using OpenAI CLIP without DINO.

Thank you for pointing out that the sources of the weights, input sizes, and additional encoders were not clearly disclosed. We agree that this information is necessary to assess whether the comparisons are fair. We therefore summarize the settings of several key methods in Table 1 and provide results for CoDR using OpenAI CLIP without DINO:

ViT-B/16

MethodCLIP weightsDense representationsCLIP inputAdditional encoderADE847ADE150PC459PC59VOC20
CAT-SegOpenAI CLIP-B/16-384×384—12.031.819.057.594.6
CAT-Seg + EVA-CLIPEVA02-CLIP-B/16r₁384×384—14.435.821.260.096.6
DeCLIPDeCLIP (EVA02-CLIP-B/16)-384×384DINOv2-B/14†15.336.321.460.696.6
CoDREVA02-CLIP-B/16r₁–r₄384×384DINOv3-B/1615.836.822.661.597.1

ViT-L/14

MethodCLIP weightsDense representationsCLIP inputAdditional modelADE847ADE150PC459PC59VOC20
CAT-SegOpenAI CLIP-L/14@336-336×336—16.037.923.863.397.0
CoDR, without DINOOpenAI CLIP-L/14@336r₁–r₄336×336—16.438.124.263.597.1
CAT-Seg + EVA-CLIP (shared, Table 2 A)EVA02-CLIP-L/14@336r₁336×336—17.940.325.863.597.8
CAT-Seg + EVA-CLIPEVA02-CLIP-L/14@336r₂336×336—17.540.325.663.697.8
DeCLIPDeCLIP (EVA02-CLIP-L/14@336)-336×336DINOv2-L/14†17.640.725.963.997.7
CoDR, DINOv2EVA02-CLIP-L/14@336r₁–r₄336×336DINOv2-L/1419.642.127.164.198.0
CoDR, DINOv3EVA02-CLIP-L/14@336r₁–r₄336×336DINOv3-L/1619.842.127.364.297.9

† denotes a distillation teacher.

Our method uses CLIP models from the same family and at the same scale as DeCLIP, PCA-Seg, and DenseRC, all of which also incorporate information from external models such as DINO. Therefore, the use of information from an external model does not, in itself, make a direct comparison unfair. To rule out the influence of using DINOv3 rather than DINOv2, we add a DINOv2 version of CoDR: only DINOv3 is replaced, while all other settings remain unchanged. This version achieves 19.6 / 42.1 / 27.1 / 64.1 / 98.0, further supporting the effectiveness of our method.

You also noted that row A exceeds DeCLIP on ADE847. Here, we clarify how the representations are constructed: row A in the original Table 2 uses a shared r₁. To align with DeCLIP's construction, we provide an EVA02-CLIP-L + r₂ control, which achieves 17.5 / 40.3 / 25.6 / 63.6 / 97.8, slightly below DeCLIP on the first four benchmarks. Therefore, the higher score in row A is not achieved by using a stronger VLM to improve the baseline. Different dense representations from the same model behave differently, which is also a key motivation for this work. This also explains our decision not to directly adopt DeCLIP weights: distillation targeting a single dense representation may weaken the capabilities and complementarity of other representations.

We additionally provide a control using OpenAI CLIP-L without DINO. CoDR improves over CAT-Seg on all five benchmarks, with ADE847 and PC459 each improving by 0.4, providing further evidence that the method can improve performance without an additional DINO encoder.

Q2: The accuracy of CAT-Seg with the two CLIP variants after adding DINOv3 spatial guidance.

As requested, we report the performance of the two CLIP variants after adding DINO:

Shared representation settingADE847ADE150PC459PC59VOC20
OpenAI CLIP-L + DINOv316.238.124.063.697.1
EVA02-CLIP-L + DINOv3 (Table 2 B)18.541.526.763.897.8

Both CLIP variants improve after adding DINO, but this does not mean that all gains from our method can be attributed to it. On ADE847, EVA+DINO achieves 18.5, compared with 19.8 for the complete CoDR model; the path using CSD and Cost RD without DINO also improves from 17.9 to 18.4 and 19.0. Meanwhile, the OpenAI CLIP + CoDR results in Q1 also show improvements without DINO. These controls indicate that the gains cannot be attributed entirely to DINO.

In fact, the EVA02-CLIP + DINOv3 control was already reported in Table 2 B. We apologize that the explanation in the paper was unclear and caused confusion; we will revise it accordingly.

AuthorsOpenReview ↗

Response to Reviewer TUbh(Q3&Q4)

Q3: Conceptual differences from PCA-Seg and comparisons using the same backbone.

Thank you for carefully examining the relationship between PCA-Seg and our work. The original manuscript did not discuss this relationship sufficiently. We will clarify the following differences:

  1. The targets of modeling and decoupling differ. PCA-Seg operates on cost volumes that have already been constructed, decoupling spatial aggregation from class aggregation to alleviate interference between these two aggregation processes. CoDR operates on the dense representations supplied to the aggregator, decoupling cost representations for image-text matching from representations for spatial guidance. This alleviates the conflicting objectives that arise when the same feature must support both semantic discrimination and spatial consistency modeling.

  2. The sources of complementary information differ. PCA-Seg obtains complementary aggregation results through different aggregation experts. CoDR constructs different dense representations from the same CLIP encoder and combines spatial representations from CLIP and DINO. CoDR therefore focuses on complementarity in the input representations themselves, rather than differences between aggregation experts.

  3. The composition mechanisms and supervision differ. PCA-Seg uses a learnable module to predict pixel-wise weights for fusing the outputs of different aggregation experts. SARC instead constructs preference supervision from the per-image segmentation losses of the branches, learning scene-level composition weights that adaptively exploit the complementary representations already constructed by RD.

The original manuscript keeps the CAT-Seg aggregator fixed to focus on evaluating representation design. Because the two methods operate at different levels, they can also be combined. In response to your suggestion, we will include the performance comparison with PCA-Seg and further report results from integrating PCA-Seg aggregation modules into CoDR:

MethodAggregatorADE847ADE150PC459PC59VOC20
CoDR-BCAT-Seg15.836.822.661.597.1
CoDR-BPCA-Seg16.137.022.861.897.2
CoDR-LCAT-Seg19.842.127.364.297.9
CoDR-LPCA-Seg19.942.427.464.498.1

Here, we use only PCA-Seg's aggregation modules, without its additional losses. Replacing the aggregator improves all five results at both B and L scales, supporting the compatibility of the two methods and showing that CoDR's representation design can benefit from another aggregation approach.

We also agree that omitting relevant methods from 2026 was inappropriate, and thank you for pointing out specific methods for comparison. We will improve the comparisons and discussion in the revised manuscript.

Q4: The shared baseline with a matched ViT-B backbone and variance across random seeds for the Table 2 ablation.

We first provide the shared r₁ baseline with ViT-B and CoDR without DINO:

EVA02-CLIP-B settingADE847ADE150PC459PC59VOC20
Shared r₁ baseline14.435.821.260.096.6
CoDR, w/o DINO15.436.722.360.196.9

These results show that our method is also effective in the ViT-B ablation setting.

Regarding random seeds, we share your concern. However, repeating all six settings in Table 2 with two additional seeds each would require 2×6=12 additional complete training runs. Given the additional experiments in Q1–Q3, limited computing resources, and the response deadline, we cannot complete random-seed experiments for all settings. We therefore first add two seed runs for each of A and B to examine the sensitivity of these two settings to the random seed:

SettingSeedADE847ADE150PC459PC59VOC20
A5851659517.9240.2825.8163.4797.82
A5851659617.8440.1525.8463.3997.91
A5851659717.9840.4125.7263.6697.75
AMean ± std17.91 ± 0.0740.28 ± 0.1325.79 ± 0.0663.51 ± 0.1497.83 ± 0.08
B5851659518.4641.4926.6963.8297.85
B5851659618.3641.4026.8163.8597.79
B5851659718.6041.6926.6363.9297.89
BMean ± std18.47 ± 0.1241.53 ± 0.1526.71 ± 0.0963.86 ± 0.0597.84 ± 0.05

The maximum cross-seed ranges across the five datasets are 0.27 for A and 0.29 for B, with sample standard deviations of 0.05–0.15. These results indicate that the two settings exhibit small cross-seed fluctuations and consistent main performance trends, and establish the validity of the results in Table 2.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer TUbh,

Thank you again for your thoughtful review. We wanted to follow up on our response and the additional experiments. When you have a chance, we would appreciate hearing whether they address your concerns. We would be happy to clarify any points that remain open.

Dear Reviewer TUbh,

Thank you for your time and thoughtful reviews. Please be reminded that the author-reviewer discussion period ends on November 18.

Since the authors have procvided their feedback, please review the feedback and actively participate in the discussion. If you have any remaining questions or concerns, please raise them before the deadline so the authors have sufficient time to respond. Thanks.

Best regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.