acceptodds
Withdrawn from ICLR 2027

Dense Similarity as a Design Choice: Consistency and Utility in CLIP Readouts

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

CLIP's native image–text function does not uniquely determine the internal similarities used by dense readers. We turn this freedom into Learned G, a dense adaptation method that learns task-directed metrics within a family of native-equivalent Q/K implementations. Reciprocal query/key transforms preserve native attention while changing the self-similarities used for dense aggregation. The transforms fold into existing projections without online adapter multiplications. On ViT-B/16, G improves matched SCLIP by 3.49 mIoU points over learned per-head scalars on VOC. Removing isotropic scale retains gains after scalar recalibration; spectrum-preserving rotations reduce performance. Retraining on ViT-L/14 yields gains over per-head scalars of 8.13 points on VOC and 4.89 on a fixed COCO-Object panel. With a trained output head held fixed, learning G upstream adds 1.97 points on VOC, showing that metric adaptation can complement output-side adaptation. Canonicalization gives qualifying readers consistent predictions across equivalent implementations; metric–bias transport retains a selected task across coordinate changes. Dense supervision can thus improve segmentation without changing CLIP's native function.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
open-vocabulary semantic segmentation, vision-language models, CLIP, function-preserving adaptation, attention geometry, parameter-efficient fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_iwVfOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper observes that CLIP's native image-text function does not determine the Q/Q and K/K self-similarities used by training-free dense readers. Based on this, it proposes Learned G: a bounded low-rank per-head transform on the last visual block, trained with 256 VOC images and folded into the Q/K weights at no inference cost. It beats learned per-head scalars of ViT-B/16 and ViT-L/14 on VOC, and improves on top of a fixed trained output head. It also proposes a canonicalization that makes dense readers consistent across equivalent parameterizations, and a transport procedure that keeps a learned task metric under coordinate changes.

Critical strengths
  • The observation is interesting.
  • The method preserves CLIP's native function exactly, has very few parameters, and folds into existing weights with no inference overhead.
  • The control experiments are well designed and the results are reported carefully.
Critical weaknesses
  • Reader-only LoRA4 beats G by 1.72 mIoU on ViT-B/16 and by 6.5 on ViT-L/14 VOC. A rank-12 output head with a similar number of parameters beats G by 3.89. The only stated advantage of G is that it preserves CLIP's native function through the same Q/K weights. However, reader-only LoRA also preserves the native function by keeping the original path. So I think the motivation for G is weak.
  • Once 256 labeled VOC images are used, the method is no longer training-free, and it should be compared with supervised or few-shot open-vocabulary segmentation methods.
  • Evaluation uses VOC val, a custom 1000-image COCO-Object subset and partial ADE panels, instead of the standard full benchmarks. As a result, the numbers cannot be compared with the literature. Besides, the authors state that the VOC evaluation images were used in earlier experiments, and the main LoRA comparison (Table 3) was completed after earlier 512-image results were known, so it is post-hoc. With many studies, separately trained G baselines and different protocols per table, repeated iteration on the same evaluation set is a real risk of overfitting.
  • The writing is very hard to follow. For example, they used many coined acronyms and many separately run studies with different protocols and baselines.
Questions for authors

Please refer to weaknesses.

Reviewer_p6oJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper intoduces Learned G, in this authors adapt CLIP to dense prediction tasks while preserving its native image-text function. The main idea is to exploit the freedom in query/key parametrization by learning transformations that preserve native attention while changing the self-similarities used for dense aggregation. The transforms fold into existing projections without online adapter multiplications. The authors evaluates the method on ViT-B/16 and ViT-L/14 backbones. The results show improvements over global and per-head scalar adaptations of SCLIP and NACLIP. The authors also introduce canonicalization and metric–bias transport to maintain prediction consistency and task behavior across equivalent model parameterizations.

Critical strengths

1.The paper exploits the non uniqueness of quesry/key parametrizations to learn task specific similarity metrics while preserving CLIP's native attention computaion. The reciprocal Q/K formulation is mathematically well motivated, and folding the learned transformations into existing projections avoids additional online adapter multiplications. This provides a distinct approach to adapting pretrained representations for dense prediction. 2. The authors provides strong experimental evaluations across ViT-B/16 and ViT-L/14, with improvements over scalar baselines under matched training budgets. The ablation studies on metric structure and orientation, along with transfer experiments and comparisons with other adaptation methods gives better support for the proposed method.

Critical weaknesses
  1. The proposed method outperforms baselines but the reeader only LORA4 and standalone output-head adaptation achieve higher VOC segmentation accuracy (Table 3). While preserving CLIP's native function is a key contribution, the paper does not clearly demonstrate its practical advantage over these less constrained adaptation methods.

  2. While Learned G improves performance when added to a fixed output head, jointly training G and the output head does not gives statistically conclusive improvements. This suggests that the benefits of combining metric and output-side adaptation may depend on the training procedure.

Questions for authors

Given that reader-only LoRA4 and standalone output-head adaptation achieve higher VOC segmentation accuracy than Learned G (Table 3), can the authors demonstrate a practical setting where preserving CLIP's native function provides a clear advantage over these alternatives?

Reviewer_ffpkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This manuscript proposes a method called G that improves the semantic segmentation performance of CLIP without altering its original image-text matching function. The core idea involves applying reciprocal transformations to the queries and keys. These transformations preserve the original attention computation and enable the model to learn more appropriate internal similarities.

Critical strengths

The research motivation and initial research objectives of the manuscript are clear.

Critical weaknesses

W1: The proposed method appears closely related to SCLIP. Further theoretical analysis and direct experimental comparisons are needed to clarify its distinct contributions. W2: The figures, as well as the derivations of key formulas and descriptions of the methodology, are riddled with signs of AI generation. They are difficult to understand and lack any research value.

Questions for authors

The AI tools contribute to nearly every stage of the research process. Could the authors clarify which methodological ideas and technical contributions were independently developed by the research team? For other questions, refer to the weaknesses mentioned above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.