acceptodds
Under review as a conference paper at ICLR 2027

From Global to Dense Representational Convergence: Shared Semantic Geometry Simplifies Vision-Language Connection

Abstract

Recent work on representational convergence suggests that independently trained vision and language models can develop similar semantic geometries despite differing modalities and training objectives. Existing studies, however, have mainly examined global image and sentence embeddings. In this paper, we ask whether this convergence also emerges in the dense patch representations of vision transformers. Using images with detailed pixel annotations, we capture how a broad range of concepts is organized in the dense vision space and compare this structure with the corresponding text representations. Interestingly, we find substantial alignment at this level as well, with consistently stronger alignment for larger and more recent vision and language models. Furthermore, we learn linear mappings between the vision and text representation spaces to exploit the consequences of this compatibility. Better-aligned spaces are easier to connect: their mappings are more accurate, require fewer training concepts, and perform better in downstream tasks. Specifically, stronger intrinsic alignment is associated with greater generalization to held-out categories in cross-modal retrieval, higher mIoU in text-guided semantic segmentation, and higher scores in zero-shot image captioning. Altogether, these findings extend evidence of representational convergence from global embeddings to spatially grounded concepts and link intrinsic alignment to easier cross-modal mapping and better downstream performance.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Representation Learning, Deep Neural Networks, Multimodal AI, Platonic Representation Hypothesis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Z6AqOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates representational convergence between vision and language models in dense patch-level representations, extending prior work on global image and sentence embeddings. Using images with pixel-level annotations, the authors demonstrate substantial alignment between dense visual representations and corresponding text representations, with stronger alignment observed in larger and more recent models. They further show that stronger intrinsic alignment facilitates linear cross-modal mapping, improves generalization to unseen categories, and benefits downstream tasks, including cross-modal retrieval, text-guided semantic segmentation, and zero-shot image captioning.

Critical strengths
  1. The paper investigates the alignment of learned concepts across models from different modalities.
  2. The paper conducts experiments across multiple models.
  3. The paper employs multiple representation similarity metrics in its experiments.
Critical weaknesses
  1. The authors lack sufficient discussion of relevant prior work. For example, VL-SAE [1] investigates vision-language semantic alignment in VLMs, while PatchSAE [2] studies the interpretability of patch embeddings.
  2. The proposed method lacks novelty, as it relies solely on a learnable linear mapping to align representations, which is a fairly general and widely used approach.
  3. The experiments do not compare the proposed method against any existing approaches. For example, in semantic segmentation, the authors report only the performance of their own method, completely overlooking existing methods.
  4. The paper is limited to representational analysis and does not demonstrate how the insights gained could be leveraged to improve existing models or suggest concrete directions for such improvements.

[1] Shen, S., Sun, J., Huang, Q., & Wang, S. (2026). Vl-sae: Interpreting and enhancing vision-language alignment with a unified concept set. Advances in Neural Information Processing Systems, 38, 45235-45265.

[2] Lim, H., Choi, J., Choo, J., & Schneider, S. (2025, May). Sparse autoencoders reveal selective remapping of visual concepts during adaptation. In International Conference on Learning Representations (Vol. 2025, pp. 24444-24469).

Questions for authors

Given the lack of discussion and comparisons with existing work, I have serious doubts about the novelty and significance of this work. Furthermore, the paper reads more like an analysis report of internal model representations, and its contributions appear insufficient for acceptance at ICLR.

4 comments

On weakness 1. Thanks for pointing these papers out, we have included a discussion about them in the related work section of the revised manuscript. The aim of our paper is to explore dense representational convergence in independently trained models. The works the Reviewer suggests refer to CLIP-style models, in which two encoders are jointly trained through cosine similarity and their alignment is measured according to this metric, therefore, their scope is substantially different from our work. Moreover, [1] focuses on global feature alignment, while we focus on patch-level features. Despite the substantially different scope, we included the suggested papers in the related work section as suggested by the Reviewer.

On weakness 2. We would like to thank the Reviewer for giving us the opportunity to better clarify the novelty of our work. Indeed, we do not aim at proposing a novel mapping between two spaces, but rather showing novel insights on dense representational convergence and, more importantly, on the implications that such convergence has on task performance.

More in detail, while the Platonic Representation Hypothesis investigates the emergence of shared semantic structures at the level of global representations, our work explores a complementary and fundamental phenomenon: dense representational convergence. We show that independently trained vision and language models develop remarkably compatible semantic geometries even at the patch level, suggesting that cross-modal convergence is not confined to global representations but emerges throughout the semantic structure of the visual world. More importantly, we show that representational convergence is not merely an emergent phenomenon, but a key determinant of cross-modal learnability: stronger intrinsic alignment predicts how effectively two independently trained models can be connected, how much supervision they require, and how well they generalize to unseen concepts. Given these insights, we deliberately select the simplest method to connect two spaces, which is linear mapping, to further highlight how much the two spaces are compatible. Indeed, the novelty does not lie in the mapping itself, but in the implications it enables.

Together, our novel findings suggest a new perspective on multimodal learning, where increasingly powerful unimodal foundation models can be connected through lightweight mappings that exploit their naturally emerging semantic compatibility. Rather than treating cross-modal alignment solely as something to be learned through joint training, our results highlight the potential of leveraging dense semantic structures that independently trained models already share.

On weakness 3. We would like to recall that the scope of the paper is not proposing a novel method, but rather to disclose novel findings regarding dense representational convergence. Nevertheless, we compared our mapping in the segmentation task with 11 different methods, and the results was already available in Appendix A.8.2. We report the table below for an easier visualization of the results, in which our full mapping solution outperforms previous methods. Once again, our aim is not to compete with tailored architectures developed for this specific task. Nonetheless, we did further experiments with Talk2Dino, a specific architecture for Semantic Segmentation. This results in an improvement of the performance thanks to a careful selection of a text-vision encoder pair with higher intrinsic alignment, showing the positive implications of our study. See details in the next answer.

ModelVision EncoderContextStuffCityADEAvg
MaskCLIPCLIP12.48.811.57.210.0
SCLIPCLIP25.217.621.310.918.7
ClearCLIPCLIP29.619.927.915.023.1
NACLIPCLIP32.121.431.417.325.5
dino.txtDINOv2(reg)30.920.932.120.626.1
FreeDADINOv235.424.232.319.427.8
FreeDACLIP+DINOv239.726.333.621.430.2
ProxyCLIPCLIP+DINOv2(reg)36.224.635.221.629.4
ProxyCLIPCLIP+DINO37.725.640.122.631.5
Talk2DINODINOv2(reg)39.127.035.821.130.7
Our Open-VocabularyDINOv3 - QWEN3-Embeddings-8B35.319.735.724.528.8
Our Full-MappingDINOv3 - QWEN3-Embeddings-8B44.130.446.831.338.1

Details of the experiment can be found in the Appendix.

Nevertheless, we would like to highlight again that this paper does not aim to propose a novel mapping method to align representations, but rather to show that dense representations converge and that their alignment predicts performance on tasks that require vision and text to work together.

On weakness 4. We thank the Reviewer for giving us the opportunity to clarify the importance of our work. We agree that understanding representational convergence becomes particularly valuable when it can inform the design and improvement of existing vision-language systems.

Indeed, to investigate this, we conducted additional experiments using Talk2DINO, an existing open-vocabulary segmentation framework, reported in the new Appendix A.9. Rather than relying on our linear mapping, we retain Talk2DINO original two-layer projection, training objective, and evaluation setting. We fix the DINOv3 ViT-L visual encoder and evaluate 18 independently trained text encoders, keeping both encoders frozen and retraining only the projection layers under the same training conditions.

Across these 18 encoder configurations, we find that intrinsic alignment measured before mapping strongly correlates with both the final mapping loss and downstream segmentation performance. Specifically, linear CKA achieves Spearman correlations of ρ=−0.959\rho = −0.959 with final InfoNCE loss and ρ=0.895\rho = 0.895 with mean mIoU across four segmentation benchmarks. The other two alignment metrics show consistent trends, with CKNNA and k-NN overlap achieving correlations of ρ=0.897\rho = 0.897 and ρ=0.913\rho = 0.913 with mean mIoU, respectively.

Encoder pairPASCAL-ContextCOCO-StuffCityscapesADE20KMean
DINOv2 ViT-L + CLIP text42.6929.6338.4722.9433.43
DINOv3 ViT-L + CLIP text44.4133.7640.4824.7735.86
DINOv3 ViT-L + Qwen3-Embedding-8B45.7633.2140.6425.9636.39

Notably, Qwen3-Embedding-8B, which exhibits the strongest intrinsic alignment, also achieves the highest average mIoU among the 18 retrained configurations (36.39). For comparison, the released Talk2DINO configurations using DINOv2–CLIP and DINOv3–CLIP achieve 33.43 and 35.85 mIoU, respectively.

These results demonstrate a concrete practical implication of our analysis: intrinsic alignment can serve as an informative criterion for selecting compatible pretrained encoders within existing vision-language architectures. Rather than choosing encoder combinations solely through downstream trial and error, pre-mapping alignment can help prioritize promising configurations before training their cross-modal connections.

Thank the authors for their response. I now better understand the authors’ claim after reading the rebuttal. However, the Platonic Representation Hypothesis has already suggested underlying correspondences between visual and textual representations. As the authors emphasize the observed phenomenon as their main contribution, this work appears to merely provide further empirical validation of an existing hypothesis at a finer granularity (patch-level), without offering fundamentally new insights.

Furthermore, I have concerns about the experiments presented in the rebuttal and Appendix A.8.2. The proposed method aligns two separate encoders using task-specific datasets, such as ADE20K and MSCOCO, giving the aligned model access to domain-specific training data for segmentation and image caption. Therefore, the comparisons in the segmentation and zero-shot image caption appear potentially unfair. This experimental setup seems closer to conventional task-specific fine-tuning than to learning general vision-language alignment that enables broad multimodal capabilities. Given these concerns, the experimental results do not appear to provide sufficiently strong evidence to support the authors' claims.

We thank the Reviewer for the follow-up and for taking the time to consider our rebuttal. We would like to further address the concerns regarding novelty and experimental supervision, hoping that these clarifications will help the Reviewer better understand the contributions of our paper.

Novelty and scientific contribution. The Reviewer correctly notes that the Platonic Representation Hypothesis suggests correspondences between visual and textual representations, but at the global representation level, as already acknowledged in our paper. Crucially, global and dense representational convergence address two distinct scientific questions, as they concern fundamentally different levels of visual representation. While the PRH primarily investigates the former, our work focuses on the latter, demonstrating that independently trained vision and language models develop shared semantic geometry at the level of spatially grounded concepts. This finding advances our understanding of learned representations and, importantly, has practical implications for downstream tasks such as cross-modal retrieval, semantic segmentation, and image captioning, as demonstrated by our experiments. In our controlled patch-versus-CLS experiment (Appendix A.7) demonstrates that granularity matters: with Qwen3-Embedding-8B, kNN overlap increases from 0.40 to 0.51, and CKNNA from 0.48 to 0.63 when using patch-based rather than global CLS prototypes.

Theoretical connection. The PRH also suggests that better-aligned representations should be easier to connect, leaving this intuition without a formal demonstration. We take a step further by providing a precise theoretical characterization in Section 3.2 and Appendix A.2: for centered, equal-rank, whitened representations, the minimum normalized linear reconstruction error satisfies ET→V∗=1−CKA(QT,QV)E^*_{T\rightarrow V}=1-\mathrm{CKA}(Q_T,Q_V). This exact result, under clearly stated assumptions, complements our empirical analysis and provides a mathematical foundation for the relationship between representational similarity and linear recoverability.

Segmentation experiments. We also address the Reviewer's concern about the results in Appendix A.8.2. The Reviewer states that our approach gives "...the aligned model access to domain-specific training data...". However, Appendix A.8.2 explicitly distinguishes our "Open-Vocabulary" setting, which excludes evaluation categories when fitting the mapping, from "Full-Mapping", which includes their visual prototypes. Full-Mapping illustrates what a simple linear connector can achieve when target-category prototypes are available; it is not intended as a supervision-matched claim of superiority over specialized open-vocabulary architectures.

We emphasize that the central finding of our segmentation experiments (and experimental section more broadly) is not achieving state-of-the-art performance with a linear mapping, but demonstrating a strong positive correlation between intrinsic encoder alignment and downstream segmentation performance (Figure 6).

Moreover, as discussed in our previous response, we directly investigated practical implications using Talk2DINO rather than our own linear mapping (Appendix A.9). We retained its original projection architecture, training objective, training strategy, and inference procedure. Thus, concerns regarding fairness in the experimental pipeline are substantially reduced in this setting. Addressing the concern about new insights, selecting a more aligned encoder pair improves upon the original released Talk2DINO configuration by 2.96 mIoU percentage points. This demonstrates a concrete practical implication of our findings within an existing architecture, without modifying its training or inference procedure.

Captioning experiments. Regarding zero-shot image captioning, the mapping is learned from segmentation/object category annotations in training splits, not from image–caption pairs. In this context, we believe the Reviewer's concern about the fairness of the captioning experiments may stem from a misunderstanding of the evaluation protocol. Evaluation is performed on MS COCO and Flickr8k validation data. The caption generator remains frozen and receives only a textual summary derived from mapped visual patches; neither the generator nor the mapping is trained against captioning targets (Section 4.5, Appendix A.10).

We strongly believe that our work investigates a novel and meaningful phenomenon in representation learning, offering both scientific insights and practical implications that are highly relevant to the international representation learning community, whose research interests lie at the heart of ICLR. We hope these additional clarifications help address the Reviewer's remaining concerns. We remain available for further discussion and would welcome any additional questions or suggestions that could help us further strengthen our contribution.

Reviewer_YqTdOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds on prior observations that the image and text representations converge even when trained independently. However, if prior work showed this for global representations, this work argues that the same happens between dense patch representations of images and text. The authors find that vision and language models that were not trained together show alignment according to CKK, kNN overlap, and CKNNA metrics. The alignment scores increase as the models get bigger. The authors find the mapping between two modalities based on concept representations and use it for downstream tasks. The alignment scores correlate with better cross-modal retrieval even for held-out concepts, better text-guided image segmentation, and better image captioning.

Critical strengths

The authors extend the interesting work on representational convergence from global image representation to dense patch representation, addressing the apparent gap. The authors use both global and local alignment metrics in their experiments. This is important because reporting only local alignment could inflate the results [1].

[1] Gröger, Fabian, Shuo Wen, and Maria Brbic. "Revisiting the platonic representation hypothesis: An Aristotelian view, 2026." https://arxiv.org/abs/2602.14486

Critical weaknesses

The authors do not provide baselines and meaningful comparisons. We see an increase in alignment metrics in Figure 2. However, without appropriate baselines, it is hard to tell how good or bad the alignment is. Increases in alignment scores for larger and more recent models are not informative by themselves. Similarly, the authors show a correlation between the alignment and downstream task scores. However, how much better is the downstream performance for better-aligned models (they show this for segmentation but not for retrieval and image captioning)? How much of it comes from models being bigger, rather than being aligned?

The significance and implications of the work are not clear. Even if dense patch representations and separately trained text representations converge, it is not clear what insights or practical implications can be taken away from the paper. For example, what does the work inform us about the learned representations or new method design? I believe discussing these would strengthen the paper's significance.

The clarity of the paper could be improved further. For example, the organization follows the standard methods (Section 3) and results (Section 4) structure, but some results are reported in Section 3 (lines 221-229), and some methodological details are reported in Section 4.1. Figure 1 has the same numbered annotation in different parts of the figure. It's not clear which number or annotation refers to what. Overall, I believe the authors could provide illustrations and examples of how obtaining prototypes, text-guided segmentation, or zero-shot image captioning work in their setting.

Questions for authors
  1. Lines 210 and 214: Could the authors explain the difference between 'local structural agreement' in kNN and 'local geometric agreement' in CKNNA?
  2. Line 228: the sentence 'It is precisely these features that facilitate the learning of a cross-modal mapping' seems not supported to me. Could the authors cite relevant work, explain why this is the case, or indicate whether it's a hypothesis?
  3. The authors use the term 'intrinsic alignment' throughout the paper. Could the authors define it early in the methods section?
  4. 'Exploring dense representational convergence' is a vague title for a section that introduces the concept of prototypes and alignment metrics.
7 comments

W1.1 On Alignment Baselines: We thank the Reviewer for highlighting the need for an interpretable alignment baseline. According to this suggestion, we compare the observed scores in Figure 2 with a random-pairing baseline obtained from 200 permutations of the visual–textual concept correspondence, preserving each modality internal geometry. We report the results here:

Text encoderMetricObservedRandom baselineObserved / Random
Qwen3-Embedding-0.6BCKA0.5040.0687.4×
CKNNA0.3420.00557.7×
kNN overlap0.2900.00929.7×
Qwen3-Embedding-4BCKA0.5470.0886.2×
CKNNA0.3860.00567.6×
kNN overlap0.3320.00934.3×
Qwen3-Embedding-8BCKA0.5660.0866.6×
CKNNA0.3980.00570.8×
kNN overlap0.3410.00935.3×

Across all metrics and model configurations evaluated in this experiment, the observed alignment scores substantially exceed their corresponding random-pairing baselines. This provides strong evidence that the alignment between visual and textual representation spaces is neither trivial nor merely the result of chance correspondences.

Importantly, these results provide a clear reference for interpreting Figure 2, showing that the measured alignment is meaningful rather than just a numerical increase. Moreover, the large difference between the least and most aligned model pairs (CKA = 0.203 for DINOv1–BERT-mini vs. 0.566 for DINOv3–Qwen3-Embedding-8B) highlights how strongly alignment depends on the choice of models.

W1.2 On alignment vs donwstream

We thank the Reviewer for giving us the opportunity to provide more information about this point. To complement Figures 4, 6, and 7, the tables below report absolute retrieval, segmentation, and captioning scores for selected encoder pairs and explicitly quantify the corresponding differences.

All pairs use DINOv3 ViT-L/16 as the visual encoder, and CKA is measured before learning the cross-modal mapping.

Cross-modal retrieval (Figure 4).

Text encoderCKAR@1R@5R@10
BERT-mini0.34711.9927.5434.67
Qwen3-Embedding-0.6B0.50429.2957.9265.84
Qwen3-Embedding-4B0.54736.9965.9974.42
Qwen3-Embedding-8B0.56738.4466.7975.44

Choosing which text model pair with DINOV3 is crucial; R@1 increases from 11.99% (BERT-Mini vs DINOv3) up to 38.44% (Qwen3-Embeddings-8B). More than three times more.

Semantic segmentation (Figure 6). Scores are mIoU in percent.

Text encoderCKAPASCAL-ContextCOCO-StuffCityscapesADE20K
BERT-mini0.34728.6015.4129.9517.02
Qwen3-Embedding-0.6B0.50441.7927.8140.3128.05
Qwen3-Embedding-4B0.54743.8529.9046.1030.62
Qwen3-Embedding-8B0.56744.1130.4046.7831.27

Under the same Semantic Segmentation pipeline here BERT-mini text signals brings to averaged final mIoU = 22.74 across four tested datasets. QWEN3-Embeddings-8B pipeline = 38.14% (+15,4%).

Zero-Shot Image captioning (Figure 7).

DatasetText encoderCKACIDEr
Flickr8kBERT-mini0.34713.41
Flickr8kQwen3-Embedding-0.6B0.50419.76
Flickr8kQwen3-Embedding-4B0.54721.10
Flickr8kQwen3-Embedding-8B0.56720.94
----
MS COCOBERT-mini0.34720.69
MS COCOQwen3-Embedding-0.6B0.50434.77
MS COCOQwen3-Embedding-4B0.54738.32
MS COCOQwen3-Embedding-8B0.56739.10

W1.3 on bigger model and better alignment: As the Reviewer correctly noted, model size can influence alignment metrics, as demonstrated by Gröger et al. (ICML 2026), who propose null calibration to account for this effect. Indeed, following their approach, we calibrated all three metrics; the results are reported in Appendix A.5, “Null-calibrated alignment metrics,” Figure 14. Even after calibration, newer and larger models continue to exhibit clearly stronger alignment, being consistent with our discussion.

W2 On implications of the works: We thank the Reviewer for giving us the opportunity to further clarify the implications of our work. In detail, the main implications of this work are:

  • Understanding representations: As the reviewer recognizes among the strengths, our work addresses a gap in the study of representational convergence by extending the analysis beyond whole-image embeddings. This provides a fuller understanding of the learned spaces, showing that shared semantic structure also emerges in category prototypes built from localized visual regions.
  • Connecting modalities: Better-aligned frozen encoders can be connected effectively with a simple linear map, requiring fewer training concepts to achieve strong retrieval performance.
  • Choosing encoders: Alignment can help identify compatible encoder pairs for tasks that require fine-grained image details and therefore rely on patch-level information.

To further investigate the latter point on encoder selection, we conducted additional experiments using Talk2DINO. We replaced its encoders, kept them frozen, and retrained the projection layers using the same training and evaluation setting. With DINOv3 fixed, across 18 text encoders, pre-mapping CKA strongly correlates with lower final loss (Spearman ρ\rho = -0.959) and higher mean mIoU across four datasets (ρ\rho = 0.895). CKNNA and kNN overlap show consistent relationships too.

Encoder pairPASCAL-ContextCOCO-StuffCityscapesADE20K
DINOv2 ViT-L + CLIP text42.6929.6338.4722.94
DINOv3 ViT-L + CLIP text44.4133.7640.4824.77
DINOv3 ViT-L + Qwen3-Embedding-8B45.7633.2140.6425.96

These results show how encoder selection can improve an existing method while keeping its training procedure unchanged. We clearly state this experiment in a dedicated section A.9 of the revised manuscript, thanks for the valuable suggestion.

W3.1 On clarity: To improve the clarity and textual flow of the manuscript, we have moved the discussion of representational convergence from methodological Section 3 (lines 221–229) to the experimental Section 4, integrating it into the existing discussion in Section 4.2. These changes are highlighted in the revised manuscript.

W3.2 on Fig 1: We have revised the figure to enhance its readability by removing the numbered descriptions from the bottom and incorporating the relevant text directly into the corresponding panels. This revision makes the relationship between the visual elements and their descriptions more explicit. The numbers in the figure correspond to the numbered research questions introduced in the Introduction and addressed throughout the paper.

W3.3 on more illustrations: We have added Figures 8, 16, and 18 to the Appendix of the revised manuscript to illustrate prototype construction, text-guided segmentation, and zero-shot image captioning, respectively. We appreciate this suggestion, which has helped us improve the clarity of the manuscript.

We appreciate the Reviewer’s suggestions for improving clarity, which have helped us make the manuscript clearer and easier to follow.

Q1 on Difference between k-NN Overlap and CKNNA: We thank the Reviewer for giving us the opportunity to further clarify this point. kk-NN overlap measures agreement in neighborhood membership, considering only the identities of shared neighbors. CKNNA additionally accounts for their centered similarity values. Consequently, two spaces can have identical neighbor sets and perfect kk-NN overlap without attaining maximal CKNNA. We have revised the metric descriptions to make the distinction explicit, lines 220-221 of the revised manuscript.

Q2 on Local similarity and alignment: We thank the Reviewer for pointing this out. Shared neighborhood structure provides relational information that can be exploited to connect representation spaces. For example, Wang and Mahadevan [1] learn projections by matching local geometry and preserving neighborhood relationships. In the vision–language setting, Maniparambil et al. [2] exploit shared similarity structure for cross-modal matching, while Li et al. [3] demonstrate that independently learned visual and textual concept representations can be connected through linear mappings. These findings support the role of shared semantic structure in cross-modal connection, consistent with Huh et al. [4], who explicitly discuss easier translation between aligned representations as an implication of representational convergence. We have revised the sentence to clarify this rationale and added these supporting references.

  1. Wang, C., et al. (2009). Manifold alignment without correspondence. IJCAI-09
  2. Maniparambil, M., et al. Do vision and language encoders represent the world similarly? CVPR24
  3. Li, J. et al. Do vision and language models share concepts? A vector space alignment study. TACL24
  4. Huh, M. et al. Position: The Platonic representation hypothesis. ICML24.

Q3 on Intrinsic alignment: We have added an explicit definition at the end of Section 3.1: intrinsic alignment denotes agreement between the similarity structures of visual and textual prototypes in their original representation spaces, measured using linear CKA, kk-NN overlap, and CKNNA before learning any cross-modal mapping.

Q4 on Subsection title: We have renamed Section 3.1 "Category Prototypes and Alignment Metrics'' to reflect its methodological content.

We truly thank the Reviewer once again for the careful review and valuable suggestions. The comments have helped us clarify our experimental results, make the practical implications of our findings more explicit, and improve the presentation of our methodology. We welcome any further questions or discussion.

Reviewer_VHM8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies alignment between independently trained vision and text models using image patches. The authors average labelled patch features and text embeddings into category prototypes, then learn linear maps between the two spaces. They report stronger alignment for larger and newer models, and that better-aligned pairs need fewer training categories for retrieval and perform better in text-guided segmentation and captioning.

Critical strengths

The question is clear and interesting: does the alignment between independently trained vision and text models also hold for dense patch features, and not only for global image embeddings. The prototypes are built at a large scale, from 7 segmentation datasets, 3,440 categories and about 332M patches (Table 1).

The study is broad in models and metrics. It uses 18 text encoders from 6 families and different sizes, three DINO generations, three DINOv3 sizes and three alignment metrics. The appendix adds checks for k (Fig. 12), the number of concepts (Fig. 13) and null-calibrated metrics (Fig. 14).

The relation between alignment and the downstream tasks is consistent. Retrieval, segmentation and captioning all improve with alignment, with Spearman correlations of 0.97 to 0.98 for retrieval (Sec. 4.3) and above 0.8 for segmentation and captioning (Figs. 6 and 7).

The method is simple. One linear map, without a decoder, gives 28.8 average mIoU in the open-vocabulary setting (Table 3), although with a newer backbone than the baselines.

Critical weaknesses

Please check the questions.

Questions for authors

correlation and the text encoder: The results show a strong association between alignment and performance, and most of the paper describes it this way ("is associated with", L25 and Fig. 1). A few sentences go further, for example "stronger initial alignment yields better cross-modal retrieval" (L102–103).

relation to prior work: Li et al. (TACL 2024) is cited only once (L113), but its setup is close to Sec. 3.2 and 4.3. They average image embeddings per concept, fit a linear map (orthogonal Procrustes) between vision and language spaces from concept pairs, evaluate retrieval of held-out concepts, and find that alignment grows with the size of the language model. Their visual representations are image-level, while this paper uses patches selected by pixel labels. It would help to discuss this difference, and an image-level baseline like theirs would show what the patch-level prototypes add.

segmentation setting: The main-text segmentation results (Sec. 4.4, Fig. 6) use a map fitted on the complete vocabulary (L425). This vocabulary includes the evaluation classes, and their visual prototypes come from the training annotations of the same datasets. A.6.2 states this clearly ("target-category supervision"), but Sec. 4.4 and the caption of Fig. 6 do not, so a reader may take Fig. 6 as transfer to new classes (L449–451). It would help to say this in Sec. 4.4.

Line 186 would benefit from a clearer definition of Pc.

3 comments

We sincerely thank the Reviewer for the thoughtful review and constructive comments. We appreciate the positive assessment of our research question, the breadth of our experiments, and the consistent relationship between alignment and downstream performance. Below, we address each concern and welcome further discussion to improve the clarity and quality of our manuscript.

On correlation and the text encoder: We thank the Reviewer for this comment. We understand that it may concern the distinction between association and causation in the wording of our findings. If this is the intended point, “yields” was meant to describe the empirical trend observed across the evaluated encoder pairs: pairs with stronger intrinsic alignment achieve better cross-modal retrieval performance. Nevertheless, we agree that this wording is imprecise given the evidence presented in the paper. We have therefore revised this and similar statements to describe the observed associations more accurately and avoid implying causality. However, since the comment does not include an explicit question, we may be missing the intended concern and we hope this clarification addresses the Reviewer’s concern. In any case, we would be happy to provide further responses if needed.

On relation to prior work: We thank the Reviewer for the opportunity to further discuss the connection with Li et al. We agree that this comparison deserves a more explicit discussion in the manuscript. Our studies share methodological elements, including the construction of concept-level prototypes, linear cross-modal mapping, and retrieval of held-out concepts. As the Reviewer notes, however, Li et al. study global image representations, whereas we investigate dense patch representations associated with spatially localized semantic categories. Indeed, our contribution is to examine whether representational convergence extends to this finer granularity and how alignment at this level relates to cross-modal mapping and downstream performance.

To further investigate this distinction, we conducted a controlled experiment comparing the alignment of patch-based and global CLS-based concept prototypes with text representations. We used DINOv3 ViT-L/16 on 13,621 COCO images containing exactly one annotated object instance, covering 79 concepts. Both types of visual prototypes were constructed from the same images and compared against Qwen3-Embedding-0.6B, 4B, and 8B text embeddings, using k = 10 for neighborhood metrics.

Text encoderCKA: PatchCKA: CLSkNN overlap: PatchkNN overlap: CLSCKNNA: PatchCKNNA: CLS
Qwen3-Embedding-0.6B0.79120.74710.49240.38990.61720.4824
Qwen3-Embedding-4B0.79100.81370.49490.40250.58950.4669
Qwen3-Embedding-8B0.80740.80970.51520.40380.63460.4865

Patch-based prototypes consistently achieve higher kNN overlap and CKNNA than CLS-based prototypes, while linear CKA shows no consistent advantage for either representation. These results highlight a distinction in representational granularity: although both representations exhibit global structural alignment with text, patch-based prototypes show stronger agreement in local semantic neighborhoods. This is consistent with object-selected patches emphasizing the target concept, whereas CLS representations also incorporate scene and background information. We have included this comparison and the discussion in Appendix A.6 of the revised manuscript, thanks again for the insightful comment.

On Segmentation setting: We thank the Reviewer for raising this point. The mapping used in Section 4.4 and Figure 6 is fitted on the complete vocabulary C, including all evaluation categories, whose visual prototypes are constructed from the training annotations of the corresponding datasets. We have now explicitly clarified this target-category supervision in Section 4.4 and in the caption of Figure 6, thanks!

On Line 186: We thank the Reviewer for pointing this out. PcP_c denotes the collection of normalized patch features retained for category c across all images and datasets. Accordingly, ∣Pc∣|P_c| is the total number of retained patches used to compute the category prototype. We have clarified this definition in Section 3.1.

We thank the Reviewer once again for the careful and constructive review. The comments and valuable suggestions have helped us clarify our claims, better contextualize our contribution, and improve the presentation of the paper. We welcome any further questions or discussion.

I thank the authors for the rebuttal and will keep my initial score.

We would like to thank the Reviewer for taking the time to read our responses. We are pleased to know that we have addressed all the Reviewer's concerns, and we particularly appreciate the positive assessment expressed regarding our manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.