- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates representational convergence between vision and language models in dense patch-level representations, extending prior work on global image and sentence embeddings. Using images with pixel-level annotations, the authors demonstrate substantial alignment between dense visual representations and corresponding text representations, with stronger alignment observed in larger and more recent models. They further show that stronger intrinsic alignment facilitates linear cross-modal mapping, improves generalization to unseen categories, and benefits downstream tasks, including cross-modal retrieval, text-guided semantic segmentation, and zero-shot image captioning.
Critical strengths
- The paper investigates the alignment of learned concepts across models from different modalities.
- The paper conducts experiments across multiple models.
- The paper employs multiple representation similarity metrics in its experiments.
Critical weaknesses
- The authors lack sufficient discussion of relevant prior work. For example, VL-SAE [1] investigates vision-language semantic alignment in VLMs, while PatchSAE [2] studies the interpretability of patch embeddings.
- The proposed method lacks novelty, as it relies solely on a learnable linear mapping to align representations, which is a fairly general and widely used approach.
- The experiments do not compare the proposed method against any existing approaches. For example, in semantic segmentation, the authors report only the performance of their own method, completely overlooking existing methods.
- The paper is limited to representational analysis and does not demonstrate how the insights gained could be leveraged to improve existing models or suggest concrete directions for such improvements.
[1] Shen, S., Sun, J., Huang, Q., & Wang, S. (2026). Vl-sae: Interpreting and enhancing vision-language alignment with a unified concept set. Advances in Neural Information Processing Systems, 38, 45235-45265.
[2] Lim, H., Choi, J., Choo, J., & Schneider, S. (2025, May). Sparse autoencoders reveal selective remapping of visual concepts during adaptation. In International Conference on Learning Representations (Vol. 2025, pp. 24444-24469).
Questions for authors
Given the lack of discussion and comparisons with existing work, I have serious doubts about the novelty and significance of this work. Furthermore, the paper reads more like an analysis report of internal model representations, and its contributions appear insufficient for acceptance at ICLR.
4 comments
On weakness 3. We would like to recall that the scope of the paper is not proposing a novel method, but rather to disclose novel findings regarding dense representational convergence. Nevertheless, we compared our mapping in the segmentation task with 11 different methods, and the results was already available in Appendix A.8.2. We report the table below for an easier visualization of the results, in which our full mapping solution outperforms previous methods. Once again, our aim is not to compete with tailored architectures developed for this specific task. Nonetheless, we did further experiments with Talk2Dino, a specific architecture for Semantic Segmentation. This results in an improvement of the performance thanks to a careful selection of a text-vision encoder pair with higher intrinsic alignment, showing the positive implications of our study. See details in the next answer.
| Model | Vision Encoder | Context | Stuff | City | ADE | Avg |
|---|---|---|---|---|---|---|
| MaskCLIP | CLIP | 12.4 | 8.8 | 11.5 | 7.2 | 10.0 |
| SCLIP | CLIP | 25.2 | 17.6 | 21.3 | 10.9 | 18.7 |
| ClearCLIP | CLIP | 29.6 | 19.9 | 27.9 | 15.0 | 23.1 |
| NACLIP | CLIP | 32.1 | 21.4 | 31.4 | 17.3 | 25.5 |
| dino.txt | DINOv2(reg) | 30.9 | 20.9 | 32.1 | 20.6 | 26.1 |
| FreeDA | DINOv2 | 35.4 | 24.2 | 32.3 | 19.4 | 27.8 |
| FreeDA | CLIP+DINOv2 | 39.7 | 26.3 | 33.6 | 21.4 | 30.2 |
| ProxyCLIP | CLIP+DINOv2(reg) | 36.2 | 24.6 | 35.2 | 21.6 | 29.4 |
| ProxyCLIP | CLIP+DINO | 37.7 | 25.6 | 40.1 | 22.6 | 31.5 |
| Talk2DINO | DINOv2(reg) | 39.1 | 27.0 | 35.8 | 21.1 | 30.7 |
| Our Open-Vocabulary | DINOv3 - QWEN3-Embeddings-8B | 35.3 | 19.7 | 35.7 | 24.5 | 28.8 |
| Our Full-Mapping | DINOv3 - QWEN3-Embeddings-8B | 44.1 | 30.4 | 46.8 | 31.3 | 38.1 |
Details of the experiment can be found in the Appendix.
Nevertheless, we would like to highlight again that this paper does not aim to propose a novel mapping method to align representations, but rather to show that dense representations converge and that their alignment predicts performance on tasks that require vision and text to work together.
On weakness 4. We thank the Reviewer for giving us the opportunity to clarify the importance of our work. We agree that understanding representational convergence becomes particularly valuable when it can inform the design and improvement of existing vision-language systems.
Indeed, to investigate this, we conducted additional experiments using Talk2DINO, an existing open-vocabulary segmentation framework, reported in the new Appendix A.9. Rather than relying on our linear mapping, we retain Talk2DINO original two-layer projection, training objective, and evaluation setting. We fix the DINOv3 ViT-L visual encoder and evaluate 18 independently trained text encoders, keeping both encoders frozen and retraining only the projection layers under the same training conditions.
Across these 18 encoder configurations, we find that intrinsic alignment measured before mapping strongly correlates with both the final mapping loss and downstream segmentation performance. Specifically, linear CKA achieves Spearman correlations of with final InfoNCE loss and with mean mIoU across four segmentation benchmarks. The other two alignment metrics show consistent trends, with CKNNA and k-NN overlap achieving correlations of and with mean mIoU, respectively.
| Encoder pair | PASCAL-Context | COCO-Stuff | Cityscapes | ADE20K | Mean |
|---|---|---|---|---|---|
| DINOv2 ViT-L + CLIP text | 42.69 | 29.63 | 38.47 | 22.94 | 33.43 |
| DINOv3 ViT-L + CLIP text | 44.41 | 33.76 | 40.48 | 24.77 | 35.86 |
| DINOv3 ViT-L + Qwen3-Embedding-8B | 45.76 | 33.21 | 40.64 | 25.96 | 36.39 |
Notably, Qwen3-Embedding-8B, which exhibits the strongest intrinsic alignment, also achieves the highest average mIoU among the 18 retrained configurations (36.39). For comparison, the released Talk2DINO configurations using DINOv2–CLIP and DINOv3–CLIP achieve 33.43 and 35.85 mIoU, respectively.
These results demonstrate a concrete practical implication of our analysis: intrinsic alignment can serve as an informative criterion for selecting compatible pretrained encoders within existing vision-language architectures. Rather than choosing encoder combinations solely through downstream trial and error, pre-mapping alignment can help prioritize promising configurations before training their cross-modal connections.
Thank the authors for their response. I now better understand the authors’ claim after reading the rebuttal. However, the Platonic Representation Hypothesis has already suggested underlying correspondences between visual and textual representations. As the authors emphasize the observed phenomenon as their main contribution, this work appears to merely provide further empirical validation of an existing hypothesis at a finer granularity (patch-level), without offering fundamentally new insights.
Furthermore, I have concerns about the experiments presented in the rebuttal and Appendix A.8.2. The proposed method aligns two separate encoders using task-specific datasets, such as ADE20K and MSCOCO, giving the aligned model access to domain-specific training data for segmentation and image caption. Therefore, the comparisons in the segmentation and zero-shot image caption appear potentially unfair. This experimental setup seems closer to conventional task-specific fine-tuning than to learning general vision-language alignment that enables broad multimodal capabilities. Given these concerns, the experimental results do not appear to provide sufficiently strong evidence to support the authors' claims.
We thank the Reviewer for the follow-up and for taking the time to consider our rebuttal. We would like to further address the concerns regarding novelty and experimental supervision, hoping that these clarifications will help the Reviewer better understand the contributions of our paper.
Novelty and scientific contribution. The Reviewer correctly notes that the Platonic Representation Hypothesis suggests correspondences between visual and textual representations, but at the global representation level, as already acknowledged in our paper. Crucially, global and dense representational convergence address two distinct scientific questions, as they concern fundamentally different levels of visual representation. While the PRH primarily investigates the former, our work focuses on the latter, demonstrating that independently trained vision and language models develop shared semantic geometry at the level of spatially grounded concepts. This finding advances our understanding of learned representations and, importantly, has practical implications for downstream tasks such as cross-modal retrieval, semantic segmentation, and image captioning, as demonstrated by our experiments. In our controlled patch-versus-CLS experiment (Appendix A.7) demonstrates that granularity matters: with Qwen3-Embedding-8B, kNN overlap increases from 0.40 to 0.51, and CKNNA from 0.48 to 0.63 when using patch-based rather than global CLS prototypes.
Theoretical connection. The PRH also suggests that better-aligned representations should be easier to connect, leaving this intuition without a formal demonstration. We take a step further by providing a precise theoretical characterization in Section 3.2 and Appendix A.2: for centered, equal-rank, whitened representations, the minimum normalized linear reconstruction error satisfies . This exact result, under clearly stated assumptions, complements our empirical analysis and provides a mathematical foundation for the relationship between representational similarity and linear recoverability.
Segmentation experiments. We also address the Reviewer's concern about the results in Appendix A.8.2. The Reviewer states that our approach gives "...the aligned model access to domain-specific training data...". However, Appendix A.8.2 explicitly distinguishes our "Open-Vocabulary" setting, which excludes evaluation categories when fitting the mapping, from "Full-Mapping", which includes their visual prototypes. Full-Mapping illustrates what a simple linear connector can achieve when target-category prototypes are available; it is not intended as a supervision-matched claim of superiority over specialized open-vocabulary architectures.
We emphasize that the central finding of our segmentation experiments (and experimental section more broadly) is not achieving state-of-the-art performance with a linear mapping, but demonstrating a strong positive correlation between intrinsic encoder alignment and downstream segmentation performance (Figure 6).
Moreover, as discussed in our previous response, we directly investigated practical implications using Talk2DINO rather than our own linear mapping (Appendix A.9). We retained its original projection architecture, training objective, training strategy, and inference procedure. Thus, concerns regarding fairness in the experimental pipeline are substantially reduced in this setting. Addressing the concern about new insights, selecting a more aligned encoder pair improves upon the original released Talk2DINO configuration by 2.96 mIoU percentage points. This demonstrates a concrete practical implication of our findings within an existing architecture, without modifying its training or inference procedure.
Captioning experiments. Regarding zero-shot image captioning, the mapping is learned from segmentation/object category annotations in training splits, not from image–caption pairs. In this context, we believe the Reviewer's concern about the fairness of the captioning experiments may stem from a misunderstanding of the evaluation protocol. Evaluation is performed on MS COCO and Flickr8k validation data. The caption generator remains frozen and receives only a textual summary derived from mapped visual patches; neither the generator nor the mapping is trained against captioning targets (Section 4.5, Appendix A.10).
We strongly believe that our work investigates a novel and meaningful phenomenon in representation learning, offering both scientific insights and practical implications that are highly relevant to the international representation learning community, whose research interests lie at the heart of ICLR. We hope these additional clarifications help address the Reviewer's remaining concerns. We remain available for further discussion and would welcome any additional questions or suggestions that could help us further strengthen our contribution.
On weakness 1. Thanks for pointing these papers out, we have included a discussion about them in the related work section of the revised manuscript. The aim of our paper is to explore dense representational convergence in independently trained models. The works the Reviewer suggests refer to CLIP-style models, in which two encoders are jointly trained through cosine similarity and their alignment is measured according to this metric, therefore, their scope is substantially different from our work. Moreover, [1] focuses on global feature alignment, while we focus on patch-level features. Despite the substantially different scope, we included the suggested papers in the related work section as suggested by the Reviewer.
On weakness 2. We would like to thank the Reviewer for giving us the opportunity to better clarify the novelty of our work. Indeed, we do not aim at proposing a novel mapping between two spaces, but rather showing novel insights on dense representational convergence and, more importantly, on the implications that such convergence has on task performance.
More in detail, while the Platonic Representation Hypothesis investigates the emergence of shared semantic structures at the level of global representations, our work explores a complementary and fundamental phenomenon: dense representational convergence. We show that independently trained vision and language models develop remarkably compatible semantic geometries even at the patch level, suggesting that cross-modal convergence is not confined to global representations but emerges throughout the semantic structure of the visual world. More importantly, we show that representational convergence is not merely an emergent phenomenon, but a key determinant of cross-modal learnability: stronger intrinsic alignment predicts how effectively two independently trained models can be connected, how much supervision they require, and how well they generalize to unseen concepts. Given these insights, we deliberately select the simplest method to connect two spaces, which is linear mapping, to further highlight how much the two spaces are compatible. Indeed, the novelty does not lie in the mapping itself, but in the implications it enables.
Together, our novel findings suggest a new perspective on multimodal learning, where increasingly powerful unimodal foundation models can be connected through lightweight mappings that exploit their naturally emerging semantic compatibility. Rather than treating cross-modal alignment solely as something to be learned through joint training, our results highlight the potential of leveraging dense semantic structures that independently trained models already share.