- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper argues that vision and language embeddings are not perfectly alignable because their embedding spaces are too different. It first distinguishes between functional and representational equivalence and alignment. Then it connects perfect linear alignability to analogies, similarity structures, and generalized analogies. Empirically, it finds that similarity structures are less aligned across vision and language models than between language models, and that textual analogies do not transfer perfectly to visual representations. Finally, it shows that training only the aligner often fails to generalize to classes not seen during training, and that representational alignment (mutual k-NN) correlates with performance on vision-language benchmarks.
Critical strengths
The paper provides some examples where vision and language embeddings differ. In particular, differences in the similarity structure and text analogies are quantified. Furthermore, representational alignment is connected to failures of VLMs.
Critical weaknesses
-
Many statements are strongly overstated given the empirical evidence:
- “we show that existing vision and language representations are linearly unalignable” (ll. 15-16), “existing representations are unalignable” (l. 45), “we provide evidence that existing vision and language representations are linearly unalignable” (ll. 457-458): The definition of alignability in the paper means perfect alignability (see e.g. Ziyin & Chuang 2025), where all points are perfectly mapped to the other space. Under the strict definition in the paper, word and sentence embeddings also couldn’t be aligned, as they don't have exactly the same distances across models.
- “This misalignment is an inherent property of the data and hence expected to persist regardless of either scale or the alignment’s complexity” (ll. 16-17), “alignment will likely not naturally emerge with scale” (l. 46): The paper compares the cosine similarities on COCO with DINOv3 base and all-MiniLM-L12-v2 and shows that “[f]or about 2% of COCO image/caption pairs, their cosine similarity difference exceeds [0.4]” (ll. 212-213). In addition, some qualitative samples from COCO are shown where these similarities are different, and the similarity is compared with language-language combinations. However, it is not shown to hold for any dataset, model, or scale. In contrast, the authors even note themselves that “the vision models’ retrieval accuracy increases with scale” (l. 467). It is not clear why this cannot further improve with a higher-quality dataset, more data, or better models.
- “The generalization failure described in the last section stems from representational misalignment.” (l. 405): This conclusion has no theoretical or empirical justification in the paper.
- “we demonstrate how this misalignment […] leads to a predictably induced generalization failure” (ll. 17-18), “This misalignment predictably leads to an induced generalization failure in vision-language models” (ll. 46-47): Only a correlation not a causality is shown in Sec. 5. Sec. 4 only shows that an adapter trained on some classes doesn’t generalize to unseen classes. The paper doesn’t build a predictive relationship between alignment quality and general generalization failures.
- “we demonstrate how this misalignment […] correlates with existing failure modes” (ll. 17-19): The specific failure modes are never discussed. The paper only shows that the performance correlates with the alignment score.
- “Overall, these results imply representations will not become aligned simply given sufficient scale, providing evidence against a strong version of the Platonic Representation Hypothesis.” (ll. 20-22), “a diverse set of VLM failures likely stem from representational misalignment and will not be resolved solely by increasing model scale” (ll. 51-52): This implication is not shown in the paper. Moreover, the platonic representation hypothesis doesn’t just discuss the convergence in terms of model size but also in terms of data size, which is not part of the paper.
This is also admitted in Appendix L: “Most results are circumstantial in supporting that many VLM failures likely arise due to representational misalignment. In this sense the results are consistent with the hypothesis but do not causally show that representational misalignment is what caused a failure.” (ll. 1342-1344)
-
The sections are not well connected. Sec. 2 introduces functional and representational equivalence and alignment without defining them properly. The only concept that is used subsequently is representational alignment. Sec. 3 discusses the connection of perfect linear alignability with analogies, similarity structures, and generalized analogies and shows empirically that similarity structures and analogies are not perfectly preserved across vision and language models. Sec. 4 discusses how a VLM performs on held-out data when no such data is used in training the adapter. The connection to alignment is mainly motivated by the fact that word embedding spaces can be aligned across languages on a subset of words and still perform well on held-out words. However, the analogue for this would be to align the vision and language embedding spaces instead of creating a generative VLM. Finally, Sec. 5 shows the correlation between mutual k-NN and downstream performance on VLM benchmarks. This is not connected sufficiently to previous sections. The theory in Sec. 3 uses the Gram matrix. However, mutual k-NN only consideres the overlap in the nearest neighbors and the causality that representational misalignment from Sec. 5 is the source of the problem of Sec. 4 is also not well explored. Altogether, the different sections seem detached.
Ziyin & Chuang 2025: “Proof of a perfect platonic representation hypothesis”
Questions for authors
- Most VLMs are end-to-end fine-tuned after initializing the adapter, and even a coarse adapter works well when fine-tuned (see Gröger et al. 2026). How is the generalization performance after fine-tuning the VLMs in Sec. 4?
- Can one show that misalignment in the vision and language model is the reason for the held-out results in Sec. 4?
- Which explicit failure modes come from representational misalignment?
- Why is CosSimRMSE introduced instead of comparing the established CKA score?
- For many vision and language models, the embeddings are normalized and lie on the unit sphere. Computing the class average and an analogy as A - B + C would move the vectors away from the manifold. Is anything done to keep the analogy on the unit sphere?
Gröger et al. 2026: “Rethinking Projector Training in Multimodal LLMs”
5 comments
W1.2…“alignment will likely not naturally emerge with scale” … However, it is not shown to hold for any dataset, model, or scale. In contrast, the authors even note themselves that “the vision models’ retrieval accuracy increases with scale” (l. 467). It is not clear why this cannot further improve with a higher-quality dataset, more data, or better models.
When referring to alignment not naturally emerging with scale, we refer specifically to full alignment. Partial alignment very well may and has been shown numerous times to improve with scale - we argue there is a ceiling to this alignment, namely due to inherent differences in information between modalities, and this ceiling cannot be breached with data/model scale. Specifically, this is a statement on certain data distributions - if one uses a dataset of book scans then the dataset contains effectively the same information as the words in those books, in which case a vision and language model may converge to the same representations. However, there are concepts which might be expressible in one modality but inexpressible in another, as exemplified in thought experiments like [Mary’sRoom].
We hope these and other, prior answers help clarify the stated weaknesses regarding accidental overstatements, and we will update the manuscript to make these sharper. If you feel like any are not well addressed, please do let us know.
W3: The sections are not well connected
This is a difficulty arising due to there being many different notions of representational similarity. At a high level, we aimed for different sections to touch on different notions of representational equivalence/alignment and functional equivalence/alignment, so the resulting statements would touch on a broad set of existing definitions and tests for representational similarity. We will make it clearer in the updated manuscript which section touches on what and how, which would hopefully make it clear how all sections effectively make statements about representational alignment (see the diagram in section 2).
We hope this answered your questions and helped clarify the paper. Thanks again for the feedback and pushback, we will use both to improve the paper.
References
[WhyWinoHard] Diwan, Anuj, et al. "Why is winoground hard? investigating failures in visuolinguistic compositionality."
[Mary’sRoom] Nida-Rümelin, Martine, and Donnchadh O Conaill. "Qualia: The knowledge argument."
Thank you for the insights and additional information.
- Q1: Fig. 3 (right) depicts the LLM as frozen. This seems to be wrong then. How is the performance before the fine-tuning? Are the concepts also left out for full finetuning?
- Q2: Comparing CLIP and a vision-only ViT is not a comparison that isolates misalignment as the reason for the held-out results because many things are changed between a CLIP ViT and a vision-only ViT. This is not enough evidence for the claim that misalignment causes the discrepancy. Moreover, the Gram matrices of the vision and language model for CLIP are not very similar (see e.g. Klebe et al., 2023 and Soares et al., 2026). So, CLIP might not be the best choice for a well-aligned model.
- Q3: This is a correlation. The PRH already showed that model size is correlated with alignment. If model size is correlated with a better performance on Winoground, VLMBias, VLMBlind, MMVP, the correlation between alignment and this performance would be trivial. Altogether, this correlation doesn’t show that representational misalignment is the reason for this failure modes. Much stronger results would be if the failures could be solved by improving the alignment without changing other components.
- Q4: CKA = 1 implies that the representations have the same similarity structure. CKA is the cosine similarity between the centered and vectorized Gram matrices. Therefore it is equal to 1 if and only if the centered Gram matrix are scaled versions of each other. Because they are centered, they can also be translated arbitrary. Both, centering and scaling, doesn’t change the distance structure. I don’t see how CosSimRMSE helps.
- W1.1: I ran this experiment on the standard MUSE setup (Conneau et al. 2018) with fastText wiki vectors (300 dim, top 200k words) on multiple language combinations (en-es, en-fr, en-de, en-ru, en-zh) and for all of them the residual even on the training data is non-zero. The relative residual is 0.56, 0.56, 0.59, 0.61, and 0.48 respectively. Therefore, even in the word translation setting the spaces are not perfectly linearly alignable. The same holds for the Gram matrices. So, even for this setting all theorems in the paper fail. This also holds for two English models (en-wiki to en-cc) with a relative residual of 0.34. Altogether, this shows that the paper acts in a very idealized setting that is irrelevant for practical scenarios as perfect alignment will probably never be reached but is also not required.
- W1.2: I agree. The original PRH already stated that concepts like “I believe in the freedom of speech” are present in only one modality and this is also part of multiple thought experiments. From the abstract and some parts of the paper, it is not clear that perfect linear alignment (with a zero residual) is meant with alignment and it could be misunderstood as that alignment cannot improve. It would be good to clarify this.
Would it be possible to specify exactly what you plan to update in the manuscript? You are allowed to upload an updated version of the paper. This would help to judge if the statements are corrected sufficiently.
Conneau, Alexis, et al. "Word translation without parallel data." ICLR 2018.
Klebe, Dustin, et al. "GeRA: Label-efficient geometrically regularized alignment." arXiv preprint arXiv:2310.00672 (2023).
Soares, Diogo, et al. "Scalable and Interpretable Representation Alignment with Ordinal Similarity." ICML 2026.
Thanks for the feedback, we clarify different points below and address your questions - going one by one:
Yes - note that the large scale experiments in section 4 already do finetune the LLM and the adapter together. LLaVA consists of an adapter only training stage and an adapter+LLM finetuning stage, which we replicate here. The given generalization performance already is post this finetuning.
Yes, there is evidence that better alignment can substantially reduce the generalization failure. When training the 7B LLaVA model using a CLIP backbone - which was already trained on joint image/text data, including over the held-out concepts - we find the held-out concept detection accuracy is about 61%, while for the vision-only ViT it is 21.5%. This does not test generalization as CLIP already got joint image/text supervision, but does show that when using better aligned backbones then the generalization failure is substantially reduced.
The failure in section 4 is the clearest example, with the left four in figure 5 being in the very least correlated if not induced from the representational misalignment. These include Winoground (matching semantically different but syntactically similar captions to their corresponding images), as also speculated by [WhyWinoHard], and VLMBias (detecting oddities in out of distribution images, e.g. number of legs on a five-legged lion). VLMBias is a nice example of how generalization failures like the one given in section 4 can happen more broadly, and we speculate that a range of similar out of distribution failures are, at least in part, due to representational misalignment. Many if not all of the failures in lines 417-424 seem to fit this category.
CosSimRMSE measures the extent to which two representations have the same similarity structure (Gram matrices) and hence the extent to which the conditions for theorem 3.5/corollary 3.6 hold/don’t hold respectively. CKA is a more restricted version thereof - having the same similarity structure implies CKA=1 but CKA=1 does not imply the representations have the same similarity structure. Thus, we choose to measure CosSimRMSE as it better corresponds to the relevant quantities for the given theorems.
Yes - concepts and analogies are renormalized to be on the unit sphere. Basically, the average of unit vectors is taken and then renormalized to be a unit vector, and A-B+C is also renormalized to be on the unit sphere. Thus, all comparisons are done over the same manifold.
Now, we would like to try clarifying some of the mentioned weaknesses as well:
Before addressing specific statements - some of which we hope the previous answers already addressed - we would like to clarify that we generally agree overly strong statements can accidentally be drawn from the paper, and it is very much not our intent for it not to read as making such statements. As we mentioned in the paper and you also noted, it is very difficult making these claims in a fully general, precise way, partially due to some of the given notions - e.g. representational similarity - having so many definitions and generally being vague. We do our best to make these statements as precise and quantitative as possible, e.g. hence the theory, but they certainly have some limitations to their generality. We will update the manuscript to improve this precision further and make the limitations of our conclusion clearer as well - thanks for the pushback.
If you are referring to Theorem 3.6, note that it refers to a certain form of alignability, with Theorem 3.7 referring to general linear alignability. Theorem 3.7, and hence linear alignability, would likely hold for sentences and words, especially as words are a subset of sentences.