acceptodds
Under review as a conference paper at ICLR 2027

Persistent Representational Misalignment Induces Vision-Language Model Failures

Abstract

Vision-language models (VLMs) fail at some tasks that are simple for humans—but why? We hypothesize that many such failures stem from VLMs’ difficulties translating between vision and language representations—a misalignment the Platonic Representation Hypothesis predicts should disappear with scale. This paper is composed of two parts: in the first, we show that existing vision and language representations are linearly unalignable. This misalignment is an inherent property of the data and hence expected to persist regardless of either scale or the alignment's complexity. In the second, we demonstrate how this misalignment both leads to a predictably induced generalization failure and correlates with existing failure modes, where more representationally aligned VLM backbones accordingly exhibit fewer failures. Overall, these results imply representations will not become aligned simply given sufficient scale, providing evidence against a strong version of the Platonic Representation Hypothesis.

Primary area:
interpretability and explainable AI
Keywords:
vision-language models, representational misalignment, platonic representation hypothesis, failure modes

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VcsQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that vision and language embeddings are not perfectly alignable because their embedding spaces are too different. It first distinguishes between functional and representational equivalence and alignment. Then it connects perfect linear alignability to analogies, similarity structures, and generalized analogies. Empirically, it finds that similarity structures are less aligned across vision and language models than between language models, and that textual analogies do not transfer perfectly to visual representations. Finally, it shows that training only the aligner often fails to generalize to classes not seen during training, and that representational alignment (mutual k-NN) correlates with performance on vision-language benchmarks.

Critical strengths

The paper provides some examples where vision and language embeddings differ. In particular, differences in the similarity structure and text analogies are quantified. Furthermore, representational alignment is connected to failures of VLMs.

Critical weaknesses
  1. Many statements are strongly overstated given the empirical evidence:

    • “we show that existing vision and language representations are linearly unalignable” (ll. 15-16), “existing representations are unalignable” (l. 45), “we provide evidence that existing vision and language representations are linearly unalignable” (ll. 457-458): The definition of alignability in the paper means perfect alignability (see e.g. Ziyin & Chuang 2025), where all points are perfectly mapped to the other space. Under the strict definition in the paper, word and sentence embeddings also couldn’t be aligned, as they don't have exactly the same distances across models.
    • “This misalignment is an inherent property of the data and hence expected to persist regardless of either scale or the alignment’s complexity” (ll. 16-17), “alignment will likely not naturally emerge with scale” (l. 46): The paper compares the cosine similarities on COCO with DINOv3 base and all-MiniLM-L12-v2 and shows that “[f]or about 2% of COCO image/caption pairs, their cosine similarity difference exceeds [0.4]” (ll. 212-213). In addition, some qualitative samples from COCO are shown where these similarities are different, and the similarity is compared with language-language combinations. However, it is not shown to hold for any dataset, model, or scale. In contrast, the authors even note themselves that “the vision models’ retrieval accuracy increases with scale” (l. 467). It is not clear why this cannot further improve with a higher-quality dataset, more data, or better models.
    • “The generalization failure described in the last section stems from representational misalignment.” (l. 405): This conclusion has no theoretical or empirical justification in the paper.
    • “we demonstrate how this misalignment […] leads to a predictably induced generalization failure” (ll. 17-18), “This misalignment predictably leads to an induced generalization failure in vision-language models” (ll. 46-47): Only a correlation not a causality is shown in Sec. 5. Sec. 4 only shows that an adapter trained on some classes doesn’t generalize to unseen classes. The paper doesn’t build a predictive relationship between alignment quality and general generalization failures.
    • “we demonstrate how this misalignment […] correlates with existing failure modes” (ll. 17-19): The specific failure modes are never discussed. The paper only shows that the performance correlates with the alignment score.
    • “Overall, these results imply representations will not become aligned simply given sufficient scale, providing evidence against a strong version of the Platonic Representation Hypothesis.” (ll. 20-22), “a diverse set of VLM failures likely stem from representational misalignment and will not be resolved solely by increasing model scale” (ll. 51-52): This implication is not shown in the paper. Moreover, the platonic representation hypothesis doesn’t just discuss the convergence in terms of model size but also in terms of data size, which is not part of the paper.

    This is also admitted in Appendix L: “Most results are circumstantial in supporting that many VLM failures likely arise due to representational misalignment. In this sense the results are consistent with the hypothesis but do not causally show that representational misalignment is what caused a failure.” (ll. 1342-1344)

  2. The sections are not well connected. Sec. 2 introduces functional and representational equivalence and alignment without defining them properly. The only concept that is used subsequently is representational alignment. Sec. 3 discusses the connection of perfect linear alignability with analogies, similarity structures, and generalized analogies and shows empirically that similarity structures and analogies are not perfectly preserved across vision and language models. Sec. 4 discusses how a VLM performs on held-out data when no such data is used in training the adapter. The connection to alignment is mainly motivated by the fact that word embedding spaces can be aligned across languages on a subset of words and still perform well on held-out words. However, the analogue for this would be to align the vision and language embedding spaces instead of creating a generative VLM. Finally, Sec. 5 shows the correlation between mutual k-NN and downstream performance on VLM benchmarks. This is not connected sufficiently to previous sections. The theory in Sec. 3 uses the Gram matrix. However, mutual k-NN only consideres the overlap in the nearest neighbors and the causality that representational misalignment from Sec. 5 is the source of the problem of Sec. 4 is also not well explored. Altogether, the different sections seem detached.

Ziyin & Chuang 2025: “Proof of a perfect platonic representation hypothesis”

Questions for authors
  1. Most VLMs are end-to-end fine-tuned after initializing the adapter, and even a coarse adapter works well when fine-tuned (see Gröger et al. 2026). How is the generalization performance after fine-tuning the VLMs in Sec. 4?
  2. Can one show that misalignment in the vision and language model is the reason for the held-out results in Sec. 4?
  3. Which explicit failure modes come from representational misalignment?
  4. Why is CosSimRMSE introduced instead of comparing the established CKA score?
  5. For many vision and language models, the embeddings are normalized and lie on the unit sphere. Computing the class average and an analogy as A - B + C would move the vectors away from the manifold. Is anything done to keep the analogy on the unit sphere?

Gröger et al. 2026: “Rethinking Projector Training in Multimodal LLMs”

5 comments

Thanks for the feedback, we clarify different points below and address your questions - going one by one:

Q1: does the generalization failure persist when finetuning the VLM?

Yes - note that the large scale experiments in section 4 already do finetune the LLM and the adapter together. LLaVA consists of an adapter only training stage and an adapter+LLM finetuning stage, which we replicate here. The given generalization performance already is post this finetuning.

Q2: Can one show that misalignment in the vision and language model is the reason for the held-out results in Sec. 4?

Yes, there is evidence that better alignment can substantially reduce the generalization failure. When training the 7B LLaVA model using a CLIP backbone - which was already trained on joint image/text data, including over the held-out concepts - we find the held-out concept detection accuracy is about 61%, while for the vision-only ViT it is 21.5%. This does not test generalization as CLIP already got joint image/text supervision, but does show that when using better aligned backbones then the generalization failure is substantially reduced.

Q3: Which explicit failure modes come from representational misalignment?

The failure in section 4 is the clearest example, with the left four in figure 5 being in the very least correlated if not induced from the representational misalignment. These include Winoground (matching semantically different but syntactically similar captions to their corresponding images), as also speculated by [WhyWinoHard], and VLMBias (detecting oddities in out of distribution images, e.g. number of legs on a five-legged lion). VLMBias is a nice example of how generalization failures like the one given in section 4 can happen more broadly, and we speculate that a range of similar out of distribution failures are, at least in part, due to representational misalignment. Many if not all of the failures in lines 417-424 seem to fit this category.

Q4: Why is CosSimRMSE introduced instead of comparing the established CKA score?

CosSimRMSE measures the extent to which two representations have the same similarity structure (Gram matrices) and hence the extent to which the conditions for theorem 3.5/corollary 3.6 hold/don’t hold respectively. CKA is a more restricted version thereof - having the same similarity structure implies CKA=1 but CKA=1 does not imply the representations have the same similarity structure. Thus, we choose to measure CosSimRMSE as it better corresponds to the relevant quantities for the given theorems.

Q5: For many vision and language models, the embeddings are normalized and lie on the unit sphere. Computing the class average and an analogy as A - B + C would move the vectors away from the manifold. Is anything done to keep the analogy on the unit sphere?

Yes - concepts and analogies are renormalized to be on the unit sphere. Basically, the average of unit vectors is taken and then renormalized to be a unit vector, and A-B+C is also renormalized to be on the unit sphere. Thus, all comparisons are done over the same manifold.

Now, we would like to try clarifying some of the mentioned weaknesses as well:

W1: statements seem overstated

Before addressing specific statements - some of which we hope the previous answers already addressed - we would like to clarify that we generally agree overly strong statements can accidentally be drawn from the paper, and it is very much not our intent for it not to read as making such statements. As we mentioned in the paper and you also noted, it is very difficult making these claims in a fully general, precise way, partially due to some of the given notions - e.g. representational similarity - having so many definitions and generally being vague. We do our best to make these statements as precise and quantitative as possible, e.g. hence the theory, but they certainly have some limitations to their generality. We will update the manuscript to improve this precision further and make the limitations of our conclusion clearer as well - thanks for the pushback.

W1.1 …under the strict definition in the paper, word and sentence embeddings also couldn’t be aligned, as they don't have exactly the same distances across models.

If you are referring to Theorem 3.6, note that it refers to a certain form of alignability, with Theorem 3.7 referring to general linear alignability. Theorem 3.7, and hence linear alignability, would likely hold for sentences and words, especially as words are a subset of sentences.

W1.2…“alignment will likely not naturally emerge with scale” … However, it is not shown to hold for any dataset, model, or scale. In contrast, the authors even note themselves that “the vision models’ retrieval accuracy increases with scale” (l. 467). It is not clear why this cannot further improve with a higher-quality dataset, more data, or better models.

When referring to alignment not naturally emerging with scale, we refer specifically to full alignment. Partial alignment very well may and has been shown numerous times to improve with scale - we argue there is a ceiling to this alignment, namely due to inherent differences in information between modalities, and this ceiling cannot be breached with data/model scale. Specifically, this is a statement on certain data distributions - if one uses a dataset of book scans then the dataset contains effectively the same information as the words in those books, in which case a vision and language model may converge to the same representations. However, there are concepts which might be expressible in one modality but inexpressible in another, as exemplified in thought experiments like [Mary’sRoom].

We hope these and other, prior answers help clarify the stated weaknesses regarding accidental overstatements, and we will update the manuscript to make these sharper. If you feel like any are not well addressed, please do let us know.

W3: The sections are not well connected

This is a difficulty arising due to there being many different notions of representational similarity. At a high level, we aimed for different sections to touch on different notions of representational equivalence/alignment and functional equivalence/alignment, so the resulting statements would touch on a broad set of existing definitions and tests for representational similarity. We will make it clearer in the updated manuscript which section touches on what and how, which would hopefully make it clear how all sections effectively make statements about representational alignment (see the diagram in section 2).

We hope this answered your questions and helped clarify the paper. Thanks again for the feedback and pushback, we will use both to improve the paper.

References

[WhyWinoHard] Diwan, Anuj, et al. "Why is winoground hard? investigating failures in visuolinguistic compositionality."

[Mary’sRoom] Nida-Rümelin, Martine, and Donnchadh O Conaill. "Qualia: The knowledge argument."

Thank you for the insights and additional information.

  • Q1: Fig. 3 (right) depicts the LLM as frozen. This seems to be wrong then. How is the performance before the fine-tuning? Are the concepts also left out for full finetuning?
  • Q2: Comparing CLIP and a vision-only ViT is not a comparison that isolates misalignment as the reason for the held-out results because many things are changed between a CLIP ViT and a vision-only ViT. This is not enough evidence for the claim that misalignment causes the discrepancy. Moreover, the Gram matrices of the vision and language model for CLIP are not very similar (see e.g. Klebe et al., 2023 and Soares et al., 2026). So, CLIP might not be the best choice for a well-aligned model.
  • Q3: This is a correlation. The PRH already showed that model size is correlated with alignment. If model size is correlated with a better performance on Winoground, VLMBias, VLMBlind, MMVP, the correlation between alignment and this performance would be trivial. Altogether, this correlation doesn’t show that representational misalignment is the reason for this failure modes. Much stronger results would be if the failures could be solved by improving the alignment without changing other components.
  • Q4: CKA = 1 implies that the representations have the same similarity structure. CKA is the cosine similarity between the centered and vectorized Gram matrices. Therefore it is equal to 1 if and only if the centered Gram matrix are scaled versions of each other. Because they are centered, they can also be translated arbitrary. Both, centering and scaling, doesn’t change the distance structure. I don’t see how CosSimRMSE helps.
  • W1.1: I ran this experiment on the standard MUSE setup (Conneau et al. 2018) with fastText wiki vectors (300 dim, top 200k words) on multiple language combinations (en-es, en-fr, en-de, en-ru, en-zh) and for all of them the residual even on the training data is non-zero. The relative residual is 0.56, 0.56, 0.59, 0.61, and 0.48 respectively. Therefore, even in the word translation setting the spaces are not perfectly linearly alignable. The same holds for the Gram matrices. So, even for this setting all theorems in the paper fail. This also holds for two English models (en-wiki to en-cc) with a relative residual of 0.34. Altogether, this shows that the paper acts in a very idealized setting that is irrelevant for practical scenarios as perfect alignment will probably never be reached but is also not required.
  • W1.2: I agree. The original PRH already stated that concepts like “I believe in the freedom of speech” are present in only one modality and this is also part of multiple thought experiments. From the abstract and some parts of the paper, it is not clear that perfect linear alignment (with a zero residual) is meant with alignment and it could be misunderstood as that alignment cannot improve. It would be good to clarify this.

Would it be possible to specify exactly what you plan to update in the manuscript? You are allowed to upload an updated version of the paper. This would help to judge if the statements are corrected sufficiently.


Conneau, Alexis, et al. "Word translation without parallel data." ICLR 2018.

Klebe, Dustin, et al. "GeRA: Label-efficient geometrically regularized alignment." arXiv preprint arXiv:2310.00672 (2023).

Soares, Diogo, et al. "Scalable and Interpretable Representation Alignment with Ordinal Similarity." ICML 2026.

Reviewer_VwuHOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper tries to explain why VLMs still make mistakes on simple visual tasks, even when they’re very good at understanding images and text separately. The authors argue that a big reason is due to the fact that images and text contain different kinds of info; this makes it hard for models to connect what they see with what they know. A key finding is that models can fail to recognize objects they already know if those objects were not included when the vision and language parts were trained together. Models with better connections between the two tend to make fewer mistakes.

Critical strengths
  • paper both points out the problems with VLMs AND tries to explain why these problems occur
  • has very clear examples demonstrating the problem at hand (Section 4, Figure 3, Table 2)
  • tests the idea across different models and tasks
  • compares tasks where alignment matters more or less makes the overall argument stronger
Critical weaknesses
  • some concepts in experiments, like countries or cities, are naturally easier to describe with words than pictures; unclear how much of the difference comes from the choices of examples (Section 3.4, Fig 2)
  • removing certain objects entirely from the training data seems a little unrealistic since models usually see at least some examples of common objects
Questions for authors
  • even if images and text cannot be perfectly aligned, how much alignment is actually necessarily for the model to perform well?
  • could you improve the connection between the vision and language parts (holding the rest of the model constant) to test whether that alone reduces mistakes? aka using more advanced methods for connecting vision and language
1 comment

Thanks for the feedback, we clarify different points below and address your questions - questions are paraphrased as we understood them so in case of any misunderstandings it would hopefully be easy to catch them:

“Would stronger nonlinear text-vision adapters result in better performance, with respect to the generalization failure in section 4?”

These results appear in the paper’s Appendix, with the short answer being that stronger adapters result in similar performance with respect to the generalization failure. Specifically, in Appendix F.1 we find for the small-scale setting that better adapters, e.g. an MLP or attention based adapter, do not lead to meaningfully different performance relative to the linear adapter used in section 4. For the large scale setting, note that LLaVA by default uses an MLP adapter and one of its stages of training includes finetuning the entire LLM, making the MLP+LLM technically be the entire adapter and a very expressive one indeed. Thus, we do not find evidence that the generalization failure is due to insufficiently strong adapters.

“How much multimodal alignment is necessary for a model to perform well?”

This is a good question albeit one which is inherently difficult to answer, given both notions of alignment and what constitutes good performance not always being well defined. What we can confidently say is that the generalization failure in section 4 shows that without better techniques for multimodal alignment one should not expect a multimodal model to perform well on concepts not in its joint finetuning data. In the case of joint data then a form of memorization, not generalization, is tested, as it is possible for the adapter to act like a dictionary where it just maps a visual representation of a concept (e.g. zebra) to the lingual version thereof (e.g. the word “zebra”). Section 5 provides some quantitative correlations between multimodal alignment and performance on VLM failures.

“”Some concepts are easier to describe textually than visually, which may lead to the differences observed between models in Figure 2”

We wholeheartedly agree. This gets at one of the paper’s main points, where imperfect alignability of vision and language is an inherent property of the data, as there is some information loss when translating text to vision and vice versa. Figure 2 shows one direction thereof, where taking concepts in text and translating them to visual representations loses information. This is also why a strong version of the Platonic Representation Hypothesis is likely incorrect, as there is some information in each modality which is not in the other.

Removing certain objects entirely from the training data seems a little unrealistic

For realistic training pipelines and common concepts, certainly. The generalization failure in section 4 evidences that simple, surprising failures can occur due to language/vision misalignment. In practice, frontier labs try to train their models on giant datasets so memorizing everything is “good enough”, but even SOTA models like Qwen3-VL fail on many of the VLM failures given in section 5. For a well documented case, see the tests in VLMBias over SOTA VLMs - https://vlmsarebiased.github.io/ . There they show that VLMs fail at detecting oddities in out of distribution images, e.g. finding the number of legs in an image of a five-legged lion. It would be exceedingly difficult to train on all possible out of distribution images like these, and some level of generalization is generally desired.

Thanks for the questions, we hope these clarifications were helpful. Please do follow up in case you would like to discuss any of these further.

Reviewer_UuiPOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors argue that vision and language representations are linearly unalignable and that this misalignment will persist “regardless of either scale or the alignment’s complexity”. The authors derive a condition when a linear map exists, which essentially states that any linear relation among concepts in one modality must hold in the other (eg king - queen = man - woman). The authors then show this condition is violated by comparing the cosine similarities of COCO images using DINOv3 with those of their captions (MiniLM). They find this cross-modal gap is 3 times larger than the gap between other models of the same modality. They train VLMs whose adapters has never seen particular concepts, eg zebra, and find that even while the individual backbones recognize zebras, the VLM rarely detects these concepts (1-3.8%). The authors then build a dataset, VisLingA, that contains images and text of various analogies. They find that analogy retrieval is much higher for text than for vision. They then show that higher representational alignment between modalities correlates with better performance on benchmarks that focus on VLM failures. The authors conclude that this suggests a strong version of the Platonic Representation Hypothesis is false.

Critical strengths

I appreciated the adapter training experiments that withheld certain concepts, e.g. zebra, to test if the VLM could generalize. They show that the backbones in each modality could recognize a zebra (the vision model can classify zebra and the LLM can describe a zebra), but when the adapter is trained without any zebra image-text pairs, the resulting VLM rarely recognizes zebras: image retrieval drops to 3.8% and generation drops to 1%, compared with full dataset fine tuning of 84%. I think this analysis is useful since it shows the independent modalities can do well but the alignment does not generalize.

Critical weaknesses

Prior work has suggested that COCO captions were designed to be short (avg ~10 words) and lack details of the images’ visual features (Onoe et al., 2024). It is unclear if the modality gap is due to shorter, less descriptive captions. Figure 1 is a good example of this: the two images are nearly identical (cosine similarity of 0.96) but the captions differ greatly (cosine similarity of 0.44). If this modality gap shrinks with densely captioned data (e.g. from DOCCI), then the central claim of the paper that the misalignment “is an inherent property of the data” and will persist “regardless of either scale or alignment’s complexity” may not hold.

Yasumasa Onoe, Sunayana Rane, Zachary Berger, Yonatan Bitton, Jaemin Cho, Roopal Garg, Alexander Ku, Zarana Parekh, et al. DOCCI: Descriptions of Connected and Contrasting Images. In European Conference on Computer Vision (ECCV), 2024. arXiv:2404.19753.

Questions for authors
  1. Does the modality gap (Table 1) persist when COCO is replaced with densely captioned data (e.g. DOCCI)?
  2. Likewise, does the held-out concept failure persist when the adapter is trained on densely captioned data (e.g. DOCCI) instead of COCO?
  3. Held-out concept detection goes from 1-3.8% with small models to 21.5% with LLaVA-1.5-7B. Can the authors test different scales of these LLMs? It would be interesting to see if the alignment is partially improved by scale.
2 comments

Thanks for the feedback and questions, addressing them one by one:

Does the modality gap (Table 1) persist when COCO is replaced with densely captioned data (e.g. DOCCI)?

The short answer is that yes, the qualitative gap remains when using DOCCI, with details given below. First, note that COCO has 5 captions per image with their embeddings being averaged, so we do calculate something roughly corresponding to the average semantic embedding of an image. When recreating Table 1 using DOCCI there is still a large gap between vision/text relative to vision/vision and text/text:

RepresentationsCOCODOCCI
Text, Text′1.001.00
Vis, Vis′1.330.99
Vis, Text3.324.47
Vis, CLIP Vis10.938.03
Text, CLIP Text*11.231.83
CLIP Vis, CLIP Text*2.872.99

Interesting, the text CLIP gap is much lower due to the denser captions, but the main result – the vis/text gap – remains qualitatively the same.

Likewise, does the held-out concept failure persist when the adapter is trained on densely captioned data (e.g. DOCCI) instead of COCO?

DOCCI is too small of a dataset for recreating the failure, having ~12x less data than COCO. To still test whether the failure persists given denser captions, we use Localized Narratives [LN], a dataset with the same images as COCO but denser captions, having on average 42 words per caption for the COCO images. Rerunning the small scale failures setup on Localized Narratives, we find that the failure persists even given the denser captions - the held out concept detection accuracies are:

SettingTraining dataCOCOLN
Generationfull data84.3%67.8%
Generationconcepts held out1.0%9.8%
Retrievalfull data81.0%74.0%
Retrievalconcepts held out4.2%0.8%

For generation the denser captions perhaps help somewhat, but the large qualitative gap still remains.

Held-out concept detection goes from 1-3.8% with small models to 21.5% with LLaVA-1.5-7B. Can the authors test different scales of these LLMs? It would be interesting to see if the alignment is partially improved by scale.

The alignment seems to be very slightly improved with scale, as noted in the discussion section and numerous prior work, chiefly the Platonic Representation Hypothesis. We run the large-scale failure experiment also with a 13B LLaVA model to get an additional data point. While working on this experiment we found some potential sources of leakage in the data due to misspelled words, so we used a stronger filter and also reran the 7B experiments. This better filter very mildly changed the 7B results, with the qualitative performance gap still remaining. The results are:

ModelHeld-out concept det. acc.
7B conceptless19.8%
7B full data87.3%
13B conceptless24.6%
13B full data87.2%

(for reference, the original filter, used in the paper, yielded for the 7B model an accuracy of 21.5%)

So while the 13B does a bit better than the 7B model, it is only mildly so, and the failure remains.

Thanks for the feedback and questions, we hope this helped show the robustness of our results. In case you have any more questions please do follow up.

References

[LN] Pont-Tuset, Jordi, et al. "Connecting vision and language with localized narratives."

Thank you for the additional experiments. The DOCCI and Localized Narratives results address my main concern that the modality gap does not shrink, and the held-out concept failure persists, with denser captions. The 13B result is helpful, but I think it is difficult to distinguish a ceiling from slow improvement with scale (19.8% at 7B to 24.6% at 13B). I suggest adding larger scale to see if there is a trend.

Since my main concern has been addressed, I will raise my score to 3 - assuming that the DOCCI, Localized Narratives, and 13B results are added to the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.