acceptodds
Under review as a conference paper at ICLR 2027

A failure of Imagination: VLMs fail to use visual representations for text, but self-distillation can help

Abstract

Vision-language (VL) models are typically built by fine-tuning a language model to attend to visual features in service of a Visual Question-Answering objective. We ask whether this alignment changes how language models represent text—especially on tasks where visualization would help, like geometry questions posed in words. We distinguish weak multimodality (the model can answer questions about an input image) from strong multimodality (the model's text representations themselves become vision-aligned, improving performance on text-only tasks with latent visual structure). To test for strong multimodality, we construct a task of shape-counting problems with paired equivalent text and image specifications. This task is trivial when the image is provided, but requires visualization when specified in text as line coordinates. We compare instruction-tuned Qwen-2.5 and Mistral models against their vision-aligned counterparts on the text variant of the task. We find that the multimodal models do no better in this setting, despite strong performance on the image version of the task. Probing experiments corroborate this: vision-aligned decoder embeddings carry no more visual structure than their text-only counterparts. Finally, we provide an optimization objective that distills in-context visual embeddings into text embeddings. This alignment induces gains on the text variant of the task without explicit task supervision, even on samples more complex than seen during alignment. This suggests that VQA-style alignment leaves available cross-modal transfer untapped.

Primary area:
interpretability and explainable AI
Keywords:
Vision Language Models, Multimodality

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PygYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether vision-language alignment improves a language model’s ability to solve text-only problems that could benefit from visual representations. The authors distinguish between “weak multimodality,” referring to the ability to process visual inputs, and “strong multimodality,” referring to the transfer of visual structure into text processing. They introduce a synthetic shape-counting task with paired image and textual specifications, where the latter consists of shuffled drawing commands. Comparisons between Qwen2.5 and Qwen2.5-VL, and between Mistral-NeMo and Pixtral, reveal little consistent advantage from multimodal training on the textual task. Linear classification and cross-modal retrieval probes show similar trends. Finally, the authors propose self-distillation that aligns the student’s text-conditioned hidden states with the frozen teacher’s image-conditioned hidden states. Without shape-count labels or task instructions during alignment, this approach improves text-only performance, including on scenes containing more shapes than those used for training.

Critical strengths

The paper asks a meaningful question about cross-modal transfer.

The distinction between successfully processing images and benefiting from visual training when processing text is conceptually useful. The study shifts attention from conventional image-conditioned performance to whether multimodal training changes the capabilities of the language pathway itself.

The paired synthetic setting enables informative comparisons.

Rendering images from the same drawing commands provides corresponding inputs with the same underlying scene content. Shuffling edges also creates a controlled challenge involving composition across separated textual elements. Comparing multimodal models with their corresponding language backbones is more informative than comparing unrelated models.

The label-free distillation result is promising.

Improving shape counting without providing counting instructions or answer labels during alignment is an interesting result. Evaluation beyond the training cutoff of four shapes adds evidence that the improvement extends beyond the exact scene complexities encountered during training. The objective is simple and clearly specified.

Critical weaknesses

The experiments do not uniquely identify a failure to construct visual representations.

Poor performance on shuffled coordinate-based drawing commands may reflect difficulties with numerical computation, code interpretation, long-range grouping, or accessing relevant representations. It does not necessarily establish that visual representations are absent. Likewise, solving this task does not inherently require visualization: shapes can be identified through symbolic connectivity and geometric calculations. Controls using grouped edges, structured vertex lists, or alternative coordinate formats would help distinguish these explanations. The current evidence supports limited transfer on this task more directly than the broader claim that VLMs fail to “think visually about text.”

The scope of evaluation is narrow relative to the central claims.

All substantive results concern one synthetic generator with two quadrilateral categories. Increasing the number of shapes tests extrapolation within this generator, but does not establish transfer to new shape categories, spatial relations, rendering styles, or other text-only reasoning tasks. Although this limitation is acknowledged, the title and discussion make broader claims about multimodal alignment. Additional task families, ideally including a more naturalistic setting, would substantially strengthen the paper.

The distillation experiment lacks controls needed to attribute gains specifically to visual transfer.

The distilled student receives additional training on paired examples from the target domain, whereas the main baselines are unadapted checkpoints. Consequently, the experiment does not isolate how much improvement comes from visual targets rather than domain adaptation or representation regularization. Useful controls include continued text-only training on the same code, distillation from a text-conditioned teacher, and shuffled image–text pairings. Ablations over target layers and normalized versus unnormalized losses would also clarify which components are responsible for the gains.

The probes provide limited evidence about the presence or absence of visual structure.

Shape-count classification measures task information rather than specifically visual information. Cross-modal retrieval may exploit shared properties such as scene complexity or layout, while failure of a linear probe on mean-pooled states cannot rule out information encoded nonlinearly or at particular token positions. Furthermore, similarity to DINOv2 or SigLIP embeddings is not a necessary property of useful internal visualization. The paper should qualify these interpretations and include stronger task-relevant controls. The uniform 1/9 chance reference also needs justification given the nonuniform count distribution induced by the generator.

Some descriptions overstate the quantitative findings.

Table 1 reports an MMLU decline from 72.49% for the VL checkpoint to 70.14% after distillation, alongside a commonsense decline from 60.33% to 59.10%. Describing these results as “minimal to no degradation” is insufficiently justified without uncertainty estimates or discussion of the trade-off. Similarly, Figure 6 shows positive extrapolation, but accuracy remains low on the hardest scenes, particularly under ICL-CoT. The results should distinguish improvement over weak baselines from strong absolute generalization.

Questions for authors

Please refer to the weaknesses.

4 comments
AuthorsOpenReview ↗

Additional controls

Thanks for your review!

W1

We recognize that this task can be done symbolically, in fact any task specified entirely in text can be done symbolically. Our claim is more about the comparison – we claim that VLM training should make the aligned text decoder better at this task than the vanilla text counterpart as the task clearly lends itself to visualization. Furthermore, we rule out long-range dependencies and numerical computation as the primary explanations by providing the distillation result (Fig. 6) because the same architecture is able to do much better when trained to have representations that are more vision-aligned.

We add the un-shuffled edges baseline numbers below. The accuracies are a little bit better however the task remains hard and there is no conclusive winner between the VL aligned text decoder and the pure text decoder for the tested models (and they always lag significantly below the image ceiling)
https://ibb.co/YBRcMv68

W2

We appreciate your concern about a single task, to mitigate them we conduct our contrastive probe experiment on the naturalistic data COCO. The representational quality claims stand on this setting i.e. the VLM text decoder’s representations of captions shows alignment with image embeddings that is on par with the representations from the vanilla text decoder while out-performing representations from a randomly initialized backbone. We attach the COCO probe results here:

https://ibb.co/ZpC5kgJ3

Unfortunately, due to computational constraints we are unable to run the distillation alignment on large-scale real data.
We will soften the language in the discussion to reflect this!

W3

Thanks for suggesting these thoughtful controls!

Direct Label Finetuning

As a text baseline on the data, we train the model to directly answer with labels on samples with shapes (1-4) and evaluate on shapes (1-8). We post the table here:

Model / checkpoint1–45–8
Qwen2.5-7B-Instruct (text, base)21.2 ± 1.52.1 ± 0.5
Qwen2.5-VL-7B-Instruct (base)13.6 ± 1.20.1 ± 0.1
Qwen2.5-VL-7B-Instruct (base, image input)79.5 ± 1.439.3 ± 1.7
Label SFT99.7 ± 0.20.0 ± 0.0
Distilled74.5 ± 1.617.7 ± 1.4

We note that while the SFT checkpoint does almost perfectly on the in-distribution held out samples (1-4 shapes) but it goes to 0 for the harder out-of-distribution samples. On the other hand, the distilled checkpoint shows gains beyond the distribution seen during training suggesting transfer of visual priors as opposed to generic distributional priors.

Text-Conditioned teacher

The distillation objective is not defined for a text-conditioned teacher because the teacher and student are initialized from the same model. Therefore, calculating both teacher and student embeddings from the same text input leads to the same embeddings (and no loss).
If you meant letting the teacher see un-shuffled / grouped edges and the student see shuffled edges, then the issue is that the difference in accuracy is not huge. The current distilled checkpoint on shuffled edges already beats the un-shuffled baseline by a strong margin. As the student will be capped by the teacher's performance, we don’t expect a student trained on a text conditioned teacher to come close. We attach a plot to demonstrate this:

https://ibb.co/WvhFcNTZ

Shuffled Image-text pairings

For this experiment we simply randomly shuffle the image-text mapping for the training data used for distillation alignment. After this, we run a search over all hyperparameters we ran for the distillation checkpoint (including our best) – all of which break the model and lead to an accuracy of 0 and an output parse-rate near 0.

The data, objective, and training budget are identical to the matched setting, which at the same hyperparameters improves the task performance(0.136 → 0.745 on 1-4 shapes; 0.001 → 0.177 on 5-8) while preserving general language ability (Appendix A.6). The only difference is whether the teacher’s hidden-states are semantically relevant to the student input. The training regime is therefore not destructive in itself; the correspondence is important. This indicates that our checkpoint’s gains depend on transferring structure from image-conditioned representations as opposed to a generic effect of the distillation objective.

Ablations over Target layers

That’s an interesting addition! But attribution to the layers is a question we consider outside the scope of this work. We would be very interested to see future investigation on this.

(continued in the next comment)

AuthorsOpenReview ↗

Continuation W4 and W5

W4

We thank the reviewer for this careful critique. We agree that our probes measure task-relevant structure and that alignment with DINOv2/SigLIP is one operationalization of visual structure, not a necessary condition. Our probe claim is comparative: vision-language alignment does not add more of this structure than the text-only backbone already has. We believe that the probes trained on randomly-initialized backbones serve as controls for how far distributional shortcuts can take you on the probe accuracy. Given that all of our probes perform significantly better than this control, we believe that their results can be read as claims about the pretrained models and not about the task distribution. Additionally the naive baseline is 1/9 for the linear probe because our metrics are macro-averaged.
We acknowledge that linear probes on mean-pooled states cannot rule out information encoded non-linearly; however this is standard for interpretability claims because making the probe more complex (MLP) runs the risk of the probe learning a task the pretrained model doesn’t lend itself to (Hewitt & Liang (2019)).

W5

We appreciate your concern here and will appropriately soften the language in discussion of this degradation. However, we stress that our artifact is intended only as a proof of concept to illustrate that gains beyond just text training are possible with alternate alignment objectives.

We hope that we were able to address your main concerns!

References:

Designing and Interpreting Probes with Control Tasks (Hewitt & Liang, EMNLP-IJCNLP 2019)

I thank the authors for conducting the additional experiments. However, my main concern remains unresolved. The paper makes broad claims about multimodal alignment, but its experiments are limited to simple synthetic shape-construction tasks. This narrow experimental scope provides insufficient evidence to support the broader claims. I encourage the authors to reframe the current main experiments as preliminary, exploratory studies, make experiments on datasets such as COCO the primary evaluation, and revise the manuscript so that its conclusions reflect the results of this broader evaluation.

Thank you for reading our response and additional experiments! However, we can't make COCO the primary evaluation, because it cannot answer our question.

Our question is: "Does VLM alignment as it stands now make the language decoder better at tasks that might benefit from visualization?". To answer this cleanly in a scientific way, we come up with a constructed setting where:

1.⁠ ⁠Each sample / question should be representable equivalently in text and image.

2.⁠ ⁠The task should be significantly and clearly easier in image than in text.

Without (2) there is nothing to transfer hence a null result is uninterpretable. Only with (2) does a null result become informative: the model shows visual competence but the attached text decoder does not benefit from it. This was mentioned in our exclusion of LLaVa (section 4.1). It is also why existing tasks don't serve as a diagnostic: for instance, in Xu et al. (2026) the tested models do best on the text-only variants, and in Zeller et al. (2026) nearly all tasks include an image in the input, with models doing worse with visual context on the one exception.

COCO fails on both the above requirements as the captions are not equivalent representations of the image; furthermore, the captions give away the answer to any question one might pose for the image directly. Our contrastive probe analysis can be run on paired data that don't pass the requirements strictly and that we do run on COCO and show results to be consistent (W2 of our response to your review).

We hope our argument convinces you that the primary results need to be on a controlled synthetic setting as it was designed for the diagnostic. However, if you have a dataset with the above requirements in mind, we would be happy to conduct our experiments on it.

References:

  1. Yige Xu, Yongjie Wang, Zizhuo Wu, Kaisong Song, Jun Lin, and Zhiqi Shen. Do vision-language models truly perform vision reasoning? a rigorous study of the modality gap. arXiv preprint arXiv:2604.16256, 2026.
  2. Jana Zeller, Thaddäus Wiedemer, Fanfei Li, Thomas Klein, Prasanna Mayilvahanan, Matthias Bethge, Felix Wichmann, Ryan Cotterell, and Wieland Brendel. Mentisoculi: Revealing the limits of reasoning with mental imagery. arXiv preprint arXiv:2602.02465, 2026.
Reviewer_xYXtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper explores whether vision-language models that have trained with vision-language alignment can leverage visual structural information to perform text-only reasoning. Through a counting task based on image-code pairs, the authors reveal the limitations of these models in this regard. To address this issue, the authors propose a cross-modal hidden state distillation method; experiments show that this method enhances the model's text reasoning ability and improves the alignment of its cross-modal representations.

Critical strengths
  1. The paper explores a novel perspective on whether visual-language alignment can improve pure text reasoning problems.
  2. The proposed cross-modal distillation method is simple and has achieved positive results in experiments.
Critical weaknesses
  1. The paper lacks crucial comparisons such as text-only training and mismatched image-text distillation with equivalent training budgets. The current results cannot definitively determine whether the performance improvement truly stems from visual knowledge transfer.

  2. In the synthetic task, each shape is fixed to consist of four line segments, and the total number of shapes can be directly inferred from the code. This may lead to non-visual statistical shortcuts and affect the evaluation of geometric reasoning ability.

Questions for authors
  1. Could you compare the method described in this paper with text-only training and distillation methods using randomly mismatched image-text pairs, under the same training budget, to verify the role of visual knowledge transfer in performance improvement?

  2. Could you add a simple baseline to predict shape counts using only the number of drawing commands and shape distribution?

2 comments
AuthorsOpenReview ↗

Distillation Controls and Baselines

Thanks for your review, we deeply appreciate your feedback!

W1 & Q1

Thank you for suggesting further controls to isolate the evidence for visual knowledge transfer.

Text-only training

We take "text-only" training to mean fine-tuning the model on the same data but on the actual labels this time. This means we fine tune the Qwen 2.5 VL 7B model on labels of samples with 1-4 shapes (With the answer dictionary) and evaluate it on samples with 1-8 shapes. This is the same data setting as the distillation experiment in the paper (except in the distillation experiment the model is never shown the labels). We present those results here.

Model / checkpoint1–45–8
Qwen2.5-7B-Instruct (text, base)21.2 ± 1.52.1 ± 0.5
Qwen2.5-VL-7B-Instruct (base)13.6 ± 1.20.1 ± 0.1
Qwen2.5-VL-7B-Instruct (base, image input)79.5 ± 1.439.3 ± 1.7
Label SFT99.7 ± 0.20.0 ± 0.0
Distilled74.5 ± 1.617.7 ± 1.4

We note that while the SFT checkpoint does almost perfectly on the in-distribution held out samples (1-4 shapes) but it goes to 0 for the harder out-of-distribution samples. On the other hand, the distilled checkpoint shows gains beyond the distribution seen during training suggesting that it has learnt more generalizable, possibly graphical, priors with respect to the task -- thereby giving us confidence in the transfer of visual knowledge.

Mismatched Images Distillation

For this experiment we simply randomly shuffle the image-text mapping for the training data used for distillation alignment. After this, we run a search over all hyperparameters we ran for the distillation checkpoint (including our best) – all of which break the model and lead to an accuracy of 0 and an output parse-rate near 0.

The data, objective, and training budget are identical to the matched setting, which at the same hyperparameters improves the task performance(0.136 → 0.745 on 1-4 shapes; 0.001 → 0.177 on 5-8) while mostly preserving general language ability (Appendix A.6). The only difference is whether the teacher’s hidden-states are semantically relevant to the student input. The training regime is therefore not destructive in itself; the correspondence is important. This indicates that our checkpoint’s gains depend on transferring structure from image-conditioned representations as opposed to a generic effect of the distillation objective.

W2 & Q2

That is a fair concern and we restricted the setting to shapes of the same number of sides so that the split is not trivial to calculate and requires an evaluation of the angles. We note that the models are never explicitly informed of the distribution over shapes.

Assuming the shape-count is done right, these are the baseline numbers with different assumptions of the prior distribution:

PriorBaseline
Uniform (uninformed)23%
Binomial (informed)36%

However, getting the shape counts right seems to be a hard task in itself as all models struggle at the exact shape count – even the image performance on the total shape-count is not perfect for the largest model we tested (as seen in appendix A.7 of the paper). Of the settings we evaluate, only SFT receives explicit supervision linking code to shape counts. It solves the in-distribution task (0.997) but does not extrapolate (0.000 on 5–8), whereas the distilled model, which never sees counts, does extrapolate beyond the base models. We reattach those plots here for your convenience.

https://ibb.co/xqfGVq3M

We will add these baselines to the appropriate plots in the draft with the disclaimer about the assumptions the baseline requires (i.e. perfect count).

We thank you for your insightful review and hope we addressed your primary concerns! Please let us know if we can provide anything else.

Dear Reviewer,

The discussion phase is coming to a close, please let us know if you have further concerns. We would be happy to address or clarify!

Reviewer_moSZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that standard VLMs, trained on visual QA tasks, exhibit multimodality only in a 'weak' sense: they can answer questions about an image, but they do not build visual representations when facing a text-only task. To show this, the authors construct a synthetic task that consists of counting rectangles and trapezia, either in an image or from the code that generates it. On two models, they show that the vision-aligned model does not improve over its text-only counterpart on the textual task, despite performing much better on the image version. Linear and contrastive probes on the decoder’s hidden states support this finding. Finally, the paper introduces a self-distillation objective: a text-only student is trained to reproduce, from the code, the hidden states that the VL teacher produces from the corresponding image, without task labels.

Critical strengths
  • The paper is well written and easy to follow.
  • The synthetic setting is very controlled: each instance can be specified equivalently as an image or as code, so the two modalities can be compared on identical content.
Critical weaknesses
  • Limited model coverage. Only two model families are tested. Both are relatively old and follow the same paradigm (LLM + vision encoder + MLP adapter), so it is unclear how far the conclusions extend to other VLMs. Most importantly, unified multimodal models, which are also trained to generate images, are a natural candidate for strong multimodality as defined. Some of them are initialized from available LLMs (e.g., Janus-Pro). Testing them would considerably strengthen the conclusions.

  • The practical impact of strong multimodality is unclear. The paper does not identify a setting where 'strong multimodality', as defined, is practically useful. Where paired text/image data exists, one can simply render the input at inference time and pass the image to the VLM, which the paper shows to produce stronger results. Where such data does not exist, the proposed distillation cannot be trained. In addition, all experiments are confined to a single synthetic task, and it is unclear whether the conclusions transfer to tasks of practical interest.

  • Additional evaluations are needed. The only evidence that distillation does not degrade the model comes from multiple-choice benchmarks, which can hide damage to generation. Perplexity and generative benchmarks should also be reported. Moreover, since the decoder weights change during distillation, the model should be evaluated with the vision encoder reattached, on image captioning and image QA benchmarks and on the image variant of the shape task itself. It would also be useful to test whether the gains transfer to other text-only tasks with spatial structure.

Questions for authors

Is the textual task a valid test of strong multimodality? I suspect the textual task would be very hard for humans without pen and paper, even though humans seem a natural example of a strongly multimodal system. Could the authors comment on this? In particular, does failing the task necessarily indicate a lack of visual representations, or could it also reflect other limitations, such as working memory or numerical precision?

4 comments
AuthorsOpenReview ↗

Reproduced results on new model family, conceptual discussion, and degradation checks

Thanks for your effort in providing us with valuable feedback. Your suggestions have been very helpful!

W1

We ran all of the analysis experiments on the Janus Pro and the Deepseek LLM (the pure text counterpart). We report that the results are consistent with our findings from the Qwen-style VLMs i.e. the vision aligned text decoder shows no evidence of visual structure in its embeddings beyond what the pure text model already has.

Shape-count evals

ModelInputICL+COTICL+No COTNo ICL + No COTImage Input
Janus-Pro-7BImages––-0.317 ± 0.012
Janus-Pro-7B LMText0.045 ± 0.0050.079 ± 0.0070.028 ± 0.004-
DeepSeek-LLM-7B-chatText0.052 ± 0.0060.082 ± 0.0070.020 ± 0.004-

The image-input performance is significantly higher than text-input performances across the board, making the evaluation valid. However, the text-input performance for both the models is at par with image-aligned text decoder performing slightly worse on the ICL settings.

Probes

Linear Probes

https://ibb.co/Kp0kGrzB

https://ibb.co/TFRXD8x

Contrastive Probes

https://ibb.co/35qWhKrs

All the probe results show a picture similar to the one reported in the draft for the other models. Across layers the models typically do significantly better than the randomly initialized backbone baseline and the image-aligned text-decoder's representations are at par or a little worse than the vanilla model.

We will add these to the main draft and include the architectural distinctions from the other models! Thanks!

W2

We believe strong modality would help with anything we traditionally associate with visualization for instance writing geometry proofs, designing and coding up webpages, game development etc. We never claim that such capabilities cannot come from extensive text training but simply argue that the paired image-text data currently used in VLM training should help in obtaining more strongly multimodal systems.

To scientifically check whether VLM training made a language decoder better in this way, we use a controlled 1-to-1 renderable setting. However, the renderable setting is not the only one where one can expect strong multimodality to help. The renderer is what makes our study controlled, it is not the intended downstream use case.

Strong multimodality would be useful so far as cross modal generalization can be useful and in principle the distillation alignment can be done on generic the text-image data typically used for VLM training. However, we don’t do this because of computational constraints.

We were able to run our contrastive probe experiments on the coco captioning dataset i.e. are caption representations from vision-aligned text decoders better aligned to appropriate images than their pure-text counterpart. The results show this not to be the case, representations from the VL-aligned decoder outperform representations from a randomly-initialized decoder but are at par with the pure text decoder. Please find them attached here:
https://ibb.co/ZpC5kgJ3

We will include the coco results as part of the revised draft!

To summarize, we pose strong multimodality as a way to get more out of the paired image-text data that is already used in the VLM training pipeline and pose our synthetic setting as a way to bring this problem out in a controlled and scientific way.

(continued in the next comment)

AuthorsOpenReview ↗

Continuation (W3 and Q1)

W3

We appreciate your concern about model degradation, but we stress that the artifact is just a proof of concept and we take no special care in protecting existing capabilities. Following your advice, we ran some generative benchmarks and observed low to moderate degradation on unrelated text tasks.

BenchmarkMetricQwen2.5-7B-InstructQwen2.5-VL-7B-InstructDistillation
TruthfulQA (generation)BLEU acc / ROUGE-1 acc47.7 / 52.444.8 / 48.842.2 / 47.5
NQ-open, 5-shotExact match19.519.518.4
TriviaQA, 5-shotExact match53.151.850.6

We see no degradation on the 500 sample val set of Nocaps. However, the performance for the base model is low to begin with. The table reports un-scaled scores.

BLEU-4CIDEr
Base Qwen2.5-VL-7B-Instruct0.14690.3805
Distilled0.19440.4771

We also benchmark on Visual question answering tasks using lmms-eval

We note that for VQAv2-lite the exact match often doesn’t work despite the right answer being there because the distilled model talks more. We use a lenient parser as well that checks whether the exact answer substring exists in the output after normalization (lowercase etc) – the parser also accounts for words that occur in the question (and excludes them). We present the table below, “official” numbers are from lmms-eval as-is and “lenient” numbers are from the lenient parser. There is some degradation, but the model is mostly intact despite no protections. We believe this is because the teacher was initialized from the same model. The table:

TaskMetricBase officialBase lenientDistilled officialDistilled lenient
VQAv2-liteexact match0.78520.80930.30980.7453
AI2Dexact match0.82710.82710.80510.8051
MMStaraverage0.61730.61730.58400.5840

We will include these in the draft!

We note that on the image-variant of the task, the student does degrade (59% to 31%). This is expected as the student never sees image inputs during alignment and thus nothing constrains the image pathway. The distilled artifact is a proof of concept that VQA-style alignment leaves cross-modal transfer untapped. It is not intended as a deployable multimodal model, there are several possibilities to protect the model during this alignment (like keeping image-input representations from changing, regularization to the original model, or LORA) to make this fit for deployment and we leave that exploration to future work.

Q1

There is reason to believe that human cognitive representations are not purely symbolic and are grounded in perception (Glenberg et al., 2003) and we are looking for a variant of such grounding in multimodal models.

Our question boils down to whether VLM training improves this kind of cross-modal association and phrases our analyses as comparisons. Our claim rests on the fact that vision aligned models do no better at any level when compared to the pure text model. Our claim is that VLM training should make it more so a strongly multimodal system and that does not seem to be the case and we don’t argue anything about the absolute performance.

To rule out alternative explanations like failure in precision and working memory, we attach the per-shape breakdown of the task performance to demonstrate that even for the easy cases where working memory shouldn't be an issue, the vision-aligned decoder does not do better either-way.

https://ibb.co/k2Yj1CCj

Additionally, We argue that the distillation checkpoint (Fig. 6) further rules out working memory and numerical precision as primary explanations. On the same input, with identical architecture, the checkpoint performs significantly higher (this is stark in the single-shape setting where the default models are at chance ~52% and the distillation checkpoint is at 96%) without ever being trained on the actual labels.

We thank you for your insightful review and suggestions and hope we were able to address at least some of those concerns! Please let us know if we can provide any additional information or clarification!

Reference:

Glenberg AM, Robertson DA, Kaschak MP, Malter AJ. Embodied meaning and negative priming. Behavioral and Brain Sciences. 2003;26(5):644-648. doi:10.1017/S0140525X03240140

Dear Reviewer,

The discussion phase is coming to a close. We hope we were able to address at least some of your primary concerns, please let us know if you would like us to provide anything else!

Thank you for the extensive rebuttal and the additional experiments. I believe the Janus-Pro results and the degradation analysis would significantly strengthen the paper.

However, I remain unconvinced on two points. First, the practical impact of weak multimodality: both the shape-counting task and the new COCO probe are evaluations constructed for this paper, and neither shows that the gap is substantial for a text-only task of practical interest. Second, the self-distillation approach degrades the model substantially, including on the image variant of the task itself and on VQAv2. I understand the distilled artifact is a proof of concept, but I believe the paper should show that this approach can, in principle, improve performance without harming unrelated capabilities, for instance with one of the regularization strategies you mention. Hence, I keep my score as is.

Comments on the paper

1 comment

Dear reviewers,

We hope we were able to resolve most of your primary concerns through our rebuttal. We would be happy to provide further clarification on any point!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.