- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether vision-language alignment improves a language model’s ability to solve text-only problems that could benefit from visual representations. The authors distinguish between “weak multimodality,” referring to the ability to process visual inputs, and “strong multimodality,” referring to the transfer of visual structure into text processing. They introduce a synthetic shape-counting task with paired image and textual specifications, where the latter consists of shuffled drawing commands. Comparisons between Qwen2.5 and Qwen2.5-VL, and between Mistral-NeMo and Pixtral, reveal little consistent advantage from multimodal training on the textual task. Linear classification and cross-modal retrieval probes show similar trends. Finally, the authors propose self-distillation that aligns the student’s text-conditioned hidden states with the frozen teacher’s image-conditioned hidden states. Without shape-count labels or task instructions during alignment, this approach improves text-only performance, including on scenes containing more shapes than those used for training.
Critical strengths
The paper asks a meaningful question about cross-modal transfer.
The distinction between successfully processing images and benefiting from visual training when processing text is conceptually useful. The study shifts attention from conventional image-conditioned performance to whether multimodal training changes the capabilities of the language pathway itself.
The paired synthetic setting enables informative comparisons.
Rendering images from the same drawing commands provides corresponding inputs with the same underlying scene content. Shuffling edges also creates a controlled challenge involving composition across separated textual elements. Comparing multimodal models with their corresponding language backbones is more informative than comparing unrelated models.
The label-free distillation result is promising.
Improving shape counting without providing counting instructions or answer labels during alignment is an interesting result. Evaluation beyond the training cutoff of four shapes adds evidence that the improvement extends beyond the exact scene complexities encountered during training. The objective is simple and clearly specified.
Critical weaknesses
The experiments do not uniquely identify a failure to construct visual representations.
Poor performance on shuffled coordinate-based drawing commands may reflect difficulties with numerical computation, code interpretation, long-range grouping, or accessing relevant representations. It does not necessarily establish that visual representations are absent. Likewise, solving this task does not inherently require visualization: shapes can be identified through symbolic connectivity and geometric calculations. Controls using grouped edges, structured vertex lists, or alternative coordinate formats would help distinguish these explanations. The current evidence supports limited transfer on this task more directly than the broader claim that VLMs fail to “think visually about text.”
The scope of evaluation is narrow relative to the central claims.
All substantive results concern one synthetic generator with two quadrilateral categories. Increasing the number of shapes tests extrapolation within this generator, but does not establish transfer to new shape categories, spatial relations, rendering styles, or other text-only reasoning tasks. Although this limitation is acknowledged, the title and discussion make broader claims about multimodal alignment. Additional task families, ideally including a more naturalistic setting, would substantially strengthen the paper.
The distillation experiment lacks controls needed to attribute gains specifically to visual transfer.
The distilled student receives additional training on paired examples from the target domain, whereas the main baselines are unadapted checkpoints. Consequently, the experiment does not isolate how much improvement comes from visual targets rather than domain adaptation or representation regularization. Useful controls include continued text-only training on the same code, distillation from a text-conditioned teacher, and shuffled image–text pairings. Ablations over target layers and normalized versus unnormalized losses would also clarify which components are responsible for the gains.
The probes provide limited evidence about the presence or absence of visual structure.
Shape-count classification measures task information rather than specifically visual information. Cross-modal retrieval may exploit shared properties such as scene complexity or layout, while failure of a linear probe on mean-pooled states cannot rule out information encoded nonlinearly or at particular token positions. Furthermore, similarity to DINOv2 or SigLIP embeddings is not a necessary property of useful internal visualization. The paper should qualify these interpretations and include stronger task-relevant controls. The uniform 1/9 chance reference also needs justification given the nonuniform count distribution induced by the generator.
Some descriptions overstate the quantitative findings.
Table 1 reports an MMLU decline from 72.49% for the VL checkpoint to 70.14% after distillation, alongside a commonsense decline from 60.33% to 59.10%. Describing these results as “minimal to no degradation” is insufficiently justified without uncertainty estimates or discussion of the trade-off. Similarly, Figure 6 shows positive extrapolation, but accuracy remains low on the hardest scenes, particularly under ICL-CoT. The results should distinguish improvement over weak baselines from strong absolute generalization.
Questions for authors
Please refer to the weaknesses.
4 comments
Continuation W4 and W5
W4
We thank the reviewer for this careful critique. We agree that our probes measure task-relevant structure and that alignment with DINOv2/SigLIP is one operationalization of visual structure, not a necessary condition. Our probe claim is comparative: vision-language alignment does not add more of this structure than the text-only backbone already has. We believe that the probes trained on randomly-initialized backbones serve as controls for how far distributional shortcuts can take you on the probe accuracy. Given that all of our probes perform significantly better than this control, we believe that their results can be read as claims about the pretrained models and not about the task distribution. Additionally the naive baseline is 1/9 for the linear probe because our metrics are macro-averaged.
We acknowledge that linear probes on mean-pooled states cannot rule out information encoded non-linearly; however this is standard for interpretability claims because making the probe more complex (MLP) runs the risk of the probe learning a task the pretrained model doesn’t lend itself to (Hewitt & Liang (2019)).
W5
We appreciate your concern here and will appropriately soften the language in discussion of this degradation. However, we stress that our artifact is intended only as a proof of concept to illustrate that gains beyond just text training are possible with alternate alignment objectives.
We hope that we were able to address your main concerns!
References:
Designing and Interpreting Probes with Control Tasks (Hewitt & Liang, EMNLP-IJCNLP 2019)
I thank the authors for conducting the additional experiments. However, my main concern remains unresolved. The paper makes broad claims about multimodal alignment, but its experiments are limited to simple synthetic shape-construction tasks. This narrow experimental scope provides insufficient evidence to support the broader claims. I encourage the authors to reframe the current main experiments as preliminary, exploratory studies, make experiments on datasets such as COCO the primary evaluation, and revise the manuscript so that its conclusions reflect the results of this broader evaluation.
Thank you for reading our response and additional experiments! However, we can't make COCO the primary evaluation, because it cannot answer our question.
Our question is: "Does VLM alignment as it stands now make the language decoder better at tasks that might benefit from visualization?". To answer this cleanly in a scientific way, we come up with a constructed setting where:
1. Each sample / question should be representable equivalently in text and image.
2. The task should be significantly and clearly easier in image than in text.
Without (2) there is nothing to transfer hence a null result is uninterpretable. Only with (2) does a null result become informative: the model shows visual competence but the attached text decoder does not benefit from it. This was mentioned in our exclusion of LLaVa (section 4.1). It is also why existing tasks don't serve as a diagnostic: for instance, in Xu et al. (2026) the tested models do best on the text-only variants, and in Zeller et al. (2026) nearly all tasks include an image in the input, with models doing worse with visual context on the one exception.
COCO fails on both the above requirements as the captions are not equivalent representations of the image; furthermore, the captions give away the answer to any question one might pose for the image directly. Our contrastive probe analysis can be run on paired data that don't pass the requirements strictly and that we do run on COCO and show results to be consistent (W2 of our response to your review).
We hope our argument convinces you that the primary results need to be on a controlled synthetic setting as it was designed for the diagnostic. However, if you have a dataset with the above requirements in mind, we would be happy to conduct our experiments on it.
References:
- Yige Xu, Yongjie Wang, Zizhuo Wu, Kaisong Song, Jun Lin, and Zhiqi Shen. Do vision-language models truly perform vision reasoning? a rigorous study of the modality gap. arXiv preprint arXiv:2604.16256, 2026.
- Jana Zeller, Thaddäus Wiedemer, Fanfei Li, Thomas Klein, Prasanna Mayilvahanan, Matthias Bethge, Felix Wichmann, Ryan Cotterell, and Wieland Brendel. Mentisoculi: Revealing the limits of reasoning with mental imagery. arXiv preprint arXiv:2602.02465, 2026.
Additional controls
Thanks for your review!
W1
We recognize that this task can be done symbolically, in fact any task specified entirely in text can be done symbolically. Our claim is more about the comparison – we claim that VLM training should make the aligned text decoder better at this task than the vanilla text counterpart as the task clearly lends itself to visualization. Furthermore, we rule out long-range dependencies and numerical computation as the primary explanations by providing the distillation result (Fig. 6) because the same architecture is able to do much better when trained to have representations that are more vision-aligned.
We add the un-shuffled edges baseline numbers below. The accuracies are a little bit better however the task remains hard and there is no conclusive winner between the VL aligned text decoder and the pure text decoder for the tested models (and they always lag significantly below the image ceiling)
https://ibb.co/YBRcMv68
W2
We appreciate your concern about a single task, to mitigate them we conduct our contrastive probe experiment on the naturalistic data COCO. The representational quality claims stand on this setting i.e. the VLM text decoder’s representations of captions shows alignment with image embeddings that is on par with the representations from the vanilla text decoder while out-performing representations from a randomly initialized backbone. We attach the COCO probe results here:
https://ibb.co/ZpC5kgJ3
Unfortunately, due to computational constraints we are unable to run the distillation alignment on large-scale real data.
We will soften the language in the discussion to reflect this!
W3
Thanks for suggesting these thoughtful controls!
Direct Label Finetuning
As a text baseline on the data, we train the model to directly answer with labels on samples with shapes (1-4) and evaluate on shapes (1-8). We post the table here:
We note that while the SFT checkpoint does almost perfectly on the in-distribution held out samples (1-4 shapes) but it goes to 0 for the harder out-of-distribution samples. On the other hand, the distilled checkpoint shows gains beyond the distribution seen during training suggesting transfer of visual priors as opposed to generic distributional priors.
Text-Conditioned teacher
The distillation objective is not defined for a text-conditioned teacher because the teacher and student are initialized from the same model. Therefore, calculating both teacher and student embeddings from the same text input leads to the same embeddings (and no loss).
If you meant letting the teacher see un-shuffled / grouped edges and the student see shuffled edges, then the issue is that the difference in accuracy is not huge. The current distilled checkpoint on shuffled edges already beats the un-shuffled baseline by a strong margin. As the student will be capped by the teacher's performance, we don’t expect a student trained on a text conditioned teacher to come close. We attach a plot to demonstrate this:
https://ibb.co/WvhFcNTZ
Shuffled Image-text pairings
For this experiment we simply randomly shuffle the image-text mapping for the training data used for distillation alignment. After this, we run a search over all hyperparameters we ran for the distillation checkpoint (including our best) – all of which break the model and lead to an accuracy of 0 and an output parse-rate near 0.
The data, objective, and training budget are identical to the matched setting, which at the same hyperparameters improves the task performance(0.136 → 0.745 on 1-4 shapes; 0.001 → 0.177 on 5-8) while preserving general language ability (Appendix A.6). The only difference is whether the teacher’s hidden-states are semantically relevant to the student input. The training regime is therefore not destructive in itself; the correspondence is important. This indicates that our checkpoint’s gains depend on transferring structure from image-conditioned representations as opposed to a generic effect of the distillation objective.
Ablations over Target layers
That’s an interesting addition! But attribution to the layers is a question we consider outside the scope of this work. We would be very interested to see future investigation on this.
(continued in the next comment)