acceptodds
Under review as a conference paper at ICLR 2027

Where Cross-Modal Reasoning Distillation Hurts, and How to Undo It

Abstract

Distilling chain-of-thought traces from a text-only language model into a vision-language model is a standard route to multimodal reasoning, and it carries a known side effect: the student sees worse and hallucinates more. The field prices this as the cost of the reasoning gained. But the two instruments behind that price are bent: the answer extractor misses a concise model's answers, in favor of the model that writes more, and the hallucination matcher mistakes similarity for identity and under-counts hallucination. Repaired, they show no measurable gain for the student over the base it started from on any of five benchmarks, while the cost is paid in full. That cost is not one thing but three, and each has its own remedy. (i) Answers the student never states: forcing a conclusion at inference brings them back, and no weight changes. (ii) Hallucination: it is mostly the cost of length, not of reasoning, and a teacher with shorter traces removes it. (iii) Perception cost: it has an address, resolvable only on questions that cannot be answered without the picture — a blind teacher takes the sight its student already had. Averaging the distilled weights back toward the original ones, with no training, recovers most of the lost accuracy while keeping most of the reasoning habit. We call it behavior-preserving interpolation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
cross-modal distillation, vision-language models, chain-of-thought distillation, object hallucination, model merging, evaluation methodology

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9ZwJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the impact of textual reasoning distillation on visual language models, and uses forced termination of answers, short reasoning teachers, and weighted interpolation to mitigate the effects.

Critical strengths

A thorough audit was conducted on evaluation factors such as answer extraction, object matching, and output length.

Weighted interpolation requires no additional training and restores some accuracy while preserving longer outputs.

Critical weaknesses
  1. This paper demonstrates the problem of rule extractors missing answers, but the original VOLD paper explicitly uses VLMEvalKit and GPT-4o-mini for answer extraction. It needs to be proven that these errors also exist in the original evaluation process.

  2. Teachers answering incorrectly without looking at the image does not necessarily mean the question must rely on the image; it could also stem from reasoning failure or output truncation.

Questions for authors

After using VOLD's original answer extraction process, does the spurious performance gain reported in the text still exist

After using an independent model to determine image necessity, are the inter-group differences valid

Reviewer_qCRmOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper analyzes three concepts: answer extraction, hallucination, and perception in five benchmarks (MathVista, MMStar, LogicVista, WeMath, and MathVision). To remedy the unstated answers, forced conclusions with a set of rules were designed, which increased the result by 10 points. For the hallucination problem, the teachers switched teh shorter traces, and for perception performance, they applied weight interpolation.

Critical strengths
  • The Appendix includes detailed analysis and explanations.
  • The quantitative experimental results are shared in detail.
  • The paper clearly states the takeaways after each point.
Critical weaknesses
  • As the experiments and findings are completed mostly on Qwen models, the generalization of the insights and solutions is questionable.
  • The solution for answer format and hallucination is not technical but mainly configuration settings.
  • The motivation and aim of the study are hard to understand from the Abstract and Introduction.
Questions for authors
  • How is the author's system different than VOLD?
  • Is there any performance or insight affected by selecting the distilled (Qwen 2.5-VL) and teacher (Qwen3-8B) models at the same model group?
  • Why are the authors taking only the first 300 questions of the benchmarks?
  • The authors mention an answer template for long-reasoning models; what are these models? Can we generate this response format for all models or just some group of them?
  • For open-ended questions, how does the author's set of rules work?
  • Why do we need a set of rules? Instead, by defining the answer set in the prompt and writing the JSON parsing code, we can prevent format errors.
  • For image-requiring and text-based questions, can the authors verify that the textual questions do not include visual clues/information in the text that directly affect the model performance?
  • Are the takeaways for the three concepts mentioned in the paper valid for all models or just the Qwen family?
  • How can these findings and leverages be generalized for other models and other kinds of tasks, not just the math domain?
  • Why is this study important? What is the main motivation behind this study?
Reviewer_bJdfOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper finds that distilling long chain-of-thought reasoning from a text-only large language model into a vision-language model does not lead to significant accuracy improvements. Instead, it results in degraded visual perception and increased hallucinations, while existing evaluation methods may overestimate the benefits of distillation and underestimate the extent of hallucinations. To address these issues, the paper revises the evaluation protocol, decomposes the performance degradation into three categories and mitigates them through forced answer generation, teachers with shorter reasoning traces, and weight interpolation, respectively.

Critical strengths
  • This paper explores an important problem and examines the biases in conventional evaluation methods, finding that some apparent gains from distillation may actually stem from biases in answer extraction.
  • By using blind-teacher stratification to localize performance degradation, the paper proposes simple and low-cost mitigation methods that do not depend on changes to the model architecture.
Critical weaknesses
  • Overall, the paper demonstrates sufficient novelty. However, considering that many previous studies have reported that longer reasoning texts can lead to worse visual reasoning performance in multimodal tasks, it is unfortunate that the paper does not clearly articulate how previous work and the present study collectively advance our understanding of this problem, despite their somewhat different scopes.
  • The coverage of the core experiments is limited. Although this is understandable as a trade-off due to computational costs, could the authors conduct some experiments on models with different architectures to further validate their findings?
Questions for authors
  • Do similar results hold for larger models? What about smaller models? Is there any relationship with model scale? How about the size of the visual component?
  • Some previous studies have attempted to introduce more fine-grained visual-textual anchors, such as points, during distillation or training, and have also reported positive results. Could such approaches be applied to the present work? Are the methods proposed in this paper orthogonal to these approaches?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.