acceptodds
Withdrawn from ICLR 2027

Same Problem, Different Start: Controlling Text-Image Modality Gaps via First-Decision Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent work has shown that presenting the same problem in Image rather than Text form can induce substantial performance and reasoning differences in multimodal large language models (MLLMs), while distilling full Text-side reasoning traces can substantially recover Image-side performance. But does visual input truly impair the model's ability to reason, or does it simply send a capable model down the wrong generation trajectory, such that minimal steering can redirect it? We find that a remarkably localized intervention can suffice: altering only the first generation decision can substantially redirect the frozen continuation. On a held-out set of Image-presented GSM8K problems, suppressing answer entry only at the first generation position raises accuracy from 41.7% to 93.0%; conversely, forcing naturally non-answer-start Text responses toward answer entry reduces accuracy from 94.8% to 28.9%. Motivated by this finding, we introduce First-Decision Distillation (FDD), which distills only the teacher's first-step decision and uses a clean handoff to leave all subsequent generation to the frozen base model. With only 2 paired training examples and 16 presentations per task, FDD-Logit reduces the Qwen3-VL-8B Text–Image accuracy gap from 12.10 to 1.03 percentage points (pp), a 91.5% reduction. Evaluations across model families, presentation directions, prompt formulations, and structured-chart inputs further test the scope of first-decision control, including an improvement of 10.92 pp from reverse Image-to-Text FDD on Gemma-3-12B. These results identify response onset as a high-leverage, context-dependent control point for reducing Text–Image modality gaps without imitating the teacher's full response.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multimodal large language models, modality gap, visualized text, self-distillation, reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_zNqaOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates why multimodal large language models (MLLMs) can perform substantially differently when the same problem is presented as text versus as an image. The central hypothesis is that visual inputs may not fundamentally impair the model's reasoning ability, but instead may steer generation toward an unfavorable response trajectory at the very beginning of decoding.

The authors first identify the first generation decision as a high-leverage control point. On image-presented GSM8K problems, suppressing answer entry at the first generation position increases accuracy from 41.7% to 93.0%, while forcing naturally non-answer-starting Text responses toward answer entry reduces accuracy from 94.8% to 28.9%. Based on this observation, the paper proposes First-Decision Distillation (FDD), which distills only the teacher's first-step decision and then hands generation back to the frozen base model, avoiding full-response or reasoning-trace distillation.

The paper evaluates FDD across multiple model families, presentation directions, prompt formulations, and structured-chart inputs. With only two paired training examples and 16 presentations per task, FDD-Logit reduces the Qwen3-VL-8B Text–Image accuracy gap from 12.10 percentage points to 1.03 percentage points, corresponding to a 91.5% reduction. The paper further reports a 10.92 percentage-point improvement from reverse Image-to-Text FDD on Gemma-3-12B.

Overall, the paper provides evidence that the onset of generation can act as a highly sensitive control point for multimodal reasoning. Its main contribution is both the empirical characterization of this first-decision effect and the resulting lightweight FDD framework, which attempts to correct modality-dependent generation trajectories while leaving the subsequent reasoning process of the base model unchanged.

Critical strengths
  1. A strong and non-trivial empirical finding about the role of response onset.

The most compelling strength of the paper is the discovery that the first generation decision can exert a remarkably large influence on downstream multimodal reasoning performance. The bidirectional intervention results are particularly compelling: suppressing answer entry for Image-presented GSM8K problems raises accuracy from 41.7% to 93.0%, while forcing answer entry for naturally non-answer-starting Text responses reduces accuracy from 94.8% to 28.9%. This goes beyond simply reporting a Text--Image performance gap and provides evidence that the gap can be substantially manipulated through a highly localized intervention. This observation is conceptually interesting and could motivate further work on generation-onset dynamics in MLLMs.

  1. FDD demonstrates an unusually lightweight way of exploiting this phenomenon.

The proposed First-Decision Distillation is attractive because it does not require distilling the teacher's complete reasoning trajectory. Instead, it supervises only the first generation decision and then allows the frozen base model to continue generation. The reported results are particularly notable given the extremely small amount of supervision: FDD-Logit reduces the Qwen3-VL-8B Text--Image accuracy gap from 12.10 pp to 1.03 pp using only two paired training examples per task. The evaluations across model families, modality directions, prompt formulations, and structured-chart inputs further suggest that this is not merely a single-model or single-setting observation. Taken together, these results make the work potentially valuable both as an empirical study of modality gaps and as a lightweight intervention strategy.

Critical weaknesses
  1. The central mechanistic interpretation is stronger than what the current evidence establishes.

The paper provides compelling evidence that the first generation decision is a high-leverage control point, but this finding does not by itself establish that the Text--Image modality gap is primarily caused by an unfavorable generation trajectory. The intervention experiments demonstrate that changing the first decision can substantially change the final outcome, but they do not fully distinguish the proposed explanation from alternative mechanisms involving visual representation, modality-specific calibration, token-level uncertainty, or prompt-dependent decoding behavior.

This distinction is consequential because the paper's conceptual contribution goes beyond showing that first-token intervention can improve accuracy: it argues that the modality gap can be understood and substantially corrected through response-onset control. Stronger causal controls or more direct mechanistic analyses would be needed to establish this interpretation. Otherwise, the results may support a narrower conclusion that the first generation decision is an effective intervention point, without necessarily establishing it as the primary source of the modality gap.

  1. The methodological contribution of FDD is not yet sufficiently separated from simpler first-token or prefix-level interventions.

FDD is appealingly lightweight, but its methodological novelty is somewhat difficult to isolate from simpler alternatives. Since the method supervises only the first generation decision and then relies entirely on the frozen base model for subsequent generation, it is important to establish whether the gains specifically arise from the proposed distillation formulation and clean handoff, rather than from simply learning an appropriate first-token or short-prefix policy.

For example, comparisons against matched interventions that supervise the first few tokens, distill a short response prefix, or directly optimize the first-token distribution without the proposed distillation procedure would help clarify what is essential to FDD. This is particularly important given the unusually large gains obtained from only two paired examples per task. Without such controls, the paper convincingly demonstrates the effectiveness of first-decision steering, but provides weaker evidence that FDD itself constitutes a substantially new methodological contribution.

Questions for authors
  1. Mechanistic interpretation

The paper argues that the Text–Image modality gap can be understood largely as an unfavorable generation trajectory initiated at response onset. How do the authors distinguish this interpretation from alternative explanations such as modality-specific representation differences, calibration differences, or prompt-dependent decoding effects? In particular, do the authors have controlled experiments showing that the first-decision effect remains after controlling for these factors?

  1. FDD versus simpler first-token interventions

What evidence shows that the gains of FDD come specifically from the proposed First-Decision Distillation formulation and clean handoff, rather than simply from supervising or optimizing the first token (or a very short response prefix)? Do the authors have results comparing FDD with matched first-token, short-prefix, or partial-response distillation baselines?

Reviewer_cTPKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether the accuracy gap between giving an MLLM a problem as text and giving it as a rendered image reflects degraded reasoning, or a different decision at the first decoding step (e.g., answering immediately). A single-position intervention at t = 0 supports the second view: masking the answer-opening token lifts Image/AF-XML GSM8K accuracy from 41.7% to 93.0%. First-Decision Distillation (FDD) then trains a rank-8 LoRA to match the teacher presentation's first-step decision, after which the frozen base model continues. With 10 training pairs across five tasks, the Qwen3-VL-8B gap drops from 12.10 pp to 1.03 (FDD-Logit) / 0.65 (Hard FDD).

Critical strengths

A clearly posed question with striking causal evidence. The two-way intervention is compelling. Masking the answer-opening token at t = 0 lifts Image GSM8K accuracy from 41.7% to 93.0%, and promoting it on reasoning-first Text responses drops accuracy from 94.8% to 28.9% (Table 1 / Table 13). The clean-handoff design separates "the first token changed" from "the continuation changed."

Honest controls and caveats. The paper includes constant-bias, mean-teacher and shuffled-pair controls, and five random training subsets (FDD-Logit is stable at 1.05 ± 0.34). It also openly reports the compliance drop, the PlotQA case where the gap grows, and that wall-time doesn't track the supervised-position savings.

Critical weaknesses

1. Much of the main result reflects an answer-format bias and a non-compliant Text teacher, not recovered reasoning. Under AF-XML ("Give only the final answer"), the Text teacher is strictly compliant only 60.70% of the time, vs. 80.95% for Base Image (Table 24). FDD-Logit lowers Image compliance to 58.25%, and correct-and-compliant accuracy drops from 55.90 to 46.82. In other words, FDD recovers CoT by making the Image side non-compliant too. A constant bias reproduces most of the effect. The Global Constant Bias reaches a gap of 1.00, vs. 1.03 for FDD-Logit (Table 7), and the Task-level Constant Bias, shown only in Table 22, gets 0.85. §4.6 concedes that "a shared first-step bias therefore explains much of the average Qwen accuracy recovery," but the abstract and contribution (1) don't mention this.

2. The generality claims go beyond the evidence. Under RN-XML, no Image response starts with the answer, yet MMLU still shows a −6.0 pp gap and GPQA −5.0 pp (Table 11). On 2B CommonsenseQA, an 11.00 pp gap coexists with perfect phase agreement (F.2). Outside the main Qwen setting, the student moves away from the teacher. On Gemma-3-12B the student–teacher difference goes from 0.15 to 11.07 pp. On InternVL3.5-8B it goes from 0.35 to 2.03, with Image accuracy falling, and on PlotQA the gap grows from 20.75 to 25.83 pp. The Gemma result is an accuracy gain, not a gap reduction, yet the abstract presents it under "reducing Text–Image modality gaps."

Questions for authors
  1. Outside the main Qwen setting, the student moves away from the teacher (Gemma 0.15 to 11.07 pp, InternVL 0.35 to 2.03 pp, PlotQA gap 20.75 to 25.83 pp). Is the objective gap reduction, student accuracy, or onset alignment? Please pick one primary metric and add the macro absolute gap to Table 4.
  2. The Task-level Constant Bias reaches a gap of 0.85, better than the 1.03 of FDD-Logit. Why does main-text Table 7 show only the Global Constant Bias? On the main five-task test, what do zero-training t = 0 masking and a "please reason" instruction achieve?
  3. FDD lowers Image compliance from 80.95% to 58.25%, close to the 60.70% of the Text teacher. How is this different from teaching the Image presentation to break the answer-only instruction? How much does FDD help under contracts that allow reasoning (RN-XML, RR-text)?
  4. How was the training subset "validation-selected," and how many candidates were screened? Its Hard FDD gap (0.65) beats all five random subsets (0.80 to 3.90).
Reviewer_31ftOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces FDD, a localized intervention for multi-modal reasoning models. FDD attempts to provide a minimal steering of MLRMs inorder to recover their reasoning trajectories when visual inputs to reasoning tasks can derail performance. This is based on the premise that visual input primarily re-directs generation at response onset. FDD shows mixed empirical improvments, with certain datasets such as GSM8K demonstrating strong performance improvements.

Critical strengths

The key premise identified is plausible and very interesting on it's own. If the key-premise is true then FDD represents a very promising methodology in advancing the reasoning in MLRMs. FDD's empirical record on the GSM8K dataset is also very strong, which extends to a few other conditions as well. The figures are well done and I would rate the intuitive arguments the paper presents as a strong point.

Critical weaknesses
  1. With the paper's enumerated contributions is both a novel idea from the perspective of mechanistic interpretability, and a new system designed to work around this problem at the same time. While this is not inherently illegitimate, I feel that either one would be sufficient for a full-length paper submission. In the case of the former, I feel as though this premise is highly plausible but not concrete established within an expected level of rigor for the scientific community. I would strongly encourage the author to reframe this as a plausible premise that received supporting evidence as opposed to a phenomena that this research establishes. If the authors are infact convinced on proving this phenomena, then I would front-load the details of how they demonstrated this and encourage iteration during rebuttal.

  2. The actual gains to the model itself seem to vary significantly depending on the dataset. The abstract currently only mentions the GSM8K, but several others were run. Rather than selectively report it's best performance, I think further work should be undertaken to identify why this is the case. Are there commonalities in the task and data? The lesser performance gains on other datasets also challenge the characterization of how effective the specific first-step decision intervention actually is... perhaps it is only true for certain kinds of tasks or data? Perhaps similar interventions work AND the aforementioned premise is true at the same time? Perhaps the aformentioned premise is NOT true, and the first-decision intervention happens to overlap with a more correct decision? I think the experimental work needs more maturity if these questions cannot be answered in rebuttal.

Paper Writing:

  • The results section is largely a datadump with miminal interpretation following it. I would strongly encourage the authors to dedicate more attention there.
  • The related work and conclusion also need work in how they present details in a very dense way. This is possibly because they still retain a bit of an LLM-dialect from text reduction from a previous draft and still need to recieve a bit more human attention for flow and readability (not an accusation).
Questions for authors

See 'Weaknesses'

Reviewer_WpgfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies accuracy gaps between Text and rendered-Image presentations of the same problem in multimodal LLMs. It argues that the gap is largely determined by the model’s first generated token.

On Qwen3-VL-8B/GSM8K, masking the answer-opening token at t=0 raises Image accuracy from 41.7% to 93.0%. Promoting that token on Text lowers accuracy from 94.8% to 28.9%.

Critical strengths

The clean-handoff design is well motivated, and it is tested against One-Decision Full-LoRA under matched training. Hard FDD has the smaller gap at both scales (Table 16). The method is very data-efficient: 10 optimizer updates in total.

Critical weaknesses

The contract. The main evaluation uses AF-XML, which instructs the model to give only the final answer. The Text model often ignores this and reasons, so it scores higher.

Wang & Zhou (2024, CoT-decoding) show that choosing an alternative token at only the first decoding step, followed by greedy decoding, switches models between direct answers and chain-of-thought. This holds for instruction-tuned models too. CoT-decoding is cited only in Appendix A.

Questions for authors

The main evaluation instructs the model to output only the final answer (AF-XML), yet headline accuracy is scored with a parser that also credits responses that reason before answering. Under Text input the model usually ignores the instruction and reasons, which raises accuracy; under Image input it complies more often. Scoring the task as instructed (correct and compliant) reverses the gap: Base Image 55.90% vs. Text 48.25% (Table 24). FDD closes the headline gap by inducing the same non-compliance, reducing strict compliance from 80.95% to 58.25% and correct-and-compliant accuracy from 55.90% to 46.82% (Table 22). When the instruction instead requests reasoning (RR-text), the GSM8K gap falls from 30.0 to 2.0 pp (Table 9).

so it seem the result is a bit cherry picked, or at least the prompt themselves shall be optimized for a more comparable/solid results

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.