- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates why multimodal large language models (MLLMs) can perform substantially differently when the same problem is presented as text versus as an image. The central hypothesis is that visual inputs may not fundamentally impair the model's reasoning ability, but instead may steer generation toward an unfavorable response trajectory at the very beginning of decoding.
The authors first identify the first generation decision as a high-leverage control point. On image-presented GSM8K problems, suppressing answer entry at the first generation position increases accuracy from 41.7% to 93.0%, while forcing naturally non-answer-starting Text responses toward answer entry reduces accuracy from 94.8% to 28.9%. Based on this observation, the paper proposes First-Decision Distillation (FDD), which distills only the teacher's first-step decision and then hands generation back to the frozen base model, avoiding full-response or reasoning-trace distillation.
The paper evaluates FDD across multiple model families, presentation directions, prompt formulations, and structured-chart inputs. With only two paired training examples and 16 presentations per task, FDD-Logit reduces the Qwen3-VL-8B Text–Image accuracy gap from 12.10 percentage points to 1.03 percentage points, corresponding to a 91.5% reduction. The paper further reports a 10.92 percentage-point improvement from reverse Image-to-Text FDD on Gemma-3-12B.
Overall, the paper provides evidence that the onset of generation can act as a highly sensitive control point for multimodal reasoning. Its main contribution is both the empirical characterization of this first-decision effect and the resulting lightweight FDD framework, which attempts to correct modality-dependent generation trajectories while leaving the subsequent reasoning process of the base model unchanged.
Critical strengths
- A strong and non-trivial empirical finding about the role of response onset.
The most compelling strength of the paper is the discovery that the first generation decision can exert a remarkably large influence on downstream multimodal reasoning performance. The bidirectional intervention results are particularly compelling: suppressing answer entry for Image-presented GSM8K problems raises accuracy from 41.7% to 93.0%, while forcing answer entry for naturally non-answer-starting Text responses reduces accuracy from 94.8% to 28.9%. This goes beyond simply reporting a Text--Image performance gap and provides evidence that the gap can be substantially manipulated through a highly localized intervention. This observation is conceptually interesting and could motivate further work on generation-onset dynamics in MLLMs.
- FDD demonstrates an unusually lightweight way of exploiting this phenomenon.
The proposed First-Decision Distillation is attractive because it does not require distilling the teacher's complete reasoning trajectory. Instead, it supervises only the first generation decision and then allows the frozen base model to continue generation. The reported results are particularly notable given the extremely small amount of supervision: FDD-Logit reduces the Qwen3-VL-8B Text--Image accuracy gap from 12.10 pp to 1.03 pp using only two paired training examples per task. The evaluations across model families, modality directions, prompt formulations, and structured-chart inputs further suggest that this is not merely a single-model or single-setting observation. Taken together, these results make the work potentially valuable both as an empirical study of modality gaps and as a lightweight intervention strategy.
Critical weaknesses
- The central mechanistic interpretation is stronger than what the current evidence establishes.
The paper provides compelling evidence that the first generation decision is a high-leverage control point, but this finding does not by itself establish that the Text--Image modality gap is primarily caused by an unfavorable generation trajectory. The intervention experiments demonstrate that changing the first decision can substantially change the final outcome, but they do not fully distinguish the proposed explanation from alternative mechanisms involving visual representation, modality-specific calibration, token-level uncertainty, or prompt-dependent decoding behavior.
This distinction is consequential because the paper's conceptual contribution goes beyond showing that first-token intervention can improve accuracy: it argues that the modality gap can be understood and substantially corrected through response-onset control. Stronger causal controls or more direct mechanistic analyses would be needed to establish this interpretation. Otherwise, the results may support a narrower conclusion that the first generation decision is an effective intervention point, without necessarily establishing it as the primary source of the modality gap.
- The methodological contribution of FDD is not yet sufficiently separated from simpler first-token or prefix-level interventions.
FDD is appealingly lightweight, but its methodological novelty is somewhat difficult to isolate from simpler alternatives. Since the method supervises only the first generation decision and then relies entirely on the frozen base model for subsequent generation, it is important to establish whether the gains specifically arise from the proposed distillation formulation and clean handoff, rather than from simply learning an appropriate first-token or short-prefix policy.
For example, comparisons against matched interventions that supervise the first few tokens, distill a short response prefix, or directly optimize the first-token distribution without the proposed distillation procedure would help clarify what is essential to FDD. This is particularly important given the unusually large gains obtained from only two paired examples per task. Without such controls, the paper convincingly demonstrates the effectiveness of first-decision steering, but provides weaker evidence that FDD itself constitutes a substantially new methodological contribution.
Questions for authors
- Mechanistic interpretation
The paper argues that the Text–Image modality gap can be understood largely as an unfavorable generation trajectory initiated at response onset. How do the authors distinguish this interpretation from alternative explanations such as modality-specific representation differences, calibration differences, or prompt-dependent decoding effects? In particular, do the authors have controlled experiments showing that the first-decision effect remains after controlling for these factors?
- FDD versus simpler first-token interventions
What evidence shows that the gains of FDD come specifically from the proposed First-Decision Distillation formulation and clean handoff, rather than simply from supervising or optimizing the first token (or a very short response prefix)? Do the authors have results comparing FDD with matched first-token, short-prefix, or partial-response distillation baselines?