acceptodds
Withdrawn from ICLR 2027

Beyond Imitation: Multi-Teacher On-Policy Distillation for Multi-Modal LLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

On-policy distillation (OPD) has demonstrated its effectiveness in improving multimodal reasoning with multimodal large language model (MLLM) teachers. In this paper, we investigate how a text-only LLM teacher can complement MLLM supervision in multimodal OPD. We find that directly distilling from the LLM teacher can substantially degrade student performance. Although its predictions lack direct visual grounding, they reveal which reasoning continuations are favored by textual context alone. We therefore use these predictions as a reference to constrain excessive text-only alignment rather than as targets to imitate. Based on this insight, we propose Role-Aware Heterogeneous Teacher Guidance (RouTe), which combines distillation from the MLLM teacher at visually dependent positions with controlled separation from the LLM teacher's prediction distribution at high-entropy positions in reasoning trajectories. RouTe dynamically adjusts the separation strength according to the student's relative divergence from the two teachers. Experiments across multiple multimodal reasoning benchmarks and student–teacher configurations demonstrate consistent improvements over existing OPD and multimodal distillation methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
MLLM, OPD

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_T5MMOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses how to combine a multimodal teacher and a text-only LLM in on-policy distillation. It argues that direct imitation of the text-only teacher suppresses image-dependent variation and creates token-level conflicts with the multimodal teacher. RouTe therefore uses different roles: it selects visually dependent positions from the student's with-image versus without-image log-probability difference and applies multimodal-teacher distillation there; it selects high-entropy nonvisual positions and reverses the text-only teacher's distillation signal there; and it sets the separation coefficient from batch-level student-teacher log-ratio statistics. Training on Geometry3K is evaluated on MathVerse, MathVision, MathVista, WeMath, and HallusionBench with 3B and 7B Qwen-family students and multimodal teachers. RouTe improves average performance over standard multimodal OPD in all three reported student-teacher configurations.

Critical strengths

Writing is clear. The writing is overall clear and easy to understand.
Idea is straightforward. Use heterogenous teachers as supervision is straightforward in post-training.

Critical weaknesses

Baseline is not strong enough. Tthe base model authors use in their experiments, Qwen2.5-VL. Improvements over these models (even in large margins) cannot guarantee the same improvements on frontier models (e.g., Qwen3.5). The paper ignores the direct comparison with Vision-OPD, which uses Qwen3.5 as the base model.

Should include extra compute. Multi-teacher could bring significant additional burden during training, so it is crucial to at least include how many additional compute their design brings, in comparison to OPSD itself.

Questions for authors

See Weakness. Better do experiments on Qwen3.5 as well to prove me wrong. Will the multi-teacher design benefit Qwen3.5 and some benchmarks like HRBench?

Reviewer_qKPEOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Role-Aware Heterogeneous Teacher Guidance (RouTe), an on-policy distillation method for multimodal reasoning that combines a multimodal teacher with a text-only LLM. The authors analyze how directly imitating an image-blind LLM can suppress image-dependent predictions and produce updates that conflict with multimodal supervision. RouTe instead assigns different roles to the teachers: it aligns the student with the multimodal teacher at positions selected using image-conditioned probability changes, while reversing LLM distillation guidance at high-entropy positions outside the visual mask. A batch-dependent coefficient controls the relative strength of these objectives. Experiments on five benchmarks report average improvements of 2.0–3.6 percentage points over standard multimodal-only OPD across three student–teacher configurations.

Critical strengths
  1. RouTe outperforms standard multimodal-only OPD on all five benchmarks across the three reported student–teacher configurations. Although the magnitude varies, this consistency provides empirical support for the combined method within the evaluated settings.
  2. The paper explores assigning complementary roles to teachers with different input modalities. In particular, using the text-only LLM as a separation reference, rather than an additional imitation target, offers a concrete alternative to directly combining teacher predictions.
  3. The experiments compare the full method against visual token selection without LLM guidance and against a fixed LLM coefficient. The full method performs best across all five benchmarks in the reported ablation setting, supporting the usefulness of the combined design, even though further controls are needed to establish the proposed explanation.
  4. Including HallusionBench alongside four mathematical reasoning benchmarks broadens the empirical evaluation.
Critical weaknesses

References: [1] Wang et .al, Perception Aware Policy Optimization (PAPO) for Multimodal Reasoning, ICLR 2026 (https://arxiv.org/pdf/2507.06448).

I have the following major concerns. If I have a misunderstanding or if the authors provide the relevant justification and experiments, I am open for discussion:

  1. I am unclear about the point of section 3.1. In the related work, authors note that prior work uses the LLM teacher for improving a MLLM student on text-only reasoning tasks. This makes sense since MLLMs may encourage more attention on the image and have a harder time distributing attention to text only sequences and reasoning chains. However, in section 3.1 the inputs are multimomal, so it may already be clear that the only LLM input can hurt the MLLM since the LLM teacher, based on the set up, does not have information of the image.
  2. In section 3.2, disagreement does not establish that the MLLM is using the image correctly. For example, in a study done in PAPO, preliminary results observed that 67% of the reasoning errors in Qwen 2.5 VL model were attributed to perception errors. Further, the disagreement could also reflect different language preferences, confidence, or errors. 3.Equation 12, selecting the visual dependent tokens: (1) Image sensitivity does not establish correct visual grounding as mentioned in second concern; (2) Visual facts already written in the prefix can make later image-dependent reasoning appear insensitive to removing the image; (3) Selecting across the entire batch can concentrate supervision on particular examples; per-response averaging does not eliminate that selection imbalance.
  3. Section 3: calculating the γL\gamma_L. The coefficient γL\gamma_L can become unstable as uLu_L can approach 0. See question 1.
  4. Section 4.2 "The student may remain uncertain about how to continue the reasoning trajectory even with access to visual information" is the most important motivation for adding a text-only LLM to a MLLM and yet it is not motivated. Useful tests would include manually categorized errors, token-selection precision, and comparisons against random selection, entropy-only regularization, and a caption-conditioned LLM. See also question 2.
  5. The benefit for RouTE is not widespread and this deserves discussion. For all datasets except HallusionBench, which has increase of 5.2%, Qwen 2.5. VL 7B model only has <=3.3% increase over adding just OPD (which I assume is standard MLLM teacher training). why are we seeing more improvements on HallusionBench? what characteristics of the datasets make RouTE have higher gains than others? Similar question for Qwen 2.5 VL 3B: why does RouTE do much better on only OPD on MathVista, WeMath, but only just a bit better on MathVerse? Question 3 is this concern in question format.
  6. The paper should report peak memory, GPU-hours, throughput, and accuracy at matched compute. RouTe requires two teacher models, student evaluations with and without visual input, and additional token-selection and calibration procedures. Its average improvements over standard OPD are 2.0–3.6 percentage points across the reported configurations and is the performance increase over GRPO on average across all datasets is 5%. These gains warrant evaluation against the additional training cost. See also question 4.
  7. An error analysis of the reasoning traces of RouTE is missing. See question 5 for more details.
  8. My understanding of the paper is that we can use MLLM and LLM teacher in the form of a contrastive objective: in the case of multimodal input, push the student distribution towards the correct MLLM teacher distribution and away from a distribution of the text only LLM, which, based on set up, does not use the image. Vice versa for text-only. In this case, I would suggest the authors compare RouTE to simpler visual grounding methods like PAPO [1].
  9. The experiments use a limited set of student and teacher configurations. It remains uncertain whether the proposed teacher roles and calibration strategy transfer across model families (e.g., InternVL, GLM), alternative text-only teachers, and alternate RL objectives like DAPO.
Questions for authors

[1] Wang et .al, Perception Aware Policy Optimization (PAPO) for Multimodal Reasoning, ICLR 2026 (https://arxiv.org/pdf/2507.06448).

  1. What guarantees or implementation safeguards keep (\lambda_L) negative and numerically stable? How often do the discrepancy estimates have opposite signs or a near-zero denominator, and what is the observed distribution of coefficient values?
  2. What evidence shows that the high-entropy, non-visual-mask positions selected in Section 4.2 correspond to reasoning-critical decisions? How frequently do they instead reflect unresolved perception errors, ambiguity, or alternative valid wordings? In what situations does the LLM teacher corrects a reasoning trajectory? When the teachers conflict, does the disagreement between teachers disappear when the LLM receives an accurate image description?
  3. Why is the largest 7B improvement observed on HallusionBench? Do breakdowns by task type, visual dependence, answer bias, or error category explain the different gains across HallusionBench, MathVision, MathVista, MathVerse, and WeMath? Similar question for Qwen 2.5 VL?
  4. How does RouTe compare against equal-budget random token selection, entropy-only token selection or regularization, and a caption-conditioned LLM teacher? These controls would help isolate the contributions of token selection and heterogeneous-teacher separation.
  5. Which error categories decrease after RouTe, and which increase? Can the authors report corrected versus newly introduced errors, supported by reasoning-trace examples and uncertainty estimates across training runs? Authors can follow the error analysis done in PAPO [1]
Reviewer_qgh3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how multimodal and text-only teachers should contribute to multimodal on-policy distillation. A fixed-context reverse-KL decomposition identifies a pressure toward image-invariant predictions, and a sampled-token analysis characterizes local conflicts between teacher signals. RouTe assigns multimodal-teacher supervision to visually dependent positions and intends to reverse text-teacher guidance at disjoint high-entropy positions. A ratio of selected-token student–teacher log-ratio means calibrates the text coefficient. Training uses approximately 2.1K Geometry3K examples. Table 1 reports average improvements over multimodal-only OPD of 2.0, 3.6 and 2.7 points across three configurations on five benchmarks. Table 2 compares visual selection alone, fixed negative guidance and the adaptive method.

Critical strengths
  1. A mathematically grounded and conditional diagnosis. Equation (6) is a valid decomposition at fixed textual context, and Equations (8)–(10) correctly characterize opposing local updates for the same sampled-token score vector. The paper acknowledges that the prefix can already contain sufficient visual information. Figure 1 also compares direct and combined teacher guidance, lending empirical support to the stated risk.
  2. Consistent reported gains with useful initial controls. Table 1 improves all five scores in each matched OPD comparison. Table 2's means are 49.64 for standard OPD, 50.16 for visual selection alone, 51.52 for fixed negative guidance and 53.16 for RouTe. This progression supports investigating the additional reference term and calibration, while leaving their precise mechanism open.
Critical weaknesses
  1. The calibration rule does not guarantee separation. The sampled log-ratio means in Eq. (18) can have either sign, so Eq. (19) guarantees neither a negative coefficient nor bounded magnitude. Sign/magnitude safeguards and empty-mask handling are unspecified. Whether these cases affected the reported runs remains unknown.
  2. Attribution and reproducibility remain incomplete. No matched control isolates high-entropy selection; the fixed-negative coefficient and tuning protocol are unspecified, and the random visual-mask test lacks numerical results (p. 8). Missing K/N and training settings, plus inconsistent 7B model names in Table 1 and §5.1, hinder reproduction.
Questions for authors
  1. Clarification and decisive evidence: How does the actual implementation handle coefficient signs, near-zero denominators and empty masks? Please provide coefficient statistics/traces and identify any differences from Eq. (19) and the affected results.
  2. Additional evidence: What gains remain against comparably tuned fixed-negative guidance and an equal-count random reasoning mask? Please also report the numerical random visual-mask result described on p. 8.
  3. Clarification: Please reconcile the 7B checkpoint names and specify K/N, loss averaging, optimizer/rollout settings and tokenizer alignment.
  4. Additional evidence: How do gains and coefficient behavior vary across independent training runs? Please state hyperparameter-selection data and extra-pass costs. Fixed-output evaluation intervals would quantify sample uncertainty separately.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.