acceptodds
Withdrawn from ICLR 2027

Is Better Teacher Supervision Enough? Unlocking Student-Side Learning in Multimodal On-Policy Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

23 views

Abstract

On-policy distillation (OPD) improves reasoning by providing token-level supervision from a teacher on a student's own trajectories. Existing methods primarily focus on enhancing this teacher-side guidance (e.g., by enriching teacher inputs and refining teacher feedback), yet we find that limited student perception is another critical bottleneck in multimodal OPD. By providing oracle visual facts, the performance of OPD-trained students can still be substantially improved for both weak and strong teachers. To address this bottleneck, we propose S-OPD, a simple multimodal on-policy distillation framework that explicitly strengthens student perceptual learning through two objectives. Specifically, *Teacher-calibrated Policy Contrast* separates student policies under original and masked images with teacher-based token-level gating, strengthening the student's reliance on visual evidence during reasoning. *Policy Agreement* aligns student policies under original and noise-perturbed images, further improving perceptual robustness to visual noise. Notably, our method can be seamlessly plugged into existing OPD frameworks, requiring no additional data annotations, model parameters or inference operations. Extensive experiments on eight benchmarks across student scales and distillation paradigms demonstrate consistent performance improvements, with gains of up to 4.25 points on LogicVista. When combined with existing teacher-side supervision methods, our method can yield further gains. Code is available at https://anonymous.4open.science/r/S-OPD-0322/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Learning, On-Policy Distillation, Knowledge Distillation, Multimodal Large Language Models, Visual Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Zm62OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies students’ use of visual evidence in multimodal on-policy distillation (OPD). It proposes S-OPD, which augments standard OPD with two objectives: Teacher-calibrated Policy Contrast (TPC), which increases the divergence between original- and masked-image student predictions at teacher-selected tokens, and Policy Agreement (PA), which encourages consistency between original and noisy images. Experiments with Qwen3-VL 2B/4B students, different teacher strengths, and multiple reasoning benchmarks show improvements in average performance.

Critical strengths

1.The paper addresses a meaningful question: whether students effectively use visual evidence, complementing work focused on improving teacher supervision.

2.The method is simple and integrates into existing OPD pipelines without additional annotations, parameters, or inference-time operations.

3.The experiments cover multiple student–teacher configurations and include component ablations, perception benchmarks, and training-cost measurements.

Critical weaknesses
  1. The evidence does not sufficiently establish improved visual perception as the source of the gains. With the masked branch detached, TPC amplifies existing original–masked probability differences. Teacher gating selects and weights tokens but neither verifies their correctness nor constrains the direction of the student’s response, so incorrect predictions can also be reinforced. The gating ablations in Table 3 demonstrate useful selection but do not rule out selective confidence reinforcement. The oracle-fact intervention provides supporting evidence, although the 4B gap reduction from 15.1 to 12.1 points partly reflects a decrease in oracle-assisted accuracy from 65.9 to 64.2. Analyses of updates on correct versus incorrect tokens, a direct token-reinforcement control with the same gate, and controlled visual counterfactual tests would help distinguish these explanations.

  2. Baseline comparisons need stronger controls for compute and reproduction quality. S-OPD adds 16–28.6% training time without a compute-matched OPD comparison. VA-OPD underperforms vanilla OPD at both scales; for example, its 4B average is 46.51 versus 48.01, and adding S-OPD reaches only 47.23. This warrants clarification of implementation and tuning. S-OPD with the standard teacher also remains below GRPO, such as 37.46 versus 38.64 for the 2B student. The results therefore support improvements over particular OPD baselines, but do not establish a broader performance or efficiency advantage.

  3. Statistical uncertainty is missing, and some conclusions are overstated. Main average gains are 0.61–1.37 points, while perception-benchmark changes in Table 8 range approximately from −0.52 to +1.05 points, without multi-seed results, confidence intervals, or paired significance tests. Several individual tasks also regress, so the maximum gain of 4.25 points should not characterize the overall effect. Moreover, a noise-consistency training objective does not itself demonstrate robustness, and two student sizes from one model family provide limited evidence of generality. Repeated runs, paired statistical analyses, and evaluations on unseen perturbations are needed, alongside narrower claims.

Questions for authors

see the weakness

Reviewer_keEuOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates student visual perception as a bottleneck in multimodal on-policy distillation (OPD), motivated by oracle visual-fact interventions and an observed association between visual sensitivity and reasoning accuracy. It proposes S-OPD, which augments standard OPD with two objectives: teacher-calibrated policy contrast, which selectively maximizes divergence between student policies under original and masked images, and policy agreement, which encourages consistency under Gaussian image perturbations. Experiments across eight reasoning benchmarks, two student sizes, and two teacher configurations show consistent average improvements. Component and gating ablations support the proposed design, while experiments combining S-OPD with VA-OPD and OPSD demonstrate compatibility with existing distillation approaches.

Critical strengths
  1. Teacher-calibrated token selection is a useful refinement of visual policy contrast. Rather than encouraging divergence between original- and masked-image policies at every token, S-OPD uses the teacher’s cross-image probability changes to select and weight the contrastive objective. The controlled comparisons against ungated, student-gated, and count-matched random gating in Table 3 support the value of this design.

  2. The experiments demonstrate consistent average gains and compatibility with existing distillation methods. S-OPD improves average performance across eight benchmarks for both student sizes and both teacher configurations, and provides additional gains when combined with VA-OPD and OPSD.

Critical weaknesses

1.Limited methodological novelty relative to PAPO. The central mechanism—maximizing policy divergence between original and masked images to encourage visual grounding—has already been explored in PAPO. S-OPD mainly adapts this mechanism to OPD, adds teacher-calibrated token selection, and introduces a noise-consistency objective. Although the gating ablations support this refinement, the paper lacks a direct comparison with PAPO adapted to OPD, including its stabilization mechanism, leaving the advance over prior work insufficiently established.

2.Evidence for improved visual perception is limited. Oracle visual facts also simplify reasoning, and visual sensitivity–accuracy correlations do not establish causality. The case in Appendix C.1.2 reflects better reasoning rather than perception, while gains on perception benchmarks are modest and mixed.

3.Unclear benefits relative to training cost and complexity. Additional teacher scoring and student image branches increase per-step training time by 16.0%–28.6%(Table 11), for average gains of 0.61–1.37 points over OPD(Table 1). With the original teacher, S-OPD also trails direct GRPO at both student sizes. Comparisons under matched computational budgets are needed to justify the added complexity.

Questions for authors

1.How does S-OPD compare with PAPO adapted to the same OPD setting?

2.What evidence, such as attention-map analysis or controlled perception tests, distinguishes improved visual perception from better reasoning or general regularization?

Reviewer_wxY2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on a bottleneck in multimodal on-policy distillation (OPD): student models learn to mimic the teacher's text reasoning style, but still fail to read the image properly.

To fix this, S-OPD takes the student's existing response and scores it on two modified images: a 60%-masked image and a noisy image. It adds two simple rules:

  1. Contrast on masked images (TPC): Push the student's predictions on the original image away from its predictions on the masked image, forcing it to rely on visual evidence. Because doing this on all tokens ruins basic grammar tokens, the authors use the teacher as a filter. They only apply this push on tokens where the teacher's confidence drops when the image is masked.
  2. Consistency on noisy images (PA): Pull the student's predictions on the original and noisy images together, so the model doesn't overreact to minor pixel noise.

Across 8 benchmarks, S-OPD beats vanilla OPD by +0.6 to +1.4 average points with zero extra test-time cost.

Critical strengths
  1. Clean bottleneck test

Rendering Geometry3K diagram annotations into text (without leaking answers or theorems) is a neat, direct way to separate perception failures from reasoning failures.

  1. Using the teacher as a token filter

Forcing masked-image contrast on all tokens hurts fluency, and student-based gating is even worse. Reusing the teacher's probability drop under masking to select vision-dependent tokens is practical and well-motivated.

  1. Low overhead

Because perturbed images are only scored on the student's existing rollout, training time increases by only 16%–29% and inference is unchanged.

Critical weaknesses
  1. The claim that S-OPD complements teacher-side methods does not hold up in Table 1 and Fig. 5.

In the authors' runs, VA-OPD and 2B OPSD both score 1.0 to 1.5 points below vanilla OPD. As a result, combining them with S-OPD is actually worse than running S-OPD alone in 3 out of 4 settings. Improving slightly over a degraded baseline does not show complementarity when dropping VA-OPD/OPSD entirely works better.

  1. Direct student GRPO beats base-teacher S-OPD, and the PAPO baseline is missing (Table 1).

With the base 8B teacher, S-OPD loses to running GRPO directly on the student (37.46 vs. 38.64 on 2B; 48.62 vs. 49.72 on 4B), and only wins after the 8B teacher is first trained with GRPO on ViRL39K.

  1. Weak 4B results and a misleading Oracle Gap reduction on 4B (Table 1 & Fig. 8).

On the 4B student (base teacher), S-OPD only gains +0.61 on average over OPD, and scores lower than the untuned 4B base model on MathVision and WeMath. In Fig. 8, the authors claim S-OPD shrinks the 4B oracle gap from 15.1% to 12.1%, but over half of that reduction comes from lowering the oracle upper bound (65.9% to 64.2%), while image-only accuracy barely moves (50.7% to 52.1%).

Questions for authors
  1. Why do VA-OPD (Table 1) and 2B OPSD (Fig. 5) score lower than vanilla OPD, making their combination with S-OPD worse than S-OPD alone? Can you test S-OPD on top of a visual teacher-side baseline that actually beats vanilla OPD (e.g., Vision-OPD or VAD)?
  2. Since student GRPO beats base-teacher S-OPD, how does S-OPD compare against running PAPO (or Student GRPO + your TPC/PA losses) directly on the student?
  3. Why does distilling from the base 8B teacher hurt the 4B student on MathVision and WeMath compared to the 4B base model, and why does S-OPD drop the 4B upper-bound score in Fig. 8 (65.9% to 64.2%)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.