- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies students’ use of visual evidence in multimodal on-policy distillation (OPD). It proposes S-OPD, which augments standard OPD with two objectives: Teacher-calibrated Policy Contrast (TPC), which increases the divergence between original- and masked-image student predictions at teacher-selected tokens, and Policy Agreement (PA), which encourages consistency between original and noisy images. Experiments with Qwen3-VL 2B/4B students, different teacher strengths, and multiple reasoning benchmarks show improvements in average performance.
Critical strengths
1.The paper addresses a meaningful question: whether students effectively use visual evidence, complementing work focused on improving teacher supervision.
2.The method is simple and integrates into existing OPD pipelines without additional annotations, parameters, or inference-time operations.
3.The experiments cover multiple student–teacher configurations and include component ablations, perception benchmarks, and training-cost measurements.
Critical weaknesses
-
The evidence does not sufficiently establish improved visual perception as the source of the gains. With the masked branch detached, TPC amplifies existing original–masked probability differences. Teacher gating selects and weights tokens but neither verifies their correctness nor constrains the direction of the student’s response, so incorrect predictions can also be reinforced. The gating ablations in Table 3 demonstrate useful selection but do not rule out selective confidence reinforcement. The oracle-fact intervention provides supporting evidence, although the 4B gap reduction from 15.1 to 12.1 points partly reflects a decrease in oracle-assisted accuracy from 65.9 to 64.2. Analyses of updates on correct versus incorrect tokens, a direct token-reinforcement control with the same gate, and controlled visual counterfactual tests would help distinguish these explanations.
-
Baseline comparisons need stronger controls for compute and reproduction quality. S-OPD adds 16–28.6% training time without a compute-matched OPD comparison. VA-OPD underperforms vanilla OPD at both scales; for example, its 4B average is 46.51 versus 48.01, and adding S-OPD reaches only 47.23. This warrants clarification of implementation and tuning. S-OPD with the standard teacher also remains below GRPO, such as 37.46 versus 38.64 for the 2B student. The results therefore support improvements over particular OPD baselines, but do not establish a broader performance or efficiency advantage.
-
Statistical uncertainty is missing, and some conclusions are overstated. Main average gains are 0.61–1.37 points, while perception-benchmark changes in Table 8 range approximately from −0.52 to +1.05 points, without multi-seed results, confidence intervals, or paired significance tests. Several individual tasks also regress, so the maximum gain of 4.25 points should not characterize the overall effect. Moreover, a noise-consistency training objective does not itself demonstrate robustness, and two student sizes from one model family provide limited evidence of generality. Repeated runs, paired statistical analyses, and evaluations on unseen perturbations are needed, alongside narrower claims.
Questions for authors
see the weakness