acceptodds
Withdrawn from ICLR 2027

MagistrateOPD: Verify Before Distill for Reliable On-Policy Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Existing On-Policy Distillation (OPD) methods combine policy learning from student-generated trajectories with dense token-level teacher guidance. Subsequent methods further strengthen this supervision by equipping a positive teacher with an evidence-centered zoom-in crop and constructing a negative teacher from a masked image. However, such dense supervision implicitly relies on the teacher responses being sufficiently reliable. For smaller models such as 4B-scale multi-modal large language models(MLLMs), additional zoom-in evidence may improve the positive teacher's correct-response rate while its absolute accuracy remains relatively low, substantially weakening the effective supervision signal available to the student. In this paper, we propose MagistrateOPD, which applies pre-judge verification to both teacher responses before token-level on-policy distillation. The magistrate uses an MLLM to compare both teacher responses with the reference answer at the semantic level before admitting a sample into distillation. Pre-judge verification requires the positive-teacher response to match the reference answer and the negative-teacher response not to match it. In addition, we introduce a negative-teacher loss that pushes the student away from the masked-image negative-teacher distribution, thereby strengthening the negative-teacher supervision signal. We evaluate MagistrateOPD on V* Bench, ZoomBench, HR-Bench, and MME-RealWorld. MagistrateOPD achieves 92.15 on V* Bench, 84.12 on HR-Bench-4K, and 72.94 on MME-RealWorld-EN, as well as 61.89 on ZoomBench. Across all six evaluation splits, MagistrateOPD achieves an average score of 78.02, outperforming the base model, SFT, and OPSD by 13.72, 11.17, and 8.76 points, respectively.

Primary area:
reinforcement learning
Keywords:
On-policy distillation, Multi-modal Large Language Model, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1uhXOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes MagistrateOPD, a method for improving visual on-policy self-distillation through dynamic verification of teacher responses. Building on a framework with a crop-conditioned positive teacher and a masked-image negative teacher, the method admits a training example only when the positive teacher’s answer matches the reference and the negative teacher’s answer does not. On admitted examples, it combines positive-teacher distillation with a negative-teacher loss that increases the Jensen–Shannon divergence between the student and the negative teacher. Experiments on Qwen3.5-4B across four benchmarks, comprising six evaluation splits, report improvements over existing visual OPD methods. Ablation studies examine the two proposed components, dynamic versus static filtering, and verification of different teacher responses.

Critical strengths

The paper identifies a meaningful problem and proposes a well-motivated solution. Privileged visual information does not automatically make a teacher reliable. Checking whether the positive and negative teachers fulfill their intended semantic roles is a sensible way to reduce misleading supervision, particularly when explicitly optimizing against a negative teacher.

Dynamic verification is a useful design choice. Since the teacher evolves during self-distillation, the usefulness of a training example can change. Reassessing teacher responses throughout training is therefore well motivated, and the reported improvement over static filtering provides empirical support for this design.

The reported improvements are encouraging and consistent across evaluation splits. MagistrateOPD improves the average score from 74.66 to 78.02 over OPD-V and outperforms it on all six splits. It also exceeds Vision-OPD by 2.85 points on average. The method adds no corresponding verification requirement at inference time.

The ablations provide an informative starting point. The paper evaluates verification and the negative loss separately and jointly, as well as alternative verification criteria. The substantially stronger performance of the combined method suggests a potentially useful interaction between reliable sample selection and negative supervision.

Critical weaknesses

The necessity of jointly gating both losses is not established. Figure 1 shows that 33.33% of examples have both teachers answering correctly. Such examples are discarded entirely, although their positive-teacher supervision may remain useful. An important missing baseline would apply positive distillation whenever the positive teacher is correct, while enabling the negative loss only when the negative teacher is also incorrect. The positive-only verification setting in Table 4 does not resolve this question because it retains the negative loss without checking the negative teacher’s correctness.

Response-level correctness does not directly establish the reliability of the distributions used for distillation. Verification evaluates independently generated teacher answers, whereas distillation queries the teachers on student-generated prefixes. A teacher that answers correctly on its own may behave differently after an erroneous student prefix. Similarly, an incorrect negative-teacher answer does not imply that its full next-token distribution should be discouraged at every selected position. This does not invalidate the method, but the connection between the proposed gate and improved token-level supervision needs stronger empirical analysis.

The generality of the method remains insufficiently demonstrated. All experiments use Qwen3.5-4B and a single training dataset. Since the motivation concerns teacher reliability, model capacity and model family may affect both the proportion of admitted examples and the benefit of verification. Experiments with additional backbones spanning different parameter scales or model families are needed to establish whether the gains and the complementary effects of the two components extend beyond the current setting.

Reproducibility and training-cost comparisons need strengthening. Important details are insufficiently specified, including the teacher update schedule, training duration, judge prompt and decoding settings, and handling of rejected examples. It is also unclear whether static and dynamic filtering are matched in effective training exposure or optimization budget. The appendix acknowledges substantial API latency and GPU idle time, but provides no quantitative cost analysis. Since the method relies on an external judge and reference answers, its gains should be assessed alongside these additional training resources.

Questions for authors

Would retaining positive distillation whenever the positive teacher is correct, while applying the negative loss only when the negative teacher is incorrect, improve performance?

Do the gains generalize to other backbones, model sizes, or model families?

Addressing these concerns in the weakness, I will support an increase to 3/4.

Reviewer_UxmeOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Prior multimodal On-Policy Distillation (OPD) assumes that privileged visual information provides reliable teacher supervision. This paper challenges this assumption, particularly for small teachers, and proposes MagistrateOPD, which verifies that positive-teacher responses match reference answers while negative-teacher responses do not. They further introduce a negative-teacher loss to push the student away from the negative teacher model. Experiments show that MagistrateOPD outperforms OPD-V and traditional OPSD.

Critical strengths
  • Empirical results show that MagistrateOPD can achieve strong performance over other baselines with a detailed ablation study over the effect of pre-judge verification and negative teacher loss.
Critical weaknesses
  • The motivation is somewhat unclear to me. The authors define a sample as valid only when the positive-teacher response is correct and the negative-teacher response is incorrect. However, there is little to no explanation of why these samples are considered valid. Fig. 1 only reports the fraction of valid samples in the training dataset; it does not explain why using "such responses as distillation targets can misdirect the student across many tokens of its on-policy trajectories" (Line 49-51). A rigorous explanation is needed to motivate the use of pre-judge verification, especially since the proposed method appears largely incremental and engineering-driven to me.

  • The paper claims that this problem is severe with small teacher models; this claim needs further clarity. Furthermore, an intuitive fix is to increase the size of the teacher. While this may increase computational overhead, MagistrateOPD also utilizes a large pre-judge, complicating the training pipeline and potentially making the comparison unfair.

  • Compared to previous approaches, MagistrateOPD requires an additional model (pre-judge model) and sampling from both teachers, increasing the number of training models to 4, which significantly increases computational and memory costs.

Questions for authors
  • Can the authors rigorously explain why samples in which the positive-teacher response is correct and the negative-teacher response is incorrect are considered valid? Besides providing an explanation, an additional ablation study examining learning from correct positive-teacher and correct negative-teacher responses, as well as other combinations, is needed to further justify this choice.
  • Could the authors report the computational cost (e.g., FLOPs, training GPU hours) of MagistrateOPD? How does it compare to other approaches?
  • How does MagistrateOPD compare to OPD-V with larger teachers? Since MagistrateOPD also requires a large teacher for pre-verification, this could make the comparison fairer.
Reviewer_5pvFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper addresses unreliable teacher supervision in on-policy distillation in VLMs. The proposed method, MagistrateOPD, exploits an online judge and reference answers to dynamically filter samples, selecting those where the positive teacher provides a correct response while the negative teacher provides an incorrect one. Experiments based on Qwen3.5-4B across four benchmarks and six evaluation subsets demonstrate the effectiveness of the method.

Critical strengths

The paper is clear and easy to follow.

The experimental results show the improvement of the proposed method, and the authors compare their method against many others baselines.

Critical weaknesses
  • A negative teacher does not necessarily need to provide an incorrect answer to be useful. In OPD with negative teachers, it serves as a contrast to the positive teacher and to normalize the rewards; even when the negative teacher provides a correct answer, the difference in probability distributions can still be informative. The requirement for the negative teacher to provide an incorrect answer might primarily be to facilitate the implementation of the additional divergence from the negative teacher loss term.

  • It remains unclear whether the performance gains justify the additional costs. Compared to standard on-policy distillation (even with negative teacher ones), the inclusion of an online judger explicitly increases training costs, yet no cost-adjusted comparison is provided. Furthermore, the use of reference answers introduces an additional dependency; whether this entails new annotation costs depends on whether such answers were already present in the original dataset.

  • The experimental evaluation is insufficient. It is uncertain whether the method has consistent improvement over student-teacher-judger triplets, and there is a lack of validation regarding sensitivity to the choice of the judge itself. The experiments are limited to a single combination of backbone model and judge; therefore, it cannot be concluded that the performance gains remain stable when the judge or the model is changed. Further experiments, such as swapping the judge, assessing sensitivity to noise in the judge's outputs, and conducting multiple training runs, are required.

Questions for authors

Please see the weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.