- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes MagistrateOPD, a method for improving visual on-policy self-distillation through dynamic verification of teacher responses. Building on a framework with a crop-conditioned positive teacher and a masked-image negative teacher, the method admits a training example only when the positive teacher’s answer matches the reference and the negative teacher’s answer does not. On admitted examples, it combines positive-teacher distillation with a negative-teacher loss that increases the Jensen–Shannon divergence between the student and the negative teacher. Experiments on Qwen3.5-4B across four benchmarks, comprising six evaluation splits, report improvements over existing visual OPD methods. Ablation studies examine the two proposed components, dynamic versus static filtering, and verification of different teacher responses.
Critical strengths
The paper identifies a meaningful problem and proposes a well-motivated solution. Privileged visual information does not automatically make a teacher reliable. Checking whether the positive and negative teachers fulfill their intended semantic roles is a sensible way to reduce misleading supervision, particularly when explicitly optimizing against a negative teacher.
Dynamic verification is a useful design choice. Since the teacher evolves during self-distillation, the usefulness of a training example can change. Reassessing teacher responses throughout training is therefore well motivated, and the reported improvement over static filtering provides empirical support for this design.
The reported improvements are encouraging and consistent across evaluation splits. MagistrateOPD improves the average score from 74.66 to 78.02 over OPD-V and outperforms it on all six splits. It also exceeds Vision-OPD by 2.85 points on average. The method adds no corresponding verification requirement at inference time.
The ablations provide an informative starting point. The paper evaluates verification and the negative loss separately and jointly, as well as alternative verification criteria. The substantially stronger performance of the combined method suggests a potentially useful interaction between reliable sample selection and negative supervision.
Critical weaknesses
The necessity of jointly gating both losses is not established. Figure 1 shows that 33.33% of examples have both teachers answering correctly. Such examples are discarded entirely, although their positive-teacher supervision may remain useful. An important missing baseline would apply positive distillation whenever the positive teacher is correct, while enabling the negative loss only when the negative teacher is also incorrect. The positive-only verification setting in Table 4 does not resolve this question because it retains the negative loss without checking the negative teacher’s correctness.
Response-level correctness does not directly establish the reliability of the distributions used for distillation. Verification evaluates independently generated teacher answers, whereas distillation queries the teachers on student-generated prefixes. A teacher that answers correctly on its own may behave differently after an erroneous student prefix. Similarly, an incorrect negative-teacher answer does not imply that its full next-token distribution should be discouraged at every selected position. This does not invalidate the method, but the connection between the proposed gate and improved token-level supervision needs stronger empirical analysis.
The generality of the method remains insufficiently demonstrated. All experiments use Qwen3.5-4B and a single training dataset. Since the motivation concerns teacher reliability, model capacity and model family may affect both the proportion of admitted examples and the benefit of verification. Experiments with additional backbones spanning different parameter scales or model families are needed to establish whether the gains and the complementary effects of the two components extend beyond the current setting.
Reproducibility and training-cost comparisons need strengthening. Important details are insufficiently specified, including the teacher update schedule, training duration, judge prompt and decoding settings, and handling of rejected examples. It is also unclear whether static and dynamic filtering are matched in effective training exposure or optimization budget. The appendix acknowledges substantial API latency and GPU idle time, but provides no quantitative cost analysis. Since the method relies on an external judge and reference answers, its gains should be assessed alongside these additional training resources.
Questions for authors
Would retaining positive distillation whenever the positive teacher is correct, while applying the negative loss only when the negative teacher is incorrect, improve performance?
Do the gains generalize to other backbones, model sizes, or model families?
Addressing these concerns in the weakness, I will support an increase to 3/4.