- Rating
- 2(+1)
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies whether a model weaker than the student can provide useful supervision in OPD. It first runs a controlled token-takeover experiment showing that positions selected by strong/weak teacher probability contrasts affect student accuracy more than KL-based selection. It then proposes cOPD: at student-generated prefixes, a margin-gated mask selects tokens whose reference-student probability lies between a strong and a weak teacher; a tanh-weighted log-probability contrast between the two teachers reinforces or suppresses student probability on these tokens. A proposition under an idealized shared-component mixture model is given to justify the mask. Experiments span three tasks across three teacher-student configurations, against six baselines.
Critical strengths
-
Mechanistic failure-mode analyses. Beyond aggregate scores, the paper diagnoses why competing methods fail.
-
Coverage of teacher-student configurations. Three configurations spanning post-training background, parameter scale, and model family, crossed with three tasks.
Critical weaknesses
- Headline results are obtained in a regime selected to favor the method, and the unfiltered margins are small.
The training set is filtered to prompts where the weak teacher fails on all 8 rollouts and the strong teacher succeeds on at least one — precisely the prompts with the cleanest three-way ordering signal. All main-table results use this filtered set, and the abstract's +7.51pp figure comes from the most extreme teacher configuration (4B/0.6B). On full data the margin over the best single-teacher baseline shrinks to 1.16pp, and no unfiltered control is provided for the IF and QA tasks. As presented, the paper cannot establish how much of the reported gain is attributable to the contrastive mechanism versus the data-selection procedure.
- Limited mechanistic novelty relative to closely related work that is not compared.
The core supervision signal — a log-probability ratio between two models applied at the student's on-policy states — is the mechanism of Direct-OPD (Feng et al., 2026), which shares this paper's motivating question (can a weaker model improve a stronger student) and is cited in related work but not compared experimentally. The margin-gated token selection parallels existing selective-OPD and teachability-margin approaches. Given how crowded this line is in 2026, the defensible contribution narrows to "the contrast pair can be a strong/weak teacher pair rather than a pre/post-RL checkpoint pair" — an instantiation choice whose incremental value is not established against the existing alternatives.
Questions for authors
-
Why is Direct-OPD[1] not included as a baseline? It is the closest prior method in both motivation (weaker model supervising a stronger student) and mechanism (log-ratio contrast as dense token-level signal).
-
What is the mask's empirical behavior: fraction of positions with at least one selected token, positive/negative branch ratio, and average number of selected tokens per position?
-
How sensitive are the results to δ ∈ {0.01, 0.05, 0.1, 0.2} and τ ∈ {0.5, 1, 2}?
-
The Llama configuration is excluded from math experiments because no clear teacher-student gap exists. Doesn't this delimit the method's applicability (it requires a sufficiently weak weak-teacher)?
-
MAD-OPD[2] was proposed and validated on agentic/code tasks and reportedly ranks first in its original settings, yet it underperforms single-teacher KL in nearly all of your configurations. What hyperparameters/tuning budget were used for MAD-OPD and W2S-OPD[3] relative to cOPD?
-
Refer to "Critical Weaknesses"
[1] Feng S, Gao H, Chi H, et al. Weak-to-Strong Generalization via Direct On-Policy Distillation[J]. arXiv preprint arXiv:2607.05394, 2026.
[2] Wang J, Liu Y, Chen J, et al. Mad-opd: Breaking the ceiling in on-policy distillation via multi-agent debate[J]. arXiv preprint arXiv:2605.01347, 2026.
[3] Yu F, Xu W, Xu M, et al. Weak-to-strong on-policy distillation[J]. arXiv preprint arXiv:2607.26246, 2026.
4 comments
Q4: In the Llama configuration, Omni-Math accuracy is 8.93% for strong teacher, 9.25% for student, and 3.68% for weak teacher. We excluded this configuration from the math tasks because the strong teacher did not outperform the student and all three models had low accuracy on math reasoning problems.
Q5: We implemented MAD-OPD using its official repository. All methods, including cOPD, shared common training hyperparameters, such as the learning rate and student rollout settings, adapted from [1]. Method specific parameters for MAD-OPD and W2S-OPD followed their original default configurations from their public code.
We include MAD-OPD to examine whether an existing multi teacher approach can benefit from a weaker teacher. However the weak teacher errors affect the debate generated guidance. These results suggest that incorporating a weak teacher into existing multi teacher methods does not consistently improve upon single teacher OPD, motivating our contrastive design.
Reference
[1] Li Y, Zuo Y, He B, et al. Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe. arXiv:2604.13016, 2026.
Thank you for the response. I’ve raised my rating to 2.
Thank you for reconsidering our work and updating your assessment. Are there any remaining concerns that you feel we have not fully addressed? We would love to clarify and provide additional evidence.
We thank the reviewer for recognizing our failure mode analyses and the coverage of our experimental configurations.
W1: Filtering generated answers for correctness is commonly used in OPD works and has been shown effective in boosting distillation performance [1-3]. Table 4 examines the effect of data filtering in the JustRL configuration by comparison between full data training and filtered training.
We further evaluate filtering in the Qwen3 configuration on IF and SQuAD. We randomly sample data from the original training dataset, matching the filtered datasets in size and training budget. cOPD consistently outperforms all evaluated baselines on all four metrics. All methods score lower than with filtered data, showing that filtering benefits every method evaluated, and cOPD keep its advantage when filtering is removed.
The magnitude of improvement varies across tasks and model configurations. We will revise the abstract to emphasize the overall empirical findings rather than the maximum gain of 7.51 percentage points. Also we want to clarify that the number 7.51 pp and 1.16 pp were from different configurations.
Additional ablation on Qwen3 without teacher performance filtering.
W2/Q1: We included Direct-OPD in our related work. One of our main contributions is selecting informative supervision, motivated in Section 3.1. Direct-OPD transfers the policy shift between checkpoints before and after RL at each training position, using a steering signal similar to proxy steering [4]. cOPD selects candidates through the relative probabilities of the strong teacher, reference student, and weak teacher, without requiring checkpoints from the same RL run.
At a fixed prefix, Direct-OPD’s teacher log ratio is independent of the student. In cOPD, the mask additionally depends on the reference student. After each reference update, candidates stop receiving direct contrastive supervision if they no longer satisfy the margin conditions. The tanh weights further limit the influence of extreme teacher contrasts.
We have added Direct-OPD experiments using its official implementation, adapted to the same teacher pairs, training data, and budget as cOPD. cOPD improves average pass@16 by 5.78 and 9.83 percentage points in the JustRL and Qwen3 configurations, respectively. On OmniMath, cOPD performs better with both JustRL and Qwen3.
Additional experiments on Direct-OPD
Q2: Thank you for this helpful question about the empirical behavior of our masks. We conducted an additional analysis using the Qwen3 configuration on DAPO, IF, and SQuAD. For each task, we randomly sampled 256 questions and generate 2 rollouts per question. Table 4 also reports ablations of the positive and negative branches.
Q3: We provide additional hyper parameter sensitivity analyze in JustRL configuration. Performance remains relatively stable across the tested settings in this configuration. The default setting achieves the highest OmniMath score while maintaining competitive AIME24/25/26/AMC performance.
Reference
[1] Zhang Z, Sun Z, Zhao F, et al. Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillation. arXiv:2609.02998, 2026.
[2] He X, Li X, Wu B, et al. Learn from Whoever Is Right: Answer-Verified Multi-Teacher Distillation for Multi-Domain LLMs. arXiv:2609.02548, 2026.
[3] Hübotter J, Lübeck F, Behric L, et al. Reinforcement Learning via Self-Distillation. arXiv:2601.20802, 2026.
[4] Liu A, Han X, Wang Y, et al. Tuning Language Models by Proxy. arXiv:2401.08565, 2024.