acceptodds
Under review as a conference paper at ICLR 2027

Contrastive On-Policy Distillation via Dual-Branch Supervision

Abstract

For on-policy distillation (OPD), more capable models are a natural choice as teachers. However, higher task performance does not necessarily imply greater teachability. A controlled token intervention study suggests that even a model weaker than the student can serve as an informative contrastive reference. Building on this observation, we propose Contrastive On-Policy Distillation (cOPD), which provides dual branch supervision by contrasting strong teacher, weak teacher, and student predictions at student generated prefixes. Our analysis explains how this three way comparison helps select token candidates associated with better or worse reasoning outcomes under an idealized model. We evaluate cOPD across diverse teacher student configurations spanning different training stages, parameter scales, and model families, on tasks including mathematical reasoning, instruction following, and question answering with abstention. cOPD achieves the best or top-tier performance among the evaluated baselines across all three tasks, improving overall Pass@16 on mathematical reasoning by up to 7.51% over the strongest baseline. Further analyses show that cOPD suppresses undesirable response patterns and suggest that it facilitates exploration.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-policy distillation, LLM post-training, Knowledge distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Q5kWOpenReview ↗
Rating
2(+1)
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies whether a model weaker than the student can provide useful supervision in OPD. It first runs a controlled token-takeover experiment showing that positions selected by strong/weak teacher probability contrasts affect student accuracy more than KL-based selection. It then proposes cOPD: at student-generated prefixes, a margin-gated mask selects tokens whose reference-student probability lies between a strong and a weak teacher; a tanh-weighted log-probability contrast between the two teachers reinforces or suppresses student probability on these tokens. A proposition under an idealized shared-component mixture model is given to justify the mask. Experiments span three tasks across three teacher-student configurations, against six baselines.

Critical strengths
  1. Mechanistic failure-mode analyses. Beyond aggregate scores, the paper diagnoses why competing methods fail.

  2. Coverage of teacher-student configurations. Three configurations spanning post-training background, parameter scale, and model family, crossed with three tasks.

Critical weaknesses
  1. Headline results are obtained in a regime selected to favor the method, and the unfiltered margins are small.

The training set is filtered to prompts where the weak teacher fails on all 8 rollouts and the strong teacher succeeds on at least one — precisely the prompts with the cleanest three-way ordering signal. All main-table results use this filtered set, and the abstract's +7.51pp figure comes from the most extreme teacher configuration (4B/0.6B). On full data the margin over the best single-teacher baseline shrinks to 1.16pp, and no unfiltered control is provided for the IF and QA tasks. As presented, the paper cannot establish how much of the reported gain is attributable to the contrastive mechanism versus the data-selection procedure.

  1. Limited mechanistic novelty relative to closely related work that is not compared.

The core supervision signal — a log-probability ratio between two models applied at the student's on-policy states — is the mechanism of Direct-OPD (Feng et al., 2026), which shares this paper's motivating question (can a weaker model improve a stronger student) and is cited in related work but not compared experimentally. The margin-gated token selection parallels existing selective-OPD and teachability-margin approaches. Given how crowded this line is in 2026, the defensible contribution narrows to "the contrast pair can be a strong/weak teacher pair rather than a pre/post-RL checkpoint pair" — an instantiation choice whose incremental value is not established against the existing alternatives.

Questions for authors
  1. Why is Direct-OPD[1] not included as a baseline? It is the closest prior method in both motivation (weaker model supervising a stronger student) and mechanism (log-ratio contrast as dense token-level signal).

  2. What is the mask's empirical behavior: fraction of positions with at least one selected token, positive/negative branch ratio, and average number of selected tokens per position?

  3. How sensitive are the results to δ ∈ {0.01, 0.05, 0.1, 0.2} and τ ∈ {0.5, 1, 2}?

  4. The Llama configuration is excluded from math experiments because no clear teacher-student gap exists. Doesn't this delimit the method's applicability (it requires a sufficiently weak weak-teacher)?

  5. MAD-OPD[2] was proposed and validated on agentic/code tasks and reportedly ranks first in its original settings, yet it underperforms single-teacher KL in nearly all of your configurations. What hyperparameters/tuning budget were used for MAD-OPD and W2S-OPD[3] relative to cOPD?

  6. Refer to "Critical Weaknesses"

[1] Feng S, Gao H, Chi H, et al. Weak-to-Strong Generalization via Direct On-Policy Distillation[J]. arXiv preprint arXiv:2607.05394, 2026.

[2] Wang J, Liu Y, Chen J, et al. Mad-opd: Breaking the ceiling in on-policy distillation via multi-agent debate[J]. arXiv preprint arXiv:2605.01347, 2026.

[3] Yu F, Xu W, Xu M, et al. Weak-to-strong on-policy distillation[J]. arXiv preprint arXiv:2607.26246, 2026.

4 comments

We thank the reviewer for recognizing our failure mode analyses and the coverage of our experimental configurations.

W1: Filtering generated answers for correctness is commonly used in OPD works and has been shown effective in boosting distillation performance [1-3]. Table 4 examines the effect of data filtering in the JustRL configuration by comparison between full data training and filtered training.

We further evaluate filtering in the Qwen3 configuration on IF and SQuAD. We randomly sample data from the original training dataset, matching the filtered datasets in size and training budget. cOPD consistently outperforms all evaluated baselines on all four metrics. All methods score lower than with filtered data, showing that filtering benefits every method evaluated, and cOPD keep its advantage when filtering is removed.

The magnitude of improvement varies across tasks and model configurations. We will revise the abstract to emphasize the overall empirical findings rather than the maximum gain of 7.51 percentage points. Also we want to clarify that the number 7.51 pp and 1.16 pp were from different configurations.

Additional ablation on Qwen3 without teacher performance filtering.

MethodIF Pass@1SQuAD EMSQuAD F1Correct abstention
RKL20.9365.2270.9758.45
FKL21.9567.1272.8562.76
JSD21.3466.1871.8361.06
cOPD23.9268.9773.4564.32
All scores are percentages.

W2/Q1: We included Direct-OPD in our related work. One of our main contributions is selecting informative supervision, motivated in Section 3.1. Direct-OPD transfers the policy shift between checkpoints before and after RL at each training position, using a steering signal similar to proxy steering [4]. cOPD selects candidates through the relative probabilities of the strong teacher, reference student, and weak teacher, without requiring checkpoints from the same RL run.

At a fixed prefix, Direct-OPD’s teacher log ratio is independent of the student. In cOPD, the mask additionally depends on the reference student. After each reference update, candidates stop receiving direct contrastive supervision if they no longer satisfy the margin conditions. The tanh weights further limit the influence of extreme teacher contrasts.

We have added Direct-OPD experiments using its official implementation, adapted to the same teacher pairs, training data, and budget as cOPD. cOPD improves average pass@16 by 5.78 and 9.83 percentage points in the JustRL and Qwen3 configurations, respectively. On OmniMath, cOPD performs better with both JustRL and Qwen3.

Additional experiments on Direct-OPD

Model familyAIME24AIME25AIME26AMC23AverageOmniMath
JustRL76.67%46.67%33.33%92.77%71.68%21.87%
Qwen346.67%30.00%26.67%78.31%55.49%14.53%

Q2: Thank you for this helpful question about the empirical behavior of our masks. We conducted an additional analysis using the Qwen3 configuration on DAPO, IF, and SQuAD. For each task, we randomly sampled 256 questions and generate 2 rollouts per question. Table 4 also reports ablations of the positive and negative branches.

Taskm+m-DroppedAvg Token/Position
DAPO12.74%25.74%61.52%6.16
IF12.05%33.69%54.26%7.32
SQuAD10.78%46.43%42.79%9.15

Q3: We provide additional hyper parameter sensitivity analyze in JustRL configuration. Performance remains relatively stable across the tested settings in this configuration. The default setting achieves the highest OmniMath score while maintaining competitive AIME24/25/26/AMC performance.

(δ, τ)AIME24AIME25AIME26AMC23AverageOmniMath
(0.05, 1)80.0056.6746.6795.1877.4623.82
(0.1, 1)76.6756.6746.6795.1876.8822.76
(0.2, 1)76.6763.3340.0093.9876.3021.91
(0.05, 0.5)76.6760.0053.3395.1878.6122.83
(0.05, 2)76.6756.6746.6792.7775.7223.47

Reference

[1] Zhang Z, Sun Z, Zhao F, et al. Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillation. arXiv:2609.02998, 2026.

[2] He X, Li X, Wu B, et al. Learn from Whoever Is Right: Answer-Verified Multi-Teacher Distillation for Multi-Domain LLMs. arXiv:2609.02548, 2026.

[3] Hübotter J, Lübeck F, Behric L, et al. Reinforcement Learning via Self-Distillation. arXiv:2601.20802, 2026.

[4] Liu A, Han X, Wang Y, et al. Tuning Language Models by Proxy. arXiv:2401.08565, 2024.

Q4: In the Llama configuration, Omni-Math accuracy is 8.93% for strong teacher, 9.25% for student, and 3.68% for weak teacher. We excluded this configuration from the math tasks because the strong teacher did not outperform the student and all three models had low accuracy on math reasoning problems.

Q5: We implemented MAD-OPD using its official repository. All methods, including cOPD, shared common training hyperparameters, such as the learning rate and student rollout settings, adapted from [1]. Method specific parameters for MAD-OPD and W2S-OPD followed their original default configurations from their public code.

We include MAD-OPD to examine whether an existing multi teacher approach can benefit from a weaker teacher. However the weak teacher errors affect the debate generated guidance. These results suggest that incorporating a weak teacher into existing multi teacher methods does not consistently improve upon single teacher OPD, motivating our contrastive design.

Reference

[1] Li Y, Zuo Y, He B, et al. Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe. arXiv:2604.13016, 2026.

Thank you for the response. I’ve raised my rating to 2.

Thank you for reconsidering our work and updating your assessment. Are there any remaining concerns that you feel we have not fully addressed? We would love to clarify and provide additional evidence.

Reviewer_V6LvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes to use a weak teacher that performs worse than the student in addition to a strong teacher in on-policy distillation. The authors compare their new method, which makes use of a tempered loss, against using only a strong teacher, a weak-to-strong baseline and a multi-teacher baseline and show favorable results on math reasoning, instruction following, and question answering.

Critical strengths
  • The method is quite intuitive and it does make sense that a weak model could help stabilize on-policy distillation by providing a negative signal that can be contrasted with a strong teacher

  • The method provides favorable results on a few benchmarks, though the models that are tried are comparably small

Critical weaknesses
  • While the overall idea seem very intuitive, some of the choices in the main method in Eq. 4 appear a bit out of nowhere. For example, why is a tanh-based loss used instead of a more intuitive divergence-based formulation like in standard OPD? These choices also are not ablated, leaving the reader questioning why they were made.

  • In addition, I found both the motivation in 3.1 and the analysis in 3.3 not very informative. The motivation mainly reiterates empirically that using a worse teacher more often to continue sequences results in worse results while using a stronger model results in better results. The analysis in 3.3 mainly seems to say that an idealized distribution can be viewed as a mixture of both teachers and the student. Both seem rather clear and not warranting such a long discussion. What I would have liked to see instead would be a clearer description of how this work fits into related works but these are only discussed at the end of the paper. I would suggest making this clearer.

Questions for authors
  • The paper motivates its contribution with "higher teacher accuracy does not necessarily translate into greater student improvement through OPD" (which is not necessarily wrong) but could I not rephrase it as "lower teacher accuracy does not necessarily translate into greater student improvement through OPD"? Could you please clarify this a bit as it starts the motivation for your work?

Suggestions for future work:

  • The paper needs a thorough proofreading, there are various spelling and grammar mistake, for example, cf. L141-145
2 comments

We thank the reviewer for recognizing the intuitive behind our design and performance across benchmarks. Below are our responses to the concerns.

W1: The contrast between our two teachers provides signed supervision, but does not directly define a target distribution for KL style loss. Constructing such a target requires additional designs, with potential risks. Inspired by DPO, we instead use the contrast directly to guide the student.

W2S-OPD constructs a proxy teacher distribution and applies reverse KL. Figure 3(b) expose a problem of this construction in our Llama SQuAD setting, where the proxy increases an invalid opening token’s probability from 1.25% under the strong teacher to 71.10%.

Recent work [1] also reports that KL loss may contain extreme negative log ratio and can dominate updates in OPD. Our tanh transform bounds the contrast weights within (-1,1), limiting the influence of extreme teacher log ratios. Also during our preliminary exploration of this idea, we also tried a sigmoid loss over tanh loss, which gave comparable performance on JustRL math setting (AIME24/25/26/AMC: 76.88% vs 77.46% and Omni-Math 23.75% vs 23.82%). We chose tanh because its loss is centered at zero and directly expresses both positive and negative supervision.

W2: We appreciate that the reviewer finds the underlying idea intuitive. Section 3.1 aims to empirically examine the token selection rule by comparing the effects of teacher takeovers at positions selected by our ordering mask and by magnitude of KL divergence. The key observation is that, for the same teacher, our ordering based selection produces larger changes in downstream accuracy than KL based selection. This suggests that teacher contrasts provide useful information beyond the magnitude of teacher student disagreement. Whereas section 3.3 explains how this ordering identifies positive and negative supervision under the stated assumptions.

We agree that the connections to prior work should be clearer earlier in the paper. We will move the related work discussion after the introduction.

Q1: Yes, neither higher nor lower teacher accuracy alone guarantees greater student improvement through OPD. To better identify useful supervision, we seek external information beyond the strong teacher student comparison itself, and considered an additional model’s predictions as a comparative reference. This led us to ask whether even a model weaker than the student could provide useful comparative information alongside a stronger teacher. We will revise the Introduction to clarify this motivation and the role of the weak teacher as an additional source of comparative information.

Q2: Thank you for pointing out these errors. We will thoroughly proofread the paper and correct the grammar and spelling in the revised version.

Reference

[1] Ko J, Abdali S, Kim Y J, et al. Scaling Reasoning Efficiently via Relaxed On-Policy Distillation. arXiv:2603.11137, 2026.

Dear Authors,

Thank you for your response!

I do not have any follow-up questions and will raise my score accordingly.

Reviewer_jCEyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether a model that performs worse than the student can still provide useful supervision in on-policy distillation. The authors first conduct a controlled token-takeover experiment, showing that adopting tokens favored by a strong teacher over both the student and a weak teacher generally improves downstream accuracy, whereas adopting tokens with the reverse preference ordering tends to reduce accuracy. Motivated by this observation, the authors propose Contrastive On-Policy Distillation. At student-generated prefixes, cOPD compares the next-token distributions of a strong teacher, a reference student, and a weak teacher. Its positive branch reinforces tokens relatively favored by the strong teacher, while its negative branch suppresses tokens relatively favored by the weak teacher; tokens that do not satisfy the required three-way probability ordering and margin conditions receive no contrastive supervision. The paper also presents an idealized mixture-model analysis to motivate this selection rule.

The method is evaluated on mathematical reasoning, multi-constraint instruction following, and question answering with abstention, using teacher–student configurations spanning different post-training backgrounds, parameter scales, and model families. Across these settings, cOPD generally achieves the best or near-best student performance among the evaluated distillation methods. Its largest reported gain is 7.51 percentage points over the strongest baseline on pooled mathematical Pass@16. Additional analyses suggest that cOPD can suppress undesirable response patterns, including truncation, instruction-constraint violations, and formatting errors. Ablations on mathematical reasoning further support the effectiveness of the data-selection strategy and the dual-branch supervision design.

Critical strengths
  1. The paper turns a counterintuitive observation into a simple and useful distillation method. Its central insight is that a model weaker than the student may still be valuable—not as a target to imitate, but as a contrast that helps identify undesirable token preferences. The controlled takeover experiment gives this idea a concrete behavioral basis, and cOPD implements it through a selective dual-branch objective without constructing an artificial proxy teacher. This is the paper’s strongest conceptual contribution and offers a meaningful alternative to existing multi-teacher OPD methods.

  2. The empirical results are consistent across tasks, model configurations, and failure modes. cOPD is evaluated on mathematical reasoning, instruction following, and question answering with abstention, and it achieves the best or near-best student performance across the tested settings. Importantly, the paper goes beyond reporting aggregate scores: the truncation and formatting analyses illustrate how cOPD can exploit information from a weak teacher without inheriting—or amplifying—its characteristic errors. This combination of broad performance gains and interpretable behavioral evidence makes the empirical case for the method considerably more convincing.

Critical weaknesses
  1. The evaluation favors cOPD’s intended setting. The main training sets include only examples where the weak teacher fails all eight samples and the strong teacher succeeds at least once, creating an unusually clear teacher contrast. The full-data ablation covers only one mathematical configuration, and no multi-run variance or significance results are reported. It is therefore unclear whether the gains remain robust on unfiltered data and across random seeds.

  2. The proposed mechanism is not directly validated. The theory assumes shared, disjoint good, bad, and neutral token components with favorably ordered mixture weights, largely encoding the desired mask behavior. Beyond the small takeover study, the paper does not test whether positive- and negative-branch tokens actually improve and harm downstream success. The exploration claim also relies mainly on a UMAP plot and one example. Thus, the results support cOPD’s performance more strongly than the paper’s explanation of why it works.

Questions for authors
  1. Clarification on data filtering and generality: The main experiments retain only prompts for which the weak teacher fails all eight sampled responses while the strong teacher succeeds at least once. This creates a particularly clear separation between the two teachers and may be especially favorable to cOPD’s contrastive supervision. Could the authors clarify how much of the reported improvement depends on this filtering criterion? In particular, do similar gains hold on unfiltered training data for instruction following and QA with abstention, or under less restrictive thresholds where the strong and weak teachers have more overlapping performance? This is not merely a suggestion for broader future evaluation: evidence that cOPD remains effective outside the deliberately teacher-separated regime would materially strengthen the paper’s claim of general applicability.

  2. Clarification on robustness and statistical reliability: Were the reported results obtained from a single training run for each method and model configuration? Several improvements over the strongest baseline are relatively modest, while benchmarks such as AIME contain few evaluation problems and may be sensitive to training and sampling variation. Could the authors report results across independent training seeds, uncertainty estimates, or appropriate significance tests for the main comparisons? It would also be helpful to clarify whether the observed ranking of cOPD against the strongest baselines is stable across runs. This information would materially affect whether I interpret the reported gains as robust evidence for the method or as potentially within experimental variance.

1 comment

We thank the reviewer for recognizing our use of weak teachers as contrastive references as a meaningful alternative to existing multi-teacher OPD methods. We also appreciate the reviewer’s assessment that our broad performance gains and interpretable behavioral evidence together make a convincing empirical case for cOPD. Below are our responses to the concerns.

W1/Q1: Filtering generated answers for correctness is commonly used in OPD works and has been shown effective in boosting distillation performance [1-3]. Table 4 examines the effect of data filtering in the JustRL configuration by comparison between full data training and filtered training.

We further evaluate filtering in the Qwen3 configuration on IF and SQuAD. We randomly sample data from the original training dataset, matching the filtered datasets in size and training budget. cOPD consistently outperforms all evaluated baselines on all four metrics. All methods score lower than with filtered data, showing that filtering benefits every method evaluated, and cOPD keep its advantage when filtering is removed.

Additional ablation on Qwen3 without teacher performance filtering.

MethodIF Pass@1SQuAD EMSQuAD F1Correct abstention
RKL20.9365.2270.9758.45
FKL21.9567.1272.8562.76
JSD21.3466.1871.8361.06
cOPD23.9268.9773.4564.32
All scores are percentages.

W1/Q2: Additional training runs with independent seeds were way beyond our computational budget. The consistent gains across different tasks and model configurations support that our method is generalizable and robust. Also our mathematical evaluation uses 16 sampled responses per problem on AIME24, AIME25, AIME26, and AMC23, and also use Omni Math with over 4,000 problems to reduce evaluation variation.

W2: Our motivation experiment in Section 3.1 tests how teacher preferences affect student continuations through temporary teacher takeovers, where m+ takeover improve downstream accuracy and m- reduce it, and both effect are stronger than under KL based selection. Also Table 4 in appendix provides complimentary training evidence, where learning toward positive and learning away negative both improve the student. Combining them achieves the highest average pass@16 accuracy of 77.46% and intermediate Omni-Math score, balancing their strengths.

Figure 3(a) further shows that cOPD reduces truncation errors from 321 to 56 while achieving the highest accuracy, which is consistent with our theory that bad token is identified and penalized. Figure 2 is an additional observation that we discuss briefly in the paper. We will further narrow the exploration claim to better reflect the scope of this evidence.

Reference

[1] Zhang Z, Sun Z, Zhao F, et al. Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillation. arXiv:2609.02998, 2026.

[2] He X, Li X, Wu B, et al. Learn from Whoever Is Right: Answer-Verified Multi-Teacher Distillation for Multi-Domain LLMs. arXiv:2609.02548, 2026.

[3] Hübotter J, Lübeck F, Behric L, et al. Reinforcement Learning via Self-Distillation. arXiv:2601.20802, 2026.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers and AC

We truly thank all reviewers and Area Chairs for time and constructive feedback. The discussion has greatly strengthened the paper. During the discussion, the scores changed as follows: jCEy (2 → 2), V6Lv (2 → 3), and Q5kW (1 → 2). Below we summarize the rebuttal period, including the strengths reviewers identified, and how we addressed the concerns by new evidences.

Strengths Recognized by Reviewers

  • A new role for weak teachers Reviewers found it valuable that a model weaker than the student can still give useful supervision, as a contrastive reference rather than a target to imitate. Reviewer jCEy called this “the paper’s strongest conceptual contribution” and “a meaningful alternative to existing multi-teacher OPD methods.” Reviewer V6Lv found the method “quite intuitive.”

  • Motivation of the design. Reviewer jCEy mentioned that “the controlled takeover experiment gives this idea a concrete behavioral basis.” Reviewer Q5kW also agreed that “positions selected by strong/weak teacher probability contrasts affect student accuracy more than KL-based selection.”

  • Consistent empirical gains with broad coverage. Reviewer jCEy noted that “the empirical results are consistent across tasks, model configurations, and failure modes,” with cOPD achieving “the best or near-best student performance across the tested settings.” Reviewer Q5kW appreciate our “three configurations spanning post-training background, parameter scale, and model family, crossed with three tasks.”

  • Mechanistic failure mode analyses. Reviewer Q5kW highlighted as a key strength that “beyond aggregate scores, the paper diagnoses why competing methods fail.” Reviewer jCEy appreciated how “the truncation and formatting analyses illustrate how cOPD can exploit information from a weak teacher without inheriting—or amplifying—its characteristic errors,” and that this “makes the empirical case for the method considerably more convincing.”

Additional Evidence from the Rebuttal

During the rebuttal, we added the following experiments to address the reviewers’ main concerns

  • Unfiltered training data (jCEy, Q5kW). We trained all methods on randomly sampled, unfiltered data for IF and SQuAD (Qwen3), matched in size and training budget, in addition to our original ablation study on Table 4 for math reasoning. cOPD keep outperforms all single teacher OPD baselines on all four metrics (e.g., IF 23.92 vs. 21.95; correct abstention 64.32 vs. 62.76). This indicates that cOPD’s advantage is not a result of training data filtering.

  • Hyperparameter sensitivity (Q5kW). Across δ ∈ {0.05, 0.1, 0.2} and τ ∈ {0.5, 1, 2}, average pass@16 stays between 75.72 and 78.61. Every setting exceeds the strongest single teacher baseline (75.14). This experiment also reveal that our default (δ = 0.05, τ = 1) is not the best on pass@16, as (0.05, 0.5) reaches 78.61.

  • Mask statistics (jCEy, Q5kW). Across DAPO, IF and SQuAD, 11–13% of positions activate the positive branch, 26–46% activate the negative branch, and 43–62% receive no contrastive supervision. This further confirms our selection behavior in practice.

  • Loss design (V6Lv). A sigmoid variant performs comparably to tanh loss (76.88 vs. 77.46 pass@16). We keep tanh because its loss is centered at zero and directly expresses both positive and negative supervision which better aligns with our design.

  • Comparison with Direct-OPD (Q5kW). As requested by reviewer, we add Direct-OPD as an additional baseline, using its official implementation. cOPD improves average Pass@16 by +5.78 pp (JustRL) and +9.83 pp (Qwen3), and is better on Omni-Math in both configurations. Unlike Direct-OPD, which transfers the policy shift between pre- and post-RL checkpoints, cOPD selects supervision by comparing both teachers against the reference student, so it works with independently trained strong and weak teachers.

In summary, cOPD shows that a model weaker than the student can serve as an informative contrastive reference in On-Policy Distillation, and the additional evidence above further supports its effectiveness and robustness. We will incorporate these suggestions and the additional results into the revised version. We thank all reviewers and the AC again for their valuable feedback, and we are happy to address any remaining questions before the discussion period ends.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.