- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes COPD and SACOPD. COPD combines both positive and negative teachers in a two-sided conditional KL objective to extrapolate beyond the positive teacher. Based on the teacher-defined axis, the authors further propose SACOPD, which projects the current student onto this axis and adaptively places the target a fixed distance ahead of the student. Experiments on math and code benchmarks outperforms standard OPD, with SACOPD further improving the performance over COPD.
Critical strengths
- I think the idea of SACOPD is interesting. Instead of using a fixed target throughout training, it tries to adapt the target based on the current student state.
- The paper provides reasonable theoretical analysis of the virtual teacher and student projection, which helps explain the proposed method.
- The method does show improvements over standard OPD on both math and code benchmarks.
Critical weaknesses
- The authors mentioned G-OPD [3] and ExOPD in the related work. But to me, it is still not clear how COPD is fundamentally different from them. Specifically, by setting the reference model in G-OPD to the negative teacher and using , isn't the proposed COPD objective in Eq. (2) exactly equivalent to for a fixed token context? The negative teacher here also seems to play the same role as the reference model in G-OPD. I understand that the formulation and training implementation can be different, but the objective itself looks mathematically equivalent up to a scaling factor. In particular, COPD uses for math, which corresponds to , while the ExOPD baseline uses . Then, how much of the improvement comes simply from using a larger extrapolation coefficient? I think a strength-matched ExOPD comparison under the same optimization settings would be important here.
- Also, in general, using contrastive signals in distillation itself does not look that novel to me. Contrastive supervision has been quite common even before these recent RL-style OPD formulations like DistiLLM-2 [2], although the exact formulation is different. I understand that the novelty may instead come from projecting the student onto the teacher axis and using this to adjust the target. But then, why is this projection necessary? The authors provide some mathematical properties of the projection, but it is still not clear why this is a better way to adjust the extrapolation strength compared to simpler alternatives (something like directly using the teacher-student KL or disagreement). The paper does compare with a global replay schedule, but the improvement is rather small. I think more justification or ablations are needed to show why this particular projection-based controller is needed and important.
- There is not enough explanation of what SACOPD actually does in the introduction, or what exactly is meant by "state-aware." Although some explanation is given in the contribution bullet points and later sections, I think the main idea (placing the target a fixed distance ahead of the student's projected position) should be introduced more clearly at the beginning.
- The main result tables do not provide error bars or standard deviations in the main tables. For math benchmarks like AIME, they have only 30 problems per benchmark and can be quite noisy. It is hard to tell whether some of the relatively small improvements are statistically meaningful.
- The improvement from COPD to SACOPD is very marginal for math mean@32 (58.91 to 59.48). Since SACOPD seems to be the main new contribution beyond the existing extrapolation methods, I am not sure if this is a sufficiently strong gain. If the main benefit is also improving pass@K or maintaining answer coverage, then I think the paper should compare with methods specifically designed to preserve diversity in OPD, such as EOPD [1]. Otherwise, it is hard to understand whether the improvement in pass@K comes from the proposed state-aware target selection or whether it could also be achieved by better preserving generation diversity.
- Also, I think the matched training budget is quite small, with only 50 updates and a global batch size of 6 (300 rollouts). It seems like the appendix includes longer training exposure results for COPD and OPD, but I would like to see what happens to SACOPD with more compute. Does the improvement remain when training for more steps or using a larger training budget? Since the proposed controller is supposed to adjust the target as the student progresses, I think showing its benefit under longer training would be particularly important.
- In Section 5.2, the authors seem to argue that the method is robust to different hyperparameter choices. But I am not sure if this is necessarily strong evidence for the proposed method. For example, changing the lookahead from 1 to 3 results in relatively small changes in mean@8. Similarly, different values of lead to similar performance, but this may also be expected from the parameterization equivalence in Section 3.3. If the performance barely changes across different target positions, it is not clear how important the precise state-aware target selection actually is. I think the authors should better explain what these robustness results tell us about the effectiveness of the proposed controller, rather than just showing that several hyperparameter choices work similarly.
References
[1] Jin et al. "Entropy-Aware On-Policy Distillation of Language Models." ICML 2026.
[2] Ko et al. "DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs." ICML 2025.
[3] Yang et al. "Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation." 2026.
Questions for authors
Please check the weaknesses for the details.
- How is COPD fundamentally different from G-OPD/ExOPD?
- Why is the KL projection necessary? Would simpler alternatives work?
- Are the marginal improvements of SACOPD statistically significant?
- How does SACOPD compare with diversity-preserving methods like EOPD?
- What happens with more training compute? How important is the lookahead hyperparameter?
8 comments
Response (Part 2/4)
Q2 (W2 & W3): Why is reverse-KL projection necessary compared to simpler alternatives (e.g., raw teacher-student KL), and how will "state-aware" be clarified in the introduction?
R2:
Difference from DistiLLM-2 [2] (Rollout-level vs. Model/Logit-level contrast). We agree that contrastive supervision has proven highly valuable in distillation, with DistiLLM-2 [2] being a prominent milestone that effectively pairs teacher and student trajectories. Specifically, DistiLLM-2 performs a sample/rollout-level contrast under a single teacher : it applies forward Skew-KL on teacher-generated rollouts () and reverse Skew-KL on student-generated rollouts (). Because both terms match the student to the same teacher , its optimum remains bounded by and requires generating separate teacher trajectories. In contrast, COPD/SACOPD operates purely on on-policy student prefixes and performs a model/token-logit-level contrast between two teachers ( vs. ) on the exact same prefix . Contrasting next-token logit distributions via constructs an extrapolated virtual teacher () beyond (lifting math mean@32 from 's 49.04% to 59.48%) and defines the 1D exponential-family axis required for student localization .
Why raw or disagreement cannot replace . Evaluating our Bregman decomposition (Appendix A.4, Eq. 22) at the positive teacher () gives:
Using raw or unsigned disagreement as a controller has two limitations:
- Sign ambiguity: Because on both sides of (), a lagging student (, e.g., Step 10 in Table D9) and an extrapolated student that has already surpassed (, Step 20) can have the exact same . Worse, as the student progresses further beyond from Step 20 to Step 150 (), monotonically increases. An unsigned KL controller therefore confuses progress beyond with falling behind.
- Off-axis confounding: A rollout with large off-axis residual yields a large even when its on-axis position is . In contrast, is signed, orthogonal to , and measured in the same natural-parameter units as .
Ablations across domains (Global Replay and Within-Batch Shuffle). We compared Static COPD, Global Replay, Within-Batch Shuffle (permuting the six online projected targets within each batch), and SACOPD on both code and math:
| Method | Control Granularity | Code Overall (mean@4 / pass@4) | Math Overall (mean@32 / pass@32) | Math Solved |
|---|---|---|---|---|
| Static COPD | Fixed constant target | 67.98 / 76.93 | 58.91 / 79.17 | 95 / 120 |
| Global Replay | Offline step-mean schedule | 69.57 / 77.02 | 58.88 / 80.00 | 96 / 120 |
| Within-Batch Shuffle | Online batch projection (permuted) | 68.83 / 77.22 | 59.51 / 80.83 | 97 / 120 |
| SACOPD | Online per-rollout projection | 69.77 / 78.30 | 59.48 / 80.83 | 97 / 120 |
Online projection outperforms Static COPD and Global Replay in both domains, and on code (where prefix heterogeneity is higher), per-rollout matching beats Global Replay by +1.28 pp (pass@4) and Within-Batch Shuffle by +0.94 pp (mean@4) / +1.08 pp (pass@4).
Clarifying "state-aware" in the Introduction (W3). We will revise Paragraph 4 of Section 1 to define "state-aware" explicitly at first mention: "Specifically, at each update, SACOPD localizes the student policy on each rollout prefix along the 1D teacher-defined exponential-family axis via convex reverse-KL projection (), and places that rollout's distillation target a fixed coordinate distance ahead ()."
Response (Part 3/4)
Q3 (W4): Are the improvements of SACOPD statistically significant?
R3: We will move the 3-seed mean std from Appendix Table B6 into Table 1 alongside 10,000-resample paired problem-level bootstrap 95% confidence intervals:
- Code (K=4, 3 seeds): SACOPD achieves 70.11 0.32% (mean@4) / 77.97 0.29% (pass@4) vs. COPD's 68.36 0.70% / 76.69 0.22%. The paired bootstrap 95% CI for SACOPD Static COPD on code mean@4 is +1.79 pp (), and for SACOPD W2S-OPD () on code pass@4 is +2.48 pp ().
- Math (K=32, 3 seeds): SACOPD achieves 59.57 0.21% (mean@32) / 80.83 0.83% (pass@32) vs. COPD's 59.14 0.28% / 80.00 0.83% (leading in all 3 seeds). Over W2S-OPD () and Matched ExOPD (), SACOPD improves pass@32 by +3.33 pp (95% CI , solving 97 vs. 93 problems).
Q4 (W5): How does SACOPD compare with diversity-preserving methods like EOPD [1]?
R4: Thank you for pointing us to EOPD (Jin et al., ICML 2026 [1]), which introduces an elegant and effective entropy-aware weighting mechanism to preserve generation diversity in OPD. Following your suggestion, we evaluated EOPD [1] under the exact matched 50-update (300-rollout) budget on both math and code:
| Method | Supervision Mechanism | Math mean@32 / pass@32 (%) | Math Solved | Code Overall mean@4 / pass@4 (%) |
|---|---|---|---|---|
| Matched OPD | Single-teacher reverse KL | 46.17 / 73.33 | 88 / 120 | 66.79 / 75.95 |
| EOPD (Jin et al., 2026) | Single-teacher + entropy weighting | 52.99 / 75.83 | 91 / 120 | 68.52 / 77.40 |
| Static COPD | Two-teacher static extrapolation | 58.91 / 79.17 | 95 / 120 | 67.98 / 76.93 |
| SACOPD | Two-teacher state-aware extrapolation | 59.48 / 80.83 | 97 / 120 | 69.77 / 78.30 |
EOPD improves over standard OPD (+6.82 pp mean@32 and +2.50 pp pass@32 on math; +1.73 pp mean@4 and +1.45 pp pass@4 on code), confirming the benefit of entropy preservation in single-teacher imitation. However, because EOPD uses a single teacher without an extrapolation direction , SACOPD outperforms EOPD by +6.49 pp (mean@32) and +5.00 pp (pass@32, 97 vs. 91 solved) on math, and by +1.25 pp (mean@4) and +0.90 pp (pass@4) on code. Notably, EOPD's entropy-aware token weighting is orthogonal to our state-aware two-teacher target construction and could naturally be combined with SACOPD; we will add these results and highlight this synergy in the revised manuscript.
Response (Part 1/4)
Thank you for your thoughtful and constructive feedback. Below, we respond to your questions and concerns.
Q1 (W1): How is COPD fundamentally different from G-OPD/ExOPD, and how does strength-matched ExOPD perform?
R1: Thank you for this insightful observation. G-OPD/ExOPD [3] is a pioneering work on extrapolative OPD that inspired our study (and whose open-sourced RL checkpoints and evaluation protocols we gratefully build upon). As noted in Sections 2.2 and 3.2 (Proposition 1), at a fixed prefix h, setting πref=π− and λG-OPD=1/(1−λ)=α makes the pointwise minimizer of G-OPD coincide with our closed-form local virtual teacher πlocal⋆(v∣h)∝π+(v∣h)απ−(v∣h)1−α. However, our framework differs in three ways:
Strength- and optimization-matched ExOPD (α=5.0). We ran ExOPD at λExOPD=5.0 (matching COPD's α=5.0,ρ=4.0) under the exact same 50-update (300-rollout) budget, learning rate, and rollout settings:
Crucially, raising ExOPD to α=5.0 lifts its mean@32 by +6.25 pp to 59.09%, demonstrating the strong effectiveness of G-OPD's reward-extrapolation principle and empirically confirming Proposition 1. At the same time, optimizing α=5.0 sampled-token PPO rewards from step 0 narrows answer diversity, leaving Matched ExOPD at 77.50% pass@32 (93/120 solved), whereas full-vocabulary state-aware SACOPD reaches 59.48% mean@32 and further improves coverage to 80.83% pass@32 (97/120 solved, +3.33 pp). We will highlight this strength-matched comparison and expand our discussion of G-OPD/ExOPD [3] in the revised paper.