acceptodds
Under review as a conference paper at ICLR 2027

State-Aware Contrastive On-Policy Distillation

12 views

Abstract

On-policy distillation trains a student on its own generations, while a single-teacher objective specifies only the distribution to imitate and no explicit extrapolation direction. We propose Contrastive On-Policy Distillation (COPD), which combines positive-teacher attraction and negative-teacher repulsion in a conditional KL objective on student-generated prefixes. We derive its closed-form virtual teacher and characterize the induced targets as an exponential-family path. The teacher pair supplies both an extrapolation direction and a coordinate system for targets and students. We localize the student along this path with a one-dimensional convex reverse-KL projection, whose residual measures off-axis mismatch. We extend COPD with State-Aware Contrastive On-Policy Distillation (SACOPD), which places each rollout's target a fixed coordinate increment ahead of the student's projection. With a Qwen3-4B student and 300 training rollouts, static COPD raises mathematics sample accuracy from matched OPD's to , compared with the positive teacher's . SACOPD reaches and improves both accuracy and coverage over COPD on all three code benchmarks. Together, these results connect effective teacher amplification with interpretable student localization and state-aware target selection.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, knowledge distillation, large language models, contrastive distillation, reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_RZXfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes COPD and SACOPD. COPD combines both positive and negative teachers in a two-sided conditional KL objective to extrapolate beyond the positive teacher. Based on the teacher-defined axis, the authors further propose SACOPD, which projects the current student onto this axis and adaptively places the target a fixed distance ahead of the student. Experiments on math and code benchmarks outperforms standard OPD, with SACOPD further improving the performance over COPD.

Critical strengths
  • I think the idea of SACOPD is interesting. Instead of using a fixed target throughout training, it tries to adapt the target based on the current student state.
  • The paper provides reasonable theoretical analysis of the virtual teacher and student projection, which helps explain the proposed method.
  • The method does show improvements over standard OPD on both math and code benchmarks.
Critical weaknesses
  • The authors mentioned G-OPD [3] and ExOPD in the related work. But to me, it is still not clear how COPD is fundamentally different from them. Specifically, by setting the reference model in G-OPD to the negative teacher and using α=1/(1−ω)\alpha=1/(1-\omega), isn't the proposed COPD objective in Eq. (2) exactly equivalent to (1−ω)LGOPD(1-\omega)\mathcal{L}_{\mathrm{GOPD}} for a fixed token context? The negative teacher here also seems to play the same role as the reference model in G-OPD. I understand that the formulation and training implementation can be different, but the objective itself looks mathematically equivalent up to a scaling factor. In particular, COPD uses ω=0.8\omega=0.8 for math, which corresponds to α=5\alpha=5, while the ExOPD baseline uses α=1.25\alpha=1.25. Then, how much of the improvement comes simply from using a larger extrapolation coefficient? I think a strength-matched ExOPD comparison under the same optimization settings would be important here.
  • Also, in general, using contrastive signals in distillation itself does not look that novel to me. Contrastive supervision has been quite common even before these recent RL-style OPD formulations like DistiLLM-2 [2], although the exact formulation is different. I understand that the novelty may instead come from projecting the student onto the teacher axis and using this to adjust the target. But then, why is this projection necessary? The authors provide some mathematical properties of the projection, but it is still not clear why this is a better way to adjust the extrapolation strength compared to simpler alternatives (something like directly using the teacher-student KL or disagreement). The paper does compare with a global replay schedule, but the improvement is rather small. I think more justification or ablations are needed to show why this particular projection-based controller is needed and important.
  • There is not enough explanation of what SACOPD actually does in the introduction, or what exactly is meant by "state-aware." Although some explanation is given in the contribution bullet points and later sections, I think the main idea (placing the target a fixed distance ahead of the student's projected position) should be introduced more clearly at the beginning.
  • The main result tables do not provide error bars or standard deviations in the main tables. For math benchmarks like AIME, they have only 30 problems per benchmark and can be quite noisy. It is hard to tell whether some of the relatively small improvements are statistically meaningful.
  • The improvement from COPD to SACOPD is very marginal for math mean@32 (58.91 to 59.48). Since SACOPD seems to be the main new contribution beyond the existing extrapolation methods, I am not sure if this is a sufficiently strong gain. If the main benefit is also improving pass@K or maintaining answer coverage, then I think the paper should compare with methods specifically designed to preserve diversity in OPD, such as EOPD [1]. Otherwise, it is hard to understand whether the improvement in pass@K comes from the proposed state-aware target selection or whether it could also be achieved by better preserving generation diversity.
  • Also, I think the matched training budget is quite small, with only 50 updates and a global batch size of 6 (300 rollouts). It seems like the appendix includes longer training exposure results for COPD and OPD, but I would like to see what happens to SACOPD with more compute. Does the improvement remain when training for more steps or using a larger training budget? Since the proposed controller is supposed to adjust the target as the student progresses, I think showing its benefit under longer training would be particularly important.
  • In Section 5.2, the authors seem to argue that the method is robust to different hyperparameter choices. But I am not sure if this is necessarily strong evidence for the proposed method. For example, changing the lookahead δ\delta from 1 to 3 results in relatively small changes in mean@8. Similarly, different values of ω\omega lead to similar performance, but this may also be expected from the parameterization equivalence in Section 3.3. If the performance barely changes across different target positions, it is not clear how important the precise state-aware target selection actually is. I think the authors should better explain what these robustness results tell us about the effectiveness of the proposed controller, rather than just showing that several hyperparameter choices work similarly.

References

[1] Jin et al. "Entropy-Aware On-Policy Distillation of Language Models." ICML 2026.

[2] Ko et al. "DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs." ICML 2025.

[3] Yang et al. "Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation." 2026.

Questions for authors

Please check the weaknesses for the details.

  1. How is COPD fundamentally different from G-OPD/ExOPD?
  2. Why is the KL projection necessary? Would simpler alternatives work?
  3. Are the marginal improvements of SACOPD statistically significant?
  4. How does SACOPD compare with diversity-preserving methods like EOPD?
  5. What happens with more training compute? How important is the lookahead hyperparameter?
8 comments
AuthorsOpenReview ↗

Response (Part 1/4)

Thank you for your thoughtful and constructive feedback. Below, we respond to your questions and concerns.

Q1 (W1): How is COPD fundamentally different from G-OPD/ExOPD, and how does strength-matched ExOPD perform?

R1: Thank you for this insightful observation. G-OPD/ExOPD [3] is a pioneering work on extrapolative OPD that inspired our study (and whose open-sourced RL checkpoints and evaluation protocols we gratefully build upon). As noted in Sections 2.2 and 3.2 (Proposition 1), at a fixed prefix hh, setting πref=π−\pi_{\text{ref}} = \pi^- and λG-OPD=1/(1−λ)=α\lambda_{\text{G-OPD}} = 1/(1-\lambda) = \alpha makes the pointwise minimizer of G-OPD coincide with our closed-form local virtual teacher πlocal⋆(v∣h)∝π+(v∣h)απ−(v∣h)1−α\pi_{\text{local}}^\star(v|h) \propto \pi^+(v|h)^\alpha \pi^-(v|h)^{1-\alpha}. However, our framework differs in three ways:

  1. Full-vocabulary analytic KL vs. sampled-token PPO gradients. ExOPD treats the log-ratio as a scalar reward on the single sampled token and optimizes it via clipped policy gradients. At large extrapolation (α=5.0\alpha=5.0), sampled-token log-ratio rewards have higher variance and reduce coverage. COPD minimizes the exact full-vocabulary conditional KL KL(πθ(⋅∣h)∥πlocal⋆(⋅∣h))\text{KL}(\pi_\theta(\cdot|h)\|\pi_{\text{local}}^\star(\cdot|h)) (Eq. 4), providing dense gradients across all tokens.
  2. General teacher pairs and controlled negatives π−β\pi_{-\beta}. Rather than restricting πref\pi_{\text{ref}} to the student's pre-RL base model, our formulation supports general capability axes—such as our Cross-Specialist (4B Math-RL / 4B Code-RL) pair, where SACOPD reaches 60.21% mean@8 / 76.67% pass@8 (+14.48 pp over OPD)—and parameterizes continuous target movement via the controlled negative π~β−=π−β\tilde\pi_\beta^- = \pi_{-\beta}.
  3. Reverse-KL projection for state-aware control (SACOPD). We use the exponential-family path as a coordinate system, projecting the student via ρ^s\hat{\rho}_s (Proposition 2) to place targets a fixed increment ahead (ρ^s+δ\hat{\rho}_s + \delta).

Strength- and optimization-matched ExOPD (α=5.0\alpha=5.0). We ran ExOPD at λExOPD=5.0\lambda_{\text{ExOPD}} = 5.0 (matching COPD's α=5.0,ρ=4.0\alpha=5.0, \rho=4.0) under the exact same 50-update (300-rollout) budget, learning rate, and rollout settings:

Math MethodExtrapolation & Optimizationmean@32 (%)pass@32 (%)Correct / 3,840Solved / 120
Matched OPDα=1.0\alpha=1.0, Full-vocab KL46.1773.331,77388
ExOPD (Default α=1.25\alpha=1.25)α=1.25\alpha=1.25, Sampled-token PPO52.8476.672,02992
Matched ExOPD (α=5.0\alpha=5.0)α=5.0\alpha=5.0, Sampled-token PPO59.0977.502,26993
Static COPD (α=5.0\alpha=5.0 eq.)α=5.0\alpha=5.0 (ρ=4\rho=4), Full-vocab KL58.9179.172,26295
SACOPDρ^s+2\hat{\rho}_s + 2, Full-vocab KL59.4880.832,28497

Crucially, raising ExOPD to α=5.0\alpha=5.0 lifts its mean@32 by +6.25 pp to 59.09%, demonstrating the strong effectiveness of G-OPD's reward-extrapolation principle and empirically confirming Proposition 1. At the same time, optimizing α=5.0\alpha=5.0 sampled-token PPO rewards from step 0 narrows answer diversity, leaving Matched ExOPD at 77.50% pass@32 (93/120 solved), whereas full-vocabulary state-aware SACOPD reaches 59.48% mean@32 and further improves coverage to 80.83% pass@32 (97/120 solved, +3.33 pp). We will highlight this strength-matched comparison and expand our discussion of G-OPD/ExOPD [3] in the revised paper.

AuthorsOpenReview ↗

Response (Part 2/4)

Q2 (W2 & W3): Why is reverse-KL projection necessary compared to simpler alternatives (e.g., raw teacher-student KL), and how will "state-aware" be clarified in the introduction?

R2:

Difference from DistiLLM-2 [2] (Rollout-level vs. Model/Logit-level contrast). We agree that contrastive supervision has proven highly valuable in distillation, with DistiLLM-2 [2] being a prominent milestone that effectively pairs teacher and student trajectories. Specifically, DistiLLM-2 performs a sample/rollout-level contrast under a single teacher π+\pi^+: it applies forward Skew-KL on teacher-generated rollouts (yt∼π+y_t \sim \pi^+) and reverse Skew-KL on student-generated rollouts (ys∼πθy_s \sim \pi_\theta). Because both terms match the student to the same teacher π+\pi^+, its optimum remains bounded by π+\pi^+ and requires generating separate teacher trajectories. In contrast, COPD/SACOPD operates purely on on-policy student prefixes hh and performs a model/token-logit-level contrast between two teachers (π+\pi^+ vs. π−\pi^-) on the exact same prefix hh. Contrasting next-token logit distributions via Δ(v∣h)=log⁡π+(v∣h)−log⁡π−(v∣h)\Delta(v|h) = \log\pi^+(v|h) - \log\pi^-(v|h) constructs an extrapolated virtual teacher πlocal⋆∝(π+)α(π−)1−α\pi_{\text{local}}^\star \propto (\pi^+)^\alpha (\pi^-)^{1-\alpha} (α>1\alpha > 1) beyond π+\pi^+ (lifting math mean@32 from π+\pi^+'s 49.04% to 59.48%) and defines the 1D exponential-family axis {πρ}\{\pi_\rho\} required for student localization ρ^s\hat\rho_s.

Why raw KL(πθ∥π+)\text{KL}(\pi_\theta\|\pi^+) or disagreement cannot replace ρ^s\hat\rho_s. Evaluating our Bregman decomposition (Appendix A.4, Eq. 22) at the positive teacher π+=π0\pi^+ = \pi_0 (r=0r=0) gives:

KL(πθ∥π+)=Φs(0)=Rs+DFs(0,ρ^s)\text{KL}(\pi_\theta \| \pi^+) = \Phi_s(0) = R_s + D_{F_s}(0, \hat\rho_s)

Using raw KL(πθ∥π+)\text{KL}(\pi_\theta\|\pi^+) or unsigned disagreement as a controller has two limitations:

  1. Sign ambiguity: Because DFs(0,ρ^s)≥0D_{F_s}(0, \hat\rho_s) \ge 0 on both sides of π+\pi^+ (ρ=0\rho=0), a lagging student (ρ^s<0\hat\rho_s < 0, e.g., Step 10 in Table D9) and an extrapolated student that has already surpassed π+\pi^+ (ρ^s>0\hat\rho_s > 0, Step 20) can have the exact same KL(πθ∥π+)\text{KL}(\pi_\theta\|\pi^+). Worse, as the student progresses further beyond π+\pi^+ from Step 20 to Step 150 (ρ^s:0.78→1.76\hat\rho_s: 0.78 \to 1.76), KL(πθ∥π+)\text{KL}(\pi_\theta\|\pi^+) monotonically increases. An unsigned KL controller therefore confuses progress beyond π+\pi^+ with falling behind.
  2. Off-axis confounding: A rollout with large off-axis residual RsR_s yields a large KL(πθ∥π+)\text{KL}(\pi_\theta\|\pi^+) even when its on-axis position is ρ^s=0\hat\rho_s = 0. In contrast, ρ^s\hat\rho_s is signed, orthogonal to RsR_s, and measured in the same natural-parameter units as πρ\pi_\rho.

Ablations across domains (Global Replay and Within-Batch Shuffle). We compared Static COPD, Global Replay, Within-Batch Shuffle (permuting the six online projected targets within each batch), and SACOPD on both code and math:

MethodControl GranularityCode Overall (mean@4 / pass@4)Math Overall (mean@32 / pass@32)Math Solved
Static COPDFixed constant target67.98 / 76.9358.91 / 79.1795 / 120
Global ReplayOffline step-mean schedule69.57 / 77.0258.88 / 80.0096 / 120
Within-Batch ShuffleOnline batch projection (permuted)68.83 / 77.2259.51 / 80.8397 / 120
SACOPDOnline per-rollout projection69.77 / 78.3059.48 / 80.8397 / 120

Online projection outperforms Static COPD and Global Replay in both domains, and on code (where prefix heterogeneity is higher), per-rollout matching beats Global Replay by +1.28 pp (pass@4) and Within-Batch Shuffle by +0.94 pp (mean@4) / +1.08 pp (pass@4).

Clarifying "state-aware" in the Introduction (W3). We will revise Paragraph 4 of Section 1 to define "state-aware" explicitly at first mention: "Specifically, at each update, SACOPD localizes the student policy on each rollout prefix along the 1D teacher-defined exponential-family axis via convex reverse-KL projection (ρ^s\hat\rho_s), and places that rollout's distillation target a fixed coordinate distance ahead (ρstarget=ρ^s+δ\rho_s^{\text{target}} = \hat\rho_s + \delta)."

AuthorsOpenReview ↗

Response (Part 3/4)

Q3 (W4): Are the improvements of SACOPD statistically significant?

R3: We will move the 3-seed mean ±\pm std from Appendix Table B6 into Table 1 alongside 10,000-resample paired problem-level bootstrap 95% confidence intervals:

  • Code (K=4, 3 seeds): SACOPD achieves 70.11 ±\pm 0.32% (mean@4) / 77.97 ±\pm 0.29% (pass@4) vs. COPD's 68.36 ±\pm 0.70% / 76.69 ±\pm 0.22%. The paired bootstrap 95% CI for SACOPD −- Static COPD on code mean@4 is +1.79 pp ([+0.31,+3.33][+0.31, +3.33]), and for SACOPD −- W2S-OPD (α=5\alpha=5) on code pass@4 is +2.48 pp ([+0.70,+4.46][+0.70, +4.46]).
  • Math (K=32, 3 seeds): SACOPD achieves 59.57 ±\pm 0.21% (mean@32) / 80.83 ±\pm 0.83% (pass@32) vs. COPD's 59.14 ±\pm 0.28% / 80.00 ±\pm 0.83% (leading in all 3 seeds). Over W2S-OPD (α=5\alpha=5) and Matched ExOPD (α=5\alpha=5), SACOPD improves pass@32 by +3.33 pp (95% CI [+0.83,+6.67][+0.83, +6.67], solving 97 vs. 93 problems).

Q4 (W5): How does SACOPD compare with diversity-preserving methods like EOPD [1]?

R4: Thank you for pointing us to EOPD (Jin et al., ICML 2026 [1]), which introduces an elegant and effective entropy-aware weighting mechanism to preserve generation diversity in OPD. Following your suggestion, we evaluated EOPD [1] under the exact matched 50-update (300-rollout) budget on both math and code:

MethodSupervision MechanismMath mean@32 / pass@32 (%)Math SolvedCode Overall mean@4 / pass@4 (%)
Matched OPDSingle-teacher reverse KL46.17 / 73.3388 / 12066.79 / 75.95
EOPD (Jin et al., 2026)Single-teacher + entropy weighting52.99 / 75.8391 / 12068.52 / 77.40
Static COPDTwo-teacher static extrapolation58.91 / 79.1795 / 12067.98 / 76.93
SACOPDTwo-teacher state-aware extrapolation59.48 / 80.8397 / 12069.77 / 78.30

EOPD improves over standard OPD (+6.82 pp mean@32 and +2.50 pp pass@32 on math; +1.73 pp mean@4 and +1.45 pp pass@4 on code), confirming the benefit of entropy preservation in single-teacher imitation. However, because EOPD uses a single teacher π+\pi^+ without an extrapolation direction Δ(v∣h)\Delta(v|h), SACOPD outperforms EOPD by +6.49 pp (mean@32) and +5.00 pp (pass@32, 97 vs. 91 solved) on math, and by +1.25 pp (mean@4) and +0.90 pp (pass@4) on code. Notably, EOPD's entropy-aware token weighting is orthogonal to our state-aware two-teacher target construction and could naturally be combined with SACOPD; we will add these results and highlight this synergy in the revised manuscript.

Reviewer_85UVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper formulates contrastive on-policy distillation using positive and negative teachers. It derives a teacher-defined exponential-family path, projects the student onto this path using reverse KL, and introduces SACOPD to place each rollout's target a fixed coordinate increment ahead of the student.

Critical strengths

Provides a coherent mathematical interpretation connecting the contrastive objective, virtual teachers, and student localization, with a residual measuring off-axis mismatch.

Includes strong extrapolative baselines and target-selection controls, while openly discussing teacher-pair sensitivity and the limits of interpreting coordinate progress as capability improvement.

Critical weaknesses
  1. The empirical advantage over a strong baseline is modest. In Table 1, SACOPD improves average accuracy over W2S-OPD (α=5) by 0.86 percentage points on mathematics and 1.02 on code, but remains lower on AIME24 and MBPP+. COPD’s code average is also below W2S-OPD (67.98 vs. 68.75). The main comparison does not quantify uncertainty for these differences.

  2. The benefit of per-rollout state awareness is only partially isolated. Table 4 reports mathematical mean@32 of 58.91 for COPD, 58.88 for global replay, and 59.48 for SACOPD. The code gain over COPD is larger, but the corresponding global-replay control is not shown. Evidence that student-specific localization improves on a shared target schedule therefore remains limited across domains.

Questions for authors

On code, does SACOPD outperform global replay and a tuned fixed target under matched training and evaluation budgets?

5 comments

Thank you for your thoughtful and constructive feedback. Below, we respond to your concerns and questions.

Q1 & W2: On code, does SACOPD outperform global replay and a tuned fixed target under matched training and evaluation budgets?

R1: Yes. Under the exact same training budget (50 updates, 300 rollouts) and evaluation protocol, we evaluated Global Replay (replaying the donor SACOPD run's step-mean target schedule ρˉt+1\bar{\rho}_t + 1), Within-Batch Shuffle (computing each batch's six state-aware targets online, then randomly permuting their assignment across the six rollouts), and a 6-point static strength sweep (β∈{0.25,0.50,0.75,0.90,1.00,1.10}\beta \in \{0.25, 0.50, 0.75, 0.90, 1.00, 1.10\} at λ=0.20\lambda=0.20):

SACOPD vs. Global Replay and Within-Batch Shuffle on code.

Code MethodTarget-Selection GranularityHumanEval+MBPP+LiveCodeBenchOverall (mean@4 / pass@4)
Static COPD (ρ=0.25\rho=0.25)Fixed constant target87.20 / 95.1273.61 / 81.4843.13 / 54.2067.98 / 76.93
Global ReplayShared step-mean schedule89.33 / 93.2976.06 / 82.8043.32 / 54.9669.57 / 77.02
Within-Batch ShuffleOnline batch targets (permuted)89.02 / 93.9076.06 / 82.8041.41 / 54.9668.83 / 77.22
SACOPDPer-rollout state-aware89.79 / 96.3476.19 / 83.6043.32 / 54.9669.77 / 78.30

While Global Replay improves mean@4 over static COPD, its pass@4 (77.02%) barely moves above the static baseline (76.93%). SACOPD achieves the highest mean@4 (69.77%) and pass@4 (78.30%), outperforming Global Replay by +1.28 pp in pass@4 and Within-Batch Shuffle by +0.94 pp in mean@4 and +1.08 pp in pass@4.

SACOPD vs. tuned fixed targets on code.

Code Method / Fixed StrengthHumanEval+MBPP+LiveCodeBenchOverall (mean@4 / pass@4)
Static COPD (β=0.25\beta=0.25)86.13 / 93.9072.42 / 82.8041.03 / 52.6766.53 / 76.46
Static COPD (β=0.50\beta=0.50)87.96 / 95.7370.90 / 80.9542.18 / 53.4467.01 / 76.71
Static COPD (β=0.75\beta=0.75)89.48 / 96.3472.22 / 82.5440.84 / 51.1567.51 / 76.68
Static COPD (β=0.90\beta=0.90)88.57 / 95.1272.82 / 83.3340.84 / 50.3867.41 / 76.28
Static COPD (β=1.00\beta=1.00, Table 1)87.20 / 95.1273.61 / 81.4843.13 / 54.2067.98 / 76.93
Static COPD (β=1.10\beta=1.10)89.33 / 94.5173.61 / 82.8042.18 / 51.1568.37 / 76.15
SACOPD (δ=1.0\delta=1.0)89.79 / 96.3476.19 / 83.6043.32 / 54.9669.77 / 78.30

Tuning fixed β\beta up to 1.10 reaches at most 68.37% mean@4 while reducing pass@4 to 76.15%. SACOPD outperforms every fixed-target configuration across all three benchmarks (+1.40 pp mean@4 and +2.15 pp pass@4 over the β=1.10\beta=1.10 run).

W1: Comparison with W2S-OPD (α=5\alpha=5), sub-benchmark trade-offs, and uncertainty quantification

R2: Comparison on mean@K and sub-benchmark behavior. Under W2S-OPD's paper-recommended coefficients (α=1.0\alpha=1.0 math, 0.750.75 code), it achieves 47.73% / 75.00% on math and 62.47% / 72.04% on code (Table 1). We constructed the stronger W2S-OPD (α=5\alpha=5) variant ourselves using our Proposition 1 (α=1/(1−λ)=5\alpha = 1/(1-\lambda) = 5) to test whether our virtual-teacher target transfers across implementations. Because W2S-OPD (α=5\alpha=5) shares our extrapolated coordinate (ρ=4.0\rho=4.0), it is expected to reach a competitive mean@K, and on individual subsets such as AIME24 and MBPP+, its top-32 token truncation sharpens the mode enough to slightly edge out or match SACOPD in mean@K.

However, on harder or broader distributions (such as HMMT-F and LiveCodeBench) and in aggregate across benchmarks, truncating distillation to top-32 tokens under strong extrapolation (α=5\alpha=5) reduces tail diversity and inflates generation length. Consequently, full-vocabulary SACOPD achieves higher aggregate mean@K (+0.86 pp on math, +1.02 pp on code) alongside statistically significant gains in coverage (pass@K) (+3.33 pp on math pass@32, +2.48 pp on code pass@4), while generating 28.4% shorter responses on code (7,645 vs. 10,675 macro-average tokens).

(Note on Table 1 code Static COPD: Static COPD had a lower code mean@4 [67.98%] than W2S-OPD (α=5\alpha=5) [68.75%] because Static COPD used a mild default λ=0.20,β=1.0\lambda=0.20, \beta=1.0 [α=1.25,ρ=0.25\alpha=1.25, \rho=0.25] whereas W2S-OPD (α=5\alpha=5) used 16×16\times stronger extrapolation [ρ=4.0\rho=4.0]; even so, full-vocabulary Static COPD achieved higher code pass@4 [76.93% vs. 75.82%].)

Uncertainty quantification. Below are 10,000-resample paired problem-level bootstrap 95% confidence intervals (stratified and equally weighted across the three benchmarks on code) for Table 1 alongside our 3-seed summary (Table B6):

Paired ComparisonMetricTable 1 Δ\Delta (pp)Paired Bootstrap 95% CI (pp)3-Seed Mean ±\pm Std (%)
SACOPD −- W2S-OPD (α=5\alpha=5)Math mean@32+0.86[−0.26,+2.01][-0.26, +2.01]59.57 ±\pm 0.21 vs. 58.62
SACOPD −- W2S-OPD (α=5\alpha=5)Code mean@4+1.02[−0.38,+2.43][-0.38, +2.43]70.11 ±\pm 0.32 vs. 68.75
SACOPD −- W2S-OPD (α=5\alpha=5)Math pass@32+3.33[+0.83,+6.67][+0.83, +6.67]80.83 ±\pm 0.83 vs. 77.50
SACOPD −- W2S-OPD (α=5\alpha=5)Code pass@4+2.48[+0.70,+4.46][+0.70, +4.46]77.97 ±\pm 0.29 vs. 75.82
SACOPD −- Static COPD (β=1.0\beta=1.0)Code mean@4+1.79[+0.31,+3.33][+0.31, +3.33]70.11 ±\pm 0.32 vs. 68.36 ±\pm 0.70
SACOPD −- Static COPD (β=1.0\beta=1.0)Math mean@32+0.57[−0.65,+1.80][-0.65, +1.80] (3-seed: [−0.25,+1.09][-0.25, +1.09])59.57 ±\pm 0.21 vs. 59.14 ±\pm 0.28

For SACOPD − W2S-OPD (α=5) on mean@K, both aggregate point estimates are positive (+0.86 pp and +1.02 pp), and the 3-seed means of SACOPD (59.57 ±\pm 0.21% on math, 70.11 ±\pm 0.32% on code) lie more than 4 standard deviations above the W2S-OPD (α=5\alpha=5) evaluation scores, although the single-seed problem-level 95% CIs slightly cross zero due to finite problem counts (e.g., 120 math problems) and the shared α=5\alpha=5 extrapolation scale. Meanwhile, the paired 95% CIs for pass@32 on math ([+0.83,+6.67][+0.83, +6.67]), pass@4 on code ([+0.70,+4.46][+0.70, +4.46]), and mean@4 over Static COPD on code ([+0.31,+3.33][+0.31, +3.33]) strictly exclude zero. We will include the 3-seed mean ±\pm std and all paired 95% CIs directly in Table 1.

AuthorsOpenReview ↗

Additional Code Results (1.7B Scale & Longer Windows) and Follow-Up

With the discussion period closing soon (Nov 18 AoE), we wanted to gently check whether our response (reporting Global Replay, Within-Batch Shuffle, the tuned static β\beta sweep on code, and paired bootstrap 95% CIs) has addressed your questions, and share two additional code generation studies (K=4K=4) that further highlight the benefit and robustness of state-aware target selection:

1. Larger SACOPD advantage on a smaller student (Qwen3-1.7B on Code, K=4K=4). Distilling from the 4B teachers into a smaller Qwen3-1.7B student using the exact same default code hyperparameters (50 updates, lr=5×10−6\text{lr}=5\times 10^{-6}, λ=0.20\lambda=0.20, fixed β=1.0\beta=1.0 for COPD, and δ=1.0\delta=1.0 with dynamic per-rollout βs\beta_s for SACOPD) yields an even larger advantage for state-aware target placement—SACOPD outperforms static COPD by +5.07 percentage points in mean@4 and +4.18 percentage points in pass@4 across all three suites:

1.7B Code MethodHumanEval+MBPP+LiveCodeBenchOverall (mean@4 / pass@4)
1.7B Matched OPD71.95 / 87.8058.40 / 73.2822.14 / 30.5350.83 / 63.87
1.7B Static COPD73.02 / 85.9860.65 / 74.0727.86 / 35.8853.84 / 65.31
1.7B SACOPD78.66 / 88.4166.20 / 78.8431.87 / 41.2258.91 / 69.49

2. Robustness across training and projection windows (256 to 4,096 tokens on 4B Code, K=4K=4). Expanding the training and projection window from 256 to 1024, 2048, and 4096 tokens preserves SACOPD's Overall mean@4 lead over static COPD at every window length (+1.79 pp at 256, +1.84 pp at 1024, +0.62 pp at 2048, and +1.58 pp at 4096):

Code WindowStatic COPD (mean@4 / pass@4)SACOPD (mean@4 / pass@4)SACOPD Δ\Deltamean@4
256 (Table 1)67.98 / 76.9369.77 / 78.30+1.79 pp
102468.89 / 77.4070.73 / 77.24+1.84 pp
204868.96 / 77.0169.58 / 75.87+0.62 pp
409669.59 / 75.9571.17 / 76.79+1.58 pp

Please let us know if you have any remaining questions—we would greatly appreciate your feedback!

Reviewer_qZYXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes State-Aware Contrastive On-Policy Distillation (SACOPD). It first introduces a conditional distillation objective combining attraction to a positive teacher with repulsion from a negative teacher on student-generated prefixes. The corresponding virtual teachers lie on a one-dimensional exponential-family path defined by the teachers’ log-probability ratio. The student is localized on this path through a reverse-KL projection, and SACOPD selects a rollout-specific target a fixed coordinate increment ahead of that projection. Experiments on mathematical reasoning and code generation show substantial gains from static contrastive distillation over matched OPD, together with smaller mathematical gains and consistent aggregate code improvements from state-aware target selection across three training seeds.

Critical strengths
  • The virtual-teacher identity establishes an explicit relationship between the two-KL loss and distillation toward an extrapolated target. The projection’s existence and uniqueness conditions, together with the residual decomposition, provide a coherent mathematical foundation under the stated assumptions. Appendix A.9 also carefully distinguishes the implemented fixed-prefix updates from optimization of a globally normalized sequence-level objective.
  • The projected coordinate, off-axis residual, and local curvature provide complementary descriptions of the student relative to the teacher-defined path. Evaluating checkpoints on a common replay set makes these measurements easier to interpret by holding observed prefixes fixed. The analysis illustrates how distributional movement can continue during a task-accuracy plateau, without treating coordinate advancement as a guarantee of improved capability.
  • Under the matched training configuration, COPD improves mathematical mean@32 from 46.17% to 58.91%, exceeding the positive teacher’s 49.04%. The additional training seeds preserve this large improvement over OPD. The smaller-student and cross-specialist experiments also provide evidence that static extrapolation is useful beyond the main 4B pre-/post-RL configuration.
  • Table B6 reports higher aggregate code mean@4 and pass@4 for SACOPD than COPD in all three training seeds. The three-seed averages increase from 68.36% to 70.11% for mean@4 and from 76.69% to 77.97% for pass@4. These results provide meaningful evidence that the observed code improvement is reproducible under the evaluated configuration.
  • The paper reports unsuccessful dense-size teacher-pair configurations, acknowledges that the controller cannot determine whether a teacher axis isolates the intended capability, and discusses non-monotonic relationships between coordinates and accuracy. It also documents the target correspondence and implementation differences relative to W2S-OPD, includes a global-replay comparison on mathematics, and provides detailed evaluation protocols.
Critical weaknesses
  • Static COPD uses a fixed target coordinate of 0.25, whereas SACOPD uses the evolving target. The comparison therefore changes both target strength over training and its dependence on individual rollouts. No static strength sweep or global-replay comparison is reported for code, where SACOPD shows its clearest benefit. Static W2S-OPD with α=5\alpha=5 already improves over the reported static COPD configuration, although their implementations differ. A static sweep with a comparable validation budget, followed by a batch-mean or replayed-target comparison, would help determine whether state conditioning provides an advantage beyond selecting a better strength schedule.
  • Across three seeds, SACOPD improves mathematical mean@32 from 59.14% to 59.57%. However, under the K=8 protocol, static COPD with β=1.10\beta=1.10 achieves 60.00% mean@8 and 75.00% pass@8, compared with 59.17% and 73.33% for the main SACOPD configuration. The stronger static configuration is not evaluated at K=32. These observations are compatible with sampling variation, but leave the incremental benefit over a well-selected static target unresolved. Independent hyperparameter selection and paired, problem-level confidence intervals would make this comparison more convincing.
  • In the shared-anchor setting, strength-matched W2S-OPD has the same untruncated target as static COPD and achieves similar mathematical accuracy: 58.62% versus 58.91%. Consequently, the large static gain alone does not establish the contribution of the projection-based controller. Comparisons with ExOPD and IW-OPD also retain differences in learning rate and rollout temperature, while ExOPD uses a different extrapolation strength in mathematics. Furthermore, the fixed-lookahead λ\lambda sweep largely tests a reparameterization that preserves the target at a given student state, up to gradient scaling. These experiments provide limited additional evidence for the distinctive value of state-aware target selection.
  • For fixed teachers and prefixes, the projection is determined by a scalar contrast moment averaged across prefixes, with vocabulary expectations weighted by the student distribution. Different patterns of alignment across positions can therefore yield the same coordinate. Although the training loss still compares full-vocabulary distributions, the target-selection rule does not use the residual or curvature, which characterize information absent from the coordinate alone. The paper acknowledges that fixed coordinate increments do not imply fixed KL discrepancies, but does not establish when coordinate-only control is reliable. Shuffling targets across rollouts, or analyzing performance by residual and within-rollout heterogeneity, would directly test this issue.
  • SACOPD is primarily evaluated with Qwen3-4B, 50 updates, and a 256-token training and projection window. The smaller-student and alternative-teacher-pair experiments evaluate static COPD rather than the adaptive method. Short-prefix supervision may generalize to later generation, but the qualitative examples do not establish how the projection contributes to long-form reasoning, and longer-window behavior remains untested. A bounded extension to another student scale or teacher pair, together with a longer-window comparison, would strengthen the robustness evidence. Reporting projection overhead, training time, and generation lengths for the main methods would also clarify whether the additional gains justify their computational costs.
Questions for authors

Please refer to the Weaknesses. Happy to adjust the assessment if my concerns are fully solved.

5 comments
AuthorsOpenReview ↗

Response (Part 1/4)

Thank you for your thoughtful and constructive feedback. Below, we respond to your concerns and questions.

W1: Static strength sweep, target-selection controls, and comparison with W2S-OPD (α=5\alpha=5) on code

R1: We provide a static β\beta sweep and target-selection controls on code under the matched 50-update (300-rollout) budget.

Comparison with W2S-OPD (α=5\alpha=5) on code. In Table 1, static COPD on code used a default setting of λ=0.20,β=1.0\lambda=0.20, \beta=1.0 (target coordinate ρ=0.25\rho=0.25, equivalent to α=1.25\alpha=1.25), whereas our constructed W2S-OPD (α=5\alpha=5) baseline used 16×16\times stronger extrapolation (ρ=4.0,α=5.0\rho=4.0, \alpha=5.0). Under the original paper's recommended code setting (α=0.75\alpha=0.75), W2S-OPD achieves 62.47% mean@4 and 72.04% pass@4. Moreover, because W2S-OPD truncates distillation to top-32 tokens, setting α=5\alpha=5 reduces tail diversity and lowers pass@4 to 75.82% (50.38% on LiveCodeBench), below both static COPD (76.93%) and SACOPD (69.77% mean@4 / 78.30% pass@4; 3-seed average 70.11 ±\pm 0.32% / 77.97 ±\pm 0.29%).

Static strength sweep on code. Holding λ=0.20\lambda=0.20 fixed, we swept static β∈{0.25,0.50,0.75,0.90,1.00,1.10}\beta \in \{0.25, 0.50, 0.75, 0.90, 1.00, 1.10\} (ρ=0.25β\rho = 0.25\beta):

Method / Static StrengthTarget ρ\rhoHumanEval+MBPP+LiveCodeBenchOverall (mean@4 / pass@4)
Static COPD (β=0.25\beta=0.25)0.062586.13 / 93.9072.42 / 82.8041.03 / 52.6766.53 / 76.46
Static COPD (β=0.50\beta=0.50)0.125087.96 / 95.7370.90 / 80.9542.18 / 53.4467.01 / 76.71
Static COPD (β=0.75\beta=0.75)0.187589.48 / 96.3472.22 / 82.5440.84 / 51.1567.51 / 76.68
Static COPD (β=0.90\beta=0.90)0.225088.57 / 95.1272.82 / 83.3340.84 / 50.3867.41 / 76.28
Static COPD (β=1.00\beta=1.00, Table 1)0.250087.20 / 95.1273.61 / 81.4843.13 / 54.2067.98 / 76.93
Static COPD (β=1.10\beta=1.10)0.275089.33 / 94.5173.61 / 82.8042.18 / 51.1568.37 / 76.15
SACOPD (δ=1.0\delta=1.0)ρ^s+1\hat{\rho}_s + 189.79 / 96.3476.19 / 83.6043.32 / 54.9669.77 / 78.30

Increasing fixed β\beta to 1.10 raises mean@4 to 68.37% at the cost of lowering pass@4 to 76.15%. SACOPD outperforms the best static configuration by +1.40 pp in mean@4 and +2.15 pp in pass@4.

Global Replay and Within-Batch Shuffle on code. To isolate per-rollout state conditioning from a global schedule or batch-level target distribution, we evaluated Global Replay (replaying the donor SACOPD run's step-mean target ρˉt+1\bar{\rho}_t + 1) and Within-Batch Shuffle (computing the six rollout targets online at each step, but randomly permuting their assignment across the six rollouts in the batch):

Target-Selection RuleControl GranularityHumanEval+MBPP+LiveCodeBenchOverall (mean@4 / pass@4)
Static COPD (ρ=0.25\rho=0.25)Fixed constant87.20 / 95.1273.61 / 81.4843.13 / 54.2067.98 / 76.93
Global ReplayStep-level mean schedule89.33 / 93.2976.06 / 82.8043.32 / 54.9669.57 / 77.02
Within-Batch ShuffleOnline batch targets (permuted)89.02 / 93.9076.06 / 82.8041.41 / 54.9668.83 / 77.22
SACOPDOnline per-rollout matched89.79 / 96.3476.19 / 83.6043.32 / 54.9669.77 / 78.30

While Global Replay recovers most of the mean@4 gain, its pass@4 (77.02%) barely changes relative to static COPD (76.93%). SACOPD outperforms Global Replay by +1.28 pp in pass@4 and improves over Within-Batch Shuffle by +0.94 pp in mean@4 and +1.08 pp in pass@4.

AuthorsOpenReview ↗

Response (Part 2/4)

W2: Evaluation of static COPD (β=1.10\beta=1.10) at K=32K=32, hyperparameter selection, and paired confidence intervals

R2: We evaluated the checkpoint from the K=8K=8 sweep peak (static COPD with λ=0.80,β=1.10,ρ=4.4\lambda=0.80, \beta=1.10, \rho=4.4) under the main-table K=32K=32 protocol (120×32=3,840120 \times 32 = 3,840 samples) and computed 10,000-resample paired problem-level bootstrap 95% confidence intervals across the 120 problems:

Math Comparison (K=32K=32, 120 problems)mean@32 (%)pass@32 (%)Correct / 3,840Solved / 120SACOPD Δ\Deltamean@32 (Paired 95% CI, pp)
Static COPD (β=1.00\beta=1.00, Seed 1)58.9179.172,26295+0.57 ([−0.65,+1.80][-0.65, +1.80])
Static COPD (β=1.10\beta=1.10, K=8K=8 peak)58.6281.672,25198+0.86 ([−0.08,+1.82][-0.08, +1.82])
SACOPD (δ=2.0\delta=2.0, Seed 1)59.4880.832,28497—
Static COPD (β=1.00\beta=1.00, 3-Seed Mean)59.14 ±\pm 0.2880.00 ±\pm 0.83——+0.43 ([−0.25,+1.09][-0.25, +1.09])
SACOPD (δ=2.0\delta=2.0, 3-Seed Mean)59.57 ±\pm 0.2180.83 ±\pm 0.83———

K=32K=32 evaluation of β=1.10\beta=1.10. Under K=32K=32 evaluation, static COPD (β=1.10\beta=1.10) drops from 60.00% (mean@8) to 58.62% (mean@32), falling below both default static COPD (β=1.00\beta=1.00, 58.91%) and SACOPD (59.48%, a +0.86 pp / +33-response difference in favor of SACOPD), confirming that the K=8K=8 difference was due to sampling variance. Pushing static strength to β=1.10\beta=1.10 trades sample accuracy for tail variance (98 vs. 97 solved), and increasing β\beta further to 1.50 degrades both metrics (57.92% / 72.50% in Table C8).

Independent selection vs. post-hoc grid search. Although the 120-problem math mean@32 intervals slightly cross zero (including +0.86 pp, 95% CI [−0.26,+2.01][-0.26, +2.01] against W2S-OPD α=5\alpha=5, and [−0.38,+2.43][-0.38, +2.43] for the +1.02 pp code mean@4 gain over W2S-OPD α=5\alpha=5), all paired mean@K point estimates are consistently positive (+0.86 pp over β=1.10\beta=1.10 and W2S-OPD α=5\alpha=5, and +0.57, +0.37, and +0.37 pp across the three math seeds against β=1.00\beta=1.00, averaging +0.43 pp). Where state-awareness and full-vocabulary supervision have their largest effect, the paired 95% CIs strictly exclude zero:

  • Code mean@4 (SACOPD −- Static COPD): +1.79 pp, 95% CI [+0.31,+3.33][+0.31, +3.33].
  • Math pass@32 (SACOPD −- W2S-OPD α=5\alpha=5): +3.33 pp, 95% CI [+0.83,+6.67][+0.83, +6.67].
  • Code pass@4 (SACOPD −- W2S-OPD α=5\alpha=5): +2.48 pp, 95% CI [+0.70,+4.46][+0.70, +4.46].
AuthorsOpenReview ↗

Response (Part 3/4)

W3: Attribution of static gains, strength-matched ExOPD (α=5.0\alpha=5.0), and role of the λ\lambda sweep

R3:

Attribution of static extrapolation gains. W2S-OPD (Yu et al., 2026) is an inspiring concurrent work; we proactively cited and adapted it to our setup for a thorough comparison, and constructed the strength-matched W2S-OPD (α=5\alpha=5) variant ourselves using our Proposition 1 (α=1/(1−λ)=5\alpha = 1/(1-\lambda) = 5; its original settings α=1.0/0.75\alpha=1.0/0.75 achieve 47.73% on math and 62.47% on code). Static COPD and its closed-form virtual-teacher formulation are an independent contribution of our work rather than a pre-existing baseline, with SACOPD further extending it through state-aware target selection.

Strength- and optimization-matched ExOPD (α=5.0\alpha=5.0). We ran ExOPD with matched extrapolation strength (λExOPD=5.0\lambda_{\text{ExOPD}}=5.0, matching α=5,ρ=4\alpha=5, \rho=4) and identical optimization settings (50 updates, 300 rollouts, matched learning rate and temperature):

Math Methodmean@32 (%)pass@32 (%)Correct / 3,840Solved / 120
Matched OPD46.1773.331,77388
Matched ExOPD (α=5.0\alpha=5.0)59.0977.502,26993
Static COPD (α=5.0\alpha=5.0 eq.)58.9179.172,26295
SACOPD59.4880.832,28497

Aligning α=5.0\alpha=5.0 lifts Matched ExOPD mean@32 to 59.09%, consistent with Proposition 1. However, optimizing high-extrapolation log-ratio rewards via sampled-token PPO gradients reduces coverage, leaving Matched ExOPD at 77.50% pass@32 (93/120 solved) compared to SACOPD's 80.83% (97/120 solved, +3.33 pp).

Role of the fixed-δ\delta parameterization λ\lambda sweep. In static COPD (β=1\beta=1), varying λ∈[0.20,0.90]\lambda \in [0.20, 0.90] shifts the target coordinate λ/(1−λ)\lambda/(1-\lambda) by 36×36\times (0.25 to 9.0). In SACOPD, Eq. (14) adjusts βs\beta_s across a 37×37\times range (13.63 to 0.37) to preserve ρstarget=ρ^s+δ\rho_s^{\text{target}} = \hat{\rho}_s + \delta, while rescaling the conditional KL loss by (1−λ)∈[0.10,0.80](1-\lambda) \in [0.10, 0.80] (an 8×8\times gradient-scale change, Eq. 25). Table 2(b) verifies that across 50 AdamW updates, this 8×8\times scaling leaves accuracy stable (58.44%–59.79%), removing λ\lambda as a sensitive target-position hyperparameter. We will clarify this in Section 5.2.


W4: Reliability of coordinate-only control and target shuffling across rollouts

R4: In addition to code (see R1), we ran Within-Batch Shuffle on mathematics (K=32K=32):

MethodControl GranularityCode Overall (m@4 / p@4)Math Overall (m@32 / p@32)Math Solved
Static COPDFixed constant target67.98 / 76.9358.91 / 79.1795 / 120
Global ReplayOffline step-mean replay69.57 / 77.0258.88 / 80.0096 / 120
Within-Batch ShuffleOnline batch targets (permuted)68.83 / 77.2259.51 / 80.8397 / 120
SACOPDOnline per-rollout matched69.77 / 78.3059.48 / 80.8397 / 120

By the Bregman decomposition Φs(ρ^s+δ)=Rs+12Isδ2+O(δ3)\Phi_s(\hat{\rho}_s+\delta) = R_s + \frac{1}{2}I_s\delta^2 + O(\delta^3) (Appendices A.4–A.5), 1D coordinate control is reliable when the off-axis residual RsR_s is small and stable and local curvature Is>0I_s > 0 is bounded away from zero. In Table D9 (steps 10–150), mean RsR_s remains within 0.0275–0.0825 and curvature is strictly positive on every rollout (mean 0.0638–0.1625, minimum 0.0219). On code, where prompts and prefixes exhibit higher within-batch heterogeneity, permuting targets degrades performance by 0.94 pp (mean@4) and 1.08 pp (pass@4). On math, where competition problems are more homogeneous, online batch-level projection already captures the gain (97/120 solved). We will add this criterion to Section 3.5.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Closing Summary of Rebuttal and Revision

As the discussion period closes, we provide a brief summary of the reviewer outcomes and the main additions incorporated into the revised PDF.

Reviewer status

  • 85UV (2 → 3, Weak Accept): Raised their score to Weak Accept after reviewing the code granularity controls and uncertainty analysis.
  • qZYX (2 → 3, Weak Accept): Appreciated the additional experiments and raised their score to Weak Accept. All rebuttal experiments (W1–W5) are now in the revised PDF, and we will expand the remaining setups in the camera-ready version.
  • RZXf (2, Conf. 4): Noted in their Nov 17 follow-up that many empirical concerns were addressed, while highlighting two central remaining concerns: (1) the distinct novelty of COPD relative to G-OPD/ExOPD, and (2) the modest incremental gains (especially on math) and necessity of SACOPD's reverse-KL projection over simpler adaptive strategies. On Nov 18–19, we responded directly with an explicit reframing of COPD's role (Sec. 2.2) and a new comparison against four projection-free adaptive controllers across three settings (Sec. 5.3.2, Table 5, App. F).

Key rebuttal findings incorporated into the revised PDF

  • Positioning and baselines (RZXf, qZYX W3): Section 2.2 now states upfront that static COPD coincides with strength-matched G-OPD/ExOPD when π−=πref\pi^- = \pi_{\text{ref}}, and frames COPD as the two-teacher exponential-family geometry supporting decoupled teacher pairs and SACOPD's signed localization. Under the same budget, SACOPD beats strength-matched ExOPD (α=5\alpha=5) in math pass@32 (80.83% vs. 77.50%, +3.33 pp; App. B.4.3) and beats EOPD on both math (+6.49/+5.00 pp) and code (+1.25/+0.90 pp; Table 1). Adding entropy weighting on top of SACOPD yields further gains (59.84%/81.67% math, 70.27% code mean@4; Sec. 5.5, Table G33).
  • Projection vs. projection-free controllers (RZXf Round 2, qZYX W4): Global Replay and Within-Batch Shuffle are granularity ablations within SACOPD (both still use ρ^s\hat{\rho}_s); per-rollout matching beats Global Replay by +1.28 pp in code pass@4 (Table 4). To test projection-free alternatives directly, we compared against four calibrated unsigned controllers (Reverse/Forward KL, TV, Token Disagreement) on 4B Math, 4B Code, and 1.7B Code (Table 5, App. F). SACOPD achieves the highest three-setting macro score (62.72%/74.26% vs. 60.23–62.18%/71.37–72.54%). Rollout diagnostics (Tables F28–F30) identify two failure modes in unsigned heuristics: zero-clamping when off-axis residuals jump 7.52×7.52\times on 1.7B Code (138/41/28 of 300 rollouts clamped for TV/Token/Reverse KL, trailing SACOPD by 3.85–6.96 pp), and target inversion past π+\pi^+ (up to 80/300 on 4B Math and 52/300 on 4B Code, vs. 0 inversions for SACOPD).
  • Controls and uncertainty (85UV, qZYX W1–W2, RZXf): Table 1 now reports three-seed mean ±\pm std and 10,000-resample paired 95% CIs (excluding zero for SACOPD vs. OPD on all four metrics, vs. COPD on code mean@4 [+0.31, +3.33], and vs. W2S-OPD α=5\alpha=5 on math/code pass@KK [+0.83, +6.67] / [+0.71, +4.41]). At K=32K=32, the K=8K=8 static math peak (β=1.10\beta=1.10) drops to 58.62% mean@32, below SACOPD's 59.48% (3-seed SACOPD vs. single-seed β=1.10\beta=1.10: +0.95 pp, CI [+0.19, +1.74]; App. B.6.2). A nine-point code static sweep shows SACOPD automatically operates inside the effective band without grid search while exceeding the best static pass@4 (78.30% vs. 77.80%; App. C.1).
  • Scaling, windows, and cost (qZYX W5, RZXf Q5): Without retuning, SACOPD improves over static COPD by +5.07 pp mean@4 ([+3.09, +7.09]) and +4.18 pp pass@4 on 1.7B Code (Table B6), transfers to cross-specialist teachers (60.21%/76.67%, 300/300 valid projections; Table 3, App. B.5.2), maintains code mean@4 gains across 256–4096 training windows (71.17% at 4096; Table 6), and widens its math mean@4 lead at 100–150 updates (+1.25 to +1.67 pp; Table D24). Projection takes 0.96 s/step (~48 s total; zero extra forwards when cached), and SACOPD uses 28.4% fewer code tokens than W2S-OPD (α=5\alpha=5) (Sec. 5.5, App. B.7).

Pointers to updates in the revised PDF

  • 85UV: Table 1, Table 4, App. B.6, C.1–C.2.
  • qZYX: W1 (Sec. 5.3.2–5.3.3, Tables 4, C17–C18), W2 (Table 1, App. B.6), W3 (Sec. 2.2, 5.1–5.2, App. B.4.3), W4 (Sec. 3.5, Tables 4–5, App. F), W5 (Sec. 5.3–5.5, Tables 3, 6, B6, B14–B15, D24, G31–G32).
  • RZXf: Sec. 1, 2.2, 5.1–5.5, Tables 1, 4–6, App. B.4.3, D.6, F, G.

Thank you again to the Area Chair and all reviewers for your time and helpful feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.