acceptodds
Under review as a conference paper at ICLR 2027

Reward-Aligned Reweighting for On- Policy Distillation

10 views

Abstract

On-policy distillation (OPD) has become a standard stage of large language model post-training: a student learns from a stronger teacher on trajectories generated by its own policy. Yet on-policy feedback does not make every teacher correction equally useful. Standard OPD weights token-level distillation terms uniformly, implicitly treating local teacher preference as a proxy for correction utility. A decision's task value, however, depends on how the student completes the subsequent reasoning. This mismatch can cause imitation to suppress viable student strategies or reinforce paths the student cannot reliably execute. Verified trajectory outcomes provide complementary evidence about continuation quality, but do not directly identify the utility of individual decisions. This motivates using outcome evidence to guide the relative strength of teacher corrections. We introduce Reward-Aligned Reweighting for On-Policy Distillation (R-OPD), which continuously reallocates teacher supervision using outcome agreement and the magnitude of teacher–student disagreement. It gives reward-aligned corrections greater relative influence while retaining dense feedback, moving beyond uniform imitation and hard filtering. Our analysis formalizes the mismatch between local teacher preference and student continuation value, and establishes sufficient conditions for reallocation to improve first-order task progress over uniform OPD. Across seven mathematical reasoning benchmarks, R-OPD achieves the highest average accuracy among the compared training methods in both cross-size and same-size distillation. It outperforms standard OPD on all seven benchmarks, with average gains of and for 1.7B and 4B students, respectively. An extension to code generation yields an average gain of over standard OPD. Together, these results highlight outcome-guided supervision allocation as an effective means of translating dense teacher feedback into stronger student performance across model scales and task domains.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
On-policy distillation, Knowledge distillation, Large language models, Post-training, Reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hW4cOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates a fundamental tension in sampled-token on-policy distillation (OPD): token-level teacher–student log-probability gaps At=log⁡πT(ot∣st)πθˉ(ot∣st)A _ t = \log \frac{\pi _ T(o _ t \mid s _ t)}{\pi _ {\bar{\theta}}(o _ t \mid s _ t)} (Eq. 3) reflect local teacher preference, which can conflict with the student's own continuation value QRθˉ(st,ot)Q _ R^{\bar{\theta}}(s _ t, o _ t) (Eq. 8). In Section 3 (Proposition 1, Eqs. 9–12), the authors show that locally interpolating the student toward the teacher in probability space always reduces local reverse KL to first order (Ds′(0)=−Vars[At]≤0D' _ s(0) = -\mathrm{Var} _ s[A _ t] \le 0), but simultaneously decreases conditional expected task reward whenever Covs(At,QRθˉ(s,ot))<0\mathrm{Cov} _ s(A _ t, Q _ R^{\bar{\theta}}(s, o _ t)) \lt 0.

To address this without discarding tokens or training a separate value network, Section 4 introduces R2-OPD\mathrm{R}^2\text{-OPD} (Reward-Aligned Reweighting for On-Policy Distillation, Eqs. 13–16, Algorithm 1). Using the signed binary trajectory outcome z(q,o)=2R(q,o)−1∈{−1,+1}z(q, o) = 2R(q, o) - 1 \in \lbrace -1, +1 \rbrace (Eq. 7) and the per-response median absolute gap νo=max⁡{median1≤j≤∣o∣∣Aj∣,ϵ}\nu _ o = \max\lbrace \mathrm{median} _ {1 \le j \le |o|} |A _ j|, \epsilon \rbrace (Eq. 13), R2-OPD\mathrm{R}^2\text{-OPD} computes a smooth gate gt=σ(βz(q,o)Atνo)g _ t = \sigma\left(\beta \frac{z(q,o)A _ t}{\nu _ o}\right) (Eq. 14, default β=10−3\beta = 10^{-3}) and normalizes it via Zo=Mo∑jgj∣Aj∣Z _ o = \frac{M _ o}{\sum _ j g _ j |A _ j|} (Mo=∑t∣At∣M _ o = \sum _ t |A _ t|) so that AtR2=ZogtAtA _ t^{\mathrm{R2}} = Z _ o g _ t A _ t preserves both token signs and total per-response absolute coefficient mass (Propositions 2–3, Eqs. 17–19). Proposition 4 (Eq. 21) characterizes when this reallocation improves first-order projection onto the true task gradient gRg _ R. Empirically, R2-OPD\mathrm{R}^2\text{-OPD} is evaluated on DAPO-Math-17k across 7 math benchmarks (Qwen3-4B-GRPO →\to Qwen3-1.7B-Base and Qwen3-4B-Base, Table 1) and on TACO across 2 code benchmarks (1.7B, Table 2), improving over vanilla sampled-token OPD by +3.5, +2.4, and +1.6 average points, respectively. While the theoretical exposition is admirably careful and the 1.7B math gains are solid, I lean toward weak rejection due to (1) an unexamined scale paradox at the default β=10−3\beta = 10^{-3} where median tokens receive virtually zero reweighting (σ(±10−3)≈0.50025\sigma(\pm 10^{-3}) \approx 0.50025) yet 25% of conflict mass is shifted via extreme heavy-tailed outliers, (2) a mismatch between statewise-centered preference in Proposition 1 and raw zAtz A _ t gating, (3) confounded component ablations and truncated benchmark/baseline matrices in Tables 2 and 3, and (4) incomplete model scaling (omitting Qwen3-8B and teacher-free GRPO while all 4B distilled students beat the frozen teacher) and missing comparisons with the closest outcome- and magnitude-aware OPD methods.

Critical strengths

1. Careful, well-scoped first-order analysis of preference–value misalignment in on-policy distillation.
Proposition 1 (Eqs. 10–12) and the analytic two-route example in Appendix G.5 cleanly formalize why minimizing local reverse KL can hurt student task return when the teacher prefers reasoning routes the student cannot complete. The authors are commendably transparent about the exact assumptions and limitations of their propositions across Appendices F–I.

2. Simple, zero-overhead reweighting rule with consistent improvements over vanilla sampled-token OPD.
R2-OPD\mathrm{R}^2\text{-OPD} requires no auxiliary value head, extra trainable parameters, or additional teacher forward passes, preserves coefficient signs and per-response absolute mass (Propositions 2–3), and improves all 7 math benchmarks over vanilla sampled-token OPD (46.2 \pm 0.4 vs. 42.7 \pm 0.6 on 1.7B and 50.8 vs. 48.4 on 4B in Table 1), with clean directional controls (Reversed alignment and Outcome-permuted in Table 3).

Critical weaknesses

1. Truncated benchmark/baseline matrices in Tables 2–3, omission of Qwen3-8B and pure GRPO, and the unexplained 4B same-size teacher paradox.

  • Truncated Table 3 and Table 2 matrices: Main ablation Table 3 (p. 8) reports results on only 3 of the 7 math benchmarks (AIME24, MATH500, Minerva), omitting AIME25, AMC23, AMC24, and OlympiadBench—even though the exact same R²-OPD (full) run (30.1, 77.8, 29.4) was already evaluated on all 7 benchmarks in Appendix Table 5 (where Success-only outperforms full R2-OPD\mathrm{R}^2\text{-OPD} on AIME24: 31.3 vs. 30.1). Reporting only a 3-benchmark subset in the main ablation table while holding back the other 4 benchmarks weakens the ablation evidence. Similarly, Table 2 (coding) omits the 4B student and drops 4 of the 6 baselines from Table 1 (Top-64 OPD, OPD + GRPO, SG-OPD, ExOPD).
  • Teacher–student checkpoint asymmetry and the 4B same-size paradox (Table 1): While the students start from raw pretrained -Base checkpoints (3.3 and 5.3 Avg.), the teacher Qwen3-4B-GRPO (46.4 Avg.) is trained from Qwen3-4B-Non-Thinking (Section 5.1, line 308). Puzzlingly, in the 4B same-size setting of Table 1, every single distilled student baseline (48.1 to 50.8, including vanilla Sampled-Token OPD at 48.4 and Top-64 OPD at 48.1) outperforms the frozen Qwen3-4B-GRPO teacher (46.4) on the same DAPO-Math-17k dataset, yet Table 1 includes no pure GRPO (teacher-free RLVR) baseline on Qwen3-1.7B-Base or Qwen3-4B-Base to isolate how much of the gain comes from on-policy exploration vs. teacher supervision.
  • Omission of Qwen3-8B and mixed per-benchmark rankings against OPDVR and ExOPD: All experiments are restricted to Qwen3-1.7B-Base and Qwen3-4B-Base, skipping Qwen3-8B-Base (as well as larger scales and non-Qwen families). Moreover, on several individual benchmarks in Table 1, R2-OPD\mathrm{R}^2\text{-OPD} trails existing gating baselines (OPDVR leads on 1.7B AIME24 [29.8 vs. 29.4], 1.7B AMC23 [63.3 vs. 63.0], and 4B MATH500 [84.6 vs. 83.6]; ExOPD leads on 4B AMC23 [68.9 vs. 68.3] and 4B Minerva [33.9 vs. 33.5]; and OPD + GRPO leads on 1.7B Minerva [30.6 vs. 29.1]).

2. Missing comparisons with the closest outcome-aware / magnitude-aware OPD methods discussed in Section 6.
Section 6 (p. 9, lines 455–461) explicitly discusses recent OPD methods that also combine trajectory outcomes, teacher–student gap magnitudes, or token importance—most notably EGRSD (Ke et al., 2026), Filter, then reweight (Li et al., 2026d), TIP (Xu et al., 2026b), Asymmetric OPD (Jia et al., 2026), and Uni-OPD (Hou et al., 2026). None of these closest competitors are evaluated in Tables 1 or 2, making it difficult to judge whether R2-OPD\mathrm{R}^2\text{-OPD}'s median-scaled sigmoid gate offers a meaningful empirical advantage over existing outcome- and magnitude-aware reweighting rules.

Questions for authors
  1. Assessment-critical (Explaining the β=10−3\beta = 10^{-3} scale paradox and testing statewise-centered preference At−Es[At]A _ t - \mathbb{E} _ s[A _ t]):

    • Why does β=10−3\beta = 10^{-3} reduce the batch conflict-mass share κˉB(β)\bar{\kappa} _ {\mathcal{B}}(\beta) from 0.53 to 0.28 in Figure 3(a) when σ(±10−3)≈0.50025/0.49975\sigma(\pm 10^{-3}) \approx 0.50025 / 0.49975 for median tokens (∣At∣/νo=1|A _ t|/\nu _ o = 1)? Please report the empirical percentiles (P50, P90, P99, Max) of νo\nu _ o and ∣At∣/νo|A _ t|/\nu _ o, as well as the fraction of responses where medianj∣Aj∣\mathrm{median} _ j |A _ j| is near zero or hits the floor ϵ=2−23\epsilon = 2^{-23}.
    • Since Proposition 1 / Eq. (12) shows that local probability updates are governed by the statewise-centered preference At−Es[At]=At+Kt(θˉ)A _ t - \mathbb{E} _ s[A _ t] = A _ t + K _ t(\bar{\theta}), how does R2-OPD\mathrm{R}^2\text{-OPD} perform if you gate on top-kk-approximated centered preference z(q,o)(At+Kt(θˉ))z(q,o)(A _ t + K _ t(\bar{\theta})) instead of raw z(q,o)Atz(q,o)A _ t?
  2. Assessment-critical (Full 7-benchmark Table 3 ablations, matched-strength gates, and Qwen3-8B / pure GRPO baselines):

    • Can you report all 7 mathematical benchmarks for every ablation row in Table 3 (just as reported for Success-only and Failure-only in Appendix Table 5), include a Sign-only gate tuned to match the conflict-mass share κˉB(10−3)≈0.28\bar{\kappa} _ {\mathcal{B}}(10^{-3}) \approx 0.28 (and a step-size-compensated No mass-norm), and include Top-64 OPD, OPD + GRPO, SG-OPD, and ExOPD in Table 2?
    • Why was Qwen3-8B-Base omitted, why do all distilled 4B students in Table 1 (48.1–50.8) outperform the Qwen3-4B-GRPO teacher (46.4), and what does pure GRPO achieve on Qwen3-1.7B-Base and Qwen3-4B-Base under the same budget?
  3. Assessment-critical (Comparisons with closest related methods and training-step clarification):

    • How does R2-OPD\mathrm{R}^2\text{-OPD} compare against EGRSD, Filter, then reweight, or TIP under the matched Table 1 setup, and how were the hyperparameters of OPD+GRPO, SG-OPD, and ExOPD tuned?
    • Finally, what was the exact number of training steps and checkpoint selection rule for Table 1 (given that Figures 1/6 end at step 140, Figures 2/4 at step 120, and Figure 5 at step 150)?
5 comments

Response to Reviewer hW4c (1/4)

We thank the reviewer for the detailed and constructive review, and for recognizing our first-order analysis of preference-value misalignment and the simplicity of the zero-overhead reweighting rule. The questions prompted several new experiments and diagnostics, reported below; all will be added to the revision.

Q1a. Why β=10−3\beta=10^{-3} changes the inconsistent mass. The reviewer is right that gt≈0.50025g_t\approx 0.50025 at a median-magnitude token, where ∣At∣/νo=1|A_t|/\nu_o=1 by construction. Such tokens, however, carry almost no coefficient mass. In sampled-token OPD, most sampled tokens are near-deterministic agreements between teacher and student, which makes νo\nu_o small, while the mass concentrates on a few fork tokens with ∣At∣/νo≫1|A_t|/\nu_o\gg 1. To measure these distributions directly, we re-ran R2-OPD (β=10−3\beta=10^{-3}, 1.7B setting) with per-token logging and analyzed every rollout of steps 60-90 (31 batches; 3,968 responses, 2,631 of them complete). The run reproduces the one behind Fig. 3: its logged inconsistent share over steps 60-120 is 28.0% (vs. 28.1%).

Quantity (complete responses)P50P90P99Max
νo\nu_o1.2×10−31.2\times10^{-3}2.2×10−22.2\times10^{-2}7.2×10−27.2\times10^{-2}0.13
∣At∣/νo\lvert A_t\rvert/\nu_o, per token0.987.1×1027.1\times10^{2}8.9×1048.9\times10^{4}2.8×1082.8\times10^{8}
∣At∣/νo\lvert A_t\rvert/\nu_o, weighted by ∣At∣\lvert A_t\rvert6.2×1026.2\times10^{2}5.9×1045.9\times10^{4}8.5×1068.5\times10^{6}2.8×1082.8\times10^{8}
  • In the median response, half of the coefficient mass lies on 3.1% of the tokens and 80% on 9.8%.
  • At β=10−3\beta=10^{-3}, 77% of the mass has β∣At∣/νo≥0.1\beta|A_t|/\nu_o\ge 0.1, and 43% has β∣At∣/νo≥1\beta|A_t|/\nu_o\ge 1, i.e. gt≤0.27g_t\le 0.27 or gt≥0.73g_t\ge 0.73. The mass-weighted mean of ∣gt−12∣|g_t-\tfrac12| is 0.30, against 0.00025 at the median token.
  • Recomputing Fig. 3(a) on these fixed rollouts gives an inconsistent share of 52.8% at β=10−8\beta=10^{-8}, 27.9% at β=10−3\beta=10^{-3}, and 6.3% as β→∞\beta\to\infty.
  • 2.7% of complete responses hit the floor ϵ=2−23\epsilon=2^{-23}. They carry 0.6% of the mass, and excluding them leaves the share at 28.4%, so the reduction comes from the bulk of responses, not from floor cases.

We will add these statistics to App. C and redraw Fig. 3(a) from the logged rollouts.

Q1b. Statewise-centered preference. Prop. 1 describes the expected update at a state, averaged over all actions. A sampled-token update acts only through the sampled action, and its effect on log⁡π(ot∣st)\log\pi(o_t|s_t) has the sign of AtA_t. Since KtK_t depends only on the state, adding it leaves the expected gradient unchanged and cannot be observed from a single sample. R2-OPD's gate asks whether the push actually applied at tt agrees with the outcome, i.e. it uses sign(zAt)\mathrm{sign}(zA_t). Centering would also require the teacher's top-kk distribution, whereas R2-OPD needs only sampled-token log-probabilities.

We quantified the alternative on the same logged rollouts, approximating KtK_t with the teacher's top-64 candidates (the student's mass outside them has a mass-weighted P90 of 1.2×10−31.2\times10^{-3}, so the approximation is tight):

  • Centering flips the sign of the gate argument (when −Kt<At<0-K_t<A_t<0) on 7% of tokens, which carry 5-6% of the mass.
  • Because KtK_t is large exactly where the mass sits (mass-weighted median 0.85 nats, against νo∼10−3\nu_o\sim10^{-3}), centering also changes gate magnitudes. With the same β\beta and mass normalization, it reallocates 13% of the coefficient mass (per-response total-variation distance, mass-weighted).
  • This reallocation goes in the wrong direction: on complete responses, the share of mass on corrections whose applied push contradicts the outcome rises from 32.5% to 46.6% (standard OPD: 70.1%).

Gating on z(At+Kt)z(A_t+K_t) therefore undoes part of the reallocation that R2-OPD is designed to perform. We will add this analysis to the appendix.

Q2a. Complete Table 2 (code; Qwen3-4B-GRPO → Qwen3-1.7B-Base; avg@4), now with all six baselines:

MethodHumanEval+LiveCodeBenchAvg.
Student64.021.142.6
Teacher78.046.562.3
Sampled-Token OPD71.337.554.4
Top-64 OPD70.937.154.0
OPD + GRPO72.538.455.5
OPDVR74.141.357.7
SG-OPD71.539.255.4
ExOPD73.940.057.0
R2-OPD75.642.258.9

All rows now use the same evaluation script. The student and teacher were already evaluated with it; we re-evaluated all distilled models with it (see Q2b), which updates the three method rows of the submitted Table 2. R2-OPD is best on both benchmarks: +4.5 over standard OPD, +1.2 over OPDVR, and +1.9 over ExOPD, the strongest of the newly added baselines.

Response to Reviewer hW4c (2/4)

Q2a (cont.). Full Table 3. All ablations on all seven benchmarks, including the two controls the reviewer requested: a sign-only gate with group multipliers matched to the full gate's inconsistent share (κˉ≈0.28\bar\kappa\approx 0.28), and a no-mass-norm variant rescaled by a single batch-level constant that restores the batch's total coefficient mass.

VariantAIME24AIME25AMC23AMC24MATH500MinervaOlympiadΔ\Deltaavg
R2-OPD (full)30.124.565.054.477.829.445.9–
Reversed alignment25.418.157.240.673.526.939.2−6.6
Outcome-permuted27.920.259.850.175.928.543.2−3.1
Sign-only gate27.320.062.952.676.628.743.1−2.3
Strength-matched sign-only26.820.761.952.075.828.643.2−2.6
Magnitude-only gate26.120.861.550.775.828.242.5−3.1
No mass-norm28.123.962.753.176.729.844.2−1.2
Step-size-compensated no mass-norm28.524.162.553.576.429.844.4−1.1
R2-OPD + GRPO31.525.865.454.978.030.246.1+0.7

The seven-benchmark averages confirm the conclusions drawn from three benchmarks in the submission:

  • Every control falls below the full gate, and reversing the alignment costs the most (−6.6).
  • Matching the inconsistent share does not rescue the sign-only gate (−2.6): the gain requires magnitude-aware weighting, not just the same amount of reallocation.
  • Compensating the global step size recovers almost nothing (−1.1 vs. −1.2), as AdamW already absorbs a global scale; the benefit of mass normalization comes from the per-response coefficient budget.

The Success-only control in App. Table 5 scores higher on AIME24, but the full gate exceeds it by 1.2 points on the seven-benchmark average. This table will replace Table 3 in the revision.

Q2b. Students above the teacher, and pure GRPO. We thank the reviewer for catching this; it exposed an error in our evaluation, for which we apologize. The teacher and the distilled models had been evaluated with scripts that were not fully aligned. In math, the distillation script contained answer-matching fixes (e.g., accepting 40.0 for the answer 40) that the GRPO script lacked, so the teacher was underestimated. In code, the distilled models had been evaluated with a different script from the one used for the student and teacher. We have now unified the evaluation: the teacher was re-evaluated with the corrected math script, and all distilled code models with the student/teacher script (Table 2 above). The corrected teacher scores 53.1, above every distilled 4B student (48.1-50.8), which resolves the inconsistency; we will correct the teacher row of Table 1.

The teacher is itself a pure-GRPO model: Qwen3-4B trained with GRPO on DAPO-Math-17k under the non-thinking chat template ("Qwen3-4B-Non-Thinking" in Sec. 5.1; we will state this explicitly). For the two Base models, we trained pure GRPO from the same checkpoints as the corresponding students for 500 steps (the last 100 after convergence), using the same evaluation and checkpoint-selection rule:

Pure GRPOAIME24AIME25AMC23AMC24MATH500MinervaOlympiadAvg.
Qwen3-1.7B-Base28.123.463.753.378.730.344.346.0
Qwen3-4B-Base33.627.466.956.284.432.547.949.8
Qwen3-4B, non-thinking (= teacher)36.530.170.761.286.335.251.553.1

Within 150 training steps, R2-OPD matches or exceeds pure GRPO from the same initialization (1.7B: 46.2 vs. 46.0; 4B: 50.8 vs. 49.8), whereas the GRPO runs train for 500 steps; standard OPD stays below GRPO in both settings (42.7 and 48.4). The two signals are also complementary: adding the GRPO term to R2-OPD yields a further +0.7 (Table 3).

Response to Reviewer hW4c (3/4)

Qwen3-8B and other model families. Compute constraints prevented these runs before the deadline, so the submission covers cross-size (4B → 1.7B) and same-size (4B → 4B) distillation within Qwen3. During the rebuttal, we added a larger teacher and a second model family, with the same baselines and benchmarks.

(a) Qwen3-8B-GRPO → Qwen3-1.7B-Base

MethodAIME24AIME25AMC23AMC24MATH500MinervaOlympiadAvg.
Student1.91.74.42.25.83.13.73.3
Teacher56.242.385.669.189.333.359.362.2
Sampled-Token OPD39.227.877.658.178.229.650.151.5
Top-64 OPD38.127.177.957.878.429.749.851.3
OPD + GRPO39.928.278.460.079.030.152.752.6
OPDVR41.829.481.062.980.131.051.854.0
SG-OPD40.628.078.160.679.230.450.952.5
ExOPD41.128.479.461.279.730.551.353.1
R2-OPD44.231.981.964.482.831.553.555.7

(b) Llama-3.2-3B-Instruct-GRPO → Llama-3.2-1B

MethodAIME24AIME25AMC23AMC24MATH500MinervaOlympiadAvg.
Student1.20.15.44.230.66.86.67.8
Teacher18.64.235.919.256.722.520.825.4
Sampled-Token OPD2.31.220.69.242.18.19.313.3
Top-64 OPD2.51.020.98.941.07.79.113.0
OPD + GRPO2.91.422.911.242.89.210.014.3
OPDVR3.12.224.012.444.59.912.415.5
SG-OPD3.31.923.811.843.29.612.115.1
ExOPD3.62.524.212.643.910.112.915.7
R2-OPD4.83.226.413.545.111.413.516.8

R2-OPD is best on every benchmark in both settings. On average, it improves over standard OPD by +4.2 and over the strongest baseline (OPDVR) by +1.7 with the Qwen3-8B teacher, and by +3.5 and +1.1 (ExOPD) in the Llama setting. Both settings will be added to the revision.

Per-benchmark rankings. Each run is reported at a single checkpoint, the one with the best average, so all of its numbers come from the same model. Individual benchmarks peak at different checkpoints; selecting a checkpoint per benchmark would raise every method's per-benchmark scores, ours included, and we deliberately avoid it. Under this stricter protocol, a baseline can lead on an individual benchmark, by at most 1.5 points, with such leads spread over three different baselines. Single small benchmarks are noisy (in Table 1, the seed standard deviation reaches 3.6 points on AMC24), so we rank methods by the average, on which R2-OPD is best in every setting. Fig. 1 also shows R2-OPD ahead of the baselines for most of training.

Reviewer_56EiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes R2-OPD, which reweights sampled-token distillation coefficients using verified response outcomes and teacher–student disagreement. A sigmoid gate changes their relative influence, and a response-level normalizer preserves their total absolute mass. The analysis separates local teacher preference from the value of a continuation generated by the student. Experiments cover two student sizes on seven mathematics benchmarks and a smaller code-generation study.

Critical strengths
  1. The method is simple and the analysis is careful about its scope. In particular, the paper distinguishes coefficient alignment from task-gradient alignment, and does not claim that preserving coefficient mass preserves the update norm or guarantees reward improvement. The implementation and detailed appendices make the proposed update easy to inspect.

  2. The main cross-size result is supported by three training seeds and several relevant baselines. R2-OPD improves the seven-benchmark average from 42.7 ± 0.6 to 46.2 ± 0.4, versus 45.0 ± 0.3 for OPDVR. The outcome-permuted control and success/failure branch ablations are useful checks. Same-size and code results provide further, although single-run, evidence.

Critical weaknesses

For the single-run same-size and code comparisons, I could not find how the checkpoints in Tables 1 and 2 were selected or which LiveCodeBench release and date range were used. Appendix A and the integration notes already provide the sampling settings and training budget. Clarifying the remaining protocol details would help assess these gains, which are about one point over the strongest reported competitor. This is a reporting concern; the main cross-size result has stronger support from three training seeds.

Questions for authors

Which checkpoints were used in Tables 1 and 2, how were they selected, and which LiveCodeBench release and date range was evaluated?

3 comments

Response to Reviewer 56Ei

We sincerely thank the reviewer for the careful reading and the positive assessment, in particular for noting that the analysis is careful about its scope and that the main cross-size result is supported by three training seeds. We answer the two reporting questions below and will add all of these details to Appendix A.3.

Checkpoint selection (Tables 1 and 2). Every method, including all baselines and R2-OPD, follows the same protocol. Checkpoints are evaluated every 20 training steps. Each method is trained until its evaluation accuracy converges and then for at least 50 additional steps, so that the plateau of every method is fully covered. For each training run, we report the single checkpoint with the best macro-average over all benchmarks of the table (the seven mathematical benchmarks in Table 1; HumanEval+ and LiveCodeBench in Table 2). All per-benchmark entries of a run therefore come from one checkpoint, not from per-benchmark maxima. For the three-seed 1.7B comparison, the checkpoint is selected within each seed, and we report the mean ± sample standard deviation over the three selected checkpoints. The single-run 4B and code experiments use the same rule.

For the 1.7B setting, Fig. 1 plots the evaluation accuracy of all methods throughout training (mean ± std over three seeds), so the reported values can be read against the whole run. R2-OPD reaches near-peak accuracy by about step 60 and generally stays above the baselines for the rest of training, so its advantage does not rest on the selected checkpoint.

LiveCodeBench version. We evaluate on LiveCodeBench release_v6 (code_generation_lite, test split): all 1,055 problems released between May 2023 and April 2025 (contest dates 2023-05-07 to 2025-04-06; 602 AtCoder, 444 LeetCode and 9 Codeforces problems). Every method is evaluated on this same set with the same sampling settings (avg@4).

We thank the reviewer again and hope these clarifications fully address the reporting concern.

Dear Reviewer,

Thank you again for your careful reading and thoughtful feedback. As the rebuttal deadline is approaching (in approximately one day), we would like to kindly follow up and ask whether our response has adequately addressed your concerns.

If so, we would be very grateful if you could consider updating your rating accordingly. Otherwise, we would greatly appreciate any further feedback on remaining concerns and would be happy to provide additional clarification.

Thank you again for your time and consideration.

Thanks for following up on the points in my review. Your explanations resolve most of my concerns. The remaining points don’t change my overall assessment, so I’ll leave my score as it is.

Reviewer_HFLeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Reward-Aligned Reweighting for On-Policy Distillation reweights token-level distillation signals using verified response outcomes and teacher–student disagreement, while preserving coefficient signs and per-response absolute mass. The paper provides conditional theoretical analysis and reports improvements over standard OPD on mathematical reasoning and code generation.

Critical strengths
  1. The main 1.7B experiments show consistent improvements over standard OPD across seven benchmarks, and they are supported by three training seeds.
  2. The method is straightforward to implement and requires no additional model passes or learned value estimator.
Critical weaknesses
  1. The distinct contribution is not convincingly isolated. Outcome-guided gating already exists in OPDVR and SG-OPD, making continuous weighting and mass normalization the core additions. However, the sign-only ablation changes weighting strength, and removing normalization changes coefficient scale. These confounds weaken the evidence that the proposed design is preferable to simpler alternatives. The single-run sharpness sweep does not establish a reliable advantage over mass-normalized hard gating.
  2. The theoretical and empirical support for the proposed mechanism remains limited. Proposition 4 establishes improvement when gate weights positively covary with task-gradient utility, but the paper does not establish that its gate satisfies this condition in practice. Reduced outcome-inconsistent mass does not demonstrate improved token-level credit assignment. In addition, gains over the strongest competing methods are modest, and the 4B, coding, and ablation results lack repeated runs. Incomplete baseline-specific settings further limit interpretation of these comparisons.
Questions for authors
  1. Does R²-OPD reliably outperform strength-matched sign-only weighting and mass-normalized hard gating across multiple seeds?
  2. Were the full SG-OPD and ExOPD methods implemented and tuned, including SG-OPD’s phased teacher sampling? Please clarify any deviations.
3 comments

Response to Reviewer HFLe (1/2)

We sincerely thank the reviewer for the careful reading and the precise, constructive questions, which helped us strengthen both the evidence and the presentation of the paper. Following the reviewer's suggestions, we have run the requested multi-seed comparisons, and we will incorporate all clarifications below into the revision.

W1 / Q1. Isolating the contribution

R2-OPD starts from two failure modes of standard OPD that the paper formalizes. Over-correction on successful trajectories: suppressing teacher-disfavored decisions can erode valid student strategies. Over-trust on failed trajectories: reinforcing teacher-preferred decisions can encourage reasoning paths the student cannot reliably complete. Prop. 1 shows why they matter: moving toward the teacher lowers the local reverse KL whenever the two disagree, but raises the student's expected reward to first order only when teacher preference covaries positively with the student's continuation value. Supervision should therefore be weighted by its likely usefulness, not by teacher–student agreement alone.

This view also separates R2-OPD from outcome-guided gating. A verified outcome is a single, noisy, trajectory-level signal: it does not identify which tokens caused the success or failure, so a reward-inconsistent correction is not necessarily harmful. Such positions also carry about half of standard OPD's coefficient mass (App. C.2, Fig. 4). Masking them, as OPDVR does, turns the noisy outcome into a token-level verdict and discards about half of OPD's dense supervision. SG-OPD keeps all tokens but routes them by sign with fixed factors. R2-OPD keeps every correction and its sign and reallocates a fixed per-response budget: the gate departs from 1/2 only where the disagreement is large relative to the response's own scale. Standard OPD (β=0\beta=0) and mass-normalized hard gating (β→∞\beta\to\infty) are the two endpoints of this family (Prop. 3).

The two requested controls, run with three seeds, each remove one of these design choices: a strength-matched sign-only gate (one multiplier per alignment group, matched to the full gate's reallocation between groups) removes magnitude awareness, and a mass-normalized hard gate (the β→∞\beta\to\infty endpoint, Eq. 78) discards the reward-inconsistent corrections.

Variant (3 seeds)AIME24AIME25AMC24MATH500MinervaAvg.
R2-OPD29.4 ± 0.824.6 ± 0.953.5 ± 2.177.6 ± 0.829.1 ± 0.642.8 ± 0.6
Strength-matched sign-only26.5 ± 1.020.4 ± 0.951.4 ± 2.275.7 ± 0.628.4 ± 0.740.5 ± 0.8
Mass-normalized hard gate29.7 ± 0.820.6 ± 0.752.6 ± 2.476.2 ± 0.428.2 ± 0.841.5 ± 0.7

R2-OPD outperforms the sign-only gate on all five benchmarks and the hard gate on four of five, by 2.3 and 1.3 points on average, with the largest gain on the hardest benchmark, AIME25 (+4.0 over the hard gate). Both magnitude-aware weighting and soft gating that retains reward-inconsistent corrections therefore contribute to the gains. Against unnormalized hard gating (OPDVR), R2-OPD also leads over three seeds (46.2 ± 0.4 vs. 45.0 ± 0.3).

Strength and scale. The ratio of about 1.001 in App. B.1 refers to a single median-magnitude token; reallocation happens mainly at large-∣At∣/νo|A_t|/\nu_o positions, which carry most of the mass. For the no-mass-norm control, AdamW largely absorbs a global rescaling of the coefficients, so this control mainly removes the per-response budget rather than changing the global step size. We will clarify both points in the revision.

W2. Support for the mechanism

Prop. 4 gives the exact condition for improvement, Covpo(g,u)>0\mathrm{Cov}_{p_o}(g,u)>0. Since gRg_R is not observable during training, we test it with controls that act on this covariance under a fixed mass budget. The reversed gate is exactly 1−gt1-g_t, so by Eq. (21) its first-order gain over standard OPD has the opposite sign to R2-OPD's. Empirically, it is at or below standard OPD on all three ablation benchmarks, while R2-OPD is above standard OPD on all three. The outcome-permuted control, which breaks the pairing between responses and outcomes, lies in between (-1.7), and the magnitude-only gate, which ignores outcomes, loses 2.4 points. This ordering is what Eq. (21) predicts when the full gate satisfies the condition. Moreover, removing more inconsistent mass (β≥1\beta\ge1) brings no further gain (App. C). The gains therefore come from how mass is allocated, not from how much inconsistent mass is removed.

Response to Reviewer HFLe (2/2)

Size of the gains. The gain over OPDVR (+1.2) is three to four times the seed-to-seed standard deviation of either method (0.3-0.4), and the mean ± std ranges of the two methods do not overlap. Measured against standard OPD, R2-OPD's improvement (+3.5) is about 1.5 times that of OPDVR (+2.3). The strongest baseline also changes across settings (OPDVR at 1.7B, ExOPD at 4B), whereas R2-OPD is best in both and improves over standard OPD on every benchmark.

Multi-seed evaluation and generalization. The main 1.7B comparison (Table 1) uses three seeds for every method, which is where we assess robustness, and the two ablation controls that bear most directly on W1 now also have three seeds (table above). Repeating the 4B experiments over seeds is considerably more costly, as each run occupies a full 8xH800 node. The 4B and code experiments are therefore designed to show that R2-OPD generalizes across model scales and domains, and in both it again improves over standard OPD and OPDVR.

Q2. Baselines

Both baselines are re-implemented in our codebase following the original papers' formulations, use the same rollout settings, data, optimizer and evaluation protocol as all other methods, and use the hyperparameters reported in the original papers without further tuning. Below, At=log⁡πT(ot∣st)−log⁡πθˉ(ot∣st)A_t=\log\pi_T(o_t|s_t)-\log\pi_{\bar\theta}(o_t|s_t) and πref\pi_{ref} is the student initialization, the reference policy of G-OPD.

  • ExOPD (G-OPD with reward extrapolation). The token reward λlog⁡πTπref−log⁡πθˉπref\lambda\log\frac{\pi_T}{\pi_{\mathrm{ref}}}-\log\frac{\pi_{\bar{\theta}}}{\pi_{\mathrm{ref}}} gives AtEx=At+(λ−1)[log⁡πT(ot∣st)−log⁡πref(ot∣st)]A_t^{\mathrm{Ex}}=A_t+(\lambda-1)\left[\log\pi_T(o_t\mid s_t)-\log\pi_{\mathrm{ref}}(o_t\mid s_t)\right], with λ=1.25\lambda=1.25, the value fixed in the original paper. Setting λ=1\lambda=1 recovers standard OPD.

  • SG-OPD. With A^o=(R(q,o)−Rˉq)/σq\hat A_o=(R(q,o)-\bar R_q)/\sigma_q the group-normalized GRPO advantage of the verifier reward, each token is routed by sign consistency:

    • consensus (A^oAt>0\hat A_o A_t>0):AtSG=At+(λhigh−1)(log⁡πT(ot∣st)−log⁡πref(ot∣st))A_{t}^{\mathrm{SG}}=A_t+(\lambda_{\mathrm{high}}-1)\left(\log\pi_T(o_t\mid s_t)-\log\pi_{\mathrm{ref}}(o_t\mid s_t)\right);
    • conflict (otherwise): AtSG=AtA^{SG}_t=A_t, i.e., the standard OPD coefficient.

    We use λhigh=1.8\lambda_{high}=1.8 and keep conflict tokens at the OPD coefficient, as in the original paper. Phased teacher sampling uses the original paper's teacher ratio (12.5% of prompt groups) and phase boundaries (30% and 35% of training): verified-correct teacher responses add an anchor α(t) LCE\alpha(t)\,\mathcal{L}_{CE}, with α(t)=1\alpha(t)=1 up to the first boundary, a cosine decay to 0 by the second, and α(t)=0\alpha(t)=0 afterwards.

Both baselines are competitive in the same-size 4B setting, where ExOPD is the strongest baseline (49.9) and SG-OPD improves over standard OPD, which suggests that their weaker 1.7B results stem from the cross-size setting rather than from the implementation. R2-OPD's advantage also does not depend on tuning β\beta: every tested value from 10−410^{-4} upward stays at or above baselines (App. C).

We hope these additions address the reviewer's concerns, and we would be grateful if the reviewer would consider them in the final assessment.

Dear Reviewer,

Thank you again for your careful reading and thoughtful feedback. As the rebuttal deadline is approaching (in approximately one day), we would like to kindly follow up and ask whether our response has adequately addressed your concerns.

If so, we would be very grateful if you could consider updating your rating accordingly. Otherwise, we would greatly appreciate any further feedback on remaining concerns and would be happy to provide additional clarification.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.