acceptodds
Under review as a conference paper at ICLR 2027

Reward-Aligned Distillation

Abstract

Reinforcement learning with verifiable rewards improves the reasoning capabilities of large language models, but remains computationally expensive and sample inefficient. On-policy distillation (OPD) provides dense token-level supervision from a stronger teacher, yet standard OPD distills every token regardless of whether the teacher's preference actually improves task reward. Existing token-selection methods rely on proxies such as uncertainty or teacher–student divergence, which need not identify reward-improving supervision. We introduce Reward-Aligned Distillation (RAD), a bilevel framework that weights distillation signals based on their predicted effect on the student's post-update reward. RAD differentiates through a temporary optimizer step to score each token's supervision, then updates the token weights by projected ascent from the initial token weights. The resulting weights can strengthen, suppress, or reverse individual distillation-gradient contributions. At a matched training-response budget, RAD improves mean accuracy on six competition-math benchmarks over the strongest evaluated baselines by 6.5 percentage points for Qwen3-1.7B (non-thinking) and 2.5 points for Qwen3-4B-Instruct-2507. On LiveCodeBench-v6, RAD reaches 54.69% and 58.67% pass@1 with Qwen3-1.7B (non-thinking) and OLMo3-7B-Instruct, respectively, exceeding the strongest evaluated baselines by 8.7 and 6.4 points at the matched training-response budget. These results support reward-guided weighting of teacher supervision across model families.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation, Reward-Aligned Distillation, Reinforcement Learning with Verifiable Rewards, Post-Training, Bilevel Optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_FBYQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Reward-Aligned Distillation (RAD), a token-level reweighting framework for on-policy distillation (OPD) under verifiable rewards. Observing that a stronger teacher's token preferences do not uniformly improve task reward, RAD formulates token weights wt(y)w _ t(y) via a one-step bilevel approximation (Section 4.1–4.2, Eqs. 1–5): starting from constant initial weights wt(0)=cw _ t^{(0)} = c, it takes a temporary optimizer step θ+=A(θ,∇θL^D(θ,c1;B))\theta^+ = \mathcal{A}(\theta, \nabla _ \theta \widehat{L} _ D(\theta, c\mathbf{1}; \mathcal{B})), generates fresh meta rollouts from πθ+\pi _ {\theta^+} on a separate meta batch to estimate the post-update reward gradient ∇θ+JR(θ+)\nabla _ {\theta^+} J _ R(\theta^+), and computes all token-level hypergradient scores It(y)I _ t(y) simultaneously via an optimizer-aware forward-mode Jacobian–vector product I(y)=−JVPθ[ℓD(θ;y)][p]I(y) = -\mathrm{JVP} _ \theta[\ell _ D(\theta; y)][p] that differentiates through AdamW moments and global gradient clipping (Section 4.3–4.4, Eqs. 6–7, Algorithm 1, Appendix G Eqs. 11–13). The updated weights wt(y)=Π[c−1,c+1](c+γIt(y))w _ t(y) = \Pi _ {[c-1, c+1]}(c + \gamma I _ t(y)) are bounded and signed—allowing RAD to amplify, suppress, or reverse teacher gradients—before restoring θ\theta and committing the weighted distillation update.

Empirically, RAD is evaluated at a matched budget of 102,400 generated training responses (100 steps of RAD with 512 actor + 512 meta responses/step vs. 200 steps of baselines with 512 responses/step) across six competition-math benchmarks (AIME24/25/26, HMMT2025-Feb/Nov, OlympiadBench, Table 1), LiveCodeBench-v6 (Figure 3, Appendix C Table 8), and a preliminary SWE-bench Verified setup (Appendix D Figure 9), using Qwen3-1.7B (non-thinking), Qwen3-4B-Instruct-2507, and OLMo3-7B-Instruct students. While the signed, optimizer-conditioned alignment formulation and custom JVP engineering (Appendix H, Algorithm 2) are appealing, I lean toward weak rejection because (1) the paper's central post-update bilevel lookahead claim is contradicted by its own c=0c = 0 operating regime and Table 2 ablations, and (2) the experimental design mixes checkpoint generations across scales (Qwen3-1.7B (non-thinking) vs. Qwen3-4B-Instruct-2507), never evaluates Qwen3-8B (or any 7B/8B model on math, or Qwen3-4B on coding), relies on a single training seed (seed = 42), and sees several distillation baselines degrade below the untuned Base checkpoint on larger models.

Critical strengths

1. Optimizer-aware, signed token reweighting with solid JVP systems engineering.
Unlike uncertainty heuristics or nonnegative token selection (e.g., TIP, RLAD), RAD allows negative weights to actively push the student away from teacher preferences that anti-align with the AdamW-preconditioned reward update direction pp (Eq. 6, Appendix G Eq. 12). Table 2 confirms that both signed weights (32.71% vs. 25.00% for Positive only) and differentiating through AdamW (32.71% vs. 16.88% for SGD lookahead) are critical. The custom Triton forward-mode attention JVP kernel (Appendix H, Eq. 14, Algorithm 2) and olmo3_layer_streamed FSDP backend make exact per-token JVP computation practical at 32k context lengths.

2. Strong gains on Qwen3-1.7B and code generation across two model families, plus compatibility with OPD2\mathrm{OPD}^2.
At matched generated-response counts (102,400 responses), RAD delivers large improvements on Qwen3-1.7B (non-thinking) for both competition math (39.07% vs. 32.59% for ExOPD, Table 1) and LiveCodeBench-v6 (54.69% vs. 45.97% for RLAD, Table 8), improves OLMo3-7B-Instruct on LiveCodeBench-v6 (58.67% vs. 52.27% for GRPO), and stacks cleanly on top of OPD2\mathrm{OPD}^2 (+2.66 and +6.51 points in Table 4).

Critical weaknesses

1. The central post-update bilevel lookahead mechanism is not empirically justified over pre-update AdamW gradient alignment.
The paper's headline methodological narrative is that RAD estimates "how each token's distillation signal affects the student's post-update reward" via a temporary distillation step θ+\theta^+. However, because the main experiments set c=0c = 0, the temporary step ignores the current batch's distillation gradient entirely, and in the only ablation that isolates the post-update step (Table 2, Qwen3-1.7B), the Pre-update gradient variant (∇θJR(θ)\nabla _ \theta J _ R(\theta) without the temporary update) beats full RAD (34.32% vs. 32.71%). In Section 5.5 (Table 3), the authors compare full RAD against RAD-Lite—which simultaneously removes three things (the temporary lookahead step, the AdamW optimizer Jacobian, and separate meta rollouts)—finding that RAD-Lite wins on Qwen3-1.7B (37.50% vs. 32.71%) but collapses on Qwen3-4B-Instruct-2507 (37.29% vs. 55.26%). Crucially, because RAD-Lite conflates dropping the lookahead step with dropping the AdamW Jacobian and fresh meta rollouts, Table 3 does not show that the post-update lookahead step caused the 4B gap—especially since Table 2 already showed that dropping the AdamW Jacobian (SGD lookahead, -15.83 pp) or reusing rollouts (Pre-update gradient + rollout reuse, -15.78 pp) causes massive degradation on its own. Without testing Pre-update gradient (retaining the AdamW Jacobian and fresh meta rollouts) on Qwen3-4B-Instruct-2507 and coding, the paper fails to justify a lookahead step that consumes 47.2% of each training step's wall-clock time (Appendix Table 12).

2. Cross-generation student mismatch, omission of Qwen3-8B (and incomplete model–task matrix), single-seed training (seed = 42), and degraded distillation baselines.

  • Cross-generation student mismatch (Qwen3-1.7B (non-thinking) vs. Qwen3-4B-Instruct-2507) and omission of Qwen3-8B: In Table 1, the 1.7B student is the original May 2025 Qwen3-1.7B (non-thinking) checkpoint (which starts at a very weak 13.39% 6-benchmark / 8.23% 4-benchmark base accuracy) distilled from the updated July 2025 Qwen3-30B-A3B-Instruct-2507 teacher (65.18%), yielding a large +14.56 pp gain over GRPO. At 4B, however, the student switches to the already heavily post-trained July 2025 Qwen3-4B-Instruct-2507 checkpoint (52.38% base), where TIP, RLAD, ExOPD, and RAD-Lite all fall below Base and RAD's macro-average gain over GRPO drops to +2.51 pp. Moreover, within the same Qwen3 (non-thinking) series where 1.7B, 4B, and 8B (Qwen3-8B) are all publicly available, the paper never evaluates Qwen3-8B (nor does it evaluate OLMo3-7B-Instruct on math, or Qwen3-4B-Instruct-2507 on coding). Furthermore, Appendix A.2 (p. 18) and Appendix H.1 (pp. 30–31) reveal that the flash_triton JVP backend used for Qwen3 models must materialize a full temporary student replica θ+\theta^+ and direction vector in GPU memory (whereas scaling to 7B required writing a custom olmo3_layer_streamed FSDP backend that still took 101.61–166.79 hours per run in Table 11 and forced omitting ExOPD and OPD2\mathrm{OPD}^2). This leaves open whether RAD can scale to Qwen3-8B both computationally and in performance over GRPO.
  • Single-seed variance and small micro-averaged margins on Qwen3-4B-Instruct-2507: Every method is trained only once (seed = 42, Section 6 p. 10 & Table 6); the reported standard deviations reflect only evaluation sampling from a single checkpoint. Because five of the six math benchmarks (AIME24/25/26, HMMT2025-Feb/Nov) contain only 30 problems each while OlympiadBench contains 675 problems (Appendix Table 5, where RAD is 70.39% vs. GRPO 69.69%, a +0.70 pp difference), weighting by problem count across all 825 problems reduces RAD's micro-averaged edge over GRPO on Qwen3-4B to roughly +1.23 percentage points.
  • Distillation baselines degrading below Base: On Qwen3-4B-Instruct-2507 (Table 1, Base 52.38%), TIP (51.93%), RLAD (50.87%), and ExOPD (51.49%) all score below the untuned Base model; on OLMo3-7B-Instruct (Table 8, Base 43.91%), TIP drops to 36.73%. Relying on untuned default hyperparameters from prior papers makes the comparison against competing distillation selectors less convincing.
Questions for authors
  1. Assessment-critical (Isolating post-update lookahead vs. pre-update AdamW alignment on Qwen3-4B and coding):
    In Table 2 (Qwen3-1.7B), the Pre-update gradient ablation (which keeps the AdamW Jacobian and separate meta rollouts, but skips the temporary update θ+\theta^+ and uses ∇θJR(θ)\nabla _ \theta J _ R(\theta)) achieves 34.32%, beating full RAD (32.71%) while avoiding the temporary update step. What happens when you run this exact Pre-update gradient variant (with AdamW Jacobian + fresh meta rollouts) on Qwen3-4B-Instruct-2507 math and on LiveCodeBench-v6? Does the post-update lookahead step provide any actual gain over pre-update AdamW gradient alignment when c=0c = 0?

  2. Assessment-critical (Why was Qwen3-8B not trained, and how does RAD scale across matched-generation Qwen3 models?):

    • In Table 1, 1.7B uses the May 2025 Qwen3-1.7B (non-thinking) checkpoint (13.39% base) whereas 4B switches to the July 2025 Qwen3-4B-Instruct-2507 checkpoint (52.38% base), where RAD's margin over GRPO narrows from +14.56 pp to +2.51 pp (+1.23 pp micro-average). Since the Qwen3 (non-thinking) series includes 1.7B, 4B, and 8B (Qwen3-8B) from the same release generation, why was Qwen3-8B never trained or evaluated?
    • Specifically, was Qwen3-8B omitted because the flash_triton JVP backend used for Qwen models (which materializes a full student replica and parameter direction in GPU memory, Appendix A.2 p. 18) runs out of memory on 8 H100 GPUs at 8B scale, or because distillation gains saturate at 8B? Also, why was Qwen3-4B-Instruct-2507 omitted from the coding experiments (Table 8) and OLMo3-7B-Instruct omitted from math?
  3. Assessment-critical (Multi-seed stability and baseline degradation below Base):

    • Can you provide at least 2–3 independent training seeds for the Qwen3-1.7B and Qwen3-4B-Instruct-2507 math settings to verify that the +2.51 pp macro-average (+1.23 pp micro-average) margin over GRPO on Qwen3-4B exceeds training-seed noise?
    • Why do TIP, RLAD, and ExOPD all degrade below the untuned Base checkpoint (52.38%) on Qwen3-4B-Instruct-2507 in Table 1 (and TIP by -7.18 pp on OLMo3-7B-Instruct in Table 8), and do they recover when their distillation weight / clipping hyperparameters are tuned?
4 comments

We thank the reviewer for their comments. We believe there are some misunderstandings. Please allow us to address your concerns point-by-point. Revisions to the paper are highlighted in blue.


Q1. The central post-update bilevel lookahead mechanism is not empirically justified over pre-update AdamW gradient alignment.

A1. Thank you for raising this question. We have extended the comparison between RAD and the pre-update gradient variant to Qwen3-4B-Instruct-2507 math in Section 5.4 of the revised paper (lines 436–438). The pre-update variant retains AdamW-aware scoring and independent fresh meta rollouts, but samples and evaluates the reward gradient at the current student rather than the temporarily updated student. On Qwen3-4B-Instruct-2507, pre-update gradient achieves 47.19% compared with 55.26% for RAD. This shows that pre-update gradient is not consistently as good as RAD: although it achieves comparable accuracy on Qwen3-1.7B (non-thinking) math, it performs substantially worse on Qwen3-4B-Instruct-2507.

Regarding computational cost, the reported 47.2% covers the entire lookahead stage, including the temporary optimizer update, weight transfer, meta rollout generation, reward-gradient computation, and restoration (Appendix E.2, Table 13, lines 1479–1486). The pre-update variant still requires fresh meta rollouts and reward-gradient computation, as well as AdamW-aware scoring and JVP computation. Consequently, 47.2% should not be interpreted as the time saved by omitting the temporary update.

Setting c=0c=0 makes the current batch's distillation gradient zero at the initial token weights. However, the temporary AdamW step can still change the student parameters through accumulated momentum and decoupled weight decay (Section 4.3, lines 250–252). Therefore, evaluating the reward gradient after this step need not be equivalent to evaluating it at the current student.

Q2. Cross-generation student mismatch, omission of Qwen3-8B.

A2. We use Qwen3-1.7B (non-thinking) and Qwen3-4B-Instruct-2507 for math tasks and OLMo3-7B-Instruct for coding instead of same-family and series models, e.g. Qwen3-4B and Qwen3-8B, because we want to show RAD works for different models instead of just a single model series. To address the reviewer’s concern, we compared RAD, OPD, and GRPO for Qwen3-8B (non-thinking) in Appendix A.5 of the revised paper (lines 1082–1089, Table 8, and Figure 7) and below. The experiments follow the same setups as Table 1.

Qwen3-8B (non-thinking) math accuracy (%). Bold and underline mark the best and second-best means within each block.

RAD at step 50 versus baselines at step 100

MethodStepAIME24AIME25HMMT25-FebHMMT25-NovMean
GRPO10057.71±6.2057.71 \pm 6.2043.96±5.3043.96 \pm 5.3029.58‾±4.70\underline{29.58} \pm 4.7038.54±5.2738.54 \pm 5.2742.45±2.6142.45 \pm 2.61
OPD10063.96‾±3.95\underline{63.96} \pm 3.9547.92‾±4.98\underline{47.92} \pm 4.9828.33±5.7728.33 \pm 5.7743.54‾±3.81\underline{43.54} \pm 3.8145.94‾±2.28\underline{45.94} \pm 2.28
RAD5070.63±3.77\mathbf{70.63} \pm 3.7760.83±5.95\mathbf{60.83} \pm 5.9538.33±3.73\mathbf{38.33} \pm 3.7350.63±5.68\mathbf{50.63} \pm 5.6855.10±2.60\mathbf{55.10} \pm 2.60
Gain over best baseline—(+6.67)(+12.92)(+8.75)(+7.08)(+9.17)

RAD at step 100 versus baselines at step 200

MethodStepAIME24AIME25HMMT25-FebHMMT25-NovMean
Teacher—74.79±3.7274.79 \pm 3.7261.46±3.5361.46 \pm 3.5343.75±5.3843.75 \pm 5.3859.17±5.5959.17 \pm 5.5959.79±2.0959.79 \pm 2.09
Base student026.67±5.5326.67 \pm 5.5319.79±5.2019.79 \pm 5.2010.63±4.2810.63 \pm 4.289.79±4.489.79 \pm 4.4816.72±2.4016.72 \pm 2.40
GRPO20071.88‾±5.27\underline{71.88} \pm 5.2762.29‾±5.74\underline{62.29} \pm 5.7440.83‾±4.93\underline{40.83} \pm 4.9351.25‾±6.96\underline{51.25} \pm 6.9656.56‾±2.99\underline{56.56} \pm 2.99
OPD20062.29±4.3762.29 \pm 4.3749.17±5.2049.17 \pm 5.2028.13±3.7228.13 \pm 3.7240.42±5.1240.42 \pm 5.1245.00±2.3945.00 \pm 2.39
RAD10073.54±7.02\mathbf{73.54} \pm 7.0265.83±4.33\mathbf{65.83} \pm 4.3342.50±4.64\mathbf{42.50} \pm 4.6453.75±3.31\mathbf{53.75} \pm 3.3158.91±2.55\mathbf{58.91} \pm 2.55
Gain over best baseline—(+1.67)(+3.54)(+1.67)(+2.50)(+2.34)

Our Qwen3-8B experiments address both concerns: the flash_triton JVP backend supports training at 8B scale without running out of memory in our setup, and the student continues to benefit from distillation at this scale.

The full student replica is an implementation choice of our current Qwen3-1.7B experiments, rather than a requirement of RAD. RAD also supports layer-streamed JVP with parameter sharding, as used in our OLMo3-7B-Instruct experiments (Appendix A.2, lines 965–971; Appendix H.1, lines 1727–1737). This follows established sharding principles used by FSDP and ZeRO and provides a path to reducing JVP memory requirements for larger students.

Q3. Single-seed variance and small micro-averaged margins on Qwen3-4B-Instruct-2507.

A3. We have now completed Qwen3-1.7B (non-thinking) RAD experiments with training seeds 42, 43, and 44. We compare RAD at 100 training updates with GRPO, OPD, and ExOPD at 200 updates on the same four benchmarks. Each run is evaluated with 16 responses per problem, and its accuracy equally averages AIME24, AIME25, HMMT2025-Feb, and HMMT2025-Nov.

MethodTraining seedUpdatesFour-benchmark mean accuracy (%)
GRPO4220018.75
OPD4220025.42
ExOPD4220026.15
RAD4210034.69
RAD4310030.68
RAD4410034.53
RAD mean ± SD across training seeds42, 43, 4410033.30 ± 2.27

The three-seed RAD mean exceeds GRPO by 14.55 percentage points, OPD by 7.88 points, and ExOPD by 7.15 points. All three RAD runs exceed the reported score of ExOPD, the strongest external baseline on these four benchmarks.

We report macro-averaged accuracy, the arithmetic mean of the six benchmark accuracies, so that each benchmark contributes equally (Table 1 caption, lines 270–273). In contrast, micro-averaging weights each problem equally: OlympiadBench accounts for 675 of the 825 problems (Table 5, lines 927–937) and would therefore receive 81.8% of the total weight. Both aggregations are meaningful but answer different evaluation questions. We use macro-averaging as our primary aggregate to measure performance across the selected benchmarks without letting their sizes determine their relative importance.

Equal task or benchmark weighting also has established precedents in LLM distillation. GKD (Agarwal et al., ICLR 2024), Section 4.4 and Figure 10 explicitly uses an "unweighted average over all tasks" when reporting accuracy within each of the MMLU and BBH benchmark suites. MiniLLM (Gu et al., ICLR 2024), Table 8 reports the average accuracy on SST2 and BoolQ. For mathematical reasoning, DistiLLM-2 (Ko et al., ICML 2025), Table 3 reports the arithmetic mean of GSM8K and MATH pass@1.

Q4. Distillation baselines degrading below Base.

A4. For the method-specific hyperparameters, we use a retention ratio of 0.5 for TIP, a mixing coefficient of 0.5 for RLAD, and an extrapolation coefficient of 1.25 for ExOPD (Table 7, lines 983–988), following values evaluated in their respective papers. Their papers include comparisons of token-retention ratios for TIP and coefficient-sensitivity analyses for RLAD and ExOPD. (TIP, Section 7.4; RLAD, Section 5.2.3; ExOPD, Section 4.1) We have tried to tune the hyperparameters for these methods, but they still degrade the student model's performance.

Distillation from a stronger teacher does not guarantee improved task accuracy over the initial student. Prior studies also report degradation under particular on-policy distillation configurations:

  • On-Policy Delta Distillation (Heo et al., 2026), Tables 4–5, reports below-initial-model results for OPD and ExOPD on Qwen3 thinking models.
  • REOPOLD (Ko et al., 2026), Section 5.1 and Figure 7, reports that vanilla reverse-KL OPD causes a sharp accuracy decline for the already strong DeepSeek-R1-Distill-Qwen-7B student and fails to improve beyond its initial performance on AIME24.
  • The Many Faces of On-Policy Distillation (Zhu et al., 2026), Section 4.1 and Figure 4, reports that Qwen3-1.7B distilled from Qwen3-8B initially improves but collapses to nearly zero accuracy on MATH500, AIME24, and AIME25 by step 1,000 when using unnormalized Top-20 reverse KL.

These findings show that distillation can degrade performance below the initial student. Our Qwen3-4B-Instruct-2507 setting may face similar challenges to the strong-student settings studied in On-Policy Delta Distillation (Heo et al., 2026) and REOPOLD (Ko et al., 2026): the student already achieves accuracy close to that of the teacher, leaving limited room for improvement through distillation (Table 1, teacher at lines 277–278 and 4B base student at lines 287–289).

We additionally evaluated the pre-update gradient ablation on Qwen3-8B (non-thinking). At the same budget of 100 training updates, RAD achieves 58.91% four-benchmark mean accuracy, compared with 55.99% for the pre-update variant, an improvement of 2.92 percentage points. Together with the 4B comparisons, these results indicate the benefit of post-update lookahead.

Thank you again for your constructive feedback. Our response adds Qwen3-8B comparisons, three-seed Qwen3-1.7B experiments, and additional pre-update gradient ablations. We also clarify why AdamW can produce a temporary update at zero initial token weights, explain the runtime breakdown and memory requirements, and clarify our use of equally weighted benchmark accuracy. Finally, we discuss baseline hyperparameter choices and prior reports of distillation degrading student performance.

We hope these additional experiments and clarifications have addressed your concerns. Please let us know if any questions remain; we would be happy to provide further clarification before the discussion period ends.

Thank you again for your time and constructive feedback. With about one day remaining in the discussion period, we would appreciate a response indicating whether our replies and additional experiments have addressed your concerns. If any substantive questions remain, please let us know so that we can provide further clarification before the discussion closes.

Reviewer_aYjGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Reward-Aligned Distillation (RAD), which assigns token-level weights in on-policy distillation according to the local effect of distillation updates on task reward. RAD computes scores through a temporary optimizer step, independent meta rollouts, and a Jacobian–vector product, followed by a projected weight update. The resulting weights can strengthen, suppress, or reverse teacher contributions. Experiments cover mathematical reasoning, code generation, and preliminary software-engineering tasks, with comparisons of generated-response budgets and training time. RAD outperforms the evaluated baselines across several students and demonstrates compatibility with an alternative distillation signal.

Critical strengths
  1. The method directly addresses misalignment between teacher supervision and task objectives. Rather than selecting tokens solely through uncertainty or teacher–student disagreement, RAD uses the reward gradient and student optimizer to assess distillation contributions, including signed weights. The derivations and practical JVP implementation effectively adapt established reweighting principles to on-policy language-model distillation.

  2. The empirical improvements are substantial and span different models and tasks. At matched generated-response budgets, RAD improves six-benchmark mean math accuracy over the strongest evaluated baselines by 6.48 and 2.51 percentage points for the two students. On LiveCodeBench-v6, the corresponding improvements are 8.72 and 6.40 points. Results with OLMo and the OPD² extension provide additional evidence of applicability.

  3. The paper carefully accounts for additional computation and experimental budgets. The main comparison counts meta rollouts, while the appendix reports training time, stage costs, and a comparable-time evaluation. On Qwen3-1.7B math, RAD retains a 5.47-point advantage over the strongest baseline at comparable training time. Positive-only weights, weight shuffling, and score reversal also provide evidence that token scoring matters.

Critical weaknesses
  1. The independent value of post-update lookahead is not fully established. In Table 2, the pre-update reward-gradient variant achieves 34.32%, compared with 32.71% for full RAD. In Table 3, RAD-Lite also performs better on the 1.7B student but substantially worse on the 4B student. The authors discuss these findings, but RAD-Lite simultaneously changes optimizer-aware scoring, rollout reuse, and update count, leaving the source of the full method’s advantage unclear. This limits understanding of the central contribution and when it is needed.

  2. The independent contribution of teacher information is insufficiently isolated. Appendix F expresses each token gradient as the student log-probability gradient multiplied by a scalar containing teacher probabilities. Improvements could therefore reflect both teacher information and reward-guided gradient reweighting. GRPO, OPD, and shuffled-weight controls do not fully separate these effects. A control preserving the scoring procedure while removing teacher information or using a fixed reference model would strengthen the interpretation that RAD makes better use of teacher supervision.

  3. The effects of score scale and projection saturation are insufficiently analyzed. Weights are obtained by scaling and clipping hypergradients, and the implemented step size additionally depends on the first batch’s token count and current length. The paper does not report step-size sensitivity, saturation rates, or score distributions. Since AdamW-aware and SGD scores can differ in magnitude as well as direction, the existing ablation does not fully distinguish the benefits of optimizer geometry from differences in effective weight scale.

Questions for authors
  1. Could pre-update and post-update reward gradients be compared while keeping independent meta data, optimizer-aware scoring, and training budgets fixed, including on the 4B student? What primarily explains the contrasting RAD-Lite results across the two students?

  2. What happens when the teacher-dependent scalar is replaced by a constant, or the teacher is replaced by the frozen initial student, while retaining the reward-guided scoring and update procedure? Can these controls identify how much improvement comes from teacher information?

  3. Could the authors report sensitivity to the weight step size and the fraction of weights reaching the clipping boundaries? Does the performance gap between SGD and AdamW scoring remain substantial after matching their score scales?

5 comments

We thank the reviewer for the positive feedback, thoughtful review, and for appreciating the merits of the work! Revisions to the paper are highlighted in blue.


Q1. The independent value of post-update lookahead is not sufficiently established. Comparison between pre-update and post-update reward gradients.

A1. Thank you for raising this question. We have extended the comparison between RAD and the pre-update gradient variant to Qwen3-4B-Instruct-2507 math in Section 5.4 of the revised paper (lines 436–438). The pre-update variant retains AdamW-aware scoring and independent fresh meta rollouts, but samples and evaluates the reward gradient at the current student rather than the temporarily updated student. On Qwen3-4B-Instruct-2507, pre-update gradient achieves 47.19% compared with 55.26% for RAD. This shows that pre-update gradient is not consistently as good as RAD: although it achieves comparable accuracy on Qwen3-1.7B (non-thinking) math, it performs substantially worse on Qwen3-4B-Instruct-2507.

We additionally evaluated the pre-update gradient ablation on Qwen3-8B (non-thinking). At the same budget of 100 training updates, RAD achieves 58.91% four-benchmark mean accuracy, compared with 55.99% for the pre-update variant, an improvement of 2.92 percentage points. Together with the 4B comparisons, these results indicate the benefit of post-update lookahead.

Q2. Contrasting RAD-Lite results across the two students.

A2. As we have shown in A1, the post-update lookahead is essential for Qwen3-4B-Instruct-2507 to perform well. RAD-Lite lacks RAD's full scoring procedure, including temporary lookahead and meta rollouts. This could cause the performance degradation for Qwen3-4B-Instruct-2507.

Q3. The independent contribution of teacher information is insufficiently isolated.

A3. Thank you for this helpful suggestion. We evaluated both proposed ablations on Qwen3-4B-Instruct-2507.

  1. Constant coefficient: Replace the teacher-dependent scalar log⁡(πθ/πT)+1\log(\pi_\theta/\pi_T)+1 with 11, giving the per-token loss gradient

    ∇θℓconst=∇θlog⁡πθ.\nabla_\theta \ell_{\mathrm{const}} = \nabla_\theta \log \pi_\theta.

  2. Frozen initial student: Replace the teacher with the initial student π0\pi_0, held fixed throughout training, giving the per-token loss gradient

    ∇θℓinit=(log⁡πθπ0+1)∇θlog⁡πθ.\nabla_\theta \ell_{\mathrm{init}} = \left(\log\frac{\pi_\theta}{\pi_0}+1\right)\nabla_\theta\log\pi_\theta.

The corresponding objectives are used consistently in both token scoring and student updates. Both ablations retain RAD's AdamW-aware lookahead, independent fresh meta rollouts, learned signed token weights, and optimization hyperparameters. We report mean accuracy across AIME24, AIME25, HMMT2025-Feb, and HMMT2025-Nov, with 16 responses per problem.

MethodFour-benchmark mean accuracy (%)
RAD (original teacher)55.26
Constant coefficient43.70
Frozen initial student44.38

RAD exceeds the constant-coefficient and frozen-initial-student ablations by 11.56 and 10.89 percentage points, respectively. These results support an additional contribution from the original teacher's token-level information in this 4B setting, beyond the retained reward-guided scoring and update procedure.

Q4. The effects of score scale and projection saturation are insufficiently analyzed.

A4. Thank you for highlighting this issue. We have added Figure 8 in Appendix A.6 (lines 1133–1140; Figure 8 caption, lines 1175–1182) to characterize score and weight dynamics for the Qwen3-1.7B (non-thinking) and Qwen3-4B-Instruct-2507 RAD math runs. It reports token-score standard deviation, the positive-score fraction, token-weight standard deviation, saturation at the clipping boundaries, pre-clipping magnitude quantiles, and mean absolute weight over 100 training steps.

Figure 8 shows that the fraction of weights at either clipping boundary decreases from over 99.9% at the first step to 4.37% and 18.53% at step 100 for the 1.7B and 4B students, respectively. At step 100, the median pre-clipping magnitudes ∣γIt∣|\gamma I_t| are 0.0088 and 0.0391, whereas their 99th percentiles are 2.82 and 41.65, above the clipping threshold of 1. The mean absolute weights are 0.119 and 0.263. These measurements show that most weights lie inside the clipping interval by step 100, while projection continues to bound large values in the score distribution's upper tail.

We also clarify the weight learning rate in Appendix A.1 (lines 874–879) and the normalized batch objective in Appendix A.2 (lines 911–917 and 962). The normalized batch objective gives the weight hypergradient (λ/N)It(\lambda/N)I_t, where NN is the current actor batch's valid response-token count and comes from the token-mean loss normalization. We apply a fixed learning rate of 105N110^5N_1 to this hypergradient, with N1N_1 as the first batch's token count. With λ=0.1\lambda=0.1, the equivalent coefficient multiplying the displayed score ItI_t in Eq. (5) is

γIt=(105N1)λNIt=104N1NIt.\gamma I_t=(10^5N_1)\frac{\lambda}{N}I_t =10^4\frac{N_1}{N}I_t.

Thus, the dependence on the current token count NN comes from token-mean loss normalization, rather than an additional learning-rate schedule. The fixed reference N1N_1 cancels NN for the first batch and sets the first-batch score multiplier to 10410^4.

To assess sensitivity to the weight learning rate, we completed a sweep on Qwen3-1.7B (non-thinking) math. All four runs use matched-BF16 JVP, training seed 42, 100 updates, and the same AdamW student learning rate of 10−610^{-6}; only the weight learning rate is varied. Accuracy equally averages AIME24, AIME25, HMMT2025-Feb, and HMMT2025-Nov, with 16 responses per problem. The weight statistics below are averages over updates 51–100.

Weight-learning-rate multiplierAccuracy at update 100 (%)Mean absolute weightWeights at either clipping boundary (%)
0.1×0.1\times32.030.0752.77
0.3×0.3\times31.670.0933.26
1×1\times32.710.1668.06
3×3\times26.980.36327.92

RAD is relatively tolerant of the tested reductions in weight learning rate: reducing it to 0.1×0.1\times or 0.3×0.3\times lowers accuracy by 0.68 and 1.04 percentage points, respectively, whereas increasing it to 3×3\times lowers accuracy by 5.73 points. The default performs best among these settings.

For fixed scores and inactive weight clipping, changing the weight learning rate uniformly rescales the weighted student gradient. AdamW's moment normalization can approximately cancel a constant positive rescaling of the gradient history when its moment estimates scale correspondingly and its numerical stabilizer is negligible (AdamW, Algorithm 2). At larger weight learning rates, more token weights reach the clipping boundaries, changing their relative magnitudes rather than only their common scale, and therefore could decrease the performance. Consistent with this explanation, the saturation rate increases from 8.06% to 27.92% between 1×1\times and 3×3\times.

Q5. Does the gap between SGD and AdamW scoring persist after calibrating their score scales?

A5. Thanks for the suggestion! We have calibrated the weight learning rate of the SGD-lookahead ablation to match the score and weight scale of RAD, but the performance gap remains substantial. Calibrated SGD lookahead remains 11.09 percentage points below RAD. This shows the importance of AdamW lookahead and scoring in RAD.

MethodFour-benchmark mean accuracy at update 100 (%)
RAD32.71
Calibrated SGD lookahead21.61

Thank you again for your constructive feedback. Our response adds the Qwen3-4B-Instruct-2507 and Qwen3-8B pre-update comparison and two teacher-information ablations, providing evidence for the contributions of post-update lookahead and the teacher information. We also report score and weight dynamics, weight-learning-rate sensitivity, and a calibrated SGD-lookahead comparison.

We hope these additional experiments and analyses have addressed your concerns. Please let us know if any questions remain; we would be happy to provide further clarification before the discussion period ends.

Reviewer_SQcEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Reward-Aligned Distillation (RAD), a method that improves on-policy distillation by weighting token-level teacher supervision according to its predicted effect on task reward. RAD uses a bilevel formulation with a temporary optimizer update and reward gradients to estimate token importance, allowing signed weights that can reverse distillation gradients. Experiments on mathematical reasoning and code generation demonstrate improvements over OPD, GRPO, and other baselines at matched training-response budgets.

Critical strengths
  • RAD demonstrates strong empirical performance across mathematical reasoning and code generation, consistently outperforming strong baselines under both matched training-response budgets and comparable training times.
  • The proposed method provides a practical extension of gradient-alignment principles to token-level on-policy distillation. The ablation studies further support the importance of token-specific and signed weighting for improving performance.
Critical weaknesses
  • The necessity of post-update lookahead is not sufficiently established. In Table 2, using the pre-update reward gradient achieves higher accuracy than full RAD (34.32% vs. 32.71%), while the lookahead stage accounts for 47.2% of training time. It is therefore unclear whether the additional lookahead computation provides sufficient benefits to justify its cost.
Questions for authors
  • Under what conditions does post-update lookahead offer advantages over the pre-update reward gradient, particularly in terms of performance and computational efficiency?
4 comments

We thank the reviewer for the positive feedback, thoughtful review, and for appreciating the merits of the work! Revisions to the paper are highlighted in blue.


Q1. The necessity of post-update lookahead is not sufficiently established.

A1. Thank you for raising this question. We have extended the comparison between RAD and the pre-update gradient variant to Qwen3-4B-Instruct-2507 math in Section 5.4 of the revised paper (lines 436–438). The pre-update variant retains AdamW-aware scoring and independent fresh meta rollouts, but samples and evaluates the reward gradient at the current student rather than the temporarily updated student. On Qwen3-4B-Instruct-2507, pre-update gradient achieves 47.19% compared with 55.26% for RAD. This shows that pre-update gradient is not consistently as good as RAD: although it achieves comparable accuracy on Qwen3-1.7B (non-thinking) math, it performs substantially worse on Qwen3-4B-Instruct-2507.

We additionally evaluated the pre-update gradient ablation on Qwen3-8B (non-thinking). At the same budget of 100 training updates, RAD achieves 58.91% four-benchmark mean accuracy, compared with 55.99% for the pre-update variant, an improvement of 2.92 percentage points. Together with the 4B comparisons, these results indicate the benefit of post-update lookahead.

Regarding computational cost, the reported 47.2% covers the entire lookahead stage, including the temporary optimizer update, weight transfer, meta rollout generation, reward-gradient computation, and restoration (Appendix E.2, Table 13, lines 1479–1486). The pre-update variant still requires fresh meta rollouts and reward-gradient computation, as well as AdamW-aware scoring and JVP computation. Consequently, 47.2% should not be interpreted as the time saved by omitting the temporary update.

Thank you again for your constructive feedback. We extended the pre-update gradient comparison to Qwen3-4B-Instruct-2507 and Qwen3-8B, where RAD improves the four-benchmark mean accuracy by 8.07 and 2.92 percentage points, respectively. We also clarified that the reported 47.2% runtime covers the entire lookahead stage, including meta rollouts and reward-gradient computation, and therefore does not represent the cost of the temporary update alone.

We hope these additional experiments and clarifications have addressed your concerns. Please let us know if any questions remain; we would be happy to provide further clarification before the discussion period ends.

Thank you again for your time and constructive feedback. With about one day remaining in the discussion period, we would appreciate a response indicating whether our replies and additional experiments have addressed your concerns. If any substantive questions remain, please let us know so that we can provide further clarification before the discussion closes.

I appreciate the authors for their response to my comments. I will maintain my positive score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.