- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Reward-Aligned Distillation (RAD), a token-level reweighting framework for on-policy distillation (OPD) under verifiable rewards. Observing that a stronger teacher's token preferences do not uniformly improve task reward, RAD formulates token weights via a one-step bilevel approximation (Section 4.1–4.2, Eqs. 1–5): starting from constant initial weights , it takes a temporary optimizer step , generates fresh meta rollouts from on a separate meta batch to estimate the post-update reward gradient , and computes all token-level hypergradient scores simultaneously via an optimizer-aware forward-mode Jacobian–vector product that differentiates through AdamW moments and global gradient clipping (Section 4.3–4.4, Eqs. 6–7, Algorithm 1, Appendix G Eqs. 11–13). The updated weights are bounded and signed—allowing RAD to amplify, suppress, or reverse teacher gradients—before restoring and committing the weighted distillation update.
Empirically, RAD is evaluated at a matched budget of 102,400 generated training responses (100 steps of RAD with 512 actor + 512 meta responses/step vs. 200 steps of baselines with 512 responses/step) across six competition-math benchmarks (AIME24/25/26, HMMT2025-Feb/Nov, OlympiadBench, Table 1), LiveCodeBench-v6 (Figure 3, Appendix C Table 8), and a preliminary SWE-bench Verified setup (Appendix D Figure 9), using Qwen3-1.7B (non-thinking), Qwen3-4B-Instruct-2507, and OLMo3-7B-Instruct students. While the signed, optimizer-conditioned alignment formulation and custom JVP engineering (Appendix H, Algorithm 2) are appealing, I lean toward weak rejection because (1) the paper's central post-update bilevel lookahead claim is contradicted by its own operating regime and Table 2 ablations, and (2) the experimental design mixes checkpoint generations across scales (Qwen3-1.7B (non-thinking) vs. Qwen3-4B-Instruct-2507), never evaluates Qwen3-8B (or any 7B/8B model on math, or Qwen3-4B on coding), relies on a single training seed (seed = 42), and sees several distillation baselines degrade below the untuned Base checkpoint on larger models.
Critical strengths
1. Optimizer-aware, signed token reweighting with solid JVP systems engineering.
Unlike uncertainty heuristics or nonnegative token selection (e.g., TIP, RLAD), RAD allows negative weights to actively push the student away from teacher preferences that anti-align with the AdamW-preconditioned reward update direction (Eq. 6, Appendix G Eq. 12). Table 2 confirms that both signed weights (32.71% vs. 25.00% for Positive only) and differentiating through AdamW (32.71% vs. 16.88% for SGD lookahead) are critical. The custom Triton forward-mode attention JVP kernel (Appendix H, Eq. 14, Algorithm 2) and olmo3_layer_streamed FSDP backend make exact per-token JVP computation practical at 32k context lengths.
2. Strong gains on Qwen3-1.7B and code generation across two model families, plus compatibility with .
At matched generated-response counts (102,400 responses), RAD delivers large improvements on Qwen3-1.7B (non-thinking) for both competition math (39.07% vs. 32.59% for ExOPD, Table 1) and LiveCodeBench-v6 (54.69% vs. 45.97% for RLAD, Table 8), improves OLMo3-7B-Instruct on LiveCodeBench-v6 (58.67% vs. 52.27% for GRPO), and stacks cleanly on top of (+2.66 and +6.51 points in Table 4).
Critical weaknesses
1. The central post-update bilevel lookahead mechanism is not empirically justified over pre-update AdamW gradient alignment.
The paper's headline methodological narrative is that RAD estimates "how each token's distillation signal affects the student's post-update reward" via a temporary distillation step . However, because the main experiments set , the temporary step ignores the current batch's distillation gradient entirely, and in the only ablation that isolates the post-update step (Table 2, Qwen3-1.7B), the Pre-update gradient variant ( without the temporary update) beats full RAD (34.32% vs. 32.71%). In Section 5.5 (Table 3), the authors compare full RAD against RAD-Lite—which simultaneously removes three things (the temporary lookahead step, the AdamW optimizer Jacobian, and separate meta rollouts)—finding that RAD-Lite wins on Qwen3-1.7B (37.50% vs. 32.71%) but collapses on Qwen3-4B-Instruct-2507 (37.29% vs. 55.26%). Crucially, because RAD-Lite conflates dropping the lookahead step with dropping the AdamW Jacobian and fresh meta rollouts, Table 3 does not show that the post-update lookahead step caused the 4B gap—especially since Table 2 already showed that dropping the AdamW Jacobian (SGD lookahead, -15.83 pp) or reusing rollouts (Pre-update gradient + rollout reuse, -15.78 pp) causes massive degradation on its own. Without testing Pre-update gradient (retaining the AdamW Jacobian and fresh meta rollouts) on Qwen3-4B-Instruct-2507 and coding, the paper fails to justify a lookahead step that consumes 47.2% of each training step's wall-clock time (Appendix Table 12).
2. Cross-generation student mismatch, omission of Qwen3-8B (and incomplete model–task matrix), single-seed training (seed = 42), and degraded distillation baselines.
- Cross-generation student mismatch (
Qwen3-1.7B (non-thinking)vs.Qwen3-4B-Instruct-2507) and omission ofQwen3-8B: In Table 1, the1.7Bstudent is the original May 2025Qwen3-1.7B (non-thinking)checkpoint (which starts at a very weak13.39%6-benchmark /8.23%4-benchmark base accuracy) distilled from the updated July 2025Qwen3-30B-A3B-Instruct-2507teacher (65.18%), yielding a large+14.56pp gain overGRPO. At4B, however, the student switches to the already heavily post-trained July 2025Qwen3-4B-Instruct-2507checkpoint (52.38%base), whereTIP,RLAD,ExOPD, andRAD-Liteall fall belowBaseand RAD's macro-average gain overGRPOdrops to+2.51pp. Moreover, within the sameQwen3 (non-thinking)series where1.7B,4B, and8B(Qwen3-8B) are all publicly available, the paper never evaluatesQwen3-8B(nor does it evaluateOLMo3-7B-Instructon math, orQwen3-4B-Instruct-2507on coding). Furthermore, Appendix A.2 (p. 18) and Appendix H.1 (pp. 30–31) reveal that theflash_tritonJVP backend used forQwen3models must materialize a full temporary student replica and direction vector in GPU memory (whereas scaling to 7B required writing a customolmo3_layer_streamedFSDP backend that still took101.61–166.79hours per run in Table 11 and forced omittingExOPDand ). This leaves open whether RAD can scale toQwen3-8Bboth computationally and in performance overGRPO. - Single-seed variance and small micro-averaged margins on
Qwen3-4B-Instruct-2507: Every method is trained only once (seed = 42, Section 6 p. 10 & Table 6); the reported standard deviations reflect only evaluation sampling from a single checkpoint. Because five of the six math benchmarks (AIME24/25/26,HMMT2025-Feb/Nov) contain only 30 problems each whileOlympiadBenchcontains 675 problems (Appendix Table 5, where RAD is70.39%vs.GRPO69.69%, a+0.70pp difference), weighting by problem count across all 825 problems reduces RAD's micro-averaged edge overGRPOonQwen3-4Bto roughly+1.23percentage points. - Distillation baselines degrading below
Base: OnQwen3-4B-Instruct-2507(Table 1,Base52.38%),TIP(51.93%),RLAD(50.87%), andExOPD(51.49%) all score below the untunedBasemodel; onOLMo3-7B-Instruct(Table 8,Base43.91%),TIPdrops to36.73%. Relying on untuned default hyperparameters from prior papers makes the comparison against competing distillation selectors less convincing.
Questions for authors
-
Assessment-critical (Isolating post-update lookahead vs. pre-update AdamW alignment on
Qwen3-4Band coding):
In Table 2 (Qwen3-1.7B), thePre-update gradientablation (which keeps the AdamW Jacobian and separate meta rollouts, but skips the temporary update and uses ) achieves34.32%, beating full RAD (32.71%) while avoiding the temporary update step. What happens when you run this exactPre-update gradientvariant (with AdamW Jacobian + fresh meta rollouts) onQwen3-4B-Instruct-2507math and onLiveCodeBench-v6? Does the post-update lookahead step provide any actual gain over pre-update AdamW gradient alignment when ? -
Assessment-critical (Why was
Qwen3-8Bnot trained, and how does RAD scale across matched-generationQwen3models?):- In Table 1,
1.7Buses the May 2025Qwen3-1.7B (non-thinking)checkpoint (13.39%base) whereas4Bswitches to the July 2025Qwen3-4B-Instruct-2507checkpoint (52.38%base), where RAD's margin overGRPOnarrows from+14.56pp to+2.51pp (+1.23pp micro-average). Since theQwen3 (non-thinking)series includes1.7B,4B, and8B(Qwen3-8B) from the same release generation, why wasQwen3-8Bnever trained or evaluated? - Specifically, was
Qwen3-8Bomitted because theflash_tritonJVP backend used for Qwen models (which materializes a full student replica and parameter direction in GPU memory, Appendix A.2 p. 18) runs out of memory on 8 H100 GPUs at 8B scale, or because distillation gains saturate at 8B? Also, why wasQwen3-4B-Instruct-2507omitted from the coding experiments (Table 8) andOLMo3-7B-Instructomitted from math?
- In Table 1,
-
Assessment-critical (Multi-seed stability and baseline degradation below
Base):- Can you provide at least 2–3 independent training seeds for the
Qwen3-1.7BandQwen3-4B-Instruct-2507math settings to verify that the+2.51pp macro-average (+1.23pp micro-average) margin overGRPOonQwen3-4Bexceeds training-seed noise? - Why do
TIP,RLAD, andExOPDall degrade below the untunedBasecheckpoint (52.38%) onQwen3-4B-Instruct-2507in Table 1 (andTIPby-7.18pp onOLMo3-7B-Instructin Table 8), and do they recover when their distillation weight / clipping hyperparameters are tuned?
- Can you provide at least 2–3 independent training seeds for the
4 comments
Q3. Single-seed variance and small micro-averaged margins on Qwen3-4B-Instruct-2507.
A3. We have now completed Qwen3-1.7B (non-thinking) RAD experiments with training seeds 42, 43, and 44. We compare RAD at 100 training updates with GRPO, OPD, and ExOPD at 200 updates on the same four benchmarks. Each run is evaluated with 16 responses per problem, and its accuracy equally averages AIME24, AIME25, HMMT2025-Feb, and HMMT2025-Nov.
| Method | Training seed | Updates | Four-benchmark mean accuracy (%) |
|---|---|---|---|
| GRPO | 42 | 200 | 18.75 |
| OPD | 42 | 200 | 25.42 |
| ExOPD | 42 | 200 | 26.15 |
| RAD | 42 | 100 | 34.69 |
| RAD | 43 | 100 | 30.68 |
| RAD | 44 | 100 | 34.53 |
| RAD mean ± SD across training seeds | 42, 43, 44 | 100 | 33.30 ± 2.27 |
The three-seed RAD mean exceeds GRPO by 14.55 percentage points, OPD by 7.88 points, and ExOPD by 7.15 points. All three RAD runs exceed the reported score of ExOPD, the strongest external baseline on these four benchmarks.
We report macro-averaged accuracy, the arithmetic mean of the six benchmark accuracies, so that each benchmark contributes equally (Table 1 caption, lines 270–273). In contrast, micro-averaging weights each problem equally: OlympiadBench accounts for 675 of the 825 problems (Table 5, lines 927–937) and would therefore receive 81.8% of the total weight. Both aggregations are meaningful but answer different evaluation questions. We use macro-averaging as our primary aggregate to measure performance across the selected benchmarks without letting their sizes determine their relative importance.
Equal task or benchmark weighting also has established precedents in LLM distillation. GKD (Agarwal et al., ICLR 2024), Section 4.4 and Figure 10 explicitly uses an "unweighted average over all tasks" when reporting accuracy within each of the MMLU and BBH benchmark suites. MiniLLM (Gu et al., ICLR 2024), Table 8 reports the average accuracy on SST2 and BoolQ. For mathematical reasoning, DistiLLM-2 (Ko et al., ICML 2025), Table 3 reports the arithmetic mean of GSM8K and MATH pass@1.
Q4. Distillation baselines degrading below Base.
A4. For the method-specific hyperparameters, we use a retention ratio of 0.5 for TIP, a mixing coefficient of 0.5 for RLAD, and an extrapolation coefficient of 1.25 for ExOPD (Table 7, lines 983–988), following values evaluated in their respective papers. Their papers include comparisons of token-retention ratios for TIP and coefficient-sensitivity analyses for RLAD and ExOPD. (TIP, Section 7.4; RLAD, Section 5.2.3; ExOPD, Section 4.1) We have tried to tune the hyperparameters for these methods, but they still degrade the student model's performance.
Distillation from a stronger teacher does not guarantee improved task accuracy over the initial student. Prior studies also report degradation under particular on-policy distillation configurations:
- On-Policy Delta Distillation (Heo et al., 2026), Tables 4–5, reports below-initial-model results for OPD and ExOPD on Qwen3 thinking models.
- REOPOLD (Ko et al., 2026), Section 5.1 and Figure 7, reports that vanilla reverse-KL OPD causes a sharp accuracy decline for the already strong DeepSeek-R1-Distill-Qwen-7B student and fails to improve beyond its initial performance on AIME24.
- The Many Faces of On-Policy Distillation (Zhu et al., 2026), Section 4.1 and Figure 4, reports that Qwen3-1.7B distilled from Qwen3-8B initially improves but collapses to nearly zero accuracy on MATH500, AIME24, and AIME25 by step 1,000 when using unnormalized Top-20 reverse KL.
These findings show that distillation can degrade performance below the initial student. Our Qwen3-4B-Instruct-2507 setting may face similar challenges to the strong-student settings studied in On-Policy Delta Distillation (Heo et al., 2026) and REOPOLD (Ko et al., 2026): the student already achieves accuracy close to that of the teacher, leaving limited room for improvement through distillation (Table 1, teacher at lines 277–278 and 4B base student at lines 287–289).
We additionally evaluated the pre-update gradient ablation on Qwen3-8B (non-thinking). At the same budget of 100 training updates, RAD achieves 58.91% four-benchmark mean accuracy, compared with 55.99% for the pre-update variant, an improvement of 2.92 percentage points. Together with the 4B comparisons, these results indicate the benefit of post-update lookahead.
Thank you again for your constructive feedback. Our response adds Qwen3-8B comparisons, three-seed Qwen3-1.7B experiments, and additional pre-update gradient ablations. We also clarify why AdamW can produce a temporary update at zero initial token weights, explain the runtime breakdown and memory requirements, and clarify our use of equally weighted benchmark accuracy. Finally, we discuss baseline hyperparameter choices and prior reports of distillation degrading student performance.
We hope these additional experiments and clarifications have addressed your concerns. Please let us know if any questions remain; we would be happy to provide further clarification before the discussion period ends.
Thank you again for your time and constructive feedback. With about one day remaining in the discussion period, we would appreciate a response indicating whether our replies and additional experiments have addressed your concerns. If any substantive questions remain, please let us know so that we can provide further clarification before the discussion closes.
We thank the reviewer for their comments. We believe there are some misunderstandings. Please allow us to address your concerns point-by-point. Revisions to the paper are highlighted in blue.
Q1. The central post-update bilevel lookahead mechanism is not empirically justified over pre-update AdamW gradient alignment.
A1. Thank you for raising this question. We have extended the comparison between RAD and the pre-update gradient variant to Qwen3-4B-Instruct-2507 math in Section 5.4 of the revised paper (lines 436–438). The pre-update variant retains AdamW-aware scoring and independent fresh meta rollouts, but samples and evaluates the reward gradient at the current student rather than the temporarily updated student. On Qwen3-4B-Instruct-2507, pre-update gradient achieves 47.19% compared with 55.26% for RAD. This shows that pre-update gradient is not consistently as good as RAD: although it achieves comparable accuracy on Qwen3-1.7B (non-thinking) math, it performs substantially worse on Qwen3-4B-Instruct-2507.
Regarding computational cost, the reported 47.2% covers the entire lookahead stage, including the temporary optimizer update, weight transfer, meta rollout generation, reward-gradient computation, and restoration (Appendix E.2, Table 13, lines 1479–1486). The pre-update variant still requires fresh meta rollouts and reward-gradient computation, as well as AdamW-aware scoring and JVP computation. Consequently, 47.2% should not be interpreted as the time saved by omitting the temporary update.
Setting c=0 makes the current batch's distillation gradient zero at the initial token weights. However, the temporary AdamW step can still change the student parameters through accumulated momentum and decoupled weight decay (Section 4.3, lines 250–252). Therefore, evaluating the reward gradient after this step need not be equivalent to evaluating it at the current student.
Q2. Cross-generation student mismatch, omission of Qwen3-8B.
A2. We use Qwen3-1.7B (non-thinking) and Qwen3-4B-Instruct-2507 for math tasks and OLMo3-7B-Instruct for coding instead of same-family and series models, e.g. Qwen3-4B and Qwen3-8B, because we want to show RAD works for different models instead of just a single model series. To address the reviewer’s concern, we compared RAD, OPD, and GRPO for Qwen3-8B (non-thinking) in Appendix A.5 of the revised paper (lines 1082–1089, Table 8, and Figure 7) and below. The experiments follow the same setups as Table 1.
Qwen3-8B (non-thinking) math accuracy (%). Bold and underline mark the best and second-best means within each block.
RAD at step 50 versus baselines at step 100
RAD at step 100 versus baselines at step 200
Our Qwen3-8B experiments address both concerns: the
flash_tritonJVP backend supports training at 8B scale without running out of memory in our setup, and the student continues to benefit from distillation at this scale.The full student replica is an implementation choice of our current Qwen3-1.7B experiments, rather than a requirement of RAD. RAD also supports layer-streamed JVP with parameter sharding, as used in our OLMo3-7B-Instruct experiments (Appendix A.2, lines 965–971; Appendix H.1, lines 1727–1737). This follows established sharding principles used by FSDP and ZeRO and provides a path to reducing JVP memory requirements for larger students.