- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Reward-Guided KD (RG-KD), which scales the distillation loss by a weight derived from the teacher's expected reward (1 + R, Eq. 3). For sequential tasks it adds a trajectory-level variant (each teacher rollout weighted by its own return) and a step-level variant (a critic estimates the value of one teacher step followed by a student continuation). Experiments cover ADE20K, a maze, three Atari games, code (HumanEval/MBPP) and math (GSM8K/MATH), plus plug-in results with GKD and DistiLLM.
Critical strengths
- The problem is practical and the method is easy to adopt. How useful teacher supervision is varies across samples and with the student's ability, and weighting the distillation loss by task utility is a simple, reward-source-agnostic idea. RG-KD only adds a scalar weight on top of an existing divergence, so it composes with other methods. In Table 5b, adding RG-KD to GKD and DistiLLM improves all four numbers (GSM8K 78.70 → 80.36 and 77.56 → 79.08; MATH 67.42 → 68.42 and 66.26 → 66.46). The step-level critic is also a sensible design: it estimates value "under the student's policy, rather than the teacher's" (Sec. 3.3.2), which matches the deployment setting and makes distillation from student-generated rollouts possible.
- The evaluation is broad, and several results are notable. The experiments span vision, maze navigation, Atari control, code and math, and vary the divergence (KLD/JSD/SKL), the weighting granularity and the rollout source. With teacher rollouts, trajectory-level RG-KD improves over Standard KD for all three divergences on every task where it is evaluated (point estimates). Step-level weighting gives +2.27 to +3.11 points over Standard KD on GSM8K (Table 4). On Phoenix, Standard KD on teacher rollouts falls far below the initial student; switching to student rollouts and adding step-level weighting recovers and greatly improves performance (Table 2). The comparisons also go beyond weak baselines: TSD-KD and ASKD (Table 5a), reward-based selection and weighting (Appendix F), and a reward-source ablation (Appendix E).
- The paper is transparent about its approximations and limitations. The authors state that the step-wise decomposition is exact only for forward KL and "should be understood as a heuristic approximation" for JSD/SKL (Sec. 3.3.1, Appendix A). Sec. 3.3.3 lays out the trade-offs between trajectory-level and step-level weighting, and Appendix D discusses reward-model error, critic error and extra cost. The weight visualization in Appendix J also helps intuition.
Critical weaknesses
Post-rebuttal update: I am raising my rating to weak accept. The matched-rollout, shuffled-weight and entropy-weighting controls, together with the multi-seed results, address my main empirical concerns. The exact weight settings and full critic cost still need clarification, and the credit-assignment claims should be narrowed. On balance, the evidence supports a useful practical contribution.
Original assessment:
- The objective actually optimized is unclear. Eq. 3 uses
1 + R, but Sec. 4.1 subtracts "the student model's expected reward as a baseline," Appendix J says "the resulting advantage is then scaled by a constant factor," and Appendices B.5/B.6 use the critic value directly. In a divergence loss, subtracting a baseline changes the objective itself (it is not a pure variance-reduction step), and the paper does not say how negative weights are handled. Suggestion: give the exact weight rule per task (baseline, scaling, clipping/normalization, stop-gradient). - The step-level claims are overstated. Sec. 3.3.3 says step-level weighting "resolves" credit assignment and "accurately" weights each term. But a positive scalar on the full per-history KL still has the teacher distribution as its optimum; it reallocates emphasis across histories and cannot suppress low-value teacher actions within a history. Also, the abstract's claim that reward "has not been used to control the impact of individual teacher samples" should be reconciled with ADPA and AdvDistill, which the paper itself discusses. Suggestion: soften these claims, and add a shuffled-weight or mean-matched constant-weight control to show that the weights track utility.
- There are no repeated runs or error bars, and "consistently outperforms" (Abstract) has exceptions. Several gains are small: HumanEval 76.83 vs. 74.39 is about 4/164 problems. Some cases are tied or worse: Qbert step-level JSD/SKL 4425 < 4450 and 8050 < 8575; HumanEval step-level KLD 73.17 < 74.39; GSM8K trajectory-level KLD ties EA-KD at 77.64. Suggestion: report mean ± std over ≥3 training seeds for a representative math setting and for Atari.
- Attribution and cost are unclear. Table 7 lacks unweighted KD with the same four rollouts, so the gain may come from more data rather than weighting. The step-level method also needs a reward model and a trained critic, but Appendix C only gives task-level run times. Suggestion: add same-rollout unweighted KD, a cheap per-token entropy-weighting control, and a per-method cost breakdown.
Minor Weaknesses:
- Table 5a vs. Appendix B.5: B.5 says "All methods are trained for one epoch" with student-generated responses, yet the GSM8K result (80.52) matches the teacher-rollout, two-epoch entry in Table 4, and the MATH result (68.14) appears in neither Table 4 nor Table 10.
- The data protocol is incomplete: dataset versions, validation splits, and whether the vision losses include ground-truth labels (the labels are used for the reward in B.1/Appendix G) are not stated.
- The LLM experiments use only the Qwen2.5 family (7B → 1.5B).
Questions for authors
- What exact weight and update rule is used in each task, and how are negative weights handled?
- Which configuration produced Table 5a?
- Do the Standard KD / EA-KD baselines receive the same ground-truth supervision as RG-KD's reward?
7 comments
Response to Reviewer eVy4 (2/3)
W3
There are no repeated runs or error bars, and "consistently outperforms" (Abstract) has exceptions. Several gains are small: HumanEval 76.83 vs. 74.39 is about 4/164 problems. Some cases are tied or worse: Qbert step-level JSD/SKL 4425 < 4450 and 8050 < 8575; HumanEval step-level KLD 73.17 < 74.39; GSM8K trajectory-level KLD ties EA-KD at 77.64. Suggestion: report mean ± std over ≥3 training seeds for a representative math setting and for Atari.
We conduct additional experiments using three different training seeds on GSM8K and Atari (Qbert) with teacher-generated trajectories, reporting the mean and standard deviation across KLD, JSD, and SKL, as shown in the tables below. On GSM8K, the multi-seed results confirm the performance trend reported in Table 4, with step-level RG-KD achieving the highest mean accuracy across all three divergence objectives (80.39 ± 0.12, 80.24 ± 0.38, and 80.59 ± 0.20). On Qbert, the results also support the overall trend in Table 2, as both trajectory-level and step-level RG-KD outperform Standard KD in mean performance across all three divergence objectives. Notably, the step-level JSD and SKL settings that underperformed Standard KD in the original single-seed results achieve higher average scores over three seeds, although the larger standard deviations indicate greater variability across runs. These results further support the effectiveness of RG-KD, showing that the overall performance trends are consistent with those observed in the original experiments, with some settings exhibiting substantial variability across training seeds. We will report these multi-seed results, along with those from the remaining experiments, in the final manuscript and revise the abstract's “consistently outperforms” claim to more accurately reflect the observed performance.
| Method | KLD (GSM8K) | JSD (GSM8K) | SKL (GSM8K) | KLD (Qbert) | JSD (Qbert) | SKL (Qbert) |
|---|---|---|---|---|---|---|
| Standard KD | 77.43 ± 0.04 | 77.61 ± 0.31 | 77.71 ± 0.20 | 8,175.00 ± 347.31 | 5,616.67 ± 1,999.11 | 7,008.33 ± 2,234.16 |
| EA-KD | 77.48 ± 0.16 | 77.71 ± 0.08 | 77.61 ± 0.16 | 8,008.33 ± 62.92 | 6,816.67 ± 2,050.97 | 9,325.00 ± 2,295.51 |
| RG-KD (Trajectory-level) | 77.69 ± 0.09 | 78.17 ± 0.08 | 78.37 ± 0.35 | 10,558.33 ± 2,175.05 | 9,300.00 ± 2,338.27 | 9,233.33 ± 2,006.29 |
| RG-KD (Step-level) | 80.39 ± 0.12 | 80.24 ± 0.38 | 80.59 ± 0.20 | 10,316.67 ± 1,978.85 | 8,233.33 ± 3,649.51 | 9,550.00 ± 2,159.43 |
W4
Attribution and cost are unclear. Table 7 lacks unweighted KD with the same four rollouts, so the gain may come from more data rather than weighting. The step-level method also needs a reward model and a trained critic, but Appendix C only gives task-level run times. Suggestion: add same-rollout unweighted KD, a cheap per-token entropy-weighting control, and a per-method cost breakdown.
Following the reviewer's suggestion, we conduct an additional unweighted Standard KD experiment using the same four rollouts as in Table 7. As shown in the table below, trajectory-level RG-KD outperforms Standard KD on GSM8K (80.06 vs. 78.54), demonstrating that the performance gains are not solely attributable to using more rollouts.
| Method | GSM8K |
|---|---|
| AdvDistill | 76.27 |
| Reward-based Resampling | 77.33 |
| Standard KD | 78.54 |
| RG-KD (Trajectory-level) | 80.06 |
We also conduct an additional experiment for the requested per-token entropy-weighting control by extending EA-KD to step-level weighting using same one teacher-generated rollout as in Table 4. As shown in the table below, step-level RG-KD achieves 80.52 on GSM8K, outperforming both step-level EA-KD (77.79) and Standard KD (77.41). These results further support the benefit of critic-based utility estimation over entropy-based timestep weighting.
| Method | GSM8K |
|---|---|
| Standard KD | 77.41 |
| EA-KD (Step-level) | 77.79 |
| RG-KD (Step-level) | 80.52 |
Regarding computational cost, we measure the cost of each method on GSM8K, distilling Qwen2.5-Math-7B-Instruct into Qwen2.5-Math-1.5B-Instruct, as shown in the table below. Compared with Standard KD and EA-KD, trajectory-level RG-KD maintains comparable training time, with increased memory usage due to the additional 8B reward model. Step-level RG-KD requires additional computation for value estimation using a 1.5B critic and continuous online critic updates based on reward-model feedback. As in critic-based reinforcement learning for LLMs, this overhead naturally arises from critic-based value estimation and online critic training.
| Method | Training Time | Peak Allocated GPU Memory |
|---|---|---|
| Standard KD | 15.93h | 48.67 GiB |
| EA-KD | 15.95h | 49.91 GiB |
| RG-KD (Trajectory-level) | 16.12h | 62.65 GiB |
| RG-KD (Step-level) | 41.55h | 64.14 GiB |
Response to Reviewer eVy4 (3/3)
Minor W1 & Q2
Table 5a vs. Appendix B.5: B.5 says "All methods are trained for one epoch" with student-generated responses, yet the GSM8K result (80.52) matches the teacher-rollout, two-epoch entry in Table 4, and the MATH result (68.14) appears in neither Table 4 nor Table 10. Which configuration produced Table 5a?
To clarify the experimental configurations, Table 5a reports the performance of step-level RG-KD trained for one epoch using student-generated responses and SKL. In contrast, Table 4 reports results obtained using teacher-generated responses with three divergence objectives (KLD, JSD, and SKL) over two training epochs. Table 10 further extends Table 4 by reporting results for both teacher-generated and student-generated responses under the same three divergence objectives, all trained for two epochs. Accordingly, the GSM8K accuracy of 80.52 in Table 5a was obtained by training step-level RG-KD for one epoch using student-generated responses and SKL, whereas the identical accuracy in Table 4 was obtained by training step-level RG-KD for two epochs using teacher-generated responses and KLD. Thus, the matching values are purely coincidental and do not indicate a reporting error. Therefore, the MATH accuracy of 68.14 reported in Table 5a cannot be found in either Table 4 or Table 10 because they report results from different experimental configurations. We will describe the experimental settings more clearly in the final manuscript.
Minor W2-1
The data protocol is incomplete: dataset versions, validation splits.
The table below summarizes the dataset versions and validation splits used across tasks. For datasets without a separate validation split, we follow common practice by training all methods for a fixed number of epochs or iterations within each task and evaluating their performance after training. We will include these details in the final version of the manuscript.
| Task | Dataset (Version) | Validation Split |
|---|---|---|
| Image Classification | CIFAR-100 (official) | None |
| Semantic Segmentation | ADE20K-150 (ADEChallengeData2016) | Official validation set |
| Maze | maze-dataset==1.4.2 | 2,500 samples |
| Atari | Qbert (QbertNoFrameskip-v4), Amidar (AmidarNoFrameskip-v4), Phoenix (PhoenixNoFrameskip-v4) | 50 validation seeds |
| Code Generation | CodeAlpaca-20K (sahil2801/CodeAlpaca-20k) | None |
| Mathematical Reasoning | GSM8K (openai/gsm8k, main), MATH (EleutherAI/hendrycks_math, all seven subjects) | None |
Minor W2-2 & Q3
Whether the vision losses include ground-truth labels (the labels are used for the reward in B.1/Appendix G) are not stated. Do the Standard KD / EA-KD baselines receive the same ground-truth supervision as RG-KD's reward?
Only for image classification and semantic segmentation, all methods use the same ground-truth labels to compute a cross-entropy loss alongside the distillation loss. No such ground-truth cross-entropy loss is used in the other tasks.
Minor W3
The LLM experiments use only the Qwen2.5 family (7B → 1.5B).
We are currently conducting additional LLM experiments with other model families and will share the results as soon as they become available.
Thank you again for your valuable feedback. We hope our responses have addressed the concerns discussed above and would be happy to provide further clarification if needed.
Response to Reviewer eVy4 (1/3)
We thank you for recognizing the practical significance of RG-KD, the breadth of our experimental evaluation. We also appreciate the detailed and constructive feedback. We address your thoughtful concerns below.
W1 & Q1
Thank you for pointing out the need to clarify the weighting rule used in practice. We agree that subtracting a baseline is not a pure variance-reduction step. We regard it as part of reward design, where scaling and shifting rewards is common practice. In all experiments, the weight is w=1+C⋅R, where C is a task-specific scaling hyperparameter and R is the reward signal. We acknowledge that the descriptions in Sec. 4.1, Appendix J, and Appendices B.5/B.6 were inconsistent; the baseline subtraction described there is part of how R is defined, not a separate step applied on top of Eq. 3. Specifically, R is measured relative to an exponential moving average (EMA) of the reward for trajectory-level RG-KD and relative to the critic-estimated value of a student-sampled action for step-level RG-KD. We will revise these sections to describe the weighting rule consistently, and add to the appendix a table listing all hyperparameters, including the value of C for each task, together with a description of the exact weighting rule used for each task.
While we have not included any explicit handling of negative weights in the formulation, no negative weights occurred in all our experiments.
Finally, we examine the effect of the EMA baseline in trajectory-level RG-KD on GSM8K and MATH (table below). The variant with the EMA baseline performs slightly better in five of the six settings. These results demonstrate that RG-KD is effective regardless of whether a baseline is used.
W2
We thank the reviewer for this constructive feedback. We will revise Sec. 3.3.3 to clarify that step-level RG-KD adaptively allocates distillation emphasis across timesteps based on estimated downstream utility, while preserving the teacher distribution. We will also revise the abstract to more precisely articulate the distinctions between RG-KD and prior work. Following the reviewer's suggestion, we conduct an additional shuffled-weight experiment, where the step-level weights are shuffled across timesteps while preserving their overall distribution. As shown in the table below, step-level RG-KD consistently outperforms its shuffled-weight counterpart on GSM8K across all three divergence objectives. These results suggest that step-level RG-KD becomes more effective when estimated utility weights are aligned with the corresponding teacher supervision at each individual timestep, supporting its ability to appropriately allocate distillation emphasis across timesteps.