acceptodds
Under review as a conference paper at ICLR 2027

Reward-Guided Knowledge Distillation

Abstract

The goal of knowledge distillation is to improve a student model’s performance using output samples of a teacher model. The performance can be measured, yet this explicit signal has not been used to control the impact of individual teacher samples during training. In this work, we establish Reward-Guided Knowledge Distillation (RG-KD), a unified and principled framework that weights the distillation loss according to the expected downstream utility of the teacher's predictive distribution. RG-KD further extends to sequential prediction through trajectory-level reward weighting and critic-based step-level value estimation. Extensive evaluations across five diverse tasks demonstrate that RG-KD consistently outperforms unweighted and entropy-weighted baselines as well as recent adaptive and selective distillation methods by effectively weighting teacher signals and aligning the student with high-reward outcomes. RG-KD also improves existing approaches through plug-and-play integration.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Knowledge Distillation, Reward Guidance, Sample-level Distillation, Trajectory-level Distillation, Step-level Distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_eVy4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Reward-Guided KD (RG-KD), which scales the distillation loss by a weight derived from the teacher's expected reward (1 + R, Eq. 3). For sequential tasks it adds a trajectory-level variant (each teacher rollout weighted by its own return) and a step-level variant (a critic estimates the value of one teacher step followed by a student continuation). Experiments cover ADE20K, a maze, three Atari games, code (HumanEval/MBPP) and math (GSM8K/MATH), plus plug-in results with GKD and DistiLLM.

Critical strengths
  1. The problem is practical and the method is easy to adopt. How useful teacher supervision is varies across samples and with the student's ability, and weighting the distillation loss by task utility is a simple, reward-source-agnostic idea. RG-KD only adds a scalar weight on top of an existing divergence, so it composes with other methods. In Table 5b, adding RG-KD to GKD and DistiLLM improves all four numbers (GSM8K 78.70 → 80.36 and 77.56 → 79.08; MATH 67.42 → 68.42 and 66.26 → 66.46). The step-level critic is also a sensible design: it estimates value "under the student's policy, rather than the teacher's" (Sec. 3.3.2), which matches the deployment setting and makes distillation from student-generated rollouts possible.
  2. The evaluation is broad, and several results are notable. The experiments span vision, maze navigation, Atari control, code and math, and vary the divergence (KLD/JSD/SKL), the weighting granularity and the rollout source. With teacher rollouts, trajectory-level RG-KD improves over Standard KD for all three divergences on every task where it is evaluated (point estimates). Step-level weighting gives +2.27 to +3.11 points over Standard KD on GSM8K (Table 4). On Phoenix, Standard KD on teacher rollouts falls far below the initial student; switching to student rollouts and adding step-level weighting recovers and greatly improves performance (Table 2). The comparisons also go beyond weak baselines: TSD-KD and ASKD (Table 5a), reward-based selection and weighting (Appendix F), and a reward-source ablation (Appendix E).
  3. The paper is transparent about its approximations and limitations. The authors state that the step-wise decomposition is exact only for forward KL and "should be understood as a heuristic approximation" for JSD/SKL (Sec. 3.3.1, Appendix A). Sec. 3.3.3 lays out the trade-offs between trajectory-level and step-level weighting, and Appendix D discusses reward-model error, critic error and extra cost. The weight visualization in Appendix J also helps intuition.
Critical weaknesses

Post-rebuttal update: I am raising my rating to weak accept. The matched-rollout, shuffled-weight and entropy-weighting controls, together with the multi-seed results, address my main empirical concerns. The exact weight settings and full critic cost still need clarification, and the credit-assignment claims should be narrowed. On balance, the evidence supports a useful practical contribution.

Original assessment:

  1. The objective actually optimized is unclear. Eq. 3 uses 1 + R, but Sec. 4.1 subtracts "the student model's expected reward as a baseline," Appendix J says "the resulting advantage is then scaled by a constant factor," and Appendices B.5/B.6 use the critic value directly. In a divergence loss, subtracting a baseline changes the objective itself (it is not a pure variance-reduction step), and the paper does not say how negative weights are handled. Suggestion: give the exact weight rule per task (baseline, scaling, clipping/normalization, stop-gradient).
  2. The step-level claims are overstated. Sec. 3.3.3 says step-level weighting "resolves" credit assignment and "accurately" weights each term. But a positive scalar on the full per-history KL still has the teacher distribution as its optimum; it reallocates emphasis across histories and cannot suppress low-value teacher actions within a history. Also, the abstract's claim that reward "has not been used to control the impact of individual teacher samples" should be reconciled with ADPA and AdvDistill, which the paper itself discusses. Suggestion: soften these claims, and add a shuffled-weight or mean-matched constant-weight control to show that the weights track utility.
  3. There are no repeated runs or error bars, and "consistently outperforms" (Abstract) has exceptions. Several gains are small: HumanEval 76.83 vs. 74.39 is about 4/164 problems. Some cases are tied or worse: Qbert step-level JSD/SKL 4425 < 4450 and 8050 < 8575; HumanEval step-level KLD 73.17 < 74.39; GSM8K trajectory-level KLD ties EA-KD at 77.64. Suggestion: report mean ± std over ≥3 training seeds for a representative math setting and for Atari.
  4. Attribution and cost are unclear. Table 7 lacks unweighted KD with the same four rollouts, so the gain may come from more data rather than weighting. The step-level method also needs a reward model and a trained critic, but Appendix C only gives task-level run times. Suggestion: add same-rollout unweighted KD, a cheap per-token entropy-weighting control, and a per-method cost breakdown.

Minor Weaknesses:

  1. Table 5a vs. Appendix B.5: B.5 says "All methods are trained for one epoch" with student-generated responses, yet the GSM8K result (80.52) matches the teacher-rollout, two-epoch entry in Table 4, and the MATH result (68.14) appears in neither Table 4 nor Table 10.
  2. The data protocol is incomplete: dataset versions, validation splits, and whether the vision losses include ground-truth labels (the labels are used for the reward in B.1/Appendix G) are not stated.
  3. The LLM experiments use only the Qwen2.5 family (7B → 1.5B).
Questions for authors
  1. What exact weight and update rule is used in each task, and how are negative weights handled?
  2. Which configuration produced Table 5a?
  3. Do the Standard KD / EA-KD baselines receive the same ground-truth supervision as RG-KD's reward?
7 comments
AuthorsOpenReview ↗

Response to Reviewer eVy4 (1/3)

We thank you for recognizing the practical significance of RG-KD, the breadth of our experimental evaluation. We also appreciate the detailed and constructive feedback. We address your thoughtful concerns below.


W1 & Q1

... In a divergence loss, subtracting a baseline changes the objective itself (it is not a pure variance-reduction step), and the paper does not say how negative weights are handled. Suggestion: give the exact weight rule per task (baseline, scaling, clipping/normalization, stop-gradient).

Thank you for pointing out the need to clarify the weighting rule used in practice. We agree that subtracting a baseline is not a pure variance-reduction step. We regard it as part of reward design, where scaling and shifting rewards is common practice. In all experiments, the weight is w=1+C⋅Rw = 1 + C·R, where CC is a task-specific scaling hyperparameter and RR is the reward signal. We acknowledge that the descriptions in Sec. 4.1, Appendix J, and Appendices B.5/B.6 were inconsistent; the baseline subtraction described there is part of how RR is defined, not a separate step applied on top of Eq. 3. Specifically, RR is measured relative to an exponential moving average (EMA) of the reward for trajectory-level RG-KD and relative to the critic-estimated value of a student-sampled action for step-level RG-KD. We will revise these sections to describe the weighting rule consistently, and add to the appendix a table listing all hyperparameters, including the value of CC for each task, together with a description of the exact weighting rule used for each task.

While we have not included any explicit handling of negative weights in the formulation, no negative weights occurred in all our experiments.

Finally, we examine the effect of the EMA baseline in trajectory-level RG-KD on GSM8K and MATH (table below). The variant with the EMA baseline performs slightly better in five of the six settings. These results demonstrate that RG-KD is effective regardless of whether a baseline is used.

MethodKLD (GSM8K)JSD (GSM8K)SKL (GSM8K)KLD (MATH)JSD (MATH)SKL (MATH)
Standard KD77.4177.5677.8666.0867.0066.42
RG-KD (w/o baseline)77.4877.7978.3266.4067.1866.88
RG-KD (w/ EMA baseline)77.6478.0978.1766.5667.2466.98

W2

The step-level claims are overstated. Sec. 3.3.3 says step-level weighting "resolves" credit assignment and "accurately" weights each term. ... Also, the abstract's claim that reward "has not been used to control the impact of individual teacher samples" should be reconciled with ADPA and AdvDistill, which the paper itself discusses. Suggestion: soften these claims, and add a shuffled-weight or mean-matched constant-weight control to show that the weights track utility.

We thank the reviewer for this constructive feedback. We will revise Sec. 3.3.3 to clarify that step-level RG-KD adaptively allocates distillation emphasis across timesteps based on estimated downstream utility, while preserving the teacher distribution. We will also revise the abstract to more precisely articulate the distinctions between RG-KD and prior work. Following the reviewer's suggestion, we conduct an additional shuffled-weight experiment, where the step-level weights are shuffled across timesteps while preserving their overall distribution. As shown in the table below, step-level RG-KD consistently outperforms its shuffled-weight counterpart on GSM8K across all three divergence objectives. These results suggest that step-level RG-KD becomes more effective when estimated utility weights are aligned with the corresponding teacher supervision at each individual timestep, supporting its ability to appropriately allocate distillation emphasis across timesteps.

MethodKLD (GSM8K)JSD (GSM8K)SKL (GSM8K)
Step-level RG-KD (shuffled weight)78.4779.4578.92
Step-level RG-KD80.5279.8380.67

AuthorsOpenReview ↗

Response to Reviewer eVy4 (2/3)

W3

There are no repeated runs or error bars, and "consistently outperforms" (Abstract) has exceptions. Several gains are small: HumanEval 76.83 vs. 74.39 is about 4/164 problems. Some cases are tied or worse: Qbert step-level JSD/SKL 4425 < 4450 and 8050 < 8575; HumanEval step-level KLD 73.17 < 74.39; GSM8K trajectory-level KLD ties EA-KD at 77.64. Suggestion: report mean ± std over ≥3 training seeds for a representative math setting and for Atari.

We conduct additional experiments using three different training seeds on GSM8K and Atari (Qbert) with teacher-generated trajectories, reporting the mean and standard deviation across KLD, JSD, and SKL, as shown in the tables below. On GSM8K, the multi-seed results confirm the performance trend reported in Table 4, with step-level RG-KD achieving the highest mean accuracy across all three divergence objectives (80.39 ± 0.12, 80.24 ± 0.38, and 80.59 ± 0.20). On Qbert, the results also support the overall trend in Table 2, as both trajectory-level and step-level RG-KD outperform Standard KD in mean performance across all three divergence objectives. Notably, the step-level JSD and SKL settings that underperformed Standard KD in the original single-seed results achieve higher average scores over three seeds, although the larger standard deviations indicate greater variability across runs. These results further support the effectiveness of RG-KD, showing that the overall performance trends are consistent with those observed in the original experiments, with some settings exhibiting substantial variability across training seeds. We will report these multi-seed results, along with those from the remaining experiments, in the final manuscript and revise the abstract's “consistently outperforms” claim to more accurately reflect the observed performance.

MethodKLD (GSM8K)JSD (GSM8K)SKL (GSM8K)KLD (Qbert)JSD (Qbert)SKL (Qbert)
Standard KD77.43 ± 0.0477.61 ± 0.3177.71 ± 0.208,175.00 ± 347.315,616.67 ± 1,999.117,008.33 ± 2,234.16
EA-KD77.48 ± 0.1677.71 ± 0.0877.61 ± 0.168,008.33 ± 62.926,816.67 ± 2,050.979,325.00 ± 2,295.51
RG-KD (Trajectory-level)77.69 ± 0.0978.17 ± 0.0878.37 ± 0.3510,558.33 ± 2,175.059,300.00 ± 2,338.279,233.33 ± 2,006.29
RG-KD (Step-level)80.39 ± 0.1280.24 ± 0.3880.59 ± 0.2010,316.67 ± 1,978.858,233.33 ± 3,649.519,550.00 ± 2,159.43

W4

Attribution and cost are unclear. Table 7 lacks unweighted KD with the same four rollouts, so the gain may come from more data rather than weighting. The step-level method also needs a reward model and a trained critic, but Appendix C only gives task-level run times. Suggestion: add same-rollout unweighted KD, a cheap per-token entropy-weighting control, and a per-method cost breakdown.

Following the reviewer's suggestion, we conduct an additional unweighted Standard KD experiment using the same four rollouts as in Table 7. As shown in the table below, trajectory-level RG-KD outperforms Standard KD on GSM8K (80.06 vs. 78.54), demonstrating that the performance gains are not solely attributable to using more rollouts.

MethodGSM8K
AdvDistill76.27
Reward-based Resampling77.33
Standard KD78.54
RG-KD (Trajectory-level)80.06

We also conduct an additional experiment for the requested per-token entropy-weighting control by extending EA-KD to step-level weighting using same one teacher-generated rollout as in Table 4. As shown in the table below, step-level RG-KD achieves 80.52 on GSM8K, outperforming both step-level EA-KD (77.79) and Standard KD (77.41). These results further support the benefit of critic-based utility estimation over entropy-based timestep weighting.

MethodGSM8K
Standard KD77.41
EA-KD (Step-level)77.79
RG-KD (Step-level)80.52

Regarding computational cost, we measure the cost of each method on GSM8K, distilling Qwen2.5-Math-7B-Instruct into Qwen2.5-Math-1.5B-Instruct, as shown in the table below. Compared with Standard KD and EA-KD, trajectory-level RG-KD maintains comparable training time, with increased memory usage due to the additional 8B reward model. Step-level RG-KD requires additional computation for value estimation using a 1.5B critic and continuous online critic updates based on reward-model feedback. As in critic-based reinforcement learning for LLMs, this overhead naturally arises from critic-based value estimation and online critic training.

MethodTraining TimePeak Allocated GPU Memory
Standard KD15.93h48.67 GiB
EA-KD15.95h49.91 GiB
RG-KD (Trajectory-level)16.12h62.65 GiB
RG-KD (Step-level)41.55h64.14 GiB

AuthorsOpenReview ↗

Response to Reviewer eVy4 (3/3)

Minor W1 & Q2

Table 5a vs. Appendix B.5: B.5 says "All methods are trained for one epoch" with student-generated responses, yet the GSM8K result (80.52) matches the teacher-rollout, two-epoch entry in Table 4, and the MATH result (68.14) appears in neither Table 4 nor Table 10. Which configuration produced Table 5a?

To clarify the experimental configurations, Table 5a reports the performance of step-level RG-KD trained for one epoch using student-generated responses and SKL. In contrast, Table 4 reports results obtained using teacher-generated responses with three divergence objectives (KLD, JSD, and SKL) over two training epochs. Table 10 further extends Table 4 by reporting results for both teacher-generated and student-generated responses under the same three divergence objectives, all trained for two epochs. Accordingly, the GSM8K accuracy of 80.52 in Table 5a was obtained by training step-level RG-KD for one epoch using student-generated responses and SKL, whereas the identical accuracy in Table 4 was obtained by training step-level RG-KD for two epochs using teacher-generated responses and KLD. Thus, the matching values are purely coincidental and do not indicate a reporting error. Therefore, the MATH accuracy of 68.14 reported in Table 5a cannot be found in either Table 4 or Table 10 because they report results from different experimental configurations. We will describe the experimental settings more clearly in the final manuscript.


Minor W2-1

The data protocol is incomplete: dataset versions, validation splits.

The table below summarizes the dataset versions and validation splits used across tasks. For datasets without a separate validation split, we follow common practice by training all methods for a fixed number of epochs or iterations within each task and evaluating their performance after training. We will include these details in the final version of the manuscript.

TaskDataset (Version)Validation Split
Image ClassificationCIFAR-100 (official)None
Semantic SegmentationADE20K-150 (ADEChallengeData2016)Official validation set
Mazemaze-dataset==1.4.22,500 samples
AtariQbert (QbertNoFrameskip-v4), Amidar (AmidarNoFrameskip-v4), Phoenix (PhoenixNoFrameskip-v4)50 validation seeds
Code GenerationCodeAlpaca-20K (sahil2801/CodeAlpaca-20k)None
Mathematical ReasoningGSM8K (openai/gsm8k, main), MATH (EleutherAI/hendrycks_math, all seven subjects)None

Minor W2-2 & Q3

Whether the vision losses include ground-truth labels (the labels are used for the reward in B.1/Appendix G) are not stated. Do the Standard KD / EA-KD baselines receive the same ground-truth supervision as RG-KD's reward?

Only for image classification and semantic segmentation, all methods use the same ground-truth labels to compute a cross-entropy loss alongside the distillation loss. No such ground-truth cross-entropy loss is used in the other tasks.


Minor W3

The LLM experiments use only the Qwen2.5 family (7B → 1.5B).

We are currently conducting additional LLM experiments with other model families and will share the results as soon as they become available.


Thank you again for your valuable feedback. We hope our responses have addressed the concerns discussed above and would be happy to provide further clarification if needed.

Reviewer_ocztOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Title: Reward-Guided Knowledge Distillation(RG-KD)

Conventional knowledge distillation methods use heuristic signals or predefined schedules to control the influence of teacher supervision during student training. However, the teacher’s supervision does not provide useful guidance for every data sample, action, or state. This creates a gap between learning by imitating the teacher’s predictive distribution and improving downstream task performance. RG-KD addresses this issue by incorporating reward-guidance into the distillation objective scale, allowing the strength of imitation to be adjusted according to the downstream utility of the teacher’s predictive distribution for each sample. The framework is further extended to sequential prediction through trajectory-level and step-level variants, and the paper discusses the trade-offs between these two approaches. Experiments across five diverse tasks demonstrate the effectiveness of RG-KD. The paper also shows its applicability and flexibility through plug-and-play integration with existing distillation approaches.

Critical strengths
  1. Rather than relying on the heuristic signals or predefined schedules used by existing KD baselines to modulate teacher supervision, RG-KD directly incorporates task rewards into the downstream objective. This provides a more practical way of controlling the strength of teacher supervision. The paper further extends this idea to sequential prediction by introducing both trajectory-level and step-level variants, whose effectiveness is supported by the experimental results.
  2. The paper provides an extensive evaluation across diverse downstream tasks, including semantic segmentation, maze reasoning, Atari, code generation, and mathematical reasoning. These experiments cover both non-sequential and sequential prediction settings, providing broad evidence for the general applicability of the proposed framework.
  3. The proposed framework demonstrates substantial flexibility by supporting different types of task-specific reward signals and KD divergence objectives. In particular, its successful integration with existing approaches such as GKD and DistiLLM highlights its potential as a practical plug-and-play component for improving existing distillation pipelines.
Critical weaknesses
  1. The RG-KD uses the reward associated with the teacher’s output as a proxy for the usefulness of teacher supervision. However, as discussed in standard KD (Hinton et al., 2015), soft teacher predictions can transfer class relationships, often called dark knowledge. Therefore, even when the teacher’s prediction is incorrect or receives a low reward, its predictive distribution may still provide useful relational information to the student. The teacher distribution may not be equally beneficial as a distillation target, but RG-KD relies on the positively rewarded teacher predictions. However, the step-level RG-KD mitigates this issue by estimating rewards under the student policy. A comparison with heuristic-based adaptive KD methods, including CA-MKD and CTKD, would strengthen the paper.

  2. As suggested by the ablation study in Appendix E, the proposed framework supports different sources of reward. However, RG-KD has the potential to be adapted more a wider variety of KD divergence methods, such as intermediate-representation distillation or inter-sample relational distillation. Evaluating these settings would provide stronger evidence for the claimed generality and plug-and-play applicability of the proposed framework.

Questions for authors
  1. As mentioned in the Critical Weaknesses section, could the authors provide comparisons with heuristic-based adaptive KD methods, including CA-MKD and CTKD? These comparisons would help establish the generalizability and effectiveness of RG-KD

  2. Although Appendix E demonstrates compatibility with different reward sources, the current experiments primarily consider predictive-distribution-based distillation objectives. Can RG-KD also be applied to intermediate-representation distillation or inter-sample relational distillation? I believe that experiments in these settings would further strengthen the paper.

3 comments
AuthorsOpenReview ↗

Response to Reviewer oczt

We sincerely thank you for recognizing the effectiveness of reward-guided distillation and the flexibility of RG-KD as a plug-and-play framework. We also appreciate the valuable suggestions, which help us further strengthen our work.


W1 & Q1

The RG-KD uses the reward associated with the teacher’s output as a proxy for the usefulness of teacher supervision. However, even when the teacher’s prediction is incorrect or receives a low reward, its predictive distribution may still provide useful dark knowledge. Could the authors provide comparisons with heuristic-based adaptive KD methods, including CA-MKD and CTKD, to further establish the generalizability and effectiveness of RG-KD?

We thank you for this insightful comment. A capacity-constrained student cannot necessarily match all teacher distributions perfectly, and the ultimate goal of KD is generally to improve the student's downstream task performance. Motivated by this, RG-KD evaluates the teacher's predictive distribution using task rewards and adjusts distillation strength accordingly, enabling the student to prioritize supervision with higher expected downstream utility. Furthermore, we agree that a low reward does not necessarily indicate that a teacher distribution lacks useful dark knowledge. In practice, all distillation weights remained positive in our experiments, without completely suppressing distillation from low-reward teacher predictions. This allows RG-KD to emphasize high-utility teacher supervision without discarding potentially useful knowledge from low-reward predictions.

Regarding the requested comparisons, we conduct additional experiments comparing RG-KD with CTKD [1] on CIFAR-100. We implement CTKD with the Global-T configuration and train all methods for 240 epochs under the same experimental settings to ensure a fair comparison. Note that CA-MKD [2] is not directly applicable to a single-teacher setting, as its weighting formulation is specifically designed for multi-teacher distillation. As shown in the table below, RG-KD achieves the highest accuracy across all three divergence objectives, outperforming CTKD, EA-KD, and Standard KD. In contrast, CTKD shows only a marginal improvement over Standard KD under KLD, with no improvement under JSD or SKL. These additional comparisons strengthen the empirical evaluation of RG-KD against heuristic-based adaptive KD methods, further supporting the effectiveness of reward-guided weighting. We will include these results in the final manuscript.

MethodKLD (CIFAR-100)JSD (CIFAR-100)SKL (CIFAR-100)
Standard KD73.3472.5973.55
CTKD73.4172.4473.06
EA-KD74.7272.5974.41
RG-KD (Trajectory-level)75.1272.6674.74

[1] Li et. al., Curriculum Temperature for Knowledge Distillation, AAAI 2023

[2] Zhang et. al., Confidence-Aware Multi-Teacher Knowledge Distillation, ICASSP 2022


W2 & Q2

The current experiments primarily consider predictive-distribution-based distillation objectives. Can RG-KD also be applied to intermediate-representation distillation or inter-sample relational distillation?

We thank you for this valuable suggestion. To further evaluate the generality and plug-and-play applicability of RG-KD beyond predictive-distribution-based distillation, we conduct additional experiments by integrating RG-KD into ReviewKD [1], an intermediate-representation distillation method. Specifically, we train on CIFAR-100 for 240 epochs and compare the original ReviewKD with its RG-KD-weighted variant (RG-ReviewKD). As shown in the table below, RG-ReviewKD outperforms the original ReviewKD, achieving higher accuracy on CIFAR-100. These results further strengthen the empirical support for RG-KD as a general and plug-and-play distillation framework. We will include these results in the final manuscript.

MethodAcc. (CIFAR-100)
ReviewKD74.95
RG-ReviewKD75.44

[1] Chen et. al., Distilling Knowledge via Knowledge Review, CVPR 2021


Thank you again for your valuable feedback. We hope our responses have addressed all your concerns and would be happy to provide any further information if needed.

Thank you for addressing all the questions I raised. Based on your responses, I have updated my comment. I appreciate your efforts.

We sincerely thank you for your thoughtful comments and for raising your rating. We greatly appreciate you for recognizing the value of our work.

Reviewer_hEw5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Reward-Guided Knowledge Distillation (RG-KD), which uses task rewards to weight the divergence between teacher and student predictions. For sequential tasks, the authors introduce trajectory-level reward weighting and step-level weighting using a critic. Experiments cover semantic segmentation, maze navigation, Atari, code generation, and mathematical reasoning. The results show improvements over standard KD in several settings, suggesting that reward-guided weighting can be useful across different tasks.

Critical strengths
  1. The main idea is straightforward and can be incorporated into existing KD objectives.
  2. The experiments cover a wide range of tasks, with encouraging improvements in several settings.
  3. The critic-based extension provides an interesting way to use step-level value estimates for sequential distillation.
Critical weaknesses
  1. The reward-weighted objective in Eq. (3) does not change the pointwise optimum of standard KD, and its distinction from existing reward-weighted distillation methods remains unclear.
  2. The transition from Eq. (6) to Eq. (7) introduces an additional covariance term, changing the original optimization objective without sufficient justification.
  3. The step-level formulation does not directly distinguish individual action values within the teacher distribution, while reward scaling, computational overhead, and experimental stability are insufficiently investigated.
Questions for authors
  1. Can the authors better explain why reward-weighted KD should improve student performance and how it differs from existing reward-weighted distillation methods?
  2. Could the authors compare Eq. (6) and Eq. (7) experimentally to clarify whether the modified objective provides additional benefits?
  3. How are reward weights normalized or constrained? Could the authors also report the cost of critic training and results across multiple random seeds?
5 comments
AuthorsOpenReview ↗

Response to Reviewer hEw5 (1/2)

We thank you for recognizing the practical applicability of RG-KD and its critic-based step-level extension. We appreciate the constructive feedback and address your concerns below.


W1 & Q1

The reward-weighted objective in Eq. (3) does not change the pointwise optimum of standard KD. Why should reward-weighted KD improve student performance, and how does it differ from existing reward-weighted distillation methods?

We thank you for raising this important point. RG-KD does not aim to change the pointwise optimum of individual teacher distributions; instead, it explicitly adjusts their relative contributions to the overall distillation objective through reward-based weighting at the sample, trajectory, or step level. As discussed in Lines 138-145, teacher distributions may not be equally beneficial as distillation targets across samples, because they may assign probability mass to outputs that are suboptimal, uninformative, or difficult for the student to benefit from. RG-KD therefore adjusts distillation strength according to the expected downstream utility of teacher predictive distributions, better aligning distillation with the goal of improving student task performance.

Unlike existing reward-weighted distillation methods that rely on ad-hoc heuristics to prioritize specific trajectories, RG-KD establishes a unified, principled framework grounded in the theoretical synergy between RL and KD. RG-KD measures the quality of the teacher's predictive distribution as a distillation target by estimating its expected task utility from rewards assigned to sampled outputs. This theoretical foundation enables the extension to step-level RG-KD by incorporating critic-based value estimation from RL into KD. By estimating utility at each individual timestep, RG-KD adaptively modulates distillation strength at a finer-grained, step-wise level—a capability not provided by prior heuristic reward-weighting distillation methods that operate at the trajectory level. Consequently, existing methods can be viewed as restricted, offline trajectory-level instantiations of the RG-KD framework that rely on heuristic reward-based selection or weighting. Establishing this theoretical foundation is significant, as it provides a rigorous, reusable paradigm for future distillation research rather than another heuristic weighting strategy.


W2 & Q2

The transition from Eq. (6) to Eq. (7) introduces an additional covariance term, changing the original optimization objective without sufficient justification. Could the authors compare Eq. (6) and Eq. (7) experimentally to clarify whether the modified objective provides additional benefits?

As explained in Lines 211–225, Eq. (7) is derived by applying a deliberate coupled sampling heuristic, rather than assuming mathematical equivalence with Eq. (6). While computing the exact expectations in Eq. (6) over all possible trajectories would be ideal, it is computationally infeasible. A practical alternative is to estimate the two expectations independently via Monte Carlo sampling, which requires NN trajectories for the expected reward and another MM trajectories for the expected distillation loss. However, this would misalign the reward signal with the supervision being weighted, particularly when NN and MM are reduced to limit computational cost. For example, when N=1N = 1 and M=1M = 1, the reward from one sampled trajectory would weight the distillation loss of another independently sampled trajectory, despite their different downstream outcomes. To address these practical limitations, we compute the reward and distillation loss for each sampled trajectory and average their products, ensuring that each reward weights its corresponding supervision, as formulated in Eq. (7).

In response to the reviewer's question, we conduct additional experiments comparing Eqs. (6) and (7) on GSM8K and MATH. For Eq. (6), we independently sample one rollout to estimate the expected reward (N=1N = 1) and another to estimate the expected distillation loss (M=1M = 1). As shown in the table below, the two objectives achieve comparable performance. We observe low reward–loss covariance on GSM8K (-0.02) and MATH (-0.01). One possible explanation is that the teachers already achieve high accuracy on GSM8K (89.39) and MATH (73.96), so independently sampling trajectories for reward and divergence estimation may still yield similarly high-quality outputs, limiting the mismatch between them. Nevertheless, such mismatch may become more pronounced in tasks where teacher predictions exhibit greater variability in quality. We will include these experimental results and analyses in the final manuscript.

MethodGSM8KMATH
Trajectory-level RG-KD (Eq. 6)77.6466.46
Trajectory-level RG-KD (Eq. 7)77.6466.56

AuthorsOpenReview ↗

Response to Reviewer hEw5 (2/2)

W3-1

The step-level formulation does not directly distinguish individual action values within the teacher distribution.

We would first like to clarify that step-level RG-KD assigns different weights to the distillation loss at each individual timestep. We understand the reviewer's comment to mean that step-level RG-KD does not separately estimate the values of sampled actions from the teacher distribution at a specific timestep. In this regard, since RG-KD is fundamentally a knowledge distillation framework that aims to match the teacher's predictive distribution, incorporating distinct sampled-action values into the distillation objective is not straightforward. If the reviewer has a particular formulation in mind, we would welcome further clarification and be happy to discuss it.


W3-2 & Q3

Reward scaling, computational overhead, and experimental stability are insufficiently investigated. How are reward weights normalized or constrained, and could the authors report the cost of critic training and results across multiple random seeds?

Reward Weight Configuration

Thank you for pointing out the need to clarify the weighting rule used in practice. In all experiments, the weight is w=1+C⋅Rw = 1 + C·R, where CC is a task-specific scaling hyperparameter and RR is the reward signal. The baseline subtraction described in Sec. 4.1 is applied. Specifically, RR is measured relative to an exponential moving average (EMA) of the reward for trajectory-level RG-KD and relative to the critic-estimated value of a student-sampled action for step-level RG-KD. We will clarify these details in the final manuscript.

Critic Training Cost

As discussed in Sec. 3.3.3 and Appendix D (Limitations), step-level RG-KD incurs additional computational cost from using a critic for value estimation and continuously updating it online using reward-model feedback. We measure the computational cost in GSM8K experiments conducted on a single NVIDIA H100 GPU. Pretraining the critic takes approximately 23 hours for 30 epochs. Since online critic training costs are difficult to measure separately during distillation, we compare the total distillation training times across methods. Standard KD, EA-KD, and trajectory-level RG-KD require 15.93, 15.95, and 16.12 hours, respectively, while step-level RG-KD requires 41.55 hours. As in critic-based reinforcement learning for LLMs, this overhead naturally arises from critic-based value estimation and online critic training.

Multi-Seed Results

We also conduct additional experiments using three different training seeds on GSM8K and Atari (Qbert) with teacher-generated trajectories, reporting the mean and standard deviation across KLD, JSD, and SKL, as shown in the tables below. On GSM8K, step-level RG-KD achieves the highest mean accuracy across all three divergence objectives (80.39 ± 0.12, 80.24 ± 0.38, and 80.59 ± 0.20). On Qbert, both trajectory-level and step-level RG-KD outperform Standard KD in mean performance across all three divergence objectives. Notably, the step-level JSD and SKL settings that underperformed Standard KD in the original single-seed results achieve higher average scores over three seeds, although the larger standard deviations indicate greater variability across runs. These results further support the effectiveness of RG-KD, with some settings exhibiting substantial variability across training seeds. We will report these multi-seed results, along with those from the remaining experiments, in the final manuscript.

MethodKLD (GSM8K)JSD (GSM8K)SKL (GSM8K)KLD (Qbert)JSD (Qbert)SKL (Qbert)
Standard KD77.43 ± 0.0477.61 ± 0.3177.71 ± 0.208,175.00 ± 347.315,616.67 ± 1,999.117,008.33 ± 2,234.16
EA-KD77.48 ± 0.1677.71 ± 0.0877.61 ± 0.168,008.33 ± 62.926,816.67 ± 2,050.979,325.00 ± 2,295.51
RG-KD (Trajectory-level)77.69 ± 0.0978.17 ± 0.0878.37 ± 0.3510,558.33 ± 2,175.059,300.00 ± 2,338.279,233.33 ± 2,006.29
RG-KD (Step-level)80.39 ± 0.1280.24 ± 0.3880.59 ± 0.2010,316.67 ± 1,978.858,233.33 ± 3,649.519,550.00 ± 2,159.43

Thank you again for your valuable feedback. We hope our responses have addressed all your concerns and would be happy to provide any further information if needed.

Thank you for the detailed response and additional experiments. The explanation of reward-based reweighting under limited student capacity, together with the new baselines and multi-seed results, strengthens the empirical evidence.

However, my main concerns are only partially resolved. The authors clarify that Eq. (7) is a coupled-sampling heuristic rather than an equivalent estimator of Eq. (6), but the new results show nearly identical performance and do not establish the benefit of changing the objective. The step-level formulation also remains a scalar reweighting of the full distillation loss at each history and does not distinguish individual teacher-action values. Finally, the reported critic cost is substantial, and the Atari results still show considerable variability.

The rebuttal improves my assessment of the paper, but I do not find the remaining methodological and practical concerns sufficiently resolved to change my recommendation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.