acceptodds
Under review as a conference paper at ICLR 2027

Hierarchical Self-Distillation for Agentic Reinforcement Learning

Abstract

On-policy self-distillation (OPSD) has emerged as a promising post-training approach that complements sparse outcome rewards in agentic reinforcement learning with dense guidance from a privileged self-teacher. Nevertheless, we reveal that the utility of teacher supervision in multi-turn OPSD is hierarchically non-uniform: step-level supervision utility is influenced by both accumulated state drift and future teacher–student divergence, while at the token level, under a limited support budget, a fixed-size support cannot maintain consistent fidelity to full-distribution supervision as teacher–student discrepancy varies. To address this, we propose HiSD, a hierarchical self-distillation framework that combines progress-aware step supervision with support-adaptive token supervision, allocating step-level distillation according to relative divergence progress and adapting the distributional scope of token-level supervision based on teacher–student discrepancy. Across ALFWorld, Search-QA, and WebShop with Qwen2.5-Instruct models, HiSD consistently outperforms GRPO and recent OPSD-based baselines. Notably, on tasks with strong dependencies across interaction steps, HiSD outperforms SDAR by an average of points on Pick2 and points on multi-hop QA, while achieving higher teacher budget utilization on WebShop. Code available: https://anonymous.4open.science/r/HiSD.

Primary area:
reinforcement learning
Keywords:
agentic reinforcement learning, on-policy self-distillation, large language model agents, multi-turn interaction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5H4iOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3(+1)
Soundness
3
Read review
Paper summary

The paper proposes HiSD for on-policy self-distillation in multi-turn agents. At the step level, it adjusts distillation weights using accumulated and remaining teacher–student divergence. At the token level, it selects an adaptive support based on the current divergence and applies forward-KL supervision. The method is evaluated on ALFWorld, Search-QA, and WebShop across three model sizes.

Critical strengths

Clear motivation and a sensible design. The paper addresses both state drift across interactions and the distributional coverage of token-level supervision. Step weighting and adaptive support selection target different aspects of the problem and appear complementary.

Broad evaluation and useful mechanism analyses. The experiments cover three tasks and three model sizes, with ablations of components, weighting strategies, support selection, and hyperparameters. The analysis of long-horizon tasks is also informative: HiSD improves over SDAR by approximately 16 points on average on Pick2.

Critical weaknesses
  1. The public implementation differs from the method definition. The paper requires at least two support tokens, while the scripts allow one; after renormalization, singleton support produces zero distillation loss and gradient. The paper also weights trajectories equally, whereas the code averages over interaction steps, giving longer trajectories greater weight. These differences change the supervision mechanism and optimization objective. The authors should clarify which implementation produced the reported results.

  2. The step-weighting ablation may not control distillation strength. The public code omits the base coefficient λbase\lambda_{\mathrm{base}} in the Relative Position branch but includes it in full HiSD. With the default coefficient of 0.01, this omission increases the positional variant’s coefficient by a factor of 100 relative to its intended value. If Table 2 uses this implementation, its lower performance could reflect excessive distillation rather than an inferior weighting strategy, weakening the evidence for the proposed design.

Questions for authors
  1. Were the reported results obtained with the paper’s minimum support size of two and equal trajectory weighting, or with the code’s allowance for singleton support and equal step weighting? Could the authors provide the actual run configurations and explain the effects of these differences?

  2. Did the Relative Position ablation in Table 2 use the corresponding public-code branch? If so, could the authors restore the λbase\lambda_{\mathrm{base}} multiplier and repeat the comparison at matched distillation strength?

2 comments

I used AI to discuss the paper, check the code, and polish the review. I evaluated its suggestions and made the final assessment and recommendation myself. After reviewing the updated code and training log, I consider my main concerns addressed and am willing to raise my rating to 3.

AuthorsOpenReview ↗

Thank you!

Dear Reviewer 5H4i,

Thank you for re-examining our updated code and log. We are glad our clarifications addressed your main concerns, and we appreciate your willingness to raise your rating to 3.

Please let us know if any specific concerns remain.

With sincere thanks,

The Authors

Reviewer_GravOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a Hierarchical Self-Distillation (HiSD) framework for agentic RL. It combines GRPO with privileged teacher supervision using two adaptive mechanisms. The first is progress-aware step supervision, which adjusts the degree of teacher guidance based on accumulated and remaining teacher-student divergence along a trajectory. Larger weights are assigned when accumulated divergence is small relative to remaining divergence, while supervision is reduced as accumulated divergence becomes dominant. The second is support-adaptive token supervision, which adjusts the number of candidate tokens used for distillation based on instantaneous teacher-student KL divergence. Larger divergence results in a smaller support focused on the teacher's high-probability tokens, whereas smaller divergence leads to a larger support. The main contribution is the design and combination of these two adaptation mechanisms. Experiments across several agentic benchmarks demonstrate improvements over GRPO and existing self-distillation approaches.

Critical strengths
  • The problem is well motivated. Allocating teacher supervision more effectively across interaction steps and tokens is relevant.
  • The adaptation mechanisms are relatively simple and can be incorporated into an existing on-policy self-distillation framework.
  • The experimental evaluation is extensive, covering several benchmarks and model sizes. The results show promising improvements in performance and efficiency, and the ablation studies provide evidence that the two adaptation mechanisms are complementary.
Critical weaknesses
  • The paper provides a good set of experiments and empirical results but the proposed rules appear largely heuristic without sufficiently convincing justification. For instance, it is not clear why accumulated KL divergence should necessarily indicate the teacher's guidance would be less reliable. One may think the opposite, namely, that large divergence could indicate that teacher supervision is valuable.

  • In terms of justification, the same goes for adapting the support size based on instantaneous KL divergence, motivated by observations about gradient coverage. I am not convinced that KL divergence alone characterizes how the discrepancy is distributed across tokens, nor that it provides a sufficiently informative criterion for determining the support required to preserve the full-distribution gradient. A more principled justification is needed.

  • The paper does not introduce a fundamentally new approach to RL or self-distillation. Rather, it combines two relatively simple adaptive heuristics for weighting and truncating an existing distillation objective. The empirical improvements are promising but it remains unclear whether the proposed approach provides substantial new methodological insights beyond an effective combination of heuristics.

Questions for authors

Please see my comments above. I am currently leaning toward a weak reject, but willing to reconsider my assessment if the authors can provide a more convincing justification for the two adaptation mechanisms and clarify the novel insights offered by their analysis.

4 comments
AuthorsOpenReview ↗

[Response 1/3 — W1]: Accumulated and remaining divergence in teacher supervision

Dear Reviewer Grav,

Thank you for your thoughtful feedback and broader perspective on our work. Your constructive comments have prompted us to think more deeply about HiSD and helped us make the paper more complete and its underlying rationale clearer.

We apologize for the delayed response, as we have been conducting additional experiments and revising the manuscript; the updated version is now available for your review.

We have revised Figures 1 and 2 and their accompanying text, and added Figures 7 and 9, Section 4.6, and Appendices A.5, B.4, and B.5. These changes are highlighted in GREEN\color{green}{\text{GREEN}} in our updated manuscript.


Weakness 1. The paper provides a good set of experiments and empirical results but the proposed rules appear largely heuristic without sufficiently convincing justification. For instance, it is not clear why accumulated KL divergence should necessarily indicate the teacher's guidance would be less reliable. One may think the opposite, namely, that large divergence could indicate that teacher supervision is valuable.

Our Response:

We agree that greater teacher–student divergence can indicate more valuable supervision. HiSD therefore distinguishes divergence accumulated before the current decision from divergence remaining in the current and subsequent decisions. The following analyses examine how these two signals relate to the utility of teacher guidance.

  • Accumulated divergence and recovery after teacher takeover. We replaced the original Figure 1(a), which showed step-wise KL increasing with step index, with a finding more directly relevant to our step-weighting design. Figures 1(a) and 9(a) show that, at matched takeover positions and with a full interaction budget, higher prefix divergence is associated with lower teacher takeover success. Figure 9(b) further shows that, even on tasks the teacher can solve from the initial state, higher prefix divergence corresponds to more takeover failures within the same suffix-divergence group. These findings suggest that student-induced states and histories can limit the effectiveness of subsequent teacher guidance.

  • Remaining divergence and opportunities for intervention. Figure 9(c) shows that, within the same prefix-divergence group, higher suffix divergence is associated with a larger proportion of tasks rescued by extending the teacher's interaction budget. Figure 1(b) also shows lower student continuation KL in the group with higher KL during teacher takeover. These observations suggest that larger remaining divergence can still offer valuable opportunities for teacher guidance, motivating joint consideration of accumulated and remaining divergence.

  • Supervision based on relative divergence progress. Section 3.2 incorporates both signals. Appendix A.6 gives the following equivalent expression for the step weight:

    λk=σ!(Dksuf−Dkpreκ).\lambda_k=\sigma\\!\left(\frac{D_k^{\mathrm{suf}}-D_k^{\mathrm{pre}}}{\kappa}\right).

    When remaining divergence dominates, HiSD retains stronger teacher guidance; when accumulated divergence dominates, it downweights distillation. Supervision strength thus accounts for both the constraints imposed by interaction history and the remaining opportunities for guidance.

  • Ablation and gradient evidence. Table 2 compares HiSD with Prefix Only, Suffix Only, and Relative Position. Jointly considering prefix and suffix divergence yields better performance on all three tasks, supporting supervision weights that adapt to the distribution of divergence along each trajectory. Figure 7(b–c) further examines how progress-aware step supervision affects joint OPSD–GRPO optimization through parameter-gradient analysis.


AuthorsOpenReview ↗

[Response 2/3 — W2]: Rationale for support adaptation based on instantaneous KL


Weakness 2. In terms of justification, the same goes for adapting the support size based on instantaneous KL divergence, motivated by observations about gradient coverage. I am not convinced that KL divergence alone characterizes how the discrepancy is distributed across tokens, nor that it provides a sufficiently informative criterion for determining the support required to preserve the full-distribution gradient. A more principled justification is needed.

Our Response:

  • Direct observations of the discrepancy distribution. Figure 1(d) shows that, at higher KL, teacher–student discrepancies tend to concentrate among high-probability teacher candidates; at lower KL, achieving the same discrepancy coverage generally requires more candidates. These observations motivate adapting the supervision range to instantaneous KL.

    This coverage measure also has a precise gradient interpretation. Proposition 5 in Appendix A.4 shows that the forward-KL gradient with respect to student logits is gV=pV−qVg_{\mathcal V}=p_{\mathcal V}-q_{\mathcal V}. Thus, discrepancy coverage is exactly the fraction of the full gradient's L1L_1 norm captured by the selected coordinates:

    C1(S)=∑v∈S∣pV(v)−qV(v)∣∑v∈V∣pV(v)−qV(v)∣=∣gS∣1∣gV∣1.C_1(S)=\frac{\sum_{v\in S}|p_{\mathcal V}(v)-q_{\mathcal V}(v)|}{\sum_{v\in\mathcal V}|p_{\mathcal V}(v)-q_{\mathcal V}(v)|}=\frac{\\|g_S\\|_1}{\\|g_{\mathcal V}\\|_1}.

    Corollary 4 further shows that, under a common coordinate-wise logit perturbation bound, this ratio equals the fraction of the maximum first-order KL decrease attainable by updating only the selected coordinates.

  • Support selection as constrained distribution compression. Building on these observations, Section 3.3 formulates support selection as distribution compression subject to a distortion constraint. With instantaneous discrepancy δ=DKL(pC∣qC)\delta=D_{\mathrm{KL}}(p_C\\|q_C) on the candidate set, we select the smallest support satisfying DKL(qS∣qC)≤αδD_{\mathrm{KL}}(q_S\\|q_C)\leq\alpha\delta, with at least two tokens retained. This constraint is equivalent to retaining at least e−αδe^{-\alpha\delta} of the teacher's probability mass. KL sets the compression budget, while the teacher distribution determines which candidates to retain and how many are needed to meet that budget. The method therefore does not require KL to uniquely determine the token-wise discrepancy distribution.

  • Gradient fidelity after renormalization. Coordinate coverage alone does not establish gradient fidelity after renormalization. Proposition 3 and Corollary 2 in Appendix A.2 address this distinction by analyzing the actual support-restricted distillation loss. Let gCg_C denote the distillation logit gradient over the full candidate set and hSh_S the gradient after restricting and renormalizing the distributions on the selected support. HiSD's retention rule guarantees

    ∣hS−gC∣2≤2(1+eα)δ.\\|h_S-g_C\\|_2\leq2(1+e\alpha)\delta.

    Corollary 2 further shows that, as the teacher distribution approaches the student distribution, the relative squared error is O(δ)O(\delta) and vanishes. This explains why the compression budget tightens as the distributions align, preserving finer distributional supervision.

  • Ablations and parameter-gradient analysis. Table 2 compares HiSD with Random Selection, Top Selection, and Student Candidate Support; HiSD performs better on all three tasks. Figure 3 further shows that adaptive support outperforms retaining all Top-256 candidates (α=0\alpha=0). Figure 7(a) in Section 4.6 further examines this comparison at the parameter-gradient level: with step weighting held fixed, adaptive support increases gradient cosine similarity and reduces the conflict rate relative to fixed Top-256 support, indicating that it helps mitigate directional conflict between distillation and policy optimization.


AuthorsOpenReview ↗

[Response 3/3 — W3]: Theoretical rationale and methodological insights


Weakness 3. The paper does not introduce a fundamentally new approach to RL or self-distillation. Rather, it combines two relatively simple adaptive heuristics for weighting and truncating an existing distillation objective. The empirical improvements are promising but it remains unclear whether the proposed approach provides substantial new methodological insights beyond an effective combination of heuristics.

Our Response:

HiSD's contribution is to identify and address the hierarchical nonuniformity of teacher supervision in multi-turn OPSD: the value of the same teacher guidance depends on the interaction history, while the required granularity of distributional supervision varies with teacher–student discrepancy. The revised manuscript explains the joint role of support adaptation and step weighting through theoretical analysis in Appendix A.5, parameter-gradient experiments in Figure 7, and ablations in Table 2.

  • Theoretical analysis. Proposition 6 in Appendix A.5 considers the joint update direction g+λhg+\lambda h, where gg and hh are the GRPO and distillation gradients. Let c=cos⁡(g,h)c=\cos(g,h) denote their directional alignment and a=λ∣h∣/∣g∣a=\lambda\\|h\\|/\\|g\\| the relative distillation strength. The fraction of first-order GRPO decrease retained relative to a GRPO-only update at the same step size is 1+ac1+ac. Thus, when the gradients conflict (c<0c<0), the negative term acac offsets part of the progress on the task objective.

    Subject to the distillation loss not increasing to first order, even optimal scalar reweighting can retain at most 1−c21-c^2, attained at a=−ca=-c. With a fixed distillation direction, strength adjustment alone cannot exceed this conflict-dependent bound; as the negative cosine cc approaches zero, the bound increases, allowing more GRPO progress while maintaining the distillation objective.

    Improving the direction of teacher supervision can therefore relax this limitation, while appropriately weighting its contribution can reduce cancellation in the actual update. This provides a unified rationale for jointly adapting distributional support and step weights.

  • Gradient conflict analysis. With step weighting held fixed, Figure 7(a) shows that HiSD's support-adaptive token supervision increases gradient cosine similarity and reduces the conflict rate, improving alignment between teacher supervision and task optimization and helping relax the conflict-dependent bound on retained first-order GRPO decrease in Appendix A.5.

    Figure 7(b) further shows that step weighting moves the negative relative projection, which corresponds to acac in the analysis, closer to zero, reducing distillation-induced cancellation in the first-order descent ratio 1+ac1+ac. Figure 7(c) also shows greater retention of aligned than conflicting projections at later interaction steps, indicating that step weighting more strongly suppresses contributions that impede task optimization while preferentially retaining useful teacher guidance. Together, these results support the complementary roles of the two adaptations: reducing directional conflict and limiting the contribution of residual conflict to the joint update.

  • Ablation evidence. Table 2 systematically evaluates variants that retain only one adaptation or replace the step-weighting or token-support selection strategy. All underperform full HiSD, supporting the effectiveness of both adaptations and their specific designs.


We sincerely thank you again for your careful reading and for raising these important questions, which have helped us substantially improve the clarity and rigor of our work. We hope that our responses and the corresponding revisions have adequately addressed your major concerns. If so, we would be very grateful if you could consider re-evaluating our paper. Should any questions or concerns remain, we would be more than happy to continue the discussion and provide further clarification.

We sincerely look forward to your reply.

Sincerely,

The Authors

AuthorsOpenReview ↗

To Reviewer Grav–Follow-up on Our Rebuttal Response

Dear Reviewer Grav,

We hope this message finds you well. We are writing to kindly follow up on our response to your review for Submission45757.

In our rebuttal, we have carefully reorganized and substantially revised our clarifications, with detailed responses to each of your concerns. We would be very grateful to know whether our revisions now meet the condition you kindly stated earlier: “I am currently leaning toward a weak reject, but willing to reconsider my assessment if the authors can provide a more convincing justification for the two adaptation mechanisms and clarify the novel insights offered by their analysis.”

We would greatly appreciate your response and any further feedback you may have. Your support would mean a great deal to us.

Thank you again for your careful reading, your hard work, and your valuable contributions to the research community.

Best regards,

The authors

Reviewer_SakkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HiSD, a method for on-policy self-distillation in multi-turn agent reinforcement learning. It employs "progress-aware step supervision" to dynamically determine the appropriate level of teacher guidance for each interaction step, and "support-adaptive token supervision" to adjust the token-level distribution support based on the discrepancy between the teacher and student models. The primary contribution lies in unifying state drift and token-level distribution mismatch within a hierarchical supervision framework. Through theoretical analysis, ablation studies, and evaluations across multiple benchmarks, the authors demonstrate the complementary nature of these two adaptive mechanisms. While the contribution is comprehensive, the approach is fundamentally built upon existing frameworks like GRPO and OPSD. Furthermore, it relies on pre-existing skills as "privileged information"; consequently, the degree of independent innovation and the necessity of the specific approach to utilizing these skills warrant further validation.

Critical strengths
  • The problem definition is well-targeted, focusing on the issue of non-uniform supervisory value during multi-turn agent self-distillation. Rather than simply applying a fixed-strength teacher supervision, the paper distinguishes between two types of non-uniformity: step-level and token-level. The former accounts for cumulative state drift and potential future divergence across multi-turn interactions, while the latter dynamically adjusts the distillation support size based on the discrepancy between the teacher and the student. This problem formulation aligns effectively with the long-horizon nature of agent-based tasks.

  • The paper presents a robust chain of evidence, encompassing theoretical analysis, ablation studies, and primary experiments. Beyond achieving strong empirical results on benchmarks such as ALFWorld, Search-QA, and WebShop, the study validates the complementarity of step-wise and token-wise adaptation. Furthermore, it provides a theoretical analysis of the relative error of the sampled-token gradient as it approaches the teacher’s distribution, as well as the effectiveness of adaptive support in approximating the full-distribution gradient. Consequently, the approach is not merely a collection of empirical modules, but is grounded in a solid conceptual framework.

Critical weaknesses
  • The core methodology carries a risk of being perceived as merely incremental innovation. The key components of HiSD have all been explored in existing research. The primary contribution of this paper lies in integrating step-level weighting with token-level adaptive support. Consequently, reviewers might view this as a hierarchical, adaptive extension of existing OPSD methods rather than an entirely novel training paradigm.
  • While the paper compares HiSD against "Skill-GRPO" and demonstrates that direct skill augmentation yields limited benefits, this baseline does not encompass all possible approaches to direct skill training. Therefore, the current experiments effectively show that HiSD outperforms Skill-GRPO, but they do not conclusively prove that self-distillation is either a necessary or an optimal method for leveraging skills.
  • Although privileged information is not required during the inference phase, the training process necessitates access to task-relevant skills, which must be provided to the self-teacher. The experiments in this paper utilize skills sourced directly from SkillRL; thus, it remains unverified whether the benefits of HiSD would hold in scenarios where high-quality skills are unavailable, contain biases, or are costly to obtain.
Questions for authors
  • Beyond merely combining step-wise OPSD and token-level adaptive distillation, does the hierarchical design of HiSD embody a more fundamental unifying principle or offer unique, irreplaceable advantages?

  • Current experiments—primarily using Skill-GRPO—suggest that the direct application of skills yields limited benefits. Would HiSD still maintain a clear advantage if more robust approaches were employed, such as advanced direct skill-conditioning, skill-based SFT, or other sophisticated skill-to-policy transfer methods?

  • HiSD’s training relies on task-relevant privileged skills. To what extent would the method’s performance and stability be affected if these skills are of low quality, contain noise, or are otherwise difficult to obtain reliably for a given task?

4 comments
AuthorsOpenReview ↗

[Response 1/3 — W1/Q1]

Dear Reviewer Sakk,

Thank you for your thoughtful feedback and valuable perspective. Your constructive comments helpe us strengthen both the presentation and evaluation of our work.

We apologize for the delayed response, as we have been conducting additional experiments and revising the manuscript; the updated version is now available for your review.

In the updated manuscript, we have added Figures 7 and 9, Section 4.6, and Appendices A.5, B.4, and B.5 to address W1/Q1 (highlighted in GREEN\color{green}{\text{GREEN}}), and Appendices C.2 and C.3 (Tables 4 and 5) to address W2/Q2 and W3/Q3 (highlighted in BLUE\color{blue}{\text{BLUE}}).


Weaknesse 1. The core methodology carries a risk of being perceived as merely incremental innovation. The key components of HiSD have all been explored in existing research. The primary contribution of this paper lies in integrating step-level weighting with token-level adaptive support. Consequently, reviewers might view this as a hierarchical, adaptive extension of existing OPSD methods rather than an entirely novel training paradigm.

Question 1. Beyond merely combining step-wise OPSD and token-level adaptive distillation, does the hierarchical design of HiSD embody a more fundamental unifying principle or offer unique, irreplaceable advantages?

Our Response:

HiSD's contribution is to identify and address the hierarchical nonuniformity of teacher supervision in multi-turn OPSD: the value of the same teacher guidance depends on the interaction history, while the required granularity of distributional supervision varies with teacher–student discrepancy. The revised manuscript explains the joint role of support adaptation and step weighting through theoretical analysis in Appendix A.5, gradient experiments in Figure 7, and ablations in Table 2.

1. A unified optimization perspective.

In Appendix A.5, Proposition 6, we analyze the joint update g+λhg+\lambda h, where gg and hh are the GRPO and distillation gradients. Let c=cos⁡(g,h)c=\cos(g,h) and a=λ∥h∥/∥g∥a=\lambda\lVert h\rVert/\lVert g\rVert. Relative to a GRPO-only update with the same step size, the fraction of first-order GRPO descent retained by the joint update is 1+ac1+ac. When the gradients conflict (c<0c<0), requiring the distillation loss not to increase to first order limits this fraction to at most 1−c21-c^2, even with an optimally chosen overall coefficient. The bound is attained at a=−ca=-c.

This exposes a limitation of adjusting distillation strength alone: for a fixed distillation direction, even the optimal strength remains subject to a bound imposed by directional conflict. As the negative cosine approaches zero, this bound increases, allowing more GRPO descent to be retained without increasing the distillation loss to first order. This provides a unified optimization rationale for considering support selection and step weighting together.

2. Gradient analysis.

Section 4.6 and Figure 7 examine these relationships using gradients collected during training. Figure 7(a) shows higher gradient cosine similarity and a lower conflict rate following support adaptation. In Figure 7(b), step weighting attenuates negative relative projections, reducing the magnitude of the acac term that counteracts first-order GRPO descent. Figure 7(c) shows greater relative retention of aligned projections at later interaction steps. Together, these results show how reducing directional conflict and adjusting supervision contributions can affect the joint update.

3. Ablation studies.

Table 2 systematically compares the full method with variants retaining only one adaptation mechanism and variants using alternative step-weighting or token-support selection strategies. Full HiSD outperforms these variants, supporting the complementarity of the two mechanisms and the effectiveness of their specific designs.


AuthorsOpenReview ↗

[Response 2/3 — W2/Q2]

Our Response:

We agree that the statement "sparse rewards are insufficient for effective skill learning" is overly strong. We have revised Section 4.2 to state that "skill-conditioned reinforcement learning does not necessarily lead to effective skill internalization".

To assess whether HiSD remains effective against stronger skill-learning approaches, we have added three sets of comparisons in Appendix C.2, Table 4 (p. 38). The results are presented below:

\begin{array}{lcrrr} \hline \textbf{Method} & \textbf{Infer. Skill} & \textbf{ALFWorld} & \textbf{WebShop} & \textbf{WebShop} \\\\ & & \textbf{Succ.} & \textbf{Score} & \textbf{Succ.} \\\\ \hline \text{Qwen2.5-7B-Instruct} & \text{No} & 26.1 & 2.1 & 0.0 \\\\ \text{Qwen2.5-7B-Instruct} & \text{Yes} & 41.4 & 2.8 & 0.0 \\\\ \hline \text{GRPO} & \text{No} & 81.2 & 80.9 & 72.6 \\\\ \text{Skill-GRPO} & \text{No} & 69.5 & 80.4 & 71.9 \\\\ \text{Skill-GRPO} & \text{Yes} & 88.3 & 87.0 & \underline{81.2} \\\\ \hline \text{Skill-SFT} & \text{No} & 27.3 & 58.0 & 27.7 \\\\ \text{Skill-SFT} & \text{Yes} & 4.7 & 12.9 & 7.8 \\\\ \text{Skill-SFT + Skill-GRPO} & \text{No} & 71.1 & 78.6 & 64.1 \\\\ \text{Skill-SFT + Skill-GRPO} & \text{Yes} & 80.5 & 81.5 & 67.2 \\\\ \hline \text{Skill0} & \text{No} & 85.9 & 83.3 & 70.9 \\\\ \text{SkillC} & \text{No} & \underline{90.6} & 85.6 & 74.0 \\\\ \textbf{HiSD (Ours)} & \text{No} & 89.8 & \underline{90.5} & \underline{81.2} \\\\ \textbf{HiSD (Ours)} & \text{Yes} & \mathbf{91.4} & \mathbf{91.1} & \mathbf{82.3} \\\\ \hline \end{array}$$ - **RL with inference-time skill access.** We include the Skill-GRPO results reported in SDAR, where skills are available during both training and inference. Even against this setting, HiSD requires no inference-time skills while achieving higher ALFWorld success and WebShop Score, and matching WebShop success. Providing HiSD with skills at inference further improves all three metrics, yielding the best results among the compared methods. These gains indicate that skill internalization and inference-time skill guidance can be complementary. - **Skill-based SFT and subsequent RL.** We add two standard training paradigms applied to skill transfer: skill-based supervised fine-tuning (Skill-SFT) and SFT followed by reinforcement learning (Skill-SFT + Skill-GRPO). Both are evaluated with and without inference-time skills. HiSD's consistent gains across all reported metrics and both inference settings indicate more effective transfer of skill supervision into task-solving capabilities that do not require external skill guidance. - **Advanced skill internalization methods.** We further include Skill0 (Lu et al., 2026) and SkillC (Lin et al., 2026). Skill0 progressively removes skill prompts based on their measured utility, whereas SkillC uses reward differences between paired rollouts with and without skills to assign greater credit to successful execution without skill guidance. Without inference-time skills, HiSD outperforms Skill0 on all three metrics. Compared with SkillC, it achieves comparable ALFWorld success, while improving WebShop Score by $4.9$ points and WebShop success by $7.2$ percentage points. These results establish HiSD as a competitive alternative to other advanced skill internalization methods. **References:** Lu, Zhengxi, et al. (2026). *SKILL0: In-Context Agentic Reinforcement Learning for Skill Internalization.* Findings of the Association for Computational Linguistics: EMNLP 2026, 4485–4505. Lin, Hongxiang, et al. (2026). *SKILLC: Learning Autonomous Skill Internalization in LLM Agents via Contrastive Credit Assignment.* arXiv:2605.27899.
AuthorsOpenReview ↗

[Response 3/3 — W3/Q3]

Our Response:

To assess HiSD's generalizability across skill libraries with varying utility and characteristics, we have added comparisons across SkillRL (Full), SkillRL (General) (Xia et al., 2026), and SkillNet (Liang et al., 2026) in Appendix C.3, Table 5 (pp. 38–39). The results are presented below:

\begin{array}{llcrrr} \hline \textbf{Skill Library} & \textbf{Method} & \textbf{Infer. Skill} & \textbf{ALFWorld} & \textbf{WebShop} & \textbf{WebShop} \\\\ & & & \textbf{Succ.} & \textbf{Score} & \textbf{Succ.} \\\\ \hline \textit{None} & \text{Qwen2.5-7B-Instruct} & \text{No} & 26.1 & 2.1 & 0.0 \\\\ & \text{GRPO} & \text{No} & 81.2 & 80.9 & 72.6 \\\\ \hline \textit{SkillRL (Full)} & \text{Qwen2.5-7B-Instruct} & \text{Yes} & 41.4 & 2.8 & 0.0 \\\\ & \text{Skill-GRPO} & \text{No} & 69.5 & 80.4 & 71.9 \\\\ & \text{Skill-GRPO} & \text{Yes} & 88.3 & 87.0 & \underline{81.2} \\\\ & \textbf{HiSD (Ours)} & \text{No} & \underline{89.8} & \underline{90.5} & \underline{81.2} \\\\ & \textbf{HiSD (Ours)} & \text{Yes} & \mathbf{91.4} & \mathbf{91.1} & \mathbf{82.3} \\\\ \hline \textit{SkillRL (General)} & \text{Qwen2.5-7B-Instruct} & \text{Yes} & 31.3 & 2.8 & 0.0 \\\\ & \text{Skill-GRPO} & \text{No} & 65.6 & 75.8 & 64.5 \\\\ & \text{Skill-GRPO} & \text{Yes} & 82.8 & 84.5 & 71.1 \\\\ & \textbf{HiSD (Ours)} & \text{No} & \underline{87.5} & \mathbf{87.2} & \mathbf{76.5} \\\\ & \textbf{HiSD (Ours)} & \text{Yes} & \mathbf{87.8} & \underline{86.1} & \underline{74.2} \\\\ \hline \textit{SkillNet} & \text{Qwen2.5-7B-Instruct} & \text{Yes} & 44.5 & 4.8 & 0.4 \\\\ & \text{Skill-GRPO} & \text{No} & 68.8 & 80.2 & 71.5 \\\\ & \text{Skill-GRPO} & \text{Yes} & 88.3 & 88.1 & 76.2 \\\\ & \textbf{HiSD (Ours)} & \text{No} & \underline{90.8} & \underline{89.7} & \underline{82.3} \\\\ & \textbf{HiSD (Ours)} & \text{Yes} & \mathbf{92.2} & \mathbf{91.8} & \mathbf{84.4} \\\\ \hline \end{array}$$ We use the base model's performance gain from skill augmentation as a proxy for library utility. The gains in ALFWorld success / WebShop score / WebShop success are $15.3$ / $0.7$ / $0.0$ for SkillRL (Full), $5.2$ / $0.7$ / $0.0$ for SkillRL (General), and $18.4$ / $2.7$ / $0.4$ for SkillNet. The utility ordering is SkillNet > SkillRL (Full) > SkillRL (General). - **Effectiveness across skill libraries.** Across these three libraries, which differ in source, utility, and task specificity, HiSD matches or exceeds Skill-GRPO with and without inference-time skills on all reported metrics, while requiring no skills at inference. These results show that HiSD's benefits are not confined to a single skill source or to more task-specific libraries. - **Performance retention under weaker skill guidance.** When switching from SkillRL (Full) to SkillRL (General), the success-rate gains from giving Skill-GRPO access to skills at inference diminish, whereas HiSD retains its advantage. HiSD without inference-time skills also exhibits a smaller decline in mean success rate across ALFWorld and WebShop than either Skill-GRPO setting, indicating better performance retention as skill utility decreases. - **HiSD relies less on inference-time skills.** With SkillRL (Full) and SkillNet, HiSD's stronger performance without skills and smaller gains from inference-time skill access, compared with Skill-GRPO, suggest more effective skill internalization. External skills provide further improvements to an already capable policy. Notably, with SkillRL (General), inference-time skills reduce WebShop performance, possibly because less task-specific guidance interferes with decisions grounded in current observations. **References** Xia, Peng, et al. (2026). *SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning.* arXiv:2602.08234. Liang, Yuan, et al. (2026). *SkillNet: Create, Evaluate, and Connect AI Skills.* arXiv:2603.04448. --- Thank you again for your careful reading and insightful questions, which have greatly helped us improve our work. **We hope our responses have addressed your major concerns; if so, we would be grateful if you could consider re-evaluating our submission.** We remain happy to continue the discussion should any questions remain. And we sincerely look forward to your reply. Sincerely, The Authors
AuthorsOpenReview ↗

Follow-up on Our Rebuttal Response

Dear Reviewer Sakk,

We hope this message finds you well. We are writing to kindly follow up on our response to your review for "Submission45757".

We have prepared our response with great care and addressed each of your concerns in detail. We would be very grateful to know whether our clarifications have resolved the questions and concerns you raised. We look forward to your response, and your support would mean a great deal to us.

Thank you again for your careful reading and your hard work.

Best regards,

The authors

Comments on the paper

4 comments
AuthorsOpenReview ↗

Key Clarifications [1/2]

Dear Reviewer Sakk,

Thank you for taking the time to clarify your concern about HiSD's novelty.

In the updated manuscript, we have updated the abstract and introduction to further clarify our work (highlighted in ORANGE\color{orange}{\text{ORANGE}}).


We would like to first make the following key clarifications.

In our work, we examine multi-turn distillation from a gradient perspective and across the two dimensions inherent in multi-turn agent-environment interaction (steps within a trajectory and tokens within each action). Our theoretical and experimental analyses clarify how teacher supervision (OPSD) and task optimization (GRPO) can reinforce each other through adaptation at these two levels, yielding new insights that motivate HiSD's adaptive hierarchical design (progress-aware step weighting and support-adaptive token supervision). Below, we explain how our theory and experiments motivate its design and how it differs from related works (SOD (Zhong et al., 2026) and TIDE (Yu et al., 2026)).

Describing HiSD as a combination of step weighting and token selection leaves out the methodological question that led to both designs: how should the teacher–student disagreement observed throughout a multi-turn rollout be translated into a useful distillation signal for the current decision? HiSD addresses this question by examining how teacher supervision signal interacts with the task update supplied by GRPO. We show theoretically that even with an optimal weight on the OPSD signal, directional conflict between OPSD and GRPO still limits the joint update (Appendix A.5). Reducing its weight can mitigate the conflict, but only by weakening the teacher's influence. Improving the guidance itself therefore requires examining which differences between the teacher and student distributions should shape the gradient being weighted. Through a series of preliminary experiments, we further show that the value of teacher guidance at a decision and the distributional detail needed to convey it need not vary together (Figs. 1, 9, and 10). The former depends on how divergence is distributed across the trajectory, while the latter depends on how broadly the discrepancy is spread across candidate tokens. These findings motivate HiSD's two design choices. At the token level, HiSD uses the current teacher–student disagreement to select which parts of the token distributions contribute to the OPSD gradient. At the step level, it compares accumulated and remaining divergence along the trajectory to adjust the contribution of that gradient relative to GRPO in the joint update.

Prior work, including SOD (Zhong et al., 2026), TCOD (Wang et al., 2026b), and Guided-OPD (Li et al., 2026a), primarily focuses on how history drift can undermine the reliability of teacher supervision in multi-turn interactions. By contrast, HiSD's takeover experiments reveal that where teacher–student disagreement occurs relative to a decision changes what it tells us about the value of intervening there. At the same takeover step, greater accumulated disagreement is associated with lower teacher success, even on tasks the teacher can solve from the initial state (Figs. 1a and 9b). Yet when accumulated disagreement is comparable, restoring the teacher's full interaction budget rescues a larger fraction of tasks on trajectories with greater remaining disagreement (Fig. 9c). In a multi-turn environment, earlier disagreement thus reflects constraints the teacher inherits from actions the student has already taken, while disagreement in decisions still open to intervention can indicate room to improve the outcome. It is this distinction between inherited constraints and remaining opportunities for recovery that motivates HiSD's step weighting. These prior works focus on how past errors can undermine teacher feedback, whereas our detailed experiments show that a large KL divergence need not imply unreliable feedback and, when it reflects remaining disagreement, may instead indicate opportunities for useful teacher intervention. This finding motivated HiSD's progress-aware step-weighting scheme. Moreover, these studies leave another source of supervision heterogeneity unexplored: within each decision, teacher–student discrepancies may be concentrated or dispersed across candidate tokens, requiring different granularities of distributional supervision.


AuthorsOpenReview ↗

About W2 and W3

Dear Reviewer Sakk,

We sincerely appreciate your constructive suggestions in W2 and W3, which have helped us make our study more comprehensive.

We have conducted further experiments on HiSD with inference-time skill access and updated the manuscript with the results and accompanying analyses:

  • Appendix C.2, Table 4. We added an evaluation of HiSD with inference-time skill access. The results show that inference-time skill access further improves HiSD's performance on all three metrics.

  • Appendix C.3, Table 5. We added HiSD's results with inference-time skill access across all three skill libraries. With SkillRL (Full) and SkillNet, HiSD further benefits from skills at inference. Compared with Skill-GRPO, HiSD's smaller performance gap between inference with and without skills, together with its stronger performance without skills, suggests more effective skill internalization. We also discuss a possible explanation for the WebShop decline with SkillRL (General): generic skill guidance may be less relevant to the current decision and interfere with the policy's use of environment observations.


We welcome any further suggestions and would be happy to address any remaining concerns.

Sincerely,

The Authors

AuthorsOpenReview ↗

Key Clarifications [1/2]

Dear Reviewer Grav,

Thank you for your follow-up and for acknowledging that our additional experiments and theoretical analysis provide a clearer justification for the adaptation mechanisms and their complementary roles. We fully understand your remaining reservation regarding methodological novelty, and we would like to address it directly rather than restate our previous points.

In the updated manuscript, we have updated the abstract and introduction to further clarify our work (highlighted in ORANGE\color{orange}{\text{ORANGE}}).


We would like to first make the following key clarifications.

In our work, we examine multi-turn distillation from a gradient perspective and across the two dimensions inherent in multi-turn agent-environment interaction (steps within a trajectory and tokens within each action). Our theoretical and experimental analyses clarify how teacher supervision (OPSD) and task optimization (GRPO) can reinforce each other through adaptation at these two levels, yielding new insights that motivate HiSD's adaptive hierarchical design (progress-aware step weighting and support-adaptive token supervision). Below, we explain how our theory and experiments motivate its design and how it differs from related works (SOD (Zhong et al., 2026) and TIDE (Yu et al., 2026)).

Describing HiSD as a combination of step weighting and token selection leaves out the methodological question that led to both designs: how should the teacher–student disagreement observed throughout a multi-turn rollout be translated into a useful distillation signal for the current decision? HiSD addresses this question by examining how teacher supervision signal interacts with the task update supplied by GRPO. We show theoretically that even with an optimal weight on the OPSD signal, directional conflict between OPSD and GRPO still limits the joint update (Appendix A.5). Reducing its weight can mitigate the conflict, but only by weakening the teacher's influence. Improving the guidance itself therefore requires examining which differences between the teacher and student distributions should shape the gradient being weighted. Through a series of preliminary experiments, we further show that the value of teacher guidance at a decision and the distributional detail needed to convey it need not vary together (Figs. 1, 9, and 10). The former depends on how divergence is distributed across the trajectory, while the latter depends on how broadly the discrepancy is spread across candidate tokens. These findings motivate HiSD's two design choices. At the token level, HiSD uses the current teacher–student disagreement to select which parts of the token distributions contribute to the OPSD gradient. At the step level, it compares accumulated and remaining divergence along the trajectory to adjust the contribution of that gradient relative to GRPO in the joint update.

Prior work, including SOD (Zhong et al., 2026), TCOD (Wang et al., 2026b), and Guided-OPD (Li et al., 2026a), primarily focuses on how history drift can undermine the reliability of teacher supervision in multi-turn interactions. By contrast, HiSD's takeover experiments reveal that where teacher–student disagreement occurs relative to a decision changes what it tells us about the value of intervening there. At the same takeover step, greater accumulated disagreement is associated with lower teacher success, even on tasks the teacher can solve from the initial state (Figs. 1a and 9b). Yet when accumulated disagreement is comparable, restoring the teacher's full interaction budget rescues a larger fraction of tasks on trajectories with greater remaining disagreement (Fig. 9c). In a multi-turn environment, earlier disagreement thus reflects constraints the teacher inherits from actions the student has already taken, while disagreement in decisions still open to intervention can indicate room to improve the outcome. It is this distinction between inherited constraints and remaining opportunities for recovery that motivates HiSD's step weighting. These prior works focus on how past errors can undermine teacher feedback, whereas our detailed experiments show that a large KL divergence need not imply unreliable feedback and, when it reflects remaining disagreement, may instead indicate opportunities for useful teacher intervention. This finding motivated HiSD's progress-aware step-weighting scheme. Moreover, these studies leave another source of supervision heterogeneity unexplored: within each decision, teacher–student discrepancies may be concentrated or dispersed across candidate tokens, requiring different granularities of distributional supervision.


AuthorsOpenReview ↗

Key Clarifications [2/2]

TIDE (Yu et al., 2026) studies on-policy distillation in mathematical reasoning, where the model extends its reasoning trace token by token without receiving new observations from an external environment during generation. However, unlike mathematical reasoning setting, our multi-turn setting requires the agent to generate its current action token by token, conditioned on its previous actions and the feedback it has received from the environment. Executing that action then changes the environment state in which the agent makes its next decision. We ask a distinct question about local predictions in these multi-turn environment interactions. When the teacher and student already have similar next-token distributions during action generation, how many candidate tokens must we retain to capture the remaining corrective signal?

Our paper answers this question through theoretical and empirical analyses of gradient coverage. We find that as the teacher's and student's next-token distributions become closer, a single sampled token provides a less faithful estimate of the full gradient (Fig. 1c), while capturing the same fraction of that gradient typically requires more candidate tokens (Appendix A.4 and Fig. 1d). HiSD therefore uses teacher–student disagreement at each prediction to determine how much teacher probability mass to retain. When the distributions are close, HiSD raises the coverage target so that small discrepancies spread across multiple candidates can contribute to the distillation gradient for the current action (Sections 3.3–3.4). Setting aside task differences and focusing on method design, TIDE uses heuristic gates for single-token and fixed-KK corrections. However, our theoretical and empirical analyses, show how gradient coverage varies with KK and the teacher–student KL divergence, leading us to design an adaptive candidate selection strategy.

Besides, our ablations comparing alternative step-level and token-level strategies (Table 2) and our gradient analyses demonstrate the effectiveness of HiSD's two design choices and offer new insights into how they complement each other in shaping the distillation gradient. Adaptive token selection reduces how often the distillation gradient conflicts with GRPO, while step weighting preferentially preserves the component of that gradient aligned with GRPO at later interaction steps (Fig. 7a,c).

In summary, inspired by the two dimensions inherent in multi-turn agent–environment interaction (steps within a trajectory and tokens within each action), our work approaches multi-turn distillation from a gradient perspective, yielding new findings and insights that set HiSD apart from both SOD and TIDE in its underlying rationale and design.


References

  • Zhong, Q., et al. (2026). SOD: Step-wise On-policy Distillation for Small Language Model Agents. arXiv:2605.07725

  • Wang, J., et al. (2026b). TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agents. arXiv:2604.24005.

  • Li, G., et al. (2026a). On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents. arXiv:2606.15912.

  • Yu, Z., et al. (2026). Mismatch Matters: On-Policy Distillation Beyond Token Agreement. arXiv:2608.09836


We recognize that this is a different kind of novelty than introducing a new RL algorithm, and we do not claim otherwise. If your reservation is specifically that the paper should more clearly foreground this problem-level and optimization-level contribution, we would be glad to further tighten the framing in the final version. If, on the other hand, the concern is that this level of contribution is not sufficient for the venue, we would appreciate any indication of what additional evidence would be needed, and we will do our best to provide it.

Thank you again for your time and for engaging with the substance of our work.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.