acceptodds
Under review as a conference paper at ICLR 2027

Self-Distilled Agentic Reinforcement Learning

Abstract

Reinforcement learning (RL) has emerged as a central paradigm for post-training LLM agents, yet its trajectory-level reward signal provides only coarse supervision for long-horizon interaction. On-Policy Self-Distillation (OPSD) complements RL by introducing dense token-level guidance from a teacher branch augmented with privileged context. However, transferring OPSD to multi-turn agents proves problematic: compounding multi-turn instability destabilizes supervision, while skill-conditioned privileged guidance requires asymmetric treatment for negative teacher rejections may arise from imperfect skills retrieval or utilization. We introduce SDAR (Self-Distilled Agentic Reinforcement Learning), which treats OPSD as a gated auxiliary objective while keeping RL as the primary optimization backbone. SDAR maps detached token-level signals into a sigmoid gate, strengthening distillation on teacher-endorsed positive-gap tokens and softly attenuating negative teacher rejections. Across the Qwen2.5 and Qwen3 families on ALFWorld, WebShop, and Search-QA, SDAR substantially improves over GRPO (up to +9.4% on ALFWorld and +8.7% on Search-QA on Qwen2.5-3B, and +10.2% on WebShop-Acc on Qwen2.5-7B), avoids the instability of naive GRPO+OPSD, and consistently outperforms hybrid RL–OPSD baselines across model scales.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reinforcement learning, agent skills, on-policy distillation, llm agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KAQNOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SDAR, a framework combining GRPO with on-policy self-distillation for multi-turn agents. Instead of applying uniform distillation, SDAR uses a sigmoid gate based on the teacher-student log-probability gap to strengthen positive teacher endorsements and attenuate negative ones. The teacher receives retrieved skills as privileged training context, while the student operates without these skills at inference time. Experiments on ALFWorld, WebShop, and Search-QA show improvements over GRPO and competitive performance against existing hybrid RL-distillation methods.

Critical strengths
  1. The paper addresses a relevant problem in combining RL with privileged self-distillation for multi-turn agents. Keeping GRPO as the primary objective while controlling auxiliary supervision at the token level is a sensible design.

  2. SDAR improves upon GRPO across the evaluated model scales and tasks. The gating ablations and skill retrieval experiments are particularly useful for understanding the method's behavior.

Critical weaknesses
  1. Ambiguity in the baseline implementation. Algorithm 5 defines GRPO+OPSD using a frozen reference policy, whereas SDAR uses a skill-conditioned self-teacher. Table 4 also lists skill retrieval for GRPO+OPSD, making the exact comparison with the ungated baseline unclear.

  2. Limited interpretation of the distillation objective. Proposition 1 shows that the auxiliary loss reduces to weighted likelihood training on student-sampled tokens. The teacher therefore controls the weights rather than directly supervising the full token distribution, and this distinction deserves a clearer explanation.

  3. The reliability of positive teacher endorsements is not fully established. A positive teacher-student gap does not necessarily imply a beneficial action, particularly when the retrieved skill is irrelevant. The retrieval robustness results are helpful, but they do not directly validate this token-level assumption.

Questions for authors
  1. Could the authors clarify whether GRPO+OPSD and the w/o Gating ablation use the same skill-conditioned teacher as SDAR? Algorithm 5 and Table 4 appear inconsistent on this point.

  2. Since the detached SDAR objective is equivalent to weighted likelihood training, how should we interpret its knowledge transfer beyond selectively reinforcing student-generated tokens?

  3. A positive teacher-student gap does not necessarily indicate a beneficial action, as the teacher may still prefer other alternatives. How does SDAR distinguish useful teacher endorsements from misleading positive gaps on student-sampled tokens?

8 comments

We sincerely thank the reviewer for the constructive feedback and for recognizing our sensible token-level gating design, consistent improvements over GRPO across model scales, and informative skill-retrieval and gating ablations. Below we address your concerns and questions in a concise and structured manner.

W1 & Q1: Baseline Specification

We thank the reviewer for this careful reading. We have rewritten Algorithms 3 and 5 and the corresponding entries in Table 4 to describe the baseline more clearly; the revised passages are highlighted in blue in the updated paper.

We would like to clarify that GRPO+OPSD and w/o Gating refer to the same method, and have made explicit that OPSD, GRPO+OPSD, and SDAR share the identical skill-conditioned self-teacher with Keyword Matching retrieval. We have updated Algorithms 3 and 5 to state this teacher explicitly rather than using the generic reference notation πref\pi_{\mathrm{ref}}. So comparing GRPO+OPSD and SDAR isolates the exact benefit of our asymmetric sigmoid gate.

W2 & Q2: Interpreting the Weighted-Likelihood Objective

We appreciate this insightful question and would like to claim that SDAR separates two distinct roles:

  • Detached Single-Sample Estimate: Following TIP [1] and OPSD [2], we estimate the token-level gap with a single student sample and detach the teacher log-probability, avoiding expensive full-vocabulary summation and the self-referential coupling in Proposition 5.
  • Teacher-Student Gap as the Weight: This estimate only fixes how gradients act on the student's support; privileged knowledge enters separately through gt=σ(βΔt)g_t=\sigma(\beta\Delta_t). Since teacher and student share θ\theta, Δt\Delta_t cancels the student prior and isolates what c+c^{+} promotes, acting as a counterfactual evaluator that up-weights endorsed tokens and suppresses unaligned ones under bounded gradients (Proposition 4).

W3 & Q3: Reliability of Positive Teacher Endorsements

We agree that Δt>0\Delta_t>0 represents a comparative endorsement relative to the current student rather than an isolated guarantee of optimality, especially under noisy retrieval. We address this concern from both mechanistic and empirical perspectives:

  • Mechanism Design: First, due to shared weights θ\theta, student prior cancellation ensures that irrelevant skills leave Δt\Delta_t near zero or negative—only skill-specific content actively pushes Δt>0\Delta_t>0. Second, SDAR retains verifier-driven LGRPO\mathcal{L}_{\mathrm{GRPO}} as the primary backbone with λ=0.01≪1\lambda=0.01\ll 1 and bounded weights gt∈(0,1)g_t\in(0,1) from Proposition 4; if a noisy skill produces a misleading positive gap on a failing trajectory, the negative group-relative advantage readily overrides the auxiliary update. We have highlighted both points more clearly in Section 4.5.
  • Token-Level Measurement: Exact-match labels are unavailable at the token level, so we label tokens with GPT-5.6 Luna judge using following prompt:
Task: {task instruction}

History:
turn 1: {action 1} -> {observation 1}
...
turn {t-1}: {action t-1} -> {observation t-1}

Current turn {t}, token #{i} of the action: "{token}"

Question: Given the task and the history above, does this
token contribute toward completing the task? Reply Yes or No.

We run this on Qwen2.5-3B over 50 ALFWorld tasks' rollout under a random skill context and under retrieved skills:

Teacher ContextPositive-Gap TokensHelpful among Positive Gaps (%)
Random skill16%72%
Retrieved skills (KM)34%86%

Retrieved skills thus produce positive gaps that are both more frequent (34%34\% vs. 16%16\%) and markedly more reliable (86%86\% vs. 72%72\% judged helpful). We will extend this judgment across backbones in the appendix.

  • Episode-Level Control: At the episode level where task credit is directly observable, Table 3 of our original paper confirms the same directional effect on Qwen2.5-7B: Soft Gating reaches 85.9%85.9\% on ALFWorld and 82.8%82.8\% on WebShop-Acc over w/o Gating at 80.4%80.4\% and 76.5%76.5\%, whereas inverting the gate in Reverse Soft Gating drops to 76.2%76.2\% and 66.4%66.4\%, a loss of −9.7-9.7 and −16.4-16.4 points.

We hope these clarifications and results address your concerns, and we look forward to your feedback.

[1] TIP: Token Importance in On-Policy Distillation
[2] Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models

AuthorsOpenReview ↗

Kind Reminder

Dear Reviewer KAQN,

Thank you once again for your valuable comments on our submission. Our responses have been available for four days now, and as the discussion phase approaches its end, we would like to kindly confirm whether they have adequately addressed your concerns.

If so, we would be grateful if you would consider raising your rating accordingly. If any questions or points still require clarification, please feel free to let us know. We look forward to your invaluable feedback.

Sincerely,

The Authors

Thank you for the detailed response and additional analysis. Overall, the rebuttal clarifies the method and provides useful supporting evidence. My remaining concern is that the token-level analysis does not directly establish the reliability of the gradient-weighted supervision, so I will maintain my current assessment.

Reviewer_i6ezOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the instability arising when merging on policy self distillation (OPSD) with GRPO for multi turn LLM agents. The authors identify asymmetric teacher student gaps from a privileged self teacher: positive gaps carry reliable supervision, while negative gaps often reflect retrieval noise or trajectory drift rather than incorrect tokens. To avoid destructive gradients from naive joint optimization, SDAR retains GRPO as the main objective and adds a token level gap sigmoid gated distillation loss as an auxiliary regularizer. Evaluations on ALFWorld, WebShop and Search QA with Qwen series models show SDAR outperforms related baselines, especially for small models where vanilla GRPO+OPSD collapses. It also demonstrates skill internalization, since retrieved privileged skills are only used during training and not required at inference.

Critical strengths
  1. The paper targets a practically important failure mode within agent RLVR pipelines, where simply stacking GRPO and on policy self distillation leads to training collapse. The asymmetric trust framing of teacher student gaps offers an intuitive explanation for why naive joint training fails, grounding the technical design in a concrete empirical pathology.
  2. The proposed method is lightweight and non intrusive, preserving the original GRPO objective as the primary optimization backbone. This design choice lowers adoption cost, as it can be plugged into existing RL training workflows without rewriting core RL logic.
  3. The gap based sigmoid gating naturally implements asymmetric trust for privileged self distillation signals. It amplifies supervision when the self teacher endorses a token and softly suppresses noisy negative guidance, aligning the weight of distillation loss with signal reliability.
Critical weaknesses
  1. All primary experimental results are reported from single training runs, with no standard deviations across multiple random seeds. LLM reinforcement learning is widely known for high run to run variance induced by trajectory sampling, initial parameter states and reward stochasticity. Without multi seed statistics, it remains difficult to distinguish consistent performance improvements from lucky random outcomes, weakening the evidential foundation of the main claims.
  2. Evaluations are restricted to three interactive agent benchmarks, and retrieved skills serve as the sole type of privileged context examined in the paper. Privileged signals in self distillation can take many other forms, such as verified solutions, off policy expert trajectories or intermediate rationales. Since the paper does not test these alternatives, it is hard to judge whether the gap gating mechanism is broadly applicable or only well suited to skill retrieval setups.
  3. The practical effectiveness of the gated distillation is fundamentally bounded by the quality of the privileged self teacher. When the teacher persistently generates flawed or misleading guidance, even carefully designed gating cannot recover trustworthy supervision. The paper lacks systematic characterization of this failure regime, including quantitative breakdowns of teacher quality, threshold analysis, or mitigation strategies for severely degraded teacher signals.
  4. The method is only evaluated on interactive multi turn agent environments and is not validated on standard single turn reasoning benchmarks like MATH or GSM8K. Single turn reasoning exhibits different dynamics: trajectories are shorter, distribution shift is less pronounced, and the source of teacher student mismatch may differ substantially. This missing validation leaves open the question of whether SDAR’s benefits generalize beyond agentic interactive settings.
  5. While the authors show SDAR outperforms naive GRPO+OPSD, the discussion does not sufficiently disentangle whether gains primarily come from asymmetric gating, the specific reverse KL distillation formulation, or the low weighting coefficient applied to the auxiliary loss. The current ablation suite isolates gating variants but does not test alternative simple heuristics for downweighting negative gaps, such as hard clipping or fixed thresholding, which would help demonstrate the added value of the sigmoid gap gating.
Questions for authors

Q1: Could you provide multi seed statistics for your primary experiments to rule out variance driven improvements?

Q2: Would the proposed gap gating transfer to other forms of privileged context beyond retrieved skills?

Q3: How does SDAR behave when the privileged self teacher is persistently low quality, and do you have strategies to mitigate such failures?

Q4: Have you considered validating the method on common standalone reasoning benchmarks such as MATH or GSM8K?

Q5: Can you compare your learned sigmoid gap gating against simpler baselines such as fixed thresholding or hard clipping of negative gaps, to highlight the unique advantage of your gated formulation?

4 comments

We sincerely thank the reviewer for the positive feedback and for recognizing our practical problem formulation, intuitive asymmetric trust framing, and lightweight non-intrusive design. Below we provide detailed responses and further experimental analysis.

W1 & Q1: Multi-Seed Statistical Validation

To rule out run-to-run variability, we report 4-seed statistics (mean ± std) across two representative settings:

SettingGRPOGRPO+OPSDSDAR (Ours)
Qwen2.5-3B76.23 ± 1.5979.78 ± 2.0582.93 ± 2.12
Qwen3.5-9B61.72 ± 1.3670.74 ± 2.0477.50 ± 1.92

SDAR consistently surpasses both baselines across seeds, indicating that the gains are systematic. Due to limited compute, we report these two representative settings here and will run the full experiments in the near future.

W2 & Q2: Generalization to Other Privileged Contexts

We would like to claim first the reason why we use skills as privilege: In long-horizon agentic tasks, a final reference solution only offers local result regardless of detailed information, while long expert trajectories are difficult for small LLM to absorb and generalize poorly across task instances; we therefore adopt concise, high-level skills with stronger transferability as our default privileged context c+c^{+}. Following your suggestion, we evaluate two more Privileged context:

Privileged Context c+c^{+}ALFWorld (3B)
w/o OPSD (GRPO)75.0
Random Context (Random)77.2
Retrieved Skills (KM, Ours)84.4
Reference Solution78.7
Expert Trajectory78.0

Our gate is agnostic to how c+c^{+} is obtained, since Propositions 1 to 4 assume only token-level log-probabilities from a privileged branch. The original paper already shows on Qwen2.5-7B that performance moves monotonically with the informativeness of c+c^{+} across five retrieval fidelity levels, which is the behaviour the mechanism predicts.

W3 & Q3: Robustness to Low-Quality Self-Teachers

We have already analyzed and evaluated this regime in our original paper (Propositions 1, 2, and 4; Section 4.4, Table 2), and we summarize again as follows:

  • The auxiliary gradient is strictly bounded in magnitude. When the self-teacher persistently produces misleading guidance (Δt≪0\Delta_t\ll 0), the gate decays toward 00, automatically suppressing the auxiliary loss and falling back to the primary GRPO update rather than injecting destructive gradients.
  • Empirical Verification: As demonstrated in Table 2, SDAR never collapses as teacher quality degrades: accuracy stays within 0.60.6 points of the GRPO baseline even when the privileged context carries no skill content, and the margin over GRPO widens monotonically as the context becomes more informative. We will report the same sweep on Qwen2.5-3B in the appendix. In practice, the logged gate activation ratio in Section 4.3 also serves as a zero-cost runtime diagnostic: a persistent drop indicates an uninformative teacher and can trigger automatic annealing (λ→0\lambda\to 0).

W4 & Q4: Single-Turn Reasoning

Our study aims to solve agentic problems of OPSD, so single-turn reasoning remains outside our main scope. To demonstrate generaliaztion to other domains, we conduct a preliminary experiment on Qwen3-1.7B trained on OpenThoughts supports this view (ground truth as self-teacher's privilege):

MethodMATH-500
GRPO74.4
SDAR (Ours)77.2

The gains are consistent to multi-turn agentic setting.

W5 & Q5: Factor Disentanglement and Simpler Gating Heuristics

We have already disentangled these factors in Section 4.5 of our original paper, and we summarize the conclusions again as follows:

  • Factor Disentanglement: Section 4.5 isolates each component individually: the distillation objective (Figure 9, sampled-token surrogate vs. forward KL and JSD under identical gating), the auxiliary coefficient (Figure 8, λ∈{0.001,0.01,0.1}\lambda\in\{0.001,0.01,0.1\}), and the gate sharpness (Figure 7, β∈{0,1,5,10}\beta\in\{0,1,5,10\}). GRPO+OPSD and SDAR further share the identical self-teacher, surrogate, and λ=0.01\lambda=0.01, isolating the gate itself.
  • Simpler Heuristics: Table 3 already covers binary Hard Gating, one-sided half-gates, and Reverse variants. Extending it to fixed thresholding and piecewise linear clipping at a matched active ratio (0.34 here) confirms the same conclusion on Qwen2.5-7B:
Gating MechanismALFWorld (%)
w/o Gating80.4
Fixed Thresholding (Δt>τ∗\Delta_t>\tau^\ast)81.2
Piecewise Linear Clipping82.6
Soft Gating (SDAR)85.9

From the results above, the conclusion is convincing: the gain is not from distilling more tokens. Smooth gate that supresses negative rejections and amplifies positive endorsements truly works in our method.

We hope these results address your concerns, and we look forward to your feedback.

AuthorsOpenReview ↗

Kind Reminder

Dear Reviewer i6ez,

Thank you once again for your valuable comments on our submission. Our responses have been available for four days now, and as the discussion phase approaches its end, we would like to kindly confirm whether they have adequately addressed your concerns.

If so, we would be grateful if you would consider raising your rating accordingly. If any questions or points still require clarification, please feel free to let us know. We look forward to your invaluable feedback.

Sincerely,

The Authors

AuthorsOpenReview ↗

Kindly Reminder

Dear Reviewer i6ez,

Thank you again for your thoughtful comments on our submission. With only a few hours remaining in the discussion period, we hope our detailed responses and revisions have fully addressed your concerns.

In light of these clarifications, we would be grateful if you would consider revising your rating upward. We appreciate your time and careful consideration and welcome any final feedback.

Sincerely,

The Authors

AuthorsOpenReview ↗

Kindly Reminder

Dear Reviewer i6ez,

Thank you again for your thoughtful comments on our submission. With only a few minutes remaining in the discussion period, we hope our responses and revisions have adequately addressed your concerns.

If so, we would be grateful if you would consider raising your rating accordingly. We sincerely appreciate your time and careful consideration, and welcome any final feedback.

Sincerely,

The Authors

Reviewer_xXGeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SDAR, which trains multi-turn language-model agents by combining GRPO with a skill-informed auxiliary objective. A privileged branch of the current policy receives retrieved skills, while the student generates trajectories without them. The teacher-minus-student log-probability gap is mapped through a detached sigmoid to weight the likelihood of student-generated tokens. This gives larger relative weights to tokens favored by the privileged branch, while retaining the original GRPO objective. Skills are not required at inference.

Experiments cover three Qwen backbones on ALFWorld, WebShop, and seven Search-QA datasets. SDAR improves Qwen2.5-3B ALFWorld success from GRPO's 75.0% to 84.4% and Qwen2.5-7B WebShop accuracy from 72.6% to 82.8%. The latter also exceeds RLSD's 77.3%. Retrieval controls, gating ablations, and training diagnostics examine the contribution of skill information and token weighting.

Critical strengths

1.The method produces useful improvements across several agent settings without requiring skills at deployment. Table 1 reports substantial gains over GRPO on ALFWorld and WebShop, with Qwen2.5-7B WebShop accuracy also improving by 5.5 percentage points over RLSD. Search-QA includes five evaluation datasets outside the NQ/HotpotQA training sources, providing evidence of transfer across datasets. Improvements over the strongest hybrids are not uniform, but the results support the practical usefulness of the overall recipe.

2.The ablations investigate both skill content and token weighting. Table 2 includes empty and random skill contexts alongside more informed retrieval strategies. On Qwen2.5-7B ALFWorld, Empty, Random, and keyword matching achieve 80.7%, 83.1%, and 85.9%, respectively. Table 3 compares soft, hard, reversed, and one-sided gates; the full soft gate achieves 85.9%/82.8% on ALFWorld/WebShop versus 80.4%/76.5% without gating. These controls provide useful evidence beyond comparisons with plain GRPO.

Critical weaknesses

1.The methodological contribution and its explanation are not sufficiently distinguished from prior work. Privileged self-distillation, skill internalization, and combining RL with distillation are established ingredients acknowledged by the paper. The identifiable addition is bounded, asymmetric token weighting within this framework. Section 2 provides only a brief characterization of related methods, without a sufficiently precise comparison of objectives and weighting mechanisms. In particular, TIP is cited as inspiration for token importance, and TCOD addresses multi-turn distillation stability, but their relationship to the proposed gate is not developed. Skill-SD and RLSD are evaluated, which is valuable; the remaining issue is establishing the specific contribution beyond these existing approaches. The mechanistic explanation also needs qualification. For student-sampled tokens at a fixed context, (mathbb{E}{p}[log q-log p]=-D{mathrm{KL}}(pVert q)leq0), regardless of teacher competence. Thus, the negative mean gap in Section 4.3 does not itself show that teacher guidance is harmful. Appendix A explicitly derives a weighted self-likelihood update, whose gradient differs from ordinary reverse-KL minimization. The method can still be effective, but the reverse-KL mode-seeking explanation in Section 4.5 does not directly establish why this surrogate works.

2.The baseline specification leaves the source of the stability gains unresolved. The main text describes privileged self-distillation, and Table 4 lists skill retrieval for OPSD and GRPO+OPSD. In contrast, Appendix B's Algorithms 3 and 5 use a frozen reference without skill context. Their sampled log-ratio losses also differ from SDAR's ungated surrogate; the pseudocode does not specify how differentiation through the sampling distribution is handled. Consequently, the reported comparison may change the teacher or gradient estimator as well as the gate. This could be a documentation issue, but it must be resolved before attributing the recovery from OPSD collapse specifically to adaptive weighting. Furthermore, (eta=0) yields a constant gate of 0.5, so the effective auxiliary strength needs to be accounted for when interpreting the no-gate comparison. Unreported run-to-run variability also limits conclusions from small margins such as 45.1% versus 44.6% Search-QA accuracy.

Questions for authors

1.What is SDAR's principal methodological distinction from the cited token-importance and skill-distillation approaches, particularly TIP and Skill-SD?

2.Which teacher and loss were actually used for GRPO+OPSD, given the discrepancy between Table 4 and Algorithm 5?

3.Does the gating advantage persist when the teacher and sampled surrogate are held fixed and the ungated baseline is tuned to a comparable auxiliary strength?

5 comments
AuthorsOpenReview ↗

Response 1/2

We sincerely thank the reviewer for the constructive comments and thoughtful suggestions. As noted, our work targets the challenge of stabilizing OPSD in agentic RL, achieving substantial gains without requiring skills at deployment. Below we address your concerns and questions in a concise and structured manner.

W1-1 & Q1: Methodological Novelty and Algorithmic Distinction

We respectfully clarify that our contributions go substantially beyond combining existing ingredients:

  • Our motivation starts naturally: As shown in our paper, multi-turn environments raise two fundamental obstacles for OPSD (Section 1, Figures 1 & 2), which we briefly reiterate: (1) Compounding Multi-Turn Drift, where early deviations render the uniform token-level supervision increasingly unreliable; and (2) Asymmetric Trust in Privileged Self-Guidance, where the teacher is not an independently stronger model but the same policy conditioned on the retrieved skill context c+c^{+}. When Δt>0\Delta_t>0, the skill provides a positive endorsement of a valid on-policy behavior; when Δt<0\Delta_t<0, the rejection frequently reflects noisy skill retrieval, imperfect skill grounding, or multi-turn drift rather than a genuinely undesirable token.

  • Algorithmic Distinction from Prior Work:

    • Unlike TIP [1], which uses an unsigned magnitude that directly filters distillation tokens, SDAR introduces a signed directional gate that softly encourages positive teacher endorsements and suppresses negative teacher rejections.

    • Unlike RLSD [5], which injects teacher divergence directly into the sequence-level RL advantage and induces early-training instability (Figure 1, right), SDAR decouples distillation as an orthogonal auxiliary objective, preserving the unbiasedness of the verifier-driven GRPO policy gradients.

    • Unlike TCOD [2] and HDPO [3], which rely on rigid temporal schedules or trajectory-depth thresholds, SDAR induces an online token-level curriculum. Moreover, Skill-SD [4] optimizes an importance-weighted K3K_3 loss ρt(exp⁡(Δt)−1−Δt)\rho_t(\exp(\Delta_t)-1-\Delta_t) whose gradient multiplier scales exponentially with Δt\Delta_t, whereas SDAR strictly bounds the auxiliary gradient multiplier in gt∈(0,1)g_t\in(0,1) (Proposition 4) to prevent gradient explosion. To verify Proposition 4 against Skill-SD's exponential K3K_3 scaling, we compare the per-token gradient multiplier, the actor gradient norm, and the success rate on Qwen2.5-3B ALFWorld:

      MethodCoef λ\lambdaToken Multiplier (Mean / P99)Aux Grad Norm (Mean / Max)Grad Clip Spike Rate (%)ALFWorld (3B)
      Skill-SD (ρtK3\rho_t K_3)0.001 (default)1.84 / 14.60.42 / 3.8511.473.4
      Skill-SD (ρtK3\rho_t K_3)0.010 (matched)2.61 / 28.43.91 / 34.264.2 (collapses)41.4
      SDAR (Ours)0.010 (default)0.34 / 0.960.18 / 0.410.884.4

      As a result, Skill-SD suffers from severe tail spikes (P99 reaching 28.428.4) due to exponential K3K_3 scaling and collapses (41.4%41.4\%) at λ=0.01\lambda=0.01.

AuthorsOpenReview ↗

Response 2/2

W1-2: Mechanistic Qualification

  • Sign Asymmetry and Empirical Harm: We agree that Eyt∼πθ[Δt]=−DKL(πθ ∥ πT)≤0\mathbb{E}_{y_t\sim\pi_\theta}[\Delta_t]=-D_{\mathrm{KL}}(\pi_\theta\,\|\,\pi_T)\le 0 holds at any fixed context, so a negative mean alone does not imply harmful guidance. Instead, Section 4.3 highlights the sign distribution: over 50%50\% of student-sampled tokens have Δt<0\Delta_t<0, causing uniform distillation to spend most of its gradient budget on negative-gap tokens. To verify at the token level, we run this on Qwen2.5-3B over 50 ALFWorld tasks' rollouts under a random skill context and under retrieved skills, with GPT-5.6 Luna judge:
Teacher ContextPositive-Gap TokensHelpful among Positive Gaps (%)
Random skill16%72%
Retrieved skills (KM)34%86%

Positive gaps cover only a minority of sampled tokens, yet the ones produced by retrieved skills are markedly more reliable than those produced by an uninformative context (86%86\% vs. 72%72\% judged helpful). Table 3 confirms the same direction: Soft Gating reaches 85.9%85.9\% on ALFWorld and 82.8%82.8\% on WebShop-Acc over w/o Gating at 80.4%80.4\% and 76.5%76.5\%, while inverting the gate drops to 76.2%76.2\% and 66.4%66.4\%; against a ratio-matched Random Gate, which activates the same fraction of tokens, SDAR still gains +4.9+4.9 and +5.5+5.5. The benefit thus comes from the direction of token selection rather than from distilling fewer tokens.

  • Mode-Seeking Behavior: We acknowledge that we did not investigate the theoretical gap between our detached surrogate and exact reverse-KL gradients in depth, as our primary contribution lies in asymmetric directional gating rather than the base surrogate itself. We directly follow prior OPD works such as TIP [1] and OPSD [6], adopt a detached sampled-token surrogate to avoid full-vocabulary summation and self-referential coupling (Proposition 5). Even so, while our update is not exact reverse-KL, it still induces effective mode-seeking on the student's support: under softmax normalization, up-weighting teacher-endorsed tokens concentrates probability mass onto teacher-preferred modes among student-sampled tokens, whereas forward KL and JSD spread mass onto unsampled vocabulary tails where the self-teacher is poorly calibrated (Figure 9).

W2 & Q2: Baseline Specification

We thank the reviewer for this careful reading. We have rewritten Appendix B and the corresponding entries in Table 4 so that the baseline is described more explicitly; the revised passages are highlighted in blue in the updated paper.

  • First of all, OPSD, GRPO+OPSD, and SDAR share the identical skill-conditioned self-teacher with Keyword Matching skill retrieval. We now update Algorithms 3 and 5 to demonstrate this teacher more explicitly.
  • We would like to claim that GRPO+OPSD and w/o Gating refer to the same method, i.e., the unweighted (ungated) sampled-token surrogate with detached teacher log-probabilities. The β=0.0\beta=0.0 entry in Table 4 is now spelled out as "no gating", consistent with Figure 7, where β=0\beta=0 denotes uniform distillation. By Proposition 2, gradients flow exclusively through log⁡πθ(yt∣st)\log\pi_\theta(y_t\mid s_t), requiring no differentiation through the sampling distribution.

Q3-1: Matched Auxiliary Strength Control

Following your advice, we conduct the experiments below:

MethodGateNominal λ\lambdaEffective λeff\lambda_{\mathrm{eff}}ALFWorld (7B)
GRPO+OPSDgt≡1g_t\equiv 10.01000.010080.4
GRPO+OPSDgt≡0.5g_t\equiv 0.50.00500.005081.2
GRPO+OPSDgt≡1g_t\equiv 1, matched strength0.00340.003481.5
SDAR (Ours)gt=σ(βΔt)g_t=\sigma(\beta\Delta_t)0.01000.003485.9

Tuning the uniform auxiliary strength in GRPO+OPSD still falls short of SDAR (81.5%81.5\% vs. 85.9%85.9\%), confirming that SDAR's gains arise from selective token-level credit assignment rather than overall loss scaling.

Q3-2 & W2: Multi-Seed Statistical Validation

To rule out run-to-run variability, we report 4-seed statistics (mean ± std) across two representative settings:

SettingGRPOGRPO+OPSDSDAR (Ours)
Qwen2.5-3B76.23 ± 1.5979.78 ± 2.0582.93 ± 2.12
Qwen3.5-9B61.72 ± 1.3670.74 ± 2.0477.50 ± 1.92

SDAR consistently surpasses both baselines across seeds, indicating that the gains are systematic. Due to limited compute, we report these two representative settings here and will run the full experiments in the near future.

We are looking forward to your feedback if any concerns arise.

[1] TIP: Token Importance in On-Policy Distillation
[2] TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agents
[3] HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation
[4] Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents
[5] Self-Distilled RLVR
[6] Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models

AuthorsOpenReview ↗

Kind Reminder

Dear Reviewer xXGe,

Thank you once again for your valuable comments on our submission. As the discussion phase approaches its end, we would like to kindly confirm whether our responses have adequately addressed your concerns.

If so, we would be grateful if you would consider raising your rating accordingly. If any questions or points still require clarification, please feel free to let us know. We look forward to your invaluable feedback.

Sincerely,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal

Dear Reviewers, AC, SAC, and PC,

We thank the reviewers for their time and thoughtful feedback. We believe we have addressed all concerns raised in the reviews through clarifications, additional experiments, and revisions. Below, we summarize the main evidence and the current discussion status for the Area Chair’s consideration.

Strengths Noted by the Reviewers

  • Consistent improvements across agent benchmarks and model scales, without requiring skills at inference time (xXGe, KAQN).
  • A lightweight design that preserves GRPO as the primary objective while controlling auxiliary supervision through token-level gating (i6ez, KAQN).
  • Informative gating ablations and retrieval controls that help explain the method’s behavior (xXGe, KAQN).

Main Concerns and Our Responses

  1. Methodological distinction and baseline consistency (xXGe, KAQN). We clarified SDAR’s bounded, asymmetric token weighting and its distinction from prior methods, and further explicitly stated the teacher settings and sampled-token loss, isolating the contribution of gating.
  2. Effectiveness of gradient weighting (xXGe, i6ez, KAQN). We clarified how privileged information enters through detached token weights. On Qwen2.5-7B ALFWorld, Soft Gating achieves 85.9%, outperforming random gating, Hard Gating, Piecewise Linear Clipping, and uniform weighting with a matched average coefficient. Reversing the gate reduces success to 76.2%, further supporting the importance of weighting direction.
  3. Run-to-run variability (xXGe, i6ez). We added four-seed results on two representative backbones, across three methods. Full multi-seed coverage remains future work.
  4. Generality and teacher quality (i6ez). We also conducted additional experiments with more privilege information (reference solutions and expert trajectories), and further demonstrate SDAR's generality on MATH-500. We also discussed retrieval-quality controls and the limitations of an imperfect self-teacher.

Discussion Status and Request for Follow-up

  • Reviewer KAQN acknowledged that the additional gating controls provide stronger evidence, stated that there were no major issues overall, and raised the rating to 3 (Weak acceptance). We also answered the further question about separating token-level allocation from auxiliary gradient magnitude, raised by KAQN.

  • Reviewers xXGe and i6ez have not yet responded to our point-by-point replies posted on November 12, despite follow-up reminders on November 16 and 18. Their current assessments therefore do not indicate whether they have considered the additional experiments and corrections.

We respectfully ask the AC to consider this evidence alongside the original reviews. We would also appreciate a brief follow-up from xXGe and i6ez indicating whether our responses address their concerns, or identifying any specific issue that remains. If the concerns have been resolved, we would be grateful if their assessments could be updated accordingly.

Sincerely, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.