acceptodds
Under review as a conference paper at ICLR 2027

DL-OPSD: Dual-Level Selective On-Policy Self-Distillation for Skill Internalization

Abstract

On-policy self-distillation combined with reinforcement learning offers a route to internalizing external skills in language agents, removing the need for skill retrieval at inference. However, local teacher–student prediction gaps alone are insufficient to determine which behaviors deserve reinforcement through distillation. We find that, when teacher–student gaps are comparable, original actions yield positive average gains in continuation success over student-sampled alternatives on positive-advantage trajectories, but negative gains on negative-advantage trajectories. This observation suggests that trajectory outcomes inform supervision selection beyond local prediction gaps, motivating Dual-Level Selective On-Policy Self-Distillation (DL-OPSD). At the trajectory level, DL-OPSD applies auxiliary distillation only to trajectories with positive group-relative advantage, while retaining reinforcement learning updates across all trajectories. At the token level, it selects positions where the teacher assigns a higher probability to the sampled token than the student does, and weights the supervision according to student uncertainty and teacher certainty. Skills are provided only to the teacher during training. Across ALFWorld, Search-QA, and WebShop, DL-OPSD outperforms GRPO and SDAR on the primary task metrics, improving Qwen3-1.7B's ALFWorld success rate from 60.9% under GRPO to 70.3%. Controlled experiments show that trajectory selection improves gradient alignment between distillation and reinforcement learning, while token-level weighting further improves task performance, supporting the complementary roles of the two levels.

Primary area:
reinforcement learning
Keywords:
LLM agents, on-policy self-distillation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_jrAdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper finds in on-policy self-distillation (OPSD), distilling tokens from positive-advantage trajectories is more valuable than distilling tokens from negative-advantage trajectories. To this end, the authors propose DL-OPSD, which operates at two levels. At the trajectory level, it applies OPSD distillation only to positive-advantage trajectories while train with GRPO on all trajectories at the same time. At the token level, it distills only tokens supported by the teacher, weighting them according to teacher certainty and student uncertainty. Experiments across ALFWorld, Search-QA, and WebShop show that DL-OPSD consistently outperforms GRPO, and ablation studies validate the effectiveness of each component.

Critical strengths
  1. This paper is well motivated and proposes a clean yet effective method that tackles a common and important problem in on-policy self-distillation (OPSD).

  2. The experimental section of this paper is solid. The authors conduct thorough ablation studies that justify the design choices made for trajectory selection and token weighting.

Critical weaknesses
  1. DL-OPSD shares similarities with several existing methods, which limits its novelty. For example, SIRI[1] also selects which trajectories to distill based on trajectory-level utility. SDAR[2] also relies on the teacher–student gap, strengthening distillation on tokens with a positive gap.

[1] Siri: Self-Internalizing Reinforcement Learning with Intrinsic Skills for LLM Agent Training

[2] Self-Distilled Agentic Reinforcement Learning

  1. The logic behind Figure 1(b) is puzzling. Actions in positive-advantage trajectories would naturally exhibit higher continuation success gains than actions in other trajectories, precisely because positive-advantage trajectories are themselves successful ones. As such, this observation does not seem to establish that distilling tokens from positive-advantage trajectories is more beneficial. Moreover, according to Appendix E.2, Figure 1(b) is computed from the step-300 DL-OPSD checkpoint, which is odd: Figure 1(b) itself serves as the motivation for DL-OPSD, so why the authors used a fully trained DL-OPSD model to validate the motivation of DL-OPSD?
Questions for authors
  1. In the gradient-alignment experiment, only the gradient alignment of the first decision response of each trajectory is measured. Why not compare over more tokens?

  2. Several ablation studies continually train models from the step-150 GRPO checkpoint, but the paper offers no corresponding explanation for this choice. Why not train them by DL-OPSD components from scratch?

Reviewer_SRD4OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes DL-OPSD, a method for internalising external skills into language agents by combining GRPO with an auxiliary on-policy self-distillation loss, where the teacher is the same policy conditioned on retrieved skills. Distillation is applied selectively at two levels: only to trajectories with positive advantage, and, within them, only to tokens that the skill-conditioned teacher supports, weighted by student uncertainty and teacher confidence. The authors claim that this selection improves gradient alignment between the distillation and RL objectives and enables effective skill internalisation without skill retrieval at inference. The method is evaluated on ALFWorld, Search-QA and WebShop with Qwen models of 1.7B–7B parameters, where it is reported to outperform GRPO and prior self-distillation baselines.

Critical strengths
  1. The paper is well written and easy to follow. The proposed selection adds little overhead to standard on-policy self-distillation: the trajectory gate reuses the GRPO advantages, and the token weights are computed from the student and teacher distributions that are already needed for the distillation loss.
  2. The authors analyse how well policy-gradient and distillation gradients agree and show that outcome-based trajectory selection increases their alignment (Fig. 4). Trajectory-level and token-level selection are ablated separately (Table 2 and Tables 3, 5), so the effect of each part can be assessed on its own.
  3. DL-OPSD is compared with many baselines, covering training-free, RL, self-distillation and hybrid methods.
Critical weaknesses
  1. No assessment of statistical significance. All main results (Table 1) and all analysis experiments (Fig. 4a, Tables 2–5) are reported from a single training run. The only multi-seed results (Table 8) cover one model on one benchmark (Qwen3-1.7B, ALFWorld), and even there the standard deviation across seeds reaches several points (from 1.47 to 5.12). Results for the same baselines also vary considerably across papers: with the same environments and backbones, SDAR (Lu et al., 2026a) reports GRPO at 75.0% on ALFWorld with Qwen2.5-3B and 38.3% WebShop accuracy with Qwen3-1.7B, compared with 90.6% and 79.7% in this paper. Given this level of variation, most of the reported gains cannot be considered statistically significant. This applies to the main comparison: gains over GRPO are mostly within 1–5 points for Qwen2.5-3B and Qwen2.5-7B on all benchmarks, and for Qwen3-1.7B on Search-QA (42.4 vs. 40.6) and WebShop (81.3 vs. 79.7 accuracy). It also applies to the analysis experiments; for example, in Fig. 4a positive-advantage selection improves over all trajectories by 0.8 points (45.3 vs. 44.5), which corresponds to a single task out of 128. All ablations are also run only on Qwen3-1.7B. Even if the gain on ALFWorld with Qwen3-1.7B is significant, an improvement on a single model and a single benchmark is not enough to support the general claims of the paper. GRPO, the baseline against which most comparisons in the paper are made, is also absent from Table 8.

  2. Evaluation protocol favours noisy methods. The paper reports the best validation checkpoint within 300 updates (Section 4.1, Appendix C.5). No separate test split is mentioned, so the same 128 tasks (ALFWorld, WebShop) appear to be used both for checkpoint selection and for reporting. With a validation interval of 5 steps, each reported number is a maximum over roughly 60 evaluations. This biases the results upwards and benefits methods with higher variance. Fig. 3a illustrates this: the DL-OPSD and GRPO+OPSD training curves plateau at roughly 60% and 55%, while Table 1 reports 70.3% and 61.7%. The paper also does not specify the decoding temperature, whether sampling or greedy decoding is used, or the number of evaluation samples per task. As far as can be understood from the text, each task is evaluated once. With non-zero temperature, repeated evaluation of the same checkpoint may give a different result.

  3. Skill internalisation is not demonstrated. Skill internalisation is presented as a main motivation of the work, but the results do not show that the gains come from skills. On ALFWorld with Qwen3-1.7B, the setting with the largest reported gain, Skill-Prompt* improves over Vanilla only on the Clean task (29.0 vs. 9.7), and the scores are identical on Heat, Cool and Pick2. Yet DL-OPSD matches GRPO on Clean (60.7 vs. 60.7) and obtains its largest gains on Cool (87.5 vs. 62.5) and Pick2 (57.9 vs. 36.8), where skills give no benefit even when provided in the prompt. More generally, Skill-Prompt* is close to Vanilla on average, and Skill-GRPO* (evaluated with skills) is below GRPO (e.g. 82.8 vs. 90.6 on ALFWorld with Qwen2.5-3B). This suggests that skills provide little useful information in these environments. The decrease of the teacher–student gap in Fig. 3b is also consistent with the student learning to ignore the skill context, not only with internalising it. The paper provides no examples of the skills used, no information on the size of the skill library or the relevance of keyword-based retrieval, and no behavioural comparison between the trained policy and the base model with skills.

  4. Several design choices are not studied. (a) According to Table 7, DL-OPSD differs from SDAR and GRPO+OPSD not only in the selection rules but also in the objective: forward KL instead of a reverse-KL-derived loss. No ablation isolates the effect of the loss, so part of the improvement may come from this change rather than from the proposed selection. (b) Negative-advantage trajectories are discarded entirely from distillation, including positions where the teacher disagrees with an action from a failed trajectory (A < 0, Δ\Delta < 0). In those positions the RL and distillation signals point in the same direction, and this case is not studied. Table 2 shows that distillation on negative-advantage trajectories outperforms distillation on all trajectories on Search-QA (39.8 vs. 36.7) and WebShop (80.4 vs. 68.8). This does not fit the motivation that failed trajectories harm distillation.

Questions for authors
  1. It would be helpful to see multi-seed results that also include GRPO, ideally for more than one backbone and benchmark. Would the authors be able to provide these?
  2. Could the authors clarify how evaluation is performed: the decoding temperature, whether sampling or greedy decoding is used, and how many samples are drawn per task? It would also be useful to see results averaged over the last several evaluations, or on a held-out test set, in addition to the best checkpoint.
  3. I am curious how DL-OPSD would perform if the teacher were conditioned on random or unrelated text instead of a skill. This experiment could help to show that the gains come from skill transfer and not from a more general regularisation effect.
  4. Could the authors share a few examples of the skills used and the size of the skill library? A short behavioural comparison would also be valuable, for instance whether skill-specific action patterns appear after training that are not present in Vanilla.
  5. Have the authors tried DL-OPSD with a reverse-KL objective? This would help to separate the effect of the selection rules from the effect of the loss.
  6. It would be interesting to see an analysis over both the sign of A and the sign of Δ\Delta, including the A < 0, Δ\Delta < 0 case. Do the authors have any results or intuition on this?
  7. Fig. 4a and 4b appear to be cropped on the right side (the “Positive-advantage” bar in 4a; the legend and last checkpoint in 4b). Could the authors please check these figures?
Reviewer_75A2OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies skill internalization for LLM agents: training a policy so that it no longer needs retrieved natural-language skills at inference. It builds on the combination of GRPO with on-policy self-distillation (OPSD), where the same model acts as a teacher (prompted with a retrieved skill) and as a student (without the skill), and the student is distilled toward the teacher's next-token distribution on its own rollouts. The authors argue that because the teacher is conditioned only on the prefix, it is blind to the trajectory outcome, so distillation on failed trajectories can reinforce tokens that GRPO is pushing down.

DL-OPSD selects distillation at two levels: (1) a trajectory-level gate that applies distillation only to trajectories with positive group-relative advantage (GRPO still uses all trajectories), and (2) a token-level weight.

Critical strengths
  1. The paper addresses a relevant problem. Removing the inference-time dependence on skill retrieval is useful and self-distillation is a cheap way to attack the problem.
  2. The trajectory gate and the token weighting are easy to implement and to reason about; Algorithm 1 is clear.
  3. DL-OPSD improves over GRPO and the hybrid baselines in all nine model/benchmark settings, and the student without skills matches or exceeds skill-conditioned baselines.
Critical weaknesses
  1. The core motivation is that the teacher "is not conditioned on the trajectory outcome." This is not an empirical finding but a property of the design: any teacher that predicts token t from the prefix c_{i,t} cannot know the episode's outcome. The interesting question is how to supply outcome information during supervision.
  2. Limited novelty relative to closely related work The two ingredients are close to prior work cited by the paper itself:
  • Outcome-based routing: SRPO (Li et al., 2026a) "routes samples between group-relative policy optimization and self-distillation according to their task outcomes, with teacher entropy further modulating the distillation signal" (Appendix A.3). This is very close to the trajectory gate plus teacher-certainty weighting, yet SRPO is not included as a baseline.
  • Token-level weighting: TIP (Xu et al., 2026) already uses student entropy and teacher–student divergence for token importance; SDAR and RLSD use teacher–student gaps to gate distillation or modulate updates.
  • Trajectory-level filtering is also closely related to filtered behavior cloning, reward-weighted regression, and rejection-sampling fine-tuning. The contribution is therefore mainly a particular combination of known ingredients. The paper should clearly state what is new relative to SRPO and TIP and compare against SRPO empirically. Related work also appears only in the appendix, which makes positioning hard to assess from the main text.
  1. Missing simple baselines: Rejection-sampling: roll out with skills and keep successful trajectories. Fine-tune the student without skills.

To the authors: I am willing to change my rating, if you address my concern around novelty and also if you can add rejection-sampling as a baseline.

Questions for authors
  1. Are the reported numbers on the same validation set used for checkpoint selection? Can you report results on a held-out test set (or with last-checkpoint evaluation), with multiple seeds?
  2. How does DL-OPSD differ from SRPO, and how does it compare empirically?
  3. Have you considered conditioning the teacher on the trajectory outcome (hindsight teacher) or using step-level credit to select tokens, so that useful supervision from failed trajectories is not discarded?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.