- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper finds in on-policy self-distillation (OPSD), distilling tokens from positive-advantage trajectories is more valuable than distilling tokens from negative-advantage trajectories. To this end, the authors propose DL-OPSD, which operates at two levels. At the trajectory level, it applies OPSD distillation only to positive-advantage trajectories while train with GRPO on all trajectories at the same time. At the token level, it distills only tokens supported by the teacher, weighting them according to teacher certainty and student uncertainty. Experiments across ALFWorld, Search-QA, and WebShop show that DL-OPSD consistently outperforms GRPO, and ablation studies validate the effectiveness of each component.
Critical strengths
-
This paper is well motivated and proposes a clean yet effective method that tackles a common and important problem in on-policy self-distillation (OPSD).
-
The experimental section of this paper is solid. The authors conduct thorough ablation studies that justify the design choices made for trajectory selection and token weighting.
Critical weaknesses
- DL-OPSD shares similarities with several existing methods, which limits its novelty. For example, SIRI[1] also selects which trajectories to distill based on trajectory-level utility. SDAR[2] also relies on the teacher–student gap, strengthening distillation on tokens with a positive gap.
[1] Siri: Self-Internalizing Reinforcement Learning with Intrinsic Skills for LLM Agent Training
[2] Self-Distilled Agentic Reinforcement Learning
- The logic behind Figure 1(b) is puzzling. Actions in positive-advantage trajectories would naturally exhibit higher continuation success gains than actions in other trajectories, precisely because positive-advantage trajectories are themselves successful ones. As such, this observation does not seem to establish that distilling tokens from positive-advantage trajectories is more beneficial. Moreover, according to Appendix E.2, Figure 1(b) is computed from the step-300 DL-OPSD checkpoint, which is odd: Figure 1(b) itself serves as the motivation for DL-OPSD, so why the authors used a fully trained DL-OPSD model to validate the motivation of DL-OPSD?
Questions for authors
-
In the gradient-alignment experiment, only the gradient alignment of the first decision response of each trajectory is measured. Why not compare over more tokens?
-
Several ablation studies continually train models from the step-150 GRPO checkpoint, but the paper offers no corresponding explanation for this choice. Why not train them by DL-OPSD components from scratch?