- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SDAR, a framework combining GRPO with on-policy self-distillation for multi-turn agents. Instead of applying uniform distillation, SDAR uses a sigmoid gate based on the teacher-student log-probability gap to strengthen positive teacher endorsements and attenuate negative ones. The teacher receives retrieved skills as privileged training context, while the student operates without these skills at inference time. Experiments on ALFWorld, WebShop, and Search-QA show improvements over GRPO and competitive performance against existing hybrid RL-distillation methods.
Critical strengths
-
The paper addresses a relevant problem in combining RL with privileged self-distillation for multi-turn agents. Keeping GRPO as the primary objective while controlling auxiliary supervision at the token level is a sensible design.
-
SDAR improves upon GRPO across the evaluated model scales and tasks. The gating ablations and skill retrieval experiments are particularly useful for understanding the method's behavior.
Critical weaknesses
-
Ambiguity in the baseline implementation. Algorithm 5 defines GRPO+OPSD using a frozen reference policy, whereas SDAR uses a skill-conditioned self-teacher. Table 4 also lists skill retrieval for GRPO+OPSD, making the exact comparison with the ungated baseline unclear.
-
Limited interpretation of the distillation objective. Proposition 1 shows that the auxiliary loss reduces to weighted likelihood training on student-sampled tokens. The teacher therefore controls the weights rather than directly supervising the full token distribution, and this distinction deserves a clearer explanation.
-
The reliability of positive teacher endorsements is not fully established. A positive teacher-student gap does not necessarily imply a beneficial action, particularly when the retrieved skill is irrelevant. The retrieval robustness results are helpful, but they do not directly validate this token-level assumption.
Questions for authors
-
Could the authors clarify whether GRPO+OPSD and the w/o Gating ablation use the same skill-conditioned teacher as SDAR? Algorithm 5 and Table 4 appear inconsistent on this point.
-
Since the detached SDAR objective is equivalent to weighted likelihood training, how should we interpret its knowledge transfer beyond selectively reinforcing student-generated tokens?
-
A positive teacher-student gap does not necessarily indicate a beneficial action, as the teacher may still prefer other alternatives. How does SDAR distinguish useful teacher endorsements from misleading positive gaps on student-sampled tokens?
8 comments
Kind Reminder
Dear Reviewer KAQN,
Thank you once again for your valuable comments on our submission. Our responses have been available for four days now, and as the discussion phase approaches its end, we would like to kindly confirm whether they have adequately addressed your concerns.
If so, we would be grateful if you would consider raising your rating accordingly. If any questions or points still require clarification, please feel free to let us know. We look forward to your invaluable feedback.
Sincerely,
The Authors
Thank you for the detailed response and additional analysis. Overall, the rebuttal clarifies the method and provides useful supporting evidence. My remaining concern is that the token-level analysis does not directly establish the reliability of the gradient-weighted supervision, so I will maintain my current assessment.
We sincerely thank the reviewer for the constructive feedback and for recognizing our sensible token-level gating design, consistent improvements over GRPO across model scales, and informative skill-retrieval and gating ablations. Below we address your concerns and questions in a concise and structured manner.
W1 & Q1: Baseline Specification
We thank the reviewer for this careful reading. We have rewritten Algorithms 3 and 5 and the corresponding entries in Table 4 to describe the baseline more clearly; the revised passages are highlighted in blue in the updated paper.
We would like to clarify that
GRPO+OPSDandw/o Gatingrefer to the same method, and have made explicit thatOPSD,GRPO+OPSD, andSDARshare the identical skill-conditioned self-teacher with Keyword Matching retrieval. We have updated Algorithms 3 and 5 to state this teacher explicitly rather than using the generic reference notation πref. So comparingGRPO+OPSDandSDARisolates the exact benefit of our asymmetric sigmoid gate.W2 & Q2: Interpreting the Weighted-Likelihood Objective
We appreciate this insightful question and would like to claim that SDAR separates two distinct roles:
W3 & Q3: Reliability of Positive Teacher Endorsements
We agree that Δt>0 represents a comparative endorsement relative to the current student rather than an isolated guarantee of optimality, especially under noisy retrieval. We address this concern from both mechanistic and empirical perspectives:
We run this on Qwen2.5-3B over 50 ALFWorld tasks' rollout under a random skill context and under retrieved skills:
Retrieved skills thus produce positive gaps that are both more frequent (34% vs. 16%) and markedly more reliable (86% vs. 72% judged helpful). We will extend this judgment across backbones in the appendix.
Soft Gatingreaches 85.9% on ALFWorld and 82.8% on WebShop-Acc overw/o Gatingat 80.4% and 76.5%, whereas inverting the gate inReverse Soft Gatingdrops to 76.2% and 66.4%, a loss of −9.7 and −16.4 points.We hope these clarifications and results address your concerns, and we look forward to your feedback.
[1] TIP: Token Importance in On-Policy Distillation
[2] Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models