- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SDIAL, a self-distilled credit assignment method that aims to address a key limitation of GRPO in multi-turn LLM agent RL under sparse outcome rewards: GRPO assigns the same advantage to every token within a trajectory. To this end, SDIAL collects (action, next observation) pairs from rollouts in the same group whose states match or closely resemble the current state, and provides them as evidence in the prompt of a teacher instantiated from the same model. The token-level log-probability difference between the teacher and the student is then transformed into a quality score via a tanh function and teacher-entropy-based weighting. This score is used to reshape the advantages asymmetrically: positive advantages are scaled multiplicatively, whereas negative advantages are adjusted additively. In experiments with Qwen2.5-7B-Instruct, the authors report substantial improvements over GRPO and modest gains over GiGPO across ALFWorld, BabyAI, and WebShop.
Critical strengths
1. Clear and practical motivation: The idea of leveraging state transitions, which are naturally obtained during rollouts, as privileged information without requiring any additional labels is well-motivated and sensible. Furthermore, the method introduces no overhead at inference time, which is a notable practical advantage.
2. Well-motivated design for the “good token in a bad trajectory” problem : The asymmetric reshaping scheme, which permits sign reversal through an additive correction in the A < 0 branch, appears intuitively reasonable.
3. Substantial and consistent improvements over the GRPO backbone: The gains are large and consistent across all three environments: +15.4points on ALFWorld, +17.1points on BabyAI, and +12.0points on WebShop (success rate).
Critical weaknesses
1. Missing comparison with the closest prior work: SDIAL combines ideas from RLSD (token reweighting via teacher–student log-probability differences) and GiGPO (step-level comparison over identical states), yet RLSD, which the authors identify as the closest related work, is not included as a baseline. It therefore remains unclear whether the gains come from the transition evidence itself or from the asymmetric reshaping. Comparisons with “RLSD + the same transition evidence” and “GiGPO + SDIAL” would help clarify the method’s unique contribution.
2. Limited statistical reliability of the reported gains: The improvements over GRPO are clear, but the margins over GiGPO fall within the standard deviations across three seeds (e.g., 63.5±1.6 vs. 62.0±2.4 on BabyAI), and SDIAL is not the best on several subtasks (Pick, Cool, and open). The claim of “consistent gains” thus appears somewhat overstated. The method is also sensitive to hyperparameters (e.g., 79.7% with K=8 vs. 66.8% with K=10 in Table 4, the latter below GRPO), but the criterion used to select λ, K, and the checkpoint is not specified.
3. The core premise of the method is not empirically verified: The central premise of SDIAL is that the teacher–student discrepancy reflects action quality, which Appendix C formalizes as Assumption 1 (positive correlation between the teacher correction and the true token-level credit). However, this assumption essentially restates the desired conclusion, and the authors acknowledge that it is not guaranteed by construction. Although they note that correlation analyses could support it, none are reported. As a result, it remains unclear whether the observed gains stem from improved credit assignment or from other effects.
Questions for authors
-
Under Eq. (10), A=0 falls into the multiplicative branch, so SDIAL appears to provide no signal when all rollouts in a group share the same reward. How does SDIAL handle this case? Could the authors also report the proportion of sign-flipped tokens and the distribution of the self-distilled scores during training?
-
Could the authors add a baseline of RLSD with the same transition evidence (or clarify whether the “Multiplicative” variant in Fig. 2a corresponds to it), as well as results for GiGPO combined with SDIAL?
-
Could the authors show that the self-distilled scores correlate with actual step-level credit (e.g., Monte Carlo value estimates)?
-
How were λ, K, and the final checkpoint selected? Could the authors provide results with more seeds and learning curves?
Dear Reviewer jHks,
Thank you for your comments. We have posted a detailed rebuttal addressing the concerns raised. We would greatly appreciate it if you could review it at your convenience and let us know if any questions remain.