acceptodds
Under review as a conference paper at ICLR 2027

State Transition Matters: Self-Distilled Credit Assignment for Multi-Turn Agentic RL

Abstract

Reinforcement learning (RL) has improved large language models (LLMs) on verifiable tasks, but training multi-turn interactive agents remains challenging due to sparse outcome rewards and coarse credit assignment. Existing methods mainly rely on outcome reward back-propagation to assign credit, ignoring the rich task-relevant information hidden in state transitions, which naturally reveal the local consequences of actions. Based on this observation, we propose SDIAL, a novel Self-Distilled Advantage reshaping method for multi-turn credit assignment in reinforcement Learning. In SDIAL, the LLM of student mode interacts with the environment, while the same LLM of teacher mode leverages the state transitions to provide hindsight evaluation of the student's actions. The difference between the teacher's and student's output distributions is distilled into token-level credit signals to refine advantage estimates at each decision step. To maintain the effectiveness of self-distillation and stabilize policy optimization, SDIAL introduces an asymmetric reshaping mechanism that applies different operators to reshape positive and negative advantages. Experiments on multiple long-horizon interactive benchmarks show that SDIAL improves training stability and robustness while achieving strong task performance across diverse environments. Code is available at https://anonymous.4open.science/r/SDIAL.

Primary area:
reinforcement learning
Keywords:
self-distillation, agentic rl, large language model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_jHksOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SDIAL, a self-distilled credit assignment method that aims to address a key limitation of GRPO in multi-turn LLM agent RL under sparse outcome rewards: GRPO assigns the same advantage to every token within a trajectory. To this end, SDIAL collects (action, next observation) pairs from rollouts in the same group whose states match or closely resemble the current state, and provides them as evidence in the prompt of a teacher instantiated from the same model. The token-level log-probability difference between the teacher and the student is then transformed into a quality score via a tanh function and teacher-entropy-based weighting. This score is used to reshape the advantages asymmetrically: positive advantages are scaled multiplicatively, whereas negative advantages are adjusted additively. In experiments with Qwen2.5-7B-Instruct, the authors report substantial improvements over GRPO and modest gains over GiGPO across ALFWorld, BabyAI, and WebShop.

Critical strengths

1. Clear and practical motivation: The idea of leveraging state transitions, which are naturally obtained during rollouts, as privileged information without requiring any additional labels is well-motivated and sensible. Furthermore, the method introduces no overhead at inference time, which is a notable practical advantage.

2. Well-motivated design for the “good token in a bad trajectory” problem : The asymmetric reshaping scheme, which permits sign reversal through an additive correction in the A < 0 branch, appears intuitively reasonable.

3. Substantial and consistent improvements over the GRPO backbone: The gains are large and consistent across all three environments: +15.4points on ALFWorld, +17.1points on BabyAI, and +12.0points on WebShop (success rate).

Critical weaknesses

1. Missing comparison with the closest prior work: SDIAL combines ideas from RLSD (token reweighting via teacher–student log-probability differences) and GiGPO (step-level comparison over identical states), yet RLSD, which the authors identify as the closest related work, is not included as a baseline. It therefore remains unclear whether the gains come from the transition evidence itself or from the asymmetric reshaping. Comparisons with “RLSD + the same transition evidence” and “GiGPO + SDIAL” would help clarify the method’s unique contribution.

2. Limited statistical reliability of the reported gains: The improvements over GRPO are clear, but the margins over GiGPO fall within the standard deviations across three seeds (e.g., 63.5±1.6 vs. 62.0±2.4 on BabyAI), and SDIAL is not the best on several subtasks (Pick, Cool, and open). The claim of “consistent gains” thus appears somewhat overstated. The method is also sensitive to hyperparameters (e.g., 79.7% with K=8 vs. 66.8% with K=10 in Table 4, the latter below GRPO), but the criterion used to select λ, K, and the checkpoint is not specified.

3. The core premise of the method is not empirically verified: The central premise of SDIAL is that the teacher–student discrepancy reflects action quality, which Appendix C formalizes as Assumption 1 (positive correlation between the teacher correction and the true token-level credit). However, this assumption essentially restates the desired conclusion, and the authors acknowledge that it is not guaranteed by construction. Although they note that correlation analyses could support it, none are reported. As a result, it remains unclear whether the observed gains stem from improved credit assignment or from other effects.

Questions for authors
  1. Under Eq. (10), A=0 falls into the multiplicative branch, so SDIAL appears to provide no signal when all rollouts in a group share the same reward. How does SDIAL handle this case? Could the authors also report the proportion of sign-flipped tokens and the distribution of the self-distilled scores during training?

  2. Could the authors add a baseline of RLSD with the same transition evidence (or clarify whether the “Multiplicative” variant in Fig. 2a corresponds to it), as well as results for GiGPO combined with SDIAL?

  3. Could the authors show that the self-distilled scores correlate with actual step-level credit (e.g., Monte Carlo value estimates)?

  4. How were λ, K, and the final checkpoint selected? Could the authors provide results with more seeds and learning curves?

1 comment

Dear Reviewer jHks,

Thank you for your comments. We have posted a detailed rebuttal addressing the concerns raised. We would greatly appreciate it if you could review it at your convenience and let us know if any questions remain.

Reviewer_9ufgOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SDIAL, a self-distillation method for credit assignment in multi-turn agentic RL. It uses state transitions as privileged information for a teacher model and adjusts token-level advantages based on the teacher-student probability difference. The method achieves promising results on ALFWorld, BabyAI, and WebShop.

Critical strengths
  1. The paper addresses an important problem in multi-turn agentic RL, where sparse rewards make credit assignment difficult.
  2. Using state transitions as privileged information is an interesting idea, and the method does not require an additional reward model.
  3. The results are promising, especially on ALFWorld and WebShop.
Critical weaknesses
  1. The comparison with related self-distillation methods is limited. The paper compares against several RL baselines, including GiGPO and RAGEN-2, but does not include other agentic self-distillation methods such as Skill-SD or closely related approaches like RLSD. Since self-distillation is central to the proposed method, I would like to see a more direct comparison, ideally using the same backbone and training budget.

  2. The computational cost of hindsight information is not fully analyzed. The authors report overall training time and memory overhead, which is helpful. However, the teacher prompt becomes longer as more transitions are retrieved. It would be useful to report the additional token cost and how training time changes with the number of retrieved transitions. The significant performance drop from K=8 to K=10 on WebShop also deserves some explanation.

  3. The advantage reshaping design needs further clarification. Appendix C.6 provides a reasonable theoretical motivation for the two branches, with the positive branch preserving update directions and the negative branch correcting potentially misassigned penalties. However, separate ablations only for positive and negative would help verify whether both components contribute to the observed performance gains.

  4. The theoretical analysis does not directly validate the proposed credit signal. The main theoretical result assumes that the self-distilled correction is aligned with the true residual credit. However, this is exactly what needs to be established. So far, the analysis only shows that credit estimation improves when the correction is sufficiently aligned with the true residual credit, but does not explain why the proposed teacher-student discrepancy should have this property.

Questions for authors
  1. Can the authors provide a more direct comparison with existing agentic self-distillation methods?

  2. How does the number of retrieved transitions affect teacher token consumption, training time, and performance?

  3. Can the authors provide separate ablations only for positive and for negative advantage reshaping?

  4. Can the authors provide empirical evidence that the self-distilled correction is aligned with the true residual credit?

1 comment

Dear Reviewer 9ufg,

Thank you for your comments. We have posted a detailed rebuttal addressing the concerns raised. We would greatly appreciate it if you could review it at your convenience and let us know if any questions remain.

Reviewer_TKvEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces SDIAL, a method that uses the consequences of an agent’s actions to improve how it learns from success and failure. During training, the same LLM acts as both student and teacher, but the teacher also sees actions and their resulting observations from comparable situations. Differences between their token probabilities adjust the training signal: positive advantages are scaled, while negative advantages receive additive corrections that can reduce penalties on useful actions. Experiments on ALFWorld, BabyAI, and WebShop show improvements over several RL baselines, including gains that persist under equal training-time budgets.

Critical strengths

The method makes practical use of feedback already produced during interaction, without requiring an external teacher, a separate critic, or additional inference-time machinery. Its treatment of failed trajectories is sensible: an unsuccessful attempt can still contain useful decisions that should not all receive the same penalty. The main experiments report three-seed results, with improvements over GiGPO of 4.3 percentage points on ALFWorld and 7.0 points on WebShop, although the BabyAI gain is smaller at 1.5 points. The appendix strengthens the evidence with Qwen3-4B experiments: SDIAL exceeds GRPO by 8.20 points on WebShop and 12.24 points on ALFWorld under approximately equal wall-clock budgets. Reported training-time overhead is roughly 13–20%, and component ablations and reward-corruption experiments provide further support for the method.

Critical weaknesses

The main unresolved issue is whether the teacher–student probability difference reliably measures action quality. Because the teacher sees the evaluated action in its evidence, increased probability could partly reflect repetition or copying rather than recognition that the action helps achieve the goal; action-only, shuffled-consequence, and leave-one-out controls would help establish this distinction.

Questions for authors

see above

2 comments

Dear Reviewer TKvE,

Thank you for your comments. We have posted a detailed rebuttal addressing the concerns raised. We would greatly appreciate it if you could review it at your convenience and let us know if any questions remain.

thank you! The additional experiment results are helpful.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.