acceptodds
Under review as a conference paper at ICLR 2027

Inside Failed Rollouts: Progressive Experience Self-Distillation for Agentic Training

Abstract

On-policy self-distillation complements sparse outcome rewards with dense supervision by re-scoring an agent's own rollouts under privileged training-time context. Failed rollouts are particularly challenging because useful progress and harmful decisions often coexist within the same trajectory. We introduce Progressive Experience Self-Distillation (PESD), a failure-aware framework that uses privileged likelihood shifts to separate realized actions into supported, rejected, and uncertain regions. Supported actions receive forward distillation, consistently rejected actions receive low-weight unlikelihood updates, and uncertain actions are left to RL. PESD further organizes outcome feedback, applicability-gated skills, and verified successful trajectories into a progressive privilege hierarchy, with stronger guidance reduced across repeated task visits and removed at inference. On ALFWorld, PESD reaches 92.2/66.1 with Qwen2.5-3B-Instruct/Qwen3-1.7B, and 46.0/44.0 on Search-QA. These results show that failed rollouts can provide useful action-level supervision when privileged signals are applied selectively rather than uniformly.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Experience distillation, On-Policy Distillation, Reinforcement learning, Policy co‑evolution

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_b8mmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PESD augments GRPO with selective self-distillation on failed on-policy rollouts. Using completed-episode hindsight, retrieved skills, and available successful references, it partitions actions by context-induced log-probability shifts into positive distillation, conservative unlikelihood updates, or RL-only optimization. Additional guidance is progressively reduced across task–outcome revisits, and privileged information is removed at inference. Experiments evaluate 1.7B and 3B models on ALFWorld and Search-QA, supplemented by component ablations and a manual audit of 1,000 actions.

Critical strengths
  1. The problem formulation is well motivated: actions within a failed trajectory receive different forms of supervision. Assigning distinct optimization roles to actions within the same failed rollout is more nuanced than suppressing all actions simply because the trajectory failed.

  2. The experiments across model scales and the evaluation on unseen tasks are reasonably comprehensive.

Critical weaknesses
  1. The core contribution is to partition actions into three categories, P, N, and U, based on log probabilities and apply a different update strategy to each. In my view, these hand-crafted decision rules may improve performance because the model's log probabilities, even with the additional context, still do not provide a reliable supervisory signal. Similarly, dividing the additional guidance into three levels appears to reflect limitations in model capability: an explicit hierarchy is needed to reduce the supervision's dependence on specific reference material. This raises the question of whether the method would remain effective for somewhat stronger models, or whether its benefits are limited to small, relatively weak models.

  2. Invalid actions account for 67.2% of the 311 actions in the positive region P, exceeding the 40% invalid-action rate across all audited actions. Specifically, 209 of these 311 actions were manually labeled invalid. Why are these invalid actions reinforced? In addition, the negative region N contains only 24 actions, providing a small sample to support the conclusions. The audit covers only 20 failed trajectories from a single model at a single training checkpoint.

  3. The Search-QA evaluation protocol does not provide sufficiently robust support for the claimed overall advantage. Appendix A explicitly states that the 125 Bamboogle questions are divided into two slices, one containing 124 questions and the other a single question, and the slice scores are then averaged with equal weight. Consequently, a single question accounts for 50% of the score on this dataset.

Questions for authors

Please refer to the weakness

4 comments
AuthorsOpenReview ↗

Response to Reviewer b8mm

We thank the reviewer for the thoughtful feedback. Our responses are below.

(W1) Decision rules, privilege hierarchy, and stronger models

these hand-crafted decision rules may improve performance because the model's log probabilities... still do not provide a reliable supervisory signal...; whether the method would remain effective for somewhat stronger models...

We agree that log-probability shifts are not a perfect signal, but this is not a weakness of small models in particular. A shift tells us how the privileged context changes the model's preference for an action, not whether the action is actually good. The thresholds simply decide when the signal is strong enough to act on. This matters most on the negative side, since unlikelihood pushes an action down without offering a better one. Without these safeguards, Bare Reverse collapses to 66.4, and dropping the consistency gate (ρ=0\rho=0) gives 85.9. SDAR and PCSD filter teacher signals for the same reason. The hierarchy is also not a workaround for weak models. It addresses the fact that privileged information is gone at inference. If strong guidance stays on throughout training, the teacher leans on the reference rather than on the policy's own behavior. To test whether the benefits extend to stronger models, we also evaluate PESD with Qwen2.5-7B-Instruct on ALFWorld (Results reported below).

(W2) Invalid actions in P and audit scale

Why are these invalid actions reinforced?...; N contains only 24 actions...; The audit covers only 20 failed trajectories...

P is defined by a likelihood increase under the privileged context, not by correctness, so it can include invalid actions. This is why the forward update on P is kept small: the coefficient is 0.01, tokens are weighted by persistence, and RL with the invalid-action penalty keeps pushing these actions down. Even so, adding forward supervision on P still improves over reverse-only (90.6 → 92.2). Filtering P by correctness during training would need a reliable process reward model, which would add another reward signal and make the effect of self-distillation harder to isolate. N is small on purpose, since reverse updates should only hit clearly rejected actions. The audit is a manual sanity check of this selectivity, not a precise estimate. On the same 1,000 hand-labeled actions, lowering the consistency threshold ρ\rho (from 0.6 to 0.3 to 0) grows N from 24 to 104 to 111 and raises the share of correct actions in it from 8.3% to 14.4% to 15.3%, in line with the performance drop (92.2 → 91.4 → 85.9).

(W3) Search-QA aggregation

a single question accounts for 50% of the score on this dataset...

We agree. This aggregation comes from the evaluation pipeline we inherited from SDAR (verl-agent), and it affects all methods equally. Bamboogle drops by 32.5–36.3 points for every method (Appendix A). We rerank all methods in two ways, using the sample-level Bamboogle score (Updated Bam.) and leaving Bamboogle out (w/o Bam.):

Method3B Updated Bam.3B w/o Bam.1.7B Updated Bam.1.7B w/o Bam.
G-OPSD39.941.037.438.3
Skill-SD39.240.735.836.9
RLSD39.240.035.736.5
SDAR38.639.637.038.0
PCSD37.639.336.638.1
PESD41.442.639.240.3

PESD stays on top in both cases, by 1.5–2.1 points, similar to the original protocol.

We thank the reviewer again for the helpful comments.

AuthorsOpenReview ↗

Additional ALFWorld Evalution on Qwen2.5-7B-Instruct

we are running PESD with Qwen2.5-7B-Instruct on ALFWorld (In Progress). -> we also evaluate PESD with Qwen2.5-7B-Instruct on ALFWorld (Results reported below).

We additionally test PESD with Qwen2.5-7B-Instruct on ALFWorld to evaluate whether its benefits extend to stronger backbones. As shown in the table below, PESD outperforms SDAR by 6.3 points on average with Qwen2.5-7B-Instruct. The results suggest that PESD maintains its advantage with a stronger backbone.

MethodPickLookCleanHeatCoolPick2Avg.
SDAR94.775.010086.768.278.985.9
PESD10091.710086.784.083.392.2

Thank you for the additional experiments. The larger-model results, comparisons using corrected Bamboogle aggregation, and experiments with matched training budgets have alleviated my concerns on these points. However, the role of positively reinforcing invalid actions remains unclear, and the existing ablations do not isolate its effect. I am still awaiting the promised control that excludes invalid actions from positive reinforcement and will reassess once the results are available. For now, I maintain my original rating.

AuthorsOpenReview ↗

Additional ablation results.

Thank you for your consideration and for acknowledging the additional experiments. We have added the ablation (excluding invalid actions from positive reinforcement) results. As shown below, excluding invalid actions from positive reinforcement does not lead to an overall improvement. We further evaluated this variant with 10 random seeds. Its best run achieved an average success rate of 93.0, matching the best performance achieved by the original PESD. We have also added an analysis of the training dynamics w/ and w/o invalid actions to the revised manuscript (Appendix D.4). We hope these additional results address your remaining concerns. Thank you again for your patience and constructive feedback.

VariantPickLookCleanHeatCoolPick2Avg.
PESD99.0 ± 1.586.1 ± 7.994.9 ± 3.697.8 ± 3.284.0 ± 0.087.0 ± 2.692.2 ± 1.3
w/o invalid in P\mathcal{P}100.0 ± 0.088.9 ± 4.894.9 ± 2.395.5 ± 3.988.0 ± 4.072.2 ± 9.691.1 ± 0.9
Reviewer_5RACOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Progressive Experience Self-Distillation (PESD), a method for training LLM agents that aims to extract fine-grained supervision signals from failed trajectories containing both useful and harmful behaviors. Specifically, the authors construct a hierarchy of privileged information using completed trajectories, retrieved skills, and verified successful trajectories. Based on the changes in action likelihood induced by the privileged context, the method partitions actions in failed trajectories into three regions: privilege-supported, privilege-rejected, and inconclusive. These regions are handled through forward distillation, low-weight unlikelihood supervision, and RL optimization without auxiliary distillation, respectively. In addition, the method progressively reduces the privileged guidance across repeated visits to the same task–outcome pair to mitigate the model's reliance on additional information. The authors evaluate PESD on two benchmarks, ALFWorld and Search-QA, using Qwen2.5-3B-Instruct and Qwen3-1.7B. The experimental results show that PESD achieves better overall performance than baseline methods such as GRPO, G-OPSD, and Skill-SD. The ablation studies further support the effectiveness of reverse supervision, skill applicability filtering, and the progressive privilege schedule.

Critical strengths
  • The paper addresses an important problem and proposes a promising fine-grained supervision mechanism to better utilize failed trajectories in reinforcement learning.

  • The experimental results and ablation studies provide empirical support for the proposed method. The authors evaluate PESD on two types of agent tasks and compare it against several reinforcement learning and self-distillation baselines, achieving strong overall performance.

Critical weaknesses
  • The algorithmic contribution feels more incremental than transformative. PESD shares considerable similarities with existing methods such as SGSD [1] and Skill-SD [2], particularly in their use of privileged information and skills to provide self-distillation signals. The current experimental evidence does not fully disentangle the benefits of the newly introduced mechanisms from those arising from the combination of existing techniques.

  • The performance improvements are inconsistent across subtasks, and the experimental evaluation provides limited evidence of broad applicability. Although PESD achieves a strong average success rate on ALFWorld, Table 1 reveals substantial variation in performance gains across different task categories. Furthermore, the evaluation is limited to two benchmarks, ALFWorld and Search-QA, which may not be sufficient to establish the general applicability of the proposed failure-aware supervision mechanism.

References

[1] Jiazhen Huang, Xiao Chen, Xiao Luo, Yong Dai, Senkang Hu, and Yuzhi Zhao. Skill-Conditioned Gated Self-Distillation for LLM Reasoning. 2026. arXiv:2605.28791. https://arxiv.org/abs/2605.28791

[2] Hao Wang et al. Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents. 2026. arXiv:2604.10674. https://arxiv.org/abs/2604.10674

Questions for authors

1. Regarding methodological novelty and distinctions from prior work: Could the authors further clarify the key differences between PESD and existing methods such as SGSD, and Skill-SD?

2. Regarding performance variations across subtasks: Could the authors provide further analysis of the factors contributing to these discrepancies? In particular, is there a relationship between the distribution of P/N/U actions in failed trajectories and the performance changes observed across different task categories? Additionally, could the authors provide results on more diverse agent benchmarks to substantiate the generalizability of the proposed method?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 5RAC

We thank the reviewer for the assessment and constructive feedback. We address the concerns point by point below.

(W1 & Q1) Novelty and differences from SGSD and Skill-SD

PESD shares considerable similarities with existing methods such as SGSD and Skill-SD...; The current experimental evidence does not fully disentangle...

PESD shares with SGSD and Skill-SD the general idea of using privileged context to score on-policy rollouts, but differs in what is supervised and how. Skill-SD summarizes completed trajectories into natural-language skills and uses them as teacher-side context. It focuses on what the teacher should know and on training stability, rather than on separating useful and harmful behavior within a failed rollout. SGSD conditions multiple teachers on retrieved skill–mistake pairs. For each teacher, it averages the teacher–student gap over the whole rollout and uses the verifier outcome to decide whether to distill or reverse. The update direction is therefore set once per teacher for the entire rollout. Our ablations help disentangle these contributions. Without failure-side supervision, PESD reaches 89.1, close to our reproduced PCSD (88.8), while adding the action-level failure supervision raises it to 92.2. We have expanded SGSD and Skill-SD methods in the revised version.

PESD is designed for multi-turn agent trajectories, where a failed rollout usually mixes useful progress with the decisions that caused the failure. Rather than treating such a rollout as a whole, PESD looks at each action separately. Actions that the privileged context supports are distilled forward, actions that it consistently rejects receive a small unlikelihood penalty, and the rest are left to RL. The privileged context is also richer than retrieved skills. It includes the completed episode and a verified successful trajectory (when available), and this guidance is gradually reduced as the same task–outcome pair is revisited. Importantly, obtaining this information adds negligible cost. The completed episode is the rollout itself, and the verified trajectory is simply a successful rollout of the same task from the current GRPO group or an earlier step. PESD needs no external teacher, no extra environment interaction, and no LLM to generate skills.

(W2 & Q2) Variation across subtasks and generality

Table 1 reveals substantial variation in performance gains across different task categories...; is there a relationship between the distribution of P/N/U actions...

We thank the reviewer for this question. We reproduced SDAR and PCSD and report all methods averaged over three runs. PESD achieves the best average on both backbones, leading the strongest baseline by 3.4 and 6.5 points. It is best or tied for best in five of six categories with Qwen2.5-3B. With Qwen3-1.7B, it is also the most balanced method, with the smallest worst-case gap to the best method across categories.

BackboneMethodPickLookCleanHeatCoolPick2Avg.
Qwen2.5-3BSDAR97.963.991.060.076.075.981.5 ± 1.2
PCSD99.072.289.788.982.788.988.8 ± 2.0
PESD99.086.194.997.884.087.092.2 ± 1.3
Qwen3-1.7BSDAR70.833.370.542.246.738.953.6 ± 0.5
PCSD61.552.856.464.470.750.059.6 ± 2.3
PESD79.241.766.771.173.344.566.1 ± 1.0

Regarding the link to the P/N/U partition, PESD's extra signal on failed trajectories comes from actions in P and N, while actions in U receive only the RL update, so we expect smaller gains on categories where failures yield few P or N actions. As for benchmark coverage, we mainly follow the benchmarks used in prior work so that results can be compared directly. ALFWorld and Search-QA cover two distinct settings, embodied household manipulation and tool-augmented multi-hop retrieval, and PESD also generalizes to the unseen ALFWorld split (89.6), outperforming PCSD and SDAR. We agree that broader coverage would further strengthen the conclusion, and we have extended the evaluation to WebShop (Results reported below).

We thank the reviewer again for the constructive feedback, which has helped us improve the clarity and positioning of our work.

AuthorsOpenReview ↗

Additional Evalution on WebShop

we are extending the evaluation to WebShop (In Progress) -> we have extended the evaluation to WebShop (Results reported below)

To further assess whether PESD’s gains generalize beyond ALFWorld and Search-QA, we extended our evaluations on WebShop benchmark. As shown below, PESD achieves the highest score and accuracy on WebShop with both backbones, improving over the best baselines by 5.0/6.5 points in score and 15.6/18.0 points in accuracy. These results provide further evidence that PESD transfers across different agent benchmarks.

BackboneMethodScoreAcc
Qwen2.5-3BSDAR85.068.0
PCSD85.067.2
PESD90.083.6
Qwen3-1.7BSDAR76.858.6
PCSD78.958.6
PESD85.476.6
Reviewer_mqAROpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Progressive Experience Self-Distillation (PESD) to address a specific and well-scoped problem in agentic RL: credit assignment in agentic RL when rollouts fail. Instead of treating a failed trajectory as uniformly negative, PESD re-scores the student's own realized actions under a privileged teacher context (the completed episode plus a guidance hierarchy), using the resulting likelihood shifts to partition actions into three regions: supported (P --> forward distillation), consistently rejected (N --> low-weight unlikelihood), and uncertain (U --> left to GRPO). The privileged context is organized as hierarchy: L1 outcome feedback ⊆ L2 (+ applicability-gated retrieved skill) ⊆ L3 (+ verified successful trajectory) which is weakened on repeated visits to the same task–outcome pair and removed at inference. Successful trajectories receive forward-only distillation scored by a frozen reference policy. Extensive experiments on ALFWorld and Search-QA with Qwen 2.5-3B-Instruct and Qwen 3-1.7B show gains over GRPO, G-OPSD, Skill-SD, RLSD, SDAR, and PCSD. Component ablations also support the validity -- a ρ sensitivity sweep, training-dynamics analysis, and unseen-task evaluation.

Critical strengths

S1. Well-controlled evidence that conservative negative supervision on failures is the key ingredient. The Bare Reverse collapse (66.4 vs 92.2), ρ\rho sweep, and manual audit (N region small, rarely contains useful actions) triangulate the central claim -- the first controlled demonstration that rejected actions can be actively exploited via unlikelihood, and when that fails.

S2. Comprehensive validation. Two benchmarks, two model scales, six baselines, unseen-task evaluation, and candid reporting of inconvenient results (non-additive branches, Bamboogle aggregation discrepancy), enabling proper calibration of confidence.

Critical weaknesses

W1. Margins are near noise and Search-QA numbers are protocol-sensitive. PESD uses 3 runs (std ~1.0–1.3); baselines appear single-run, so +1.6 (ALFWorld-3B) and +1.4 (Search-QA-3B) are not established as significant. Slice aggregation inflates Bamboogle by ~33 points, dropping PESD to 41.4/39.1 sample-level without cross-method rankings. It is hard to tell consistent gains.

W2. Self-confirmation confound and heavy engineering. The privileged context contains the very action being scored, inviting in-context copying (consistent with 209/311 P-region actions being invalid per audit); no masking control is reported. Many tuned knobs with sensitivity analysis only for ρ.

Questions for authors

Q1. Are baselines single-run? I cannot find std for PCSD/G-OPSD over matched seeds and significance tests for the ALFWorld-3B and Search-QA-3B margins.

Q2. How is the completed episode rendered in the teacher prompt, and how do you rule out self-copying inflating P-region scores? Would masking the scored suffix change results?

Q3. What is the per-step compute overhead vs GRPO, and do conclusions hold under compute-matched comparison?

Q4. How sensitive are results to tau+tau^+, tau−tau^-, λrevλ_{rev}, λ\lambda, and the revisit ratio (only ρ\rho is swept)?

Q5. How does PESD degrade when verified successful trajectories (L3) are rarely available?

Reviewer_tpPaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to use failed trajectories in multi-turn agent reinforcement learning: the same model re-scores the actions of a failed trajectory under additional hint information, actions whose score increases are reinforced and those whose score drops clearly are suppressed, and the hint information is reduced step by step as the task recurs. The authors report on ALFWorld and Search-QA that the method outperforms GRPO and several self-distillation baselines.

Critical strengths

The authors submitted their full code and manually annotated 1,000 actions from failed trajectories, comparing each against the method's partition (Appendix D). This lets readers directly check which actions the method actually reinforces and suppresses.

Critical weaknesses
  1. The baselines are trained for 150 steps (numbers taken from SDAR and PCSD), while the proposed method is trained for 300 steps (Appendix F). Since the gain over the strongest baseline is only 1.6 points (about 2 of 128 ALFWorld tasks), it cannot be attributed to the method rather than longer training.
  2. The main ideas are not new, and the related work is misdescribed. OPID already suppresses tokens in failed trajectories via negative privileged log-prob shifts, on the same benchmarks and models; the paper describes it as using "retrieved skills", although OPID explicitly avoids retrieval. Skill0 already withdraws privileged skills progressively during training, yet the paper groups it with teacher-gating methods and claims this question is left open.
  3. Most actions selected for reinforcement are invalid. Per Appendix D, 67% of the 311 reinforced actions are invalid (vs. 40% overall). Calling them "supervision regions" does not explain why reinforcing invalid actions helps.
  4. The failure-conditioned retrieval in Eq. (1) is not implemented. The default code returns the first skill of the keyword-matched task type, regardless of the failed trajectory.
Questions for authors
  1. What is the method's performance at step 150, or the baselines' performance after 300 steps?
  2. Were the reported results obtained with the default keyword-based retrieval? If so, what do failure-conditioned retrieval and the applicability gate contribute?
  3. Since 67% of reinforced actions are invalid, why does reinforcing them help? Could you report a control that reinforces only manually labeled correct actions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.