acceptodds
Under review as a conference paper at ICLR 2027

Self-Evolving Agents through Privileged Retrospective Distillation

Abstract

Self-evolving agents aim to improve from their own experience rather than remaining fixed once deployed. In practice, however, naturally collected agent trajectories often lack ground-truth solutions or verifier-derived rewards, making them difficult to learn from. We observe that these trajectories become substantially more informative in retrospect. When the agent takes an action, it knows only what has happened so far, but looking back after the trajectory ends, the same action can be judged with knowledge the agent lacked at that moment, including how the environment responded afterward and how related tasks in its accumulated experience succeeded or failed. Based on this observation, we introduce Retrospective Self-Distillation (RSD), a multi-round self-evolution framework in which the same model, given privileged retrospective context built from its own experience, acts as a teacher that re-scores its own actions, and the resulting distribution is distilled back into the policy through on-policy self-distillation. The privileged context combines cross-task precedents and lessons with turn-level reflection and future hindsight, allowing both successful and imperfect trajectories to provide dense supervision. RSD is also built to make this supervision improve alongside the agent. Every round, precedents and lessons are re-derived from the experience bank enlarged with the current policy's trajectories, and reflection and hindsight are computed fresh for each new trajectory, so the teacher always scores the policy against its most recent experience. We further introduce frontier-guided context-policy evolution, in which a stronger frontier model proposes edits to the context construction rules, for example how many precedents to retrieve or which components to include. Across ALFWorld, ScienceWorld, and SearchQA with Qwen3-8B and 14B, RSD improves performance by up to 59.7% relative over the base model, and frontier-guided context-policy evolution adds up to a further 44.7% relative improvement over a fixed context policy. These results show that agents can convert naturally accumulated experience into persistent model capability without ground-truth trajectories or verifier-derived rewards for the core policy update.

Primary area:
foundation or frontier models, including LLMs
Keywords:
self-evolving agent, knowledge distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_VFZZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Retrospective Self-Distillation (RSD), which improves language agents using their own interaction experience. A teacher receives privileged context consisting of retrieved precedents, accumulated lessons, reflection, and future observations, and its token distributions are distilled into the policy on student-generated trajectories. A further extension uses a frontier model to modify the context-construction policy. Experiments with Qwen3-8B and 14B show improvements over the base models on ALFWorld, ScienceWorld, and SearchQA, with stronger gains on the first two benchmarks.

Critical strengths
  • The method is generally easy to understand and intuitive. The gains on ALFWorld and ScienceWorld are meaningful. The trained agent also works without privileged context at inference, which is a useful practical benefit.

  • The component ablations reveal useful differences across environments: precedents are particularly important for ScienceWorld, while lessons, reflection, and hindsight contribute more strongly on Alfworld. I also appreciate the discussion of limited improvement on SearchQA and the disclosure of unstable teacher dynamics.

Critical weaknesses
  • The incremental contribution over closely related feedback- and context-based self-distillation is not sufficiently established. SDPO (Hubotter et al., 2026) already distills feedback-conditioned retrospective predictions, while OPCD internalizes accumulated experience, including settings without ground-truth labels. These works are cited, but the experiments mainly compare against successful-trajectory training, RL, and gold-trajectory OPSD. A feedback-only or hindsight-summary-only distillation baseline under the same rollout and update budget would help establish what the full retrospective context adds.

  • The overall training cost is unclear. Section 4.2 emphasizes that RSD uses fewer rollouts than BoN-16 and GRPO, but RSD also generates lessons and reflections and re-scores trajectories with additional context. Without reporting these costs, it is difficult to assess the trade-off between fewer environment interactions and additional computation. The introduction of frontier model to adjust the context also brings additional overhead.

Questions for authors
  1. How does RSD compare with feedback-only distillation under the same rollout budget?

  2. What is the total training cost, including context generation and teacher scoring?

  3. In Figure 2, does Random use the same probe and selection budget as Frontier?

  4. In Table 1, many baseline methods achieve even lower scores than the baseline. Why is this so? For instance, why does GT-OPSD drop to 3.0 SR on both ScienceWorld models?

Reviewer_xaowOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduced a new self distillation method called Retrospective Self-Distillation (RSD). This method used privileged context to support the teacher model and train the student model based on the student's rollout. The authors claimed that this method can get rid of the requirements of the verifiable rewards.

Critical strengths
  1. The problem this paper tried to solve is meaningful. In many environments, the reliable rewards and verifiers are hard to obtain. A self-improving method without relying on the rewards and verifier is meaningful.
  2. The developed method is very interesting. Its performance is also promising.
Critical weaknesses
  1. The most critical issue of this paper is the claim of 'without ground-truth trajectories or verifier-derived rewards for the core policy update'. Although in the update part, it may not use the ground-truth and the reward. But these information are heavily used during the whole process, especially during the process of constructing of the privileged context. This setting weakened the main benefits of this paper.
  2. The freeze-to-peak teacher method is described in the appendix, but seems to be very important to the whole method. If the teacher model remains updated through the iteration, the performance is quite unstable.
  3. The performance shown in Table 1. illustrated some weakness of RSD. RSD is not working on searchQA and tends to be less effective with larger base models.
Questions for authors
  1. In table 1, are the results reported from the last iteration round, or from the peak performance? The y-axis of Fig. 2 is labeled “Peak success rate”, and the values of the Fixed bars are identical to the RSD results in Table 1.
  2. Why is the performance unstable when the teacher keeps being updated?
Reviewer_zAnoOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Retrospective Self-Distillation (RSD), which trains an agent on its own trajectories using a teacher supplied with retrieved precedents, accumulated lessons, reflection, and future observations. An additional variant uses a frontier model to adapt this context. Experiments show persistent improvements on ALFWorld and ScienceWorld, with more limited gains on search-based question answering.

Critical strengths
  1. Supervising decisions at the states actually visited by the agent is a useful way to exploit imperfect trajectories. The improvements without privileged context at inference—particularly 62.9 to 76.6 on ALFWorld-8B and 15.6 to 23.4 on ScienceWorld-8B—make this more compelling than improvements confined to an expanding inference scaffold.

  2. The component ablations reveal a meaningful distinction between environments: removing precedents nearly eliminates the ScienceWorld gain, whereas lessons and retrospective information matter more on ALFWorld. This provides useful evidence about which experience can be transferred, beyond a single aggregate performance claim.

Critical weaknesses
  1. The successful training procedure relies on an outcome-based teacher-selection mechanism missing from the main algorithm. Appendix B freezes the teacher at the best checkpoint when benchmark performance declines, whereas Eq. 2 and Algorithm 1 describe a teacher tracking the current policy. This matters: the fully evolving teacher in Table 9 collapses to 0% success. Together with completion-based precedent filtering, this leaves the method's dependence on reliable outcome supervision unresolved, even though its loss contains no reward term. The paper also does not identify a separate evaluation set insulated from teacher selection and context-policy acceptance.

  2. The frontier comparison confounds proposal quality with access to empirical selection. Table 6 gives Frontier + Probe candidate-level probe feedback but gives Random none; matching the edit budget does not isolate the value of frontier guidance. The probe itself is inconsistently specified: Appendix D.8 reports sensitivity to removing lessons and describes matching the training context, while Table 16 explicitly says the probe does not respond to lessons, reflection, or future hindsight. Consequently, the additional gains cannot yet be attributed to the claimed context-policy optimization mechanism.

Questions for authors
  1. Which disjoint splits determine teacher freezing, checkpoint selection, and context-policy acceptance, and which produce the reported scores? Please report the teacher-freeze rounds and clarify whether Table 1 contains final-round or selected-checkpoint results. Does a predetermined teacher schedule retain the gains without outcome-based teacher selection?

  2. How does random proposal generation perform with the same candidate probes, selection budget, and acceptance rule as the frontier method? Please also resolve the discrepancy between Appendix D.8 and Table 16 about which context components the probe actually evaluates.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Updated Manuscript and Summary of Revisions

We thank all reviewers for their time and constructive feedback. We have updated the PDF manuscript, with revisions highlighted in blue, to clarify teacher stabilization and supervision requirements, add controlled distillation and Random + Probe comparisons, and report computational costs and baseline diagnostics. We have also corrected the final-round reporting and probe descriptions.

With only two days remaining in the discussion period, we would appreciate any further feedback. We hope these revisions address your concerns and would be grateful if you would reconsider your assessment in light of the additional evidence and clarifications.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.