- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Retrospective Self-Distillation (RSD), which improves language agents using their own interaction experience. A teacher receives privileged context consisting of retrieved precedents, accumulated lessons, reflection, and future observations, and its token distributions are distilled into the policy on student-generated trajectories. A further extension uses a frontier model to modify the context-construction policy. Experiments with Qwen3-8B and 14B show improvements over the base models on ALFWorld, ScienceWorld, and SearchQA, with stronger gains on the first two benchmarks.
Critical strengths
-
The method is generally easy to understand and intuitive. The gains on ALFWorld and ScienceWorld are meaningful. The trained agent also works without privileged context at inference, which is a useful practical benefit.
-
The component ablations reveal useful differences across environments: precedents are particularly important for ScienceWorld, while lessons, reflection, and hindsight contribute more strongly on Alfworld. I also appreciate the discussion of limited improvement on SearchQA and the disclosure of unstable teacher dynamics.
Critical weaknesses
-
The incremental contribution over closely related feedback- and context-based self-distillation is not sufficiently established. SDPO (Hubotter et al., 2026) already distills feedback-conditioned retrospective predictions, while OPCD internalizes accumulated experience, including settings without ground-truth labels. These works are cited, but the experiments mainly compare against successful-trajectory training, RL, and gold-trajectory OPSD. A feedback-only or hindsight-summary-only distillation baseline under the same rollout and update budget would help establish what the full retrospective context adds.
-
The overall training cost is unclear. Section 4.2 emphasizes that RSD uses fewer rollouts than BoN-16 and GRPO, but RSD also generates lessons and reflections and re-scores trajectories with additional context. Without reporting these costs, it is difficult to assess the trade-off between fewer environment interactions and additional computation. The introduction of frontier model to adjust the context also brings additional overhead.
Questions for authors
-
How does RSD compare with feedback-only distillation under the same rollout budget?
-
What is the total training cost, including context generation and teacher scoring?
-
In Figure 2, does Random use the same probe and selection budget as Frontier?
-
In Table 1, many baseline methods achieve even lower scores than the baseline. Why is this so? For instance, why does GT-OPSD drop to 3.0 SR on both ScienceWorld models?
Updated Manuscript and Summary of Revisions
We thank all reviewers for their time and constructive feedback. We have updated the PDF manuscript, with revisions highlighted in blue, to clarify teacher stabilization and supervision requirements, add controlled distillation and Random + Probe comparisons, and report computational costs and baseline diagnostics. We have also corrected the final-round reporting and probe descriptions.
With only two days remaining in the discussion period, we would appreciate any further feedback. We hope these revisions address your concerns and would be grateful if you would reconsider your assessment in light of the additional evidence and clarifications.