- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Reflection-Enhanced Self-Distillation (RESD) for learning in environments with rare successful trajectories but informative execution feedback. It augments a feedback-conditioned self-teacher with a reflection on the current failure, a persistent global playbook of reusable lessons, and cached successful solutions when available. The student learns from the enriched teacher through on-policy token-level distillation. The main contribution is the construction and maintenance of teacher context rather than a new distillation objective. Experiments on MANUFACTORIA-HAS, two BOUNCINGSIM tasks, and FINER report improvements over SDPO variants, with the largest gains on MANUFACTORIA-HAS. Additional results examine matched interaction and training-time budgets, component ablations, instruction following, and LiveCodeBench, where the authors appropriately characterize the gains over competing methods as statistically inconclusive.
Critical strengths
Substantial improvement in a relevant low-success regime. On MANUFACTORIA-HAS, Table 2 reports per-task mean@4 accuracy of 35.80% for RESD, compared with 0.95% for SDPO and 0.57% for SDPO+ss. Per-test-case mean@4 rises to 76.95%, versus 37.66% and 37.97%. This is a substantial practical improvement in a setting where successful demonstrations are scarce. Appendix D also provides evidence beyond equal-step comparisons: RESD reaches 76.9% per-test-case accuracy within 2,976 environment interactions, whereas GRPO remains at 29.1% within that budget. The matched training-time results on BOUNCINGSIM-EASY further support the intended interaction-efficiency claim despite the extra reflection and curation cost.
Critical weaknesses
The contribution is not sufficiently distinguished from closely related experience-distillation methods. The paper cites SDPO and ACE, but the supplied reference list omits OEL [1], Skill-SD [2], and ERL [3]. OEL accumulates transferable knowledge from textual interaction feedback and internalizes it through on-policy context distillation. Skill-SD summarizes trajectory successes, mistakes, and workflows into teacher-only skill context for token-level self-distillation. ERL combines explicit reflection, cross-episode reflection memory, and distillation of successful revised attempts. Consequently, the contrast in Section 5 between instance-local self-distillation and training-free experience accumulation is incomplete. RESD may still contribute a useful single-rollout, failure-driven implementation with an evolving global playbook, but the current comparisons do not establish its incremental benefit over these closer alternatives. A focused comparison on the rare-success setting is more consequential than adding citations alone; I would not require reproducing every related method. 2. The specified training rule is undefined for a central regime of the paper. Appendix B assigns successful examples weight (1 − s)^α and failed examples weight s^β, then normalizes the weights to have batch mean one; α = β = 1. If a batch contains only failures (s = 0), every raw weight is zero and the normalization is undefined. The same issue occurs for an all-success batch (s = 1). he missing boundary-case behavior is particularly important because learning without successful rollouts is a central claim. References [1] Ye et al. Online Experiential Learning for Language Models. https://arxiv.org/abs/2603.16856 [2] Wang et al. Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents. https://arxiv.org/abs/2604.10674 [3] Shi et al. Experiential Reinforcement Learning. https://arxiv.org/abs/2602.13949
Questions for authors
Clarification of the implemented loss. What exactly happens when s = 0 or s = 1 in Appendix B? Please give the implemented normalization and fallback, the frequency of these cases in the affected runs, and whether the same weighting and fallback are used for the relevant baselines. This would directly address weakness 2. 2. Clarification of the contribution and a targeted comparison. How does RESD differ operationally from OEL, Skill-SD, and ERL in experience extraction, memory persistence, and parameter updates? Can you provide a matched-budget comparison with a closely related experience-conditioned distillation method on at least the principal rare-success task, or existing evidence isolating the benefit of the proposed online reflection/playbook update? This would directly affect the novelty and significance assessment. 3. Clarification of interaction accounting. Section 4.1 says each training example is seen at most once, but also specifies up to K = 4 updates on the same prompts, and Appendix K shows successive generated attempts. Does each inner iteration generate and evaluate a fresh rollout? Is each such iteration counted as one step in Table 5, so that its interaction count includes all retries? Please distinguish unique problem arrivals, policy updates, and environment evaluations. I am not assuming that the reported budget is incorrect.