acceptodds
Under review as a conference paper at ICLR 2027

Learning with Rare Success but Rich Feedback via Reflection-Enhanced Self-Distillation

Abstract

Enabling Large Language Models (LLMs) to continuously improve from environmental interactions is a central challenge in post-training. While on-policy self-distillation offers a promising paradigm, existing methods predominantly treat environmental feedback as a passive conditioning signal. Consequently, they heavily rely on successful demonstrations and struggle to learn in rare-success regimes. To bridge this gap, we introduce Reflection-Enhanced Self-Distillation (RESD), a framework that transforms raw failure feedback into an active source of corrective supervision. Instead of passively appending feedback, RESD interprets failed trajectories by generating retrospective reflections to diagnose local errors, and curates a persistent global playbook to preserve reusable lessons across training steps. The enriched context enables the self-teacher to provide actionable token-level supervision even in the absence of successful rollouts. Empirical evaluations on multiple continual learning tasks demonstrate that RESD substantially outperforms standard self-distillation baselines. Furthermore, RESD achieves significantly faster early-stage improvement than GRPO with samples using only a single rollout per prompt, highlighting its superior interaction efficiency.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, self-reflection, feedback, self-distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Xd4QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Reflection-Enhanced Self-Distillation (RESD) for learning in environments with rare successful trajectories but informative execution feedback. It augments a feedback-conditioned self-teacher with a reflection on the current failure, a persistent global playbook of reusable lessons, and cached successful solutions when available. The student learns from the enriched teacher through on-policy token-level distillation. The main contribution is the construction and maintenance of teacher context rather than a new distillation objective. Experiments on MANUFACTORIA-HAS, two BOUNCINGSIM tasks, and FINER report improvements over SDPO variants, with the largest gains on MANUFACTORIA-HAS. Additional results examine matched interaction and training-time budgets, component ablations, instruction following, and LiveCodeBench, where the authors appropriately characterize the gains over competing methods as statistically inconclusive.

Critical strengths

Substantial improvement in a relevant low-success regime. On MANUFACTORIA-HAS, Table 2 reports per-task mean@4 accuracy of 35.80% for RESD, compared with 0.95% for SDPO and 0.57% for SDPO+ss. Per-test-case mean@4 rises to 76.95%, versus 37.66% and 37.97%. This is a substantial practical improvement in a setting where successful demonstrations are scarce. Appendix D also provides evidence beyond equal-step comparisons: RESD reaches 76.9% per-test-case accuracy within 2,976 environment interactions, whereas GRPO remains at 29.1% within that budget. The matched training-time results on BOUNCINGSIM-EASY further support the intended interaction-efficiency claim despite the extra reflection and curation cost.

Critical weaknesses

The contribution is not sufficiently distinguished from closely related experience-distillation methods. The paper cites SDPO and ACE, but the supplied reference list omits OEL [1], Skill-SD [2], and ERL [3]. OEL accumulates transferable knowledge from textual interaction feedback and internalizes it through on-policy context distillation. Skill-SD summarizes trajectory successes, mistakes, and workflows into teacher-only skill context for token-level self-distillation. ERL combines explicit reflection, cross-episode reflection memory, and distillation of successful revised attempts. Consequently, the contrast in Section 5 between instance-local self-distillation and training-free experience accumulation is incomplete. RESD may still contribute a useful single-rollout, failure-driven implementation with an evolving global playbook, but the current comparisons do not establish its incremental benefit over these closer alternatives. A focused comparison on the rare-success setting is more consequential than adding citations alone; I would not require reproducing every related method. 2. The specified training rule is undefined for a central regime of the paper. Appendix B assigns successful examples weight (1 − s)^α and failed examples weight s^β, then normalizes the weights to have batch mean one; α = β = 1. If a batch contains only failures (s = 0), every raw weight is zero and the normalization is undefined. The same issue occurs for an all-success batch (s = 1). he missing boundary-case behavior is particularly important because learning without successful rollouts is a central claim. References [1] Ye et al. Online Experiential Learning for Language Models. https://arxiv.org/abs/2603.16856 [2] Wang et al. Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents. https://arxiv.org/abs/2604.10674 [3] Shi et al. Experiential Reinforcement Learning. https://arxiv.org/abs/2602.13949

Questions for authors

Clarification of the implemented loss. What exactly happens when s = 0 or s = 1 in Appendix B? Please give the implemented normalization and fallback, the frequency of these cases in the affected runs, and whether the same weighting and fallback are used for the relevant baselines. This would directly address weakness 2. 2. Clarification of the contribution and a targeted comparison. How does RESD differ operationally from OEL, Skill-SD, and ERL in experience extraction, memory persistence, and parameter updates? Can you provide a matched-budget comparison with a closely related experience-conditioned distillation method on at least the principal rare-success task, or existing evidence isolating the benefit of the proposed online reflection/playbook update? This would directly affect the novelty and significance assessment. 3. Clarification of interaction accounting. Section 4.1 says each training example is seen at most once, but also specifies up to K = 4 updates on the same prompts, and Appendix K shows successive generated attempts. Does each inner iteration generate and evaluate a fresh rollout? Is each such iteration counted as one step in Table 5, so that its interaction count includes all retries? Please distinguish unique problem arrivals, policy updates, and environment evaluations. I am not assuming that the reported budget is incorrect.

Reviewer_dPQ2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Reflection-Enhanced Self-Distillation (RESD) for learning in environments where successful rollouts are rare but execution feedback is informative. RESD augments the self-teacher’s context with retrospective reflections on failed trajectories, a persistent playbook of reusable lessons, and cached successful solutions when available. The student learns from the resulting teacher distributions through on-policy self-distillation. Experiments across four tasks show improvements over self-distillation baselines, with particularly large gains on MANUFACTORIA-HAS. Additional analyses examine component contributions and performance under matched interaction and wall-clock budgets.

Critical strengths

The method provides a useful integration of failure reflection and experience accumulation into self-distillation. Explicitly interpreting feedback and preserving reusable lessons offers a practical way to improve supervision when successful demonstrations are scarce.

Critical weaknesses

The separation between training, validation, and test data is unclear. Table 1 lists training and test splits, while Section 4.1 describes checkpoint selection using validation metrics. Without an explicit description of an independent validation split, it is difficult to determine whether the reported test results provide an evaluation independent of checkpoint selection.

Questions for authors
  1. Could you clarify the training, validation, and test splits for each task? Is there an independent validation set for checkpoint selection, with the test set evaluated only after selecting the checkpoint?

  2. How does the implementation handle batches containing only successes or only failures when normalizing the weights in Appendix B? Please specify any smoothing or fallback rule and whether these implementation details will be included in the released code.

Reviewer_LFKKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies self-distillation when successful rollouts are rare. The authors observe that SDPO relies heavily on successful peer demonstrations, while raw failure feedback alone provides weak supervision. They propose RESD, which generates reflections on failed rollouts and maintains a persistent playbook of reusable lessons. These are added to the self-teacher's context to produce token-level supervision. Successful solutions are also cached for reuse. Experiments on Manufactoria, BouncingSim, and FINER show improvements over SDPO, especially on Manufactoria where the initial success rate is near zero. RESD also improves faster than GRPO in early training, with fewer environment interactions.

Critical strengths
  1. Learning from failures is an important problem, and the proposed approach is simple and effective. The SDPO batch-size ablation (Fig. 3) provides a clear motivation: without successful peer demonstrations, standard SDPO struggles, while adding reflection substantially improves performance.

  2. The results on Manufactoria are impressive. RESD improves per-task accuracy from below 1% to 35.8%, despite using only one rollout per prompt per update. The matched interaction and wall-clock comparisons also help substantiate its efficiency advantage in early training.

Critical weaknesses
  1. The gains are less convincing outside Manufactoria. On BouncingSim, the final per-task accuracy remains quite low (4.25% and 7.25%). Improvements on FINER are more modest. The paper also does not report variance across training seeds, which is concerning given the substantial fluctuations in the training curves (Fig. 6). It is hard to judge how robust the improvements are.

  2. The novelty relative to existing reflection and memory methods is somewhat limited. Reflection and persistent playbooks have been studied previously. The main contribution here is incorporating them into self-distillation. While this combination works, it is not entirely clear how much of the improvement comes from the distillation mechanism itself versus additional reasoning and experience accumulation in the teacher context.

Questions for authors
  1. Can the authors report results across multiple training seeds, particularly for BouncingSim and FINER? How consistent are the gains and late-stage regressions?

  2. How would a training-free approach using the same reflection and playbook compare? This would help clarify the additional benefit of distilling the accumulated knowledge into model weights.

  3. The method uses K=4 inner-loop iterations on the same prompts. Does "single rollout per prompt" refer to each iteration rather than the entire training encounter? Please clarify this distinction in the interaction-efficiency claim.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.