- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper propses ERL, which augments language-model reinforcement learning with a first attempt, feedback-conditioned self-reflection, and a revised second attempt. The same model is trained on attempts and reflections using reward-based updates; successful second attempts are additionally distilled into the policy without reflection context. A failure gate restricts retries, and a textual memory carries successful reflections across episodes. The intended deployment policy acts without the training-time reflection-and-retry loop.
Experiments compare ERL with GRPO-based RLVR on FrozenLake, Sokoban, and retrieval-based HotpotQA using Qwen3-4B and Olmo-3-7B. Table 1 reports higher final reward in all six model-task settings: Qwen Sokoban improves from 0.06 to 0.87 and Qwen HotpotQA from 0.45 to 0.56. Wall-clock learning curves and ablations of memory and explicit reflection provide additional evidence.
Critical strengths
-
The reported gains are substantial and extend across two model families and different task types. Besides Qwen's 0.06-to-0.87 Sokoban improvement, Olmo improves from 0.39 to 0.66 on FrozenLake and from 0.04 to 0.20 on Sokoban (Table 1, p. 8). The control environments use disjoint training and evaluation layouts (Appendix C.1-C.2, pp. 20-24). Figure 4 (p. 6) also reports reward against actual training wall-clock time on the stated hardware, providing practical efficiency evidence beyond iteration counts. The intended ability to retain improvements without deployment-time reflection is valuable.
-
The ablations address a meaningful alternative explanation and reveal a limitation. The no-reflection variant retains two attempts and supplies the first trajectory, feedback, and a generic improvement instruction (Section 4.4, p. 8; Table 15, p. 30). Full ERL nevertheless improves Qwen Sokoban from 0.59 to 0.87 and FrozenLake from 0.60 to 0.94, supporting the usefulness of explicit reflection beyond simply retrying with history. The paper also reports that removing memory improves Olmo Sokoban from 0.20 to 0.24 and discusses the possibility of propagating inaccurate reflections, rather than hiding this negative result.
Critical weaknesses
1.The core internalization and memory mechanisms are not consistently specified. Appendix A sets the retry threshold to 1, while the distillation loss on and memory update in Algorithm 2 require r^(2) > tau. All three tasks have rewards bounded above by 1, so the literal specification would never distill a second attempt or update memory. This may simply be an inequality typo, but it concerns two central claimed mechanisms and must be reconciled with the implementation. In addition, Section 3.2 (p. 5) says clipping, importance sampling, and KL regularization also apply during internalization, whereas the only explicit main-method distillation objective is a gated negative log-likelihood. The actual loss, its weighting relative to the three policy losses, and the GRPO grouping for reflections are not specified sufficiently to reproduce the method. I regard this as a fixable specification problem, not evidence that these operations were absent from the experiments.
2.The evidence does not yet isolate the distinctive training contribution or support the breadth of the interpretation. The paper appropriately acknowledges feedback-conditioned self/context-distillation approaches in Section 5 (p. 9). Its experiments establish an improvement over standard RLVR and a useful advantage over retries without explicit reflection, but do not isolate the added value of optimizing the reflections themselves or of distilling the revised attempts. Both are central to the claimed training paradigm. A focused comparison with the same reflection-and-retry pipeline but without the reflection policy loss, and an internalization ablation, would materially clarify what the new combination contributes. The broad long-horizon interpretation also exceeds the demonstrated setting: Sokoban has one box and one goal, accepts only layouts solvable within eight moves, and imposes an eight-action budget (Appendix C.2, pp. 21-23). This is meaningful evidence for learning unfamiliar rules in short tasks, but not yet for complex long-horizon planning. Smaller effects, such as Olmo HotpotQA's 0.47-to-0.50 improvement, also lack reported training-seed variability or uncertainty estimates.
Questions for authors
1.Should the successful-retry condition be r^(2) >= 1, or is a different threshold used for memory and distillation? Please give the actual internalization objective, its loss weighting, and the reward-normalization groups used for reflection updates, including how failed-only gating affects those groups. 2.Under the same retry and internalization protocol, how much improvement remains when only the reflection RL loss is removed? Conversely, what happens when only internalization is removed? A focused comparison on a representative setting would clarify whether the gains specifically require jointly learning reflection and consolidating its outputs, beyond the existing no-reflection ablation.
3 comments
Response to Reviewer 67V6 (2/3)
W2 / Q2. Isolating the reflection RL loss and internalization.
We ran both comparisons under the same retry and internalization protocol. They are in the new §4.5 and Table 2 (p.9), with full curves and pass@4 in App. F (Tables 19–20).
Setup of the new runs. To fulfill the requested experiments as fully as we could, we spent about 1,770 H100 GPU-hours on new runs during this rebuttal, roughly 9 days of continuous compute on a full 8×H100 node. Because our compute is limited and the rebuttal period is short, we ran these experiments with Qwen3-4B-Instruct-2507 on FrozenLake and Sokoban. We are working to get more compute and will add full experiments (all tasks, both models, multiple seeds) to the camera-ready version.
- w/o reflection RL loss: the policy still writes reflections, and they still drive the retry, memory and internalization. Only the policy-gradient loss on reflection tokens is removed.
- w/o internalization: the distillation of successful second attempts is removed. Everything else is unchanged.
Sokoban (validation reward)
| Method | 0 | 25 | 50 | 75 | 100 | 125 |
|---|---|---|---|---|---|---|
| GRPO | 0.060 | 0.072 | 0.150 | 0.140 | 0.177 | 0.200 |
| ERL | 0.060 | 0.050 | 0.292 | 0.510 | 0.710 | 0.762 |
| ERL w/o internalization | 0.060 | 0.080 | 0.115 | 0.160 | 0.160 | 0.205 |
| ERL w/o reflection RL loss | 0.060 | 0.055 | 0.115 | 0.185 | 0.542 | 0.762 |
FrozenLake (validation reward)
| Method | 0 | 20 | 40 | 60 | 80 |
|---|---|---|---|---|---|
| GRPO | 0.240 | 0.282 | 0.440 | 0.605 | 0.772 |
| ERL | 0.240 | 0.672 | 0.863 | 0.935 | 0.877 |
| ERL w/o internalization | 0.240 | 0.320 | 0.485 | 0.680 | 0.757 |
| ERL w/o reflection RL loss | 0.240 | 0.352 | 0.757 | 0.830 | 0.873 |
Our new runs use the latest version of our training stack (newer rLLM on verl 0.9.1, with better handling of the training–inference mismatch). On it, GRPO rises from 0.06 (paper) to 0.200 at step 125, and ERL keeps a 3.8× margin.
So the gains require consolidating the revised attempts. Without internalization, Sokoban falls to GRPO level. Jointly learning the reflection adds sample efficiency: without it, ERL's step-125 Sokoban score is still reached, but more slowly (0.542 vs. 0.710 at step 100; FrozenLake 0.352 vs. 0.672 at step 20). We now state the contribution in exactly these terms (§4.5, l.480–485).
Response to Reviewer 67V6 (3/3)
Does the choice of internalization objective matter? Your W1 asks what the internalization loss actually is. Beyond specifying it, we also tested alternatives. We replaced our supervised objective with three on-policy distillation objectives that match the reflection-free policy to the reflection-conditioned policy : reverse KL, forward KL and JSD. The setup is otherwise identical (App. B, l.965; Table 4 and Fig. 8, p.19; full curves in App. F, Tables 21–22).
| Sokoban | │ | FrozenLake | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Step | 0 | 25 | 50 | 75 | 100 | 125 | │ | 0 | 20 | 40 | 60 | 80 |
| GRPO | 0.060 | 0.072 | 0.150 | 0.140 | 0.177 | 0.200 | │ | 0.240 | 0.282 | 0.440 | 0.605 | 0.772 |
| ERL (supervised, default) | 0.060 | 0.050 | 0.292 | 0.510 | 0.710 | 0.762 | │ | 0.240 | 0.672 | 0.863 | 0.935 | 0.877 |
| ERL (reverse KL) | 0.060 | 0.085 | 0.403 | 0.613 | 0.690 | 0.787 | │ | 0.240 | 0.512 | 0.863 | 0.965 | 0.932 |
| ERL (forward KL) | 0.060 | 0.083 | 0.062 | 0.090 | 0.195 | 0.203 | │ | 0.240 | 0.562 | 0.927 | 0.968 | 0.950 |
| ERL (JSD) | 0.060 | 0.062 | 0.083 | 0.755 | 0.762 | 0.823 | │ | 0.240 | 0.383 | 0.830 | 0.955 | 0.938 |
All methods start from the same checkpoint. Reverse KL and JSD match or exceed our supervised objective, while forward KL stays at GRPO level on Sokoban. So the gain does not depend on our specific supervised loss.
Long-horizon interpretation. We agree. With one box, solutions of at most 8 moves and an 8-action budget, Sokoban is evidence for learning unknown rules in short tasks, as you put it, not for complex long-horizon planning. §4.1 no longer says that Sokoban requires long-horizon planning; it now says ERL helps most where the agent must infer unknown dynamics from sparse feedback (l.362). App. D states that episodes are short-horizon (l.1169).
Small effects and variability. All runs use one seed (App. F states this for the new runs, l.1294), and the revision limits our claims to large effects. For the Olmo HotpotQA gap you mention: the test set has 1,024 questions, each answered once, with a per-question SD of about 0.45. Each score therefore carries a 95% interval of about ±0.028 from test-set and sampling noise. So 0.47 vs. 0.50 is within noise (p ≈ 0.13), and we no longer claim it (§4.1, l.355). Qwen's 0.45 vs. 0.56 is not within noise (z ≈ 5.5). Multi-seed mean ± std for all settings will be in the camera-ready version.
Additional baseline. We also compare against GEPA [Agrawal et al., ICLR 2026], a reflective prompt-evolution method that does not update the weights, using the same 4B model and up to 12k rollouts. It reaches 0.223 on Sokoban, compared with 0.762 for ERL at step 125, which further indicates that consolidating reflections into the weights is what matters (Table 2; App. F, Tables 23–24).
Thank you again for the precise questions; the specification is much clearer as a result. We hope these results address your concerns, and we would appreciate it if you would reconsider your score.
Response to Reviewer 67V6 (1/3)
Thank you for a careful and constructive review. Your two requested comparisons, removing only the reflection RL loss and removing only internalization, are the core new experiments of this rebuttal. We also agree that the specification problems were real, and we fix them below. Changes in the revised PDF are in blue; p./l. below refer to it.
W1 / Q1. Threshold, internalization objective, weighting and grouping.
Threshold. You are right that, as written, r(2)>τ can never hold. It is a typo: the implementation uses r(2)≥τ with τ=1. So memory updates and internalization run on every successful second attempt. The new w/o-internalization ablation (below) confirms that this step is active and matters. The fix is in Alg. 1 (l.177), the equations in §2, and Alg. 2 (App. B, l.917).
Internalization objective. Each second attempt y(2) is also added to the batch as a "distilled" copy: the reflection-conditioned prompt is replaced by the initial prompt x. The copy's advantage is A=r(2), with no group normalization. Failed copies therefore have A=0 and contribute only the KL term. With 0/1 rewards, this is gated imitation of successful second attempts, −1[r(2)≥τ]logπθ(y(2)∣x), optimized through the same clipped surrogate as the RL terms. This is why §3.2 mentions clipping, importance sampling and KL during internalization. The revision now says this in §2 (l.219).
Loss and weighting. All four sample types (first attempt, reflection, second attempt, distilled copy) share one per-token loss: ℓ=−min(ρA, clip(ρ,0.8,1.28)A)+0.001⋅KL(πθ∥πinit),ρ=πθ/πold. The KL term uses verl's low-variance estimator; dual clipping caps the policy term at 3∣A∣ when A<0, and there is no entropy term. Only generated tokens count (observations are masked), and each trajectory's advantage is copied to all of its tokens. Each training step's samples are split into mini-batches of 256, with one optimizer step per mini-batch, and the loss is L=2561∑samples∑tokensℓ (verl's
seq-mean-token-sum). There is no separate coefficient or per-type normalization: a type's weight grows with its number of samples and tokens, and every mini-batch mixes all four types.Groups and rewards. GRPO normalizes, as (r−rˉ)/(std+10−6), within groups defined by task × role, so the types never share a group:
The failure gate thus shapes the groups: a task whose 4 first attempts all succeed contributes no reflection or second-attempt group, and groups with identical rewards get zero advantage.
We have added these details to §2 (grouping, l.204; internalization, l.219) and to a new "Exact Training Objective" paragraph in App. B (l.1009–1057), so that the method can be reproduced from the text.