acceptodds
Under review as a conference paper at ICLR 2027

Experiential Reinforcement Learning

18 views

Abstract

Reinforcement learning has become the central approach for language models (LMs) to learn from environmental reward or feedback. In practice, the environmental feedback is usually sparse and delayed. Learning from such signals is challenging, as LMs must implicitly infer how observed failures should translate into behavioral changes for future iterations. We introduce Experiential Reinforcement Learning (ERL), a training paradigm that embeds an explicit experience-reflection-consolidation loop into the reinforcement learning process. Given a task, the model generates an initial attempt, receives environmental feedback, and produces a reflection that guides a refined second attempt, whose success is reinforced and internalized into the base policy. This process converts feedback into structured behavioral revision, improving exploration and stabilizing optimization while preserving gains at deployment without additional inference cost. Across sparse-reward control environments and agentic reasoning benchmarks, ERL consistently improves learning efficiency and final performance over strong reinforcement learning baselines, achieving gains of up to +81% in complex multi-step environments and up to +11% in tool-using reasoning tasks. These results suggest that integrating explicit self-reflection into policy training provides a practical mechanism for transforming feedback into durable behavioral improvement.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Agent, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_67V6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper propses ERL, which augments language-model reinforcement learning with a first attempt, feedback-conditioned self-reflection, and a revised second attempt. The same model is trained on attempts and reflections using reward-based updates; successful second attempts are additionally distilled into the policy without reflection context. A failure gate restricts retries, and a textual memory carries successful reflections across episodes. The intended deployment policy acts without the training-time reflection-and-retry loop.

Experiments compare ERL with GRPO-based RLVR on FrozenLake, Sokoban, and retrieval-based HotpotQA using Qwen3-4B and Olmo-3-7B. Table 1 reports higher final reward in all six model-task settings: Qwen Sokoban improves from 0.06 to 0.87 and Qwen HotpotQA from 0.45 to 0.56. Wall-clock learning curves and ablations of memory and explicit reflection provide additional evidence.

Critical strengths
  1. The reported gains are substantial and extend across two model families and different task types. Besides Qwen's 0.06-to-0.87 Sokoban improvement, Olmo improves from 0.39 to 0.66 on FrozenLake and from 0.04 to 0.20 on Sokoban (Table 1, p. 8). The control environments use disjoint training and evaluation layouts (Appendix C.1-C.2, pp. 20-24). Figure 4 (p. 6) also reports reward against actual training wall-clock time on the stated hardware, providing practical efficiency evidence beyond iteration counts. The intended ability to retain improvements without deployment-time reflection is valuable.

  2. The ablations address a meaningful alternative explanation and reveal a limitation. The no-reflection variant retains two attempts and supplies the first trajectory, feedback, and a generic improvement instruction (Section 4.4, p. 8; Table 15, p. 30). Full ERL nevertheless improves Qwen Sokoban from 0.59 to 0.87 and FrozenLake from 0.60 to 0.94, supporting the usefulness of explicit reflection beyond simply retrying with history. The paper also reports that removing memory improves Olmo Sokoban from 0.20 to 0.24 and discusses the possibility of propagating inaccurate reflections, rather than hiding this negative result.

Critical weaknesses

1.The core internalization and memory mechanisms are not consistently specified. Appendix A sets the retry threshold to 1, while the distillation loss on and memory update in Algorithm 2 require r^(2) > tau. All three tasks have rewards bounded above by 1, so the literal specification would never distill a second attempt or update memory. This may simply be an inequality typo, but it concerns two central claimed mechanisms and must be reconciled with the implementation. In addition, Section 3.2 (p. 5) says clipping, importance sampling, and KL regularization also apply during internalization, whereas the only explicit main-method distillation objective is a gated negative log-likelihood. The actual loss, its weighting relative to the three policy losses, and the GRPO grouping for reflections are not specified sufficiently to reproduce the method. I regard this as a fixable specification problem, not evidence that these operations were absent from the experiments.

2.The evidence does not yet isolate the distinctive training contribution or support the breadth of the interpretation. The paper appropriately acknowledges feedback-conditioned self/context-distillation approaches in Section 5 (p. 9). Its experiments establish an improvement over standard RLVR and a useful advantage over retries without explicit reflection, but do not isolate the added value of optimizing the reflections themselves or of distilling the revised attempts. Both are central to the claimed training paradigm. A focused comparison with the same reflection-and-retry pipeline but without the reflection policy loss, and an internalization ablation, would materially clarify what the new combination contributes. The broad long-horizon interpretation also exceeds the demonstrated setting: Sokoban has one box and one goal, accepts only layouts solvable within eight moves, and imposes an eight-action budget (Appendix C.2, pp. 21-23). This is meaningful evidence for learning unfamiliar rules in short tasks, but not yet for complex long-horizon planning. Smaller effects, such as Olmo HotpotQA's 0.47-to-0.50 improvement, also lack reported training-seed variability or uncertainty estimates.

Questions for authors

1.Should the successful-retry condition be r^(2) >= 1, or is a different threshold used for memory and distillation? Please give the actual internalization objective, its loss weighting, and the reward-normalization groups used for reflection updates, including how failed-only gating affects those groups. 2.Under the same retry and internalization protocol, how much improvement remains when only the reflection RL loss is removed? Conversely, what happens when only internalization is removed? A focused comparison on a representative setting would clarify whether the gains specifically require jointly learning reflection and consolidating its outputs, beyond the existing no-reflection ablation.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 67V6 (1/3)

Thank you for a careful and constructive review. Your two requested comparisons, removing only the reflection RL loss and removing only internalization, are the core new experiments of this rebuttal. We also agree that the specification problems were real, and we fix them below. Changes in the revised PDF are in blue; p./l. below refer to it.

W1 / Q1. Threshold, internalization objective, weighting and grouping.

Threshold. You are right that, as written, r(2)>τr^{(2)} > \tau can never hold. It is a typo: the implementation uses r(2)≥τr^{(2)} \ge \tau with τ=1\tau = 1. So memory updates and internalization run on every successful second attempt. The new w/o-internalization ablation (below) confirms that this step is active and matters. The fix is in Alg. 1 (l.177), the equations in §2, and Alg. 2 (App. B, l.917).

Internalization objective. Each second attempt y(2)y^{(2)} is also added to the batch as a "distilled" copy: the reflection-conditioned prompt is replaced by the initial prompt xx. The copy's advantage is A=r(2)A = r^{(2)}, with no group normalization. Failed copies therefore have A=0A = 0 and contribute only the KL term. With 0/1 rewards, this is gated imitation of successful second attempts, −1[r(2)≥τ]log⁡πθ(y(2)∣x)-\mathbb{1}[r^{(2)} \ge \tau]\log\pi_\theta(y^{(2)}\mid x), optimized through the same clipped surrogate as the RL terms. This is why §3.2 mentions clipping, importance sampling and KL during internalization. The revision now says this in §2 (l.219).

Loss and weighting. All four sample types (first attempt, reflection, second attempt, distilled copy) share one per-token loss: ℓ=−min⁡(ρA, clip(ρ,0.8,1.28) A)+0.001⋅KL(πθ ∥ πinit),ρ=πθ/πold.\ell = -\min\big(\rho A,\ \mathrm{clip}(\rho, 0.8, 1.28)\,A\big) + 0.001\cdot \mathrm{KL}(\pi_\theta \,\|\, \pi_{\text{init}}),\qquad \rho = \pi_\theta/\pi_{\text{old}}. The KL term uses verl's low-variance estimator; dual clipping caps the policy term at 3∣A∣3|A| when A<0A<0, and there is no entropy term. Only generated tokens count (observations are masked), and each trajectory's advantage is copied to all of its tokens. Each training step's samples are split into mini-batches of 256, with one optimizer step per mini-batch, and the loss is L=1256∑samples∑tokensℓL = \frac{1}{256}\sum_{\text{samples}}\sum_{\text{tokens}}\ell (verl's seq-mean-token-sum). There is no separate coefficient or per-type normalization: a type's weight grows with its number of samples and tokens, and every mini-batch mixes all four types.

Groups and rewards. GRPO normalizes, as (r−rˉ)/(std+10−6)(r-\bar r)/(\mathrm{std}+10^{-6}), within groups defined by task × role, so the types never share a group:

  • First attempts: the task's 4 rollouts.
  • Reflections: one per failed first attempt (0–4 per task). Each reflection's reward is the reward of the second attempt it led to.
  • Second attempts: only the retries of that task (0–4), compared with each other and never with first attempts. A group with a single retry uses A=rA = r.

The failure gate thus shapes the groups: a task whose 4 first attempts all succeed contributes no reflection or second-attempt group, and groups with identical rewards get zero advantage.

We have added these details to §2 (grouping, l.204; internalization, l.219) and to a new "Exact Training Objective" paragraph in App. B (l.1009–1057), so that the method can be reproduced from the text.

AuthorsOpenReview ↗

Response to Reviewer 67V6 (2/3)

W2 / Q2. Isolating the reflection RL loss and internalization.

We ran both comparisons under the same retry and internalization protocol. They are in the new §4.5 and Table 2 (p.9), with full curves and pass@4 in App. F (Tables 19–20).

Setup of the new runs. To fulfill the requested experiments as fully as we could, we spent about 1,770 H100 GPU-hours on new runs during this rebuttal, roughly 9 days of continuous compute on a full 8×H100 node. Because our compute is limited and the rebuttal period is short, we ran these experiments with Qwen3-4B-Instruct-2507 on FrozenLake and Sokoban. We are working to get more compute and will add full experiments (all tasks, both models, multiple seeds) to the camera-ready version.

  • w/o reflection RL loss: the policy still writes reflections, and they still drive the retry, memory and internalization. Only the policy-gradient loss on reflection tokens is removed.
  • w/o internalization: the distillation of successful second attempts is removed. Everything else is unchanged.

Sokoban (validation reward)

Method0255075100125
GRPO0.0600.0720.1500.1400.1770.200
ERL0.0600.0500.2920.5100.7100.762
ERL w/o internalization0.0600.0800.1150.1600.1600.205
ERL w/o reflection RL loss0.0600.0550.1150.1850.5420.762

FrozenLake (validation reward)

Method020406080
GRPO0.2400.2820.4400.6050.772
ERL0.2400.6720.8630.9350.877
ERL w/o internalization0.2400.3200.4850.6800.757
ERL w/o reflection RL loss0.2400.3520.7570.8300.873

Our new runs use the latest version of our training stack (newer rLLM on verl 0.9.1, with better handling of the training–inference mismatch). On it, GRPO rises from 0.06 (paper) to 0.200 at step 125, and ERL keeps a 3.8× margin.

So the gains require consolidating the revised attempts. Without internalization, Sokoban falls to GRPO level. Jointly learning the reflection adds sample efficiency: without it, ERL's step-125 Sokoban score is still reached, but more slowly (0.542 vs. 0.710 at step 100; FrozenLake 0.352 vs. 0.672 at step 20). We now state the contribution in exactly these terms (§4.5, l.480–485).

AuthorsOpenReview ↗

Response to Reviewer 67V6 (3/3)

Does the choice of internalization objective matter? Your W1 asks what the internalization loss actually is. Beyond specifying it, we also tested alternatives. We replaced our supervised objective with three on-policy distillation objectives that match the reflection-free policy πθ(⋅∣x)\pi_\theta(\cdot\mid x) to the reflection-conditioned policy πθ(⋅∣x,Δ)\pi_\theta(\cdot\mid x,\Delta): reverse KL, forward KL and JSD. The setup is otherwise identical (App. B, l.965; Table 4 and Fig. 8, p.19; full curves in App. F, Tables 21–22).

Sokoban│FrozenLake
Step0255075100125│020406080
GRPO0.0600.0720.1500.1400.1770.200│0.2400.2820.4400.6050.772
ERL (supervised, default)0.0600.0500.2920.5100.7100.762│0.2400.6720.8630.9350.877
ERL (reverse KL)0.0600.0850.4030.6130.6900.787│0.2400.5120.8630.9650.932
ERL (forward KL)0.0600.0830.0620.0900.1950.203│0.2400.5620.9270.9680.950
ERL (JSD)0.0600.0620.0830.7550.7620.823│0.2400.3830.8300.9550.938

All methods start from the same checkpoint. Reverse KL and JSD match or exceed our supervised objective, while forward KL stays at GRPO level on Sokoban. So the gain does not depend on our specific supervised loss.

Long-horizon interpretation. We agree. With one box, solutions of at most 8 moves and an 8-action budget, Sokoban is evidence for learning unknown rules in short tasks, as you put it, not for complex long-horizon planning. §4.1 no longer says that Sokoban requires long-horizon planning; it now says ERL helps most where the agent must infer unknown dynamics from sparse feedback (l.362). App. D states that episodes are short-horizon (l.1169).

Small effects and variability. All runs use one seed (App. F states this for the new runs, l.1294), and the revision limits our claims to large effects. For the Olmo HotpotQA gap you mention: the test set has 1,024 questions, each answered once, with a per-question SD of about 0.45. Each score therefore carries a 95% interval of about ±0.028 from test-set and sampling noise. So 0.47 vs. 0.50 is within noise (p ≈ 0.13), and we no longer claim it (§4.1, l.355). Qwen's 0.45 vs. 0.56 is not within noise (z ≈ 5.5). Multi-seed mean ± std for all settings will be in the camera-ready version.

Additional baseline. We also compare against GEPA [Agrawal et al., ICLR 2026], a reflective prompt-evolution method that does not update the weights, using the same 4B model and up to 12k rollouts. It reaches 0.223 on Sokoban, compared with 0.762 for ERL at step 125, which further indicates that consolidating reflections into the weights is what matters (Table 2; App. F, Tables 23–24).

Thank you again for the precise questions; the specification is much clearer as a result. We hope these results address your concerns, and we would appreciate it if you would reconsider your score.

Reviewer_6rtROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Experiential Reinforcement Learning (ERL), a framework that helps language models learn from environmental feedback through self-reflection and reinforcement learning. Instead of relying solely on scalar rewards, ERL allows the model to make an initial attempt, reflect on failures using environmental feedback, and generate a refined second attempt. The framework uses reinforcement learning to optimize both the attempts and reflections, maintains a cross-episode memory of successful reflections, and introduces a self-distillation step to internalize successful behaviors into the base policy without requiring reflection at inference time.

The authors evaluate ERL against standard RLVR on three tasks (FrozenLake, Sokoban, and HotpotQA) using two language models (Qwen3-4B and Olmo3-7B). ERL achieves improvements across all six settings, with particularly large gains on Sokoban. The paper also includes ablation studies on reflection and memory, providing evidence for the contributions of these components.

Critical strengths
  1. Interesting and practically motivated approach. I think the paper addresses an important problem: how to help language models learn more effectively from sparse environmental feedback. The idea of using self-reflection to guide a second attempt and then internalizing the improved behavior is intuitive and potentially useful. I also find the experimental results encouraging, especially the large improvements on Sokoban across both models.

  2. Meaningful ablation studies. I appreciate that the authors do not simply compare ERL against standard RLVR, but also investigate which components contribute to the improvements. In particular, the no-reflection ablation preserves the second attempt and access to previous feedback, making it a useful comparison for understanding the additional benefit of structured reflection. This makes the empirical findings more convincing.

Critical weaknesses
  1. The contribution of internalization is not sufficiently established. I appreciate the ablation studies on reflection and memory, but I think an important missing comparison is ERL without self-distillation. Since the first and second attempts are already optimized through RL using the same model, it is unclear how much additional improvement comes from the internalization step. I also noticed that removing memory has no effect in some settings and even improves performance on Olmo Sokoban. These results make me less convinced about the individual contributions of the proposed consolidation mechanisms, which are central to the paper.

  2. The experiments are limited to relatively simple, short-horizon tasks. The paper emphasizes the benefits of ERL for long-horizon planning and learning from delayed feedback, but FrozenLake and Sokoban have at most eight actions per episode, and Sokoban only involves a single box and goal. HotpotQA also allows at most five interaction turns. While the results are encouraging, I am not sure whether the improvements would extend to more complex agentic tasks with longer trajectories, where identifying errors and generating useful reflections may be substantially more difficult. This limits how strongly the experiments support the broader claims of the paper.

Questions for authors
  1. Clarification on the reward threshold. In Appendix A, the reflection threshold is set to τ=1\tau=1, but Algorithm 2 and the distillation objective use r(2)>τr^{(2)}>\tau to trigger memory updates and internalization. Since rewards in all three environments are bounded by 1, this condition appears impossible to satisfy. Is the intended condition r(2)≥τr^{(2)}\geq\tau, or is a different threshold used in the actual implementation? Could the authors clarify how memory updates and distillation are triggered during training?

  2. Contribution of internalization. The paper includes useful ablations on reflection and memory, but I could not find an ablation removing self-distillation. Since the first and second attempts are already optimized through RL using the same model, how much additional improvement comes from the internalization step? Would it be possible to report results without self-distillation to better isolate its contribution?

  3. Generalization to longer-horizon tasks. The paper emphasizes the benefits of ERL for long-horizon planning and learning from delayed feedback, but the evaluated environments have relatively short interaction horizons. Have the authors investigated whether the improvements persist as the number of steps increases? This would help clarify whether the benefits extend beyond the relatively short tasks considered here.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 6rtR (1/2)

Thank you for the thoughtful review and for recognizing the value of the no-reflection ablation. Your two main concerns are the role of internalization and the short horizon of the tasks. Below, we answer the first with a new experiment and the second by narrowing our claims. Changes in the revised PDF are in blue; p./l. below refer to it.

Setup of the new runs. To fulfill the requested experiments as fully as we could, we spent about 1,770 H100 GPU-hours on new runs during this rebuttal, roughly 9 days of continuous compute on a full 8×H100 node. Because our compute is limited and the rebuttal period is short, we ran these experiments with Qwen3-4B-Instruct-2507 on FrozenLake and Sokoban. We are working to get more compute and will add full experiments (all tasks, both models, multiple seeds) to the camera-ready version.

W1 / Q2. "The contribution of internalization is not sufficiently established."

We ran exactly the ablation you asked for. "ERL w/o internalization" keeps the reflection, the retry, memory and RL on both attempts and the reflection, and removes only the self-distillation step. It is in the new §4.5 and Table 2 (p.9), with full curves and pass@4 in App. F (Tables 19–20).

Sokoban (validation reward)

Method0255075100125
GRPO0.0600.0720.1500.1400.1770.200
ERL0.0600.0500.2920.5100.7100.762
ERL w/o internalization0.0600.0800.1150.1600.1600.205
ERL w/o reflection RL loss0.0600.0550.1150.1850.5420.762

FrozenLake (validation reward)

Method020406080
GRPO0.2400.2820.4400.6050.772
ERL0.2400.6720.8630.9350.877
ERL w/o internalization0.2400.3200.4850.6800.757
ERL w/o reflection RL loss0.2400.3520.7570.8300.873

Our new runs use the latest version of our training stack (newer rLLM on verl 0.9.1, with better handling of the training–inference mismatch). On it, GRPO rises from 0.06 (paper) to 0.200 at step 125, and ERL keeps a 3.8× margin (§4.5, l.469–485).

As you suspected, optimizing both attempts with RL is not enough on its own. Without internalization, Sokoban stays at GRPO level, and FrozenLake keeps only a small part of ERL's lead. The improvements of the second attempt during training reach the reflection-free policy used at test time only through the distillation step. So internalization is the essential component. We also added "ERL w/o reflection RL loss": it catches up with ERL on Sokoban by step 125 (0.762) but more slowly, so training the reflection mainly improves sample efficiency.

AuthorsOpenReview ↗

Response to Reviewer 6rtR (2/2)

On memory. We agree that memory's contribution is small and depends on the setting. It speeds up convergence on Qwen FrozenLake (Fig. 7) and is neutral or slightly harmful elsewhere (Olmo Sokoban, +0.04 without it). The new ablations show that the main gain comes from reflection plus internalization (§4.5), and memory is a secondary component. For the Olmo Sokoban case, the revision also notes that memory is overwritten whenever a later reflection succeeds, which limits but does not prevent inaccurate reflections from persisting (§4.4, l.466).

Q1. Threshold.

You are right; this is a typo. The implementation uses r(2)≥τr^{(2)} \ge \tau with τ=1\tau = 1: memory is updated and internalization runs whenever a second attempt succeeds (reward 1). We fixed Alg. 1 (l.177), the equations in §2, and Alg. 2 (App. B, l.917). We also describe the internalization step in §2 (l.219), with the full objective in App. B (l.1009–1057): each second attempt is added to the batch again with the reflection removed from the prompt, with advantage r(2)r^{(2)}, and trained with the same clipped loss as the RL terms. With 0/1 rewards, this imitates only the successful second attempts.

W2 / Q3. Short-horizon tasks.

We agree, and we have narrowed our claims. FrozenLake and Sokoban have budgets of at most 8 actions, and Sokoban uses one box. Our experiments therefore support a narrower claim than "long-horizon planning". They show that ERL helps an agent learn unknown environment rules and dynamics from sparse 0/1 feedback. Here the difficulty comes from inferring rules that are never stated, not from the length of the episode. Concretely:

  • §4.1 no longer says that Sokoban requires long-horizon planning. It now says ERL helps most where the agent must infer unknown dynamics from sparse feedback (§4.1, l.362). App. D states that layouts are solvable in at most 8 moves, keeping episodes short-horizon (l.1169).
  • On longer trajectories, it is harder to find where an error happened and to write a useful reflection. We expect ERL's benefit to depend on how often a reflection can fix a failed attempt, not only on episode length.
  • With more compute, we plan to add a harder-horizon setting (e.g., Sokoban with more boxes and a larger step budget) to the camera-ready version.

We hope the new internalization ablation and the narrower claims address your concerns. We would be glad to clarify anything further, and we would appreciate it if you would reconsider your score in light of these additions.

Thank you for the detailed response and for running the additional experiments. I appreciate the new ablation without internalization, which directly addresses my main concern. The results are convincing, particularly on Sokoban, where removing internalization brings performance close to the GRPO baseline. I also appreciate the clarification of the threshold issue and the more careful framing of the short-horizon experiments.

One clarification: the rebuttal describes internalization as using an advantage-weighted clipped loss, whereas the original paper presents it as a supervised self-distillation objective. Could the authors clarify the exact objective used in the experiments and how it relates to the original formulation?

Overall, the additional evidence addresses most of my original concerns and improves my assessment of the paper.

AuthorsOpenReview ↗

Follow-up reply to Reviewer 6rtR

Thank you for the follow-up, and we are glad the new ablation addressed your main concern. The paper describes the internalization objective, and our response described its implementation. They are the same, and we will make this explicit in §2.

Objective (paper). Supervised self-distillation of successful second attempts, with the reflection removed from the input:

L(θ)=−E[I(r(2)≥τ)⋅log⁡πθ(y(2)∣x)]L(\theta) = -\mathbb{E}\left[ I(r^{(2)} \ge \tau) \cdot \log \pi_\theta(y^{(2)} | x) \right]

Implementation. Each second attempt is added to the RL batch again, with the reflection removed from its prompt and advantage A=r(2)A = r^{(2)} (1 for a success, 0 for a failure). It then goes through the same clipped policy-gradient loss as the RL terms, with ratio ρ=πθ/πold\rho = \pi_\theta / \pi_{old}, both conditioned on xx.

Why they are equivalent. At the start of each update, πθ=πold\pi_\theta = \pi_{old}, so ρ=1\rho = 1, and the gradient of −ρA-\rho A is

−A⋅∇θlog⁡πθ(y(2)∣x),-A \cdot \nabla_\theta \log \pi_\theta(y^{(2)} | x),

which is exactly the gradient of the SFT loss above. This is the standard identity that a weighted log-likelihood loss is a policy-gradient loss with the weight as the advantage. Failed attempts (A=0A = 0) contribute nothing.

The only differences from plain SFT are the stabilizers mentioned in §3.2:

  1. Clipping as a trust region. Each training step is split into about 4 mini-batches. Once a token's probability has risen more than 28% within a step, its gradient is set to zero.
  2. The same small KL penalty to the initial model (coefficient 0.001) as for all other tokens.
  3. Joint optimization. These samples share mini-batches and loss normalization with the RL terms, with no extra weighting coefficient, instead of a separate SFT phase.

Without these stabilizers, the update is exactly gated SFT on successful second attempts. We implement it inside the RL loss so that the off-policy imitation data shares the RL trust region. We will add this to §2 and App. A. Thank you again for the careful reading and for updating your assessment.

Reviewer_vuvrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Experiential Reinforcement Learning (ERL). When the model’s first attempt at a task does not fully succeed, it writes a reflection and tries the same task again with that reflection. In practice, the reflection is a rewritten system prompt containing the rules and strategy the model has guessed. Both attempts and the reflection are trained with GRPO, and the reflection receives the second attempt’s reward. Successful reflections are stored in a memory. Successful second attempts are distilled into the model without the reflection (“internalization”), so that no reflection is needed at test time. Experiments use FrozenLake, Sokoban, and HotpotQA with Qwen3-4B and Olmo-3-7B. ERL beats plain GRPO in all six settings, most clearly on Sokoban with Qwen model.

Critical strengths

1.ERL works where plain GRPO stalls. On Sokoban with Qwen, plain GRPO stays near 0 reward for the whole run (about 36 hours in Figure 4), while ERL, trained for about the same time, reaches 0.87 (Table 1). With 0/1 rewards, a GRPO group in which every attempt fails has no reward difference to learn from, and ERL’s second attempt is a plausible way around this. This is the main reason I find the paper interesting.

2.Appendix A reports a useful lesson. Reflecting on successful attempts led to instance-specific shortcuts and unstable training, so reflection is applied only when the first attempt does not get full reward.

Critical weaknesses

1.Internalization, the second main contribution, is not tested on its own. The ablations remove memory and reflection, but no variant removes only internalization. RL on the second attempts also updates the model’s weights, so it is unclear how much of the gain comes from internalization.

2.The evidence is thin. The paper does not report random seeds, the number of runs, or error bars, so we cannot tell how much the results vary between runs. FrozenLake and Sokoban are evaluated on only 100 instances each, the HotpotQA test size is not given, and the only baseline is plain GRPO. Without this information, a gap like 0.50 vs. 0.47 (Olmo, HotpotQA) tells us little.

3.There is no comparison with closely related published work. Three related ideas are already published: training reflections with RL (Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization, ICLR 2024), two-attempt RL for self-correction (Training Language Models to Self-Correct via Reinforcement Learning, ICLR 2025), and rewriting the system prompt through reflection (GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning, ICLR 2026). The paper neither cites nor compares with any of them, so it is unclear what combining these pieces adds.

Questions for authors

1.Appendix A sets τ = 1, and the reward is at most 1, so the condition r(2) > τ can never hold. Do memory storage and internalization actually run, and how often?

2.At test time, does the system prompt include the rulebook stored in memory?

3.How do the numbers in Table 1 relate to Figure 4? Table 1 reports “final evaluation reward” and Figure 4 reports “validation reward”, but the paper does not say whether they use the same instances or which checkpoint Table 1 reports. This matters because most Table 1 numbers are above the peaks of the smoothed curves. For example, ERL with Olmo on FrozenLake is 0.66 in the table, but its curve peaks at about 0.56.

2 comments
AuthorsOpenReview ↗

Response to Reviewer vuvr (1/2)

Thank you for the detailed review. We agree with your reading of why ERL works on Sokoban: with 0/1 rewards, all-fail GRPO groups give no signal, and the second attempt turns some of them into useful learning signal. Below, we address each weakness with new experiments. Changes in the revised PDF are in blue; p./l. below refer to it.

Setup of the new runs. To fulfill the requested experiments as fully as we could, we spent about 1,770 H100 GPU-hours on new runs during this rebuttal, roughly 9 days of continuous compute on a full 8×H100 node. Because our compute is limited and the rebuttal period is short, we ran these experiments with Qwen3-4B-Instruct-2507 on FrozenLake and Sokoban. We are working to get more compute and will add full experiments (all tasks, both models, multiple seeds) to the camera-ready version.

W1. "Internalization is not tested on its own."

We added this ablation. "ERL w/o internalization" keeps reflection, retry, memory and RL on both attempts and the reflection, and removes only the distillation of successful second attempts. We also added the converse, "ERL w/o reflection RL loss". Both are in the new §4.5 and Table 2 (p.9), with full curves and pass@4 in App. F (Tables 19–20).

Sokoban (validation reward)

Method0255075100125
GRPO0.0600.0720.1500.1400.1770.200
ERL0.0600.0500.2920.5100.7100.762
ERL w/o internalization0.0600.0800.1150.1600.1600.205
ERL w/o reflection RL loss0.0600.0550.1150.1850.5420.762

FrozenLake (validation reward)

Method020406080
GRPO0.2400.2820.4400.6050.772
ERL0.2400.6720.8630.9350.877
ERL w/o internalization0.2400.3200.4850.6800.757
ERL w/o reflection RL loss0.2400.3520.7570.8300.873

This directly answers your question: RL on second attempts alone does not explain the gain. Without internalization, ERL performs at GRPO level on Sokoban and loses most of its early lead on FrozenLake. The second attempts improve during training, but this does not carry over to the reflection-free policy used at test time. Internalization is the step that carries it over. Training the reflection with RL mainly speeds up learning.

W2. "The evidence is thin": seeds, test sizes, a single baseline.

  • Seeds. Every configuration uses one seed. App. F states this for the new runs (l.1294), the revision reports pass@4 alongside mean reward (App. F, Tables 19–20), and limits our claims to large effects. Multi-seed mean ± std for all settings will be in the camera-ready version.
  • Test sizes. FrozenLake and Sokoban each use 100 held-out layouts × 4 samples (400 episodes), disjoint from training at the layout level. HotpotQA uses the first 1,024 questions of the official HotpotQA distractor development split, each answered once with the validation sampling settings (temperature 0.7, top-p 0.8, top-k 20). The HotpotQA appendix did not state the test size; the revision adds it (App. D, l.1228), and the grid-task appendices now give the number of samples per layout (l.1158).
  • How noisy is a gap like 0.50 vs. 0.47? On HotpotQA the per-question reward SD is about 0.45, so the standard error is 0.45/√1024 ≈ 0.014, and each score carries a 95% interval of about ±0.028. Olmo's 0.47 vs. 0.50 is within noise (unpaired z-test, p ≈ 0.13), and we no longer claim it as an improvement (§4.1, l.355). Qwen's 0.45 vs. 0.56 is not (z ≈ 5.5). These intervals cover test-set and sampling noise only, not training-seed variance. The unpaired test is conservative, because all runs answer the same questions.
  • Baselines. We added GEPA and ablations that isolate each component; see W3.
  • Sokoban baseline. Our new runs use the latest version of our training stack (newer rLLM on verl 0.9.1, with better handling of the training–inference mismatch). On it, GRPO rises from 0.06 (paper) to 0.200 at step 125, and ERL keeps a 3.8× margin (§4.5, Table 2).

W3. No comparison with Retroformer, SCoRe, or GEPA.

We apologize for omitting these; they are closely related. We ran GEPA and position ERL against all three (see Part 2).

AuthorsOpenReview ↗

Response to Reviewer vuvr (2/2)

GEPA (new experiment). GEPA evolves the system prompt by reflecting on execution traces and feedback; the weights stay fixed. It uses the same Qwen3-4B as policy and reflection LM, starts from ERL's initial prompt, never sees gold answers, uses up to 12k rollouts, and is evaluated on the same validation split. It is in Table 2 (p.9), with details in App. F (l.1308; Tables 23–24).

BaseGEPAGRPOERL
FrozenLake0.240.620.770.88
Sokoban0.060.220.200.76
HotpotQA0.250.330.450.56

GEPA validation reward vs. rollouts used

Task01k2k4k8kFinal (rollouts)H100s × wall-clock = GPU-h
FrozenLake0.2400.3120.5930.6230.6230.623 (12.0k)4 × 6.1 h = 24
Sokoban0.0600.2250.2280.2280.2280.223 (9.9k, 16 h cap)4 × 16.3 h = 65
HotpotQA0.2520.2520.2820.3020.3140.331 (12.0k)8 × 1.3 h = 11

GEPA's reflections rewrite a prompt. ERL's reflections are internalized into the weights. On Sokoban, this gives 0.762 vs. 0.223. GEPA levels off within about 2–4k rollouts.

How ERL differs from Retroformer and SCoRe

Who reflectsWhat is trainedFeedbackTest-time cost
RetroformerA separate retrospective LMOnly the reflector (the actor is frozen)Env. reward/feedbackReflection and retries at test time
SCoReThe policy (intrinsic, no external feedback)Both attempts, multi-turn RLNone between attemptsTwo attempts at test time
GEPAThe policy, as a prompt optimizerNothing (prompt only)Traces + feedbackEvolved prompt
ERLThe policyAttempt 1, the reflection and attempt 2, plus internalizationEnv. feedbackSingle attempt, initial prompt

What the combination adds is the one property none of them has: the improvement found by reflection ends up in a policy that needs no reflection, retry or special prompt at test time. The new ablation shows that this step (internalization) is what makes the combination work. Without it, two-attempt RL with trained reflections is no better than GRPO on Sokoban. We have added this discussion to §5 (l.497) and the table to App. A (Table 3, p.18).

Q1. τ = 1 and r(2)>τr^{(2)} > \tau can never hold. Do memory and internalization actually run?

Thank you for catching this. It is a typo: the code uses r(2)≥τr^{(2)} \ge \tau with τ=1\tau = 1. Memory updates and internalization therefore run on every successful second attempt, which is how the w/o-internalization ablation can show a large effect. The fix is in Alg. 1 (l.177), the equations in §2, and Alg. 2 (App. B, l.917).

Q2. Does the test-time system prompt include the memory rulebook?

No. Evaluation uses only the initial system prompt (Table 11), a single attempt, and no memory, reflection or retry. The revision now states this in §3.2 ("Evaluation protocol", l.267).

Q3. Table 1 vs. Figure 4.

Both use the same held-out validation set. Table 1 reports the unsmoothed validation reward of the best checkpoint over the full training run. Figure 4 shows a 5-point trailing moving average and is cut at a fixed wall-clock budget per setting, which for some runs ends before training does. So table values can exceed the plotted curve. In your example, Olmo FrozenLake, ERL reaches 0.66 after the plotted window ends. Because the best checkpoint is chosen on the same set it is scored on, Table 1 is optimistic for every method, ERL and RLVR alike. In the revision, we state this rule (§3.2, l.269; App. E, l.1258), add the unsmoothed curves (App. E, Fig. 9, p.24), and report the new runs at fixed steps (as in our tables above) rather than at the best step.

We hope the new ablations and baselines address your soundness concerns, especially about internalization. If so, we would appreciate it if you would reconsider your assessment. We are happy to answer any further questions during the discussion.

Reviewer_FFSSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper adds a reflect-and-retry loop to RL training for LM agents. The model makes an attempt, gets env feedback, writes a reflection, then makes a second attempt conditioned on that reflection. All three (attempt 1, reflection, attempt 2) are trained with GRPO. Reflections that work get saved in a cross-episode memory, and successful second attempts are distilled back into the policy without the reflection in context, so at test time it's just a normal single-shot policy. Experiments are on FrozenLake and Sokoban (no rules given, sparse 0/1 reward) and HotpotQA with search tools, using Qwen3-4B-Instruct and Olmo-3-7B-Instruct. ERL beats RLVR in all 6 settings, and by a lot on Sokoban with Qwen (0.06 -> 0.87). There's also an ablation dropping memory or reflection.

Critical strengths
  1. The idea is simple and it has a clear practical payoff. You get reflection during training, but the deployed policy doesn't need it, so there's no extra inference cost. That matters because most reflection work (Reflexion, Self-Refine) only helps if you keep paying for reflection at test time.
  2. Some of the gains are big. Sokoban going from 0.06 to 0.87 and FrozenLake from 0.39 to 0.66 (Olmo) aren't the kind of +1 point gains you can explain away, and the pre- vs post-reflection training curves (Fig 6) make a reasonable case that the second attempt is where the useful signal comes from.
Critical weaknesses
  1. It isn't clear where the gain comes from. ERL sees the env's textual feedback f and the RLVR baseline only sees a scalar reward, so part of the improvement may just be extra information rather than "reflection" as such. The w/o-reflection ablation helps a bit, but it still runs a second attempt with the full first-attempt history. I'd want an RLVR baseline that also gets the textual feedback, and a comparison against the concurrent feedback-distillation methods you cite (Hübotter et al., Song et al., Ye et al. 2026), which look like the closest competitors. Right now the only baseline is plain GRPO.
  2. The experimental rigor is thin for the size of the claims. It looks like there's a single run per setting with no seeds or error bars, training was "manually early stopped upon convergence", and the compute matching is described two ways (main text says half as many rollouts per attempt, so 5; Appendix D says 4 per attempt). On HotpotQA the Olmo gap is 0.47 vs 0.50, which could easily be noise. Also, RLVR almost completely fails on Sokoban (0.06 / 0.04). That might be real, but it could also be an under-tuned baseline, and that one result carries a lot of the "+81%" headline.
Questions for authors
  • How many seeds per setting? Can you add variance, at least for HotpotQA and the ablation table?
  • Is the cross-episode memory used at evaluation time, or only during training? If it's used at eval, the "no extra inference cost" claim needs qualifying.
  • Which is right for compute matching, 4 or 5 rollouts per attempt? And is the comparison matched on total generated tokens, given the reflection step adds tokens?
  • Can you add an RLVR baseline that gets the same textual feedback f (e.g. multi-turn RL where the feedback is in context), so we can separate "more information" from "reflection"?
  • For Sokoban, did you try tuning RLVR (more rollouts, longer training, a different temperature)? Why does it stay near 0?
  • In the Olmo Sokoban case the no-memory variant beats full ERL. Do you see bad reflections getting stuck in memory in other settings too?
2 comments
AuthorsOpenReview ↗

Response to Reviewer FFSS (1/2)

Thank you for the constructive review. We are glad you found the zero-inference-cost design practical and the Sokoban and FrozenLake gains hard to explain away. Changes in the revised PDF are in blue; p./l. below refer to it.

Setup of the new runs. To fulfill the requested experiments as fully as we could, we spent about 1,770 H100 GPU-hours on new runs during this rebuttal, roughly 9 days of continuous compute on a full 8×H100 node. Because our compute is limited and the rebuttal period is short, we ran these experiments with Qwen3-4B-Instruct-2507 on FrozenLake and Sokoban. We are working to get more compute and will add full experiments (all tasks, both models, multiple seeds) to the camera-ready version.

W1 / Q4. "ERL sees textual feedback f, RLVR only sees a scalar reward. Is the gain from more information rather than reflection?"

This is an important confound, and our evidence addresses it in three ways.

  • The paper's no-reflection ablation already gives feedback to a second attempt (§4.4; prompt in Table 18). The second attempt sees the full first-attempt trajectory, its rewards and its feedback, and both attempts are trained with RL. It still trails full ERL by 0.28 on Sokoban and 0.34 on FrozenLake (Qwen). So the same information without a structured reflection does not explain the gain.
  • A feedback-only method that does not update the weights is far weaker. We ran GEPA [Agrawal et al., ICLR 2026], in which the same 4B model reads execution traces and feedback and rewrites the system prompt while the weights stay fixed (up to 12k rollouts). It improves on the base model but stays far below ERL (table below; §4.5, Table 2, p.9; App. F, Tables 23–24).
  • What carries the gain is internalization. If RL keeps the reflection and second attempt but drops internalization, Sokoban falls to GRPO level (0.205 vs. 0.200 at step 125; §4.5, Table 2). Feedback in context therefore helps only once the successful revisions are distilled into the reflection-free policy.
BaseGEPAGRPOERL
FrozenLake0.240.620.770.88
Sokoban0.060.220.200.76
HotpotQA0.250.330.450.56

GEPA validation reward vs. rollouts used (score of the best candidate found so far; the best candidate is chosen on GEPA's own selection set, so its validation score can dip slightly)

Task01k2k4k8kFinal (rollouts)H100s × wall-clock = GPU-h
FrozenLake0.2400.3120.5930.6230.6230.623 (12.0k)4 × 6.1 h = 24
Sokoban0.0600.2250.2280.2280.2280.223 (9.9k, 16 h cap)4 × 16.3 h = 65
HotpotQA0.2520.2520.2820.3020.3140.331 (12.0k)8 × 1.3 h = 11

GEPA's gains level off within about 2–4k rollouts.

Concurrent feedback-distillation methods (Hübotter et al.; Song et al.; Ye et al. 2026). Our "ERL w/o Refl." variant (§4.4, Table 1) is almost identical to these methods. In both, a second attempt conditioned on the first attempt and its raw feedback acts as the teacher and is distilled into the policy without that context. The only difference is the internalization loss: we use RAFT/SFT on successful second attempts, while they use on-policy reverse KL. This variant trails full ERL by 0.28 on Sokoban and 0.34 on FrozenLake (Qwen), so the gain comes from the explicit reflection, not from feedback distillation alone. To check that the choice of loss does not drive this, we also ran full ERL with three on-policy distillation objectives in place of RAFT: reverse KL, forward KL and JSD (App. B, l.965, Table 4 and Fig. 8; full curves in App. F, Tables 21–22). Best validation reward over training (Sokoban up to step 150, FrozenLake up to step 100):

InternalizationSokobanFrozenLake
RAFT (default ERL)0.8020.935
Reverse KL0.8680.978
Forward KL0.2350.970
JSD0.8530.993

Reverse KL and JSD, the objectives used by these concurrent methods, match or exceed RAFT, so ERL's reflection step is complementary to them and can benefit from them. Forward KL fails on Sokoban, so the choice of divergence does matter. The revision states the relation to these methods in §4.4 (l.451).

AuthorsOpenReview ↗

Response to Reviewer FFSS (2/2)

W2 / Q1. Seeds and variance. All runs use one seed. App. F states this for the new runs (l.1294), and we limit our claims to effects that are large compared with the step-to-step changes in each curve. We will report mean ± std over multiple seeds in the camera-ready version. On the HotpotQA gaps you mention: the test set has 1,024 questions, each answered once, with a per-question reward SD of about 0.45. Each score therefore carries a 95% interval of about ±0.028 from test-set and sampling noise. Olmo's 0.47 vs. 0.50 is within this noise (unpaired z-test, p ≈ 0.13), and we no longer claim it as an improvement (§4.1, l.355). Qwen's 0.45 vs. 0.56 is well outside it (z ≈ 5.5).

Q2. Is the cross-episode memory used at evaluation?

No. At evaluation the model uses only the first attempt with the initial system prompt, with no memory, reflection or retry. Memory only seeds the reflection prompt during training, so the "no extra inference cost" claim holds as stated. The revision now states this in §3.2 ("Evaluation protocol", l.267).

Q3. 4 or 5 rollouts per attempt? Token matching?

The answer is 4 (the training-details appendix is correct; "half as many" in §3.2 was imprecise and is now fixed, l.261). We matched rollouts per task, not generated tokens. Reflection and second attempts run only after a failed first attempt, so ERL's extra tokens shrink as the policy improves. In practice, ERL costs about the same wall-clock time as GRPO on the same 8×H100 node: 3.7 vs. 3.9 minutes per training step on FrozenLake, and 7.3 vs. 8.5 minutes per step on Sokoban (new runs; App. F, l.1296). In the camera-ready version, we will report generated tokens per step for both methods. Fig. 4 also reports wall-clock time on identical hardware, and on that axis ERL is still ahead on Sokoban and FrozenLake.

Q5. Was RLVR under-tuned on Sokoban?

Thank you for pushing on this. To run the new experiments, we moved to the latest version of our training stack (newer rLLM on verl 0.9.1), which handles the mismatch between the training and inference engines better. With hyperparameters unchanged, GRPO on this stack improves from 0.06 to 0.200 at step 125, and its curve is still rising slowly. ERL reaches 0.762. The gap shrinks but remains large (3.8×). We keep Table 1 as the original runs and report the new runs separately (new §4.5 and Table 2, p.9; App. F). The likely reason GRPO stays low is that, with 0/1 rewards and few successes, most GRPO groups have zero advantage. ERL's second attempt turns some of these all-fail groups into learning signal.

Q6. Do bad reflections get stuck in memory elsewhere?

In Table 1, removing memory makes no difference on Qwen Sokoban and helps only on Olmo Sokoban (+0.04). Olmo is the weakest setting: its first-attempt success rate is low, so early reflections that happen to succeed can encode wrong rules and persist. Memory is overwritten whenever a later reflection succeeds, which limits but does not prevent this; the revision adds this to the discussion of this failure mode in §4.4 (l.466). Note also that memory is not needed for ERL's main gain: the new ablations show that internalization is the essential component.

We hope these additions clarify where ERL's gain comes from. We are happy to discuss further.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response (1/2): new ablations isolating internalization and reflection training

We thank all four reviewers for their careful reviews. We have uploaded a revised PDF in which all changes are in blue; page (p.) and line (l.) numbers below refer to it. Reviewers noted that ERL keeps the benefit of reflection without any test-time cost (FFSS, 67V6). They also noted that it learns where GRPO stalls, notably on Sokoban (all reviewers), and that the no-reflection ablation addresses a real alternative explanation (6rtR, 67V6). Below we cover the concerns raised by several reviewers. Each reviewer thread answers the remaining questions.

Scope of the new experiments. To fulfill the requested experiments as fully as we could, we spent about 1,770 H100 GPU-hours on new runs during this rebuttal, roughly 9 days of continuous compute on a full 8×H100 node. Because our compute is limited and the rebuttal period is short, we ran these experiments with Qwen3-4B-Instruct-2507 on FrozenLake and Sokoban, the two settings where ERL's effect is largest and where the reviewers' questions matter most. We are working to get more compute and will add full experiments (all tasks, both models, multiple seeds) to the camera-ready version.

Summary of new results and clarifications

  1. Internalization is now tested on its own (vuvr W1, 6rtR, 67V6 Q2). We add "ERL w/o internalization" and "ERL w/o reflection RL loss". Removing internalization brings ERL down to GRPO level on Sokoban (§4.5, Table 2, p.9).
  2. The success threshold (vuvr Q1, 6rtR, 67V6 Q1). r(2)>τr^{(2)} > \tau is a typo. The code uses r(2)≥τ=1r^{(2)} \ge \tau = 1, so memory updates and internalization run on every successful second attempt (fixed in Alg. 1 l.177, the equations in §2, Alg. 2 and App. B l.917).
  3. No memory or reflection at test time (FFSS, vuvr Q2). Evaluation uses only the first attempt with the initial system prompt (§3.2, "Evaluation protocol", l.267).
  4. New baseline: GEPA (vuvr W3), a reflective prompt-optimization method run with the same 4B model. It improves the base model but stays far below ERL (Part 2; Table 2 and App. F, Tables 23–24).

Setup for all new runs. Qwen3-4B-Instruct-2507 on an updated training stack (newer rLLM + verl 0.9.1), with the paper's hyperparameters unchanged (ERL: 4 rollouts per attempt; GRPO: 10). Each configuration uses one seed. Evaluation uses only the first attempt, with no reflection or memory, on 100 held-out layouts × 4 samples.

  • ERL w/o internalization: removes the distillation of successful second attempts. Reflection, retry, memory and RL on all three parts are unchanged.
  • ERL w/o reflection RL loss: the policy still writes reflections, and they still drive the retry, memory and internalization, but reflection tokens get no policy-gradient loss.

Sokoban (validation reward)

Method0255075100125
GRPO0.0600.0720.1500.1400.1770.200
ERL0.0600.0500.2920.5100.7100.762
ERL w/o internalization0.0600.0800.1150.1600.1600.205
ERL w/o reflection RL loss0.0600.0550.1150.1850.5420.762

FrozenLake (validation reward)

Method020406080
GRPO0.2400.2820.4400.6050.772
ERL0.2400.6720.8630.9350.877
ERL w/o internalization0.2400.3200.4850.6800.757
ERL w/o reflection RL loss0.2400.3520.7570.8300.873

Takeaways

  • Internalization is necessary. Without it, Sokoban falls to GRPO level (0.205 vs. 0.200 at step 125). On FrozenLake, ERL loses most of its early lead (0.485 vs. 0.863 at step 40). RL on the second attempts alone does not carry the improvement over to the reflection-free deployment policy.
  • Training the reflection with RL mainly improves sample efficiency. Without it, ERL catches up on Sokoban by step 125 (0.762 for both) but learns more slowly: it first reaches 0.5 at step 95 instead of 75 (0.542 vs. 0.710 at step 100). FrozenLake shows the same pattern (0.352 vs. 0.672 at step 20).

Table 2 (p.9) reports these steps for all four methods together with GEPA. Full curves (every 10 steps) with pass@4 are in the new Appendix F (Tables 19–20, p.35).

AuthorsOpenReview ↗

General response (2/2): GEPA baseline, seeds, and list of revisions

GEPA baseline (vuvr W3; also relevant to FFSS's question about learning from feedback alone). GEPA [Agrawal et al., ICLR 2026] evolves the deployment system prompt by reflecting on execution traces and feedback; the model weights stay fixed. We seed it with ERL's initial prompt and use the same Qwen3-4B as the reflection LM, matching GEPA's open-model setup. Rollouts use ERL's solver and sampling settings. The budget is up to 12k rollouts or 16 h, and scores are on ERL's validation split, which GEPA never sees.

TaskBase modelGEPA (best prompt)GRPOERL
FrozenLake0.240.620.770.88
Sokoban0.060.220.200.76
HotpotQA0.250.330.450.56

GEPA validation reward vs. rollouts used (score of the best candidate found so far; the best candidate is chosen on GEPA's own selection set, so its validation score can dip slightly)

Task01k2k4k8kFinal (rollouts)H100s × wall-clock = GPU-h
FrozenLake0.2400.3120.5930.6230.6230.623 (12.0k)4 × 6.1 h = 24
Sokoban0.0600.2250.2280.2280.2280.223 (9.9k, 16 h cap)4 × 16.3 h = 65
HotpotQA0.2520.2520.2820.3020.3140.331 (12.0k)8 × 1.3 h = 11

On all three tasks, GEPA's gains level off within about 2–4k rollouts. Reflection that only rewrites the prompt helps, but far less than ERL, whose reflections are internalized into the weights. A stronger reflection LM might help GEPA; we used the policy itself so both methods use the same model.

On seeds (FFSS, vuvr, 67V6). All runs in the paper and in this response use one seed per configuration. We do not want to overstate small gaps. The revision limits our claims to effects that are large compared with step-to-step fluctuations in each curve (e.g., Sokoban), and labels small differences as such. For example, on the HotpotQA test set (1,024 questions), each score carries a 95% interval of about ±0.028 from test-set and sampling noise. Olmo's 0.47 vs. 0.50 is therefore within noise, while Qwen's 0.45 vs. 0.56 is not (z ≈ 5.5); the revision states this in §4.1 (l.355) and App. D (l.1228). We will report mean ± std over multiple seeds in the camera-ready version.

Changes to the revised PDF (highlighted in blue; p./l. refer to the revised PDF)

  1. Threshold and objective. Fix r(2)>τr^{(2)} > \tau to r(2)≥τr^{(2)} \ge \tau in Alg. 1 (l.177), the equations in §2, and Alg. 2 (App. B, l.917). State how the internalization copy is trained (§2, l.219) and how advantages are grouped (§2, l.204). The full objective, weighting and grouping are in App. B, "Exact Training Objective" (l.1009–1057).
  2. New experiments. The new ablations (w/o internalization, w/o reflection RL loss) and GEPA are in the new §4.5 (l.469–485) and Table 2 (p.9). Full curves with pass@4 are in App. F (Tables 19–20), and the GEPA details are in App. F (Tables 23–24). Alternative internalization objectives (on-policy reverse KL, forward KL, JSD) are in App. B (l.965, Table 4, Fig. 8), with full curves in App. F (Tables 21–22).
  3. Compute matching. ERL uses 4 rollouts per attempt, not "half as many" (§3.2, l.261). Per-step training time is in App. F (l.1296).
  4. Evaluation protocol. First attempt only, with no memory or reflection at test time, plus evaluation set sizes (§3.2, l.267–269 and p.6, l.307; App. D, l.1158 and l.1228). How Table 1 relates to Fig. 4, and the new unsmoothed curves, are in App. E (l.1258, Fig. 8).
  5. Related work. Retroformer, SCoRe and GEPA are added (§5, l.497), with a comparison table (App. A, Table 3).
  6. Claims. Table 1 is unchanged; the new runs on the updated training stack are reported separately (§4.5, App. F). We no longer claim the Olmo HotpotQA gap (§4.1, l.355), and §4.1 no longer says Sokoban requires long-horizon planning.

We thank the reviewers again, and we are happy to answer any follow-up questions before the discussion closes.

AuthorsOpenReview ↗

Reminder: discussion period ends today

Thank you again for your time and feedback. The author-reviewer discussion ends today. If you haven't had a chance yet, we'd really appreciate it if you could look at our responses and the revised PDF, where changes are highlighted in blue. It includes the requested internalization and reflection-loss ablations, a GEPA baseline, and fixes to the issues you raised. Please let us know if any concerns remain, and we're happy to clarify anything before the deadline.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.