acceptodds
Under review as a conference paper at ICLR 2027

VEL: Verifiable Experiential Learning for Recursive Self-Improving Agents

Abstract

Large language model (LLM) agents are increasingly applied to long-horizon tasks that require sustained interactions and iterative decision-making. In such environments, agents need to learn from their previous attempts, improving future behaviors by extracting and reusing useful knowledge from past experiences. This capability, known as experiential learning, is essential for agents to progressively adapt and improve beyond their initial capabilities. Recent approaches have sought to enhance experiential learning through reinforcement learning (RL), yet they typically rely on coarse-grained and implicit outcome-based rewards, providing limited guidance on how agents should leverage prior experiences. We propose VEL, an RL framework for verifiable experiential learning. VEL isolates the experiential learning capability through a clear cross-episode formulation: the agent makes multiple sequential attempts on the same task and leverages experience distilled from previous attempts to guide future decisions. To reinforce experiential learning with fine-grained and verifiable feedback, VEL introduces the experience-guided advancement reward (EGAR), which is derived from improvements in verifiable task progress between consecutive episodes. Experiments on four representative benchmarks show that VEL outperforms RL baselines and existing experiential learning approaches, yielding an average relative improvement of 49.3% from the first to the third attempt by leveraging prior episodes. We further demonstrate that agents trained with VEL in a single environment exhibit transferable experiential learning capabilities in out-of-distribution environments, enabling both same-task generalization and cross-task transfer. These results indicate that VEL equips LLM agents with transferable experiential learning capabilities, supporting recursive self-improving agents.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agent, Reinforcement Learning, Experiential Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_oBm5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VEL trains agents to distill experience using environment-verifiable progress differences between consecutive attempts, so that reflection after failure actually helps the next retry rather than merely producing plausible-sounding summaries. Across four task types, cumulative success over the first three attempts improves by 49.3% on average relative to the first attempt, and the method shows preliminary cross-environment transfer.

Critical strengths
  1. For long-horizon interactive tasks, distilling useful information from failures may be more valuable than simply raising single-attempt success rates.

  2. In environments with computable progress, this reduces noise from subjective evaluation.

  3. Table 3 examines the reward and grouping design, and Table 6 examines the cross-episode information carrier.

Critical weaknesses

W1. Insufficient comparison with the closest baseline (Table 1). All 63 baseline success rates in Table 1 match LaMer's Table 1 exactly, yet the paper claims all methods are evaluated under its own cross-episode, experience-only protocol.

Please mark which results are cited versus re-run, and explain how LaMer's GiGPO optimizer is reconciled with VEL's GRPO.

More importantly, LaMer's reflection-only configuration, closest to VEL's experience-only setting, is not compared. Its reported p@3 (80.5 / 56.4 / 92.8) exceeds the LaMer numbers here (74.4 / 51.6 / 89.1) and VEL on Sokoban (55.5).

Please compare against it under the same split, optimizer, test budget, and checkpoint selection.

W2. Limited novelty relative to prior work. The novelty of VEL is not sufficiently established.

  • SCoRe [1] already uses the reward difference between consecutive attempts to encourage improvement, although its shaping bonus is applied to the second response rather than an intermediate experience summary.
  • Moreover, LaMer [2] already trains cross-episode self-reflection using subsequent-episode rewards and evaluates a reflection-only memory configuration.

Thus, neither improvement-based reward shaping nor the cross-episode reflection framework is new in itself. The paper should explicitly position these components against the existing mechanisms and provide controlled evidence that their combination offers a substantive advance, rather than primarily benefiting from additional dense progress supervision.

[1] Training Language Models to Self-Correct via Reinforcement Learning. ICLR, 2025.

[2] Meta-RL Induces Exploration in Language Agents. ICLR, 2026.

W3. EGAR does not causally attribute progress to experience quality (Eq. 7). Even with a fixed task and policy, the progress difference also reflects action sampling, environment stochasticity, and regression to the mean. Moreover, given h=(x,τn)h = (x, \tau^n), vnv^n acts as a constant baseline that leaves the expected policy gradient unchanged. EGAR's gains may thus come from altered baselines, variance, and advantage ranking rather than causal attribution.

Please fix the previous trajectory and actor, and compare real, empty, shuffled, and other-task experience over multiple rollouts.

W4. Weak evidence for generalization and recursive improvement (Table 2). Transfer from MineSweeper to ALFWorld is a valuable experiment, but one model outperforming another does not show a stronger experience-transfer mechanism. The gap may reflect general execution ability, prompt following, or training recipe. Table 2 compares only training methods and cannot isolate the contribution of cross-task experience transfer.

Please compare the same VEL model with

  • no prior-task experience
  • correct same-category experience
  • shuffled experience
  • unrelated-category experience
  • fixed generic strategy text

W5. The method relies on hand-constructed verifiers. All four benchmarks have well-defined progress structures, but open-ended real-world tasks may lack computable shortest-solution distances, full environment states, reliable intermediate milestones, or continuous completion scores.

For example, BFS-based progress in Sokoban is reasonable for small boards but does not scale to large, open-ended planning problems. The paper thus addresses how to learn from experience given a reliable progress signal, but not how to obtain such a signal automatically in general environments.

Please discuss this limitation and how progress signals could be constructed beyond these benchmarks.


This work reads more as an effective reward/advantage recipe than as convincing evidence of the claimed independent, attributable, and recursively transferable experience-learning capability.

Questions for authors

See Weakness

Reviewer_719QOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

To address the problem of how agent reinforcement learning can extract and reuse experience from failed or incomplete attempts, this manuscript proposes VEL, a framework that explicitly integrates cross-episode experiential learning into reinforcement learning optimization. By defining an experience reward based on verifiable task progress differences alongside an advantage estimation normalized by attempt stage, VEL achieves competitive multi-attempt success rates across four benchmarks including MineSweeper, Sokoban, WebShop, and ScienceWorld, while demonstrating preliminary cross-environment transfer on ALFWorld. In my view, this manuscript presents clear research value, but it still requires key control experiments and statistical verification to substantiate its core arguments.

Critical strengths
  1. The manuscript addresses an important and practically meaningful research problem, with a sound alignment between the research motivation and the methodological design.
  2. The method design is concise and intuitive. By optimizing experience distillation through verifiable progress difference signals between consecutive attempts, it requires neither additional LLM-as-a-Judge training nor reliance on complex external retrieval pipelines, demonstrating sound technical rationality.
  3. The experiments achieve optimal multi-attempt success rates across four benchmark environments with distinct characteristics, and the manuscript provides ablation experiments in the appendix comparing passing no experience, passing full trajectories, and passing distilled experience, which offers solid empirical support.
  4. The manuscript further examines both same-task multi-attempt and cross-task experience transfer in the out-of-distribution environment ALFWorld, presenting a relatively comprehensive experimental design.
  5. The manuscript is well structured. The mathematical formulations of the method, the definitions of progress signals across environments, and the prompt templates are described in detail, which contributes to good readability and potential reproducibility.
Critical weaknesses
  1. The manuscript repeatedly highlights that EGAR provides direct, attributable rewards for experiential learning. However, rEGARnr_{\mathrm{EGAR}}^n in the formulation is essentially just the difference between two actual execution outcomes rather than the counterfactual gain brought by utilizing a specific piece of experience. In my view, the current experiments are insufficient to rigorously validate the claimed precise credit assignment mechanism.
  2. The core mechanism of the manuscript relies heavily on the verifiable progress signals provided by the environment. However, the verifiability of VAS does not equate to its alignment with the true task objective, and its scalability remains questionable.
  3. The manuscript lacks statistical significance analysis across multiple random seeds, which casts doubt on the robustness of some key findings. In particular, in the ALFWorld cross-task generalization experiment in Table 2, the test set contains only 78 tasks across 6 categories. VEL only succeeds in approximately 6 more tasks compared to GRPO, which I find insufficient to support a strong claim of generalization.
  4. The title and main text of the manuscript employ terms such as Recursive Self-Improving Agents and transferable atomic capability, yet the empirical evidence can hardly substantiate such broad claims. The primary training environments are relatively small in scale with short interaction horizons. Furthermore, during test time, the model only takes experience text as context input without online parameter updates, which is essentially test-time in-context adaptation.
  5. Although the manuscript aligns the upper limit on the number of trajectories per parameter update, having the same trajectory limit does not equate to identical training computational costs and inference overhead. The computational budget matching and baseline coverage in the comparative experiments remain somewhat incomplete. Additionally, while closely related methods such as SAGE and ERL are discussed in the related work, the main experiments do not include direct comparisons against them.
Questions for authors
  1. Regarding the first weakness, could the authors provide paired control experiments comparing ground-truth experience, empty experience, and randomly permuted experience under fixed tasks and prior trajectories, in order to verify whether the performance gain truly originates from the experience content rather than random resampling variance?
  2. Regarding the second weakness, could the authors provide an alignment analysis between VAS and final task success rates across environments, and discuss the performance of VEL when progress signals are noisy or when exact optimal distances cannot be obtained?
  3. Regarding the third weakness, could the authors provide statistical results across multiple random seeds to substantiate the findings?
  4. Regarding the fourth and fifth weaknesses, I suggest providing performance comparisons under a unified generation token budget or training compute budget, clarifying why SAGE and ERL were not included as direct baselines, and appropriately toning down the scope of the claims.
  5. When early task completion reduces the number of valid samples within a group, what proportion of zero advantages or extremely small sample groups occurs in Transition-Wise Advantage, and how does this affect the late-stage training dynamics?
Reviewer_cEFwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces VEL, a RL framework of experiential learning for LLM agents. In particular, VEL adapts a cross-episode formulation of training where the agent make multiple attempts on the same task sequentially, and leverage the experience distilled from previous attempt to guide the following execution. The authors further present experience-guided advantage reward (EGAR), which encourages the improvement between consecutive progress. Experiments on four agentic environments demonstrate the effectiveness of VEL compared to existing RL baselines.

Critical strengths

The paper tackles an important challenge in training LLM agents to learn from previous experiences. The proposed method provides a simple and intuitive way to reward experience distillation based on improvements in task progress, rather than relying solely on binary task outcomes. The experiments demonstrate strong empirical results of VEL compared to the baselines. The results of ablation and generalization are also interesting.

Critical weaknesses

My major concerns is the reliance to the enviroment specific verifiable signal vn=V(x,τn)v^n=V(x,\tau^n). I understand that vnv^n would provide richer learning signal compared to binary reward, which is especially useful for experiential learning. However, it requires environment-specific reward signal and might not be applicable to all scenarios. In some realistic problem, the environment might only contains binary reward, and then rEGARn=vn+1−vnr^n_{EGAR} = v^{n+1} − v^n would be {-1, 0, 1}, which could be sparse and less effective as the learning signal.

Questions for authors
  1. How would VEL work in environments with only binary reward? I see there is an ablation in Table 3, but I'm wondering if there're better and more uniform way to estimate the partial success in this case, such as using a learned value network?
  2. In VEL, the Experience-Guided Advancement Reward rEGARnr^n_{EGAR} is only assigned to the distillation step, not the task execution tokens. Can the authors explain the intuition? Have you tried to train actions also with rEGARnr^n_{EGAR}?
  3. The cross-episode learning framework has been presented in several exsiting works, such as LaMer, SAGE, and ERL as describeed in Section 2. Can the authors further clarify the key insights of VEL compared to the previous works? This would be helpful to understannd the contribution of VEL.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.