VEL: Verifiable Experiential Learning for Recursive Self-Improving Agents

Large language model (LLM) agents are increasingly applied to long-horizon tasks that require sustained interactions and iterative decision-making. In such environments, agents need to learn from their previous attempts, improving future behaviors by extracting and reusing useful knowledge from past experiences. This capability, known as experiential learning, is essential for agents to progressively adapt and improve beyond their initial capabilities. Recent approaches have sought to enhance experiential learning through reinforcement learning (RL), yet they typically rely on coarse-grained and implicit outcome-based rewards, providing limited guidance on how agents should leverage prior experiences. We propose VEL, an RL framework for verifiable experiential learning. VEL isolates the experiential learning capability through a clear cross-episode formulation: the agent makes multiple sequential attempts on the same task and leverages experience distilled from previous attempts to guide future decisions. To reinforce experiential learning with fine-grained and verifiable feedback, VEL introduces the experience-guided advancement reward (EGAR), which is derived from improvements in verifiable task progress between consecutive episodes. Experiments on four representative benchmarks show that VEL outperforms RL baselines and existing experiential learning approaches, yielding an average relative improvement of 49.3% from the first to the third attempt by leveraging prior episodes. We further demonstrate that agents trained with VEL in a single environment exhibit transferable experiential learning capabilities in out-of-distribution environments, enabling both same-task generalization and cross-task transfer. These results indicate that VEL equips LLM agents with transferable experiential learning capabilities, supporting recursive self-improving agents.