Frontier-RSI: Advancing the Agent Capability Frontier through Continual Training and Environment Evolution

Recursive self-improvement (RSI) seeks to enable agents to continually expand their capabilities through iterative learning and evolution. However, existing approaches largely treat evolution as an open-loop process: as the agent improves, the evolving training environments are not systematically adapted to its changing capabilities, leaving no closed loop that continually pushes the agent’s capability frontier. We introduce Frontier-RSI, a closed-loop framework that couples agent continual learning, policy-relative task states, and environment evolution. After each learning round, Frontier-RSI measures each task relative to the updated agent and categorizes it as unsolved, frontier, or saturated. A curriculum operator then adapts the environment through two complementary knobs, difficulty and scaffolding, to continuously reshape the task distribution around the agent’s learnable frontier. Specifically, saturated tasks that have become too easy are evolved into harder escalation tasks, while unsolved tasks that remain beyond the agent’s reach are transformed into learnable prefix tasks by providing verified intermediate progress. Experiments across foundation models with different capability levels and coding tasks of varying difficulty demonstrate that Frontier-RSI consistently sustains improvement across successive rounds, effectively advancing the agent’s capability frontier. Code is open-sourced at https://anonymous.4open.science/r/Frontier-RSI-1C0D/.