- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper first demonstrates that current agent self-evolution methods suffer from evolutionary stagnation, meaning the effectiveness of self-evolution gradually diminishes as the policy improves. The primary reason is insufficient skill extraction. To address this issue, the authors propose a Skill–Policy Co-Evolution training method (SCALE). It comprises three components: skill-induced trajectory velocity field, which uses a fixed encoder for flow-mathcing in the representation space; simulated evolution and skills evaluation, which scores a skill based on its proximity to the target distribution after simulated and fully evolved evolution; skill-policy self-evolution, which alternately trains the skill extractor and distills skill-conditioned behaviors into the policy. The authors validate the effectiveness of their method using extensive experiments.
Critical strengths
- The authors analyze the problems with current methods, pointing out the insufficient skill extraction. This makes their method more motivated, rather than incremental. The authors provide diagnostic analyses, including skill extraction evaluation and gradient cosine similarity analysis (Figures 1 and 2), making their arguments conceptually clear.
- The authors conduct extensive empirical and ablation studies to demonstrate the effectiveness of their methods. The paper covers nine benchmarks across various domains. Experiments are performed on models of different sizes, extending to a 397B model, verifying that their methods can scale to larger models. Multiple ablation experiments are conducted to demonstrate the necessity of each component. In addition, the method appears to perform very well, achieving state-of-the-art results on all benchmarks tested.
Critical weaknesses
- Some of the authors' statements seem more like plausible guesses or assumptions, lacking necessary and convincing evidence. For example, the authors believe that the reduced progress in self-evolution is due to insufficient skill extraction. While the skill score does decrease, this doesn't necessarily have a causal relationship with the reduced progress. For instance, any training process eventually saturates, and problems the model struggles with naturally become more difficult as training progresses. Furthermore, the skill score measures how much improvement a skill brings to the current model. However, as the model becomes stronger, the room for improvement shrinks, naturally causing the score to drop close to zero. In other words, the score decline might simply mean the model no longer needs the skill as much, rather than a decline in the extraction ability as the authors claim.
- The fixed encoder may be inherently inaccurate. The authors use T5 as a fixed encoder, which, as a general text encoder, may primarily capture word similarity rather than correctness. The paper does not prove that a smaller encoded distance equates to better performance.
- The target reweights existing trajectories. It rewards skills that gravitate towards embeddings of high-reward trajectories within the same set. This target does not incorporate any new or superior behaviors. Its essence remains to align behavior with existing successful patterns, rather than long-term strategy improvement.
Questions for authors
See Weaknesses.
7 comments
W3: The target reweights existing trajectories. It rewards skills that gravitate towards embeddings of high-reward trajectories within the same set. This target does not incorporate any new or superior behaviors. Its essence remains to align behavior with existing successful patterns, rather than long-term strategy improvement.
To clarify, the target is constructed from the current policy's rollouts and updated across co-evolution rounds, rather than fixed to an initial set of successful trajectories. Reinforcing promising behaviors discovered through interaction is a central mechanism of reinforcement learning: an individual update uses already collected experience, while subsequent exploration under the updated policy can discover new trajectories. Thus, using observed successful behaviors as an improvement signal does not restrict the entire learning process to reproducing a fixed set of behaviors.
In our framework, skills that promote desirable behavior are internalized into the policy. The updated policy then interacts with the environment to collect fresh trajectories, which provide experience for extracting new skills and reconstructing the target distribution. As the policy improves, it may reach states and discover successful strategies that were rarely or never observed in earlier rounds. Long-term improvement therefore arises from the repeated cycle of exploration, skill extraction, and policy optimization, not from reweighting alone. The reweighted target supplies a policy-dependent improvement signal within each round; new behaviors arise through subsequent interaction and exploration. This mechanism enables improvement beyond the initial trajectory set, although it does not guarantee the discovery of superior behaviors in every round.
The response partially addresses my concerns.
For W1, the authors lack direct experimental or causal evidence demonstrating that slow progress in self-evolution is caused by insufficient skill extraction. Although the observations and ablation studies provide some support, they still do not sufficiently distinguish this explanation from other possible causes. I still consider this overclaiming, and the claims need to be narrowed.
For W2, the paper’s appendix also acknowledges that "this visualization does not establish that embedding distance is a direct measure of correctness or that reducing this distance necessarily improves task performance". Therefore, the separation between successful and failed trajectories does not establish whether the adopted distance can reliably reflect actual behavioral improvement. Of course, this does not mean that T5 is necessarily the wrong choice. The method performs well overall.
For W3, I have no further questions.
W1: Some of the authors' statements seem more like plausible guesses or assumptions, lacking necessary and convincing evidence. For example, the authors believe that the reduced progress in self-evolution is due to insufficient skill extraction. While the skill score does decrease, this doesn't necessarily have a causal relationship with the reduced progress. For instance, any training process eventually saturates, and problems the model struggles with naturally become more difficult as training progresses. Furthermore, the skill score measures how much improvement a skill brings to the current model. However, as the model becomes stronger, the room for improvement shrinks, naturally causing the score to drop close to zero. In other words, the score decline might simply mean the model no longer needs the skill as much, rather than a decline in the extraction ability as the authors claim.
We would like to clarify the causal relationship between skill extraction degradation and the stagnation of policy improvement from two important observations.
First, the observed performance stagnation does not necessarily indicate that the agent has reached its performance ceiling. Our experiments show that the current agents still have substantial room for improvement, particularly on challenging agentic tasks. The objective of self-evolution is precisely to discover new behavioral strategies that allow the agent to overcome the limitations of its current policy, rather than merely optimizing within its existing capability boundary. Therefore, the diminishing improvement brought by extracted skills should not be simply attributed to the decreasing need for skills as the policy becomes stronger.
Second, importantly, the skill score does not merely approach zero but can even become negative. While a score close to zero may indicate diminishing marginal gains, a negative score suggests that the extracted skills can actually deteriorate the current policy's performance. This goes beyond the explanation that the agent simply needs skills less as it becomes stronger, and instead indicates that the skill extraction process may fail to consistently identify beneficial behavioral changes, potentially producing detrimental skills that interfere with policy optimization.
In addition, the skill score in Figure.1 is intended to measure the agent's skill extraction capability, rather than the quality or utility of the specific extracted skills. Specifically, it reflects the agent's ability to identify and extract reusable behavioral patterns from its interaction experience, which is one of the core capability to achieve consistent policy self-improvement. Therefore, the declining score should not simply be interpreted as the agent becoming less dependent on previously extracted skills as its policy improves. Our concern is whether the agent can continuously discover new and useful skills to overcome the limitations of its current policy.
W2: The fixed encoder may be inherently inaccurate. The authors use T5 as a fixed encoder, which, as a general text encoder, may primarily capture word similarity rather than correctness. The paper does not prove that a smaller encoded distance equates to better performance.
Our choice of a fixed trajectory encoder is motivated by the semantic structure of the learned representation space. Specifically, the encoder maps trajectories with semantically similar content to nearby regions while distinguishing trajectories with different semantic characteristics. In our setting, this property allows the representation space to capture task-relevant differences in reasoning behaviors: trajectories exhibiting correct reasoning and successful task completion are expected to be closer to desirable regions than those containing erroneous reasoning or unsuccessful outcomes. As shown in Figure.20, we provide a visualization of successful and failed trajectories in the learned representation space, which empirically demonstrates their distinguishable distributional patterns. To be specific, successful trajectories form a relatively compact cluster, while failed trajectories generally lie farther from this cluster. This separation suggests that the encoder captures task-relevant semantic distinctions associated with success and failure, providing qualitative support for using the representation space to assess alignment with desirable behaviors.