acceptodds
Under review as a conference paper at ICLR 2027

Learning to Evolve: Skill–Policy Co-Evolution for Self-Evolving Agents

Abstract

Agent self-evolution has garnered increasing attention, where LLM agents improve their policies by extracting and internalizing reusable skills from their own interactions. However, we find that current methods can exhibit evolutionary stagnation, where policy improvement gradually diminishes over successive evolution rounds as skill-extraction quality degrades and skill extraction becomes increasingly misaligned with policy optimization. In this paper, we introduce a skill-policy co-evolution method (SCALE) that evaluates candidate skills by their effects on long-term policy evolution. Specifically, SCALE learns a skill-conditioned velocity field to model how each skill transforms the trajectory distribution and constructs an evolution-aware reward based on the Wasserstein distance to a reward-reweighted target distribution. It enables the agent to continuously extract skills aligned with its evolving policy and sustain improvement across evolution rounds. We evaluate SCALE on nine challenging benchmarks spanning scientific reasoning, coding, agentic tasks, GUI control, and real-world environment learning. The results show that SCALE consistently surpasses existing methods across diverse model scales and exhibits strong active learning capability from interaction.

Primary area:
generative models
Keywords:
Self-Evolution, Reinforcement Learning, Skill Agent

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_QWQFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper first demonstrates that current agent self-evolution methods suffer from evolutionary stagnation, meaning the effectiveness of self-evolution gradually diminishes as the policy improves. The primary reason is insufficient skill extraction. To address this issue, the authors propose a Skill–Policy Co-Evolution training method (SCALE). It comprises three components: skill-induced trajectory velocity field, which uses a fixed encoder for flow-mathcing in the representation space; simulated evolution and skills evaluation, which scores a skill based on its proximity to the target distribution after simulated and fully evolved evolution; skill-policy self-evolution, which alternately trains the skill extractor and distills skill-conditioned behaviors into the policy. The authors validate the effectiveness of their method using extensive experiments.

Critical strengths
  1. The authors analyze the problems with current methods, pointing out the insufficient skill extraction. This makes their method more motivated, rather than incremental. The authors provide diagnostic analyses, including skill extraction evaluation and gradient cosine similarity analysis (Figures 1 and 2), making their arguments conceptually clear.
  2. The authors conduct extensive empirical and ablation studies to demonstrate the effectiveness of their methods. The paper covers nine benchmarks across various domains. Experiments are performed on models of different sizes, extending to a 397B model, verifying that their methods can scale to larger models. Multiple ablation experiments are conducted to demonstrate the necessity of each component. In addition, the method appears to perform very well, achieving state-of-the-art results on all benchmarks tested.
Critical weaknesses
  1. Some of the authors' statements seem more like plausible guesses or assumptions, lacking necessary and convincing evidence. For example, the authors believe that the reduced progress in self-evolution is due to insufficient skill extraction. While the skill score does decrease, this doesn't necessarily have a causal relationship with the reduced progress. For instance, any training process eventually saturates, and problems the model struggles with naturally become more difficult as training progresses. Furthermore, the skill score measures how much improvement a skill brings to the current model. However, as the model becomes stronger, the room for improvement shrinks, naturally causing the score to drop close to zero. In other words, the score decline might simply mean the model no longer needs the skill as much, rather than a decline in the extraction ability as the authors claim.
  2. The fixed encoder may be inherently inaccurate. The authors use T5 as a fixed encoder, which, as a general text encoder, may primarily capture word similarity rather than correctness. The paper does not prove that a smaller encoded distance equates to better performance.
  3. The target ptarp_{tar} reweights existing trajectories. It rewards skills that gravitate towards embeddings of high-reward trajectories within the same set. This target does not incorporate any new or superior behaviors. Its essence remains to align behavior with existing successful patterns, rather than long-term strategy improvement.
Questions for authors

See Weaknesses.

7 comments

W1: Some of the authors' statements seem more like plausible guesses or assumptions, lacking necessary and convincing evidence. For example, the authors believe that the reduced progress in self-evolution is due to insufficient skill extraction. While the skill score does decrease, this doesn't necessarily have a causal relationship with the reduced progress. For instance, any training process eventually saturates, and problems the model struggles with naturally become more difficult as training progresses. Furthermore, the skill score measures how much improvement a skill brings to the current model. However, as the model becomes stronger, the room for improvement shrinks, naturally causing the score to drop close to zero. In other words, the score decline might simply mean the model no longer needs the skill as much, rather than a decline in the extraction ability as the authors claim.

We would like to clarify the causal relationship between skill extraction degradation and the stagnation of policy improvement from two important observations.

First, the observed performance stagnation does not necessarily indicate that the agent has reached its performance ceiling. Our experiments show that the current agents still have substantial room for improvement, particularly on challenging agentic tasks. The objective of self-evolution is precisely to discover new behavioral strategies that allow the agent to overcome the limitations of its current policy, rather than merely optimizing within its existing capability boundary. Therefore, the diminishing improvement brought by extracted skills should not be simply attributed to the decreasing need for skills as the policy becomes stronger.

Second, importantly, the skill score does not merely approach zero but can even become negative. While a score close to zero may indicate diminishing marginal gains, a negative score suggests that the extracted skills can actually deteriorate the current policy's performance. This goes beyond the explanation that the agent simply needs skills less as it becomes stronger, and instead indicates that the skill extraction process may fail to consistently identify beneficial behavioral changes, potentially producing detrimental skills that interfere with policy optimization.

In addition, the skill score in Figure.1 is intended to measure the agent's skill extraction capability, rather than the quality or utility of the specific extracted skills. Specifically, it reflects the agent's ability to identify and extract reusable behavioral patterns from its interaction experience, which is one of the core capability to achieve consistent policy self-improvement. Therefore, the declining score should not simply be interpreted as the agent becoming less dependent on previously extracted skills as its policy improves. Our concern is whether the agent can continuously discover new and useful skills to overcome the limitations of its current policy.

W2: The fixed encoder may be inherently inaccurate. The authors use T5 as a fixed encoder, which, as a general text encoder, may primarily capture word similarity rather than correctness. The paper does not prove that a smaller encoded distance equates to better performance.

Our choice of a fixed trajectory encoder is motivated by the semantic structure of the learned representation space. Specifically, the encoder maps trajectories with semantically similar content to nearby regions while distinguishing trajectories with different semantic characteristics. In our setting, this property allows the representation space to capture task-relevant differences in reasoning behaviors: trajectories exhibiting correct reasoning and successful task completion are expected to be closer to desirable regions than those containing erroneous reasoning or unsuccessful outcomes. As shown in Figure.20, we provide a visualization of successful and failed trajectories in the learned representation space, which empirically demonstrates their distinguishable distributional patterns. To be specific, successful trajectories form a relatively compact cluster, while failed trajectories generally lie farther from this cluster. This separation suggests that the encoder captures task-relevant semantic distinctions associated with success and failure, providing qualitative support for using the representation space to assess alignment with desirable behaviors.

W3: The target reweights existing trajectories. It rewards skills that gravitate towards embeddings of high-reward trajectories within the same set. This target does not incorporate any new or superior behaviors. Its essence remains to align behavior with existing successful patterns, rather than long-term strategy improvement.

To clarify, the target is constructed from the current policy's rollouts and updated across co-evolution rounds, rather than fixed to an initial set of successful trajectories. Reinforcing promising behaviors discovered through interaction is a central mechanism of reinforcement learning: an individual update uses already collected experience, while subsequent exploration under the updated policy can discover new trajectories. Thus, using observed successful behaviors as an improvement signal does not restrict the entire learning process to reproducing a fixed set of behaviors.

In our framework, skills that promote desirable behavior are internalized into the policy. The updated policy then interacts with the environment to collect fresh trajectories, which provide experience for extracting new skills and reconstructing the target distribution. As the policy improves, it may reach states and discover successful strategies that were rarely or never observed in earlier rounds. Long-term improvement therefore arises from the repeated cycle of exploration, skill extraction, and policy optimization, not from reweighting alone. The reweighted target supplies a policy-dependent improvement signal within each round; new behaviors arise through subsequent interaction and exploration. This mechanism enables improvement beyond the initial trajectory set, although it does not guarantee the discovery of superior behaviors in every round.

The response partially addresses my concerns.

For W1, the authors lack direct experimental or causal evidence demonstrating that slow progress in self-evolution is caused by insufficient skill extraction. Although the observations and ablation studies provide some support, they still do not sufficiently distinguish this explanation from other possible causes. I still consider this overclaiming, and the claims need to be narrowed.

For W2, the paper’s appendix also acknowledges that "this visualization does not establish that embedding distance is a direct measure of correctness or that reducing this distance necessarily improves task performance". Therefore, the separation between successful and failed trajectories does not establish whether the adopted distance can reliably reflect actual behavioral improvement. Of course, this does not mean that T5 is necessarily the wrong choice. The method performs well overall.

For W3, I have no further questions.

Reviewer_uVmyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper provides one new paradigm for agent self evolution from the entry point of skill. Specifically, it involves co training of skill extractions by using GRPO with scores trained by one flow matching policy to predict the agent trajectory, and the policy optimization by self distillation with the extracted skill information.

Critical strengths
  1. The paper is written in very intuitive and clear focus on the motivation and contribution for each part of the method.
  2. The method has superior performance in the listed benchmarks with sufficient empirical supports, ablations, and mechanism explanations.
Critical weaknesses

I think the biggest contribution of this work is to introduce the flow matching surrogate with one Wasserstein metric to evaluate the quality of the skill candidate. Then it provides one way to have the positive cycle of generating good value skills and distill the poilcy from the useful skills. The paper provides figures to show the flow matching can predict the trajectory with good fidelity. However, I think it can be more clear if the papers show why their Wasserstein metric is positively correlated with one good skill.

Questions for authors

This is one very solid and complete work, but I want to get more details and supporting evidences that using the flow matching policy with one fixed encoder, and using one metric to extract the good encoding skill is more convincing. As the skill and agent co-evolves, I think the distribution of new skill that can further improve the current agent can also change, so that with this thinking W distance might not be effective since your positive historical distributions might not be valuable for the next improvement.

3 comments

W1: However, I think it can be more clear if the papers show why their Wasserstein metric is positively correlated with one good skill.

Thank you for this insightful comment. To clarify, the Wasserstein distance measures the minimum transport cost required to transform one trajectory distribution into another, which can be interpreted as the remaining effort needed to align the evolved trajectory distribution with the desired target distribution. In our formulation, a skill is considered effective if it can drive the current trajectory distribution closer to the target through skill-induced evolution. Consequently, a more effective skill leaves a smaller distributional discrepancy, resulting in a lower Wasserstein distance and a higher skill reward. We have clarified this intuition in Section 5.2.

Q1: This is one very solid and complete work, but I want to get more details and supporting evidences that using the flow matching policy with one fixed encoder, and using one metric to extract the good encoding skill is more convincing. As the skill and agent co-evolves, I think the distribution of new skill that can further improve the current agent can also change, so that with this thinking W distance might not be effective since your positive historical distributions might not be valuable for the next improvement.

Our choice of a fixed trajectory encoder is motivated by the semantic structure of the representation space. Specifically, the encoder maps trajectories with semantically similar content to nearby regions while distinguishing trajectories with different semantic characteristics. In our setting, this property allows the representation space to capture task-relevant differences in reasoning behaviors: trajectories exhibiting correct reasoning and successful task completion are expected to be closer to desirable regions than those containing erroneous reasoning or unsuccessful outcomes. As shown in Figure.20, we provide a visualization of successful and failed trajectories in the learned representation space, which empirically demonstrates their distinguishable distributional patterns.

Based on this property, the Wasserstein distance measures the distributional discrepancy between the evolved trajectories and desirable trajectories in a semantically meaningful space. A smaller distance therefore indicates that the evolved behavior is distributionally closer to the high-quality behavior represented by the target distribution.

Importantly, using a fixed encoder does not imply that the desirable skill distribution remains unchanged throughout co-evolution. The target distribution is dynamically reconstructed from the current policy's sampled trajectories through advantage-based reweighting. Thus, the target can adapt to the evolving policy, while the fixed encoder provides a consistent semantic space for measuring distributional changes across training stages.

Reviewer_uVmyOpenReview ↗

Reply to rebuttal

Thanks for your explanation which solves most of my concern. I will keep my positive score.

Thank you so much for your timely response and support for acceptance!

Reviewer_t3bPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes SCALE, a skill-policy co-evolution framework to address evolutionary stagnation in self-improving LLM agents. It introduces a skill-conditioned velocity field and an evolution-aware reward to evaluate the long-term utility of extracted skills. Experiments across nine benchmarks demonstrate consistent improvements over existing approaches.

Critical strengths
  1. The problem is well-motivated. The paper identifies skill-policy misalignment as an important bottleneck in agent self-evolution, supported by empirical gradient analysis.

  2. Interesting methodological contribution with comprehensive evaluation. The proposed flow-based skill evaluation offers a novel perspective on estimating long-term skill utility without repeatedly performing expensive policy rollouts. The authors evaluate SCALE across nine benchmarks and multiple model scales, with encouraging improvements and supporting ablation studies.

Critical weaknesses
  1. Computational overhead and comparison fairness. SCALE introduces an additional 1.3B-parameter velocity model and approximately 1.48× runtime compared with GRPO. Although runtime is reported, compute-matched comparisons are missing, making it difficult to isolate algorithmic improvements from additional computation.
Questions for authors

I noticed that the velocity field is learned from the distribution shift between the original and skill-conditioned trajectories (Eq. 10), but is then repeatedly applied to approximate long-term policy evolution (Eqs. 11–15). However, it is unclear why repeatedly applying the same velocity field in the trajectory embedding space can reliably approximate the actual evolution of a policy undergoing parameter updates. In particular, the skill-induced transformation may change as the policy evolves, and approximation errors may accumulate over multiple steps. Could the authors provide stronger theoretical justification or empirical validation, such as comparing the simulated evolution with actual multi-round policy optimization trajectories?

2 comments

W1: Computational overhead and comparison fairness. SCALE introduces an additional 1.3B-parameter velocity model and approximately 1.48× runtime compared with GRPO. Although runtime is reported, compute-matched comparisons are missing, making it difficult to isolate algorithmic improvements from additional computation.

Thank you for pointing out the importance of computational efficiency and fair comparisons. To address this concern, we provide additional runtime-performance comparisons in Figure.17, demonstrating task success rates against wall-clock training time across different model scales and environments. The results show that SCALE\texttt{SCALE} consistently achieves better performance under comparable time budgets and reaches higher success rates with less training time than the baselines. These results demonstrate that the additional velocity model introduces computational overhead but also brings substantial improvements in training efficiency, allowing SCALE\texttt{SCALE} to achieve better performance within a given computational budget.

Q1: I noticed that the velocity field is learned from the distribution shift between the original and skill-conditioned trajectories (Eq. 10), but is then repeatedly applied to approximate long-term policy evolution (Eqs. 11–15). However, it is unclear why repeatedly applying the same velocity field in the trajectory embedding space can reliably approximate the actual evolution of a policy undergoing parameter updates. In particular, the skill-induced transformation may change as the policy evolves, and approximation errors may accumulate over multiple steps. Could the authors provide stronger theoretical justification or empirical validation, such as comparing the simulated evolution with actual multi-round policy optimization trajectories?

The purpose of the repeated application in Eqs. 11–15 is to estimate the stationary behavioral outcome of skill internalization. Our goal is to obtain the trajectory distribution that results from fully internalizing the skill, i.e., the distribution on which further conditioning on the skill no longer changes the behavior (Eq. 12). Directly evaluating each candidate skill through multi-round policy optimization is prohibitively expensive, and since the quality of extracted skills is not guaranteed, most models trained this way would be discarded. We therefore take an alternative route to reach this fixed point. To be specific, the velocity field quantifies how a trajectory is transformed when conditioning on the skill. Based on this, we introduce an operator Fc\mathcal{F}_c mapping a trajectory distribution to its evolved one, and define the stationary distribution as p∗=Fc(p∗)p*=\mathcal{F}_c(p*) (Eq. 12). To obtain this stationary distribution, we iteratively apply Fc\mathcal{F}_c to the initial distribution until convergence. This iteration reaches the fixed point because the learned velocity field is a continuous transformation that generalizes to the broader trajectory-representation space (Remark 5.1), so an evolved representation can still be fed back into the same field; Eq. 14 thus realizes the standard fixed-point iteration for Eq. 12, and the stopping criterion in Eq. 15 detects when the distribution is approximately invariant under Fc\mathcal{F}_c.

To validate this computation, Figure 19 provides an empirical comparison with actual multi-round policy optimization. As described in Appendix C.3, we start from the same initial trajectory distribution and compare (i) actual policy optimization obtained by repeatedly internalizing a fixed skill into the policy with (ii) recursive evolution under the learned velocity field. The PCA visualizations show close agreement between the predicted terminal distribution and the outcome of actual optimization, together with behavior consistent with attracting fixed points. These observations support using the simulated endpoint as a surrogate for the stabilized behavioral effect of skill internalization.

Dear Reviewer,

As the author–reviewer discussion period is approaching its end, we wanted to kindly follow up to ask whether our rebuttal and ubsequent clarifications have addressed your concerns, or if there are any remaining issues that we could further clarify.

We would be more than happy to provide additional explanations or evidence before the discussion closes. If you feel that your concerns have been sufficiently addressed, we would sincerely appreciate it if you could acknowledge this in the discussion.

Thank you again for your time and consideration!

Best regards, The Authors

Comments on the paper

1 comment

Dear Area Chair, we summarize our responses and the updates made to the paper as follows:

Reviewer t3bP

We would like to clarify that Reviewer t3bP’s score remains at 2 due to the absence in the discussion period. We have provided responses to their concerns; however, without further feedback.

  • Reviewer t3bP asked for computational efficiency-performance comparisons. To address this concern, we provide additional runtime-performance comparisons in Figure.17, and the results show that our method consistently achieves better performance under comparable time budgets and reaches higher success rates with less training time than the baselines.

  • Reviewer t3bP asked for empirical validation of the relationship between skill internalization and policy improvement. We have already provided evidence in the Figure.19, which shows close agreement between the predicted terminal distribution and the outcome of actual optimization.

Reviewer uVmy

Reviewer uVmy described our paper as "one very solid and complete work" and maintained the original positive score as we have address the concerns in the rebuttal.

  • Reviewer uVmy asked for clarification on the Wasserstein metric for skill evaulation. We have further explained the correlation between small Wasserstein distances and high skill scores as better skill can evolve current policy closer to the target one.

  • Reviewer uVmy asked for evidence on the choice of encoder. We have clarified that this encoder maps trajectories with semantically similar content to nearby regions while distinguishing trajectories with different semantic characteristics and provided a visualization of successful and failed trajectories in the learned representation space to demonstrate their distinguishable distributional patterns in Figure.20.

Reviewer QWQF

Reviewer QWQF has raised the score from 2 to 3 for addressing the concerns in the rebuttal.

  • Reviewer QWQF questioned on the correlation between skill extraction capability and improvement stagnation. We have provided additional explanations and evidence to

  • Reviewer QWQF raised concern on the inaccuracy of the encoder. We have clarified that this encoder maps trajectories with semantically similar content to nearby regions while distinguishing trajectories with different semantic characteristics and provided a visualization of successful and failed trajectories in the learned representation space to demonstrate their distinguishable distributional patterns in Figure.20.

  • Reviewer QWQF questioned on the construction of the target distribution. We have clarified that the target is constructed from the current policy's rollout and the updated policy will collect fresh trajectories, which provide experience for extracting new skills and reconstructing the target distribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.