acceptodds
Under review as a conference paper at ICLR 2027

Beyond Endpoint Performance: Process-Level Evaluation of Self-Evolving Agents

Abstract

Self-evolving agents convert interaction feedback into persistent artifacts, such as memories or skills, which in turn guide subsequent decisions. As these artifacts are iteratively updated throughout an experience stream, the capabilities they support may evolve. Consequently, endpoint performance alone offers an incomplete view of self-evolution. Process-level evaluation is therefore essential to identify when a target capability emerges and whether later updates strengthen, preserve, or weaken it. Motivated by this, we propose EvoPathBench, a benchmark that tracks individual capabilities during artifact-level self-evolution. EvoPathBench fixes the base model and tools, freezes evolving artifacts at successive checkpoints, and evaluates the target capability on held-out episodes. This benchmark evaluates agent self-evolution using public trading data and calibrated trajectories. It tests three capabilities: generalization to unseen tasks, retention after unrelated learning, and rule adaptation to new evidence. Experimental results show that gains on similar unseen tasks often weaken under distribution shift, retention losses are concentrated in a minority of evolution paths, and no method achieves reliable rule adaptation. Moreover, while self-evolution enables agents to generate candidate artifacts with substantial held-out gains, the selected updates consistently fall short of realizing this potential. Together, these findings establish capability-level process evaluation as a foundation for analyzing self-evolution, identifying candidate evaluation and selection as key targets for improvement.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Self-evolving agent, agent evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_cFyMOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces EvoPathBench, a benchmark for evaluating the evolution of agent capabilities over time. It uses three types of task streams to assess learning generalization, capability retention, and rule adaptation. Experiments on memory- and skill-based methods reveal limitations in transfer, retention, and adaptation. The authors also find that selecting effective skill updates remains a major challenge.

Critical strengths
  1. The paper studies an important problem in self-evolving agents. Evaluating capability changes throughout the evolution process provides more information than relying on final performance alone.

  2. The benchmark has a clear experimental design, with controlled task streams and held-out evaluations. I also find the analysis of candidate selection and the trade-off between accepting harmful updates and missing useful ones interesting.

Critical weaknesses
  1. The paper focuses on artifact-level self-evolution, which limits the scope of the evaluation. The experiments mainly cover memory- and skill-based methods, without considering parameter-based self-evolution. Although the authors also mention workflows as evolving artifacts, I could not find a workflow-evolution method in the experiments. I would like the authors to clarify how broadly the findings can be applied to other forms of self-evolving agents.

  2. I have some concerns about the novelty of this work. As shown in Table 1, existing benchmarks such as PAST-Bench[1] and SEA-Gym[2] have already studied several aspects of agent performance during self-evolution, including generalization, retention, and adaptation. The paper also cites SEA-Eval[3] but does not include it in Table 1, despite its closely related evaluation settings. I think the authors should provide a more thorough comparison with these works and better explain what EvoPathBench offers beyond existing benchmarks. This would help readers understand the paper's contribution.

  3. All experiments are conducted in the financial trading domain. Although the authors design six task families and two market layers, they still belong to the same domain. I am concerned about whether the findings can generalize to other agent tasks. It would be helpful to evaluate the framework in at least one additional domain.

  4. The benchmark uses three types of task streams: Accumulation, Interference, and Reversal. I find these settings somewhat simple compared with how agents evolve in real-world deployments. In practice, agents may need to learn new skills, retain existing ones, and revise outdated rules within the same sequence of tasks. Have the authors considered more complex streams that combine these settings, such as Accumulation with Interference, or all three together? I think such experiments would make the evaluation more realistic.

  5. In Section 3.2, the authors state that "the focal capability is established at K2." However, completing two learning episodes does not necessarily mean that the agent has acquired the target capability. This raises a concern about the retention evaluation in the Interference stream. If the agent has not learned the capability at K2, a low retention loss at K5 does not necessarily indicate successful retention. A similar issue exists in the Reversal stream, where the agent may not have learned the original rule before receiving contradictory evidence. I think the authors should provide additional experiments to verify that the target capability has actually been acquired before evaluating retention or rule adaptation.

[1] PAST-Bench: Benchmarking the Foundations of Recursive Self-Improvement in Personal Agents

[2] SEAGym: An Evaluation Environment for Self-Evolving LLM Agents

[3] SEA-Eval: A Benchmark for Evaluating Self-Evolving Agents Beyond Episodic Assessment

Questions for authors

See Critical Weaknesses.

I would be willing to reconsider my score if the authors can more clearly articulate the paper's contributions and distinguish them from prior work.

3 comments
AuthorsOpenReview ↗

Revised Manuscripts

We have added the supplementary experiments to the revised manuscript (Discussion section and rebuttal). We also discuss the positioning of our work and its comparison with related work in Appendix A. The nonfinancial table tasks are provided in the Appendix E as well.

Due to the page limit of the main text, we are unable to include all supplementary experiments in the paper. Please also refer to the rebuttal content. Should you have any further questions, we would be happy to address them.

We sincerely appreciate your time and valuable feedback.

AuthorsOpenReview ↗

Gentle Follow-up on Our Rebuttal

Dear Reviewer cFyM,

Thank you again for your constructive feedback. We have addressed the issues you raised in our rebuttal. Specifically, we have added supplementary experiments to the revised manuscript (see the Discussion section) and clarified the positioning of our work alongside comparisons with related studies in Appendix A. Additionally, results for non-financial table tasks are provided in Appendix E.

As the discussion period is drawing to a close, we would greatly appreciate any further thoughts or questions you may have regarding these updates. We remain fully available to respond promptly.

Thank you for your time and consideration

Dear authors,

I apologize for my late reply.

I have carefully read your rebuttal and revised manuscript. Although the additional experimental section addressed some of my previous concerns, I still have concerns about the overall contribution, as the differences from the existing benchmark still feel incremental.

Overall, I decide to raise my score to 2.

Reviewer_kYyAOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EvoPathBench uses 1,800 trading streams to track artifact-level learning through held-out probes of frozen checkpoints. It measures whether gains transfer, survive later updates, and adapt to changed rules. Ten configurations are compared. Transfer gains are limited, some paths incur large losses, and reliable rule revision remains unestablished.

Critical strengths
  • The protocol offers a useful way to assess whether learned capabilities persist through continued evolution. Its checkpoint-based design gives benchmark results a clear interpretation at the level of individual capabilities.
  • The candidate-selection analysis gives the benchmark explanatory value as well as comparative value. Independently evaluating proposed artifacts supports a concrete research direction for improving update selection.
Critical weaknesses
  • Table 7 leaves the aggregation unclear. Under common weighting, E[L] = Pr(L > 0) E[L | L > 0], yet the Baseline values give 0.236 × 3.113 = 0.735 rather than 0.926. Separate macro-averages could explain this difference. The weighting needs clarification to support the frequency-versus-severity interpretation.
  • The reversal results lack a reference showing how much adaptation is achievable within two post-change episodes under the same observations and update budget. This leaves the contribution of task difficulty to the negative result unclear.
Questions for authors
  • When estimating the best available candidate's gain, is the winner chosen and scored on the same held-out runs? Does the selection gap persist with independent scoring runs?
  • How are stored artifacts interpreted after the observation mapping changes in a transfer probe? Can you separate mapping mismatch from failure of the learned decision rule?
2 comments

Thanks for the follow-up. The per-model breakdown clears up my concern about Table 7. The reversal experiment gives me the comparison I was looking for. With two post-change episodes, the updated skill document outperforms the frozen copy on all three evaluation sets.

The candidate-selection gap holds up on fresh execution repeats, and the clarification that the input fields keep the same meaning answers my question about transfer. With those points addressed, I’m raising my rating from 2 to 3.

We sincerely thank the reviewer for the careful re-evaluation and the increased score. We are grateful for the constructive feedback throughout the review process, which has substantially improved the quality of the manuscript.

Reviewer_dhzEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EvoPathBench to evaluate how agent capabilities change as memories and skills are updated over time. It uses ordered task streams and held-out checkpoint evaluations in controlled trading environments to measure generalization, retention, and rule adaptation, while keeping the base model, tools, and executor fixed. The experiments show that gains on familiar tasks can weaken under distribution shift, retention losses can concentrate in a minority of paths, and reliable adaptation to changed rules remains challenging.

Critical strengths
  1. Clear motivation and well-organized experimental design. The paper clearly explains the limitations of endpoint evaluation and organizes the study around generalization, retention, and adaptation. The three corresponding task-stream templates, fixed execution setup, and held-out checkpoint evaluations provide a sensible way to examine these questions separately.
  2. Extensive experiments with useful insights. Beyond comparing multiple agent configurations, the paper examines computational cost, candidate selection, forgetting patterns, task order, and skill transfer across models. These analyses offer useful insights.
Critical weaknesses
  1. The scope of the research is somewhat narrow. The study focuses primarily on prompts (skills), which represent only a small fraction of the broader topic of "self-evolving" systems. Consequently, the generalizability of the findings to the evolution of other components within an agent framework remains limited.
  2. The appendix mentions the use of both Qwen3.8-Max and Kimi-K3; however, the paper fails to present full primary results for each model individually, nor does it specify which base model corresponds to the data in the main tables. It is therefore unclear whether the conclusions hold consistently across different models.
  3. The claims regarding candidate selection are somewhat overreaching. The authors argue that when the number of candidates increases, better candidates are generated but fail to be selected—implying that selection is the primary bottleneck. Yet, it remains unproven whether a "higher-scoring candidate" in a single update step actually leads to superior downstream performance. The current results could alternatively suggest that local gains in candidate quality do not reliably translate into long-term, transferable improvements.
  4. Regarding the statement on line 427, "...compare them with SFT...", the origin of the SFT models is not clearly defined. For instance, details such as the number of trajectories used, or the criteria for filtering and transforming training samples, are missing.
Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.