- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces EvoPathBench, a benchmark for evaluating the evolution of agent capabilities over time. It uses three types of task streams to assess learning generalization, capability retention, and rule adaptation. Experiments on memory- and skill-based methods reveal limitations in transfer, retention, and adaptation. The authors also find that selecting effective skill updates remains a major challenge.
Critical strengths
-
The paper studies an important problem in self-evolving agents. Evaluating capability changes throughout the evolution process provides more information than relying on final performance alone.
-
The benchmark has a clear experimental design, with controlled task streams and held-out evaluations. I also find the analysis of candidate selection and the trade-off between accepting harmful updates and missing useful ones interesting.
Critical weaknesses
-
The paper focuses on artifact-level self-evolution, which limits the scope of the evaluation. The experiments mainly cover memory- and skill-based methods, without considering parameter-based self-evolution. Although the authors also mention workflows as evolving artifacts, I could not find a workflow-evolution method in the experiments. I would like the authors to clarify how broadly the findings can be applied to other forms of self-evolving agents.
-
I have some concerns about the novelty of this work. As shown in Table 1, existing benchmarks such as PAST-Bench[1] and SEA-Gym[2] have already studied several aspects of agent performance during self-evolution, including generalization, retention, and adaptation. The paper also cites SEA-Eval[3] but does not include it in Table 1, despite its closely related evaluation settings. I think the authors should provide a more thorough comparison with these works and better explain what EvoPathBench offers beyond existing benchmarks. This would help readers understand the paper's contribution.
-
All experiments are conducted in the financial trading domain. Although the authors design six task families and two market layers, they still belong to the same domain. I am concerned about whether the findings can generalize to other agent tasks. It would be helpful to evaluate the framework in at least one additional domain.
-
The benchmark uses three types of task streams: Accumulation, Interference, and Reversal. I find these settings somewhat simple compared with how agents evolve in real-world deployments. In practice, agents may need to learn new skills, retain existing ones, and revise outdated rules within the same sequence of tasks. Have the authors considered more complex streams that combine these settings, such as Accumulation with Interference, or all three together? I think such experiments would make the evaluation more realistic.
-
In Section 3.2, the authors state that "the focal capability is established at K2." However, completing two learning episodes does not necessarily mean that the agent has acquired the target capability. This raises a concern about the retention evaluation in the Interference stream. If the agent has not learned the capability at K2, a low retention loss at K5 does not necessarily indicate successful retention. A similar issue exists in the Reversal stream, where the agent may not have learned the original rule before receiving contradictory evidence. I think the authors should provide additional experiments to verify that the target capability has actually been acquired before evaluating retention or rule adaptation.
[1] PAST-Bench: Benchmarking the Foundations of Recursive Self-Improvement in Personal Agents
[2] SEAGym: An Evaluation Environment for Self-Evolving LLM Agents
[3] SEA-Eval: A Benchmark for Evaluating Self-Evolving Agents Beyond Episodic Assessment
Questions for authors
See Critical Weaknesses.
I would be willing to reconsider my score if the authors can more clearly articulate the paper's contributions and distinguish them from prior work.
3 comments
Gentle Follow-up on Our Rebuttal
Dear Reviewer cFyM,
Thank you again for your constructive feedback. We have addressed the issues you raised in our rebuttal. Specifically, we have added supplementary experiments to the revised manuscript (see the Discussion section) and clarified the positioning of our work alongside comparisons with related studies in Appendix A. Additionally, results for non-financial table tasks are provided in Appendix E.
As the discussion period is drawing to a close, we would greatly appreciate any further thoughts or questions you may have regarding these updates. We remain fully available to respond promptly.
Thank you for your time and consideration
Dear authors,
I apologize for my late reply.
I have carefully read your rebuttal and revised manuscript. Although the additional experimental section addressed some of my previous concerns, I still have concerns about the overall contribution, as the differences from the existing benchmark still feel incremental.
Overall, I decide to raise my score to 2.
Revised Manuscripts
We have added the supplementary experiments to the revised manuscript (Discussion section and rebuttal). We also discuss the positioning of our work and its comparison with related work in Appendix A. The nonfinancial table tasks are provided in the Appendix E as well.
Due to the page limit of the main text, we are unable to include all supplementary experiments in the paper. Please also refer to the rebuttal content. Should you have any further questions, we would be happy to address them.
We sincerely appreciate your time and valuable feedback.