- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents EvolAudit, a framework for evaluating self-evolving agents beyond their overall task success rates. The main idea is to track how individual task outcomes change throughout evolution, including both improvements and regressions. The authors introduce several metrics, including Evolution Efficiency (EE), Evolution Retention (ER), and observed clade success, to measure these changes. They also analyze which agent components are updated and whether these updates actually affect agent behavior. Based on this analysis, the framework guides further evolution toward underexplored components. Experiments on AHE and ReasoningBank show that self-evolving agents frequently experience instance-level regressions despite improvements in overall performance. The authors further demonstrate that targeting long-term memory in AHE improves its final performance.
Critical strengths
-
The paper highlights how aggregate performance metrics can conceal instance-level regressions during self-evolution, motivating more fine-grained evaluation.
-
The framework introduces multiple metrics, including Evolution Efficiency, Evolution Retention, and observed clade success, to characterize different aspects of agent evolution. I think these metrics are useful.
-
The paper examines how component updates relate to task outcomes and reveals that some updated components remain inactive or unused during execution.
Critical weaknesses
-
The paper does not adequately distinguish genuine evolution-induced regressions from natural execution fluctuations, particularly since ReasoningBank uses only one rollout per evaluation point.
-
The proposed metrics largely extend straightforward instance-level performance comparisons, while the interpretation and improvement mechanisms rely on relatively simple component analysis and selection strategies.
-
The targeted evolution experiment does not compare against simpler strategies, such as directly enabling long-term memory or randomly selecting underexplored components, making it difficult to isolate the contribution of EvolAudit.
-
The evaluation covers only two self-evolving frameworks, and the AHE held-out set contains just 17 tasks, limiting the generality and statistical reliability of the findings.
Questions for authors
-
How do the authors distinguish regressions caused by evolution from natural execution variability?
-
How does the proposed improvement strategy compare with simpler alternatives, such as directly enabling long-term memory or randomly selecting components that have not been updated?