acceptodds
Withdrawn from ICLR 2027

EvolAudit: An Evaluation Framework For Measuring, Interpreting, and Improving Self-Evolving Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Self-evolving agents autonomously improve their capabilities by updating their components based on available feedback. They are typically evaluated by the final task performance. However, aggregate performance may hide performance drops on individual instances even when the overall score improves. Analyzing these performance declines provides fine-grained guidance for improvement. To fill this gap, we propose an audit framework EvolAudit that measures evolution, interprets performance changes, and provides guidance for improving self-evolving agents. During evolution, EvolAudit tracks instance-level changes between iterations, recording a gain when an instance fails in one iteration but succeeds in the next and a regression when the opposite occurs. To assess evolution quality, we define Evolution Efficiency as the proportion of gains among all gains and regressions. For evolution interpretation, we examine component updates at each iteration, and then relate them to instance-level gains/regressions to assess their contribution to performance changes. Building on this analysis, EvolAudit translates instance-level diagnostic insights into targeted component improvements to increase gains relative to regressions. We apply EvolAudit to harness evolution on terminal tasks and memory evolution on web tasks. Experimental results reveal frequent instance-level regressions throughout evolution and show that some updated components are rarely activated during execution. Guided by these findings, targeting previously untouched components improves final performance from 61.77% to 72.06%, showing that EvolAudit can effectively identify evolution failures and guide further improvement.

Primary area:
datasets and benchmarks
Keywords:
Self-Evolving Agents, Agent Evaluation, Instance-Level Analysis, Component-Level Interpretability, Evolution Stability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1PBPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents EvolAudit, a framework for evaluating self-evolving agents beyond their overall task success rates. The main idea is to track how individual task outcomes change throughout evolution, including both improvements and regressions. The authors introduce several metrics, including Evolution Efficiency (EE), Evolution Retention (ER), and observed clade success, to measure these changes. They also analyze which agent components are updated and whether these updates actually affect agent behavior. Based on this analysis, the framework guides further evolution toward underexplored components. Experiments on AHE and ReasoningBank show that self-evolving agents frequently experience instance-level regressions despite improvements in overall performance. The authors further demonstrate that targeting long-term memory in AHE improves its final performance.

Critical strengths
  1. The paper highlights how aggregate performance metrics can conceal instance-level regressions during self-evolution, motivating more fine-grained evaluation.

  2. The framework introduces multiple metrics, including Evolution Efficiency, Evolution Retention, and observed clade success, to characterize different aspects of agent evolution. I think these metrics are useful.

  3. The paper examines how component updates relate to task outcomes and reveals that some updated components remain inactive or unused during execution.

Critical weaknesses
  1. The paper does not adequately distinguish genuine evolution-induced regressions from natural execution fluctuations, particularly since ReasoningBank uses only one rollout per evaluation point.

  2. The proposed metrics largely extend straightforward instance-level performance comparisons, while the interpretation and improvement mechanisms rely on relatively simple component analysis and selection strategies.

  3. The targeted evolution experiment does not compare against simpler strategies, such as directly enabling long-term memory or randomly selecting underexplored components, making it difficult to isolate the contribution of EvolAudit.

  4. The evaluation covers only two self-evolving frameworks, and the AHE held-out set contains just 17 tasks, limiting the generality and statistical reliability of the findings.

Questions for authors
  1. How do the authors distinguish regressions caused by evolution from natural execution variability?

  2. How does the proposed improvement strategy compare with simpler alternatives, such as directly enabling long-term memory or randomly selecting components that have not been updated?

Reviewer_a3wzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduce EvolAudit, a framework for evaluating self-evolving agents beyond their final performance scores and the gain of the single instance. More specifically, it focuses on how individual instances change over time. Their empirical results show that overall performance can improve even when many individual instances regress, and some updated components are rarely used during execution. By targeting an underexplored long-term memory component, they improve final performance from 61.77% to 72.06%.

Critical strengths
  1. I think the authors capture an important challenge of current benchmark evaluation: improve aggregated results are not equal to the individual instances gain. This is a good motivation for the paper. Also, the evolution retention is interesting, it evaluates not only the capability acquisition but also the capability retentions.
  2. They show that EvolAudit can provide actionable information instead of only the post-hoc results and analysis. For example, the analysis of whether the component actually influencing behavior is interesting.
Critical weaknesses
  1. While the motivation and goals are clear and straightforward, the evaluation metric is difficult to understand and follow. I read the paper multiple times to better understand all metrics. I would suggest restructuring this section to make the metric more prominent and briefly introducing it in the introduction.
  2. Evolution Retention might use a relatively loose definition of an acquired capability. For a stochastic agent, one successful run may not indicate a reliably acquired capability. So there is a chance that ER just treat an isolated success as a capability that was subsequently lost.
Questions for authors
  1. Why you set up H=2? Have you try different number for H?
  2. If multiple components change, can the current approach quantify each component’s contribution?
  3. The evolution process tends to favor familiar, simpler component updates. I think this is a common complain for AI agent across multiple tasks. Based on your experiments, is this kind of conservative selection something self-evolving agents should avoid?
Reviewer_MLGBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces EvolAudit to analyze instance-level changes as self-evolving agents evolve. It tracks fail-to-pass and pass-to-fail transitions between consecutive versions. Evolution Efficiency measures the ratio of gains to regressions, while Evolution Retention tracks whether previously solved instances remain solved. The framework also uses component changes, activation records, and execution traces to help explain performance changes. Then they apply it to AHE harness evolution on Terminal-Bench and ReasoningBank memory evolution on WebArena Shopping. The experiments show that overall improvement can come with many flips in individual instance outcomes. Some updated components also do not take effect during execution. Based on the finding that certain components are under explored, the authors guide further evolution of long-term memory. They also report improved performance on held-out tasks.

Critical strengths

What I find most interesting is the distinction between updating a component and having that update actually affect execution. The experiments show that some updated components are not activated. An agent may also take different actions even when given the same memory. This helps the audit pinpoint where improvement gets stuck. It can distinguish between a component not being used and its guidance not being followed. I find this distinction useful for understanding and improving self-evolving systems. I also appreciate the focus on tracking the same set of instances over time. In both AHE and ReasoningBank, some previously successful tasks fail again even as the overall score improves. Tracking these changes shows which successes are retained and which instances keep switching between success and failure. This gives useful information about the reliability of self-evolving systems that the final success rate alone cannot provide.

Critical weaknesses

My main concern is that the link between the audit findings and the final improvement strategy is not clear enough. The paper finds that some updated components are not used. Agents may also take different actions even when given the same memory. These findings raise questions about component triggers and whether agents can correctly apply existing guidance. However, the later intervention mainly targets long-term memory. A component being modified less often does not mean it is the current performance bottleneck. It also does not show that modifying it will address the problems found by the audit. e.g., an agent may fail because it does not follow useful guidance it has already received. The paper needs to explain more clearly why further memory evolution would help in this case. The current results suggest that exploring long-term memory can be useful. But I am still unclear about how instance level gains, regressions, and execution analysis led to this particular intervention. This also limits how well the framework can guide improvements in other systems. Researchers can use EvolAudit to find problems, but still lack a clear basis for deciding what to change. I would like to see a complete example showing how the audit identifies a failure mechanism and guides a corresponding change. It should then show whether the change actually reduces that failure. This evidence would make the claim that auditing can guide improvement much more convincing.

Questions for authors
  1. Which audit findings led you to focus on long-term memory in AHE? How did they shape the specific memory changes? What role did instance-level failure analysis play in this decision?

  2. Could you give a concrete example linking a failure found by the audit to a memory change? Did that change help address the identified failure?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.