- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes EVOTRACE, a benchmark for evaluating self-evolving LLM agents on procedural tasks. The core concept, AE-SOP (Agent-Executable Standard Operating Procedure), formalizes standard operating procedures as a graph structure with operation nodes, decision nodes, and dependency edges, hidden from the agent and used only as evaluation ground truth. The benchmark covers 115 AE-SOP families and 10,792 executable instances, drawn from six major procedural sources (τ-bench, SOPBench, etc.) and seven application domains. The core contribution is an "experience → retention → execution" three-stage tracing framework that can pinpoint whether a specific procedural requirement was never encountered in experience, encountered but not retained, or retained but incorrectly executed, accompanied by a metric suite (FSR, AMR, KP/KR/KC, K2B). The authors evaluate 11 representative self-evolving mechanisms on two backbones, GPT-5.6-Terra and Qwen3.8-Flash-Next.
Critical strengths
This paper establishes a requirement-level pathway tracing mechanism (experience → retention → execution), supported by theory. Theorems H.1-H.4 prove that aggregate metrics cannot uniquely determine the execution success rate distribution of retained requirements, providing formal justification for the necessity of fine-grained diagnosis rather than relying on intuition alone. The experiments show that broader coverage ≠ more reliable execution and validate the actionability of diagnostic conclusions through targeted interventions.
Critical weaknesses
-
In the experiment, the false acceptance rate for execution-based judgments was as high as 34.5%, the inconsistency rate between native and procedural successful judgments was 20.5%, and the AMR corrections showed opposite signs on the two backbones. The authors should provide more discussion on these biases.
-
The paper does not provide a sensitivity analysis of the errors on downstream diagnostic conclusions, making it difficult for readers to determine how many specific "bottleneck attributions" are reliable.
-
The diagnostic framework is essentially more of a descriptive association than a causal mechanism. Although the intervention experiments showed positive effects, there was a lack of control over confounding variables (e.g., retention repair might simultaneously improve product quality and trigger better in-context inference on the backbone). The authors should supplement this with more ablation experiments or case studies.
Questions for authors
-
Could you provide an estimate of the propagated uncertainty of measurement error on core diagnostic metrics such as K2B and KC?
-
AE-SOP extraction uses Claude Opus 4.8, GPT-5.6-sol, and Qwen3.8-2.4T-A95B, while the evaluation backbones are GPT-5.6-Terra and Qwen3.8-Flash-Next from the same families. Could this model-family overlap cause the extracted AE-SOPs to implicitly favor the procedural reasoning patterns of the same model family, thereby affecting the relative ranking of different methods?
-
Do you plan to conduct causal mediation analysis to verify which stage truly causes execution failure, rather than merely identifying bottlenecks at the correlational level?
6 comments
Response to Reviewer nR1o – 2
3. Corrected states and the reliability of conclusions (W2: bottleneck attributions; Q1: downstream diagnostic conclusions).
- Record-level changes. For the 50 linked records per backbone, the percentages of diagnostic-state and retained-miss changes are:
| Record changes (%, 95% interval) | GPT | Qwen |
|---|---|---|
| Changes in diagnostic states | 18.0 [8.0, 30.0] | 22.0 [12.0, 34.0] |
| Changes in retained-miss states | 22.0 [12.0, 34.0] | 18.0 [8.0, 30.0] |
- Method-level comparisons. We recompute artifact-absent execution misses and execution misses for operation, decision, and dependency requirements across all methods (Figure 37). The direction of the method-specific contrast remains stable on both backbones. For example, we compare decision artifact-miss rates for MemP-online and ReasoningBank:
| Backbone | Gap before correction | Corrected MemP-online | Corrected ReasoningBank | Gap after correction |
|---|---|---|---|---|
| GPT | 55.2 pp | 41.6% | 8.5% | 33.1 pp |
| Qwen | 57.8 pp | 43.0% | 7.9% | 35.1 pp |
- Process–outcome associations. After calibration, we report changes in the Pearson and Spearman correlations between the four process metrics and FSR/AMR, together with their intervals (Table 25). K2B remains the process metric with the strongest positive association with FSR and AMR on both backbones.
| Pearson r: corrected minus original (interval) | GPT | Qwen |
|---|---|---|
| K2B–FSR | +0.077 [0.046, 0.109] | +0.080 [0.050, 0.104] |
| KC–FSR | +0.048 [0.021, 0.130] | +0.043 [0.019, 0.117] |
- Together with the record-level changes, specific method comparisons, and learning curves across all five checkpoints, these analyses show how measurement errors affect states and effect sizes. They support the stability of the reported key diagnostic comparisons under the calibration.
Revision: §4.1, §6.6; Appendix P, Tables 23–25, Figures 36–39.
Response to Reviewer nR1o – 3
Q2: Model-family overlap and relative ranking
Concern: Model-family overlap between AE-SOP extraction and the evaluation backbones may cause the extracted AE-SOPs to favor the procedural reasoning patterns of the same model family, affecting the relative ranking of different methods.
Response: We add a leave-one-family-out reference sensitivity analysis. For each evaluation backbone, we exclude the proposer from the same model family when constructing the references, then evaluate the same 11 methods using the newly extracted references. The reference content changes, while the rankings for FSR, KC, and K2B remain highly consistent. All four core contrasts retain their directions on both backbones. Thus, extracted AE-SOPs generally do not favor the same model family or affect the relative ranking of different methods.
Main results:
| Metric | GPT backbone / No-GPT | Qwen backbone / No-Qwen |
|---|---|---|
| FSR ranking ρ | 0.982 | 0.982 |
| KC ranking ρ | 1.000 | 1.000 |
| K2B ranking ρ | 0.991 | 0.991 |
| FSR pairwise reversals | 2/55 | 2/55 |
The Jaccard similarity between the family-excluded and original references is approximately 0.86.
Revision: Appendix N.1, Table 21, Figure 28.
Response to Reviewer nR1o – 1
Thank you for your review and helpful comments.
W1/W2/Q1: Measurement errors, diagnostic metrics, and bottleneck attribution reliability
Response: We add analyses of measurement errors, metric corrections, and conclusion stability (Appendix P). We use 60 complete trajectories and 50 linked records per backbone. Based on measurement errors, we recompute calibrated metrics and learning profiles for all 13 conditions, both backbones, and all five checkpoints. The corrections change some diagnostic states and reduce specific gaps between methods, while the key bottleneck contrasts and the association between diagnostic metrics and execution performance retain their original directions. Below, we address the biases, error propagation, and the reliability of diagnostic conclusions.
1. Error sources and discussion of biases (W1: execution-based judgments, native–procedural disagreement, and AMR corrections).
Two independent reviewers and one adjudicator establish the human reference. We calculate precision and recall for artifact extraction, and false acceptance and false rejection for exposure, artifact matching, and execution judgments (Table 23).
2. Error propagation, metric correction, and intervals (Q1: propagated uncertainty; KC/K2B and FSR/AMR).