- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a system called ADIAS for the automatic evolution of intelligent agent frameworks. The system is capable of tracking recurring failures, recovery states, and intervention histories. This information guides the selection of parent models and code revisions. In five interactive benchmarks, ADIAS outperformed existing baseline models, as demonstrated by corresponding ablation experiments and cross-model evaluations.
Critical strengths
- Persistent issue tracking provides a structured approach to guiding agent improvement. The ablation experiment results in Table 3 confirm its effectiveness.
- Excellent performance. ADIAS achieved the best performance on all five benchmarks. Additional experiments show that it consistently improved performance across four backbone models on τ-Bench and in multiple optimization runs on ALFWorld.
Critical weaknesses
- ADIAS uses far more optimization tokens than the baseline model (Table 11). Ablation experiments have addressed this issue to some extent. However, the specific advantages of structured issue tracking over simpler, persistent fault summaries remain unclear.
- The reliability of issue tracking remains unclear. The identification and resolution status of issues depend on the judgment of large language models (LLMs) and have not been directly verified. Appendix F.2 also reports modifications to the optimization mechanism itself. Since this mechanism is not explicitly described in the main paper, it makes performance attribution difficult.
Questions for authors
- Could the author provide a comparative analysis involving a comparable computational load?
- How does ADIAS perform compared to a simpler baseline based on persistent fault summaries?
- How reliable are problem identification and lifecycle transitions? Which optimization components can be modified during the evolution process?
- Please explain the reasons for the inconsistencies in the scores in tables 6 and 8, as well as the method used to calculate the confidence intervals in table 10.
3 comments
Response to Reviewer zKKa [Part 2]
Question 3: How reliable are problem identification and lifecycle transitions? Which optimization components can be modified during the evolution process?
Response 3: We have added a direct evaluation against expert-established gold answers to assess failure diagnosis, issue association, and lifecycle tracking. ADIAS achieves higher mean accuracy than the three human participants on all three tasks under the same inputs and scoring criteria, with the largest advantage in failure diagnosis.
-
Evaluation purpose. We evaluate whether ADIAS correctly identifies failures, associates diagnoses with the appropriate existing issue or creates a new issue, and assigns the appropriate lifecycle state. This directly evaluates the correctness of intermediate decisions.
-
Real cases and expert-established gold answers. We randomly sample five training trajectories from ADIAS’s optimization process on each of τ-Bench-Return, ALFWorld, and TextCraft, covering early, middle, and late optimization rounds and obtaining 15 diagnosis questions. The association and lifecycle evaluation contains 15 round-level cases, comprising 28 diagnosis-to-issue decisions and 50 issue-lifecycle decisions. Two experts familiar with the tasks discuss each question and establish a consensus gold answer before evaluation. These answers are withheld from ADIAS and the human participants.
-
Identical inputs and strict scoring. ADIAS and the human participants receive the same inputs for each question. For both association and lifecycle evaluation, the previous round’s issue state and the current round’s diagnosis results are held fixed, allowing us to assess these decisions independently of differences in diagnosis generation. Each answer receives a binary score based on exact agreement with the gold answer: (1) Diagnosis accuracy: An answer is correct if the predicted error set matches the gold set; otherwise, it is incorrect. Accuracy is the number of correct answers divided by 15. (2) Association accuracy: Each diagnosis must be assigned to the gold existing issue or NEW. Accuracy is the number of correct assignments divided by 28. (3) Lifecycle accuracy: Each issue must receive the gold state: ACTIVE, TENTATIVELY_FIXED, CONFIRMED_FIXED, or REGRESSED. Accuracy is the number of correct state assignments divided by 50.
-
Averaging protocol. ADIAS completes the same evaluation in three independent runs, and we report the mean accuracy across runs. Three human participants independently complete the same evaluation, and we report their mean accuracy. The three human participants were PhD students with experience in LLM agents and the evaluated tasks, and were distinct from the two experts who established the gold answers.
-
Results and conclusions. The results are shown in the following table. ADIAS shows the largest advantage in diagnosis accuracy, while both ADIAS and human participants achieve high accuracy in association and lifecycle decisions, with ADIAS obtaining slightly higher averages. One possible explanation for the larger diagnosis gap is that identifying the complete set of errors requires integrating evidence across long, complex trajectories, where human participants may overlook individual errors. These results provide direct evidence that ADIAS can accurately diagnose failures, link diagnoses to the appropriate issues, and update issue states.
| Method | Diagnosis accuracy | Association accuracy | Lifecycle accuracy |
|---|---|---|---|
| Human participants | 62.2% ± 3.9% | 94.1% ± 5.5% | 95.3% ± 4.2% |
| ADIAS | 82.2% ± 3.9% | 96.4% ± 3.6% | 96.0% ± 2.0% |
Thank you for highlighting the need to clarify the optimization scope. ADIAS supports evolution over both the task agent and meta-level components, including diagnosis, profile maintenance, and improvement logic. We will make this scope explicit in the main paper and document the actual modifications to clarify performance attribution. Across the three benchmark histories in main experiments, meta-level code was modified in 5 of 30 generations (16.7%), primarily to normalize profile/revision-plan fields and simplify target parsing. No diagnostic-agent code was modified. The final selected lineages on TauBench and TextCraft contained only task-agent changes. ALFWorld additionally retained a profile-normalization edit whose outputs were unchanged when replayed on the saved profiles. These findings show that the evaluated agents’ substantive inherited improvements were concentrated at the task level, while ADIAS retained the flexibility to maintain its optimization machinery. The recorded patches and parent lineages make this distinction directly auditable, providing a concrete account of what evolved and what was ultimately evaluated.
Response to Reviewer zKKa [Part 3]
Question 4: Please explain the reasons for the inconsistencies in the scores in tables 6 and 8, as well as the method used to calculate the confidence intervals in table 10.
Response 4: Thank you for requesting clarification. Tables 6 and 8 report results on different tasks: Table 6 evaluates τ-Bench-Return, whereas Table 8 evaluates ALFWorld. Their scores therefore measure performance on different benchmarks and are not expected to match. We also correct two typos. In Table 6, we have corrected the Pass 2 score for Archive-Wide Synthesis from 62.6 to 62.5 and updated the corresponding Avg. ± Std. from 64.1 ± 2.1 to 64.1 ± 2.2. In Table 8, we have corrected the Clean-and-Place test score for SkillOpt from 5.0 to 6.7.
We clarify that Table 10 uses two confidence-interval procedures. For each method’s success rate, we report a 95% Wilson score interval based on the number of successful tasks and the test-set size. For the performance difference between ADIAS and the baseline, we report a 95% confidence interval using paired nonparametric task-level bootstrap resampling with 10,000 replicates. Each replicate samples N task indices with replacement and uses the same indices for both methods, preserving the pairing of outcomes when computing their mean score difference. We will correct the description in Lines 1171–1175 to distinguish these two procedures.
Response to Reviewer zKKa [Part 1]
We thank the reviewer for insightful feedback. We make the following clarification to address the concerns.
Question 1: Could the author provide a comparative analysis involving a comparable computational load?
Response 1: We add token-budget-controlled experiments on τ-Bench-Return, ALFWorld, and TextCraft. ADIAS retains the highest mean task score on all three benchmarks.
For each benchmark, we use the strongest baseline in the main experiments as the reference and set two simultaneous optimization budgets: its meta-level token cost and its total token cost (meta + task). The meta-level cap constrains improvement effort, while the total-token cap also accounts for task execution and processing of environment feedback. The respective limits are 4.7M / 26.4M for τ-Bench-Return (4.7M meta + 21.7M task), 3.9M / 35.2M for ALFWorld (3.9M + 31.3M), and 6.5M / 15.1M for TextCraft (6.5M + 8.6M). These budgets are determined by baseline costs rather than tuned to favor ADIAS. Optimization stops as soon as either the meta-level or total-token budget is exceeded. We then select the checkpoint with the highest validation score among completed rounds that remained within both limits and evaluate it on the held-out test set. All methods follow the same budget-accounting, stopping, checkpoint-selection, and test-evaluation procedures to ensure a fair comparison.
We run each experiment for three independent runs and results are reported as mean ± standard deviation in the following table. Score uses the benchmark's task-performance metric and Eff. denotes task interaction efficiency. ADIAS also achieves both the highest mean task performance and interaction efficiency on all three benchmarks. These results support the conclusion that its advantage persists when both meta-level and total token expenditure are constrained.
Question 2: How does ADIAS perform compared to a simpler baseline based on persistent fault summaries?
Response 2: We add a persistent fault summary baseline that replaces ADIAS’s structured issue state with a continuously updated free-text summary. All other experimental conditions remain identical, such as the model, task interfaces, data splits, initial agent, evaluation protocol, and meta-level and total-token budgets. ADIAS achieves higher mean task performance and interaction efficiency on all three benchmarks, supporting the benefit of structured issue tracking beyond maintaining persistent textual memory.
After each optimization round, the baseline receives the previous summary, the current training diagnosis, the attempted modification, and evaluation feedback. It rewrites the summary to retain current faults, relevant repair attempts and their outcomes, and the next optimization priority. It can consolidate repeated observations and describe improvements or recurring failures in natural language, without maintaining explicit issue identities. The results are shown in the table below.