acceptodds
Under review as a conference paper at ICLR 2027

ADIAS: Automated Design of Interactive Agentic Systems

Abstract

Self-evolving agents can improve without updating model weights by iteratively optimizing the harness based on evaluation feedback. Existing methods are largely candidate-centric: they record evolution outcomes (which agent variants have been produced and how they perform), rather than evolution progress (which issues, i.e., recurring failure patterns, have been fixed and which remain). Such methods pose two difficulties for cumulative harness evolution. First, repair context is scattered across candidate records and must be reconstructed every round, which leads to inefficient, redundant, or misdirected revisions. Second, because candidates are evaluated as a whole, partial fixes across different agents are difficult to combine, while ineffective changes persist alongside beneficial ones. To address these problems, we formulate issue-centric agent optimization, which explicitly carries forward evolutionary progress through a persistent issue state. We instantiate this paradigm in ADIAS, a harness-evolution framework with two key components. An Issue Manager maintains the issue state with stable issue identities, lifecycle status, supporting evidence, and intervention-outcome histories. An Issue-Guided Optimizer proposes repair targets from the issue state and performs focused harness revision. On five interactive benchmarks, ADIAS outperforms the strongest baseline by 25.2% on average and achieves consistent gains across four backbone models. Controlled ablations show that removing the Issue Manager or the Issue-Guided Optimizer reduces performance by up to 40.7%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language models, LLM agent, Agent harness, Self-evolving agent

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zKKaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a system called ADIAS for the automatic evolution of intelligent agent frameworks. The system is capable of tracking recurring failures, recovery states, and intervention histories. This information guides the selection of parent models and code revisions. In five interactive benchmarks, ADIAS outperformed existing baseline models, as demonstrated by corresponding ablation experiments and cross-model evaluations.

Critical strengths
  1. Persistent issue tracking provides a structured approach to guiding agent improvement. The ablation experiment results in Table 3 confirm its effectiveness.
  2. Excellent performance. ADIAS achieved the best performance on all five benchmarks. Additional experiments show that it consistently improved performance across four backbone models on τ-Bench and in multiple optimization runs on ALFWorld.
Critical weaknesses
  1. ADIAS uses far more optimization tokens than the baseline model (Table 11). Ablation experiments have addressed this issue to some extent. However, the specific advantages of structured issue tracking over simpler, persistent fault summaries remain unclear.
  2. The reliability of issue tracking remains unclear. The identification and resolution status of issues depend on the judgment of large language models (LLMs) and have not been directly verified. Appendix F.2 also reports modifications to the optimization mechanism itself. Since this mechanism is not explicitly described in the main paper, it makes performance attribution difficult.
Questions for authors
  1. Could the author provide a comparative analysis involving a comparable computational load?
  2. How does ADIAS perform compared to a simpler baseline based on persistent fault summaries?
  3. How reliable are problem identification and lifecycle transitions? Which optimization components can be modified during the evolution process?
  4. Please explain the reasons for the inconsistencies in the scores in tables 6 and 8, as well as the method used to calculate the confidence intervals in table 10.
3 comments
AuthorsOpenReview ↗

Response to Reviewer zKKa [Part 1]

We thank the reviewer for insightful feedback. We make the following clarification to address the concerns.

Question 1: Could the author provide a comparative analysis involving a comparable computational load?

Response 1: We add token-budget-controlled experiments on τ-Bench-Return, ALFWorld, and TextCraft. ADIAS retains the highest mean task score on all three benchmarks.

For each benchmark, we use the strongest baseline in the main experiments as the reference and set two simultaneous optimization budgets: its meta-level token cost and its total token cost (meta + task). The meta-level cap constrains improvement effort, while the total-token cap also accounts for task execution and processing of environment feedback. The respective limits are 4.7M / 26.4M for τ-Bench-Return (4.7M meta + 21.7M task), 3.9M / 35.2M for ALFWorld (3.9M + 31.3M), and 6.5M / 15.1M for TextCraft (6.5M + 8.6M). These budgets are determined by baseline costs rather than tuned to favor ADIAS. Optimization stops as soon as either the meta-level or total-token budget is exceeded. We then select the checkpoint with the highest validation score among completed rounds that remained within both limits and evaluate it on the held-out test set. All methods follow the same budget-accounting, stopping, checkpoint-selection, and test-evaluation procedures to ensure a fair comparison.

We run each experiment for three independent runs and results are reported as mean ± standard deviation in the following table. Score uses the benchmark's task-performance metric and Eff. denotes task interaction efficiency. ADIAS also achieves both the highest mean task performance and interaction efficiency on all three benchmarks. These results support the conclusion that its advantage persists when both meta-level and total token expenditure are constrained.

Methodτ-Bench-Return Score ↑τ-Bench-Return Eff. ↑ALFWorld Score ↑ALFWorld Eff. ↑TextCraft Score ↑TextCraft Eff. ↑
SkillOpt11.5 ± 4.80.5 ± 0.20.0 ± 0.00.0 ± 0.057.3 ± 8.33.7 ± 1.3
Meta-Harness54.2 ± 18.13.6 ± 2.275.1 ± 13.45.4 ± 1.877.7 ± 3.55.8 ± 0.6
AHE56.3 ± 18.83.6 ± 2.085.3 ± 6.16.4 ± 1.776.7 ± 5.05.4 ± 0.6
DGM-H62.5 ± 10.54.5 ± 1.977.6 ± 7.44.8 ± 1.279.7 ± 7.44.8 ± 1.2
ADIAS (Ours)80.2 ± 1.86.2 ± 0.187.8 ± 5.47.6 ± 1.288.0 ± 2.78.1 ± 0.8

Question 2: How does ADIAS perform compared to a simpler baseline based on persistent fault summaries?

Response 2: We add a persistent fault summary baseline that replaces ADIAS’s structured issue state with a continuously updated free-text summary. All other experimental conditions remain identical, such as the model, task interfaces, data splits, initial agent, evaluation protocol, and meta-level and total-token budgets. ADIAS achieves higher mean task performance and interaction efficiency on all three benchmarks, supporting the benefit of structured issue tracking beyond maintaining persistent textual memory.

After each optimization round, the baseline receives the previous summary, the current training diagnosis, the attempted modification, and evaluation feedback. It rewrites the summary to retain current faults, relevant repair attempts and their outcomes, and the next optimization priority. It can consolidate repeated observations and describe improvements or recurring failures in natural language, without maintaining explicit issue identities. The results are shown in the table below.

Methodτ-Bench-Return Score ↑τ-Bench-Return Eff. ↑ALFWorld Score ↑ALFWorld Eff. ↑TextCraft Score ↑TextCraft Eff. ↑
ADIAS (Ours)80.2 ± 1.86.2 ± 0.187.8 ± 5.47.6 ± 1.288.0 ± 2.78.1 ± 0.8
Replace issue state with fault summaries66.7 ± 11.85.2 ± 0.825.9 ± 3.01.0 ± 0.283.7 ± 1.57.2 ± 0.3
AuthorsOpenReview ↗

Response to Reviewer zKKa [Part 2]

Question 3: How reliable are problem identification and lifecycle transitions? Which optimization components can be modified during the evolution process?

Response 3: We have added a direct evaluation against expert-established gold answers to assess failure diagnosis, issue association, and lifecycle tracking. ADIAS achieves higher mean accuracy than the three human participants on all three tasks under the same inputs and scoring criteria, with the largest advantage in failure diagnosis.

  • Evaluation purpose. We evaluate whether ADIAS correctly identifies failures, associates diagnoses with the appropriate existing issue or creates a new issue, and assigns the appropriate lifecycle state. This directly evaluates the correctness of intermediate decisions.

  • Real cases and expert-established gold answers. We randomly sample five training trajectories from ADIAS’s optimization process on each of τ-Bench-Return, ALFWorld, and TextCraft, covering early, middle, and late optimization rounds and obtaining 15 diagnosis questions. The association and lifecycle evaluation contains 15 round-level cases, comprising 28 diagnosis-to-issue decisions and 50 issue-lifecycle decisions. Two experts familiar with the tasks discuss each question and establish a consensus gold answer before evaluation. These answers are withheld from ADIAS and the human participants.

  • Identical inputs and strict scoring. ADIAS and the human participants receive the same inputs for each question. For both association and lifecycle evaluation, the previous round’s issue state and the current round’s diagnosis results are held fixed, allowing us to assess these decisions independently of differences in diagnosis generation. Each answer receives a binary score based on exact agreement with the gold answer: (1) Diagnosis accuracy: An answer is correct if the predicted error set matches the gold set; otherwise, it is incorrect. Accuracy is the number of correct answers divided by 15. (2) Association accuracy: Each diagnosis must be assigned to the gold existing issue or NEW. Accuracy is the number of correct assignments divided by 28. (3) Lifecycle accuracy: Each issue must receive the gold state: ACTIVE, TENTATIVELY_FIXED, CONFIRMED_FIXED, or REGRESSED. Accuracy is the number of correct state assignments divided by 50.

  • Averaging protocol. ADIAS completes the same evaluation in three independent runs, and we report the mean accuracy across runs. Three human participants independently complete the same evaluation, and we report their mean accuracy. The three human participants were PhD students with experience in LLM agents and the evaluated tasks, and were distinct from the two experts who established the gold answers.

  • Results and conclusions. The results are shown in the following table. ADIAS shows the largest advantage in diagnosis accuracy, while both ADIAS and human participants achieve high accuracy in association and lifecycle decisions, with ADIAS obtaining slightly higher averages. One possible explanation for the larger diagnosis gap is that identifying the complete set of errors requires integrating evidence across long, complex trajectories, where human participants may overlook individual errors. These results provide direct evidence that ADIAS can accurately diagnose failures, link diagnoses to the appropriate issues, and update issue states.

MethodDiagnosis accuracyAssociation accuracyLifecycle accuracy
Human participants62.2% ± 3.9%94.1% ± 5.5%95.3% ± 4.2%
ADIAS82.2% ± 3.9%96.4% ± 3.6%96.0% ± 2.0%

Thank you for highlighting the need to clarify the optimization scope. ADIAS supports evolution over both the task agent and meta-level components, including diagnosis, profile maintenance, and improvement logic. We will make this scope explicit in the main paper and document the actual modifications to clarify performance attribution. Across the three benchmark histories in main experiments, meta-level code was modified in 5 of 30 generations (16.7%), primarily to normalize profile/revision-plan fields and simplify target parsing. No diagnostic-agent code was modified. The final selected lineages on TauBench and TextCraft contained only task-agent changes. ALFWorld additionally retained a profile-normalization edit whose outputs were unchanged when replayed on the saved profiles. These findings show that the evaluated agents’ substantive inherited improvements were concentrated at the task level, while ADIAS retained the flexibility to maintain its optimization machinery. The recorded patches and parent lineages make this distinction directly auditable, providing a concrete account of what evolved and what was ultimately evaluated.

AuthorsOpenReview ↗

Response to Reviewer zKKa [Part 3]

Question 4: Please explain the reasons for the inconsistencies in the scores in tables 6 and 8, as well as the method used to calculate the confidence intervals in table 10.

Response 4: Thank you for requesting clarification. Tables 6 and 8 report results on different tasks: Table 6 evaluates τ-Bench-Return, whereas Table 8 evaluates ALFWorld. Their scores therefore measure performance on different benchmarks and are not expected to match. We also correct two typos. In Table 6, we have corrected the Pass 2 score for Archive-Wide Synthesis from 62.6 to 62.5 and updated the corresponding Avg. ± Std. from 64.1 ± 2.1 to 64.1 ± 2.2. In Table 8, we have corrected the Clean-and-Place test score for SkillOpt from 5.0 to 6.7.

We clarify that Table 10 uses two confidence-interval procedures. For each method’s success rate, we report a 95% Wilson score interval based on the number of successful tasks and the test-set size. For the performance difference between ADIAS and the baseline, we report a 95% confidence interval using paired nonparametric task-level bootstrap resampling with 10,000 replicates. Each replicate samples N task indices with replacement and uses the same indices for both methods, preserving the pairing of outcomes when computing their mean score difference. We will correct the description in Lines 1171–1175 to distinguish these two procedures.

Reviewer_VGnTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ADIAS, a framework for automatically evolving an AI agent’s runtime harness while keeping the underlying model fixed. It augments candidate histories with persistent issue records that track issue identities, repair status, supporting evidence, and previous interventions. An Issue Manager maintains these records, while an Issue-Guided Optimizer uses them to select repair targets and parent candidates for targeted code revisions. Experiments report improvements over the evaluated baselines across five interactive benchmarks, with additional evidence from ablations, four backbone models, and ALFWorld task-family generalization tests.

Critical strengths
  • Overall, the paper is well written and easy to follow. The overview figures clearly explain the main components and optimization workflow.
  • The paper addresses a relevant problem: preserving partial repairs and avoiding repeated unsuccessful revisions during iterative agent optimization.
  • Table 1 reports higher task scores than the evaluated baselines on all five interactive benchmarks, supporting the effectiveness of the overall framework.
Critical weaknesses
  • Although AHE is included, the paper does not explain why AgentSquare’s explicit module-recombination approach is omitted from experiments on overlapping benchmarks.
  • ADIAS uses more optimization tokens than the external baselines in Table 11. Internal cost controls are informative, but comparisons under matched optimization-token budgets remain missing.
  • The ablations support the combined design, but do not isolate the contributions of persistent issue identities, lifecycle tracking, and evidence summarization.
  • Variability across independent complete optimization runs is reported only for ALFWorld, leaving search stability on the other benchmarks less clear.
  • Cost reporting focuses on optimization tokens in two environments. End-to-end token consumption and latency are not quantified, limiting assessment of practical efficiency.
Questions for authors
  • Why was AgentSquare’s explicit module-recombination approach not evaluated on the overlapping environments, alongside the existing AHE baseline?
  • Does ADIAS retain its advantage over external baselines when optimization-token budgets are matched?
  • With the same diagnostic evidence and intervention history, what additional benefit do persistent issue identities and lifecycle states provide over a compact structured repair summary?
  • Beyond ALFWorld, are the reported gains stable across independent complete optimization runs?
  • In the ALFWorld generalization experiment, how do the gains vary across task families, and which unseen families remain difficult?
2 comments
AuthorsOpenReview ↗

Response to Reviewer VGnT [Part 1]

We thank the reviewer for constructive comments. We conduct additional experiments to address weaknesses and questions raised.

To address Questions 1–4, we conduct additional experiments on τ-Bench-Return, ALFWorld, and TextCraft under a unified evaluation protocol. Table R1 includes AgentSquare to address Question 1, compares methods under matched token-budget limits to address Question 2, reports four component ablations to address Question 3, and summarizes three independent complete optimization runs as mean ± standard deviation to address Question 4.

All methods use the same domain interfaces, data splits, initial task agent, and evaluation protocol. The token-budget and checkpoint-selection procedures are detailed in Response 2. In Table R1, Score uses the benchmark’s task-performance metric, and Eff. denotes task interaction efficiency.

Table R1. Additional baseline comparisons and component ablations under matched token-budget limits. Results are mean ± standard deviation across three independent complete optimization runs.

Methodτ-Bench-Return Score ↑τ-Bench-Return Eff. ↑ALFWorld Score ↑ALFWorld Eff. ↑TextCraft Score ↑TextCraft Eff. ↑
Baseline comparisons
SkillOpt11.5 ± 4.80.5 ± 0.20.0 ± 0.00.0 ± 0.057.3 ± 8.33.7 ± 1.3
AgentSquare70.8 ± 6.55.6 ± 0.484.3 ± 2.76.5 ± 0.365.0 ± 2.04.0 ± 0.3
Meta-Harness54.2 ± 18.13.6 ± 2.275.1 ± 13.45.4 ± 1.877.7 ± 3.55.8 ± 0.6
AHE56.3 ± 18.83.6 ± 2.085.3 ± 6.16.4 ± 1.776.7 ± 5.05.4 ± 0.6
DGM-H62.5 ± 10.54.5 ± 1.977.6 ± 7.44.8 ± 1.279.7 ± 7.44.8 ± 1.2
ADIAS (Ours)80.2 ± 1.86.2 ± 0.187.8 ± 5.47.6 ± 1.288.0 ± 2.78.1 ± 0.8
Component ablations
w/o issue identities79.2 ± 3.66.0 ± 0.379.6 ± 1.15.3 ± 0.271.3 ± 1.54.9 ± 0.3
w/o lifecycle tracking75.0 ± 5.46.1 ± 0.469.9 ± 1.93.9 ± 0.281.3 ± 3.16.8 ± 0.5
w/o evidence summarization79.2 ± 1.86.0 ± 0.456.5 ± 1.12.7 ± 0.185.7 ± 3.28.0 ± 0.6
Replace issue state with compact structured repair summary66.7 ± 6.55.5 ± 0.857.7 ± 3.54.0 ± 0.873.7 ± 3.56.2 ± 0.6

Weakness & Question 1: AgentSquare method is not evaluated.

Response 1: We have added AgentSquare, using the same domain interfaces, data splits, initial task agent, and evaluation protocol while retaining its module-recombination procedure. All methods are evaluated under the same meta-level and total-token budget limits. As shown in Table R1, ADIAS achieves higher mean task performance and interaction efficiency than AgentSquare on every evaluated benchmark.

Weakness & Question 2: Comparisons under matched optimization-token budgets remain missing.

Response 2: We have added token-budget-controlled experiments on τ-Bench-Return, ALFWorld, and TextCraft. For each benchmark, we use the strongest baseline in the original main experiments as the reference and set two simultaneous optimization budgets: its meta-level token cost and its total token cost (meta + task). The meta-level cap constrains improvement effort, while the total-token cap also accounts for task execution and processing of environment feedback.

The respective limits are 4.7M / 26.4M for τ-Bench-Return (4.7M meta + 21.7M task), 3.9M / 35.2M for ALFWorld (3.9M + 31.3M), and 6.5M / 15.1M for TextCraft (6.5M + 8.6M). These budgets are determined by baseline costs rather than tuned to favor ADIAS.

Optimization stops as soon as either the meta-level or total-token budget is exceeded. We then select the checkpoint with the highest validation score among completed rounds that remained within both limits and evaluate it on the held-out test set. All methods follow the same budget-accounting, stopping, checkpoint-selection, and test-evaluation procedures to ensure a fair comparison.

As shown in Table R1, ADIAS retains the highest mean task score on all three benchmarks. ADIAS also achieves the highest mean interaction efficiency. These results support the conclusion that its advantage persists when both meta-level and total token expenditure are constrained.

AuthorsOpenReview ↗

Response to Reviewer VGnT [Part 2]

Weakness & Question 3: The experiments do not isolate the contributions of persistent issue identities, lifecycle tracking, and evidence summarization.

Response 3: We have added four controlled ablation variants to assess the contributions of individual components and compare ADIAS with a compact structured repair summary:

  • w/o issue identities: appends diagnoses without associating them with existing issues.
  • w/o lifecycle tracking: removes explicit issue-status tracking and transitions.
  • w/o evidence summarization: removes the requirement to summarize and record supporting evidence.
  • replace issue state with compact structured repair summary: replaces persistent issue records with a continuously updated structured summary of failure patterns, supporting evidence, attempted repairs and their outcomes, and next repair priorities. The summary does not maintain stable issue identifiers or explicit lifecycle states.

The first three variants each remove one component. The fourth jointly replaces explicit issue identities and lifecycle tracking with summary-based memory while retaining access to diagnostic evidence and intervention history. All variants use the same backbone model, diagnostic procedure, data splits, evaluation protocol, and meta-level and total-token budget caps.

The component-ablation rows in Table R1 show that ADIAS achieves higher mean performance than the first three variants on every benchmark. The effects vary across tasks: removing issue identities has the largest impact on TextCraft, while removing evidence summarization has the largest impact on ALFWorld. These results support the incremental value of each component within ADIAS.

ADIAS also achieves higher mean performance than the compact structured repair summary on every benchmark, supporting the additional value of persistent issue identities and explicit lifecycle tracking beyond structured retention of diagnostic evidence and repair history.

Weakness & Question 4: Beyond ALFWorld, are the reported gains stable across independent complete optimization runs?

Response 4: We have added repeated independent complete optimization runs on τ-Bench-Return and TextCraft, complementing the ALFWorld evaluation. Every setting in Table R1 is evaluated over three independent complete optimization runs, starting from the same initial task agent, and we report mean performance and standard deviation. ADIAS maintains the highest mean task performance across all three benchmarks, providing evidence that the gains extend beyond a single optimization run.

Weakness 5: End-to-end token consumption and latency are not quantified, limiting assessment of practical efficiency.

Response 5: In addition to the budget-controlled performance comparisons in Table R1, we report actual optimization-token consumption and wall-clock time.

ADIAS uses 22.2M tokens on τ-Bench-Return (3.9M meta + 18.3M task), below the baseline-derived budget of 26.4M; 27.6M on ALFWorld (3.9M + 23.7M), below 35.2M; and 14.8M on TextCraft (5.6M + 9.2M), below 15.1M. Thus, token accounting covers both improvement generation and task execution, and ADIAS remains within both the meta-level and total-token limits on every benchmark.

We use end-to-end wall-clock time as a practical approximation of optimization latency. ADIAS takes 5.4 hours on τ-Bench-Return, 3.1 hours on ALFWorld, and 2.4 hours on TextCraft, compared with 6.5, 3.3, and 2.6 hours for the best-performing baseline on each benchmark, respectively. These results indicate competitive observed optimization latency under our experimental setup. All wall-clock comparisons were conducted with the same concurrency level, model-serving configuration, and hardware setup.

Question 5: In the ALFWorld generalization experiment, how do the gains vary across task families, and which unseen families remain difficult?

Response 6: The following breakdown corresponds to the original ALFWorld generalization experiment in Submission Table 1. It is separate from the additional token-budget-controlled experiments in Table R1. ADIAS outperforms AHE, the strongest baseline in our original main experiment, and the improvement extends across task families, with particularly strong gains on tasks requiring multiple coordinated operations. ADIAS matches AHE on Look and improves on all other task families. Among the unseen families, Heat remains the most challenging for ADIAS, although it still improves over AHE by 17.4 percentage points.

Table R2. Success rate by ALFWorld task family (%).

MetricLook at Object in LightPick & PlacePick, Clean & PlacePick, Cool & PlacePick, Heat & PlacePick Two & PlaceOverall
AHE100.091.774.271.465.270.678.4
ADIAS100.0100.090.395.282.6100.094.0
Reviewer_kjHzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes ADIAS, an automated agent-optimization framework that shifts the focus from tracking candidate agents to tracking failures and repair progress through a persistent issue state. The method maintains structured records of identified issues, uses them to guide future revisions, and selects parent agents based on the issue being addressed. Experiments on five interactive-agent benchmarks show consistent improvements over several recent agent-optimization baselines, supported by ablations and cross-model evaluations. The main empirical finding is that maintaining and leveraging issue-level optimization history can lead to better-performing agent systems.

Critical strengths

ADIAS proposes a clear and well-motivated framework that explicitly tracks failures and repair progress across optimization rounds, providing a structured alternative to existing candidate-centric optimization approaches. The idea is intuitive, and the issue lifecycle offers a practical mechanism for accumulating knowledge about past failures and interventions over time. Empirically, the method achieves the best performance across all five evaluated benchmarks and maintains improvements across multiple backbone models, suggesting that the overall approach is effective across different settings. The paper is also supported by an extensive experimental study, including comparisons with several recent agent-optimization methods, ablations, and qualitative analyses, which together provide a strong empirical evaluation of the proposed framework.

Critical weaknesses
  1. The paper does not directly test whether failures are diagnosed correctly or whether related failures are consistently linked to the same issue. This evaluation is important because persistent issue tracking is the main contribution of ADIAS.

  2. ADIAS uses substantially more meta-level tokens than the main baselines. For example, it uses 8.1M tokens on τ-Bench, compared with 4.7M for DGM-H. A compute-matched comparison would clarify whether the gains come from the proposed method or greater model usage.

  3. Issue-guided parent selection is compared only with two simple alternatives: selecting the best candidate and selecting the latest candidate. Testing broader parent-selection strategies would provide stronger evidence that this component drives the improvement.

  4. The main experiments use selected parts of several benchmarks. For example, τ-Bench focuses on return-related tasks, while ALFWorld optimization uses Clean-and-Place tasks. Evaluation on complete task distributions across all benchmarks would better support the generalization claims.

  5. Diagnosis, issue association, and prioritization depend on LLM judgments. The paper does not test how stable these decisions are across repeated runs or how incorrect decisions affect final performance. A robustness analysis would strengthen the empirical evidence.

  6. The related-work discussion mainly covers candidate-centric agent optimization. It gives limited attention to broader work on persistent memory, failure-driven refinement, experience reuse, and policy repair. A more systematic comparison with these research directions would better clarify what is new in issue-centric optimization.

Questions for authors

Check my comments in the Weaknesses

9 comments
AuthorsOpenReview ↗

Response to Reviewer kjHz [Part 1]

We thank the reviewer for the constructive comments, and we clarify each concern below.

Weakness 1: The paper does not directly test whether failures are diagnosed correctly or whether related failures are consistently linked to the same issue.

Response 1: Thank you for this suggestion. We have added a direct evaluation against expert-established gold answers to assess failure diagnosis, issue association, and lifecycle tracking. ADIAS achieves higher mean accuracy than the three human participants on all three tasks under the same inputs and scoring criteria, with the largest advantage in failure diagnosis.

  • Evaluation purpose. We evaluate whether ADIAS correctly identifies failures, associates diagnoses with the appropriate existing issue or creates a new issue, and assigns the appropriate lifecycle state. This directly evaluates the correctness of intermediate decisions.

  • Real cases and expert-established gold answers. We randomly sample five training trajectories from ADIAS’s optimization process on each of τ-Bench-Return, ALFWorld, and TextCraft, covering early, middle, and late optimization rounds and obtaining 15 diagnosis questions. The association and lifecycle evaluation contains 15 round-level cases, comprising 28 diagnosis-to-issue decisions and 50 issue-lifecycle decisions. Two experts familiar with the tasks discuss each question and establish a consensus gold answer before evaluation. These answers are withheld from ADIAS and the human participants.

  • Identical inputs and strict scoring. ADIAS and the human participants receive the same inputs for each question. For both association and lifecycle evaluation, the previous round’s issue state and the current round’s diagnosis results are held fixed, allowing us to assess these decisions independently of differences in diagnosis generation. Each answer receives a binary score based on exact agreement with the gold answer: (1) Diagnosis accuracy: An answer is correct if the predicted error set matches the gold set; otherwise, it is incorrect. Accuracy is the number of correct answers divided by 15. (2) Association accuracy: Each diagnosis must be assigned to the gold existing issue or NEW. Accuracy is the number of correct assignments divided by 28. (3) Lifecycle accuracy: Each issue must receive the gold state: ACTIVE, TENTATIVELY_FIXED, CONFIRMED_FIXED, or REGRESSED. Accuracy is the number of correct state assignments divided by 50.

  • Averaging protocol. ADIAS completes the same evaluation in three independent runs, and we report the mean accuracy across runs. Three human participants independently complete the same evaluation, and we report their mean accuracy. The three human participants were PhD students with experience in LLM agents and the evaluated tasks, and were distinct from the two experts who established the gold answers.

  • Results and conclusions. The results are shown in the following table. ADIAS shows the largest advantage in diagnosis accuracy, while both ADIAS and human participants achieve high accuracy in association and lifecycle decisions, with ADIAS obtaining slightly higher averages. One possible explanation for the larger diagnosis gap is that identifying the complete set of errors requires integrating evidence across long, complex trajectories, where human participants may overlook individual errors. These results provide direct evidence that ADIAS can accurately diagnose failures, link diagnoses to the appropriate issues, and update issue states.

MethodDiagnosis accuracyAssociation accuracyLifecycle accuracy
Human participants62.2% ± 3.9%94.1% ± 5.5%95.3% ± 4.2%
ADIAS82.2% ± 3.9%96.4% ± 3.6%96.0% ± 2.0%
AuthorsOpenReview ↗

Response to Reviewer kjHz [Part 2]

Weakness 2: A compute-matched comparison would clarify whether the gains come from the proposed method or greater model usage.

Response 2: We add token-budget-controlled experiments on τ-Bench-Return, ALFWorld, and TextCraft. ADIAS retains the highest mean task score on all three benchmarks.

For each benchmark, we use the strongest baseline in the main experiments as the reference and set two simultaneous optimization budgets: its meta-level token cost and its total token cost (meta + task). The meta-level cap constrains improvement effort, while the total-token cap also accounts for task execution and processing of environment feedback. The respective limits are 4.7M / 26.4M for τ-Bench-Return (4.7M meta + 21.7M task), 3.9M / 35.2M for ALFWorld (3.9M + 31.3M), and 6.5M / 15.1M for TextCraft (6.5M + 8.6M). These budgets are determined by baseline costs rather than tuned to favor ADIAS. Optimization stops as soon as either the meta-level or total-token budget is exceeded. We then select the checkpoint with the highest validation score among completed rounds that remained within both limits and evaluate it on the held-out test set. All methods follow the same budget-accounting, stopping, checkpoint-selection, and test-evaluation procedures to ensure a fair comparison.

We run each experiment for three independent runs and results are reported as mean ± standard deviation in the following table. Score uses the benchmark's task-performance metric and Eff. denotes task interaction efficiency. ADIAS also achieves both the highest mean task performance and interaction efficiency on all three benchmarks. These results support the conclusion that its advantage persists when both meta-level and total token expenditure are constrained.

Methodτ-Bench-Return Score ↑τ-Bench-Return Eff. ↑ALFWorld Score ↑ALFWorld Eff. ↑TextCraft Score ↑TextCraft Eff. ↑
SkillOpt11.5 ± 4.80.5 ± 0.20.0 ± 0.00.0 ± 0.057.3 ± 8.33.7 ± 1.3
Meta-Harness54.2 ± 18.13.6 ± 2.275.1 ± 13.45.4 ± 1.877.7 ± 3.55.8 ± 0.6
AHE56.3 ± 18.83.6 ± 2.085.3 ± 6.16.4 ± 1.776.7 ± 5.05.4 ± 0.6
DGM-H62.5 ± 10.54.5 ± 1.977.6 ± 7.44.8 ± 1.279.7 ± 7.44.8 ± 1.2
ADIAS (Ours)80.2 ± 1.86.2 ± 0.187.8 ± 5.47.6 ± 1.288.0 ± 2.78.1 ± 0.8

Weakness 3: Testing broader parent-selection strategies would provide stronger evidence that this component drives the improvement.

Response 3.: We have extended the ablation with three parent-selection strategies adapted from prior work. Each replaces only the selection rule within ADIAS under the same meta and total token budget with other settings remain unchanged. The results show that ADIAS's issue-guided selection outperforms other parent selection methods on all three benchmarks.

The experiments of parent-selection methods include:

  • DGM-H-style performance–offspring sampling [1]: favor higher-performing candidates while reducing the weight of candidates with many offspring. This tests performance-guided exploration of less-expanded branches.
  • AFlow-style soft mixed selection [2]: mix uniform sampling with score-based softmax sampling over high-performing candidates. This tests stochastic exploration combined with performance-based selection.
  • HGM-style clade-metaproductivity selection [3]: aggregate outcomes across a candidate and its descendants to estimate future improvement potential, and select parents through Thompson sampling with decreasing exploration as the budget is consumed. This tests lineage-level guidance without issue-conditioned parent selection.

The results are reported in the following table. Issue-guided selection achieves the highest mean score and interaction efficiency on all three benchmarks.

Methodτ-Bench-Return Score ↑τ-Bench-Return Eff. ↑ALFWorld Score ↑ALFWorld Eff. ↑TextCraft Score ↑TextCraft Eff. ↑
Issue-guided selection (ADIAS)80.2 ± 1.86.2 ± 0.187.8 ± 5.47.6 ± 1.288.0 ± 2.78.1 ± 0.8
Performance–offspring sampling79.2 ± 3.65.7 ± 0.529.1 ± 3.31.2 ± 0.279.3 ± 1.25.8 ± 0.1
Soft mixed selection68.8 ± 3.15.2 ± 0.274.1 ± 1.64.2 ± 0.178.3 ± 4.25.7 ± 0.6
HGM-style clade-metaproductivity selection70.8 ± 6.55.7 ± 0.473.6 ± 3.04.4 ± 0.361.3 ± 1.23.8 ± 0.2

[1] Jenny Zhang et al. Hyperagents. arXiv:2603.19461, 2026.

[2] Jiayi Zhang et al. AFlow: Automating Agentic Workflow Generation. arXiv:2410.10762, 2024.

[3] Wenyi Wang et al. Huxley-Gödel Machine: Human-Level Coding Agent Development by an Approximation of the Optimal Self-Improving Machine. arXiv:2510.21614, 2025.

AuthorsOpenReview ↗

Response to Reviewer kjHz [Part 3]

Weakness 4: The main experiments use selected parts of several benchmarks. For example, τ-Bench focuses on return-related tasks, while ALFWorld optimization uses Clean-and-Place tasks.

Response 4: The submission already includes complete task-distribution evaluations. Results on all τ-Bench retail tasks are in Table 9. Evaluation on complete task distributions of ALFWorld is in Table 1, while the results on Clean-and-Place tasks are Table 8.

Weakness 5: The paper does not test decision stability across repeated runs or how incorrect decisions affect final performance.

Response 5: Thank you for raising this concern. Because LLM-based diagnosis is stochastic, identical intermediate decisions are not required for reliable optimization. The key question is whether this variability undermines optimization outcomes. We therefore evaluate whether the performance advantage is reproducible across complete optimization runs and whether observed diagnostic errors cause persistent performance degradation under a fixed optimization budget.

  • Reproducibility across independent optimization runs. We conduct three independent end-to-end optimization runs on each benchmark, starting from the same initial agent and following the same evaluation protocol. These runs include stochastic diagnosis, issue association, prioritization, parent selection, and agent modification. ADIAS consistently achieves higher mean performance than the competing baselines, supporting the reproducibility of its overall advantage despite variability in intermediate decisions.

  • Recovery from intermediate errors. We consider three error types: incorrect diagnoses, incorrect issue associations, and incorrect lifecycle updates. We randomly sample three instances of each type from the optimization trajectories, yielding nine cases in total. For each case, we resume optimization from the corresponding checkpoint in three independent continuation runs, retaining the erroneous decision and the history available at that point. In all error-retaining continuations, subsequent optimization rounds resolve the associated task failure despite the earlier error. To assess the impact of these errors beyond eventual recovery, we compare two continuation conditions from each checkpoint: one retains the erroneous decision, while the other corrects only that decision and leaves the remaining history unchanged. Both conditions receive the same remaining optimization budget and follow the same evaluation protocol, with three independent continuation runs per condition. Across the nine checkpoints, the mean final task score under the error-retaining condition is no more than 1.5 percentage points lower than under the corrected condition and is higher in some cases. Together with the observed recovery, these results suggest that ADIAS can recover from the sampled errors in diagnosis, issue association, and lifecycle updates within the allotted budget, with limited impact on final task performance.

Together, these experiments support reproducible optimization outcomes and recovery from the sampled diagnostic errors.

Weakness 6: The related-work discussion mainly covers candidate-centric agent optimization.

Response 6: Section 2 (lines 114-133) already considers two complementary perspectives. The first paragraph reviews the broader literature on self-evolving agents and agent harnesses. The second paragraph, “Experience and Reusable Knowledge,” discusses reflection, experiential learning, editable playbooks, and reusable artifacts, including Reflexion, ExpeL, ACE, and SkillOpt. We will strengthen this discussion with A-MEM [4], Self-Refine [5], Agent Workflow Memory [6], and POLARIS [7], covering persistent memory organization, feedback-driven refinement, workflow reuse, and policy repair. The comparison will clarify what each approach preserves across iterations, how that information guides subsequent changes, and how repair outcomes are tracked. We will explicitly acknowledge that retaining experience, learning from failures, and making persistent policy improvements are established ideas. Our specific emphasis is on maintaining the evolving status, evidence, and intervention history of recurring issues across candidate revisions, and using this issue state to guide subsequent optimization. This expanded discussion will clarify both the shared foundations and the positioning of issue-centric optimization.

[4] Wujiang Xu, et at. A-Mem: Agentic Memory for LLM Agents. NeurIPS, 2025.

[5] Aman Madaan, et al. Self-Refine: Iterative Refinement with Self-Feedback. NeurIPS, 2023.

[6] Zora Zhiruo Wang, et al. Agent Workflow Memory. ICML, 2025.

[7] Aditya Namdev Kakade, et al. POLARIS: A Gödel Agent Framework for Small Language Models through Experience-Abstracted Policy Repair. Findings of ACL, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.