acceptodds
Under review as a conference paper at ICLR 2027

Ecdysis: Rethinking Failure-Driven Evolution of LLM Agent Harnesses

Abstract

Self-evolving runtime harnesses can substantially improve the capabilities of large language model (LLM) agents and provide a promising paradigm for optimizing agent execution. Existing failure-driven approaches often treat observed agent failures as direct evidence for harness modification. A key challenge in failure-driven harness evolution is that observed failures can reflect either limitations of the underlying model or systematic deficiencies of the harness. Directly optimizing against individual failures can therefore induce model-specific accommodation and impair generalization across tasks and models. We study whether failure evidence accumulated across task instances can provide a more reliable signal for harness training. Our key insight is that failures recurring across distinct tasks provide stronger inductive evidence for systematic harness deficiencies than isolated failures. Based on this insight, we propose Ecdysis, which aggregates failure evidence across task instances before promoting recurring failure patterns into persistent harness evolution, biasing evolution toward repairs that are more likely to generalize beyond individual model behaviors. Ecdysis further employs collaborative failure analysis to refine modification specifications, trading additional evolution-time reasoning for improved modification quality. Across multiple LLMs and benchmarks, Ecdysis improves the reasoning accuracy of evolved harnesses by 18.56% over existing harness evolution while achieving up to 1.84 faster harness training. Ecdysis also enables more data-efficient training. Fine-grained analysis shows that Ecdysis reduces model-specific accommodation during evolution, while the resulting harnesses exhibit stronger cross-LLM generalization and lower inference-time token consumption. These results suggest that effective harness training depends not only on how failures are repaired, but also on which failures provide reliable evidence for persistent harness changes.

Primary area:
generative models
Keywords:
agent, LLM, harness, evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PtuQOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies failure-driven self-evolution of runtime harnesses for LLM agents and proposes ECDYSIS. ECDYSIS distinguishes task-model failures from systematic harness defects by aggregating recurring failures across tasks. It then applies Failure-Driven Collaborative Refinement (FDCR), where multiple roles analyze the evidence and produce a structured modification specification. A candidate harness is accepted only if it improves the training-set score. The method is evaluated on AgentBench and the Airline and Retail subsets of τ2\tau^2-Bench. Harnesses evolved with Qwen3-8B are tested on five models. ECDYSIS with FDCR achieves over 10% average accuracy improvement versus Self-Evolution, while also reducing token usage, trajectory time, and the manually audited model-specific accommodation (MSA) ratio.

Critical strengths
  1. The paper clearly distinguishes task-model errors from persistent harness defects and uses cross-task recurrence plus refinement to avoid overfitting the harness to individual model failures, which should be a clear and useful reframing of failure-driven harness evolution.

  2. The experiments cover five task models and three benchmarks, showing consistent accuracy gains together with lower token usage and runtime. Cross-model evaluation is particularly relevant to the paper’s goal of reducing model-specific accommodation.

Critical weaknesses
  1. Comparisons are not compute-matched. Self-Evolution and ECDYSIS use different numbers and types of LLM calls and follow different search trajectories. It is difficult to isolate whether the gains come from the proposed method or unequal search opportunities.

  2. The MSA/HLR analysis is insufficiently validated. The classification is based on manually merged decisions labeled by five author annotators, but the paper does not report decision counts, inter-rater agreement, blinding, or sensitivity to the merging procedure. Since this analysis supports a central mechanistic claim, stronger reliability evidence is needed.

Questions for authors
  1. Can the authors provide compute-matched comparisons across Self-Evolution, ECDYSIS without FDCR, and ECDYSIS with FDCR, using equal LLM-call, token, or wall-clock budgets?

  2. How many modification decisions were labeled for each method and dataset?

Reviewer_LLiVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies failure-driven self-evolution of runtime harnesses for LLM agents. It argues that existing methods treat every observed failure as evidence for harness modification, without distinguishing failures caused by model-level limitations (Model-Specific Accommodation, MSA) from failures caused by genuine runtime deficiencies (Harness-Level Repair, HLR). The proposed method, ECDYSIS, aggregates failures across task instances and promotes only recurring patterns, followed by a multi-role collaborative refinement (FDCR) step before committing harness changes. The paper reports accuracy and efficiency gains over a single self-evolution baseline on 5 LLMs and 3 benchmarks.

Critical strengths
  1. The motivation is clear and the pipeline is simple and easy to reimplement. The MSA vs. HLR distinction is useful and well-motivated, and the overall loop (collect trajectories → aggregate failures → propose modifications → validate on training set) is straightforward to follow.

  2. FDCR is a practically interesting design: role-separated multi-round discussion (Analyst → Critic → Engineer → Moderator) forces the pipeline to examine root-cause evidence, side effects, and unintended blocking of legitimate behavior before any harness code is written.

Critical weaknesses
  1. The load-bearing Batch-Level Failure Aggregation step is underspecified. Algorithm 1 lists Group(B_i) as a single call, but the paper never describes whether grouping is LLM-based, embedding-based, or rule-based, nor what counts as a "distinct task" at deployment time. If the method relies on pre-labeled task categories to cluster failures, its practicality is substantially weaker than claimed.

  2. The empirical comparison is narrow and the FDCR ablation is thin. Only Life-Harness's SE is benchmarked, while concurrent work in the same space — including batch-clustered failure adaptation (Mistake Notebook Learning), structural root-cause extraction from noisy traces (From Noisy Traces to Root Causes), and reusable-skill mining from batched experience (Skill-Pro) — is neither compared against nor discussed in relation to ECDYSIS. On the ablation side, there is no sweep over the number of refinement rounds K, no sweep over the failure threshold λ, no breakdown of the marginal contribution of each of the four FDCR roles, and no comparison between the full multi-role debate and a single binary MSA/HLR judge, even though FDCR roughly doubles evolution-time API cost (e.g., $2.49 → $5.76 on τ²-Retail).

Questions for authors
  1. Figure 2 shows that without FDCR, the batch-clustered modifications are dominated by MSA — and the MSA ratio actually increases relative to Self-Evolution. This seems to contradict the intro's motivation, which frames the core problem as the lack of MSA/HLR attribution. Can the authors clarify this tension?

  2. Batch-Level Failure Aggration may need to cite below papers

From Noisy Traces to Root Causes: Structural Trajectory Analysis and Causal Extraction for Agent Optimization https://arxiv.org/pdf/2607.07702

Skill-Pro: Learning Reusable Skills from Experience via Non-Parametric PPO for LLM Agents https://arxiv.org/pdf/2602.01869

Mistake Notebook Learning: Batch-Clustered Failures for Training-Free Agent Adaptation https://aclanthology.org/2026.findings-acl.719/

Reviewer_tNVLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how execution failures should guide the evolution of LLM agent harnesses. It distinguishes model-specific accommodation from repairs to systematic harness deficiencies and proposes Ecdysis, which aggregates failure evidence across tasks before generating harness modifications. A collaborative refinement stage involving Analyst, Critic, Engineer, and Moderator roles scrutinizes proposed changes before implementation. Candidates are retained when they improve training-set performance. Experiments evolve harnesses using Qwen3-8B and evaluate the frozen harnesses across five task models on τ²-Airline, τ²-Retail, and AgentBench. The full method improves aggregate accuracy from 58.67% to 69.56% over the serial self-evolution baseline while reducing training cost and average inference token consumption.

Critical strengths
  1. The paper identifies a consequential failure mode of harness optimization. Its distinction between model-specific accommodation and harness-level repair is useful because improving a triggering task can introduce inappropriate global restrictions. The concrete code examples in Appendices A and B make this concern persuasive. In particular, the refund example demonstrates how a locally beneficial modification can remove a legitimate action permitted by the environment.

  2. The reported improvements extend beyond the model used during evolution. The frozen harnesses are evaluated on additional task models without further modification, providing useful evidence of transfer. Table 1 reports an improvement of 10.89 percentage points over self-evolution, together with lower average inference token consumption. The aggregation-only and full-method variants also expose a meaningful trade-off between evolution efficiency and modification quality.

Critical weaknesses
  1. The experiments do not adequately isolate the contribution of recurrence-based evidence selection. Aggregation changes both the organization of evidence and the number of coding-agent calls, while FDCR adds structured deliberation. The reported MSA ratio actually increases from 60.00% under self-evolution to 75.61% with aggregation alone, before falling to 45.45% with FDCR. The authors appropriately acknowledge this result. However, without controls such as serial evolution with FDCR and ordinary batching without recurrence prioritization, it remains unclear how much of the improvement specifically requires the proposed recurrence mechanism. This limits the strength of the paper’s explanatory contribution.

  2. The experimental reporting leaves substantial uncertainty about reproducibility and stability. Appendix C specifies Qwen3-8B evolution with at most three candidate-generation rounds, but the PDF does not clearly report repeated independent evolution runs. Repeated evaluation of a frozen harness does not measure variability in the optimization process itself. Exact training and held-out set sizes, task identifiers, and the evaluated AgentBench environments are also insufficiently clear. These omissions make it difficult to assess the robustness and coverage of the aggregate improvements.

Questions for authors
  1. Can you separate recurrence prioritization from ordinary batched failure processing while keeping the number of coding-agent calls and FDCR procedure fixed? Does adding FDCR to the serial baseline substantially close the performance and MSA gaps?

  2. Please specify the exact training and held-out task counts, split construction, and AgentBench environments. How many independent harness-evolution runs were performed, and what variability is observed across those runs?

  3. The MSA audit includes modifications that were subsequently rolled back. Does the reduction remain when considering only accepted modifications that contribute to the deployed harness? Please also provide the annotation counts and inter-annotator agreement.

Comments on the paper

1 comment
Area_Chair_P15aOpenReview ↗

urgent!

I noticed that none of the reviewers has provided follow-up feedback so far. As the discussion period is approaching its end, I would strongly encourage all reviewers to take a moment to review the authors' responses and revised manuscript and provide your feedback before the discussion closes.

Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.