- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies failure-driven self-evolution of runtime harnesses for LLM agents and proposes ECDYSIS. ECDYSIS distinguishes task-model failures from systematic harness defects by aggregating recurring failures across tasks. It then applies Failure-Driven Collaborative Refinement (FDCR), where multiple roles analyze the evidence and produce a structured modification specification. A candidate harness is accepted only if it improves the training-set score. The method is evaluated on AgentBench and the Airline and Retail subsets of -Bench. Harnesses evolved with Qwen3-8B are tested on five models. ECDYSIS with FDCR achieves over 10% average accuracy improvement versus Self-Evolution, while also reducing token usage, trajectory time, and the manually audited model-specific accommodation (MSA) ratio.
Critical strengths
-
The paper clearly distinguishes task-model errors from persistent harness defects and uses cross-task recurrence plus refinement to avoid overfitting the harness to individual model failures, which should be a clear and useful reframing of failure-driven harness evolution.
-
The experiments cover five task models and three benchmarks, showing consistent accuracy gains together with lower token usage and runtime. Cross-model evaluation is particularly relevant to the paper’s goal of reducing model-specific accommodation.
Critical weaknesses
-
Comparisons are not compute-matched. Self-Evolution and ECDYSIS use different numbers and types of LLM calls and follow different search trajectories. It is difficult to isolate whether the gains come from the proposed method or unequal search opportunities.
-
The MSA/HLR analysis is insufficiently validated. The classification is based on manually merged decisions labeled by five author annotators, but the paper does not report decision counts, inter-rater agreement, blinding, or sensitivity to the merging procedure. Since this analysis supports a central mechanistic claim, stronger reliability evidence is needed.
Questions for authors
-
Can the authors provide compute-matched comparisons across Self-Evolution, ECDYSIS without FDCR, and ECDYSIS with FDCR, using equal LLM-call, token, or wall-clock budgets?
-
How many modification decisions were labeled for each method and dataset?
urgent!
I noticed that none of the reviewers has provided follow-up feedback so far. As the discussion period is approaching its end, I would strongly encourage all reviewers to take a moment to review the authors' responses and revised manuscript and provide your feedback before the discussion closes.
Thank you!