- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper argues that propose-evaluate-select optimizers for skills, harnesses and pipelines discard rejected candidates that may carry partial fixes. Mara Chain instead keeps a rejected candidate and refines it for up to d steps from its own files and history, and bounds the pool with Pareto filtering plus top-N truncation. With GLM-5 as the main target model, it reports gains over GEPA, ACE and SkillOpt-Lite on AppWorld, 71.9 percent against 51.7 for AHE on TerminalBench 2.1, and +0.104 nDCG@10 over a hand-written pipeline on MuSiQue.
Critical strengths
- The topic is timely and worth studying. Self-evolving harnesses and skills are an active direction, and the observation that a rejected edit can carry a partial fix is a reasonable motivation, shown clearly by the sanitize-git-repo case.
- The scope is broad and the mechanism is simple. Three artifact types and three target models are covered, and the chain is training-free and bounded by a fixed depth, so it is easy to adopt.
Critical weaknesses
- Novelty is limited. Keeping non-improving candidates as stepping stones is central to archive-based search such as MAP-Elites and the Darwin Gödel Machine, and agentic proposers that read all prior candidates' code, scores and traces already exist (Meta-Harness). Continuing from the rejected candidate's files instead of the parent's is a small step beyond these.
- The experiments are not solid. The main body does not state the TerminalBench split, and the appendix shows that the 89 test tasks are also the training tasks and that the AppWorld test set contains the 200 validation tasks. Baselines differ per benchmark, MuSiQue has no optimizer baseline although GEPA handles multi-hop retrieval, the reproduced AHE and Meta-Harness do not beat the unoptimized base, and all results are single runs. Without the chain the method already reaches 57.3 percent on TerminalBench, above both baselines.
- The paper is hard to follow. The method is spread across a dense figure, an algorithm and several loop concepts, and the proposer model, the budgets and the 1-slot and 2-slot settings are not explained in the main body.
Questions for authors
- In the depth-0 ablation, can later proposals still read the logs and analyses of rejected candidates? The answer decides whether the gain comes from keeping the files or from keeping the history.
- How were the AppWorld curves and baselines produced, given that the appendix states optimizer settings, budgets and baseline launch parameters are unavailable? Were rollout and proposer budgets matched across methods?
- What are the TerminalBench and AppWorld results on tasks not seen during optimization or selection?
17 comments
Response on Novelty
Response on Novelty
Contributions
Our core contributions are three:
Core contribution 1 (problem formulation): an acceptance signal measures a candidate's immediate performance, not the value of the rejected attempt for subsequent optimization. Candidates that fail the bar are rejected alike—whether a hard error (score 0) or a near miss.We distinguish three forms of such value:
- Partial fix: a sub-condition improved but the whole still does not clear the bar, and descendants can extend on top of it;
- Right direction, but the implementation still has bugs: descendants only need to repair rather than rebuild from scratch (for example, a prompt that is only 30% likely to get it right, where the agent tries switching to a script but the script has a bug — both the directional judgment "a script is the right tool" and the buggy first version of the script are inheritable assets);
- Informative negative evidence: an unsuccessful modification establishes that a particular change did not work under the observed artifact and task conditions. Its value lies in preserving this negative result together with the associated change and outcome, so that descendants can avoid repeating the same attempt and redirect their search.
These cases arise from the combination of stochastic LLM behavior and the multi-step, dependency-heavy nature of real tasks: meaningful progress often requires several dependent modifications, and an intermediate attempt may contain useful artifact state and diagnostic evidence without producing an immediate improvement in the aggregate acceptance score. The distinction from prior optimizers is therefore not whether they can ever retain or revisit a failed attempt, but whether the search procedure is explicitly designed to identify and exploit these different forms of value. This is the design starting point of Mara Chain.
Core contribution 2 (the core algorithm, bounded, history-conditioned local deep search): Mara Chain, designed around capturing these three kinds of value. For each candidate that does not clear the bar, we perform a bounded (≤d steps, i.e., under the given budget) depth-first refinement on its minibatch; the entire pipeline — evidence aggregation, memory recall, context engineering, analysis, and modification — is carried out by a workflow-driven, purpose-built Mara Chain analysis agent and mutation agent. The goal of the algorithm is to let experience and assets accumulate and reinforce each other across successive attempts on the same task, producing stronger descendants. (We embed Mara Chain within a population-search outer loop: broad, population-based exploration proposes candidates, and only a rejected candidate triggers a bounded (≤d-step), history-conditioned local deep search refining that same candidate on its own chain and minibatch — we inherit the classic global–local skeleton and redesign its two joints: the rejection-based trigger, and the history-conditioned refinement operator.)
Core contribution 3 (empirical validation): We demonstrate experimentally that Mara Chain's bounded, history-conditioned local deep search is effective for candidates that fail to clear a well-defined acceptance bar, improving both the optimization ceiling and optimization efficiency across diverse system architectures and tasks.
We elaborate in three parts below: ① a point-by-point comparison with OpenEvolve/MAP-Elites, DGM, and Meta-Harness along two angles — "compute allocation" and "evidence aggregation"; ② an empirical audit of Meta-Harness; ③ our design philosophy (all facts come from a line-by-line inspection of the official OpenEvolve and DGM code and from tool-call-level forensics on a real Meta-Harness optimization run, and are reproducible).
Response on Novelty(continue)
I. Differences in Search Control and Cross-Attempt Evidence Utilization
The central distinction is not simply whether a method retains non-improving candidates, nor whether it performs broad or deep search. It is how the optimizer determines whether a particular unsuccessful attempt receives sustained, evidence-conditioned refinement.
OpenEvolve/MAP-Elites and DGM perform broad, archive or population based search: non-improving candidates may remain available and later influence the search through parent selection. Meta-Harness instead allows the agentic proposer to decide which prior candidates and traces to inspect and how to continue the search. Our framework adds Mara Chain to an outer broad-search loop (currently using Pareto-filtered Top-N, with MAP-Elites supported as an alternative evolutionary algorithm). Mara Chain provides budget-bounded depth-first search on a given minibatch: when a direct proposal misses the acceptance bar, it starts a refinement episode from that rejected proposal on the same minibatch. The overall procedure therefore combines broad candidate-pool exploration with explicit local refinement.
We distinguish the methods along two dimensions: (1) what determines subsequent investment in a particular rejected attempt, and (2) whether successive attempts on the same task are organized into a coherent, history-conditioned refinement process. We address the first dimension here and the second in Angle 2.
Angle 1: What Determines Subsequent Investment in a Rejected Attempt?
All three prior methods may revisit a rejected candidate; none makes rejection itself a control event that guarantees sustained, evidence-conditioned refinement.
| Method | Mechanism determining subsequent work on a particular unsuccessful candidate |
|---|---|
| OpenEvolve / MAP-Elites | The candidate may remain eligible for future parent selection depending on its archive or niche status. Its subsequent use is governed by stochastic selection within the population; there is no rejection-triggered guarantee of consecutive refinement from that candidate. |
| DGM | The candidate may be revisited through weighted archive sampling. Each selection also increases children_count, which further reduces its subsequent selection weight. This diversity-promoting penalty contrasts with Mara Chain’s bounded but sustained refinement of the same rejected lineage. |
| Meta-Harness | The proposer decides whether to revisit a prior candidate, how much of its code and execution evidence to inspect, and whether to continue from it. History access is available, but a rejected-candidate refinement episode is not prescribed by the optimizer. |
| Mara Chain | A rejected direct proposal becomes the starting state of a bounded local refinement episode. Subject to the remaining global budget, the procedure performs up to the configured depth on the same minibatch, with early stopping when the acceptance condition is met. Artifact inheritance, refinement triggering, and termination are specified by the algorithm rather than left to archive resampling or proposer discretion. |
This distinction does not imply that archive-based reuse or agent-directed search is ineffective. Our claim is more specific: the possibility of revisiting a rejected candidate is not equivalent to a mechanism that organizes a sequence of related attempts into a coherent refinement process. Mara Chain deliberately reserves part of the search budget for extracting value from a rejected attempt and couples that allocation to the lineage-evidence mechanism described in Angle 2.
As an implementation-level illustration, the official OpenEvolve selection rule contains exploration and random channels that sample candidates independently of whether they improved upon their parents. Under the representative configuration we inspected—100 candidates distributed evenly across five islands and 20 archive elites—these channels correspond to approximately a 0.3% per-round selection probability for a particular candidate, while an archive elite may receive approximately 3.5% through the exploitation channel. In DGM, parent selection is weighted by score and inversely by children_count; in our representative 60-commit example, a low-scoring candidate has an approximately 1% selection probability, which decreases after subsequent selections.
These numbers are intended only to make the control policies concrete, not as universal performance bounds or as evidence that lower selection probability is inherently undesirable. The relevant distinction is that repeated refinement of a particular rejected candidate is contingent on global sampling in these methods, whereas Mara Chain makes it an explicit local-search operation with bounded cost.
Begin words
Dear Reviewer,
Thank you very much for investing a great deal of time in reviewing our paper in depth. Your questions hit the mark and prompted us to re-examine how the paper presents its evidence; at the same time, we sincerely thank you for affirming our core observation and the simplicity of the method.
In particular, thank you for suggesting that we add a comparison against GEPA on the MuSiQue RAG pipeline. We have completed this set of experiments strictly under the same budget, and it also made us see more clearly the value of the problem (results below).
Your review shows first-hand research experience in this area, and we take every one of your criticisms seriously. We reply point by point below, and we are very glad to share some of our views with you in the hope of answering your questions; wherever anything is unclear, we would be grateful for further correction.
We apologize for the late reply — during the rebuttal period we adopted the reviewers' suggestions, ran a substantial number of additional experiments, and also found a wording error in the paper regarding reproducibility, which we commit to correcting in the revision (see the "Reproducibility" section):
These suggestions benefited us greatly and helped us polish the paper; we thank you for them as well.