acceptodds
Under review as a conference paper at ICLR 2027

Mara Chain: Rethinking Failure as a Stepping Stone for AI System Auto-Evolution

Abstract

Optimizing deployed AI systems increasingly amounts to editing prompts, skills, harnesses, and code rather than model weights. Existing approaches commonly optimize these artifacts through propose–evaluate–select procedures, where candidate configurations are evaluated and only those meeting an acceptance criterion are selected. Yet our analysis shows that discarded candidates often contain information critical for subsequent optimization. Discarding them causes later proposals to revisit the same failure modes. We introduce Mara Chain, a refinement procedure that turns rejected candidates into stepping stones. Rather than discarding a rejected candidate, Mara Chain retains and iteratively refines it using evidence accumulated across preceding attempts. The procedure limits each refinement chain to a fixed depth and applies Pareto-filtered Top- selection to bound the candidate pool. Across AppWorld skill optimization, TerminalBench 2.1 harness optimization, and MuSiQue retrieval-pipeline optimization, Mara Chain delivers greater task-performance gains with fewer rollouts. It outperforms GEPA, ACE, and SkillOpt-Lite by up to in relative performance on AppWorld, reaching the target score with fewer rollouts than GEPA. It improves the pass rate by and percentage points over AHE and Meta-Harness on TerminalBench 2.1, respectively, and improves MuSiQue test nDCG@10 and Recall@10 by and over a hand-written retrieval pipeline.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, self-evolving AI systems, prompt optimization, harness optimization, evolutionary search, natural language reflection, failure-driven optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XAdVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that propose-evaluate-select optimizers for skills, harnesses and pipelines discard rejected candidates that may carry partial fixes. Mara Chain instead keeps a rejected candidate and refines it for up to d steps from its own files and history, and bounds the pool with Pareto filtering plus top-N truncation. With GLM-5 as the main target model, it reports gains over GEPA, ACE and SkillOpt-Lite on AppWorld, 71.9 percent against 51.7 for AHE on TerminalBench 2.1, and +0.104 nDCG@10 over a hand-written pipeline on MuSiQue.

Critical strengths
  1. The topic is timely and worth studying. Self-evolving harnesses and skills are an active direction, and the observation that a rejected edit can carry a partial fix is a reasonable motivation, shown clearly by the sanitize-git-repo case.
  2. The scope is broad and the mechanism is simple. Three artifact types and three target models are covered, and the chain is training-free and bounded by a fixed depth, so it is easy to adopt.
Critical weaknesses
  1. Novelty is limited. Keeping non-improving candidates as stepping stones is central to archive-based search such as MAP-Elites and the Darwin Gödel Machine, and agentic proposers that read all prior candidates' code, scores and traces already exist (Meta-Harness). Continuing from the rejected candidate's files instead of the parent's is a small step beyond these.
  2. The experiments are not solid. The main body does not state the TerminalBench split, and the appendix shows that the 89 test tasks are also the training tasks and that the AppWorld test set contains the 200 validation tasks. Baselines differ per benchmark, MuSiQue has no optimizer baseline although GEPA handles multi-hop retrieval, the reproduced AHE and Meta-Harness do not beat the unoptimized base, and all results are single runs. Without the chain the method already reaches 57.3 percent on TerminalBench, above both baselines.
  3. The paper is hard to follow. The method is spread across a dense figure, an algorithm and several loop concepts, and the proposer model, the budgets and the 1-slot and 2-slot settings are not explained in the main body.
Questions for authors
  1. In the depth-0 ablation, can later proposals still read the logs and analyses of rejected candidates? The answer decides whether the gain comes from keeping the files or from keeping the history.
  2. How were the AppWorld curves and baselines produced, given that the appendix states optimizer settings, budgets and baseline launch parameters are unavailable? Were rollout and proposer budgets matched across methods?
  3. What are the TerminalBench and AppWorld results on tasks not seen during optimization or selection?
17 comments
AuthorsOpenReview ↗

Begin words

Dear Reviewer,

Thank you very much for investing a great deal of time in reviewing our paper in depth. Your questions hit the mark and prompted us to re-examine how the paper presents its evidence; at the same time, we sincerely thank you for affirming our core observation and the simplicity of the method.

In particular, thank you for suggesting that we add a comparison against GEPA on the MuSiQue RAG pipeline. We have completed this set of experiments strictly under the same budget, and it also made us see more clearly the value of the problem (results below).

Your review shows first-hand research experience in this area, and we take every one of your criticisms seriously. We reply point by point below, and we are very glad to share some of our views with you in the hope of answering your questions; wherever anything is unclear, we would be grateful for further correction.

We apologize for the late reply — during the rebuttal period we adopted the reviewers' suggestions, ran a substantial number of additional experiments, and also found a wording error in the paper regarding reproducibility, which we commit to correcting in the revision (see the "Reproducibility" section):

  • AppWorld strict held-out evaluation: after removing the val tasks used for optimization, re-evaluate all methods and all ablation configurations on the remaining tasks;
  • MuSiQue optimizer baseline: a GEPA comparison under the same budget, with the same pipeline and the same evaluator;
  • Ours(−history) ablation: downgrade the Mara Chain analysis agent to the default analysis agent, removing the chain-injected history while keeping the changelog, compared against the full configuration at matched budget on AppWorld and MuSiQue;
  • mean@3 scores added to each experimental evaluation: independently evaluate the final artifact of each configuration three times and average (test); val takes the first run (pass@1), and the optimization process remains a single run;
  • Cross-model validation: re-evaluate on the held-out tasks after swapping the system and proposer models;
  • Reproducibility: clarify that "manifests/budgets unavailable" in the appendix was an archiving mistake rather than missing data, and commit to releasing all configurations, task splits, and final artifacts with the public package;

These suggestions benefited us greatly and helped us polish the paper; we thank you for them as well.

AuthorsOpenReview ↗

Response on Novelty

Response on Novelty

Contributions

Our core contributions are three:

Core contribution 1 (problem formulation): an acceptance signal measures a candidate's immediate performance, not the value of the rejected attempt for subsequent optimization. Candidates that fail the bar are rejected alike—whether a hard error (score 0) or a near miss.We distinguish three forms of such value:

  1. Partial fix: a sub-condition improved but the whole still does not clear the bar, and descendants can extend on top of it;
  2. Right direction, but the implementation still has bugs: descendants only need to repair rather than rebuild from scratch (for example, a prompt that is only 30% likely to get it right, where the agent tries switching to a script but the script has a bug — both the directional judgment "a script is the right tool" and the buggy first version of the script are inheritable assets);
  3. Informative negative evidence: an unsuccessful modification establishes that a particular change did not work under the observed artifact and task conditions. Its value lies in preserving this negative result together with the associated change and outcome, so that descendants can avoid repeating the same attempt and redirect their search.

These cases arise from the combination of stochastic LLM behavior and the multi-step, dependency-heavy nature of real tasks: meaningful progress often requires several dependent modifications, and an intermediate attempt may contain useful artifact state and diagnostic evidence without producing an immediate improvement in the aggregate acceptance score. The distinction from prior optimizers is therefore not whether they can ever retain or revisit a failed attempt, but whether the search procedure is explicitly designed to identify and exploit these different forms of value. This is the design starting point of Mara Chain.

Core contribution 2 (the core algorithm, bounded, history-conditioned local deep search): Mara Chain, designed around capturing these three kinds of value. For each candidate that does not clear the bar, we perform a bounded (≤d steps, i.e., under the given budget) depth-first refinement on its minibatch; the entire pipeline — evidence aggregation, memory recall, context engineering, analysis, and modification — is carried out by a workflow-driven, purpose-built Mara Chain analysis agent and mutation agent. The goal of the algorithm is to let experience and assets accumulate and reinforce each other across successive attempts on the same task, producing stronger descendants. (We embed Mara Chain within a population-search outer loop: broad, population-based exploration proposes candidates, and only a rejected candidate triggers a bounded (≤d-step), history-conditioned local deep search refining that same candidate on its own chain and minibatch — we inherit the classic global–local skeleton and redesign its two joints: the rejection-based trigger, and the history-conditioned refinement operator.)

Core contribution 3 (empirical validation): We demonstrate experimentally that Mara Chain's bounded, history-conditioned local deep search is effective for candidates that fail to clear a well-defined acceptance bar, improving both the optimization ceiling and optimization efficiency across diverse system architectures and tasks.

We elaborate in three parts below: ① a point-by-point comparison with OpenEvolve/MAP-Elites, DGM, and Meta-Harness along two angles — "compute allocation" and "evidence aggregation"; ② an empirical audit of Meta-Harness; ③ our design philosophy (all facts come from a line-by-line inspection of the official OpenEvolve and DGM code and from tool-call-level forensics on a real Meta-Harness optimization run, and are reproducible).

AuthorsOpenReview ↗

Response on Novelty(continue)

I. Differences in Search Control and Cross-Attempt Evidence Utilization

The central distinction is not simply whether a method retains non-improving candidates, nor whether it performs broad or deep search. It is how the optimizer determines whether a particular unsuccessful attempt receives sustained, evidence-conditioned refinement.

OpenEvolve/MAP-Elites and DGM perform broad, archive or population based search: non-improving candidates may remain available and later influence the search through parent selection. Meta-Harness instead allows the agentic proposer to decide which prior candidates and traces to inspect and how to continue the search. Our framework adds Mara Chain to an outer broad-search loop (currently using Pareto-filtered Top-N, with MAP-Elites supported as an alternative evolutionary algorithm). Mara Chain provides budget-bounded depth-first search on a given minibatch: when a direct proposal misses the acceptance bar, it starts a refinement episode from that rejected proposal on the same minibatch. The overall procedure therefore combines broad candidate-pool exploration with explicit local refinement.

We distinguish the methods along two dimensions: (1) what determines subsequent investment in a particular rejected attempt, and (2) whether successive attempts on the same task are organized into a coherent, history-conditioned refinement process. We address the first dimension here and the second in Angle 2.

Angle 1: What Determines Subsequent Investment in a Rejected Attempt?

All three prior methods may revisit a rejected candidate; none makes rejection itself a control event that guarantees sustained, evidence-conditioned refinement.

MethodMechanism determining subsequent work on a particular unsuccessful candidate
OpenEvolve / MAP-ElitesThe candidate may remain eligible for future parent selection depending on its archive or niche status. Its subsequent use is governed by stochastic selection within the population; there is no rejection-triggered guarantee of consecutive refinement from that candidate.
DGMThe candidate may be revisited through weighted archive sampling. Each selection also increases children_count, which further reduces its subsequent selection weight. This diversity-promoting penalty contrasts with Mara Chain’s bounded but sustained refinement of the same rejected lineage.
Meta-HarnessThe proposer decides whether to revisit a prior candidate, how much of its code and execution evidence to inspect, and whether to continue from it. History access is available, but a rejected-candidate refinement episode is not prescribed by the optimizer.
Mara ChainA rejected direct proposal becomes the starting state of a bounded local refinement episode. Subject to the remaining global budget, the procedure performs up to the configured depth dd on the same minibatch, with early stopping when the acceptance condition is met. Artifact inheritance, refinement triggering, and termination are specified by the algorithm rather than left to archive resampling or proposer discretion.

This distinction does not imply that archive-based reuse or agent-directed search is ineffective. Our claim is more specific: the possibility of revisiting a rejected candidate is not equivalent to a mechanism that organizes a sequence of related attempts into a coherent refinement process. Mara Chain deliberately reserves part of the search budget for extracting value from a rejected attempt and couples that allocation to the lineage-evidence mechanism described in Angle 2.

As an implementation-level illustration, the official OpenEvolve selection rule contains exploration and random channels that sample candidates independently of whether they improved upon their parents. Under the representative configuration we inspected—100 candidates distributed evenly across five islands and 20 archive elites—these channels correspond to approximately a 0.3% per-round selection probability for a particular candidate, while an archive elite may receive approximately 3.5% through the exploitation channel. In DGM, parent selection is weighted by score and inversely by children_count; in our representative 60-commit example, a low-scoring candidate has an approximately 1% selection probability, which decreases after subsequent selections.

These numbers are intended only to make the control policies concrete, not as universal performance bounds or as evidence that lower selection probability is inherently undesirable. The relevant distinction is that repeated refinement of a particular rejected candidate is contingent on global sampling in these methods, whereas Mara Chain makes it an explicit local-search operation with bounded cost.

Reviewer_VKYAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to optimize AI systems without updating the weights of the underlying foundation model. Instead, the optimization target consists of external system artifacts, such as prompts, skills, scripts, agent harnesses, and pipeline code. The authors argue that conventional propose-evaluate-select frameworks typically discard candidates that fail to exceed a selection threshold, even though such candidates may already contain useful partial fixes, expose new downstream failure modes, or rule out unproductive directions.

To address this issue, the paper introduces Mara Chain, which retains a candidate artifact together with its execution trajectories, structured failure analyses, and modification history, and then continues refining the same lineage until it either satisfies an acceptance criterion or reaches a maximum chain depth. At the outer-loop level, the method maintains a Pareto-filtered Top-N population. The framework is evaluated on three substantially different optimization targets: an AppWorld skill, a TerminalBench agent harness, and a MuSiQue retrieval pipeline.

The final systems achieve competitive performance and optimization efficiency. The case studies also illustrate a meaningful progression from modifying natural-language instructions to introducing scripts and, when necessary, changing harness or pipeline code. Overall, the paper addresses an important problem, is clearly written, demonstrates some methodological novelty, and presents a relatively complete engineering system. I lean toward acceptance.

Critical strengths

The problem is important and the motivation is clear. Treating rejected candidates as evidence rather than simply discarding them is a sensible and practically relevant observation. A candidate may fail globally while still containing a useful local improvement, revealing a newly exposed downstream bottleneck, or providing evidence that a particular direction is ineffective.

Mara Chain provides a concrete mechanism beyond single-step candidate search. A rejected candidate can become the direct parent of the next proposal. The next optimization step is conditioned not only on the current artifact, but also on evaluation deltas, raw execution trajectories, structured analyses, and an attributed changelog. This makes the method more than repeated independent prompting.

The system design is complete and reasonably traceable. The two-stage proposer, multiple analysis agents, a single mutation agent, filesystem-based lineage storage, validation isolation, and controlled changelog construction together form a coherent implementation rather than a loosely specified optimization idea.

The experimental scope is broad. The paper evaluates an AppWorld skill, a TerminalBench harness, and a non-agentic MuSiQue retrieval DAG. These targets differ considerably in structure, suggesting that the framework can adapt to different artifact types once the relevant interfaces and schemas have been defined.

Critical weaknesses
  1. AppWorld would benefit from a truly independent held-out evaluation and more complete reproducibility information

The 200-task validation set is constructed by sampling 100 tasks from test_normal and 100 tasks from test_challenge. However, the final reported results on all 585 tasks include these same 200 tasks. Therefore, the main AppWorld result is not fully independent of the optimization process.

I suggest additionally reporting performance on the remaining 68 test_normal tasks and 317 test_challenge tasks, or otherwise constructing a strictly disjoint held-out split and evaluating the same final artifacts on it. This would make it easier to assess whether the optimized skill generalizes beyond the tasks used to guide the optimization loop.

In addition, the archived AppWorld records appear to contain mainly static evaluation outputs, while some information needed to reconstruct the optimization process—such as optimizer manifests, proposer/system assignments, minibatch definitions, and certain launch parameters—is not readily available. Releasing the relevant configurations, final artifacts, task IDs, and run metadata would substantially improve reproducibility.

  1. The method relies on substantial manual design of the initial harness, artifact schema, which limits its current practicality and generality

Applying the framework to a new domain appears to require the user to define a System runner, an initial harness or pipeline. The method therefore optimizes within a human-designed and constrained search space; it does not automatically discover what should be optimized in an arbitrary system.

Moreover, all three experiments start from relatively mature initial systems: an AppWorld skill and API environment, the Kira harness for TerminalBench, and a manually written MuSiQue pipeline with predefined nodes. It remains unclear how robust the method would be when the initial harness is weak, the artifact schema is poorly designed, or a reliable evaluator is unavailable. The paper also does not quantify the human effort required to integrate a new task.

The paper should distinguish more explicitly between the generality of the optimization framework and the cost of integrating a new domain. A sensitivity study using different initial harnesses or schemas, an experiment starting from a simpler and more generic scaffold, or at least a quantitative accounting of human implementation effort would strengthen the practical claims.

Questions for authors

Can the authors report AppWorld results on a held-out set that is fully disjoint from the optimization validation set, and release the corresponding task IDs, final artifacts, evaluation commands, and optimization-run manifests?

How much human design and implementation effort is required to integrate a new system into Mara Chain? How sensitive is the method to the initial harness, artifact schema? Can a simpler or more generic initial scaffold achieve comparable improvements?

7 comments
AuthorsOpenReview ↗

Generalization: Response on held-out Results

Dear Reviewer,

Thank you very much for your careful and in-depth review. You accurately summarized our problem motivation, mechanism design, and system completeness, and gave us positive feedback, which encouraged us greatly.

Your questions on the AppWorld held-out evaluation and reproducibility hit the mark; and your question about the human integration cost and sensitivity of the framework for new domains was even more illuminating. We reply point by point below; wherever anything is unclear, we would be grateful for further correction.

We apologize for the late reply — during the rebuttal period we adopted the reviewers' suggestions, ran a substantial number of additional experiments, and also found a wording error in the paper regarding reproducibility, which we commit to correcting in the revision (see the "Reproducibility" section):

  • AppWorld strict held-out evaluation: after removing the val tasks used for optimization, re-evaluate all methods and all ablation configurations on the remaining tasks;
  • MuSiQue optimizer baseline: a GEPA comparison under the same budget, with the same pipeline and the same evaluator;
  • Ours(−history) ablation: downgrade the Mara Chain analysis agent to the default analysis agent, removing the chain-injected history while keeping the changelog, compared against the full configuration at matched budget on AppWorld and MuSiQue;
  • mean@3 scores added to each experimental evaluation: independently evaluate the final artifact of each configuration three times and average (test); val takes the first run (pass@1), and the optimization process remains a single run;
  • Cross-model validation: re-evaluate on the held-out tasks after swapping the system and proposer models;
  • Reproducibility: clarify that "manifests/budgets unavailable" in the appendix was an archiving mistake rather than missing data, and commit to releasing all configurations, task splits, and final artifacts with the public package;

These suggestions benefited us greatly and helped us polish the paper; we thank you for them as well.

Generalization: Response on held-out Results

Results on the 385 held-out tasks (TGC, %, mean@3). The task split is described in the "Reproducibility" section — the 385 tasks never participated in optimization at any point, constituting a strict held-out set. Scoring protocol: normal and challenge are not distinguished; we directly average the TGC over the 385 tasks; we report only TGC and not SGC (the val sampling was done at the task level, which splits scenarios apart, so SGC on held-out cannot be point-estimated and carries no conclusive value):

MethodTGC (385, mean@3)
Empty skill (GLM-5)41.3
SkillOpt-Lite72.6
GEPA74.0
ACE76.4
Ours80.8
Ours(−Mara Chain)75.6
Ours(−Top-N)77.2
Ours(−history)77.6

Our method is the best on the 385 tasks that never participated in optimization — the improvement is not overfitting to the optimization tasks. The ablation also holds: Ours is +5.2pp higher than Ours(−Mara Chain) (80.8 vs 75.6), so the chain's contribution persists on held-out; the comparison without chain history (Ours(−history)) scores 77.6, 3.2pp below the full configuration, so historical information contributes as well. Cross-model evaluation (the same optimization artifacts, with the system and proposer models swapped; all pass@1) generalizes consistently too: DeepSeek-V4-Pro 67.3→84.9 (+17.6pp), Qwen3.5-397B 56.9→76.6 (+19.8pp), GLM-5 42.3→79.0 (+36.7pp).

MuSiQue provides support on a second, independent benchmark. At another reviewer's suggestion, we added a comparison experiment of GEPA on the MuSiQue RAG pipeline, with very significant results; MuSiQue's test split and val do not overlap at all at either the query-id or the doc-id level, and val does not participate in the final evaluation (the split is detailed in the "Reproducibility" section). We should note: the official test split contains no annotations, so both val and test are carved out of the annotated official dev 4-hop set — 100 items for val, and 300 for test drawn from the remaining 4-hop items (a fixed id list, provided with the release package). We therefore position MuSiQue as a test comparison on unseen queries from the same distribution. We used the official GEPA engine to optimize the same RAG pipeline as in the paper, with the same budget (20h) and the same evaluator as Ours:

val nDCG@10 (pass@1)val Recall@10 (pass@1)test nDCG@10 (mean@3)test Recall@10 (mean@3)
Baseline (unoptimized pipeline)0.41850.39500.30140.3275
After GEPA optimization0.45300.46000.32720.3556
Ours0.55290.60500.40430.4601
Ours(−Mara Chain)0.50520.56000.37130.4192
Ours(−history)0.53450.54750.37740.4182
AuthorsOpenReview ↗

Reproducibility: Clarifications on Experimental Configurations, Data Splits, and Archiving Gaps

Reproducibility: Clarifications on Experimental Configurations, Data Splits, and Archiving Gaps

You noted that the archived AppWorld records are missing optimizer manifests, proposer/system assignments, minibatch definitions, and some launch parameters. Let us first clarify: the "unavailable" statement in the appendix was a mistake in an intermediate operation while using an agent to organize the archive (the archiving script omitted these files) that we failed to catch — the data does exist. The original run directories are fully preserved, and all configurations can be recovered.

We commit to releasing everything: the parameter configuration of every experiment (optimizer manifests, proposer models and configs, minibatch definitions, 1-slot/2-slot settings, rollout and token budgets, baseline launch parameters), the framework code, dataset sampling and splits (task ID lists), system artifacts (final skill/harness/pipeline), the manifests and metadata of all optimization runs, and the complete experiment and evaluation scripts.

Dataset splits:

BenchmarkSplit
AppWorld (v0.2.0.dev0)train pool 147 (official train+dev); val 200 (test_normal 100 + test_challenge 100; seed 42, stratified by GLM-5 empty-skill score: ~10% solved + ~90% failed, actually 23 solved + 177 failed); held-out 385 (the remaining 68 + 317, never used in optimization)
TerminalBench 2.189 tasks, no separate val (consistent with the AHE/Meta-Harness protocol: optimization and evaluation use the same task set; this is an in-sample protocol, already noted in the paper)
MuSiQue (4-hop subset)train 300 (official train 4-hop) / val 100 (official dev 4-hop) / test 300 (official dev 4-hop, excluding the 100 val ids); the query ids and doc ids of the three splits are mutually disjoint (the official test set has no annotations; the carving procedure is described below)

AppWorld split rationale: we did not use the official 90 train as train and 57 dev as val, because the data volume is too small: our method reaches an average score of 0.965 on the 57 dev items (out of a maximum of 1) in about 1 hour, making it very hard for the effect of optimization changes to show. We also pointed out a problem with the Pareto frontier used by GEPA, and verifying that problem requires val to be not too small and to contain a certain number of non-trivial questions. Moreover, under the official 90 train / 57 dev recipe, the final results also lack good discriminative power. We therefore adopted a larger train pool and a stratified val of 200 (test_normal 100 + test_challenge 100; stratified by GLM-5 empty-skill score, targeting ~10% solved (actually 23 solved + 177 failed); seed 42).

MuSiQue split rationale: the official musique_full_v1.0_test only provides id / paragraphs / question, without answer or is_supporting, so nDCG@10 / Recall@10 cannot be computed; val/test can therefore only be carved from the annotated official dev: val is 100 items of dev 4-hop, and test is 300 items taken from dev 4-hop after excluding val. The three splits are mutually disjoint at the query-id and doc-id levels; our usage protocol is: val is used only for development-time tuning and candidate selection and does not participate in the final evaluation; MuSiQue test results should be read as a comparison on unseen queries from the same distribution; all methods share the same split and evaluator, so comparisons between methods are unaffected.

Parameter settings for each method (the full version is released with the public package):

AuthorsOpenReview ↗

Reproducibility(continue)

MethodBenchmarkParameters
OursAppWorldsystem GLM-5 (max_steps=50, system API concurrency 25), evaluator concurrency 20, proposer concurrency 10 (GLM-5.1, context 170k), max_candidate_num=3, batch_size=4, epochs=3, max_iterations=1000, budget 40h. 1-slot (num_proposals=1): min_improvement_per_batch=3.0; 2-slot (num_proposals=2): min_improvement_per_batch=2.0; all other parameters are identical across the two configurations (max_reflection_iterations = 2 for both)
Ours / Ours(−Mara Chain)TB2.1system GLM-5, system API concurrency 4, evaluator concurrency 4, proposer concurrency 20, max_candidate_num=1, batch_size=4, budget 100h. num_proposals=1; +Mara Chain: max_reflection_iterations=5; −Mara Chain: max_reflection_iterations=0
Ours / Ours(−Mara Chain)MuSiQuenum_proposals=2 (2-slot), batch_size=6, min_improvement_per_batch=2.0, max_reflection_iterations=3; evaluator concurrency 8, proposer concurrency 10, max_candidate_num=3; budget 20h. Models inside the optimized pipeline: BAAI/bge-m3, BAAI/bge-reranker-v2-m3, GLM-5.2
Ours(−Mara Chain)AppWorldmax_reflection_iterations=0, min_improvement_per_batch disabled, everything else identical to the corresponding configuration
GEPAAppWorldopen-source gepa engine + our own adapter; system GLM-5 / reflection GLM-5.1; temperature 0.5 for both system and reflection; system API concurrency 25, evaluator concurrency 20; Pareto selection, hybrid frontier; reflection minibatch=3; merge (max 5, val overlap floor 5); skip-perfect-score; max refinements=1; max metric calls 2,000,000; seed 0; reflection max tokens 64000; budget 40h
GEPAMuSiQueoptimized pipeline identical to Ours (bge-m3, bge-reranker-v2-m3, GLM-5.2); open-source gepa engine, Pareto selection, merge (MAX_MERGE_INVOCATIONS=5); reflection GLM-5.1 (streaming, 32000 tokens); minibatch=3; generate concurrency 8; budget 20h
ACEAppWorldopen-source ACE + our own adapter (Reflector→Curator→apply); system GLM-5, reflector/curator GLM-5.1; system temperature 0.5, reflector 0.0, curator 0.3; system API concurrency 25, evaluator concurrency 20; reflect workers 6; reflector/curator max tokens 32767; max reflector rounds 5; playbook token budget 80000; budget 40h
SkillOpt-LiteAppWorldsystem GLM-5 (temperature 0.5), optimizer GLM-5.1 (CLI, no explicit temperature); system API concurrency 25, evaluator concurrency 20; max_steps 50; max_completion_tokens 64000; batch_size 40; minibatch 8; num_epochs unlimited (truncated by the 40h budget); budget 40h

Scoring protocol: AppWorld = TGC (per-task pass/fail, same evaluator as the final table); MuSiQue = per-query 0.2·nDCG@10 + 0.8·Recall@10; optimization and final testing use the same evaluator. Final evaluation protocol: test results are mean@3 (the fixed artifact is evaluated 3 times and averaged); val is the first-run result (pass@1); the optimization process is a single run. Budget matching: all compared methods use the same wall-clock budget on the same benchmark (AppWorld 40h, TB 100h, MuSiQue 20h), with the same system models and evaluation configuration.

Reviewer_FAhMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the optimization of compound AI systems through editable artifacts such as prompts, skills, harnesses, and code. The authors argue that conventional propose–evaluate–select methods often discard unsuccessful candidates that contain useful partial solutions. To address this, the paper proposes Mara Chain, a history-conditioned refinement procedure that retains rejected candidates and iteratively improves them using accumulated execution traces, failure diagnoses, and artifact-change histories. Refinement is bounded by a fixed chain depth, while Pareto-filtered Top-N selection maintains a bounded candidate population. The results show improvements over specialized baselines and a no-chain ablation, with particularly large gains on challenging tasks and reduced optimization rollouts.

Critical strengths
  1. A practical approach to a meaningful optimization problem. The paper identifies a limitation of candidate-based optimization: rejecting configurations based on immediate performance may discard partial improvements that require multiple dependent changes. Mara Chain addresses this by preserving unsuccessful artifacts and using their execution evidence to guide subsequent refinements. The idea is novel and the direction is critical.

  2. Strong empirical improvements on challenging tasks. Mara Chain outperforms several specialized optimizers on AppWorld and TerminalBench 2.1. The ablations provide useful evidence for the refinement procedure: removing Mara Chain reduces TerminalBench pass rate from 71.9% to 57.3% and AppWorld Challenge SGC from 59.0% to 51.8%. The improvement on MuSiQue over the no-chain baseline further suggests that the method can benefit different classes of tunable artifacts.

  3. A mostly general and well-structured optimization framework. The framework supports editing prompts, scripts, and executable harness code through a common artifact interface. The combination of bounded refinement depth, persistent execution records, and Pareto-filtered Top-N selection makes the search procedure reasonably practical.

Critical weaknesses
  1. The evaluation protocol substantially limits claims about generalization. AppWorld uses 200 tasks drawn from the official test splits as its optimization validation set, while the reported final results include these same tasks. Consequently, one of the paper's main evaluations do not provide fully held-out estimates of performance. This is particularly concerning given the flexibility of the optimizer to make task-specific artifact changes and overfitting on the eval set. Ideally the evolution framework should show generalizable gains, or at least gains on in-distribution completely separate test set.

  2. The central AppWorld comparisons are difficult to verify and reproduce. The paoer states that optimization manifests, proposer assignments, minibatch settings, budgets, and baseline launch parameters are unavailable for the archived AppWorld runs. This makes it difficult to assess the fairness of the reported comparisons and the claimed 65.5% reduction in rollouts relative to GEPA.

  3. The contribution of retaining rejected candidates is not fully isolated. The no-chain ablation demonstrates that iterative refinement helps, but it does not distinguish the effects of artifact inheritance, accumulated failure history, and repeated attempts on the same minibatch. As the core assumption of the paper that failure candidates also provide value, it's not validated and proved rigorously. A simpler optimizer that retries from the original parent while retaining diagnostic feedback might achieve similar gains. Since failure-driven reflection and iterative refinement have substantial prior work, stronger controlled ablations are needed to establish what specifically makes Mara Chain effective.

Questions for authors
  1. Can the authors provide evaluations that exclude tasks used during optimization? 200 AppWorld validation tasks are included in the final 585-task evaluation. Could the authors separately report results on the remaining 385 tasks, ideally using a single fixed optimized configuration for each model? This would help determine whether the improvements reflect generalizable system-level changes rather than adaptation to the optimization tasks.

  2. Can the authors clarify the AppWorld optimization protocol and provide reproducible comparisons? The paper states that the archived AppWorld records do not contain optimizer manifests or baseline launch configurations, despite the paper's central claims about sample efficiency and performance relative to GEPA, ACE, and SkillOpt-Lite. Could the authors provide the exact proposer models and configs, optimization budgets for efficiency, and evaluation procedures used for each method?

  3. Which part of Mara Chain is responsible for the gains: artifact inheritance, accumulated feedback, or repeated refinement? The current ablation compares the complete chain against single-shot optimization. Could the authors add controlled alternatives that (a) retry from the original parent using accumulated failure analyses without inheriting rejected artifacts, and (b) continue modifying rejected artifacts without the full historical analysis? These comparisons should use matched rollout and proposer-call budgets. This question echos the 3rd bullet point in the weakness, and is mainly targeted at answer the question: what is the value of retaining rejected candidates.

9 comments
AuthorsOpenReview ↗

Generalization: Response on held-out Results

Dear Reviewer,

Thank you very much for your careful and in-depth review. You accurately summarized our problem motivation, mechanism design, and system completeness, and gave us positive feedback, which encouraged us greatly.

Your questions on the AppWorld held-out evaluation and reproducibility hit the mark; and your question about the human integration cost and sensitivity of the framework for new domains was even more illuminating. We reply point by point below; wherever anything is unclear, we would be grateful for further correction.

We apologize for the late reply — during the rebuttal period we adopted the reviewers' suggestions, ran a substantial number of additional experiments, and also found a wording error in the paper regarding reproducibility, which we commit to correcting in the revision (see the "Reproducibility" section):

  • AppWorld strict held-out evaluation: after removing the val tasks used for optimization, re-evaluate all methods and all ablation configurations on the remaining tasks;
  • MuSiQue optimizer baseline: a GEPA comparison under the same budget, with the same pipeline and the same evaluator;
  • Ours(−history) ablation: downgrade the Mara Chain analysis agent to the default analysis agent, removing the chain-injected history while keeping the changelog, compared against the full configuration at matched budget on AppWorld and MuSiQue;
  • mean@3 scores added to each experimental evaluation: independently evaluate the final artifact of each configuration three times and average (test); val takes the first run (pass@1), and the optimization process remains a single run;
  • Cross-model validation: re-evaluate on the held-out tasks after swapping the system and proposer models;
  • Reproducibility: clarify that "manifests/budgets unavailable" in the appendix was an archiving mistake rather than missing data, and commit to releasing all configurations, task splits, and final artifacts with the public package;

These suggestions benefited us greatly and helped us polish the paper; we thank you for them as well.


Generalization: Response on held-out Results

Results on the 385 held-out tasks (TGC, %, mean@3). The task split is described in the "Reproducibility" section — the 385 tasks never participated in optimization at any point, constituting a strict held-out set. Scoring protocol: normal and challenge are not distinguished; we directly average the TGC over the 385 tasks; we report only TGC and not SGC (the val sampling was done at the task level, which splits scenarios apart, so SGC on held-out cannot be point-estimated and carries no conclusive value):

MethodTGC (385, mean@3)
Empty skill (GLM-5)41.3
SkillOpt-Lite72.6
GEPA74.0
ACE76.4
Ours80.8
Ours(−Mara Chain)75.6
Ours(−Top-N)77.2
Ours(−history)77.6

Our method is the best on the 385 tasks that never participated in optimization — the improvement is not overfitting to the optimization tasks. The ablation also holds: Ours is +5.2pp higher than Ours(−Mara Chain) (80.8 vs 75.6), so the chain's contribution persists on held-out; the comparison without chain history (Ours(−history)) scores 77.6, 3.2pp below the full configuration, so historical information contributes as well. Cross-model evaluation (the same optimization artifacts, with the system and proposer models swapped; all pass@1) generalizes consistently too: DeepSeek-V4-Pro 67.3→84.9 (+17.6pp), Qwen3.5-397B 56.9→76.6 (+19.8pp), GLM-5 42.3→79.0 (+36.7pp).

MuSiQue provides support on a second, independent benchmark. At another reviewer's suggestion, we added a comparison experiment of GEPA on the MuSiQue RAG pipeline, with very significant results; MuSiQue's test split and val do not overlap at all at either the query-id or the doc-id level, and val does not participate in the final evaluation (the split is detailed in the "Reproducibility" section). We should note: the official test split contains no annotations, so both val and test are carved out of the annotated official dev 4-hop set — 100 items for val, and 300 for test drawn from the remaining 4-hop items (a fixed id list, provided with the release package). We therefore position MuSiQue as a test comparison on unseen queries from the same distribution. We used the official GEPA engine to optimize the same RAG pipeline as in the paper, with the same budget (20h) and the same evaluator as Ours:

val nDCG@10 (pass@1)val Recall@10 (pass@1)test nDCG@10 (mean@3)test Recall@10 (mean@3)
Baseline (unoptimized pipeline)0.41850.39500.30140.3275
After GEPA optimization0.45300.46000.32720.3556
Ours0.55290.60500.40430.4601
Ours(−Mara Chain)0.50520.56000.37130.4192
Ours(−history)0.53450.54750.37740.4182
AuthorsOpenReview ↗

Reproducibility: Response on Experimental Configurations and Curve Protocol

Reproducibility: Response on Experimental Configurations and Curve Protocol

You asked about the "manifest/budget unavailable" statement in the appendix and whether the budgets across methods are matched. Let us first clarify: the "unavailable" statement in the appendix was a mistake in an intermediate operation while using an agent to organize the archive (the archiving script omitted these files) that we failed to catch — the data does exist. The original run directories are fully preserved, and all configurations can be recovered.

We commit to releasing everything: the parameter configuration of every experiment (optimizer manifests, proposer models and configs, minibatch definitions, 1-slot/2-slot settings, rollout and token budgets, baseline launch parameters), the framework code, dataset sampling and splits (task ID lists), system artifacts (final skill/harness/pipeline), etc.

Dataset splits:

BenchmarkSplit
AppWorld (v0.2.0.dev0)train pool 147 (official train+dev); val 200 (test_normal 100 + test_challenge 100; seed 42, stratified by GLM-5 empty-skill score: ~10% solved + ~90% failed, actually 23 solved + 177 failed); held-out 385 (the remaining 68 + 317, never used in optimization)
TerminalBench 2.189 tasks, no separate val (consistent with the AHE/Meta-Harness protocol: optimization and evaluation use the same task set; in-sample protocol, already noted in the paper)
MuSiQue (4-hop subset)train 300 (official train 4-hop) / val 100 (official dev 4-hop) / test 300 (official dev 4-hop, excluding the 100 val ids); the query ids and doc ids of the three splits are mutually disjoint (the official test set has no annotations; the carving procedure is described below)

AppWorld split rationale: we did not use the official 90 train as train and 57 dev as val, because the data volume is too small: our method reaches an average score of 0.965 on the 57 dev items (out of a maximum of 1) in about 1 hour, making it very hard for the effect of optimization changes to show. In addition, the paper's "Why top-N truncation" section analyzes a degenerate behavior of the GEPA-style per-instance Pareto selection: when candidates are probabilistic systems, the dominance probability decays exponentially with the validation dimension D, and the frontier degenerates to the entire population — to observe this phenomenon, val must be large enough and must contain a certain number of non-trivial questions. Moreover, under the official 90 train / 57 dev recipe, the final results also lack good discriminative power. We therefore adopted a larger train pool and a stratified val of 200 (test_normal 100 + test_challenge 100; stratified by GLM-5 empty-skill score, targeting ~10% solved (actually 23 solved + 177 failed); seed 42).

MuSiQue split rationale: the official musique_full_v1.0_test only provides id / paragraphs / question, without answer or is_supporting, so nDCG@10 / Recall@10 cannot be computed; val/test can therefore only be carved from the annotated official dev: val is 100 items of dev 4-hop, and test is 300 items taken from dev 4-hop after excluding val. The three splits are mutually disjoint at the query-id and doc-id levels; our usage protocol is: val is used only for development-time tuning and candidate selection and does not participate in the final evaluation; MuSiQue test results should be read as a comparison on unseen queries from the same distribution; all methods share the same split and evaluator, so comparisons between methods are unaffected.

AuthorsOpenReview ↗

Reproducibility(continue)

Parameter settings for each method (the full version is released with the public package):

MethodBenchmarkParameters
OursAppWorldsystem GLM-5 (max_steps=50, system API concurrency 25), evaluator concurrency 20, proposer concurrency 10 (GLM-5.1, context 170k), max_candidate_num=3, batch_size=4, epochs=3, max_iterations=1000, budget 40h. 1-slot (num_proposals=1): min_improvement_per_batch=3.0; 2-slot (num_proposals=2): min_improvement_per_batch=2.0; all other parameters are identical across the two configurations (max_reflection_iterations = 2 for both)
Ours / Ours(−Mara Chain)TB2.1system GLM-5, system API concurrency 4, evaluator concurrency 4, proposer concurrency 20, max_candidate_num=1, batch_size=4, budget 100h. num_proposals=1; +Mara Chain: max_reflection_iterations=5; −Mara Chain: max_reflection_iterations=0
Ours / Ours(−Mara Chain)MuSiQuenum_proposals=2 (2-slot), batch_size=6, min_improvement_per_batch=2.0, max_reflection_iterations=3; evaluator concurrency 8, proposer concurrency 10, max_candidate_num=3; budget 20h. Models inside the optimized pipeline: BAAI/bge-m3, BAAI/bge-reranker-v2-m3, GLM-5.2
Ours(−Mara Chain)AppWorldmax_reflection_iterations=0, min_improvement_per_batch disabled, everything else identical to the corresponding configuration
GEPAAppWorldopen-source gepa engine + our own adapter; system GLM-5 / reflection GLM-5.1; temperature 0.5 for both system and reflection; system API concurrency 25, evaluator concurrency 20; Pareto selection, hybrid frontier; reflection minibatch=3; merge (max 5, val overlap floor 5); skip-perfect-score; max refinements=1; max metric calls 2,000,000; seed 0; reflection max tokens 64000; budget 40h
GEPAMuSiQueoptimized pipeline identical to Ours (bge-m3, bge-reranker-v2-m3, GLM-5.2); open-source gepa engine, Pareto selection, merge (MAX_MERGE_INVOCATIONS=5); reflection GLM-5.1 (streaming, 32000 tokens); minibatch=3; generate concurrency 8; budget 20h
ACEAppWorldopen-source ACE + our own adapter (Reflector→Curator→apply); system GLM-5, reflector/curator GLM-5.1; system temperature 0.5, reflector 0.0, curator 0.3; system API concurrency 25, evaluator concurrency 20; reflect workers 6; reflector/curator max tokens 32767; max reflector rounds 5; playbook token budget 80000; budget 40h
SkillOpt-LiteAppWorldsystem GLM-5 (temperature 0.5), optimizer GLM-5.1 (CLI, no explicit temperature); system API concurrency 25, evaluator concurrency 20; max_steps 50; max_completion_tokens 64000; batch_size 40; minibatch 8; num_epochs unlimited (truncated by the 40h budget); budget 40h
AHETB2.1system GLM-5, analyzer/improve GLM-5.1 (max tokens 64000, temperature 0.2); max iterations 10, target pass rate 0.95, pass@k with k=2, evaluator concurrency 4, auto-rollback enabled, evolve-agent (the NexAU mutation agent) enabled, explore-agent (knowledge injection) not effective; budget 100h. The base model is GLM-5 (GPT-5.4 in the original paper), so absolute scores are not directly comparable with the original paper; only same-environment comparisons are valid
Meta-HarnessTB2.1system GLM-5, proposer GLM-5.1 (Claude Code style), budget 100h. The original paper's proposer is Claude Code × Opus-4.6 (max reasoning) and its system is Claude Opus 4.6 / Haiku 4.5 — the models differ, so absolute scores are not directly comparable with the original paper; only same-environment comparisons are valid

The AppWorld curve and the 65.5% rollout reduction protocol: the rollout definition is unified across methods (1 rollout = executing 1 data instance; train and val rollouts are counted separately, and the curve's x-axis is cumulative train rollouts, excluding val), and the y-axis is the best-so-far score on val; the 65.5% reduction means that upon reaching the target score (val best = 0.8, the horizontal line in the figure), Ours consumes 65.5% fewer train rollouts than GEPA (ACE and SkillOpt-Lite do not reach that target within the budget, consistent with the paper's figure caption). The rollout counts of every method were verified in each system's own source code.

Budget matching: all compared methods use the same wall-clock budget on the same benchmark (AppWorld 40h, TB 100h, MuSiQue 20h), with the same system models and evaluation configuration.

Scoring and evaluation protocol: AppWorld = TGC (per-task pass/fail, same evaluator as the final table); optimization and final testing use the same evaluator; test results on the three benchmarks are mean@3 (fixed artifact evaluated 3 times and averaged); val is the first-run result (pass@1); the optimization process is a single run.

Comments on the paper

1 comment

Dear reviewers,

The author-reviewer discussion period is approaching the deadline. Please read through the author's rebuttals and update your reviews. Thank you for your time and work.

Best Regards, AC.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.