acceptodds
Under review as a conference paper at ICLR 2027

Meta-Reflective Evolution Effectively Improves Production AI Systems

Abstract

Large Language Model (LLM)-guided evolutionary optimization has successfully improved prompts, programs, and even entire AI systems. Most optimization methods select a candidate from the search frontier, then mutate it in one LLM pass based only on its failures. This focus on the frontier can stall on production AI systems, where isolated failures reveal little about complex, hidden interactions. We introduce CoCoEvolve, an evolutionary optimizer built around what we call meta-reflection, wherein an agentic mutator selectively retrieves evidence from the entire evolutionary history and probes the live system before verifying and proposing mutations. We apply CoCoEvolve to optimize three classes of production AI systems: data agents, LLM-powered data pipelines, and data engineering workflows. On DABStep, CoCoEvolve tops the public leaderboard, reaching 92.8% held-out hard accuracy compared to 78.4% for Meta-Harness and 30.4% for GEPA under matched conditions. Across six LLM data pipeline workloads, it outperforms every evaluated baseline. On data-eng-bench, CoCoEvolve lifts test-set solve rate from 59.1% to 77.4%, past the best baseline of 73.6%. It also has 11pp higher accuracy than GEPA on HotpotQA while remaining competitive with specialized optimizers on established mathematical and systems benchmarks. Matched ablations confirm that each component of CoCoEvolve contributes to these gains, demonstrating the value of meta-reflection for optimizing production AI systems.

Primary area:
foundation or frontier models, including LLMs
Keywords:
evolutionary algorithms, genetic, optimization, code optimization, agent discovery, agent optimization, large language models, natural language, reflection, meta-reflection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_t2TNOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CoCoEvolve, an evolutionary optimizer combining a strategy ledger, on-demand retrieval of search history, and an agentic mutator that investigates and tests candidate changes. Evaluations span data agents, SQL pipelines, data engineering, and prompt, mathematical, and systems optimization.

Critical strengths
  • The evaluation is unusually broad and includes substantial held-out improvements on practically relevant workloads.
  • Mechanism ablations and detailed execution traces provide useful evidence about how historical retrieval and pre-commit testing support structural improvements beyond prompt editing.
Critical weaknesses
  • Search variability substantially weakens the component-level conclusions. Appendix E.3 reports 114/150 test successes for a second full DABStep run, versus 137/150 for the flagship. Most primary ablations use one search and repeatedly reuse the flagship. Moreover, ledger-only underperforms parent-only on DABStep.
  • Baseline comparability is incomplete. Appendix G.1 restricts GEPA to instruction edits while CoCoEvolve can modify stored procedures and dynamic tables. Equal iteration counts also do not establish equal token, probing, or evaluation costs.
  • Appendix I.1 states that adherence tests were not used uniformly in data-engineering runs. This leaves a potentially consequential experimental condition insufficiently controlled or documented.
Questions for authors
  • Do the main improvements and ablation rankings persist across independent searches under matched total computational budgets?
  • Can the authors provide comparisons with equally expressive editable artifacts and specify adherence-test usage for every data-engineering result?
2 comments

We thank the reviewer for the thoughtful feedback. We address the questions and concerns below:

Q1 / W1 / W2

Please refer to the general response.

Q2 / W2 / W3: Comparability of Editable Artifacts and Adherence-Test Usage

  • Equally expressive editable artifacts: Across Data Engineering, AI Functions, and all mathematical and systems benchmarks, every baseline (GEPA, Meta-Harness, AdaEvolve, OpenEvolve) and CoCoEvolve optimize the exact same editable artifact surface. On DABStep, all methods initialize from the identical Cortex Agent specification P0P_0; Meta-Harness (118/150 with mh_tools.py; 115/150 and 83/150 vanilla) and CoCoEvolve - as well as CoCoEvolve's internal ablation arms A1 (static one-shot mutator, 47/150 test), A2 (interactive agentic mutator without pre-commit test_draft, 98/150 test), the history ablations, and C1--C2 (Appendices E.2--E.4) - have unrestricted access to edit the entire specification, including instructions, tool schemas, stored procedures, and dynamic tables. While external GEPA is by design a reflective prompt/instruction optimizer, our internal A1/A2, the no-meta-reflection arm, and C1 controls and Meta-Harness provide direct comparisons on the exact same full-specification editing surface.
  • Adherence-test usage in Data Engineering: Across all methods and runs in Table 3, adherence-test compilation and availability were enabled identically. The statement in Appendix I.1 refers strictly to the downstream fixed Harbor coding agent's autonomous runtime behavior inside the container (i.e., whether the solver chose to execute pytest test_kg.py during a given task attempt), not to any difference in supplied inputs or harness permissions across methods. No method received assistance unavailable to the others.

I thank the authors for their detailed responses, which have largely addressed my concerns. I remain positive about the paper and maintain my original score.

Reviewer_MZm9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CoCoEvolve combines a compact search-history ledger, on-demand queries over prior programs and traces, and an interactive coding-agent mutator that can inspect and test changes before committing them. It also introduces task targeting and complementary-coverage parent selection for relevant settings. The paper evaluates agent specifications, SQL-based AI functions, data-engineering pipelines, and mathematical, systems, and prompt benchmarks. Reported gains include 92.8% versus 78.4% held-out hard accuracy against Meta-Harness on DABStep, and 79.3% versus 68.3% test accuracy against GEPA on HotpotQA. Ablations investigate mutation execution, history access, targeting, and parent selection.

Critical strengths
  1. A useful, operational approach to history-aware optimization. The ledger and structured archive queries make earlier solutions accessible without placing the entire history in context. Combined with live testing, this supports substantive repairs, such as the fee-computation changes in §3.1 and Appendix G.4. The tool interfaces and worked traces make the contribution concrete.

  2. Broad evaluation with informative controls. The paper tests several executable-artifact settings and includes held-out evaluation, an interactive Meta-Harness baseline with self-testing tools (Appendix G.1), and component ablations. The authors also disclose unfavorable results, including the weaker repeated DABStep run and backbone-dependent math/system results. This provides useful evidence beyond selected success stories.

Critical weaknesses
  1. The reliability of the DABStep history-access gain remains uncertain. Appendix E.3 (pp. 21–22) reports 137/150 test solves for the flagship full-history configuration but 114/150 for a second run with the same configuration and model. These outcomes straddle the reported parent-only (116/150) and Meta-Harness (118/150) results, which are also individual runs. The authors appropriately disclose this variation. Reusing the flagship as the DABStep full arm in Tables 10, 12, and 13 is a valid shared-control design, but adds no independent evidence about search variability. The current results therefore do not establish a stable advantage across searches. Repeating the full-history and key control configurations and reporting all outcomes would strengthen this central claim. This concern does not negate the observed flagship improvement or the multi-trial evidence on other tasks.

  2. The data-engineering comparison leaves an important inference-time condition unspecified. Appendix I.1 (pp. 40–41) describes adherence tests as feedback for the downstream solver, then states that they were not used in all runs. It does not map this qualification to the methods/results in Table 3. Thus, it is unclear whether test availability was matched, or whether “not used” refers to the solver's behavior rather than different supplied inputs. This matters for interpreting 77.4% versus 73.6% on 53 test tasks, approximately a two-task margin; that margin alone does not establish statistical insignificance. Please report test availability and usage per method/run. If supplied assistance differed, a common-condition comparison is needed to attribute the difference to the optimized pipelines. If inputs were already matched, clarification may resolve this concern.

Questions for authors
  1. Can you report all independent DABStep searches and the rule used to select the flagship run, and provide repeated full-history, parent-only, and Meta-Harness comparisons? Does the advantage persist across searches rather than repeated evaluations of one champion?

  2. In Table 3, were adherence tests made available identically to every method? Does “not used for all runs” mean they were withheld in some runs, or that the solver did not use available tests? If availability differed, does the ordering persist with identical assistance?

1 comment

We thank the reviewer for the thoughtful feedback. We address the questions and concerns below:

Q1 / W1

Please refer to the general response for independent and repeated DABStep searches. Regarding how the flagship run was selected: in the submitted paper, the 137/150 flagship run was certainly not cherry-picked from a pool of random seeds - we only ran the configuration once due to cost. We replicate the results in the general response.

Q2 / W2: Data-Engineering Adherence-Test Availability and Usage in Table 3

  • Identical availability across all methods: Yes, adherence tests were enabled and made available identically to every method and run in Table 3 (ENABLE_ADHERENCE_TESTS=1). Every evaluated pipeline had the same opportunity to compile and supply test_kg.py alongside KG.md, and no method received additional test assistance or inputs unavailable to the others.
  • Meaning of "not used in all runs" (Appendix I.1): This refers solely to the autonomous runtime behavior of the fixed downstream Harbor coding agent inside the container (i.e., whether the solver chose to invoke pytest test_kg.py during a given task trial before submitting its dbt models), and not to adherence tests being withheld in some methods or runs. Because the inference-time protocol and supplied inputs were already matched across every row of Table 3, the reported ordering directly reflects the quality of the optimized pipelines.
Reviewer_xQg9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CoCoEvolve, an evolutionary optimization framework that combines structured historical retrieval with agentic mutation. The mutator can inspect previous search trajectories, interact with live environments, and validate modifications before committing them. The framework also introduces complementary-coverage parent selection and similarity-guided task targeting. Experiments demonstrate strong improvements across production AI systems and several established optimization benchmarks.

Critical strengths

1. Strong empirical results across diverse tasks.

CoCoEvolve demonstrates substantial improvements on challenging benchmarks, particularly DABStep, where it significantly outperforms Meta-Harness and GEPA. The evaluation spans data agents, LLM-powered pipelines, data engineering, and mathematical optimization, demonstrating the framework's applicability to different optimization settings.

2. Practical and well-motivated framework design.

Combining historical retrieval, environment interaction, and self-validation provides a useful approach to diagnosing complex failures that cannot be resolved from local feedback alone. The structured history-query interface also helps recover previous discoveries, identify regressions, and avoid repeated exploration of unsuccessful strategies.

3. Informative case studies and ablations.

The paper provides detailed examples showing how agentic mutation can discover structural improvements beyond prompt optimization, such as replacing error-prone computations with deterministic tools. Component ablations further investigate the contributions of history access, self-validation, and task selection.

Critical weaknesses

1. Limited novelty relative to existing agentic evolution frameworks

Prior work such as Meta-Harness and CORAL already employs autonomous coding agents, iterative experimentation, and access to historical information. While CoCoEvolve introduces structured history retrieval, its incremental contribution over these approaches is insufficiently established. In particular, the paper neither discusses CORAL nor compares against it on overlapping benchmarks.

2. Limited validation of structural system modifications

CoCoEvolve allows simultaneous modifications to agent instructions, tools, and system architecture. Such changes may introduce cross-component dependencies and regressions that are not adequately captured by task-level evaluations or semantic similarity-based sampling. Although the framework includes regression testing, its ability to reliably validate structural changes remains unclear. The Meta-Harness failure analysis highlights structural consistency issues, but the paper does not demonstrate that CoCoEvolve systematically prevents them.

3. Limited experimental robustness and compute-efficiency analysis

Several key ablations rely on single search runs. Notably, two full-configuration DABStep runs achieve substantially different test accuracies (137/150 versus 114/150), with the latter performing slightly below the parent-only baseline. This raises concerns about the stability of the claimed improvements. Additionally, while the paper controls iteration budgets and several evaluation settings, agentic mutation may incur substantially different inference and tool-execution costs. Multi-seed evaluations and compute-normalized comparisons would strengthen the empirical conclusions.

Questions for authors

Q1. Novelty relative to existing agentic evolution frameworks. Both Meta-Harness and CORAL already support multi-step coding-agent exploration, iterative experimentation, and access to historical information. What fundamentally distinguishes CoCoEvolve from these approaches, beyond its structured history-query interface and search policies?

Q2. Benefits of structured historical retrieval. Meta-Harness provides filesystem-based access to previous experiments, while CORAL uses persistent shared memory for knowledge reuse. What additional benefits does CoCoEvolve's strategy ledger and structured history retrieval provide over these simpler memory mechanisms? Could the authors isolate these benefits through controlled comparisons?

Q3. Reliability of system-level evaluation. CoCoEvolve allows simultaneous modifications to agent instructions, tools, and system architecture. How does its task-level evaluation and similarity-guided sampling reliably capture cross-component dependencies and regressions, particularly when structural changes affect seemingly unrelated tasks?

Q4. Robustness and computational efficiency. What explains the substantial performance variation between the two full-configuration DABStep runs (137/150 versus 114/150)? Could the authors provide multi-seed evaluations and comparisons under matched total inference and tool-execution budgets?

2 comments
AuthorsOpenReview ↗

Author Response (part 1 of 2)

We thank the reviewer for the thoughtful feedback. We address the questions and concerns below.

Q4 / W3

Please refer to the general response.

Q1 / W1: Novelty Relative to Existing Agentic Evolution Frameworks

We thank the reviewer for pointing us to CORAL (Qu et al., COLM 2026) and will add a dedicated comparison in Sec. 4. While CORAL and Meta-Harness also pair a coding agent with past attempts, treating "a coding agent with history access" as a commodity overlooks a central bottleneck of agentic evolution: how trajectory evidence is curated, queried, and targeted as the historical archive scales.

Even in the single-agent regime, foundational work such as Agentic Context Engineering (ACE; Zhang et al., ICLR 2026) is devoted entirely to curating evolving context playbooks because unstructured context updates (e.g., brevity bias and context collapse) degrade reasoning and inflate cost.

In a similar vein, our central thesis with meta reflection is that careful curation of evolutionary context is similarly crucial in improving the quality of mutations, especially in more complex applications. In contrast, both CORAL and Meta-Harness do not explicitly curate context and choose to delegate context-gathering to long-running agent sessions. We first highlight very significant methodological differences to bring out technical novelty, and then compare quality/cost across methods to highlight the major impact that these techniques have on the quality and cost of optimization.

  • Dual-channel context curation (ct=Curate(Ht−1),;Et=Augment(E0;Pt′,Ht−1)\mathbf{c}_t = \mathrm{Curate}(\mathcal{H}_{t-1}),\\; \mathcal{E}_t = \mathrm{Augment}(\mathcal{E}_0; P'_t, \mathcal{H}_{t-1}), Sec. 2.2). Meta-Harness exposes raw filesystem directories of prior traces, while CORAL shares a filesystem of scalar-scored attempt records and self-authored markdown notes/skills - where CORAL's authors note that agents "may also forget to consult and contribute to shared persistent memory" (CORAL Sec. 3.3) and access notes/skills in only 7% of attempts across standard math and systems tasks (CORAL Table 4). In contrast, CoCoEvolve explicitly curates and injects a bounded strategy ledger ct\mathbf{c}_t as mandatory context, while complementing it with indexed on-demand query tools via evo.py (Et\mathcal{E}_t: per-task lineage, donor diffs, and co-regression memory, plus live validation in E0\mathcal{E}_0; App. D.2, Table 6). In our controlled 2×22\times 2 history ablations, both channels contribute and their combination is best across both Signal Processing (0.661→0.707→0.749→0.7680.661 \to 0.707 \to 0.749 \to \mathbf{0.768}; App. E.3) and our repaired multi-seed DABStep runs (84.0% without ledger and query tools →\to 87.7% without ledger →\to 88.3% without query tools →\to 89.8% ± 1.4% Full CoCoEvolve; Q2).

  • Kernel-surrogate observation curation and ℓp\ell_p coverage selection (Sec. 2.3). Neither work formalizes which instances to score, target, or re-check: CORAL evaluates only single-score objectives (CORAL Sec. 4.1), while Meta-Harness scores every candidate on a fixed evaluation set and identifies evaluation as its main bottleneck (Meta-Harness Sec. 3, 4.1). CoCoEvolve models instance-level performance via a Nadaraya--Watson kernel surrogate s^(P,x)\hat{s}(P,x) over a semantic similarity graph K(x,x′)K(x,x'), directing mutation and sparse evaluation toward high-transfer failures via information gain IG(x)=(1−s^(P,x))δ(x)∑x′K(x,x′)\mathrm{IG}(x) = (1 - \hat{s}(P,x))\delta(x)\sum_{x'} K(x,x'), re-testing only the most at-risk solved neighbors, and pairing this with ℓp\ell_p complementary-coverage parent selection (App. D.5).

As shown in the general response above, these mechanisms drive both search quality and cost efficiency: without sparse observation curation and bounded historical context, Meta-Harness and CORAL exhaust the USD 1,500 search cap in just 11--15 and 9--10 iterations, whereas Full CoCoEvolve completes all 100 iterations under USD 1,500 all-in (with its sparse in-loop surrogate staying within ≤3\le 3 pp of the full 300-question training score; Q3). Even when Meta-Harness is granted our live warehouse self-test commands and an unmatched 100 iterations (118/150, App. G.1), lacking ct\mathbf{c}_t, Et\mathcal{E}_t history queries, and kernel targeting causes it to stall for its final 47 iterations on repeated dead ends (App. G.3), while ablating CoCoEvolve's kernel targeting to uniform sampling collapses test accuracy from 91.3% to 28.0% (42/150, App. E.4).

AuthorsOpenReview ↗

Author Response (part 2 of 2)

Q2: Benefits of the Strategy Ledger and Structured History Retrieval

Our controlled 2×22\times 2 ablation (holding backbone, seed, 100-iteration budget, parent policy, and self-validation tools fixed) isolates four arms: 1) CoCoEvolve without ledger and query tools, 2) without ledger, 3) without query tools, and 4) Full CoCoEvolve.

  • Monotonic gains across Signal Processing and DABStep: Once the remote launcher bug on the paper's B1--B3 DABStep batch is fixed (see the general response), DABStep matches the monotonic progression of Signal Processing (0.661→0.707→0.749→0.7680.661 \to 0.707 \to 0.749 \to \mathbf{0.768}): held-out test accuracy rises from 84.0% ± 3.8% (without ledger and query tools) to 87.7% ± 6.1% (without ledger) and 88.3% ± 6.1% (without query tools), reaching 89.8% ± 1.4% (91.7% ± 1.2% hard) in Full CoCoEvolve.
  • Why the two channels are complementary - and why combining them collapses variance: The injected ledger ct\mathbf{c}_t provides mandatory high-level orientation (which strategy families are stagnating vs. succeeding), preventing repeated local dead ends, while query tools Et\mathcal{E}_t verify exact donor diffs and per-question failure traces on demand. On 300-question DABStep, either channel alone lifts mean accuracy by +3.7 to +4.3 pp over no meta-reflection, but leaves higher variance (±6.1% test, ±9.1% hard) because ledger summaries without code inspection can over-generalize, and query tools without a global ledger can miss macro plateaus. Pairing both in Full CoCoEvolve yields the highest accuracy and cuts test variance by over 4×4\times (±1.4% test, ±1.2% hard) at comparable cost.

Why split history into ct\mathbf{c}_t and Et\mathcal{E}_t rather than placing raw history into the prompt or filesystem? Prompt-stuffing full histories bloats context and biases the mutator toward recent local edits, whereas raw filesystem traces or unindexed notes (Meta-Harness and CORAL) waste turns and budget browsing directories or overlooking prior memory (66.0% and 54.3% test). Our 2×22\times 2 ablation isolates ct\mathbf{c}_t and Et\mathcal{E}_t within CoCoEvolve, while Meta-Harness and CORAL test filesystem and shared-note memory at the system level; we will note in the limitations that we do not run a separate in-harness raw-filesystem arm under CoCoEvolve's exact search policy.

Q3 / W2: Reliability of System-Level Evaluation and Cross-Component Regression Control

While no heuristic search guarantees zero regressions from arbitrary structural edits (which we will state in the limitations), four coupled mechanisms in CoCoEvolve directly curb cross-component regressions:

  1. End-to-end pre-commit execution: Rather than guessing cross-component impact across instructions, SQL procedures, and schemas, the mutator executes the modified system end to end before committing (evo_sweep.py on math/systems; live scratch-agent deployment via test_draft/check_deploy on DABStep; App. D.2). Removing pre-commit testing triples the regression rate on Heilbronn Convex (∼4\sim 4\\% \to \sim 12.5\\%) and drops DABStep test accuracy from 91.3% to 65.3% (98/150; App. E.2, Table 10).
  2. Risk-prioritized neighbor gating: Every candidate is gated on its target question and on the parent's solved questions that are semantically closest (most at risk of co-regression) on the similarity graph.
  3. Co-regression memory (get_question_memory): Query tools expose which questions regressed when specific components were edited, letting the mutator check known failure modes before committing.
  4. Manifold-anchored targeting: Anchoring mutations to coherent similarity-graph neighborhoods sharply curtails collateral regressions: holding the regression sampler fixed (App. E.4, Table 14), similarity-guided targeting (C3) gains 189 training questions with only 4/300 regressions vs. seed, compared to 35/300 under uniform targeting (C1) and 16/300 under a shuffled graph (C2).

Finally, to ensure sparse sampling never hides distant regressions, every champion is scored post-hoc on the full 300-question training and 150-question held-out test pools; across our 100-iteration runs, the in-loop sparse Nadaraya--Watson surrogate tracks full-pool training accuracy within ≤3\le 3 pp (93.3--95.3% vs. 0.943--0.977). For the Cortex Analyst semantic-view failure mode (App. G.3), CoCoEvolve uses no custom DDL verifier unavailable to Meta-Harness; curated history and query tools simply prevent re-entering the same dead-end edit across 60+ iterations.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely appreciate the reviewers' time, constructive engagement, and insightful critiques. We first address the shared concerns regarding DABStep search stability, multi-seed replication, and compute-normalized baseline comparisons below, and then respond to each reviewer's specific questions in their individual sections.

New Experiments

In light of the concerns regarding DABStep search variability, multi-seed replication, and compute-normalized baseline comparisons, we ran twelve new independent DABStep searches: two Full CoCoEvolve repeats, two repeats each of the three history ablations (CoCoEvolve without ledger, without query tools, and without ledger and query tools), two faithful Meta-Harness searches, and two CORAL searches. All runs were metered via Anthropic's standard pricing tables (metering all inference spent on both mutation and agent evaluation) and stopped at 100 iterations or a USD 1,500 search budget cap, whichever came first (checked before launching each iteration, so the final iteration can slightly exceed USD 1,500).

MethodRuns AveragedMean TestMean HardMean Spend (USD)Mean Iters
Full CoCoEvolve1 paper + 2 new89.8% ± 1.4%91.7% ± 1.2%1,343100
CoCoEvolve without query tools2 new88.3% ± 6.1%90.4% ± 9.1%1,455100
CoCoEvolve without ledger2 new87.7% ± 6.1%89.6% ± 9.1%1,123100
CoCoEvolve without ledger and query tools2 new84.0% ± 3.8%85.6% ± 4.5%1,240100
Meta-Harness2 new66.0% ± 15.1%62.8% ± 18.7%1,52613
CORAL2 new54.3% ± 6.1%47.6% ± 7.4%1,5669.5

On test and averaged across all runs, Full CoCoEvolve gains +23.8 pp over Meta-Harness, +35.5 pp over CORAL, and +5.8 pp over CoCoEvolve without ledger and query tools (+2.1 pp over CoCoEvolve without ledger and +1.4 pp over CoCoEvolve without query tools), while also achieving the lowest across-run variance of any method (±1.4%). In addition, CoCoEvolve is roughly 10X more cost-efficient per iteration than both baselines. CoCoEvolve beats Meta-Harness and CORAL very significantly in both quality and per-iteration cost while also exhibiting the lowest variance across runs. These improvements in quality and cost are the direct result of the novel technical contributions of CoCoEvolve as we elaborate on further in response to the first reviewer's questions w.r.t. results of ablation experiments.

Root Cause of the 137/150 vs. 114/150 Discrepancy

While our flagship 137/150 run executed locally restricting the mutator's tools to our meta-reflection query tools, we offloaded the subsequent B1--B4 ablation batch (including the 114/150 Full CoCoEvolve run) to a remote cluster runner for parallel execution; a post-submission audit revealed that this remote runner was misconfigured with unrestricted shell access. We apologize for the confusion this caused and our new experiments fully re-run the affected runs. Although the mutator issued nearly identical total tool calls per iteration (33.7 vs. 32.3; App. E.3), its share of calls to our meta-reflection query tools dropped from 95% (3,066) to 53% (1,798, with 47% diverted to ad hoc shell commands). Serendipitously, this bug acts as a natural intervention reinforcing our core thesis: when the mutator drifts from our curated query tools into unconstrained shell browsing - the exact interaction paradigm that Meta-Harness and CORAL rely on by design - search quality drops sharply, helping explain the performance gap to both baselines. As shown in the table above, once this remote launcher bug is fixed, our new repeats consistently recover the flagship performance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.