acceptodds
Withdrawn from ICLR 2027

Beyond Chain Selection: Compiling Proposal Archives into Frozen Agent Harnesses

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Agent harness evolution couples two distinct operations: proposing useful edits and selecting edits that generalize. We study whether useful proposal content can be recovered even when chain-level selection fails. We introduce cross-chain proposal compilation, which constructs a single frozen harness from logged edits with no additional model calls at compile time. We instantiate this idea in two forms: prose compilation assembles proposal text into instructions, and typed compilation selects recurrent, executable mechanisms with development-ranked implementations. On a primary pool of 1,007 BFCL v4 tasks, union prose, recurrent prose, and typed consensus improve paired held-out accuracy by +2.29, +2.02, and +1.65 percentage points. All three paired intervals exclude zero. A fresh five-arm follow-up on 400 additional tasks finds smaller effects whose intervals include zero, indicating limited category transfer. In a separate held-out comparison, single-chain finalists underperform the baseline despite positive development gains. These results support proposal-level reuse beyond chain-level selection, with bounded evidence for transfer and inference cost isolated as a separate outcome.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Self-Evolution

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_YMSrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Logged edit proposals from harness evolution chains can be reused even when chain-level selection fails

Two ways to build one frozen harness: prose compilation (union or recurrent text) and typed consensus (mechanisms proposed by at least half of 7 chains, inactive components removed, best development-ranked implementation copied)

On 1,007 held-out BFCL v4 single-turn tasks with one frozen task model, gains over bare are about +2.3, +2.0 and +1.7 points

A fresh 400-task follow-up shows smaller effects, and every interval includes zero

Single-chain finalists drop below bare on held-out tasks despite development gains; also covers noise analysis, inactive-edit checks, inference cost, pre-registration and full log release

Critical strengths

transparent measurement - pre-registration, paired intervals, minimum detectable effect, machine-graded chain exclusion, sensitivity corpus, repeated runs and a full log release

negative results are reported openly (matched prose, fresh follow-up, multi-turn floor), which makes the work easy to audit

Useful warning for self-improving agents - edits that never run still get accepted, development gains flip on held-out tasks, and 42.5% of decisions needed tie-break resampling

The released evolution logs, including rejected edits, could be a useful community resource

Critical weaknesses

Small effect - gains of about 2 points on an 80–92% baseline, all fresh follow-up intervals include zero, and the same artifact on the same tasks varies by 1.78 points across runs

Partly post hoc and weakly controlled - prose arms were registered as controls and only later called methods; prose results use an older archived bare run rather than one run at the same time; most primary tasks come from live_multiple, the same category the chains evolved on

Confounded comparisons - union prose uses a broader archive (ToolOrbit, MGSM, extra runs); recurrent prose is a single 716-character prompt; no best-chain-prompt, hand-written-prompt, or DSPy/GEPA baseline at the same compute budget

Very thin chain population - consensus built from 7 chains including 3 the authors themselves hard-exclude, only 2 clean-delta chains, and the "selection fails" claim rests on 2 Kimi finalists

Many inconsistencies - pool sizes 1,669 vs 1,567 vs 1,407, proposals 113 vs 165, MGSM −3.3 vs −3.89, the hybrid arm beating the "oracle upper bound," the C.2 verdict saying matched prose "retains the gain" when it doesn't, S6 with a positive interval but McNemar p=1, undefined gate G4, missing Table 6, a "three-model" ladder showing two, plus missing or duplicated appendix tables

Questions for authors

what is the McNemar p for typed consensus on the primary pool, does it pass your own registered rule, and were the prose arms rerun against a bare baseline done at the same time?

how does it compare with only the best chain's prompt, only the consensus system prompt, or a short hand-written function-calling prompt, and does max_steps of 2 differ from bare?

why include 3 hard-excluded chains in consensus, and what are the results with only clean chains?

please explain the inconsistent numbers, especially the C.2 verdict, the H4 hybrid-above-oracle result and S6, and say whether any headline numbers change

other prompt optimizers at the same budget, a second task model, and a multi-turn benchmark with enough statistical power

Reviewer_axTgOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies loops in which a meta-model proposes edits to an LLM agent's harness (the prompts, output-format rules, task decomposition and similar processors around a frozen model), evaluates each candidate on development tasks, and keeps or rejects it. Its thesis is that proposing and selecting are separable operations, that selection at the budgets these loops run at keeps noise and rejects real gains, and that the archive of every proposed edit, accepted or rejected, retains value that can be extracted without further model calls. The paper calls this extraction cross-chain proposal compilation.

Two forms of construction are evaluated. The typed one keeps mechanisms proposed by at least half of seven chains grown on one BFCL category, drops those that cannot execute, and copies parameters from the best development chain; the result is a short function-calling system prompt plus three formatting components. The prose one concatenates deduplicated instruction text from the archive into one system prompt, in a broad form (6,000 characters drawn from the logs of every environment the authors ran) and a narrow form (the text of the typed artifact).

On 1,007 held-out BFCL tasks the three artifacts score +2.29, +2.02 and +1.65 pp above a harness with an empty system prompt, each paired against an archived baseline from a separate run, with one evaluation per arm. A follow-up that re-ran all arms under a fresh common baseline on 400 tasks returns +0.4 to +0.7 pp with intervals including zero. The main text reads this as limited category transfer, but these are the same tasks on which the archived design had shown about +1.75 pp for the typed artifact, so the follow-up also measures how much of the primary effect is run-to-run variation, which the paper elsewhere puts near 1.8 pp. Separately, the final harnesses of two evolved chains score 4.6 pp below bare on 300 tasks despite positive development gains; those gains were measured at a resolution the paper puts near 13 pp, and no compiled artifact (assembled harness) was evaluated in that run. The analysis section quantifies evaluation noise, inactive edits and inference cost.

Critical strengths
  1. Separating proposal generation from acceptance, and treating the archive of rejected edits as an object worth evaluating, is a useful reframing. The paper names the parts that harness-evolution papers usually leave implicit: what the acceptance rule sees (a development delta at a stated resolution), what it cannot see (inactive components, run-to-run drift), and what the archive contains beyond the accepted history. The released logs with rejected edits and their rationales are, as the authors note, a resource that does not otherwise exist.
  2. The experiment details are faithfully reported with specific findings.
Critical weaknesses
  1. The performance gains are not significant. The reported gains are the size of the difference the paper itself measures between two runs of one fixed harness, and the only experiment that ran the artifacts and the bare harness together returns gains whose intervals include zero. Each primary-pool gain is computed by subtracting bare scores stored from an earlier run from artifact scores obtained in a later run, once per artifact. The paper measures how much one fixed harness moves between runs: 1.76 pp for one comparison repeated on stratum B, and a 1.78 pp spread across three runs of union prose on stratum A, of which the highest entered Table 1. In Section 5.2 the artifacts and bare were run together on 400 tasks; the gains fall to +0.4 to +0.7 pp with intervals including zero (Table 2). These are the same tasks on which the earlier method had given the typed artifact about +1.75 pp (Figure 2), so the drop is not category transfer, as the main text reads it, but the between-run difference.
  2. The evidence shows that instructions assembled from proposal text beat a harness with no instructions, not that the archive adds anything over a hand-written harness, including the seed one chain started from. The comparator has an empty system prompt (as stated in Appendix E.3.1). No compiled artifact is compared on the primary pool with any transcribed public harness, although the seed library contains four.
  3. The paper’s central contrast, that chain selection discarded value which compilation then recovered, rests on two separately measured results that were not evaluated on the same tasks against the same baseline. The first result is that two evolved chains end up 4.6 pp below bare on 300 tasks (Table 3). These chains accepted edits using 60 development tasks with one sample each. At that size the paper’s own calculation says differences below about 13 pp cannot be seen. The second result is that compiled artifacts score 1.7 to 2.3 pp above bare on 1,007 tasks (Table 1), measured in a different run against a different bare score.
Questions for authors
  1. How do the compiled artifacts compare on the primary pool with the transcribed SWE-agent and Codex harnesses, in particular with the seed of the SWE-seeded chain?
  2. Were any compiled artifacts and the single-chain finalists ever evaluated in one run against one baseline?
Reviewer_fcTYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies automated harness evolution, where a meta-model proposes edits to an agent harness and a chain accepts the edits that improve a development score. The authors observe that these chains often fail to generalize, and ask whether the proposed edits are still useful when chain-level selection fails. They propose cross-chain proposal compilation, which builds one frozen harness from the logged edits of many chains without new model calls. Prose compilation concatenates deduplicated instruction text from the proposals. Typed compilation keeps mechanisms that recur in at least half of the chains and copies their best development implementation. On held-out BFCL function-calling tasks with one task model, the compiled harnesses improve paired accuracy by about two points, while the single-chain finalists score below the bare harness. The paper also analyzes evaluation noise, inactive edits and inference cost in the evolution loop.

Critical strengths
  1. The statistical practice is careful. The held-out evaluation protocol is registered in advance, uses clean strata with paired intervals, and reports null results, such as limited transfer to new categories.
  2. Separating proposal generation from selection is useful lens for harness evolution. The finding that accepted edits can be inert, and that chain finalists can lose to the seed on held-out tasks despite positive development gains, is a concrete warning for this line of work.
Critical weaknesses
  1. The held-out gain is mostly measured on the same category the edits were tuned on. The chains are developed for one BFCL category, and about of the primary held-out pool comes from that same category. Held-out here only means unseen tasks, not unseen task types. On new categories, none of the compiled variants shows a significant gain. So the headline gain likely overstates how much the compiled harness helps beyond the development distribution.
  2. The difference from prior prompt and context optimization is unclear. If the compiled harness only helps on one task type, it plays the same role as task-specific prompt or context optimization, such as GEPA, DSPy or ACE, which also build an improved prompt or context from task feedback. The paper cites these methods but does not compare with them. Without a comparison under a matched budget, it is not clear what advantage compilation has over prior work.
Questions for authors
  1. Do the compiled harness edits generalize beyond the category they were developed on? The edits are selected on one development category, and most of the held-out pool comes from the same category. How does the gain change on task categories that were never used for development? Please report the gain on these categories separately from the gain on the development category.
  2. If the compiled harness only helps on one task type, how is it different from task-specific prompt or context optimization? For example, ACE keeps an evolving playbook of itemized strategies, tool usage and code, updated incrementally, and GEPA evolves prompts from reflective feedback. Both are cited but not compared. Please clarify the difference, and compare with these methods as baselines under a matched total budget, including the cost of generating the chains.
Reviewer_NibbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether logged self-evolution proposals retain any value once chain-level accept/reject selection fails to generalize. The answer is mixed: the dev-selected single chains and checklist aggregation that won on dev lose points on held-out tasks, but deterministically compiling logged edits into a frozen harness — with no LLM call at compile time — yields a small pre-registered gain on BFCL v4. That gain sits near the benchmark's resolution floor, and a fresh 400-task follow-up on new categories does not independently confirm it.

Critical strengths
  1. Arms, strata, decision rule and planned MDE were fixed before new evaluations (A.1), and non-significant results are reported as-is rather than pooled across strata until something appears.
  2. The chain-selection failure is measured inside the paper rather than cited from prior work. Two single-chain finalists that gained +6.67 pp on dev lose 4.56–4.67 pp on 300 held-out tasks, and checklist aggregation loses 4.37 pp on the same pool (C.1.11).
  3. The compiled artifact is deterministic and auditable — a JSON spec chosen by ≥50% recurrence across 7 chains, with no LLM call at compile time and a released replay check that verifies every archived version (C.5.1, E.1).
Critical weaknesses
  1. The primary effect is small and the fresh-category follow-up does not independently confirm it. Typed consensus is +1.65 pp [+0.58, +2.74] on 1,007 tasks, close to the 2.16 pp planning MDE; on the new 400-task categories all arms' CIs cross zero (C.3). The "archives contain recoverable value" claim therefore rests mainly on the primary live_multiple / parallel_multiple pool.
  2. The consensus vote may count shared proposals rather than independent replications. The artifact is built from 7 chains on live_multiple (3 hard-excluded), all sharing one frozen task model (Qwen3.8-27B) and overlapping initialization seeds (B.1); whether the seven chains are independent draws or correlated updates to the same task model is not tested.
  3. Prose compilation is scope-confounded and the mechanism contrast is post hoc. Union prose draws from BFCL+ToolOrbit+MGSM, while typed/recurrent use the 7-chain live_multiple corpus; prompts are 6,031 vs 716 chars (C.2). The pre-registered recurrent-minus-union contrast is −0.27 pp, inside the ±1.95 pp MDE, so why prose helps is not causally identified.
  4. The measured gain is small relative to the added inference cost. Union prose runs 2.02× tokens/task on the primary pool (2.75× on follow-up) and typed consensus 1.22× (1.32× follow-up); the MGSM cross-domain transfer also costs −3.3 pp and 1.20× tokens (Table 11), so shipping the harness may quietly hurt out-of-distribution reasoning.
  5. The evidence is concentrated in single-turn, low-multiplicity tool use. All headline results are BFCL v4 single-turn with one frozen task model; the stateful multi-turn integration sits at a 2–4% success floor with MDE 6–18 pp (C.6), so the components that should matter for multi-step agents are exactly the ones this single-step environment leaves inert.
Questions for authors

Refer to above weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.