- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
Logged edit proposals from harness evolution chains can be reused even when chain-level selection fails
Two ways to build one frozen harness: prose compilation (union or recurrent text) and typed consensus (mechanisms proposed by at least half of 7 chains, inactive components removed, best development-ranked implementation copied)
On 1,007 held-out BFCL v4 single-turn tasks with one frozen task model, gains over bare are about +2.3, +2.0 and +1.7 points
A fresh 400-task follow-up shows smaller effects, and every interval includes zero
Single-chain finalists drop below bare on held-out tasks despite development gains; also covers noise analysis, inactive-edit checks, inference cost, pre-registration and full log release
Critical strengths
transparent measurement - pre-registration, paired intervals, minimum detectable effect, machine-graded chain exclusion, sensitivity corpus, repeated runs and a full log release
negative results are reported openly (matched prose, fresh follow-up, multi-turn floor), which makes the work easy to audit
Useful warning for self-improving agents - edits that never run still get accepted, development gains flip on held-out tasks, and 42.5% of decisions needed tie-break resampling
The released evolution logs, including rejected edits, could be a useful community resource
Critical weaknesses
Small effect - gains of about 2 points on an 80–92% baseline, all fresh follow-up intervals include zero, and the same artifact on the same tasks varies by 1.78 points across runs
Partly post hoc and weakly controlled - prose arms were registered as controls and only later called methods; prose results use an older archived bare run rather than one run at the same time; most primary tasks come from live_multiple, the same category the chains evolved on
Confounded comparisons - union prose uses a broader archive (ToolOrbit, MGSM, extra runs); recurrent prose is a single 716-character prompt; no best-chain-prompt, hand-written-prompt, or DSPy/GEPA baseline at the same compute budget
Very thin chain population - consensus built from 7 chains including 3 the authors themselves hard-exclude, only 2 clean-delta chains, and the "selection fails" claim rests on 2 Kimi finalists
Many inconsistencies - pool sizes 1,669 vs 1,567 vs 1,407, proposals 113 vs 165, MGSM −3.3 vs −3.89, the hybrid arm beating the "oracle upper bound," the C.2 verdict saying matched prose "retains the gain" when it doesn't, S6 with a positive interval but McNemar p=1, undefined gate G4, missing Table 6, a "three-model" ladder showing two, plus missing or duplicated appendix tables
Questions for authors
what is the McNemar p for typed consensus on the primary pool, does it pass your own registered rule, and were the prose arms rerun against a bare baseline done at the same time?
how does it compare with only the best chain's prompt, only the consensus system prompt, or a short hand-written function-calling prompt, and does max_steps of 2 differ from bare?
why include 3 hard-excluded chains in consensus, and what are the results with only clean chains?
please explain the inconsistent numbers, especially the C.2 verdict, the H4 hybrid-above-oracle result and S6, and say whether any headline numbers change
other prompt optimizers at the same budget, a second task model, and a multi-turn benchmark with enough statistical power