acceptodds
Under review as a conference paper at ICLR 2027

More Programs or More Rolls? Separating Coverage from Specialization in LLM Harnesses

Abstract

Automated generation of LLM harnesses promises to improve inference through task specialization. Yet additional answer coverage can arise from repeated execution of the same program, making specialization difficult to identify. We introduce a controlled evaluation that separates answer coverage, repeatable task advantages, and gains from pre-execution selection. On 386 MATH-500 tasks, we compare eight generated harnesses plus a baseline with nine byte-identical baseline copies, using three executions per member. Identical programs yield 2.16 percentage points of repeat-averaged oracle headroom. Generated programs exhibit substantially more repeatable score patterns, but these chiefly reveal persistent weaknesses: losses relative to the baseline persist across all three repeats on 100 tasks, while persistent wins occur on only one task and are sensitive to answer extraction. The frozen selector gains 0.00 percentage points, and both populations reach 98.70% oracle coverage at 27 harness executions. Stable complementarity remains unresolved at three repeats. Supporting BIRD traces locate failures in mechanism implementation, activation, and output validity. Together, these findings establish why coverage and repeatability alone cannot justify claims of useful specialization. They motivate an evaluation standard for harness diversity: task advantages should persist across executions, guide usable decisions, and improve on additional fixed-program executions under matched inference budgets.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM harnesses, automated agent design, LLM evaluation, task specialization, inference-time scaling, controlled evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_q3BEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether automatically generated LLM harnesses provides useful specialization beyond repeated execution of a fixed program. Generated harnesses increases answer coverage, but the frozen selector gains 0.00 pp and full-budget oracle coverage ties. The results show that coverage and repeatability alone cannot justify useful specialization.

Critical strengths
  • The paper addresses an important evaluation problem where additional answer coverage from a harness population does not necessarily show useful specialization
  • Well-design on same-code control and differentiating categories of answer coverage, repeat-stable task advantages and usable selection
Critical weaknesses
  • The main MATH evaluation concerns only one solver and a selected low-call panel, which limits generalizability.
  • The reply matches harness executions but not model calls or tokens, which does not establish equal-cost utility.
Questions for authors
  • Would the results change with more repeats solver?
  • How would the comparison change under matched model-call or token budgets?
2 comments

Thank you for the review and for recognizing the same-code control and the separation of coverage, repeat-stable advantages, and usable selection. Please also see the general response, which lists the revisions in the PDF.

W1. One solver and a selected low-call panel.

To go beyond the low-call panel, we optimized programs with two rounds of performance feedback on 100 development tasks (13 versions from three program families). A rule fixed in advance selected a check-and-revise program with up to six calls. We ran it once per task on all 400 evaluation tasks, beside fresh baseline executions combined by a gold-free majority vote:

PolicyCorrect / 400Tokens relative to program
Optimized program3801.00× (3.49M)
Baseline, 1 run3800.22×
Baseline vote, 3 runs3880.65×
Baseline vote, 9 runs3871.91×

The optimized program matches one baseline run and trails the cheaper 3-run vote by 8 tasks (2 vs 10 discordant; exact McNemar p = 0.039, uncorrected, post hoc); the prespecified 9-run comparison gives −1.75 pp (95% CI [−4.50, 1.10]; p = 0.065). The conclusion therefore extends to an optimized multi-call program. The same holds away from the ceiling: on BIRD, where the baseline answers 63.8%, a development-selected repair program gains 0.50 pp at 0.78× tokens, within the 383–388/600 range of identical copies; the clone-adjusted gain is D = 1.30 pp [−0.93, 3.70]; and a generated vote program uses 4.04× the tokens for the same 383/600. The study keeps a single solver; we state this in the limitations.

W2 / Q2. Comparison under matched model calls or tokens.

At equal executions (10,422 per arm on the 386 tasks), the generated panel makes 2.0× the logical calls (21,131 vs 10,422) and 2.9× the sample requests (30,395 vs 10,422) of the clones, and uses 4.04× the recorded tokens (45.66M vs 11.30M). Per execution budget b:

b391527
Panel coverage (%)98.4598.7098.7098.70
Clone coverage (%)98.2498.6398.6998.70
Panel tokens per task (k)3.552.275.2118.3
Clone tokens per task (k)3.39.816.329.3

Under token matching the comparison shifts toward repeated baseline execution: the clones reach 98.70% coverage with 29.3k tokens per task, the panel only with 52.2k. For final answers, the W1 comparison is the closest test; it uses recorded usage, and we did not run a prospective fixed-token reallocation.

Q1. Would the results change with more repeats or another solver?

More repeats would change what the diagnostic can detect. With two discovery repeats (R=3), weaker members tie a strong expert so often that it is selected on only 27.4% of tasks; the exact expected gain over the best fixed member is −7.9 pp at R=3, +0.5 pp at R=5, +11.9 pp at R=10 and +16.9 pp at R=20. A prespecified confirmation at baseline accuracy 0.96 detects a 3 pp crossover in 196/200 datasets at R=80 and 200/200 at R=160, with 0/200 false support in every null setting. The usable-decision results are measured directly at the tested budgets: the selector and the full-budget comparison at R=3, and the comparison in W1, show no gain over repeated baseline execution.

Another solver is outside this study; our conclusions concern the tested solver, as stated in the limitations.

Thank you for the response and the additional experiments. The optimized-program and token-usage comparisons help address my concerns about the selected panel and resource budget. I also appreciate the clarification on repeat counts. My overall assessment remains unchanged.

Reviewer_bSWvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors study the effectiveness of a harness in terms of lift from task specialization vs executing the LLM multiple times and averaging out the 'noise'. This is an important problem to study given the growing popularity of agentic AI. The authors characterize their findings using two well-used benchmarks - MATH-500 and BIRD Text-to-SQL. On the MATH dataset, they show that repeated executions don't perform as well as the baseline. In particular, the generated harnesses suffer all-3-repeat losses against the baseline on 100 tasks (570 member–task pairs), whereas all-3-repeat wins occur on only 1 task. The produce similar results for the BIRD SQL benchmark as well. More specifically, across 12 generated discovery harnesses they test, 21/66 pairs produce identical correctness vectors, and source-code similarity has near-zero correlation with correctness disagreement (ρ = -0.010). Overall, they show a negative result w.r.t. characterizing the use of task specialization and repeated executions of the solver.

Critical strengths
  1. Empirical rigor: The introduction of byte-identical baseline clones (H_(stable) = 0 by construction) as a control group is a standout methodological contribution. In LLM agent literature, post-hoc oracle improvements across a population of agents are routinely misattributed to architectural diversity. Thus, the authors expose a critical confounding variable in automated harness evaluation.

  2. Clearer distinction between repeatability and variance : Using qualitative vs. quantitative interactions, the paper cleanly proves (theoretically via Equation 2 and empirically) that high score-pattern repeatability across runs does not imply useful specialization. Showing that ρ_(res) = 0.789 is overwhelmingly driven by consistent failures (100 tasks with persistent losses vs. 1 formatting-artifact win) is a sharp diagnostic insight that prevents researchers from mistaking reproducible variance for division of labor.

  3. Trace audits and transparency: Rather than treating harnesses as black boxes, the BIRD trace audit connects high-level statistical outcomes to concrete code-execution failures (unimplemented docstrings, untriggered syntactic error traps on semantic errors, and broken output parsers).

Critical weaknesses
  1. Ceiling effects: The single-call GLM-5.3-Flash baseline already achieves 95.85% accuracy on the 386 retained MATH-500 tasks, leaving very little error headroom for complex harnesses to improve upon, while the 14 excluded tasks with incomplete runs were substantially harder (71.43% baseline accuracy).

  2. Statistical underpowering at R = 3: As Table 8 reveals, with R=3 repeats, their v3* diagnostic has 0% statistical power to detect even a massive two-specialist crossover.

  3. Execution-Matched Rather Than Cost/Token-Matched Replay Budgets: In the portfolio replay analysis (Table 3), the budget b ∈ {3, 9, 15, 27} counts harness executions rather than LLM calls or tokens. Because a single generated harness execution can invoke up to 5 samples or multiple verification/adjudication turns, the generated panel consumes significantly more compute and tokens per execution than the single-call baseline clones. Even with this unfair compute advantage, the generated panel merely ties the clones—meaning that under a strict token-matched or dollar-matched budget, the generated harnesses would perform even worse.

Questions for authors

My main question relates to token-matched vs execution budgets used in the paper. As stated above, budgets count harness runs, ignoring that generated harnesses can use 5x more calls. How do the coverage curves in Table 3 look when plotted against total solver tokens or total LLM invocations rather than harness executions? At matched token cost, do generated harnesses significantly underperform simple repeated sampling of the baseline?

Other questions:

  1. What caused the incomplete runs on the 14 harder tasks (e.g., timeouts from multi-turn harnesses)? If those 14 tasks are completed and evaluated, does the directional win/loss balance (currently 1 win vs. 100 persistent losses) change?

  2. Why was the full clone-controlled repeat-stability and held-out transfer analysis (D, ρ_(res)) not reported on the BIRD benchmark, where the lower baseline accuracy eliminates the ceiling effect seen in MATH-500?

3 comments

Thank you for the detailed review and for recognizing the same-code control, Equation 2, and the trace audit. Please also see the general response, which lists the revisions in the PDF.

Main question (and W3): coverage against tokens and calls; do generated harnesses significantly underperform at matched token cost?

Coverage against tokens. Table 3 now carries the recorded tokens at each execution budget b:

b (executions per task)391527
Panel coverage (%)98.4598.7098.7098.70
Clone coverage (%)98.2498.6398.6998.70
Panel tokens per task (k)3.552.275.2118.3
Clone tokens per task (k)3.39.816.329.3

At b=3 both arms run only the baseline program. Plotted against tokens, both arms reach 98.70% coverage, but the clones get there at 29.3k tokens per task, whereas the panel first does at b=9 with 52.2k (1.8×); at b = 9, 15 and 27 the panel uses 5.3×, 4.6× and 4.04× the clone tokens. In calls, the full panel makes 54.7 logical calls per task versus 27 for the clones (2.0×), and 78.7 versus 27 sample requests (2.9×).

Final answers at comparable tokens. We added a stronger program: after two rounds of performance feedback on 100 development tasks, the best of 13 versions (a check-and-revise program, up to six calls) was frozen and run once per task on all 400 MATH evaluation tasks, beside fresh baseline executions combined by a gold-free majority vote:

PolicyCorrect / 400Recorded tokensRelative to program
Optimized program3803.49M1.00×
Baseline, 1 run3800.75M0.22×
Baseline vote, 3 runs3882.27M0.65×
Baseline vote, 5 runs3883.69M1.06×
Baseline vote, 9 runs3876.67M1.91×

Against the 3- and 5-run votes the program has 2 vs 10 discordant tasks (exact McNemar p = 0.039, uncorrected, post hoc); the prespecified 9-run comparison gives −1.75 pp (95% CI [−4.50, 1.10]; p = 0.065). On BIRD (200 held-out tasks, three repeats each):

BIRD policyCorrect / 600Tokens relative to baseline
Baseline3831.00×
Development-selected repair program3860.78×
Generated vote program3834.04×
Three identical baseline copies383–3880.90–0.91×

So at comparable token cost, no generated program improves on repeated baseline execution beyond the variation among identical copies. On MATH the optimized program trails the cheaper 3-run vote, significantly in the uncorrected post hoc comparison and inconclusively in the prespecified one; on BIRD the repair program's +0.50 pp at 0.78× tokens lies within the 383–388/600 range of identical copies, and the vote program spends 4.04× the tokens for the same 383/600. These comparisons use recorded token usage; we did not run a prospective fixed-token reallocation.

W1. Ceiling effect at 95.85%.

  1. Lower-accuracy benchmark. On BIRD, where the baseline answers 63.83%, the clone-controlled replication reaches the same decision as MATH (Q3, next comment).
  2. Detection at a 0.96 baseline. The unchanged diagnostic detects a 3 pp crossover at baseline accuracy 0.96 in 196/200 datasets at R=80 and 200/200 at R=160, with 0/200 false support in each of eight null settings. In an exploratory grid at BIRD's dimensions (3 members, 200 tasks), a 3 pp crossover is supported in 0/25 datasets at R=3 for every baseline accuracy in {0.3, 0.5, 0.7, 0.9}, in 3–17/25 at R=160, and in 15–25/25 at R=640, with the highest rates at 0.9. In these simulations a high baseline did not prevent detection; the number of repeats per task did.

W2. Underpowering at R=3. The 0% power comes from per-task discovery. With two discovery repeats, weaker members tie the true expert so often that it is selected on only 27.4% of tasks, and all 300 datasets fail the gain conditions before randomization. Exact expected gains over the best fixed member:

R351020
Expert selected27.39%51.37%84.09%98.36%
Expected gain−7.91 pp+0.48 pp+11.93 pp+16.93 pp

Q2. The 14 harder tasks: cause, and effect on the win/loss balance.

Cause. All 61 missing cells are request timeouts (ReadTimeout) in generated members; baseline and clone records are complete. The affected tasks are harder and longer:

Observed characteristicIncluded (386)Excluded (14)
Baseline repeat-mean accuracy (%)95.8571.43
Mean difficulty level3.474.29
Level-5 tasks (%)26.4264.29
Mean question length (characters)198.75284.71

Effect. We keep every observed score on all 400 tasks and let each missing cell take either value. Persistent-win and persistent-loss counts are monotone in each cell, so the all-correct and all-wrong completions give exact bounds. Under every completion, the 400 tasks contain exactly 1 persistent-win task and 104 persistent-loss tasks, compared with 1 and 100 on the 386 retained tasks. Completing the 14 tasks therefore does not change the directional balance. These bounds concern the directional counts; they do not bound D or H_stable.

Q3. Clone-controlled analysis on BIRD.

Design. A repair program, which revises its SQL when execution fails, was selected on two development databases (83/150 versus 70/150 for the baseline). We froze six slots: the baseline, the selected repair program, a generated execution-vote program, and three byte-identical baseline copies. Each ran three times on 200 tasks from nine held-out databases (3,600 executions). The diagnostic is the MATH procedure, with baseline, repair and vote as the real arm and the three copies as the clone arm.

SlotCorrect / 600AccuracyRepeats 1 / 2 / 3
Baseline38363.83%127 / 132 / 124
Selected repair38664.33%127 / 129 / 130
Generated vote38363.83%130 / 125 / 128
Copy 138864.67%128 / 128 / 132
Copy 238664.33%128 / 129 / 129
Copy 338363.83%129 / 124 / 130

Recorded tokens over the 600 executions: baseline 753,540; repair 587,283 (0.78×); vote 3,043,815 (4.04×); copies 675,402–687,688.

QuantityEstimate95% interval
Held-out gain G_real2.35 pp[0.62, 4.13]
Same-code gain G_clone1.05 pp—
Clone-adjusted gain D1.30 pp[−0.93, 3.70]
Residual correlation, generated0.285[−0.079, 0.567]
Residual correlation, clones0.004[−0.263, 0.274]
Difference in residual correlation0.280[−0.246, 0.746]

The lower bound of D is negative, so the randomization test is not run and the decision is Insufficient Evidence. Pooling all nine executions per arm, oracle coverage is 141/200 for the generated arm and 146/200 for the identical copies. The repair program's 8.67 pp development gain shrinks to 0.50 pp on held-out databases, within the range of identical copies.

Without the same-code control, BIRD would show a held-out gain whose interval excludes zero; after subtracting what identical copies gain, the remainder is indistinguishable from zero.

Reviewer_bSWvOpenReview ↗

Ack authors' response

Thank you for addressing my key concerns (reporting token budgets; adding more details on the incomplete runs). I will maintain my score.

Reviewer_B74NOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines whether automatically generated LLM harnesses provide useful task specialization beyond gains from repeatedly executing the same program. It distinguishes answer coverage, stable task-specific advantages, and realizable gains from pre-execution selection. On 386 MATH-500 problems, the authors compare eight generated harnesses plus a baseline against nine identical baseline copies, with three executions per member. Identical programs already yield positive empirical oracle headroom, while generated harnesses exhibit repeatable differences dominated by persistent losses. The tested selector provides no improvement, and both populations reach the same oracle coverage at 27 executions. Supporting BIRD studies identify failures in mechanism implementation, activation, and output validity. The main contribution is the evaluation framework with repeated-execution controls and usable selection, although limited statistical power leaves stable complementarity unresolved.

Critical strengths
  • Byte-identical harnesses produce positive empirical oracle headroom, demonstrating that additional answer coverage alone cannot establish program specialization. This provides a concrete, useful control for evaluating whether generated harness diversity adds value beyond repeated sampling.

  • The formulation and directional analysis show that highly repeatable performance differences can primarily reflect persistent weaknesses rather than complementary strengths. Evaluating held-out-repeat transfer and pre-execution selection separately makes this distinction operational and helps prevent misleading interpretations of harness diversity.

Critical weaknesses
  • The calibration reports 0% detection power at three repeats even for a setting with 17.5 percentage points of true stable headroom; smaller crossovers remain largely undetected at twenty repeats. The authors appropriately acknowledge this limitation, but it substantially reduces the informativeness of the empirical nonpositive result. A proposed evaluation framework needs a demonstrated ability to detect practically meaningful specialization.

  • The main study uses one solver and a small panel of independently generated harnesses without performance-driven optimization. Moreover, budget comparisons match harness executions despite differing internal call and sample counts. Consequently, the observed coverage tie does not establish whether additional programs offer value over repeated baseline execution at comparable inference cost, which is the central practical question motivating the paper.

Questions for authors
  • Which components of the decision rule account for its 0% power at three repeats in the strong two-specialist setting? Given this calibration, what substantive conclusions about specialization can the empirical diagnostic support beyond insufficient evidence?

  • What are the actual completion counts and available token costs for the generated panel versus repeated baseline execution? Does the coverage comparison change when these resources, rather than harness executions, are matched?

4 comments

Thank you for highlighting the same-code control and the directional analysis. Please also see the general response, which lists the revisions in the PDF.

W1 / Q1. Which components cause 0% power at R=3, and what can the diagnostic conclude beyond Insufficient Evidence?

Which component. Neither the randomization test nor the confidence gates. The bottleneck is the per-task choice learned from two discovery repeats. In the two-specialist setting, one member succeeds with probability 0.95 and eight with 0.60. Each weaker member scores 2/2 with probability 0.36, so on average 2.88 of them tie a perfect expert, and with symmetric tie-breaking the expert is chosen on only 27.4% of tasks. Exact expectations:

RDiscovery repeatsExpert selectedHeld-out accuracyGain vs best fixed member (77.5%)
3227.39%69.59%−7.91 pp
5451.37%77.98%+0.48 pp
10984.09%89.43%+11.93 pp
201998.36%94.43%+16.93 pp

Replaying the original 300 datasets at R=3 gives mean G_real = −7.69 pp, G_clone = −0.02 pp and D = −7.67 pp (range −16.58 to −0.58 pp). Each necessary condition (positive lower bound for G_real, positive lower bound for D, D > 1 pp) holds in 0/300 datasets, so the randomization test never decides the outcome. More tasks only estimate a poor choice more precisely; more repeats improve the choice itself.

Demonstrated detection. A separately prespecified confirmation at a MATH-like baseline uses 4 members, 400 tasks and 200 datasets per condition. In the positive condition the baseline succeeds with probability 0.96 and two specialists alternate 0.99/0.93 across complementary task halves, so the true stable headroom is 3 pp. Support decisions:

ConditionTrue H_stableR=80R=160
Complementary specialists3 pp196/200200/200
Equal members00/2000/200
Equal, varying difficulty00/2000/200
Dominance00/2000/200
Dominance, varying difficulty00/2000/200

All prespecified targets are met (positive-condition Wilson lower bound 94.97% ≥ 80%; null upper bound 1.88% ≤ 5%). The diagnostic therefore detects a practically meaningful crossover given enough repeats per task. For a design like ours, R=80 would require 256,000 executions, which we did not run.

What three repeats already establish, measured directly rather than inferred from a non-rejection:

  1. Byte-identical programs produce 2.16 pp of repeat-averaged oracle headroom, so coverage alone cannot identify specialization.
  2. Task-wise choices learned from the generated panel do not transfer better than choices learned from identical copies: D = −0.26 pp, 95% CI [−1.45, 0.95], which excludes an advantage of this selection procedure above 0.95 pp at 95% confidence.
  3. No tested way of using the generated programs gains over further baseline executions on MATH: the frozen selector keeps the baseline on every task (0.00 pp gain); both populations reach 98.70% coverage at 27 executions, while the generated panel uses 4.04× the tokens; and an optimized program trails a cheaper baseline vote (W2, next comment).

What three repeats cannot establish is that no stable specialization exists at all; we state this as a limitation.

Dear Reviewer B74N,

We are writing to ask whether our rebuttal has addressed your concerns. We would be happy to discuss any remaining issues.

W2. One solver, a small panel without performance-driven optimization, and execution-matched budgets.

We added a performance-optimized program. Three program families (self-consistency, draft–check–revise, disagreement adjudication) were developed over two rounds of performance feedback on 100 development tasks: 13 versions, each run three times. A rule fixed before the rankings selected the version with the most correct answers, a check-and-revise program with up to six calls (290/300 on development). We then ran it once per task on all 400 evaluation tasks, beside fresh baseline executions combined by a gold-free majority vote:

PolicyCorrect / 400Recorded tokensRelative to program
Optimized program3803.49M1.00×
Baseline, 1 run3800.75M0.22×
Baseline vote, 3 runs3882.27M0.65×
Baseline vote, 5 runs3883.69M1.06×
Baseline vote, 9 runs3876.67M1.91×

The prespecified comparison with the 9-run vote gives −1.75 pp (2 vs 9 discordant tasks; 95% CI [−4.50, 1.10]; exact McNemar p = 0.065). Against the 3- and 5-run votes the program has 2 vs 10 discordant tasks (p = 0.039, uncorrected, post hoc). Optimizing this program therefore does not change the conclusion: with fewer tokens, the baseline (one run or a 3-run vote) answers at least as many tasks. The study keeps a single solver; we state this in the limitations.

Q2. Actual counts and token costs; does the comparison change when resources are matched?

Recorded resources on the 386 tasks, three executions per member:

Generated panel (9 members)Nine clonesRatio
Harness executions10,42210,4221.0×
Logical solver calls21,13110,4222.0×
Sample requests30,39510,4222.9×
Recorded tokens45.66M11.30M4.04×

Oracle coverage and tokens per task at each execution budget b:

b (executions per task)391527
Panel coverage (%)98.4598.7098.7098.70
Clone coverage (%)98.2498.6398.6998.70
Panel tokens per task (k)3.552.275.2118.3
Clone tokens per task (k)3.39.816.329.3

Tokens sum recorded usage; requests without usage records are excluded. At b=3 both arms run only the baseline program.

Matched on tokens, the comparison moves toward the clones: they reach the final 98.70% coverage with 29.3k tokens per task, whereas the panel first reaches it at b=9 with 52.2k (1.8×); at b = 9, 15 and 27 the panel uses 5.3×, 4.6× and 4.04× the clone tokens. For final answers, the W2 comparison above is the closest test: one baseline run and the 3-run vote, with 0.22× and 0.65× the program's tokens, answer at least as many tasks as the optimized program. These comparisons use recorded token usage; we did not run a prospective fixed-token reallocation.

Reviewer_B74NOpenReview ↗

Response to authors

Thank you for the careful response and the additional results. The analysis largely addresses my original concerns. I have updated my overall assessment.

Comments on the paper

1 comment

We thank the reviewers. All three valued the same-code control and the separation of answer coverage, repeat-stable task advantages, and usable selection. The paper's results and conclusions are unchanged; the revision adds only the analyses and experiments the reviewers asked for, and all added text is shown in blue in the revised PDF (main text within 10 pages). We first summarize the shared concerns, then list every change in the PDF. The individual responses give the full tables.

Shared concerns, what we added, and the results

1. Budgets matched executions, not calls or tokens (B74N W2/Q2; bSWv W3/main question; q3BE W2/Q2). Added: recorded logical calls, sample requests and tokens for every member and every budget. Result: at equal executions (10,422 per arm), the generated panel uses 2.0× the calls, 2.9× the sample requests and 4.04× the tokens of the clones (45.66M vs 11.30M). The clones reach the final 98.70% coverage with 29.3k tokens per task; the panel first reaches it with 52.2k.

2. No performance-optimized program; final answers at comparable cost (B74N W2; bSWv main question; q3BE W1). Added: a program optimized with two rounds of performance feedback (13 versions on 100 development tasks), run on all 400 MATH evaluation tasks beside baseline majority votes; token accounting for the BIRD programs. Result: the optimized program answers 380/400; a 3-run baseline vote answers 388/400 with 0.65× its tokens (exact McNemar p = 0.039, uncorrected, post hoc; prespecified 9-run comparison −1.75 pp, p = 0.065). On BIRD, a repair program's +0.50 pp lies within the range of identical copies, and a vote program uses 4.04× the tokens for the same answers.

3. 0% detection power at R=3 (B74N W1/Q1; bSWv W2; q3BE Q1). Added: an exact decomposition of the decision rule and a prespecified known-truth confirmation. Result: the bottleneck is per-task discovery, not the statistical gates: with two discovery repeats the true expert is chosen on only 27.4% of tasks. The unchanged diagnostic detects a 3 pp crossover at baseline 0.96 in 196/200 datasets at R=80 and 200/200 at R=160, with 0/200 false support in every null setting.

4. Ceiling effect and no clone control on BIRD (bSWv W1, Q3). Added: a clone-controlled BIRD replication: baseline, repair, vote and three identical baseline copies on 200 held-out tasks, R=3, 3,600 executions. Result: at 63.8% baseline accuracy, G_real = 2.35 pp [0.62, 4.13], but identical copies also gain 1.05 pp, so D = 1.30 pp [−0.93, 3.70]: Insufficient Evidence, as on MATH.

5. The 14 excluded tasks (bSWv W1, Q2). Added: a cause audit of the 61 missing cells and exact completion bounds over all 400 tasks. Result: all 61 cells are request timeouts in generated members. Under every completion, the 400 tasks contain 1 persistent win and 104 persistent losses (1 and 100 on the 386 retained tasks).

6. One solver (B74N W2; q3BE W1, Q1). We keep the study to the tested solver and state this in the limitations.

Revisions in the PDF

LocationWhat we addedRaised by
Abstract, §1Panel's 4.0× token cost; R=80 calibration; optimized-program resultall
§5.3Why power is 0% at R=3; confirmation at R=80/160B74N W1/Q1; bSWv W2; q3BE Q1
§5.4, Table 3Two rows of tokens per task at b = 3, 9, 15, 27; paragraph on calls, samples, tokensB74N Q2; bSWv W3; q3BE W2/Q2
§5.4, Table 4 (new)Optimized program vs 1/3/5/9-run baseline votes (MATH); repair, vote, baseline and copies with tokens (BIRD)B74N W2; bSWv main question; q3BE W1
§5.5Clone-controlled BIRD replicationbSWv W1, Q3
§6 LimitationsOne optimized program; R=80 needs 256,000 executions; missing-cell bounds; no prospective fixed-token reallocationall
App. CCause of the 61 missing cells; completion boundsbSWv Q2
App. D.4Selection probability and expected gain at R = 3, 5, 10, 20; replay of the 300 datasetsB74N Q1; bSWv W2; q3BE Q1
App. D.5Calibration table (5 conditions, R=80/160); exploratory grids incl. BIRD dimensionsB74N W1; bSWv W1, W2; q3BE Q1
App. HPer-member calls, samples, tokens; all 13 optimized-program versions; evaluationB74N W2/Q2; bSWv main question; q3BE W1
App. IBIRD design, per-slot accuracy and tokens, diagnostic with intervalsbSWv Q3

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.