- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates whether automatically generated LLM harnesses provides useful specialization beyond repeated execution of a fixed program. Generated harnesses increases answer coverage, but the frozen selector gains 0.00 pp and full-budget oracle coverage ties. The results show that coverage and repeatability alone cannot justify useful specialization.
Critical strengths
- The paper addresses an important evaluation problem where additional answer coverage from a harness population does not necessarily show useful specialization
- Well-design on same-code control and differentiating categories of answer coverage, repeat-stable task advantages and usable selection
Critical weaknesses
- The main MATH evaluation concerns only one solver and a selected low-call panel, which limits generalizability.
- The reply matches harness executions but not model calls or tokens, which does not establish equal-cost utility.
Questions for authors
- Would the results change with more repeats solver?
- How would the comparison change under matched model-call or token budgets?
2 comments
Thank you for the response and the additional experiments. The optimized-program and token-usage comparisons help address my concerns about the selected panel and resource budget. I also appreciate the clarification on repeat counts. My overall assessment remains unchanged.
Thank you for the review and for recognizing the same-code control and the separation of coverage, repeat-stable advantages, and usable selection. Please also see the general response, which lists the revisions in the PDF.
W1. One solver and a selected low-call panel.
To go beyond the low-call panel, we optimized programs with two rounds of performance feedback on 100 development tasks (13 versions from three program families). A rule fixed in advance selected a check-and-revise program with up to six calls. We ran it once per task on all 400 evaluation tasks, beside fresh baseline executions combined by a gold-free majority vote:
The optimized program matches one baseline run and trails the cheaper 3-run vote by 8 tasks (2 vs 10 discordant; exact McNemar p = 0.039, uncorrected, post hoc); the prespecified 9-run comparison gives −1.75 pp (95% CI [−4.50, 1.10]; p = 0.065). The conclusion therefore extends to an optimized multi-call program. The same holds away from the ceiling: on BIRD, where the baseline answers 63.8%, a development-selected repair program gains 0.50 pp at 0.78× tokens, within the 383–388/600 range of identical copies; the clone-adjusted gain is D = 1.30 pp [−0.93, 3.70]; and a generated vote program uses 4.04× the tokens for the same 383/600. The study keeps a single solver; we state this in the limitations.
W2 / Q2. Comparison under matched model calls or tokens.
At equal executions (10,422 per arm on the 386 tasks), the generated panel makes 2.0× the logical calls (21,131 vs 10,422) and 2.9× the sample requests (30,395 vs 10,422) of the clones, and uses 4.04× the recorded tokens (45.66M vs 11.30M). Per execution budget b:
Under token matching the comparison shifts toward repeated baseline execution: the clones reach 98.70% coverage with 29.3k tokens per task, the panel only with 52.2k. For final answers, the W1 comparison is the closest test; it uses recorded usage, and we did not run a prospective fixed-token reallocation.
Q1. Would the results change with more repeats or another solver?
More repeats would change what the diagnostic can detect. With two discovery repeats (R=3), weaker members tie a strong expert so often that it is selected on only 27.4% of tasks; the exact expected gain over the best fixed member is −7.9 pp at R=3, +0.5 pp at R=5, +11.9 pp at R=10 and +16.9 pp at R=20. A prespecified confirmation at baseline accuracy 0.96 detects a 3 pp crossover in 196/200 datasets at R=80 and 200/200 at R=160, with 0/200 false support in every null setting. The usable-decision results are measured directly at the tested budgets: the selector and the full-budget comparison at R=3, and the comparison in W1, show no gain over repeated baseline execution.
Another solver is outside this study; our conclusions concern the tested solver, as stated in the limitations.