acceptodds
Under review as a conference paper at ICLR 2027

Automated Discovery Has No Universally Superior Harness

Abstract

Autonomous discovery systems such as OpenEvolve and TTT-Discover are often used as general-purpose harnesses. However, in practice, these are composite systems combining several design choices about archives, parent selection, exploration, and budget allocation into a single recipe. Because discovery runs are expensive and inherently stochastic, existing harnesses are often compared using too few independent trials to distinguish key methodological improvements from run-to-run variance. We systematically decompose OpenEvolve-style evolutionary search and the TTT-Discover search harness into their constituent components and systematically evaluate 30 budget-matched harnesses across 12 model–problem pairs using more than 3.1 million LLM rollouts and repeated-trial statistical analysis. Our results show that discovery harnesses have a generalization problem: No fixed harness is reliably superior across the model–problem pairs, and variants of OpenEvolve generally underperform simpler alternatives. Thus, harness choice is better viewed as a hyperparameter rather than as a universal recipe, and should be tailored to the problem and underlying model. We also find that early discovery progress predicts final performance, and use this property to present a budget-matched adaptive-allocation experiment that starts multiple harnesses, prunes weak partial runs, and reallocates compute to stronger survivors, outperforming both commitment to sampled fixed harnesses and non-adaptive harness ensembles. Together, these results motivate shifting from fixed harness selection to online adaptation guided by early performance. We will release all run pools, including baseline null distributions for every model–problem pair, as statistical infrastructure for evaluating future harness proposals.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Harness, Adaptive Harness, Automated Discovery, LLM-Driven Discovery, Algorithmic Discovery

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_SzdcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper compares 30 search configurations across 4 LLMs and 3 math tasks, using over 3.1 million candidate generations. It reports that:

  • No fixed configuration shows a statistically reliable overall advantage over a simple baseline.
  • Complex OpenEvolve-style configurations often perform worse than simpler methods.
  • Early scores help identify promising runs. Simulations suggest that stopping weak runs and continuing strong runs improves results under equal generation budgets.

The main message is that search settings should depend on the model and problem. The study treats the LLM as a candidate proposer; it does not evaluate an interactive agent within each proposal.

Critical strengths

The fact that single-turn llm proposer auto-discovery has no optimal harness reveals a fundamental instability in this auto-discovery approach. This makes a strong case for shifting toward mechanisms that encourage automated agents to carry out the discovery process, instead of leveraging fixed workflows.

Critical weaknesses

The title is misleading. The authors define automated discovery as a family of workflows following OpenEvolve style or TTT-Discover style, where each proposal is generated by a single LLM run. Many more frontier auto-discovery works have used agentic proposers, including CORAL, Co-Scientist, etc..

Questions for authors
  1. It is pretty counter-intuitive to see that OpenEvolve perform worse than baseline sequential BoN as OpenEvolve is quite a popular method in auto-discovery. Does OpenEvolve generate weaker ideas, invalid code, or proposals that exceed execution limits? Does extra context reduce proposal quality? A failure analysis would make the result much more useful.
  2. Is it possible to also include analysis including agentic proposers?
1 comment

Thank you for your review and feedback on our paper. We address your comments below.

W1

CORAL and Co-Scientist change how candidate ideas are produced. Our study focuses on the search components used to select and develop the candidates. We take apart the popular OpenEvolve-style and TTT-Discover-style harnesses and recombine their components to compare the strength of different combinations while keeping proposal generation fixed. With these comparisons, we are able to test if the search components themselves work consistently across model/task pairs. No single combination of harness components performed the best throughout all the pairs we tested. These results show how the tested combinations of search components perform across our model/task pairs, which can help guide future discovery systems, including ones with agentic proposers.

Q1

We audited three progressively configured OpenEvolve variants on Heilbronn with Qwen2.5-3B: one island without inspiration sampling, one island with the regular prompt, and four islands with the regular prompt. Each variant has five archived runs of 1,600 iterations.

VariantResponses with no valid SEARCH/REPLACE diffMedian user-prompt length
1 island, no inspiration1,316/8,000 (16.5%; 79–537 per run)2,448 characters
1 island, regular prompt7,070/8,000 (88.4%; 1,300–1,514 per run)21,467 characters
4 islands, regular prompt7,070/8,000 (88.4%; 1,316–1,518 per run)21,238 characters
The progression points to two different bottlenecks. First, moving from the no-inspiration prompt to the regular prompt coincides with a sharp increase in responses rejected before evaluation; increasing the regular setup from one to four islands leaves this rejection rate essentially unchanged. The logged user prompts had median lengths of about 2,448 characters without inspiration, 21,467 with the one-island regular prompt, and 21,238 with the four-island regular prompt. Passing the parser also did not mean the model produced a strong solution. Among proposals that returned a score, excluding each run's initial all-zero program and the 29 timeouts, the logged outcomes were:
VariantPositive-score proposalsZero with reported error (outside triangle / other code or output error)Zero without reported error
1 island, no inspiration1,762/6,651 (26.5%)4,643 (843 / 3,800)246
1 island, regular prompt116/930 (12.5%)733 (205 / 528)81
4 islands, regular prompt65/929 (7.0%)777 (267 / 510)87
The error logs include wrong output shapes, Python syntax/name errors, and points outside the triangle; invalid geometry receives score zero. Some accepted no-inspiration programs did run, but used random or heuristic samplers that did not reliably optimize the minimum triangle area. The archived final best scores were [0.0890, 0.0277, 0.0247, 0.0028, 0] for no-inspiration, [0, 0.0613, 0, 0, 0] for regular one-island, and [0, 0, 0, 0, 0.0010] for regular four-island. The averages are 0.029, 0.00362, and 0.00004, respectively. Thus, removing inspiration improved format adherence and the reported score, but did not make OpenEvolve competitive with this baseline on this task/model pair.

Q2

The addition of an agentic proposer would call for experiments that require a lot of compute. For this paper, we will keep the proposal mechanism constant so that we can focus on the search heuristics.

Reviewer_ph3MOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether the “harness” in LLM-guided discovery systems, meaning the rule that picks which program to edit next and how to spend the budget, works the same across different models and problems. It starts from a simple baseline (always edit the best program so far) and adds the parts of OpenEvolve and of TTT-Discover’s search one at a time. It tests 30 setups on 12 model–problem pairs (four models, three math tasks), with 5 runs per setup and 100 or 30 runs for the baseline. The result: no setup is significantly better than the baseline after correcting for multiple comparisons, and the full OpenEvolve setups are the worst. The paper then shows that scores early in a run (at 25–50%) predict the final score, and proposes running several harnesses at once, dropping the weak ones early, and finishing the strong ones. In a simulation on the recorded runs, this beats the baseline on 10 of 12 pairs.

Critical strengths

The question is good and the way the systems are broken into parts is clear and useful. The statistics are more careful than usual in this area: large baseline pools, proper null distributions, correction for multiple comparisons, and honest reporting that even the authors’ best setup is not significant. The promised release of all runs with per-step scores would be useful to others.

Critical weaknesses

The main claim is stronger than the evidence, and the key experiment is missing one control.

  • The negative result is underpowered. With only 5 runs per setup and 30 setups to correct for, not finding a winner is expected even if a moderate effect exists. The paper shows "nothing is significantly better", not "nothing is better". There is no power analysis.
  • Only improvements are tested. All tests ask "is this setup better?" The paper also claims some parts make things worse, but never tests that.
  • Many per-pair p-values are too small. The bootstrap test cannot produce values above the baseline pool's maximum, so any run that beats the pool max gets p ≈ 0. But by chance this happens about 14% of the time with a 30-run pool (5/35). Most of the starred GPT-OSS results in Figs. 2–4 and Tables 5–12 are of this kind.
  • The cross-pair baseline is in the wrong place. A setup equal to the baseline wins at least 7 of 12 pairs with probability about 0.39, not 0.5. So the zero line in Fig. 1b is above the true null (App. E).
  • The OpenEvolve runs look broken. On Heilbronn with Qwen2.5-3B, all OpenEvolve runs score almost exactly 0 (Tables 8–9), while the baseline median is 0.07. No other setup does this. The paper does not report how many generated programs failed, or whether the real OpenEvolve code was used. Since "OpenEvolve underperforms" is in the abstract, this needs an answer.
  • Fewer useful pairs than claimed. On 6 of the 12 pairs the baseline is already near the maximum score, so nothing can be told apart. The GPT-OSS runs use only 160–320 LLM calls, far fewer than these harnesses are built for. The real evidence is closer to 4–6 pairs at small budgets.
  • The adaptive method is missing a control. Dropping weak runs early may help simply because it drops unlucky runs, not because it picks a better harness. The needed control is the same early-dropping applied to many runs of the baseline alone. The authors already have the data to run this. Also, several of the 10/12 "wins" in Table 1 are smaller than the noise visible between the reference rows of Tables 13–16, and no confidence intervals are given.
Questions for authors
  • Does the adaptive method still win against early-dropping over baseline runs only, at the same budget?
  • What fraction of OpenEvolve programs failed to run or evaluate? Was the real OpenEvolve code used?
  • How many of the starred per-pair results survive an exact rank test?
  • Please add confidence intervals to Table 1, and say exactly which setup each point in Fig. 1b is. The OpenEvolve endpoints use N = 1 and different (K, ε) per model, so the drop at "+Depth" cannot be explained from the figure.
2 comments

Thank you for your in-depth review and feedback. We address your concerns below.

W1

Five runs per configuration and correction across 30 comparisons limit our ability to detect moderate effects; we do not present the nonsignificant results as evidence that no harness is better. The paper’s finding is the pattern across pairs: the same search choice can help on some model/task pairs and hurt on others, and no single configuration emerges as a consistent winner in the observed results. The strongest observed configuration has a cross-pair majority-win rate of 0.914 against the baseline and an unadjusted p-value of 0.023, but its Holm-adjusted p-value is 0.678. We therefore report it as promising evidence, not a statistically established winner. The cross-pair pattern and component comparisons remain useful for showing why a result on one pair may not transfer to another; more independent runs would be needed to measure moderate effects precisely.

W2

Our primary question is whether a search change improves on Sequential BoN, so the main test is one-sided. We agree that a lower score in this test is not proof that the change caused harm. The comparison still answers the improvement question we set out to study; we will describe lower scores as observed outcomes rather than causal or statistically established harm.

W3

We agree that a very small bootstrap p-value can be misleading when a result is above the largest score in a small baseline pool. In that case, resampling the same 30 baseline runs cannot produce a higher score, so the bootstrap reports a near-zero tail probability. This means the result exceeds the observed pool; it does not show that the true probability is zero. As the reviewer notes, under the null there is about a 5/35 chance that the highest score among five candidate runs and 30 baseline runs falls in the candidate group. Following the reviewer's suggestion, we compared the five individual scores with the matched baseline scores using an exact rank test: 11 of the 34 starred results have unadjusted p < 0.05. This provides evidence that a meaningful subset of the improvements is also reflected across runs. The other results are inconclusive under this small-sample check, rather than evidence that they have no effect.

W4

Thank you for pointing this out. We have made the appropriate fix to our figure.

W5

We audited the archived regular-prompt one-/multi-island OpenEvolve logs for all three tasks and four models, covering 110,400 recorded iteration outcomes. Of these, 36.3% produced no valid SEARCH/REPLACE diff, so no candidate was available to evaluate; 10.0% failed before evaluation for other reasons, including API/context-window and code-length errors; and 53.6% reached candidate evaluation. Among candidate evaluations, 23.8% had a logged program/evaluator error or timeout. A zero score without an explicit error is a completed evaluation, not a run failure. The explicit error/timeout rate among candidate evaluations varies by task: 15.1% for Circle Packing, 40.1% for Heilbronn Triangle, and 15.1% for Second Autocorrelation.

We used the actual OpenEvolve implementation, not a reimplementation. The preserved source tree is a vendored snapshot of the official repository at commit 65cbbe8 (version 0.2.26)

W6

We agree that near-ceiling baselines leave less room to show improvement, and that the GPT-OSS runs used smaller budgets than the Qwen runs. We therefore interpret the results at the budgets we tested, not as evidence about how these harnesses would perform with much longer GPT-OSS runs. Those settings still help show whether a harness can maintain strong performance or cause regressions; the other settings provide more room to observe gains. All methods were compared under the same rollout budget within each model/task pair. Thus, the evidence covers 12 tested pairs, even if not all 12 are equally informative about potential improvements.

W7

We ran the requested control using the same 12→5→2→1 checkpoint schedule and five-full-run-equivalent budget with only Sequential BoN as the harness. We compared the diverse harness and fixed harness policy using a cross-pair majority win rate: in each replay trial, a policy counts as winning if its score is higher on at least 7 of the 12 model–task pairs; ties count for neither policy. Across 100,000 replays, the diverse adaptive policy reached a majority in 49.15% of trials, the Sequential-BoN-only control did so in 24.89%, and neither reached a majority in 25.96%. This comparison suggests that harness diversity adds value beyond checkpoint pruning alone under the tested budget.

Q1

Yes, as stated above when addressing weakness 7, the adaptive harness method beats early dropping applied to baseline runs only at the same budget.

Q2

We answer this question in response to the weakness stated above.

Q3

As stated above, 11 out of the 34 starred per-pair results survive an exact rank test.

Q4

We added the confidence intervals for the results in Table 1 in Appendix section F (Tables 3-5). The +Depth setup in Fig. 1b refers to decreasing N while increasing T to increase the depth of a run while preserving the overall budget.

Reviewer_qrt9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Decomposes two discovery-harness families, OpenEvolve-style evolutionary search and TTT-Discover-style PUCT search, into their components and tests whether any fixed harness transfers across models and problems. Starting from greedy Sequential Best-of-N, 30 budget-matched harnesses are evaluated on 12 model–problem pairs (four LLMs by three mathematical-discovery tasks) with over 3.1 million rollouts and bootstrap and permutation significance tests. Finds no fixed harness is reliably superior, that early progress predicts final performance, and that an online pruning-and-reallocation policy outperforms committing to one harness or an unpruned ensemble. Releases all rollout pools and null distributions.

Critical strengths
  • Unusually rigorous for this area. Budget-matched harnesses, repeated trials, bootstrap reference pools, permutation tests with Holm correction, and 3.1M rollouts let it separate component effects from run-to-run variance.
  • The result is contrarian and actionable. It shows complex OpenEvolve machinery often underperforms a greedy baseline, reframes harness choice as a hyperparameter, and turns early-progress predictiveness into a concrete pruning policy.
Critical weaknesses
  • The adaptive-allocation result is an offline resampling simulation over the same fixed-harness runs, not a live run, and it is reported as win counts without the significance testing applied to the fixed-harness findings. The policy is also close to successive halving but is not compared to Hyperband or ASHA despite citing them.
  • The scope is narrow. Three mathematical-discovery tasks and four models support the title's broad "no universally superior harness" claim, and the adaptive gains in Table 1 are small in absolute score even where the win rate is high.
Questions for authors
  1. Finding 3 is evaluated by resampling from the fixed-harness run pool rather than running the adaptive policy end-to-end. How much would a live run differ, given that pruning decisions would then consume budget and the checkpoint ranking is itself noisy?
  2. The adaptive policy resembles successive halving, but Successive Halving, Hyperband, and ASHA are cited yet not compared. Why, and what does the adaptive policy add beyond them?
  3. After Holm correction no fixed harness is significant (strongest corrected p is 0.678). Is the headline best read as a powered null result relative to Sequential BoN, and does the conclusion extend beyond the three mathematical tasks?
  4. The adaptive gains in Table 1 are small in absolute score (for example 79.17 versus 78.42). Is the 10/12 win rate driven by a few pairs with meaningful gains, and what is the practical benefit over the added complexity?
  5. Only OpenEvolve-style and TTT-Discover-style harnesses are decomposed, and both are built from Sequential BoN. Do the no-universal-harness and hyperparameter conclusions extend to other frameworks such as FunSearch, AlphaEvolve, or reflection-based harnesses?
2 comments

Thank you for your feedback and analysis. We will address each of the points below:

W1

We use replay to test a specific part of the adaptive-allocation idea: with an equal rollout budget, do checkpoint scores help select saved trajectories that later finish with stronger results? At each checkpoint, the policy uses only the score available then to choose which trajectories continue. Since early-score ranking is the basis for pruning, this tests a necessary part of the method: whether early scores identify trajectories that finish stronger. The 100,000 trials repeatedly resample the same saved-run pool. They estimate how often each policy wins under this replay, but are not 100,000 new LLM runs or, by themselves, a significance test for performance on new runs. This offline test isolates the value of checkpoint selection without spending compute on new model calls. We answer your concerns about comparing our policy to Hyperband/ASHA below in the questions.

W2

The paper’s main value is not that one setup produces a large gain everywhere. It is that results vary by model and task, so a fixed default can miss better choices. Even within discovery tasks that are distinct but related, we find that no fixed harness performs best across all 12 tested model/task pairs. When comparing harnesses, we use the cross-pair majority-win statistic to show how often each one beats a shared Sequential BoN baseline on a majority of the pairs. For our strongest observed adaptive policy 12→5→2→1, this means the share of replay trials in which it beats the five-run Sequential BoN baseline on at least 7 of 12 pairs. In our saved-run replay, the adaptive policy does so in about 67% of trials; Sequential BoN does so in about 14%, and neither reaches seven wins in about 20%. Against other tested allocation policies, the adaptive policy has a higher majority-win rate in most comparisons. For several close schedules, the most common outcome is that neither policy reaches seven wins; however, our policy still reaches seven wins more often than all other compared harnesses.

Q1

A live run would start multiple harnesses, prune weaker runs at each checkpoint, and spend the remaining budget on the survivors. Our replay applies the same schedule to complete trajectories from live runs, so it captures real variation between runs and in their checkpoint scores. But it uses the survivors’ recorded endings instead of generating new proposals after pruning. We therefore cannot tell from the replay whether a live run would score higher or lower. Both use the same five-full-run rollout budget; pruning redirects the remaining budget but does not recover what was already spent.

Q2

The schedule is itself successive-halving-style; we do not claim a new scheduler or superiority over Hyperband/ASHA. The empirical question is whether early scores can guide allocation across discovery harnesses, and whether harness diversity adds value under the same schedule and rollout budget. We added a control baseline where we use our policy but only use Sequential-BoN as the harness, and we found that the cross-pair majority-win statistic favored our policy over the control. Our policy reaches at least 7 wins in 49.06% of trials, versus 24.89% for Sequential-BoN-only (26.05% were ties). A direct benchmark against Hyperband/ASHA remains outside this evaluation.

Q3

No. The absence of a Holm-significant result is not a powered demonstration that the methods are equal. Most configurations have five runs, so smaller gains may go undetected. The headline reflects a separate pattern across 12 model/task pairs: the best-performing fixed setup changes across pairs, and no single setup emerges as a reliable default. Together, these results show that harness choice can be model- and task-dependent in automated discovery, so a harness that works well for one pair should not be assumed to transfer to another. Testing additional domains can measure how far this transfer problem extends.

Q4

The 10/12 count compares mean scores, so it can be influenced by a few large gains and does not, by itself, show how broadly a method wins. The paper also reports a cross-pair majority-win rate: across the twelve model/task pairs in the budget-matched policy comparison, adaptive allocation beats the Sequential BoN reference on a majority of pairs in 58.2% of trials. The single-harness baseline does so in 16.4% of trials, and the unpruned diverse-harness portfolio in 38.5%. These comparisons suggest a practical benefit beyond either committing to one harness or trying several harnesses without using early scores to guide the remaining budget.

Q5

We have not evaluated those systems and cannot claim that they share the same ranking or behavior. Our evidence supports treating the evaluated harness choice as setting-dependent; whether this extends to those other frameworks remains an empirical question.

I appreciate the authors’ response and their careful answers to my questions. However, I still believe my initial overall rating reflects my assessment of the paper, and therefore I would like to keep it.

Reviewer_5Hv4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates whether complex automated discovery harnesses consistently outperform simpler search strategies. The authors decompose OpenEvolve-style evolutionary search and TTT-Discover-style tree search into individual components, evaluating 30 configurations across 12 model–problem pairs with over 3.1 million LLM rollouts. The results suggest that complex harnesses do not consistently outperform simpler baselines. The paper further proposes an Adaptive Harness Ensemble that uses early performance to prune weaker runs and reallocate computational resources.

Critical strengths

1. Extensive empirical evaluation.

The paper provides systematic component-level ablations across multiple models and tasks, supported by repeated trials and statistical significance testing.

2. Useful insights into harness design.

The finding that additional search complexity does not necessarily improve discovery performance challenges common assumptions and provides practical guidance for automated discovery systems.

3. Adaptive compute allocation.

The experiments demonstrate the potential benefits of reallocating computational resources based on early search feedback.

Critical weaknesses

1. Potential task-specific bias in experimental conclusions

All three evaluated tasks involve numerical optimization with well-defined, immediate scalar feedback, which may inherently favor greedy refinement strategies over more sophisticated exploration mechanisms. Moreover, the strong correlation between early and final performance may be specific to these search landscapes. It remains unclear whether the findings generalize to tasks requiring delayed feedback, long-horizon exploration, or substantial conceptual breakthroughs.

2. Overstated conclusions and limited harness coverage

The central claim that no discovery harness is universally superior appears broader than the experimental evidence. The study primarily evaluates search-related components from OpenEvolve and TTT-Discover, without directly comparing more recent frameworks such as ShinkaEvolve, AdaEvolve, or EvoX. It also does not investigate broader agentic mechanisms, including structured memory, hypothesis-driven proposal generation, and critic-based feedback. These limitations make it difficult to draw general conclusions about automated discovery harnesses beyond the specific search configurations evaluated.

3. Limited novelty and validation of adaptive harness allocation

The proposed Adaptive Harness Ensemble largely applies established successive-halving principles to harness selection, with limited methodological novelty. Furthermore, its effectiveness is evaluated through resampling existing trajectories rather than prospective online experiments. The evaluation also primarily matches rollout counts, without fully accounting for differences in token consumption or computational overhead across harnesses.

Questions for authors
  1. Task characteristics and generalization: Could the strong performance of simple search strategies be partly explained by the characteristics of the selected tasks, which primarily involve numerical optimization with immediate scalar feedback? How would the conclusions generalize to tasks requiring delayed feedback, long-horizon exploration, or conceptual breakthroughs?

  2. Coverage of harness components: The evaluation primarily focuses on search-related components from OpenEvolve and TTT-Discover. How might the conclusions change when considering broader agentic designs, such as structured memory, hypothesis-driven proposal generation, critic-based feedback, or more recent discovery frameworks?

  3. Adaptive search strategies: The proposed Adaptive Harness Ensemble reallocates compute across fixed harness configurations, but does not evaluate within-run adaptive search mechanisms. How would its effectiveness compare with approaches such as AdaEvolve, EvoX, or PaceEvolve, which explore adaptive resource allocation or evolving search strategies? Could adapting search policies within a run provide additional benefits over pruning entire runs?

1 comment

Thank you for the detailed review. The inherent limitations in the design of our experiment limit the interpretation of our results, but not necessarily their usefulness. We respond to all of the comments below.

W1

Our claim is empirical. None of the evaluated harness configurations are reliably the best across the 12 model–task pairs. We evaluate the OpenEvolve-style and TTT-Discover-inspired harnesses with specific configurations. Within that scope, the study still provides evidence across four models, 30 configurations, and more than 3.1 million rollouts; it is broader than a result from one task or one run. We do not claim that the results are the basis for software engineering or open-ended discovery or agentic multi-step systems ranking.

W2

Our paper focuses on search setups based on the components of the simpler discovery harnesses OpenEvolve and TTT-Discover. We use this controlled setup in order to analyze the behavior of different search harness choices across the model/problem pairs. We agree that our study does not cover systems such as ShinkaEvolve, AdaEvolve, and other more recent frameworks with newly introduced features. Those are important extensions, but they do not remove the value of measuring how the search choices we test vary across settings. We will state clearly that our conclusion concerns these setups, not all discovery systems.

W3

As the reviewer says, we have no forward-looking evaluation of our algorithm online. In our setup, we do address a related question: whether early scores allow us to predict better completion later on among the saved candidate trajectories.

We compare harness performance under a matched number of LLM proposals for each task-model pair as our allocation budget. As mentioned by the reviewer, we do not establish equivalence in terms of tokens, runtime, and GPU hours. We have explicitly mentioned this matched budget and will clarify this limitation.

Q1

Thank you for asking this clarifying question. Some of our findings may be specific to mathematical discovery. Since the three problems are concerned with mathematical discovery, we can’t directly show that the same trends hold true for all other problems. However, our three distinct math problems were analyzed using four different models. Within these 12 settings, the best harness changes with the model and task. This variation is an important result in itself. Even among related discovery tasks, we find that there isn’t a fixed search recipe that reliably transfers between them. For tasks where feedback is not as informative early on, and progression relies on conceptual breakthroughs that come later in the search, pruning could stop a promising run too soon. Our results support pruning when early scores reliably predict later performance. Tasks with weaker or delayed signals may need a different pruning schedule.

Q2

These additional features may be able to provide better search performance across a more diverse/wider range of tasks. By providing the model with better direction and context, the model may be able to generate higher-quality candidates at a higher rate. If the additional context is reliable and relevant, it's possible we may be able to find that certain components are consistently helpful. When looking at simpler components like archive, parent-selection, and exploration rules that decide which generated candidates guide later search, we believe their effect is much more influenced by the model/task, since promising parts of the search space heavily vary across settings. In general, it’s not clear if/when those additional features would provide consistently stronger context or become noisy/irrelevant. Their effect could also depend on the framework they reside in. Our component comparisons provide a baseline for testing these ideas.

Q3

Adaptive search within a run could perform better when the best search strategy changes during a run. Pruning across runs helps when early scores suggest that a particular harness is a poor fit for the model/task pair. Combining the two ideas, where we both prune whole runs and adapt search with the survivors, could also prove to be beneficial. We have not done the experiments to compare these methods, so we cannot rank them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.