- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies how initialization affects LLM-driven discovery. The authors develop 12 Modular discovery harnesses and evaluate them across five tasks. They find that sequential discovery often suffers from mode collapse and that diversity-oriented interventions do not consistently improve final discovery quality. Instead, early discovery quality is predictive of downstream success. Based on this observation, the paper proposes a parallel exploration stage to construct a strong initial population before sequential optimization, improving several existing harnesses by 2–21%.
Critical strengths
- Clear and practically useful insight about initialization. The paper identifies early population quality as a more reliable predictor of downstream discovery than simply maintaining diversity, and proposes a simple intervention applicable across different harnesses. This is a useful design principle for LLM-driven discovery systems.
- Broad empirical study across tasks and harnesses. The analysis covers multiple discovery tasks, LLMs, 12 Modular variants, and three state-of-the-art harnesses. The consistent gains from initialization across these settings make the main empirical observation reasonably convincing.
Critical weaknesses
- The initialization-budget selection is partly heuristic and potentially post-hoc. The convergence thresholds are task-specific, and the parameter N is selected through visual analysis of the discovery curves. This makes the claim of a generally applicable automatic initialization strategy less convincing.
- Evaluation artifacts affect at least one discovery task. The authors identify reward hacking in CloudCast and apply a post-hoc score cutoff to remove such discoveries. While transparently reported, this raises some concern about the robustness of the task evaluators and whether similar undetected artifacts may influence other tasks.
Questions for authors
- How sensitive are the results to the manually chosen convergence parameters (\delta, N)?
- Can the authors clarify how the convergence criterion would be selected for a completely new task without examining its discovery curves?
- How confident are the authors that reward-hacking behavior is absent from the other discovery tasks?
2 comments
With the rebuttal period coming to an end in 2 days, we thank you for your review and are wondering if you have any outstanding questions? We have updated the manuscript with new experiments and summarized said updates above.
Thank you for your review! We are glad you find our initialization method to be practically useful and our empirical analysis to be broad and comprehensive. We respond to your concerns in turn below:
Initialization Budget Selection
The initially proposed convergence criterion was intended as a proof of concept, and we previously noted that it requires calibration by a domain expert, which may be reasonable to expect of practitioners designing a discovery task. However, we agree that domain-specific convergence hyperparameters are challenging to set.
To address your concern, we revise our recommendation: instead of performing a convergence analysis to determine the initialization-to-discovery budget ratio r, we simply treat it as a hyper-parameter. Across the five (task, LLM) settings in the main paper, we find that discovery trajectories initialized with 25% of the discovery budget (r=0.25) consistently outperform uninitialized discovery.
We then also validate whether setting r = 0.25 generalizes to 15 total unique (task, LLM) settings in Appendix C, Fig 12, including two newly introduced (task, LLM) settings, DeepSeekV4 Flash (a new model family) for the CloudCast and CantBeLate task that we previously did not test. We find that serial trajectories initialized with 25% of the discovery budget beat their uninitialized counterparts (r = 0) in 12/15 settings. In the three settings where r = 0.25 did not actively beat uninitialized discovery, initialization either had a negligible impact or the LLM itself was not suited to the discovery task (i.e., lacked relevant parametric knowledge; we elaborate in Section C.3). In other words, using 25% of the discovery budget doesn’t harm the performance of discovery harnesses and in most cases improves it.
In conclusion, we find that simply using a predetermined 25% of the discovery budget for initializing serial trajectories consistently improves discovery performance across a wide range of settings, and sidesteps the problem of needing to determine task-specific convergence criteria. This validates the simplicity and utility of our initialization method.
Main paper Section 5 and Appendix C.3 have been updated with the revised r=0.25 recommendation, additional DeepSeek experiments (Figure 12), and subsequent analysis.
Reward Hacking
You raise reasonable concerns about task evaluators, and we agree! In this work, we use (and cite) the task definitions and their evaluation criteria as defined by prior published literature. For the task in question, CloudCast, we directly used the evaluation suite provided by (Agrawal et al., 2026a).
We notice, and raise concerns about, instances of reward hacking whereby the LLM (Gemini 3.5 Flash), rather than only implementing a cloud scheduling algorithm with the provided information, instead attempts to evaluate different discoveries locally and greedily choose the highest performing one. This behavior does result in a legitimate discovery, but may be undesirable in settings where evaluation should be independent of the discovery procedure. We leave this choice to the task-designer as it is outside the purview of improving discovery harnesses via interventions.
In this work, we assume this behavior is unwanted and subsequently detected reward hacking via anomalously high-scoring candidates. Therefore, we minimally augment Agrawal et al. (2026a)’s task evaluation by applying a max-score filter of 0.015 to CloudCast discoveries; only 12 out of 48817 CloudCast discoveries in this work (less than 0.025% of discoveries) exceeded this threshold. Our conclusions about the utility of initialization are unaffected by the inclusion/exclusion of these 12 candidate discoveries. We did not detect any such anomalously high-scoring candidates for other (task, LLM) pairs.
The responsibility of catching/evaluating reward hacking falls on the discovery-task designer, and appropriate edge cases should be covered in discovery evaluation. We point out that reward hacking is not a weakness of our initialization method (i.e., the scope of our paper), but rather a broader concern that task evaluations should account for.
We have updated Appendix B.1 with the analysis of reward hacking for CloudCast.
We have incorporated all updates from the review process into the paper PDF (demarcated in blue text).
[Agrawal et al., 2026a] optimize_anything: Unified text optimization can outperform specialized systems. In Proceedings of the ACM Conference on AI and Agentic Systems, 2026.