- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper addresses an overlooked issue within the LLM-driven autonomous program discovery system (such as the "LLM-generated program + evaluator scoring + selection iteration" closed loop): when the evaluator itself has noise, the selection process will overfit the evaluator's noise, resulting in the "winner's curse" a significant improvement on the training set, but an illusion on the validation set. The authors used the Ctrip hotel search ranking dataset (ICDM 2013, 9.9M impressions) as the test bed and employed Claude Sonnet 5 as the mutation operator to conduct experiments in two search spaces.
Critical strengths
- No need for new models or new search algorithms. Instead, simply adopt the mature statistical discipline from ML competitions; and use the positive-negative contrast of "failure without headroom space / success with headroom space" to prove that it can distinguish between true signals and false signals.
Critical weaknesses
- The task domain and dataset are too narrow, and the extrapolation boundary is unclear. All the evidence comes from a learning-to-rank task (Expedia/Booking.com hotel search), and the reward is pairwise NDCG@10; the typical task families in the LLM program discovery literature - combinatorial optimization, code generation, theorem proving - have zero coverage. The "discovered programs" in the pipeline space are essentially a combination of feature engineering and ranking heads, not the discovery of algorithmic logic or program structure. In the pairwise ranking scenario where "evaluators are cheap and can run tens of thousands of queries repeatedly", disjoint audit folds + paired bootstrap are easy to implement; but in tasks where the validator is expensive (each run takes 30 minutes) or noisy (human review), the cost and margin calibration methods of this mechanism have not been discussed.
- The Headroom gate has circular reasoning, and the sample size of the pipeline space is too thin. The gate requires a hand-built reference candidate to test whether there is headroom in the space, but the reference in the paper is a strong baseline "created after two weeks of work" - this is equivalent to "knowing the answer, having a strong baseline, before deciding whether it is worth searching", which is not operational for true exploratory discovery (open questions without a ready reference). More critically, the pipeline space has only run 3 independent runs, and the authors themselves admit in Limitations that "three runs are sufficient for the paired-bootstrap noise, but three is not enough to characterize how variable the search itself is" - all three runs beat the baseline, and statistically neither can rule out "the headroom of this space is so large that randomly searching can win", nor can it estimate the variance between runs.
- Key ablation is missing, making it impossible to separate the causal contribution of audited acceptance from the headroom of the space. The success in the pipeline space is observed under the condition of "having headroom + using audit". If the audit is turned off in the same pipeline space and returned to naive fitness-gated selection, what would be the final test NDCG of the three runs? The paper did not run this control. This leads to an unanswered question: What is the role of audited acceptance - is it "preventing false positives from being treated as true" (diagnostic value), or "really improving the quality of the final solution" (therapeutic value)? If shutting down the audit process still allows one to beat the baseline in this space, then the contribution of this paper would merely be "helping you avoid unnecessary searching in the objective space", while the success in the pipeline space is more a result of the existence of sufficient room in the space itself.
Questions for authors
See Critical Weaknesses
2 comments
Thanks for the response. After reading through the response, I find that my previous concerns remain unaddressed. Taking into account the comments from the other reviewers, I have decided to maintain my original score.
Audit-off control is diagnostic, wider task coverage
Thank you for the detailed review. On the audit-off ablation, the answer is that the check is mainly diagnostic. In 34 earlier Expedia pipeline-space runs, the short-list audit shipped a program with the same test score as no audit every time. In eight fresh LLM runs on MSLR-WEB30K, scoring the winner once on held-out data ships the same candidate and moves the over-statement from about a third of the test gain to about zero (resolved). Only a short list can change what ships. It does so resolvably in a 180-run hyperparameter search, a non-LLM control (+0.0036 test gain), and not in the fresh ranking runs (+0.0012). Less reporting bias alone does not show fewer harmful deployments. We see the diagnostic value as the contribution: it says what a reported gain is worth.
On the gate and the three runs, we agree with both points. That analysis is removed, and the evidence no longer rests on three runs: it now rests on eight fresh and eight earlier LLM ranking runs, 20 LLM bin-packing runs and 180 non-LLM control runs. On whether random search would win in the Expedia pipeline space, with 50 evaluations per arm it shipped the same test NDCG@10 as guided LLM search (0.4334).
On scope, the revision adds a tabular task and a code-generation task, in which an LLM writes online bin packing heuristics (20 runs). There the candidates were near-duplicates and almost nothing needed correcting. Theorem proving and human validators are not covered. The theorems are stated for any evaluation outside the loop that uses a fresh sample, such as a human rater, a simulator or an A/B test, but only held-out data was tested, and rater or simulator bias is not covered. For costly validators, one pre-chosen winner has zero expected bias at any sample size.
What is new for self-improving loops is listed in our general comment. We are glad to answer anything further.