- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposed an evidence-grounded Bayesian optimization framework under collective LLM reasoning. Instead of directly relying on an LLM, the authors first proposed using design of experiments to collect a small amount of data from the target system. Then they apply some statistical analysis and let multiple LLMs use the results to propose promising search regions. After this, the proposed method performs sequential optimization by combining LLM results, Bayesian optimization, and random exploration. The effect of LLMs decays as more and more observations are collected. The paper provides robustness results for this scheme. Some numerical studies are conducted to validate this idea.
Critical strengths
This paper provides an interesting integration of statistical evidence with LLMs. The idea of use of multiple-agent aggregation to improve robustness seems sound. The paper also provides theoretical analysis for the aggregation framework.
Critical weaknesses
1, The claim regarding the benefit of multi-agent aggregation is not properly justified by the experiment. The single-agent baseline uses only GPT-4o-Mini, while the multi-agent setting uses seven different models, so the improvements may come from stronger or more diverse models rather than from the aggregation itself.
2, There might be potential information leakage in the experimental design. For Rastrigin, the DoE configuration is explicitly chosen “because the global optimum lies at the origin,” which partially assumes that the ground truth is known before conducting the optimization algorithm.
Questions for authors
1, Empty Section A.6, “INITIALIZATION ABLATION SECOND EXPERIMENT.”
2, Incorrect figure numbering. Figure 3(a) is about the experiment on Rastrigin-2D, not FCNet Naval.
3, The notation is inconsistent. In the main theorem, it is A_s^{loc}, whereas in the appendix, it becomes A_r, which can be quite confusing. The same issue applies to the decay exponent: in the main experiment, it is k, whereas in the main text, it is \gamma.
4, Could the authors clarify how the aggregation guarantees translate into optimization robustness? In particular, the medoid theorem assumes a majority of sufficiently accurate agents, and the projected-mean theorem only provides a representation-space distance bound rather than an objective-value guarantee. What additional assumptions are required to justify the claimed robustness of the overall BO procedure?
3 comments
Dear Reviewer wr5s,
As the discussion phase comes to an end, please acknowledge the author's response and consider leaving them a comment on whether it has resolved some or all of your concerns.
Thanks!
Thank you for the detailed response and additional experiments. While several of my concerns have been addressed, I still have some reservations about the experimental design and the limited theoretical guarantees, and I believe the paper would benefit from further refinement.
We would like to express our sincere appreciation for your insightful feedback and careful assessment. Our point-by-point responses are provided below.
Response to Reviewer wr5s
We sincerely thank the reviewer for the careful reading and constructive comments. We have expanded the experiments, improved the presentation, and clarified our theoretical results and their limitations. We address each concern below.
W1. Our motivation is that the best-performing LLM for a new optimization task is generally unknown beforehand. So it would be wise to trust widom of crowds rather than betting on one individual.
We extend the ablation to three strong models (GPT-5.6, Claude Opus 5.5, and Gemini 3.1 Pro), each evaluated individually and with seven independently sampled and aggregated proposals. We also include a heterogeneous ensemble of seven strong models and a statistical-only baseline. The LLM-based comparisons use matched DoE evidence and the same aggregation procedure.
Table 1 provides three findings. First, aggregation improves all three strong individual models. For example, Strong LLM 1 improves from 0.2288 to 0.1918 in average distance and from 33.90 to 30.09 in regret. Second, our original mixed-strength ensemble (0.1308; 19.91) outperforms all three homogeneous strong-model ensembles. Third, the heterogeneous ensemble of seven strong models achieves the best results (0.1107; 13.17).
These results support the wisdom of crowds: combining different models can outperform repeatedly sampling a single stronger model.
We also clarify the metrics. Avg. Dist. measures the average normalized Euclidean distance to a global optimum, while Regret measures the objective-value gap of initialization proposals, not final BO regret at budget T. See Section 3, Table 1 and Figure 2.
W2. We deliberately excluded (0,0) from the initial DoE for Rastrigin-2D because it is the known global optimum. A standard central composite design (CCD) includes this center point, which would make the benchmark trivially easy by identifying the optimum during initialization, leaving little room to evaluate subsequent optimization. We therefore used 8 CCD points and 5 maximin points, instead of 9 CCD and 4 maximin points.
Q1. We apologize for the confusing appendix presentation. The second initialization experiment is presented in Appendix Figure 8, immediately before Appendix A.6. It compares individual LLM proposals with projected-mean aggregation over 100 Monte Carlo replications.
The results show that aggregation consistently reduces the average normalized distance to the global optima across the six benchmarks, providing further evidence for collective reasoning. See Section 3 and Appendix Figure 8.
Q2. Thank you for identifying the incorrect Figure 3(a) reference. We have corrected it now. See revised Figure 3.
Q3. We agree that consistent notation is important for understanding and reproducing our framework. We have standardized the notation and simplified the pseudocode, moving detailed auxiliary procedures to the appendix.
In particular, we now define Li,s as the scalar trust-region size around anchor xi at iteration s. We also explicitly specify min-max normalization based on the original variable bounds:
[ \phi_j(x_j)=\frac{x_j-\ell_j}{u_j-\ell_j}. ]
We further clarify the local search domain and the LLM–GP decay parameter γ. See Sections 2.3–2.5, Appendix A.8, and Appendix B (Notation).
Q4. We appreciate the reviewer's concern about the difference between proposal robustness and end-to-end optimization guarantees. We have clarified the assumptions and conclusions of both aggregation theorems.
Theorem 1 shows that when a majority of LLM proposals are close to a reference region, their medoid also remains within a bounded distance. This deterministic bound does not require independence. Independence is required only for the probability statement, and using different LLMs does not automatically guarantee it.
Theorem 2 shows that averaging proposals reduces the mean squared distance to a reference point in the chosen representation space. Projecting the average back to the feasible domain preserves a related distance bound. Neither result directly guarantees lower objective regret.
Importantly, these are proposal-level guarantees, not end-to-end BO regret guarantees. An objective-value bound would require additional conditions connecting proposal distance to objective values, such as Lipschitz continuity and proximity to a global optimizer.
We also provide a mathematical analysis of the LLM–GP transition. With λ0=0.80, choosing γ=4 limits direct LLM selection to at most 5% of non-random queries after the midpoint. This explains the decay schedule but does not imply a universal regret guarantee. We explicitly show why our method does not inherit LLINBO's cumulative no-harm guarantee.
These clarifications separate what our theory establishes from what is supported by experiments. See Sections 2.3–2.4 and Appendices A.1, A.2, and A.9.