- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes online LLM-based automated algorithm design (AAD): instead of designing an algorithm offline and deploying it fixed, LLM agents re-synthesize the algorithm at every stage of the target run, conditioned on the current optimization state. OnDesign has three modules: (1) State Analysis, where an LLM writes a state report from runtime observations, guided by a State Analysis Guideline (SAG); (2) Multi-Agent Deliberation, with exploration- and exploitation-oriented Proposers and an Arbiter that writes the code; and (3) SAG Evolution, which periodically refines the SAG using the execution gains credited to it. The method is instantiated for Bayesian optimization (BO), where it generates acquisition functions, and for evolutionary continuous and mixed-variable optimization, where it generates offspring operators. It reports the best average rank in all 17 suite–dimension configurations (Tables 1–2) and the best result on an LS-DYNA crashworthiness problem (Fig. 3).
Critical strengths
- Treating the algorithm as a state-dependent decision variable is a natural step for LLM-based AAD, and Figure 1 and Section 2 lay out the distinction from offline design well.
- The paper covers three scenarios, six benchmark suites, and one real expensive engineering problem.
- Figures 15–17 trace one design decision end to end, which makes the mechanism concrete.
Critical weaknesses
1. The BO gains are implausibly large for the design space and need an implementation-parity check
The design space is narrow. Every generated acquisition function has the signature acquisition(best_y, mean, std) (Figure 5): a pointwise function of the GP posterior mean and standard deviation, maximized to pick the next point (Appendix C, step 4). This is the same family as LogEI, LogPI, and UCB. All methods share the same GP: a Matérn-5/2 kernel on standardized observations (Appendix H.2).
The generated functions are close to existing baselines. The case-study output (Equation 37, Figure 17) is:
Since is monotone, the first term has the same maximizer as EI. Since for any non-negligible , the second term is effectively with , a smooth PI-like term. So the generated rule is approximately EI plus a small PI-like term, both already baselines (LogEI, LogPI).
Yet the reported gaps span many orders of magnitude. For example, Table 15 (BO, CEC2026, ) shows OnDesign vs. the best baseline:
- F4: vs. (LogPI)
- F6: vs. (JES)
- F15: vs. (MEoH)
- F16: vs. (JES)
- F18: vs. (ATRBO-LCB)
- F22: (std ) vs. (SETUP-BO)
- F28: vs. (LogEI)
Table 10 (BBOB, ) shows similar gaps (e.g., F9: vs. ; F20: vs. ). With the same GP, the same 161 evaluations, and acquisition rules close to EI and PI, gaps of to are hard to explain by the acquisition formula alone. The paper does not specify:
- how the argmax in Appendix C, step 4 is computed for each method (candidate set or gradient-based optimizer, number of restarts);
- whether baselines such as JES, SETUP-BO, FunBO, and TREvol run through the same candidate-selection routine or their own implementations;
- input normalization and boundary handling per method.
2. OnDesign finishes last on a consistent set of functions, which average rank hides
On the following functions, OnDesign is last among 12 methods, while the baselines cluster tightly:
- Table 13 (CEC2022, ): F3 ( vs. baselines at –); F5 ( vs. baselines at –).
- Table 14 (CEC2022, ): F3 ( vs. –); F5 ( vs. –).
- Table 11 (CEC2020, ): F8 ( vs. –).
- Table 12 (CEC2020, ): F8 ( vs. –).
- Table 15 (CEC2026, ): F5 ( vs. ); F8 ( vs. –); F9 ( vs. –); F21 ( vs. –).
These losses account for most of the baselines' "+" counts (e.g., LogEI's 4/25/0 on CEC2026 BO consists exactly of F5, F8, F9, and F21). Average rank conceals both extremes, because a win and a loss each move the rank by a fixed amount. Together, extreme wins and systematic losses suggest OnDesign's search behaves structurally differently rather than uniformly better (e.g., a different region or boundary preference in candidate selection).
3. The central claim, that state-conditioned redesign helps, is not isolated by any experiment
The paper's thesis is that the algorithm should be designed from the current optimization state (Section 2, Equation 3), but none of the ablations tests this. w/o SA (Table 3) still passes runtime observations directly to the designer, and w/o MAD and w/o SAGE keep full state conditioning.
The feedback signal is also very sparse. In BO, a new acquisition function is generated for every evaluation (Algorithm 1), so each design is judged on a single sample, and the gain is zero whenever the incumbent does not improve (Equation 31). In the case study (Figure 4), only 8 of 100 evaluations improve the incumbent, meaning roughly 92% of archived gains are zero. The SAG scores and the archive the Proposers reason over therefore carry little signal about whether a design was good. A simpler explanation is that the benefit comes from diversity (frequently switching among reasonable EI-like rules) rather than from reading the state.
4. Strong and closely related baselines are missing
- CMA-ES is cited (Hansen & Ostermeier, 2001, Section 1) but not run; it is the standard reference optimizer on BBOB.
- L-SHADE or jSO, the standard strong DE variants on CEC suites. SHADE (2013) is the strongest DE baseline included.
- TuRBO or HEBO for the 30-D BO settings, where vanilla GP-BO is known to struggle.
- Ngo et al. (2026), LLM-based adaptive acquisition selection for BO, cited in Section 1. It is the closest prior work on online LLM adaptation and directly tests whether synthesizing new logic beats selecting among predefined acquisition functions. AutoEP, the only online LLM baseline, appears only in the EC continuous setting.
Several included baselines (CoDE, CLPSO, ParEvo1) are weak and widen the rank gaps without informing the comparison.
5. LLM cost is unreported, and the fixed-evaluation comparison favours OnDesign on cheap benchmarks
The paper fixes the objective-evaluation budget but reports no LLM usage. Estimated from Algorithms 1–3:
- BO: Each of the 100 stages calls the State Analyst, two Proposers, and the Arbiter (4 calls), plus a SAG refinement every 10 stages, giving about 410 LLM calls per run, or roughly calls over 121 function–dimension configurations 10 runs.
- EC: 5,000 evaluations with gives about 49 generations, so about 200 calls per run, or roughly more over the 199 continuous and mixed-variable configurations 10 runs.
Offline AAD amortizes its design cost across tasks; OnDesign pays it on every run. A fixed-evaluation budget is the right axis for the LS-DYNA problem (Appendix H.4), where each evaluation is a finite-element simulation. For BBOB and CEC functions, which evaluate in microseconds, the same compute could buy orders of magnitude more evaluations for CMA-ES or L-SHADE.
Questions for authors
- How is the acquisition function maximized for each BO method, and do the CEC2026 gaps survive replaying OnDesign's generated acquisition functions through the LogEI pipeline?
- Why does OnDesign finish last among all 12 methods on CEC2022 F3/F5 (Tables 13–14) and CEC2026 F8/F9 (Table 15)?
- How do state-blind online redesign and random switching over an LLM-generated pool perform?
- What are the LLM calls, tokens, wall-clock time, and cost per run?
- What were the training problems and design budgets for EoH, MEoH, and ReEvo?
- Are the per-function differences statistically significant (e.g., Wilcoxon rank-sum)?
- Is Figure 13 measured data, and what does "adjusted" mean?
Thank you for the rebuttal. The reported runtimes and token counts show that online redesign is costly. OnDesign performs better than the variant without state feedback. Both use multi-agent deliberation and make a similar number of LLM requests.
The response leaves one question open. It is still unclear how the theoretical error terms relate to the observed gains. I maintain my rating of 3.