- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
NextEval is an agent-controlled derivative-free optimization system. An LLM selects numerical tools, parameters, and evaluation allocations, while numerical software computes candidate points and maintains a shared evaluation history and method-specific continuation state. The agent receives oracle observations rather than the objective formula or benchmark identity. The paper also proposes a common environment for comparing LLM–harness configurations. Experiments use 92 unconstrained problems with at most five variables and a budget of 50n objective evaluations. The primary rotated, noiseless comparison uses ten repetitions and reports the highest normalized evaluation-efficiency score against COBYQA, NEWUOA, Powell, and finite-difference BFGS.
Critical strengths
- The division between strategic control and numerical computation is sensible and reusable. The LLM decides which operations to perform, while numerical methods handle interpolation, trust-region steps, line search, and candidate evaluation. Maintaining method-specific state across switches is an important implementation detail: the system can combine methods without discarding all previous numerical work. This is a more substantive design than asking an LLM to produce numerical evaluation points directly.
Critical weaknesses
- The key comparison needed to establish the value of the LLM controller has not been completed. The system combines an LLM with a substantial numerical toolset, shared history, continuation state, and restart capabilities. Comparison with standalone solvers cannot distinguish the contribution of agent decisions from these additional numerical capabilities. A deterministic controller or fixed strategy using the same environment would directly address this issue. Appendix E explicitly lists controller-replacement and numerical-support ablations as planned tests, not results, leaving the central mechanism insufficiently isolated.
- The evidence is narrow relative to the broader promise of an optimization solver. All primary problems are public and have at most five variables, and there is no demonstrated generalization to unseen problem families. Rotations and perturbed starts do not resolve that concern. Moreover, performance is assessed through objective-evaluation counts rather than total runtime or LLM cost. The results therefore support a low-dimensional benchmark advantage, but leave both out-of-family robustness and the practical trade-off between saved objective calls and agent overhead unresolved.
Questions for authors
- The reported improvements are obtained by the complete system, which combines an LLM controller with multiple numerical optimization tools, shared evaluation history, and persistent method states. Could the authors compare NextEval against a deterministic or heuristic controller with access to exactly the same numerical tools and information? Without this comparison, it is difficult to determine whether the improvements arise from LLM-based decision making or from the underlying numerical infrastructure.
- The experiments compare NextEval against standalone classical DFO solvers. Could the authors include stronger adaptive numerical baselines, such as algorithm portfolios, restart strategies, or automated method-selection policies? These would provide a more direct comparison with NextEval's ability to switch between numerical methods.
- All 92 benchmark problems are public, low-dimensional test functions, and the paper acknowledges that development feedback informed the tools and prompts. Could the authors evaluate NextEval on previously unseen problem families or independently constructed black-box optimization tasks? This would help establish whether the reported improvements generalize beyond the development benchmark.
- The experiments are restricted to problems with at most five decision variables. How does NextEval perform on moderately higher-dimensional problems (e.g., 10, 20, or 50 variables)? Since the numerical methods and the effectiveness of their combinations can change substantially with dimension, it is unclear whether the current results extend beyond very small problems.
6 comments
Response to Reviewer mzDT (Part 2/2)
Response to Reviewer mzDT (Part 2/2)
Q3 / W2. Unseen families or independently constructed tasks. Partly. Independently constructed tasks are not in this revision; they are a main extension of the next Bench version (§6). We agree that rotations and perturbed starts do not settle the concern: they reduce but do not exclude problem recognition (§6). The evidence we have now (§4.4; App. D):
- n = 20, never used in development (8 problems, defined by a rule written before any run on it). Sol is S+ against the textbook policy (0.682, 0.701), also without simple-point problems (0.649, 0.685; four or five problems remain). Case by case, Sol (high) beats the textbook policy in 49 of the 96 cases and loses 6; Sol (medium) wins 43 and loses 9 (App. E). Six of its eight families also occur at smaller n in the Bench or Dev. 2–5 (§6).
- n = 15, never used in development (18 problems). Sixteen are DIXMAAN variants with optimum at the origin, which Sol evaluates itself in 77% and 86% of the cases, so we do not use this pool as evidence of generalization.
- Development pools share no problem with the Bench but were used in earlier development, so we do not call them held out.
Q4. Moderately higher dimensions (10, 20, 50). Up to n = 20 (§4.4, Tables 1a, 11): against the textbook policy, Sol (high) scores 0.711 on the Bench, 0.764 at n = 6–10, and 0.701 at n = 20, all S+, also without simple-point problems (0.718, 0.768, 0.685). Luna moves from inconclusive at n ≤ 5 to S+ at n = 6–10 (0.575). The textbook policy itself falls against COBYQA from 0.598 to 0.442 at n = 20. We have not run n = 50. At n = 20 a case already uses 2,012k (Sol medium) and 2,624k (Sol high) tokens on average (Table 12); larger dimensions are a main extension of the next Bench version (§6).
W2. Runtime and LLM cost. Evaluation counts remain the score, as is standard when evaluations are expensive; cost and time are now reported (§4.4; Tables 12–13; Fig. 4a; list prices of 2026-11-04).
- Bench, per case: USD 0.011 and 104 s (Luna), 0.121 and 116 s (Sol medium), 0.186 and 199 s (Sol high). At n = 6–10: USD 0.029, 0.250, and 0.391; at n = 20: USD 0.069, 0.377, and 0.498. The textbook policy alone has no model cost and takes 1.0 to 21.5 s.
- Solver 1.0 with Sol (high) costs about twice the best native agent (USD 0.186 vs. 0.099), and their scores against COBYQA have overlapping intervals (0.738 vs. 0.726). Solver 1.0 with Luna reaches 0.602 at USD 0.011; both join the cost–score front of the native agents (Fig. 4a).
- Solver sessions ran as processes on a shared host, native agents in per-case containers (1 CPU, 2 GiB), so the times describe each setting and are not a matched comparison (App. D).
LLM control is meant for objectives whose evaluations are costly, where cents and minutes per problem are small against one evaluation.
On the strength you noted (the LLM does not produce evaluation points directly). Solver 1.0 differs from the submission here: numerical methods still compute most points, but the agent can now also evaluate points of its own, stated explicitly or returned by analysis code that it writes (§2.4, App. I). These are 20.9–35.1% of the evaluations of Sol outside n = 15 and 0.8–1.6% of those of Luna (Table 19). The comparison measures the LLM policy over the whole decision space; it does not separate the campaigns from these points (App. E). With Luna, which hardly uses them (1.3% at n = 6–10), the LLM policy is still S+ there (0.575).
Thank you for the substantial additional experiments.
Two key issues remain. First, Solver 1.0 now allows autonomous point generation, so the gains do not isolate the benefit of numerical-method orchestration. A strong-model ablation disabling autonomous point generation would help clarify this. Second, six of the eight 20-dimensional families overlap with earlier pools, leaving limited evidence of generalization to genuinely unseen objective structures. Separate results for the two non-overlapping families would be informative.
I appreciate the progress, but these remaining limitations leave me unconvinced that the evidence sufficiently supports the central contribution and generality claims. I therefore maintain my rating of 2.
Response to Reviewer mzDT (Part 1/2)
Response to Reviewer mzDT (Part 1/2)
Thank you for four concrete questions; each now has an experimental answer, in part for Q3 and Q4.
Verdicts (§4.1): two-level 90% intervals, margin δ = 0.025; S+ superior beyond the margin, S superior, E equivalent, ? inconclusive.
W1 / Q1. Deterministic controller with exactly the same tools and information. Done: the LLM and a deterministic textbook controller (the textbook autopilot, §2.4) act in the same decision space, and their head-to-head comparison was fixed before the corresponding model calls (§4.4). The textbook policy runs trust-region slices with library and textbook defaults and resets after a stall. Every call it makes passes the agent's schema, so it has no tool or information that the agent lacks (App. D). Head-to-head, three repetitions (Table 1a):
† Most optima at the origin, which Sol evaluates itself; not used as evidence (Part 2).
Q2. Stronger adaptive baselines. Added, each in a fixed configuration (App. F): restarted COBYQA and NEWUOA; COBYQA with a noise-aware final radius and restarts; Py-BOBYQA with noise handling and restarts; sequential and interleaved COBYQA–NEWUOA portfolios; NGOpt (automated algorithm selection); IPOP-CMA-ES; NOMAD; GP-EI; default Py-BOBYQA. With default NEWUOA from the submission, there are 12.