acceptodds
Under review as a conference paper at ICLR 2027

ASAP: Agent Selection Across Proposers for Hyperparameter Optimization under Wall-Clock Budgets

Abstract

Hyperparameter Optimization (HPO) is essential for maximizing machine learning model performance, and its core challenge is sample efficiency: finding strong configurations within a limited budget. Because every HPO tool relies on a surrogate prior that imparts its own inductive bias, individual tools struggle once problems become sufficiently diverse and drift from these priors. Motivated by the reasoning and generalization capabilities of LLMs, recent work has explored using LLMs for HPO and reports improved per-iteration performance. Yet these methods share two limitations with a common origin: they use the LLM as a single-tool replacement evaluated by iteration count. (i) Deployed in place of prior tools, the LLM is itself constrained by its pretraining objective to one family of inductive-biased proposals; this single-source setup still fails to handle the full diversity of problems. (ii) Per-iteration evaluation ignores that, in real runs, LLM inference or tool execution is paid serially on top of model evaluation every round, so iteration-count gains do not translate into end-to-end wall-clock gains. We present ASAP, an agent-system co-design that addresses both limitations. On the agent side, ASAP uses the LLM to integrate a diverse pool of inductive-biased optimizers and to select among their proposals each round. On the system side, ASAP re-architects the loop to reduce end-to-end wall-clock while preserving regret quality: a prefix-stable prompt maximizes KV-cache reuse across rounds; speculation parallelism hides the remaining LLM and tool latency under model evaluation via a relative-error accept test; and a Self-Tuner adapts the speculation threshold from execution logs off the critical path. Extensive experiments on diverse modern HPO tasks show that ASAP consistently outperforms baselines, underscoring the value of tool integration and agent-system co-design.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Efficient LLM Agents, Agent-System CoDesign, Hyperparameter Optimization, AutoResearch

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_tnWUOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes ASAP, an HPO framework in which an LLM selects among candidate configurations generated by eight heterogeneous optimizers. The Judge estimates candidate performance through in-context regression and selects configurations using Expected Improvement. The framework also introduces cache-friendly prompts and cross-iteration speculation to overlap proposal and selection overhead with model evaluation. Experiments cover nine real-training HPOBench combinations and nineteen PD1 replay combinations. The serial framework reports lower normalized regret than individual optimizers and alternative selectors, while speculative execution is estimated to reduce runtime by 12–24% on five HPOBench combinations.

Critical strengths
  1. The wall-clock-first framing addresses a real limitation of LLM-based HPO in practice. Iteration-based comparisons can obscure substantial proposal and inference overhead, so explicitly examining when improved selection quality translates into time savings is valuable. The paper considers evaluation costs spanning several orders of magnitude and relates the usefulness of speculation to the ratio between evaluation cost and decision overhead. Appendix C.6 also reports a meaningful boundary case where TPE outperforms ASAP under a matched time budget. This makes it clear where the method actually helps.
  2. The integration mechanism is simple and well-motivated. The ablations support it. Table 1a reports average final normalized regret of 0.026 for ASAP, compared with 0.046 for both uniform and GP-EI selection over the same candidate pool, and 0.051 for standalone LLAMBO. Keeping the candidate pool fixed makes this a particularly useful comparison: it provides direct evidence that the LLM Judge contributes additional optimization value. Separately evaluating serial ASAP also helps distinguish improvements in selection quality from the proposed speculative acceleration.
Critical weaknesses
  1. The main evaluation protocol does not fully establish superiority under matched wall-clock budgets. Although wall-clock performance is the primary claim, the main experiments cap every method at five initialization points plus thirty iterations (§5.1 and Appendix B.4). Proposal overheads and configuration-dependent evaluation costs differ across methods, yet the paper never tests whether faster baselines win when allowed to run until a common deadline. Appendix C.6 appropriately extends BO baselines to 500 trials for a fair seven-minute comparison and observes a ranking reversal. This does not imply that the main-suite ranking would reverse, but it demonstrates why fixed-evaluation and fixed-time comparisons cannot be treated interchangeably. A matched-time evaluation on representative main-suite tasks, reporting deadline regret and time to target, is needed to substantiate the central claim.
  2. The systems-efficiency evidence is limited relative to the strength of the quantitative acceleration claims. The 12–24% speedup comes from five HPOBench combinations, one seed, and runtimes composed from logged medians rather than measured end-to-end (Appendix B.4). As acknowledged in Appendix D, these estimates do not capture queueing or API-latency variation. The single-seed comparison also leaves uncertainty about the stability of the achieved regret, while the realized benefit of prefix caching remains unquantified. These limitations matter because practical wall-clock efficiency is a central contribution. Repeated asynchronous runs under matched resources, with uncertainty reported for both runtime and regret, would fix this.
Questions for authors

1 Matched Wall-Clock Budget Evaluation The main experiments cap all methods at five initialization points plus thirty iterations, which does not fully establish superiority under matched wall-clock budgets—the paper's primary claim. Given that proposal overheads and evaluation costs differ across methods, and Appendix C.6 already demonstrates a ranking reversal when BO baselines are extended to 500 trials within a seven-minute limit, how do you justify the central claim of wall-clock superiority without a time-matched comparison?

  1. Evidence and Robustness of Systems-Efficiency Claims The quantitative acceleration claims (12–24% speedup) are supported by evidence from only five HPOBench combinations using a single seed, with runtimes composed from logged medians rather than measured end-to-end. As acknowledged in Appendix D, this ignores queueing and API-latency variations, while the single-seed setup leaves uncertainty about regret stability and the realized benefit of prefix caching remains unquantified. Could you provide results from repeated asynchronous runs under matched resources, reporting uncertainty estimates for both runtime and regret to strengthen the evidence for wall-clock efficiency?
2 comments

Thank you for the supportive and careful review. Your point that fixed-evaluation and fixed-time comparisons cannot be treated interchangeably became the basis of the revision.

W1 / Q1. The main protocol does not establish superiority under matched wall-clock budgets.

The main experiment is now time-matched (Sec. 5.1, Fig. 3). ASAP runs 5 + 30 evaluations in its serial loop and pays its full propose + Judge overhead every round; each baseline is given the same end-to-end wall-clock on every task and is scored by the best configuration it has found at that deadline. On the four most expensive HPOBench combinations, all seven BO tools were re-run from scratch until ASAP's deadline, with no cap on the number of evaluations.

  • Deadline regret. ASAP ends at 0.026 over the 28 tasks, against 0.062 for TuRBO, 0.063 for SKOPT and 0.071 for LLAMBO, and it is lowest on both benchmarks: 0.066 on HPOBench (best baseline 0.077) and 0.007 on PD1 (best baseline 0.036).
  • Time to target. ASAP reaches each baseline's final regret within 52% of the budget (TuRBO; 43–46% for LLAMBO, SKOPT, SMAC, Optuna and TPE).
  • Uncertainty. Appendix C.3 gives 95% bootstrap intervals, medians and worst-case regret for every method; ASAP's largest regret on any task is 0.36, against at least 0.51 for every baseline.

W2 / Q2. The systems-efficiency evidence is limited relative to the quantitative claims.

We strengthened both the claim and its evidence.

  • The saving is now characterized by its mechanism. An accepted round saves the shorter of T_ML and the decision overhead, so the saving is set by ρ = T_ML / (T_tool + T_judge). Fig. 4 shows the five combinations on the analytical curve: 20% at ρ ≈ 3, 7–9% at ρ ≈ 8–10, and 1% at ρ ≈ 50, at matching regret and with 97% of speculative rounds accepted (Table 1b). This tells a practitioner in advance how much speculation will save on a given task.
  • Composed and measured wall-clock agree (Appendix C.7). Applying the composition rule to the per-round times logged by the asynchronous implementation reproduces its measured end-to-end wall-clock to within 0.03% on all five runs (0.3–0.5 s over 30 rounds). We compose the serial-vs-speculative comparison on one trajectory so that the two variants differ only in whether the decision overlaps the evaluation.
  • Prefix caching is now quantified (Sec. 4.2). With provider-side caching switched on and off on four PD1 studies, 27% of the Judge's prompt tokens are served from the cache and input cost falls by about 20%; per-call latency is the same with and without caching at our prompt lengths (median 1.35 s vs. 1.28 s), so ASAP's wall-clock results do not rely on it.

Finally, the main result (Fig. 3) is serial ASAP with its full overhead charged, so the regret conclusions do not depend on the speculation estimates; speculation is an additional saving on top, largest where evaluation time and decision overhead are comparable.

Other changes in the revision. For completeness, two further changes made in response to the other reviewers: (i) a BO-only pool ablation (Table 2, Appendix C.4): with every LLM-generated candidate removed, the Judge over the seven BO tools alone reaches 0.029, against 0.053 for random or GP-EI selection on the same pool, 0.064 for the best single BO tool and 0.036 for the per-task hindsight oracle, and restricting LLAMBO to its top-1 candidate gives 0.046 against 0.050 for the full pool, so the gain comes from the Judge's selection rather than from the LLM proposer. (ii) Sec. 3.2 now positions the Judge's scoring against LLAMBO's in-context surrogate.

Thank you again for the constructive feedback; it made the central claim both more precise and better supported.

Thank you for the thorough responses. I will keep my positive score.

Reviewer_LgsKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that LLM-based hyperparameter optimization should be judged by regret per unit of wall-clock time, and proposes ASAP. Each round, seven BO tools and LLAMBO propose candidates from a shared history, and an LLM Judge picks one by expected improvement over sampled in-context predictions. A speculative variant starts the next round on the Judge's predicted outcome, with a serial fallback. On 28 HPOBench and PD1 tasks with 35 evaluations each, ASAP reports the lowest mean regret of all compared methods, including LLAMBO and frontier LLM agents. Its main new elements relative to LLAMBO are the heterogeneous pool and the speculative overlap.

Critical strengths
  1. Controlled evidence that LLM selection adds value per evaluation: Table 1a holds ASAP's candidate pool fixed and varies only the selector. Random or GP-EI selection from the pool already beats the best single method, and the Judge lowers regret further, from 0.046 to 0.026; an offline replay on frozen PD1 pools agrees. This separates the pooling gain from the selection gain and is the strongest evidence in the paper.

  2. Speculative overlap is a useful systems idea. The decision overhead paid every round can rival the evaluation cost. Starting the next decision on the Judge's predicted outcome, with a serial fallback, hides part of it; the 12–24% wall-clock savings on five HPOBench settings are controlled estimates composed from logs (Appendix D). It is a second contribution with its own evaluation.

Critical weaknesses
  1. The paper does not show the wall-clock advantage where per-round overhead matters. Regret per unit wall-clock is the primary metric, and the paper claims the advantage "holds from minutes-long HPOBench evaluations to multi-hour PD1 training". For HPOBench, the paper does not yet show this:

    • All methods stop after the same 35 evaluations, and the right edge of the wall-clock plot (Figure 3a) closely tracks the per-iteration results.
    • On HPOBench, ASAP's proposal-and-Judge overhead is at least as long as an evaluation (Section 5.3). By my estimate from the paper's timing reconstruction for five such settings, it is roughly 82% of serial ASAP's time.
    • When time is matched in a cheaper regime, TPE beats ASAP (Appendix C.6). The main suite has no time-matched comparison.

    Figure 3 does charge ASAP's overhead, nothing in the main suite shows that the ranking would reverse, and the fixed-budget results stand. But where the wall-clock claim goes beyond the per-evaluation results, the paper does not yet support it. Since the title and the primary metric are about wall-clock budgets, this is my main concern. I suggest a common-time comparison on representative HPOBench tasks (existing logs may suffice), or limiting the claim to fixed evaluation budgets and evaluation-dominated settings.

  2. The Judge's scoring closely follows LLAMBO's, and no experiment isolates the selector conclusion. The Judge's prompt (Table 3) and its rule of scoring sampled predictions by expected improvement closely follow how LLAMBO's discriminative surrogate scores its own candidates. Details such as the templates and example order differ. Although Section 2 notes that LLAMBO uses an LLM as surrogate and acquisition function, the paper presents LLAMBO as "one more fixed-prior proposer" and never relates the Judge's scoring to it. The Judge's new step is therefore applying this scoring to a heterogeneous pool: effective, but narrower than presented. Separately, no experiment isolates the claim that the LLM helps most "not as one more proposer but as a selector". LLAMBO supplies 10 of the 17 candidates and over half of the committed configurations, and no BO-only pool with the same Judge is reported. This narrows the contribution but matters less to me than the wall-clock issue. I suggest positioning the Judge against LLAMBO, and either narrowing the role conclusion to this mixed pool or adding that ablation. With both issues addressed, I would support acceptance.

Questions for authors
  1. How do runs that have finished their 35 evaluations enter the time average in Figure 3a? On HPOBench, does ASAP's advantage hold over the interval in which all methods are still running?
  2. What is each method's per-round overhead on HPOBench? Section 4.1 treats a classical optimizer's proposal cost as negligible, but Appendix C.2 says HPOBench rounds "are bound by the BO tools rather than the LLM".
  3. Beyond templates, example order and candidate source, how does the Judge differ from LLAMBO's surrogate-plus-EI scoring? Is there evidence, such as a BO-only pool with the same Judge, that separates the LLM's selector and proposer roles?
  4. The 28 main tasks all have four-dimensional search spaces and 35 evaluations. Does the Judge's gain over random selection persist in higher-dimensional spaces or with larger budgets?
4 comments

Thank you for the thoughtful and precise review. Your two main suggestions, a common-time comparison and a BO-only pool, are now the main experiment and a main-text table of the revision.

W1. The paper does not show the wall-clock advantage where per-round overhead matters.

We followed your suggestion and made the common-time comparison the main protocol (Sec. 5.1). ASAP runs 5 + 30 evaluations in its serial loop, paying its full propose + Judge overhead every round, and each baseline is given the same end-to-end wall-clock on every task. On the four most expensive HPOBench combinations we re-ran all seven BO tools from scratch until ASAP's deadline, with no cap on the number of evaluations.

ASAP has the lowest regret at equal wall-clock on both benchmarks (Fig. 3, Sec. 5.2):

ASAPBest baselineBest BO tool
All 28 tasks0.0260.062 (TuRBO)0.062 (TuRBO)
HPOBench (9)0.0660.077 (Claude Opus 5)0.084 (TuRBO)
PD1 (19)0.0070.036 (SKOPT)0.036 (SKOPT)

The advantage therefore holds from minutes-long HPOBench evaluations to multi-hour PD1 training, with ASAP's full overhead charged. Intervals, medians and worst-case values are in Appendix C.3.

Q1. How do finished runs enter the time average? Does the advantage hold on HPOBench?

Every method is read at the same deadline on each task, ASAP's finish time, by the best configuration it has found up to that point. On HPOBench alone ASAP also ends with the lowest regret (0.066, against 0.077 for the best baseline and 0.084 for the best BO tool), and over all 28 tasks it has the lowest curve from about 15% of the budget onward (Fig. 3).

Q2. What is each method's per-round overhead on HPOBench?

ASAP's propose + Judge overhead on HPOBench is about 6 s per round. The tools propose in parallel, so the propose latency is that of the slowest BO tool, which is what "bound by the BO tools" referred to; a single BO tool's own proposal is at most a few seconds and small beside T_ML. On the speculated combinations this gives ρ ≈ 3–50 (Fig. 4), i.e. the decision overhead is a small share of each round.

W2 / Q3. The Judge's scoring closely follows LLAMBO's; no experiment isolates the selector role.

Thank you; we addressed both points.

  • Positioning. Sec. 3.2 now states that the Judge's scoring builds on the in-context surrogate LLAMBO uses to rank its own samples. ASAP's contribution is the role: it turns this scoring into a selector over a heterogeneous pool, ranking the proposals of optimizers with different priors against one shared history.
  • Isolating the selector. We added the BO-only pool with the same Judge (Table 2, Appendix C.4). The LLM never proposes a configuration there, and the Judge reaches 0.029, against 0.053 for random and GP-EI selection on the same pool, 0.064 for the best single BO tool, and 0.036 for the per-task hindsight oracle. Restricting LLAMBO to its top-1 candidate gives 0.046 against 0.050 for the full pool, so the result does not rest on LLAMBO's share of the pool. This directly supports the conclusion that the LLM contributes as a selector rather than as one more proposer (Sec. 5.3).

Q4. Higher-dimensional spaces and larger budgets.

Our tasks use the benchmarks' standard four-dimensional search spaces; higher-dimensional spaces are a natural next step and are noted in Appendix D. For larger budgets, we extended the PD1 experiments to 100 evaluations, where ASAP still outperforms both LLAMBO and random selection from the same pool (mean regret 0.026 vs. 0.076 and 0.032).

Other changes in the revision. For completeness, two further changes made in response to the other reviewers: (i) Appendix C.3 reports 95% bootstrap intervals, medians, worst-case regret and time-to-target for every method; ASAP's largest regret on any task is 0.36, against at least 0.51 for every baseline. (ii) The speculation saving is now reported as a function of ρ (20% at ρ ≈ 3 down to 1% at ρ ≈ 50; Fig. 4, Sec. 5.3), and the composed wall-clock reproduces the end-to-end wall-clock measured by the asynchronous implementation to within 0.03% (Appendix C.7).

Thank you again for stating clearly what would change your assessment. We hope the time-matched protocol and the BO-only pool address both issues.

Thanks for the detailed response. The BO-only pool experiment and the explanation in Section 3.2 address my second concern.

However, I still have two questions about the time-matched comparison:

  1. In the submission, the reported total time for serial ASAP across the five HPOBench tasks used in the speculation experiments was 532 minutes, with ρ ≤ 1. But in the revised Figure 4, the times add up to about 279 minutes, with ρ ranging from 3 to 52.5. Could you explain what changed?
  2. Your response says that all seven BO tools were rerun on the four most expensive HPOBench combinations, with no limit on evaluations, but Appendix B.4 describes 35 evaluations per method and does not mention the reruns. Could you update this section to explain them? Also, for the other HPOBench runs, were baselines allowed to keep going until ASAP finished, or did they stop after 35 evaluations? If any stopped early, could you use the existing logs to compare all methods at a common cutoff on each task, the time when the first method finishes its 35 evaluations?
  1. Why serial ASAP's time changed from 532 to 279 min, and ρ from ≤ 1 to 3–52.5.

Thank you for checking this closely. For the revision we carefully analyzed ASAP's end-to-end time per round on HPOBench, separating the tools' proposals, the Judge and the evaluation. This led to two corrections. Neither changes the configurations ASAP evaluates, so all regret values are unchanged. (a) Decision overhead. In our original prototype, each of the seven BO tools proposed by running a full ask–evaluate–tell step, so it also trained its own candidate in every round. These trainings were redundant: the Judge sees only the proposed configurations and the shared history, and their results were never used. In the submission they were counted as overhead (24–580 s per round), which gave ρ ≤ 1 and is why we described HPOBench rounds as "bound by the BO tools". We streamlined the infrastructure so that each tool runs only its "ask" step, and propose + Judge now takes about 6 s per round. (b) Evaluation time. ASAP is now charged the recorded training time of each configuration it evaluates, instead of a fixed per-task evaluation time. This is the same accounting we use for every baseline in Fig. 3. ASAP tends to select larger, more accurate models, so this correction increases its time. Together, these give the 279 min and ρ = 3–52.5 in Fig. 4. ASAP is accounted this way throughout the revision (Appendix B.4).

  1. Re-runs and the common cutoff. Thank you for pointing this out. Following your comment, we have updated the experiments of all methods on all tasks, and Appendix B.4 now describes the protocol fully. On all HPOBench and PD1 tasks, every baseline (the seven BO tools, LLAMBO, and the frontier LLM agents) is run until ASAP's wall-clock, so each makes as many evaluations as fit in that time. Every method is therefore compared at the same wall-clock on every task, and the common cutoff is not needed. Under this protocol ASAP has the lowest mean regret over the 28 tasks (0.026), about half that of the best baselines (SKOPT and SMAC, 0.053; LLAMBO 0.071; frontier LLM agents 0.068–0.095), and Fig. 3 and Table 1 are updated accordingly.

Thank you again for your careful reading and thoughtful suggestions, which have made the evaluation clearer and more rigorous. Please let us know if you have any further questions; we would be happy to clarify.

Thank you for the thorough responses and for all the additional work during the discussion. The new experiments and explanations resolve my concerns, and I have raised my score to weak accept.

Reviewer_27XPOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents ASAP (Agent Selection Across Proposers) which uses LLMs for hyperparameter optimization. It uses the LLMs to choose between optimizer proposed candidates every round. The paper evaluates on 28 HPOBench and PD1 tasks and the method reaches 0.026 normalized regret versus 0.051 for LLAMBO and 0.039 for the per-task best tool in hindsight. The paper also suggests methods to save wall clock time on some of the tasks.

Critical strengths

The method is the best performing method with strong headline results.

The authors take steps to improve the wall clock time.

Critical weaknesses

While the paper reports strong headline results, I have significant concerns about soundness:

Much of the gain comes from pooling. Uniform random selection from the pool already beats every single tool baseline. The pool is biased towards LLAMBO which provides 10 out of 17 candidates. There are important baselines missing such as:

  1. LLAMBO contributing its top-1 instead of all its candidates
  2. Judge over BO proposals only

Latency claims rely on a closed-source opaque API. A major part of the paper is wall-clock speedup, yet all LLM calls go through the OpenAI API where the queueing, load, caching etc. are outside the authors control. Thus the speculation comparison comes from the median logged per round times and is not measured end to end. The kv cache aware prompt is not measured (for future work). The latency claims should be evaluated with a self-hosted open model with explicit serving conditions ( for e.g. vLLM/SGLang with prefix caching on/off).

The frontier baselines (Claude, GPT 5.6) use AgentHPO, which proposes configurations directly, while ASAP and LLAMBO use surrogate function with GPT-3.5. Both model and method differ so it is hard to compare across them. The GPT 5.6 in ASAP experiment covers only 9 HPOBench tasks with one seed, and there is no LLAMBO baseline with a frontier model.

Appendix B.4 states that curves show shaded bands for the ±1 standard deviation, but the main result figure (fig. 3) shows no bands or other uncertainty for any method, and the headline numbers in table 1 are reported without intervals or significance tests.

Questions for authors

Please refer to weaknesses section and run experiments on the missing baselines, using open source models with controllable and measurable serving and add standard deviation to the tables.

4 comments

Thank you for the detailed review and for the concrete list of baselines. We have added them, and they strengthen the paper's central claim.

W1. Much of the gain comes from pooling; the pool is biased towards LLAMBO; missing baselines (LLAMBO top-1, Judge over BO proposals only).

We added both baselines (Table 2, Appendix C.4; all 28 tasks, two seeds, time-matched, GPT-4.1-mini for every LLM call):

VariantRegret ↓
BO-only pool + LLM Judge0.029
BO-only pool + uniform-random selector0.053
BO-only pool + GP-EI selector0.053
Best single BO tool (SKOPT)0.054
Per-task best BO tool (hindsight oracle)0.038
BO pool + LLAMBO top-1 only + LLM Judge0.046
Full pool + LLM Judge (ASAP)0.050
  • Judge over BO proposals only. The LLM never generates a configuration here. The Judge still nearly halves the regret of the best single BO tool and passes even the hindsight oracle that picks the best BO tool per task, because it can switch tools along a run. On the same pool, random and GP-EI selection stay at 0.053, so the gain is the Judge's and not only the pool's.
  • LLAMBO top-1. When LLAMBO contributes a single candidate, so that it weighs the same as each BO tool, the regret is 0.046 against 0.050 for the full pool. ASAP's result does not depend on LLAMBO's share of the pool.

W2. Latency claims rely on a hosted API; speculation is not measured end to end; the KV-cache-aware prompt is not measured; please use a self-hosted open model.

Thank you for raising this; let us first clarify what the paper claims. Our central claim is that integrating heterogeneous optimizers under an LLM Judge reaches lower regret at equal wall-clock than any single tool, LLAMBO, or a frontier LLM agent. The strongest competitors for this claim are the frontier models, which are available only through hosted APIs, so we evaluate every LLM-based method in the same hosted setting, which is also how these methods are used in practice. A self-hosted open model would mainly lower the quality of all LLM-based methods and make the comparison with frontier agents less fair; it would not test this claim. Within this common setting we strengthened the evidence in three ways:

  • The main result does not depend on any latency optimization. The ASAP curve in Fig. 3 is the serial loop: it pays its full measured propose + Judge time in every round and is compared with every baseline at equal wall-clock. API latency can only count against ASAP.
  • Speculation is validated against end-to-end measurement (Appendix C.7). Applying our composition rule to the per-round times logged by the asynchronous implementation reproduces its measured end-to-end wall-clock to within 0.03% on all five runs (0.3–0.5 s over 30 rounds).
  • Prefix caching is measured (Sec. 4.2). Switching provider-side caching on and off on four PD1 studies, 27% of the Judge's prompt tokens are served from the cache, which lowers input cost by about 20%. Per-call latency is the same with and without caching at our prompt lengths (median 1.35 s vs. 1.28 s), so ASAP's wall-clock results do not rely on provider-side caching.

Both mechanisms are model-agnostic. The speculation saving depends on the serving stack only through ρ, and the measured points follow the analytical curve over ρ ≈ 3–50 (Fig. 4), so a different stack moves a task along that curve rather than changing the conclusion.

W3. Frontier baselines use AgentHPO, so both model and method differ; GPT-5.6 in ASAP covers only 9 tasks; no LLAMBO with a frontier model.

The paper establishes ASAP's advantage under matched settings, where only the method differs:

  • ASAP vs. LLAMBO with GPT-3.5: 0.026 vs. 0.071 on all 28 tasks (Fig. 3, Table 1a).
  • ASAP vs. LLAMBO with GPT-4.1-mini, on all 28 tasks with two seeds: 0.045 vs. 0.106, with ASAP lower on 22 of the 28 tasks (Appendix C.4).

So with the backbone held fixed, integration wins under two different LLMs, and it is also less sensitive to the backbone than a pure LLM proposer. The AgentHPO baselines then answer a separate question: can a much stronger stand-alone LLM close the gap without integration? It cannot: the three frontier agents end at 0.068–0.095, against 0.026 for ASAP with a far smaller model (Sec. 5.2). Appendix C.2 completes the picture from the other direction by placing GPT-5.6 on both sides of ASAP, where a stronger model does not lower the regret. Together these show that the gain comes from integration rather than from the model.

W4. No uncertainty bands, intervals or significance tests.

Thank you for catching this; we corrected the statement in Appendix B.4 and added Appendix C.3, which reports for every method the mean with a 95% bootstrap interval over the 28 tasks, the median and the worst-case regret (Table 1 points to it). ASAP has the lowest mean and the narrowest interval (0.026 [0.008, 0.056]; TuRBO 0.061 [0.025, 0.110]; LLAMBO 0.071 [0.015, 0.143]). ASAP's largest regret on any task is 0.36, while every baseline has at least one task above 0.51, and ASAP reaches each baseline's final regret within 60% of the budget.

Other changes in the revision. For completeness, two further changes made in response to the other reviewers: (i) the main experiment is now time-matched, so every baseline is given the same end-to-end wall-clock budget as ASAP on every task (Sec. 5.1, Fig. 3); under this protocol ASAP ends at 0.026 over the 28 tasks, against 0.062 for TuRBO and 0.071 for LLAMBO, and is lowest on both HPOBench and PD1. (ii) Sec. 3.2 now positions the Judge's scoring against LLAMBO's in-context surrogate. The speculation saving is also reported as a function of ρ rather than as one aggregate number (Fig. 4, Sec. 5.3).

We hope these additions resolve your soundness concerns, and we would be glad to clarify anything further.

Dear Reviewer, as the discussion deadline is approaching, we would greatly appreciate it if you could take a look at our responses and the revised PDF. Thank you for your time!

Dear Reviewer, thank you again for your review. As the discussion deadline is close, we would be grateful if you could take a look at our response and the revised paper.

On your two main points: (i) you noted that uniform-random selection over the pool already matches the best single tool, so much of the gain could come from pooling; we added both baselines you noted as missing (the LLM Judge over BO proposals only, and LLAMBO contributing its top-1 instead of all its candidates), and on the same BO-only pool the Judge reaches a regret of 0.029 against 0.053 for random selection, so the gain comes from the Judge rather than from pooling. (ii) Every experimental setting was chosen to make the comparison strictly fair: all LLM-based methods share the same backbone and the same hosted setting as reported in the paper, and every baseline gets the same wall-clock budget as ASAP.

We would be happy to clarify anything further before the deadline.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response (to all reviewers and the AC)

We sincerely thank the Area Chair and all three reviewers for their time, careful reading and constructive suggestions. We have revised the paper accordingly; the main changes are:

  1. The main experiment is now time-matched (Reviewers LgsK, tnWU). Every baseline is given the same end-to-end wall-clock budget as ASAP on every task and is scored by the best configuration it has found when that budget ends (Sec. 5.1, Fig. 3, Table 1a). ASAP reaches a mean normalized regret of 0.026 over the 28 tasks, against 0.062 for TuRBO, 0.063 for SKOPT and 0.071 for LLAMBO, and it is the lowest on both HPOBench and PD1.
  2. A BO-only pool and a LLAMBO top-1 ablation (Reviewers 27XP, LgsK). With every LLM-generated candidate removed, the Judge over the seven BO tools alone reaches 0.029, against 0.053 for random or GP-EI selection on the same pool and 0.064 for the best single BO tool (Table 2, Appendix C.4).
  3. Uncertainty for the main comparison (Reviewer 27XP). Appendix C.3 reports 95% bootstrap intervals, medians, worst-case regret and time-to-target for every method.
  4. Speculation is characterized by ρ and validated against end-to-end measurements (Reviewers 27XP, tnWU). The saving is reported as a function of ρ = T_ML / (T_tool + T_judge) (Fig. 4, Sec. 5.3), and the composed wall-clock matches the wall-clock measured by the asynchronous implementation (Appendix C.7).
  5. The Judge is positioned against LLAMBO's surrogate (Reviewer LgsK) in Sec. 3.2.

Each point is addressed in detail in our individual responses to the reviewers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.