- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes ASAP, an HPO framework in which an LLM selects among candidate configurations generated by eight heterogeneous optimizers. The Judge estimates candidate performance through in-context regression and selects configurations using Expected Improvement. The framework also introduces cache-friendly prompts and cross-iteration speculation to overlap proposal and selection overhead with model evaluation. Experiments cover nine real-training HPOBench combinations and nineteen PD1 replay combinations. The serial framework reports lower normalized regret than individual optimizers and alternative selectors, while speculative execution is estimated to reduce runtime by 12–24% on five HPOBench combinations.
Critical strengths
- The wall-clock-first framing addresses a real limitation of LLM-based HPO in practice. Iteration-based comparisons can obscure substantial proposal and inference overhead, so explicitly examining when improved selection quality translates into time savings is valuable. The paper considers evaluation costs spanning several orders of magnitude and relates the usefulness of speculation to the ratio between evaluation cost and decision overhead. Appendix C.6 also reports a meaningful boundary case where TPE outperforms ASAP under a matched time budget. This makes it clear where the method actually helps.
- The integration mechanism is simple and well-motivated. The ablations support it. Table 1a reports average final normalized regret of 0.026 for ASAP, compared with 0.046 for both uniform and GP-EI selection over the same candidate pool, and 0.051 for standalone LLAMBO. Keeping the candidate pool fixed makes this a particularly useful comparison: it provides direct evidence that the LLM Judge contributes additional optimization value. Separately evaluating serial ASAP also helps distinguish improvements in selection quality from the proposed speculative acceleration.
Critical weaknesses
- The main evaluation protocol does not fully establish superiority under matched wall-clock budgets. Although wall-clock performance is the primary claim, the main experiments cap every method at five initialization points plus thirty iterations (§5.1 and Appendix B.4). Proposal overheads and configuration-dependent evaluation costs differ across methods, yet the paper never tests whether faster baselines win when allowed to run until a common deadline. Appendix C.6 appropriately extends BO baselines to 500 trials for a fair seven-minute comparison and observes a ranking reversal. This does not imply that the main-suite ranking would reverse, but it demonstrates why fixed-evaluation and fixed-time comparisons cannot be treated interchangeably. A matched-time evaluation on representative main-suite tasks, reporting deadline regret and time to target, is needed to substantiate the central claim.
- The systems-efficiency evidence is limited relative to the strength of the quantitative acceleration claims. The 12–24% speedup comes from five HPOBench combinations, one seed, and runtimes composed from logged medians rather than measured end-to-end (Appendix B.4). As acknowledged in Appendix D, these estimates do not capture queueing or API-latency variation. The single-seed comparison also leaves uncertainty about the stability of the achieved regret, while the realized benefit of prefix caching remains unquantified. These limitations matter because practical wall-clock efficiency is a central contribution. Repeated asynchronous runs under matched resources, with uncertainty reported for both runtime and regret, would fix this.
Questions for authors
1 Matched Wall-Clock Budget Evaluation The main experiments cap all methods at five initialization points plus thirty iterations, which does not fully establish superiority under matched wall-clock budgets—the paper's primary claim. Given that proposal overheads and evaluation costs differ across methods, and Appendix C.6 already demonstrates a ranking reversal when BO baselines are extended to 500 trials within a seven-minute limit, how do you justify the central claim of wall-clock superiority without a time-matched comparison?
- Evidence and Robustness of Systems-Efficiency Claims The quantitative acceleration claims (12–24% speedup) are supported by evidence from only five HPOBench combinations using a single seed, with runtimes composed from logged medians rather than measured end-to-end. As acknowledged in Appendix D, this ignores queueing and API-latency variations, while the single-seed setup leaves uncertainty about regret stability and the realized benefit of prefix caching remains unquantified. Could you provide results from repeated asynchronous runs under matched resources, reporting uncertainty estimates for both runtime and regret to strengthen the evidence for wall-clock efficiency?
2 comments
Thank you for the thorough responses. I will keep my positive score.
Thank you for the supportive and careful review. Your point that fixed-evaluation and fixed-time comparisons cannot be treated interchangeably became the basis of the revision.
W1 / Q1. The main protocol does not establish superiority under matched wall-clock budgets.
The main experiment is now time-matched (Sec. 5.1, Fig. 3). ASAP runs 5 + 30 evaluations in its serial loop and pays its full propose + Judge overhead every round; each baseline is given the same end-to-end wall-clock on every task and is scored by the best configuration it has found at that deadline. On the four most expensive HPOBench combinations, all seven BO tools were re-run from scratch until ASAP's deadline, with no cap on the number of evaluations.
W2 / Q2. The systems-efficiency evidence is limited relative to the quantitative claims.
We strengthened both the claim and its evidence.
Finally, the main result (Fig. 3) is serial ASAP with its full overhead charged, so the regret conclusions do not depend on the speculation estimates; speculation is an additional saving on top, largest where evaluation time and decision overhead are comparable.
Other changes in the revision. For completeness, two further changes made in response to the other reviewers: (i) a BO-only pool ablation (Table 2, Appendix C.4): with every LLM-generated candidate removed, the Judge over the seven BO tools alone reaches 0.029, against 0.053 for random or GP-EI selection on the same pool, 0.064 for the best single BO tool and 0.036 for the per-task hindsight oracle, and restricting LLAMBO to its top-1 candidate gives 0.046 against 0.050 for the full pool, so the gain comes from the Judge's selection rather than from the LLM proposer. (ii) Sec. 3.2 now positions the Judge's scoring against LLAMBO's in-context surrogate.
Thank you again for the constructive feedback; it made the central claim both more precise and better supported.