- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces LLMSys-HPOBench, a tabular benchmark for hyperparameter optimization of LLM systems, covering seven open-source systems across RAG pipelines, inference engines, and agent frameworks. For each system, it defines a mixed space of AI and non-AI inference-time hyperparameters and a multi-factor fidelity space, and releases 364450 measured configurations with objective metrics, cost metrics, hardware statistics, and execution logs. A unified API supports lookups, with a random forest surrogate for unmeasured configurations.
Critical strengths
The authors profiled over 364k configurations across seven real LLM systems using about 94900 GPU-hours, and released the results together with cost metrics, hardware statistics, and execution logs. This is the main strength in my assessment, since evaluating configurations of LLM systems is expensive.
Critical weaknesses
My major concern is the practical value of this tabular benchmark.
- All measurements are collected on a single hardware setup (RTX 5090), with small models and mostly one dataset per system. A tabular benchmark is useful only if conclusions drawn from it hold on real tuning problems, but the paper does not test this.
- The configuration spaces leave out the components that most strongly affect current LLM systems, such as prompts and tool design, especially for the RAG and agent systems.
Although the discussion outlines several research opportunities, the paper only touches on a few of them at a preliminary level, and none leads to new methods or actionable findings. As a result, the current contribution reads mainly as a costly data collection effort rather than a benchmark with demonstrated value for HPO research or practice.
Questions for authors
- Section 3.1 claims to include all tunable hyperparameters, yet the spaces include likely irrelevant ones (e.g., log_level in SGLang) and exclude prompts, with tools limited to on/off switches. How were hyperparameters selected, and why were prompts and tool design excluded?
- Beyond the speedup in Appendix C, can the authors show a concrete case where the benchmark leads to a measurable improvement on a real LLM system?
- Appendix E only tests ranking stability within the table. Can the authors show that optimizer rankings from the table match those from running directly on a real system, or under a different model or hardware?
4 comments
| System | Method | Default | Tuned (mean ± SD) | Change (pct.) | Rank |
|---|---|---|---|---|---|
| NaiveRAG () | Hyperband | 0.5079 | 0.6832 ± 0.0505 | 34.5 | +1.00 |
| BOHB | 0.6667 ± 0.0615 | 31.3 | |||
| SMAC | 0.7090 ± 0.0330 | 39.6 | |||
| HEBO | 0.6736 ± 0.0683 | 32.6 | |||
| Random search | 0.6881 ± 0.0593 | 35.5 | |||
| LightRAG () | Hyperband | 0.0556 | 0.0639 ± 0.0260 | 15.0 | +0.80 |
| BOHB | 0.0638 ± 0.0200 | 14.9 | |||
| SMAC | 0.0655 ± 0.0170 | 17.9 | |||
| HEBO | 0.0698 ± 0.0169 | 25.7 | |||
| Random search | 0.0652 ± 0.0220 | 17.4 | |||
| vLLM () | Hyperband | 6.271 | 6.556 ± 0.316 | 4.5 | +1.00 |
| BOHB | 6.739 ± 0.521 | 7.5 | |||
| SMAC | 6.870 ± 0.959 | 9.5 | |||
| HEBO | 6.723 ± 0.482 | 7.2 | |||
| Random search | 6.592 ± 0.294 | 5.1 | |||
| HTML-RAG () | Hyperband | 0.6100 | 0.5587 ± 0.1089 | +0.40 | |
| BOHB | 0.5357 ± 0.1144 | ||||
| SMAC | 0.5795 ± 0.1004 | ||||
| HEBO | 0.5613 ± 0.0976 | ||||
| Random search | 0.5503 ± 0.0733 | ||||
| OpenHands () | Hyperband | 1.0000 | 0.9333 ± 0.1356 | +0.95 | |
| BOHB | 0.9389 ± 0.1417 | ||||
| SMAC | 0.9556 ± 0.1152 | ||||
| HEBO | 0.9556 ± 0.1152 | ||||
| Random search | 0.9278 ± 0.1496 | ||||
| SGLang () | Hyperband | 0.760 | 0.917 ± 0.249 | 20.6 | |
| BOHB | 0.957 ± 0.234 | 25.9 | |||
| SMAC | 0.941 ± 0.224 | 23.7 | |||
| HEBO | 0.904 ± 0.275 | 18.8 | |||
| Random search | 0.922 ± 0.217 | 21.2 | |||
| AutoGPT () | Hyperband | 0.00554 | 0.02394 ± 0.01330 | 332.3 | +0.40 |
| BOHB | 0.02753 ± 0.02142 | 397.1 | |||
| SMAC | 0.03748 ± 0.03552 | 576.8 | |||
| HEBO | 0.02522 ± 0.01438 | 355.4 | |||
| Random search | 0.02295 ± 0.01829 | 314.5 |
Table: R1.1. Real-system performance of the default configuration and configurations recommended by benchmark-based tuners at the target fidelity setting. indicates a maximization objective. Change reports the percentage change relative to the default. Rank () is the Spearman correlation between tuners rankings obtained on LLMSYS-HPOBench and those obtained after reevaluating the recommended configurations on the real system.
Thank you for the response and the new experiment. My main concern about practical value remains. No results are provided on other hardware, models, or datasets. Unlike prior benchmarks cited in the paper (e.g., HPOBench, NAS-Bench-201), which cover multiple datasets, six of the seven systems here use a single dataset, so it is unclear whether conclusions transfer to real tuning problems. I will keep my current score.
Thank you for the follow-up. We would like to further clarify the scope and practical value of LLMSYS-HPOBench.
- Different data-collection setting. Building a benchmark for complete LLM systems involves substantially more than evaluating a model-level hyperparameter configuration. Each measurement requires deploying and executing an end-to-end system involving model inference, retrieval, agents/tools, workloads, and system-level components, resulting in considerable computational cost and engineering effort. This makes it substantially harder to scale simultaneously across configurations, systems, datasets, models, and hardware than in model-level HPO/NAS benchmarks such as HPOBench.
- Practical value. LLMSYS-HPOBench is built from real, deployable LLM systems and their practical tuning problems. It covers heterogeneous RAG systems, inference engines, and agents, together with diverse hyperparameters, fidelity factors, performance objectives, and evaluation costs that arise in their actual operation. We therefore believe its practical value should be assessed not only by the number of datasets or hardware platforms, but also by whether it captures realistic optimization problems and enables fair, reproducible comparison of HPO methods.
- Complementary breadth. Compared with HPOBench/NAS benchmarks that provide broader coverage across datasets, LLMSYS-HPOBench emphasizes dimensions specific to end-to-end LLM system tuning: it covers both AI and non-AI hyperparameters, higher-dimensional fidelity factors, a broader range of optimization objectives, and additionally records hardware metrics and rich execution logs.
- An evolving benchmark. We agree that broader coverage of datasets, models, and hardware would further strengthen external validity. LLMSYS-HPOBench is therefore designed as an evolving benchmark rather than a fixed one-time dataset. At the time of review, it has already been expanded beyond the submitted version, with an additional 223,920 measured configurations incorporated. We will continue extending its coverage across datasets and LLM systems through versioned releases.
We thank Reviewer BJjh for the recognition of the value of our benchmark and the insightful suggestions for further improvement. We have carefully considered all the concerns and provide point-by-point responses below, including additional clarifications and experimental analyses where appropriate.
Thanks for the valuable comment.
Thanks for these helpful questions.
log_levelmay have limited impact on output quality, but it can affect logging overhead and execution latency.enabled_componentsin AutoGPT andnative_tool_callingin OpenHands. These parameters vary the set of enabled components and the tool-calling mode, respectively. We keep tool definitions and implementations fixed to compare configurations within each system.Good point. We examine this practical benefit through additional real-system evaluations:
Very good point. We further validate whether optimizer comparisons on LLMSYS-HPOBench are preserved on the real systems: