acceptodds
Under review as a conference paper at ICLR 2027

LLMSYS-HPOBench: Hyperparameter Optimization Benchmark Suite for Real-World LLM Systems

Abstract

Large Language Model (LLM) systems have been the frontier of AI in many application domains, leading to new challenges and opportunities for hyperparameter optimization (HPO) for the AutoML community. However, this type of system exhibits an unprecedented compound space of hyperparameter configuration from both the AI and non-AI components; rich and nonlinear implications from the fidelity factors; and diverse costs of measuring hyperparameter configurations, none of which have been fully captured in existing benchmarks. This paper presents the first (live) benchmark suite and datasets for HPO of real-world LLM systems, dubbed LLMSYS-HPOBench, covering data related to the inference objective values of hyperparameter configurations profiled from running the LLM systems. Currently, LLMSYS-HPOBench contains 364,450 hyperparameter configurations with a dimensionality of 12–23, 3–5 dimensions of fidelity factor leading to 932 settings, 3–9 inference objective metrics, and 2–10 cost metrics, together with generated logs from measuring the LLM systems. What we seek to advocate is not only a revalidation of the existing HPO algorithms over the frontier LLM systems, but also to provide an evolving platform for the AutoML community to explore new directions of research in this regard. The benchmark suite has been made available at: https://anonymous.4open.science/r/llmsys-hpobench-CD2B.

Primary area:
datasets and benchmarks
Keywords:
AutoML, HPO, LLM systems, Dataset, Benchmark.

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BJjhOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces LLMSys-HPOBench, a tabular benchmark for hyperparameter optimization of LLM systems, covering seven open-source systems across RAG pipelines, inference engines, and agent frameworks. For each system, it defines a mixed space of AI and non-AI inference-time hyperparameters and a multi-factor fidelity space, and releases 364450 measured configurations with objective metrics, cost metrics, hardware statistics, and execution logs. A unified API supports lookups, with a random forest surrogate for unmeasured configurations.

Critical strengths

The authors profiled over 364k configurations across seven real LLM systems using about 94900 GPU-hours, and released the results together with cost metrics, hardware statistics, and execution logs. This is the main strength in my assessment, since evaluating configurations of LLM systems is expensive.

Critical weaknesses

My major concern is the practical value of this tabular benchmark.

  1. All measurements are collected on a single hardware setup (RTX 5090), with small models and mostly one dataset per system. A tabular benchmark is useful only if conclusions drawn from it hold on real tuning problems, but the paper does not test this.
  2. The configuration spaces leave out the components that most strongly affect current LLM systems, such as prompts and tool design, especially for the RAG and agent systems.

Although the discussion outlines several research opportunities, the paper only touches on a few of them at a preliminary level, and none leads to new methods or actionable findings. As a result, the current contribution reads mainly as a costly data collection effort rather than a benchmark with demonstrated value for HPO research or practice.

Questions for authors
  1. Section 3.1 claims to include all tunable hyperparameters, yet the spaces include likely irrelevant ones (e.g., log_level in SGLang) and exclude prompts, with tools limited to on/off switches. How were hyperparameters selected, and why were prompts and tool design excluded?
  2. Beyond the speedup in Appendix C, can the authors show a concrete case where the benchmark leads to a measurable improvement on a real LLM system?
  3. Appendix E only tests ranking stability within the table. Can the authors show that optimizer rankings from the table match those from running directly on a real system, or under a different model or hardware?
4 comments

We thank Reviewer BJjh for the recognition of the value of our benchmark and the insightful suggestions for further improvement. We have carefully considered all the concerns and provide point-by-point responses below, including additional clarifications and experimental analyses where appropriate.

@R1W1: Generalization across hardware, models, and datasets

Thanks for the valuable comment.

  • Common benchmark practice. Most existing benchmarks, e.g., HPOBench, also retain a single hardware platform, since it is prohibitively expensive to collect data from multiple hardware platforms.
  • Benchmark goal. Our primary goal is to provide a reproducible and comparable benchmark for HPO of LLM systems. Therefore, we selected representative models and datasets and conducted measurements on a consistent hardware setup to support reliable comparisons across configurations.
  • Extensibility. While the current release covers a limited range of deployment settings, LLMSYS-HPOBench is designed as an evolving benchmark, allowing new hardware platforms, models, and datasets to be progressively incorporated through versioned releases.

@R1Q1: Hyperparameter selection, irrelevant parameters, and excluded prompts and tools

Thanks for these helpful questions.

  • Parameter selection: We identify tunable hyperparameters and their feasible value ranges from each LLM system's configuration interfaces, source code, and documentation.
  • Objective-dependent relevance. The importance of hyperparameters depends on the performance objective. For example, log_level may have limited impact on output quality, but it can affect logging overhead and execution latency.
  • Tool-related parameters. We already include tool-related hyperparameters, such as enabled_components in AutoGPT and native_tool_calling in OpenHands. These parameters vary the set of enabled components and the tool-calling mode, respectively. We keep tool definitions and implementations fixed to compare configurations within each system.
  • Prompt-related parameters. Our benchmark focuses on configurable parameters of the LLM systems themselves and treats user prompts as workload inputs. Within each workload, we keep the wording and structure of the prompt templates fixed so that configurations are compared under the same prompting setup.

@R1Q2: can the authors show a concrete case where the benchmark leads to a measurable improvement on a real LLM system?

Good point. We examine this practical benefit through additional real-system evaluations:

  • Real-system reevaluation. For seven LLM systems, we take the configuration recommended by benchmark-based HPO and reevaluate it on the actual system under the same target fidelity setting. We then compare its measured performance with the system's default configuration.
  • Observed improvement. The results are reported in Table R1.1. Overall, benchmark-guided tuning improves the target performance over the default configuration on five of seven systems; for example, SMAC improves NaiveRAG’s mean Lexical AC by 39.6 percent on the real system.

@R1Q3: Agreement between tabular and real-system optimizer rankings

Very good point. We further validate whether optimizer comparisons on LLMSYS-HPOBench are preserved on the real systems:

  • Real-system reevaluation. We reevaluate the configurations recommended by each optimizer on the corresponding LLM system under the same target fidelity setting.
  • Ranking agreement. We compare optimizer rankings based on benchmark-estimated and real-system-measured performance using Spearman’s rank correlation(Table R1.1). Overall, six of the seven systems show positive ranking agreement, with strong agreement on four, providing encouraging evidence that benchmark-based comparisons can reflect real-system outcomes, while the degree of agreement remains system-dependent.
SystemMethodDefaultTuned (mean ± SD)Change (pct.)Rank ρ\rho
NaiveRAG (↑\uparrow)Hyperband0.50790.6832 ± 0.050534.5+1.00
BOHB0.6667 ± 0.061531.3
SMAC0.7090 ± 0.033039.6
HEBO0.6736 ± 0.068332.6
Random search0.6881 ± 0.059335.5
LightRAG (↑\uparrow)Hyperband0.05560.0639 ± 0.026015.0+0.80
BOHB0.0638 ± 0.020014.9
SMAC0.0655 ± 0.017017.9
HEBO0.0698 ± 0.016925.7
Random search0.0652 ± 0.022017.4
vLLM (↑\uparrow)Hyperband6.2716.556 ± 0.3164.5+1.00
BOHB6.739 ± 0.5217.5
SMAC6.870 ± 0.9599.5
HEBO6.723 ± 0.4827.2
Random search6.592 ± 0.2945.1
HTML-RAG (↑\uparrow)Hyperband0.61000.5587 ± 0.1089−8.4-8.4+0.40
BOHB0.5357 ± 0.1144−12.2-12.2
SMAC0.5795 ± 0.1004−5.0\mathbf{-5.0}
HEBO0.5613 ± 0.0976−8.0-8.0
Random search0.5503 ± 0.0733−9.8-9.8
OpenHands (↑\uparrow)Hyperband1.00000.9333 ± 0.1356−6.7-6.7+0.95
BOHB0.9389 ± 0.1417−6.1-6.1
SMAC0.9556 ± 0.1152−4.4\mathbf{-4.4}
HEBO0.9556 ± 0.1152−4.4\mathbf{-4.4}
Random search0.9278 ± 0.1496−7.2-7.2
SGLang (↑\uparrow)Hyperband0.7600.917 ± 0.24920.6−0.40-0.40
BOHB0.957 ± 0.23425.9
SMAC0.941 ± 0.22423.7
HEBO0.904 ± 0.27518.8
Random search0.922 ± 0.21721.2
AutoGPT (↑\uparrow)Hyperband0.005540.02394 ± 0.01330332.3+0.40
BOHB0.02753 ± 0.02142397.1
SMAC0.03748 ± 0.03552576.8
HEBO0.02522 ± 0.01438355.4
Random search0.02295 ± 0.01829314.5

Table: R1.1. Real-system performance of the default configuration and configurations recommended by benchmark-based tuners at the target fidelity setting. ↑\uparrow indicates a maximization objective. Change reports the percentage change relative to the default. Rank (ρ\rho) is the Spearman correlation between tuners rankings obtained on LLMSYS-HPOBench and those obtained after reevaluating the recommended configurations on the real system.

Thank you for the response and the new experiment. My main concern about practical value remains. No results are provided on other hardware, models, or datasets. Unlike prior benchmarks cited in the paper (e.g., HPOBench, NAS-Bench-201), which cover multiple datasets, six of the seven systems here use a single dataset, so it is unclear whether conclusions transfer to real tuning problems. I will keep my current score.

Thank you for the follow-up. We would like to further clarify the scope and practical value of LLMSYS-HPOBench.

  • Different data-collection setting. Building a benchmark for complete LLM systems involves substantially more than evaluating a model-level hyperparameter configuration. Each measurement requires deploying and executing an end-to-end system involving model inference, retrieval, agents/tools, workloads, and system-level components, resulting in considerable computational cost and engineering effort. This makes it substantially harder to scale simultaneously across configurations, systems, datasets, models, and hardware than in model-level HPO/NAS benchmarks such as HPOBench.
  • Practical value. LLMSYS-HPOBench is built from real, deployable LLM systems and their practical tuning problems. It covers heterogeneous RAG systems, inference engines, and agents, together with diverse hyperparameters, fidelity factors, performance objectives, and evaluation costs that arise in their actual operation. We therefore believe its practical value should be assessed not only by the number of datasets or hardware platforms, but also by whether it captures realistic optimization problems and enables fair, reproducible comparison of HPO methods.
  • Complementary breadth. Compared with HPOBench/NAS benchmarks that provide broader coverage across datasets, LLMSYS-HPOBench emphasizes dimensions specific to end-to-end LLM system tuning: it covers both AI and non-AI hyperparameters, higher-dimensional fidelity factors, a broader range of optimization objectives, and additionally records hardware metrics and rich execution logs.
  • An evolving benchmark. We agree that broader coverage of datasets, models, and hardware would further strengthen external validity. LLMSYS-HPOBench is therefore designed as an evolving benchmark rather than a fixed one-time dataset. At the time of review, it has already been expanded beyond the submitted version, with an additional 223,920 measured configurations incorporated. We will continue extending its coverage across datasets and LLM systems through versioned releases.
Reviewer_7vUmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper releases LLMSYS-HPOBench, a benchmark of measured hyperparameter configurations for seven open-source LLM systems: three RAG pipelines, two inference engines and two agent frameworks. It reports 364,450 samples over 12–23 model-side and system-side hyperparameters per system, with objectives, costs, hardware metrics and logs from about 94,900 GPU-hours. A unified API returns recorded values and, according to Section 3.3, random-forest estimates for unmeasured configurations by default. A case study compares single- and multi-objective optimizers and relates the cost of fidelity settings to their rank correlation with a target setting. The central contribution is the measured resource.

Critical strengths
  1. A large measured resource spanning several kinds of LLM system: Real open-source systems are measured over search spaces that mix model-side and system-side settings. Earlier LLM-system HPO work exists, such as IBM's RAG HPO study (arXiv 2505.03452), which released its configuration results, and SCOOT's tuning of vLLM and TensorRT-LLM (WWW 2025). What appears new is the breadth across RAG, serving and agents. This costly, reusable table is the paper's main contribution.

  2. Release design and candid robustness reporting: The paper commits to versioned releases with frozen data and default protocols (Appendix G.2). Its sampling-density study reports largely stable method rankings while noting where the best method changes (Appendix E). Both add to the value of the resource once the evaluator issue below is resolved.

Critical weaknesses
  1. The paper reports no validation for the evaluator behind the optimizer comparisons and does not quantify measurement noise. It says algorithms built on the benchmark "can be applied to the practice without many barriers". However:

    • SMAC and HEBO can propose unmeasured configurations. According to Section 3.3, these are answered by default by a random-forest surrogate, which the case studies also use. Its held-out error, rank agreement and share of answered queries are not reported, although Eggensperger et al. (2015), cited for this choice, validated their surrogates this way.
    • Several objectives depend on sampled LLM outputs, yet no repeat-to-repeat agreement is reported. So the near-zero autocorrelations and fluctuating cross-setting correlations, which the paper reads as ruggedness and non-monotonic fidelity, cannot be separated from noise.
    • The released loader, as accessed during reviewing, defaults to the nearest recorded row under an unscaled distance. The inspected files contain no random-forest or case-study code, so it is unclear which evaluator ran.

    As comparing optimizers is the benchmark's central use, the missing evaluator validation is my main concern. I suggest held-out validation (or restricting the case study to measured configurations) and a repeat-agreement estimate (or narrower landscape claims).

  2. The paper's account of the case-study protocol has gaps and unexplained discrepancies, and the fidelity analysis does not separate task from cost changes.

    • HB and BOHB are said to promote configurations "to the target setting", but their top rung differs from the common target for three systems, and only in the repeat for vLLM (Tables 5 and 12). How their target results were obtained and charged is not stated.
    • Table 9's 100% row and Table 2 both use the full dataset, yet all 28 means in that row are higher and three systems' best tiers differ, without explanation.
    • Several "fidelity" factors also change the task, such as dataset category or task type. The non-monotonic cost–correlation pattern in Table 14 may therefore reflect cross-task transfer as much as fidelity.

    This makes the optimizer comparisons hard to interpret, and the paper does not yet establish the conclusion that "evaluation cost alone is not a reliable proxy for fidelity level" or support the suggestion that existing multi-fidelity methods might be invalidated. I suggest a reconciled protocol, and either cost-only correlations at a fixed task or reframed fidelity claims (task factors remain useful as auxiliary tasks). If both weaknesses are addressed, I would support accepting the measured table as a resource.

Questions for authors
  1. Which evaluator produced Tables 2, 3 and 9, and what share of case-study queries did it answer rather than look up? Please report its held-out error and rank correlation per system.
  2. Were any configurations measured more than once, as vLLM's repeat-count fidelity factor and the repository's format.md (as accessed during reviewing) suggest? If so, how well do repeats agree?
  3. Could the authors give one reconciled table of targets, HB/BOHB ladders and evaluation accounting, and explain why Table 9's 100% row differs from Table 2?
  4. Please position the benchmark against the earlier LLM-system HPO work named above, and scope the "first benchmark" claim to its cross-category coverage.
4 comments

We thank the reviewer 7vUm for the careful review and constructive feedback. We have carefully addressed the main concerns and provide detailed responses below.

@R2Q1: Which evaluator produced Tables 2, 3 and 9, and what share of case-study queries did it answer rather than look up? Please report its held-out error and rank correlation per system.

Thanks for these insightful questions.

  • Evaluator. Tables 2, 3, and 9 use Random Forest (RF) as the default surrogate for unmeasured configurations. We have also updated the released source code to make this evaluator and the corresponding case-study implementation explicit.
  • Surrogate usage. Across the studied systems, RF answers 100% of the audited single-objective queries within the base budget, for both performance and cost. This likely reflects the very large configuration spaces relative to the currently measured samples. As LLMSYS-HPOBench continues to grow, more queries may be resolved directly from measured configurations, reducing reliance on surrogate estimates.
  • Held-out error. We train RF using half of the measured configurations and evaluate its predictions on the remaining half for each system at its target fidelity. Table R2.1 reports weighted absolute percentage error (WAPE) and configuration-level Spearman correlations between RF predictions and measurements for both performance and cost. Overall, RF achieves reasonably accurate predictions.
  • Ranking agreement. We compare optimizer rankings based on benchmark-estimated and real-system-measured performance using Spearman's rank correlation. At the target fidelities, six of seven systems showed positive correlations, with ρ≥+0.80\rho \geq +0.80 in four, suggesting that benchmark-based comparisons can reflect real-system outcomes while the degree of agreement remains system-dependent.
SystemPerformance metricCost fieldPerf. WAPECost WAPEPerf. ρ\rhoCost ρ\rho
NaiveRAGLexical ACcost-Total_Time7.617.20.644 ± 0.0250.889 ± 0.014
LightRAGF1cost-total_time25.533.00.835 ± 0.0100.712 ± 0.033
vLLMOutput-token throughputcost-benchmark_duration_s7.06.20.674 ± 0.0490.678 ± 0.056
HTML-RAGAverage similaritycost-Total_Time2.715.10.685 ± 0.0230.660 ± 0.020
OpenHandsBinary accuracycost-total_time_sec49.538.10.280 ± 0.0230.191 ± 0.024
SGLangOutput-token throughputcost-duration10.611.50.881 ± 0.0530.911 ± 0.037
AutoGPTTask throughputcost-total_duration9.010.60.921 ± 0.0150.915 ± 0.018

Table: R2.1. Random forest prediction error (using 50/50 train/test splits) and configuration-level rank agreement at target fidelity setting. WAPE is reported in percent.

Prediction error and rank agreement vary across systems. At the HTML-RAG target fidelity setting, the RF achieves a performance WAPE of 2.7 percent and a Spearman correlation of 0.685.

@R2Q2: Repeated measurements and repeat-to-repeat agreement

Thank you for raising this question about repeated measurements.

  • Repeated measurements. Our original vLLM sampling protocol scheduled one measurement per configuration at each workload setting. The repeat_count parameter controls how many times requests are repeated within a single measurement for prefix-caching tests.
  • Repeat agreement. In a follow-up experiment, we independently remeasured 40 distinct vLLM configurations at the target fidelity. Pairwise Spearman correlations across measurement rounds ranged from 0.77 to 0.95, indicating that the throughput rankings were broadly consistent across repeats. Median within-configuration throughput coefficients of variation ranged from 1.30 to 1.39 percent, indicating low typical variation in throughput for the same configuration.

We clarify the distinction in the vLLM manual and format.md and will qualify our interpretation of the autocorrelation results in the appendix in light of possible measurement noise.

@R2Q3: Reconciled targets, HB/BOHB ladders, budget accounting, and Tables 2 and 9

Thanks for raising these helpful questions.

  • Target fidelity setting. The mismatch between Tables 5 and 12 is a typo. We will correct Table 5 to match the Level 4 settings in Table 12, which the single-objective runs used as their common targets.
  • HB/BOHB ladders. We use η=2\eta=2 and R=4R=4, yielding resource levels (1,2,4)(1, 2, 4). Table R2.2 lists the concrete settings from manuscript Table 12. Table 11 of manuscript specifies the number of configurations evaluated at each level in each bracket. We will consolidate these details.
  • Budget accounting. All optimizers are given the same total evaluation budget. For HB/BOHB, evaluations at different fidelity levels, including the target level after promotion, incur their corresponding costs returned by the evaluator, which are accumulated toward the same budget. The final comparison considers only the best performance achieved by each optimizer at the target fidelity setting within total budget BB, based on cumulative cost at query completion.
  • Table 2 vs. Table 9. The discrepancy arises from an inconsistent budget/reporting rule between the original case study and the sampling-density analysis. We will align both experiments to the same evaluation beudget and stopping criterion, and update the results/descriptions accordingly.
SystemLevel 1Level 2Level 4 (target)BB (s)
NaiveRAG0.5-0-easy-cs0.5-1-easy-cs0.5-2-easy-cs10,000
LightRAGbridge-0-0-2bridge-1-2-5bridge-2-5-1010,000
vLLM15.0-2.0-32-50-r115.0-2.0-32-100-r115.0-2.0-32-200-r110,000
HTML-RAG2-01-102-02-102-025-1010,000
OpenHands10-4-5-614-4-5-620-4-5-6200
SGLanginf-2.0-32-64-1024inf-2.0-32-64-2048inf-2.0-32-64-409610,000
AutoGPTcomplex-req1-code_generationcomplex-req2-code_generationcomplex-req4-code_generation2,000

Table: R2.2. Single-objective target fidelity setting, HB/BOHB resource levels, and base budgets by system.

@R2Q4: Distinction from prior LLM-system HPO studies.

Thanks for pointing us to these relevant studies.

  • IBM's RAG HPO. It focuses on HPO for RAG systems and releases the corresponding configuration evaluation results.
  • SCOOT. It proposes an optimization method for LLM inference engines, rather than a reusable benchmark suite for comparing HPO methods.
  • LLMSYS-HPOBench. Our goal is to provide a comprehensive benchmark spanning RAG systems, inference engines, and agents, with unified access to measured configurations, fidelity settings, objectives, cost, hardware metrics, and execution logs.

We will incorporate both studies into the related work and more precisely position our contribution around the benchmark scope.

Thank you for your response and the new analyses, which answer most of my questions. I have two remain questions:

First, since RF handles every single-objective query, the benchmark is only useful for comparing optimizers if its rankings carry over to the real systems. Could you share the optimizer-ranking agreement \rho for each system using the Table 6 objectives, and explain how you obtained the real-system performance?

Second, my point that several fidelity factors also change the underlying task was not covered in the response. How do you plan to phrase the claims that evaluation cost isn't a reliable proxy for fidelity level, and that existing multi-fidelity methods might be invalidated?

We thank the reviewer for the prompt follow-up. We address the two remaining questions below.

Q1: How are the real-system optimizer rankings obtained, and how well do they agree with the benchmark rankings?

Thanks for this important follow-up.

  • Benchmark evaluation. We run HB, BOHB, SMAC, and HEBO on LLMSYS-HPOBench using the single-objective metric in Table 6, and obtain their benchmark-based performance and ranking.
  • Real-system reevaluation. For each optimizer, we take its best configuration identified by LLMSYS-HPOBench at the target fidelity setting and run that again on the real LLM system under the same target setting.
  • Ranking agreement. We rank the optimizers using these real-system measurements and compute Spearman's (ρ\rho) against the benchmark-based ranking.

The resulting ρ\rho values are reported in the following table. Overall, six of the seven systems show positive ranking agreement, with strong agreement on four, providing encouraging evidence that benchmark-based comparisons can reflect real-system outcomes, while the degree of agreement remains system-dependent.

Table 1: Real-system performance of the default configuration and configurations recommended by benchmark-based tuners at the target fidelity setting. ↑\uparrow indicates a maximization objective. Change reports the percentage change relative to the default. Rank (ρ\rho) is the Spearman correlation between tuners' rankings obtained on LLMSYS-HPOBench and those obtained after reevaluating the recommended configurations on the real system.

SystemMethodDefaultTuned (mean ± SD)Change (pct.)Rank ρ\rho
NaiveRAG (↑\uparrow)Hyperband0.50790.6832 ± 0.050534.5+1.00
BOHB0.6667 ± 0.061531.3
SMAC0.7090 ± 0.033039.6
HEBO0.6736 ± 0.068332.6
Random search0.6881 ± 0.059335.5
LightRAG (↑\uparrow)Hyperband0.05560.0639 ± 0.026015.0+0.80
BOHB0.0638 ± 0.020014.9
SMAC0.0655 ± 0.017017.9
HEBO0.0698 ± 0.016925.7
Random search0.0652 ± 0.022017.4
vLLM (↑\uparrow)Hyperband6.2716.556 ± 0.3164.5+1.00
BOHB6.739 ± 0.5217.5
SMAC6.870 ± 0.9599.5
HEBO6.723 ± 0.4827.2
Random search6.592 ± 0.2945.1
HTML-RAG (↑\uparrow)Hyperband0.61000.5587 ± 0.1089−8.4-8.4+0.40
BOHB0.5357 ± 0.1144−12.2-12.2
SMAC0.5795 ± 0.1004−5.0\mathbf{-5.0}
HEBO0.5613 ± 0.0976−8.0-8.0
Random search0.5503 ± 0.0733−9.8-9.8
OpenHands (↑\uparrow)Hyperband1.00000.9333 ± 0.1356−6.7-6.7+0.95
BOHB0.9389 ± 0.1417−6.1-6.1
SMAC0.9556 ± 0.1152−4.4\mathbf{-4.4}
HEBO0.9556 ± 0.1152−4.4\mathbf{-4.4}
Random search0.9278 ± 0.1496−7.2-7.2
SGLang (↑\uparrow)Hyperband0.7600.917 ± 0.24920.6−0.40-0.40
BOHB0.957 ± 0.23425.9
SMAC0.941 ± 0.22423.7
HEBO0.904 ± 0.27518.8
Random search0.922 ± 0.21721.2
AutoGPT (↑\uparrow)Hyperband0.005540.02394 ± 0.01330332.3+0.40
BOHB0.02753 ± 0.02142397.1
SMAC0.03748 ± 0.03552576.8
HEBO0.02522 ± 0.01438355.4
Random search0.02295 ± 0.01829314.5

Q2: Fidelity definition and the related claims.

Good point. We clarify our definition and the related claims as follows:

  • Fidelity definition. We use fidelity in a broader, optimization-oriented sense. Given a target setting, other evaluation settings that share the same hyperparameter space and performance metric are treated as candidate auxiliary fidelity settings, where they might provide an imperfect but potentially useful signal for optimizing the target. Such settings may differ in resource allocation, operating environments, or even task characteristics.
  • Cost and fidelity. Our experimental results (Figure 7 of the manuscript) suggest that measurement cost is not necessarily monotonic with how informative a fidelity setting is for the target (i.e., there are fidelity settings that come with high measurement cost but exhibit weak correlation with the target fidelity setting). Therefore, cost alone cannot determine which fidelity is more useful; the relationship with the target needs to be examined when selecting auxiliary fidelity settings for optimization.
  • Implications for multi-fidelity HPO. This can challenge multi-fidelity methods that rely on a predefined cost-ordered fidelity hierarchy. In contrast, methods that adaptively select fidelity settings according to their observed utility for the target may be better suited to such heterogeneous settings.

We will make this definition and these claims clearer.

Reviewer_TWiUOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper introduces LLMSYS-HPOBench, a benchmark suite for hyperparameter optimization of LLM systems. It covers seven systems spanning RAG pipelines, inference engines, and agent frameworks, with 364,450 configuration evaluation samples across 932 fidelity settings. The benchmark provides objective values, evaluation costs, hardware measurements, and execution logs through a unified interface, using surrogate predictions for unmeasured configurations. The empirical study compares single- and multi-objective optimizers and examines relationships between evaluation cost and fidelity informativeness.

Critical strengths

The paper addresses a clear and practically relevant problem. Evaluating configurations of end-to-end LLM systems is expensive, and an accessible benchmark could facilitate research on their optimization.The collected data cover several system categories and jointly expose model-related and system-related hyperparameters. Recording costs, hardware measurements, and execution logs increases the resource’s potential usefulness.The unified interface and versioned release protocol provide a useful foundation for reuse and extension. The experiments also cover multiple optimization settings and include additional analyses of sampling density and optimization behavior.

Critical weaknesses

1. The fidelity of the surrogate-backed benchmark to real system behavior is insufficiently validated.

Unmeasured configurations are evaluated using predicted objectives and costs, but the paper does not establish whether optimizing this environment produces configurations that perform well on the actual systems. In particular, it lacks validation on independently measured configurations and real-system reevaluation of optimizer recommendations. The sampling-density analysis in Appendix E examines sensitivity to data coverage, but does not establish agreement between offline optimization results and real-system outcomes. This is my main concern because such agreement is central to the benchmark’s intended use.

2. The multi-fidelity evaluation protocol contains unresolved inconsistencies.

The highest-resource settings in Table 12 differ from the common target settings in Table 5 for several systems. For example, NaiveRAG uses different data domains and question ratios, while AutoGPT uses different task types. OpenHands and vLLM also have mismatched settings. It is unclear whether HB/BOHB recommendations are additionally evaluated at the common target, and how those evaluations are charged. Without this clarification, the comparison with target-only optimizers is difficult to assess.

3. The definition of fidelity conflates evaluation approximation with changes in the target task or operating environment.

The fidelity factors include dataset category, task type, request rate, and concurrency. These variables can change the optimization problem itself rather than simply provide a cheaper approximation to a fixed target. Consequently, observed ranking changes across settings do not by themselves establish the claimed limitations of conventional multi-fidelity optimization. The paper needs a clearer account of how these settings relate to a specified target.

4. The empirical study only partially substantiates the benchmark’s distinctive design claims.

Joint AI/non-AI hyperparameter optimization is a central motivation, but its benefit is not directly examined against tuning either group alone or tuning them sequentially. Task representativeness also requires further justification: OpenHands is presented as a software development agent, yet it is evaluated on ProofWriter reasoning tasks. The paper does not sufficiently explain which agent mechanisms this setup exercises or how it captures the associated system optimization challenges.

Questions for authors
  1. Can the authors reevaluate configurations recommended by the optimizers on the actual systems and show whether the predicted improvements and relative optimizer performance are preserved? How accurately does the surrogate identify high-performing configurations and their evaluation costs?

  2. Why do the highest-resource settings in Table 12 differ from the target settings in Table 5? Are all optimizers ultimately evaluated at the same target, and are any additional target evaluations included in the budget?

  3. How are target fidelities selected, and how do the authors distinguish lower-fidelity approximations from different tasks or deployment environments? For the cross-fidelity analysis, are correlations computed over identical configurations?

  4. Can the authors provide evidence for the benefit of jointly tuning AI and non-AI hyperparameters, and clarify which OpenHands capabilities are exercised by the ProofWriter evaluation?

4 comments

We appreciate the reviewer's thoughtful comments and detailed suggestions. We have carefully examined the concerns raised and respond to them point by point below.

@R3Q1: Real-system validation and surrogate accuracy in identifying high-performing configurations

Good point.

  • Real-system reevaluation. For seven systems, we reevaluate the configurations recommended by each optimizer on the corresponding LLM system under the same target fidelity setting.
  • Observed improvement. Table R3.2 reports performance for the default and benchmark-recommended configurations at the target fidelity setting. The recommended configurations improved the reported performance metric over the default in five of seven systems, showing that configurations identified through LLMSYS-HPOBench can deliver measurable gains on real systems.
  • Ranking agreement. We compare optimizer rankings based on benchmark-estimated and real-system-measured performance using Spearman's rank correlation. Table R3.2 reports the results. Six of the seven systems show positive correlations, with ρ≥+0.80\rho \geq +0.80 in four, suggesting that benchmark-based rankings can reflect real-system rankings, with agreement varying across systems.
  • Held-out accuracy. We train RF on half of the measured configurations at each system’s target fidelity and evaluate it on the remaining half. Table R3.1 reports the overlap between the predicted and measured top 5 percent of configurations and the weighted absolute percentage error (WAPE) for cost.
SystemPerformance metricCost fieldTop-5-percent overlapCost WAPE (pct.)
NaiveRAGLexical ACcost-Total_Time56.817.2
LightRAGF1cost-total_time24.133.0
vLLMOutput-token throughputcost-benchmark_duration_s72.16.2
HTML-RAGAverage similaritycost-Total_Time50.215.1
OpenHandsBinary accuracycost-total_time_sec5.638.1
SGLangOutput-token throughputcost-duration65.011.5
AutoGPTTask throughputcost-total_duration25.010.6

Table: R3.1. Random forest top-five-percent overlap and cost prediction error on held-out configurations at target fidelity settings. Both metrics are reported in percent.

RF recovered more than half of the top-performing configurations on average in four of seven systems, with overlap ranging from 5.6 to 72.1 percent. Held-out cost WAPE ranged from 6.2 to 17.2 percent in five of seven systems, suggesting that RF can provide useful cost estimates for most systems, with larger errors for LightRAG and OpenHands. Many OpenHands configurations tied on binary accuracy, which limits the interpretation of its overlap near the 5 percent random baseline.

SystemMethodDefaultTuned (mean ± SD)Change (pct.)Rank ρ\rho
NaiveRAG (↑\uparrow)Hyperband0.50790.6832 ± 0.050534.5+1.00
BOHB0.6667 ± 0.061531.3
SMAC0.7090 ± 0.033039.6
HEBO0.6736 ± 0.068332.6
Random search0.6881 ± 0.059335.5
LightRAG (↑\uparrow)Hyperband0.05560.0639 ± 0.026015.0+0.80
BOHB0.0638 ± 0.020014.9
SMAC0.0655 ± 0.017017.9
HEBO0.0698 ± 0.016925.7
Random search0.0652 ± 0.022017.4
vLLM (↑\uparrow)Hyperband6.2716.556 ± 0.3164.5+1.00
BOHB6.739 ± 0.5217.5
SMAC6.870 ± 0.9599.5
HEBO6.723 ± 0.4827.2
Random search6.592 ± 0.2945.1
HTML-RAG (↑\uparrow)Hyperband0.61000.5587 ± 0.1089−8.4-8.4+0.40
BOHB0.5357 ± 0.1144−12.2-12.2
SMAC0.5795 ± 0.1004−5.0\mathbf{-5.0}
HEBO0.5613 ± 0.0976−8.0-8.0
Random search0.5503 ± 0.0733−9.8-9.8
OpenHands (↑\uparrow)Hyperband1.00000.9333 ± 0.1356−6.7-6.7+0.95
BOHB0.9389 ± 0.1417−6.1-6.1
SMAC0.9556 ± 0.1152−4.4\mathbf{-4.4}
HEBO0.9556 ± 0.1152−4.4\mathbf{-4.4}
Random search0.9278 ± 0.1496−7.2-7.2
SGLang (↑\uparrow)Hyperband0.7600.917 ± 0.24920.6−0.40-0.40
BOHB0.957 ± 0.23425.9
SMAC0.941 ± 0.22423.7
HEBO0.904 ± 0.27518.8
Random search0.922 ± 0.21721.2
AutoGPT (↑\uparrow)Hyperband0.005540.02394 ± 0.01330332.3+0.40
BOHB0.02753 ± 0.02142397.1
SMAC0.03748 ± 0.03552576.8
HEBO0.02522 ± 0.01438355.4
Random search0.02295 ± 0.01829314.5

Table: R3.2. Default and benchmark-recommended performance at target fidelity settings, with offline-to-real optimizer rank agreement.

↑\uparrow indicates a maximization objective. Scores are Lexical AC for NaiveRAG and HTML-RAG, F1 for LightRAG, requests/s for vLLM and SGLang, evidence recall for OpenHands, and task attempts/s for AutoGPT. Tuned scores are means and sample standard deviations over valid measurements. Change is the percentage change relative to the default. The default and Spearman's ρ\rho are listed once per system. The correlation ranks Hyperband, BOHB, SMAC, and HEBO on common seeds.

@R3Q2: Differences between Tables 5 and 12, common targets, and budget accounting

Thanks for raising these helpful questions.

  • Target fidelity setting. The mismatch between Tables 5 and 12 is a presentation error. We will correct these tables to make the settings consistent.
  • Budget accounting. All optimizers are given the same total evaluation budget. For HB/BOHB, evaluations at different fidelity levels incur their corresponding measured costs (e.g., execution time), which are accumulated toward the same budget. The final comparison considers only the best performance achieved by each optimizer at the target fidelity setting.

@R3Q3: Target selection, lower-fidelity definitions, and matched configurations for correlations

Thanks for raising these important questions. We clarify the following points:

  • Target fidelity selection. For each LLM system, we select a high-cost setting as the target fidelity setting, representing the intended optimization scenario.
  • Low-fidelity settings. We define fidelity factors as variables that govern evaluation conditions, such as resource allocation, task characteristics, and operating environments. Relative to a specified target fidelity, other settings are treated as candidate lower-fidelity approximations, including those involving different tasks or environments. They share the same hyperparameter configuration space and performance metric, and may provide useful information for the optimization under the target fidelity setting.
  • Cross-fidelity correlation. We confirm that the correlations are computed over identical hyperparameter configurations.

We will clarify these definitions and make the corresponding claims clearer.

@R3Q4: Joint AI/non-AI tuning benefits and OpenHands capabilities exercised by ProofWriter

Thanks for these helpful comments.

  • AI/non-AI hyperparameters. Both AI and non-AI hyperparameters can shape the performance of an LLM system, motivating us to capture both groups and their interactions within a unified benchmark.
  • AI/non-AI tuning benefits. To investigate how the two groups contribute to system performance, we additionally use functional ANOVA (fANOVA) [1] to quantify the effects of AI hyperparameters, non-AI parameters, and their interactions. The results show that cross-group pairs account for 80.3 percent of the reported pairwise interaction strength for LightRAG F1, 76.0 percent for vLLM mean time to first token, and 50.1 percent for OpenHands total runtime at the target fidelities. These interactions motivate tuning both groups jointly. Quantifying the benefit of joint tuning requires comparing joint, group-only, and sequential tuning under the same budget.
  • OpenHands evaluation. Following prior literature [2], we use ProofWriter as a controlled workload for evaluating OpenHands, as it provides well-defined reasoning tasks and ground truth for reproducible evaluation. We will further clarify the OpenHands capabilities exercised in this setting and its evaluation scope.

References

[1] Frank Hutter, Holger Hoos, and Kevin Leyton-Brown. An Efficient Approach for Assessing Hyperparameter Importance. In International Conference on Machine Learning (ICML), pp. 754–762, 2014. Conference paper

[2] Xingyao Wang et al. OpenHands: An Open Platform for AI Software Developers as Generalist Agents. In International Conference on Learning Representations (ICLR), 2025. Conference paper

Dear Reviewer TWiU,

As the discussion period is nearing its end, we hope our responses have addressed your concerns. Please let us know if any further clarification would be helpful. We would be happy to answer any further questions. Thank you again for your time and thoughtful review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.