acceptodds
Under review as a conference paper at ICLR 2027

InferenceBench: A Benchmark for Open-Ended LLM Inference Optimization by AI Agents

Abstract

AI agents are increasingly used to automate research and development, but existing benchmarks often constrain them to prescribed workflows or narrow action spaces, making it difficult to distinguish actual strong performance from reproductions of pre-existing solutions. We introduce InferenceBench, where an agent must deploy an OpenAI-compatible inference server and optimize LLM inference speed. Each agent receives a target LLM, one H100 GPU, an optimization scenario, and a 2-hour time budget. The objectives each address a bottleneck in LLM inference: the delay before the first token, the delay between later tokens, and the number of requests served per second under load, with a fourth objective that balances all three at the same time. We evaluate 42 agent configurations and find that all agents improve over a na\"ive PyTorch server, and 30 of 42 exceed the best serving engine with default settings ( for vLLM). However, a simple hyperparameter search over vLLM hyperparameters with the same time budget reaches , beating all but a single agent, Claude Opus 5.5 (). Although agents enumerate many relevant optimization techniques, they overwhelmingly converge on a single inference framework, test few distinct configurations, and spend the remaining budget re-measuring, repairing, or optimizing hyperparameters rather than exploring substantially different strategies. Overall, InferenceBench measures how well agents operate in an open-ended AI-engineering setting, where memorized solutions give only limited gains and the unbounded speedup score leaves room to track stronger agents.

Primary area:
datasets and benchmarks
Keywords:
Evaluations, Agent Evaluation, Benchmarking, Agentic Systems, Inference Optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_nKxUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper suggests InferenceBench, an evaluation toolkit which is used to see how good different LLM models are at improving an inference setup for certain workloads compared to a common PyTorch baseline.

Critical strengths

The paper present a study on a field that is very of interests in the system and efficient machine learning community.

Critical weaknesses
  1. The setting is too small to be practical. The scenarios use at most 8K input and 8K output tokens. Real long-context prefill and long-generation decode now can easily exceed 100K tokens. With a 7B model on an 96 GB H100… this setup is quite impractical as weight and activation memory is never the binding constraint: Mistral-7B’s KV cache is roughly 1 GB for an 8K request. As a result, key systems decisions such as KV-cache offloading to CPU or NVMe, paged attention, KV compression, and context or sequence parallelism are never exercised. This sits poorly with the claim of evaluating “end-to-end inference systems engineering.” The Limitations section mentions only multi-GPU, not context length or memory pressure.
  2. Weakness: Limited novelty. The paper claims to be the first open-ended benchmark for end-to-end inference systems engineering, but prior work already evaluates agents on closely related tasks: ISO-Bench[1] on optimizing vLLM and SGLang, and KernelBench[2] and FlashInfer-Bench on inference kernels. The open-ended action space also collapses in practice: 91.5% of runs ship vLLM and the median run tries one configuration, so the task reduces largely to serving-flag tuning. The main findings (shallow exploration, early convergence, weak experimental control) echo prior agent benchmarks. And also people are already doing inference serving improvements using different LLMs and harnesses.
  3. Prefix caching is presented as an optimization but never tested. The paper names prefix caching as one of the main optimization levers for Scenario A (Section 2.2), includes it in the vLLM search space (Appendix B.4), and describes an agent enabling it (Example 5). However, no scenario contains shared prefixes: every request is a distinct LongBench document, so prefix caching cannot meaningfully help in Scenarios A, B, or D. The only possible reuse is an artifact of Scenario C replaying the same 256 requests across its three traffic profiles. This is not a deliberate test of prefix caching but an uncontrolled confound, since the paper does not say whether the cache is cleared between profiles. Prefix reuse is central to real deployments, such as multi-turn chat, shared system prompts, RAG, and agent workloads, so its absence further limits how practical the benchmark is. A scenario with deliberately shared prefixes would test this lever properly.
  4. vLLM hyperparameter search beats most of their tested setups.
  5. Scenarios do not reflect modern agentic workloads. All four scenarios in Table 1 send single-turn, independent requests built from LongBench documents, with fixed input and output lengths of at most 8K tokens. Modern agentic workloads look very different. They are multi-turn sessions whose context grows to tens or hundreds of thousands of tokens. They share heavy prefixes (system prompts, tool definitions, conversation history), so prefix-cache reuse dominates. Model calls alternate with idle gaps during tool execution, and outputs per turn are often short and structured. None of this is captured, so the benchmark cannot show whether an agent can optimize for the deployments where inference cost now matters most. This is especially notable for a paper about agents, whose own workloads (like the agentic coding tools it evaluates) have exactly this profile.
  6. The SGLang search baseline is inconsistent with its default. On Scenario C, all three SGLang searches (31.95–33.05×) score about 35% below SGLang with default settings (51.12×), far outside the reported standard errors. A search whose space includes the default configuration should not end well below it. The SGLang default also exceeds the best vLLM search (46.70×), which is the main search baseline in Table 2. Since “search beats nearly all agents” is a central claim, the paper should explain how the search spaces and evaluation differ from the default runs.
Questions for authors

See weakness

6 comments

We thank the reviewer for the detailed and careful review. Several of the reviewer's concerns relate to what the benchmark is meant to cover. So we first state more precisely what InferenceBench is designed to measure: whether an agent can autonomously set up and optimize an inference server, with no prescribed solution.

The main contribution that we present is the benchmark framework along with an analysis of the different runs of the benchmark showing its validity and usefulness as a measurement framework. We have revised the manuscript in order to make this more clear. We address the remaining concerns below.

Regarding the scale of the experiments

By design, the benchmark does not forbid any of the techniques that the reviewer lists. The agent has root access, internet access, and a free choice of engine and implementation. It is free to offload or compress the KV cache or change the attention implementation, and the prompt even names many of the options it has in the open ended task (Appendix B.1).

However, the current scenarios already contain large gains that agents do not take. For instance, search on SGLang on scenario A beats search on vLLM under all three optimizers (Table 14), but 91.5% of runs ship vLLM. On Scenario B, GPT-6 Sol (Ultra) never speculative decoding and reached 5.99x, against 22.56x for the best agent which did. Due to not utilizing scheduler and memory settings no agent configuration reaches the vLLM or SGLang default on scenario C.

Even when using a more complex model such as DeepSeek-V2-Lite, the agent attempted to apply the same optimization techniques that it used on the dense model, and as a result, only 5 of 12 runs passed (Appendix G.2). On a separate experiment with three GPUs, GPT-5.4 (High) failed from 6.16x to 3.04x, and its pass rate fell from 10/12 to 5/12 (Appendix G.8). Results thus show that the current failure modes of the agents identified by our benchmark are not due to a problem of the scale of the evaluations.

Regarding limited novelty

Despite there being an increasing interest in building benchmarks related to inference optimization, there are a number of factors that set InferenceBench apart from existing works. For instance, InferenceBench is open-ended by design. It is not limited to kernel optimization or to any single part of the inference stack. The agent must decide what to optimize. In addition, due to the benchmark score being based on the speedup on the PyTorch baseline, there is effectively no upper limit in how the agents can score beyond what is left theoretically available by the hardware that the model is running on. This design lets the benchmark keep measuring progress when agents begin exploring more widely and discover more ambitious optimization techniques than they are currently.

With ISO-Bench, the agent is given an existing code repository and agent submitted patches are compared against a future patch made by humans. While this a valuable contribution that is complementary to InferenceBench’s results, InferenceBench is less dependent on prior knowledge of one codebase and more on the research and development process as a whole.

Another important distinction is the coverage of current frontier capabilities. ISO-Bench's published agent evaluations include models such as Claude Sonnet 4.5 and GPT-5, whereas InferenceBench evaluates substantially newer models including Claude Opus 5.5 and GPT-6. This allows us to study the capabilities and limitations of the current generation of AI R&D agents, whose behavior may differ substantially from that of earlier models.

KernelBench and FlashInfer-Bench test single GPU kernels, which are smaller scoped and one level below a complete server. These are substantially different capabilities being tested than in InferenceBench.

These differences can also be seen empirically in the performance of frontier agents. While Claude and GPT models are much closer in performance on adjacent works that measure automated AI R&D such as PostTrainBench, Claude Opus 5.5 substantially outperforms GPT-6 configurations on InferenceBench. Similarly, newer open-weight models such as Muse Spark 1.3 fall considerably further behind the leading Claude and GPT models on InferenceBench compared to PostTrainBench. These differences in relative performance suggest that InferenceBench captures capabilities that are not fully reflected in existing AI R&D benchmarks, rather than simply reproducing their findings in a different environment.

Regarding the agents favoring vLLM, this is a failure mode of the agents and is not caused by the evaluation design in InferenceBench. The agents are not prompted to converge to vLLM and the environment clearly allows for other engines to be used as well. When forced to use SGLang in an otherwise identical environment as the main experiments, GPT-5.4 (High) even improved its Scenario A score by 28%, from 3.60x to 4.62x (Table 16), ruling out the notion that vLLM is the most optimal engine choice. This shows that the lack of exploration of agents, as mentioned by the reviewer to be externally validated in contemporary works, is due to the capabilities of the agents themselves. We observe this in our own benchmark, with Claude Opus 5.5 beating search on Scenario A by calibrating its own FP8 weights and patching vLLM's attention path (Section 3.2), something we haven’t seen previous agents explore.

Regarding prefix caching

Prefix caching is one of many levers that the open-ended setting leaves available to the agent. We do not claim that agents should use it, or that it is the correct choice for any scenario. We have revised the Section 2.2 to avoid such implications. We do agree a multi-turn scenario would be an exciting task to add to the benchmark that could require the agent to consider even more factors, and have added this to future work.

Regarding search beating most agents

This is one of the primary findings of the benchmark. Our contribution is not a state-of-the-art agent. It is a benchmark that measures how agent capability in this task grows. We noticed that many earlier agents converged to manually tuning hyperparameters, which we believed to be an unoptimal use of its time budget. This is why we added a simple non-agentic baseline that compares each agent with a cheap automated method under the same budget. A benchmark that all agents easily beat would be close to saturation and would not be as useful for tracking future progress. Here, the score has no upper limit, and the gap to search shows how far agents still have to go. The search baseline is just one simple method that agents are meant to beat as agent capability goes up.

Regarding modern agentic workloads

We agree that the current list of scenarios do not model multi-turn sessions. We believe this to be an exciting addition for future work. However, we would like to emphasize that the question the benchmark asks is whether an agent can optimize an inference server for a given workload, in an open-ended setting. The absence of a single scenario does not reduce the value of the four current tasks, each of which targets a distinct bottleneck of inference.

Regarding the SGLang search baseline

The default configuration is the one that is set by the framework developers, meaning it is already tuned for general purpose already. The search baseline initializes its search from configurations sampled from its space which are not necessarily the default configurations, and thus the search baseline has no guarantee of reaching it or something better than the default. We have expanded on the details as requested by the reviewer in Appendix B.4.

In addition, we have also run a separate set of baselines seeded from the default configuration of SGLang and have relatively minimal gains (51~53x vs 51.12x) beyond the default scores initially reported, suggesting that optimizations from tuning hyperparameters specifically for SGLang may have diminishing returns compared to the plethora of other optimization techniques the agents have access to. We have added this additional analysis in the main text.

We hope these clarifications and the additional analysis address the reviewer’s remaining concerns, and we would be grateful if the reviewer would consider raising their score.

Reviewer_uPhaOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces InferenceBench, where agents deploy and optimize an LLM inference server within a two-hour budget. It evaluates latency and throughput across four scenarios. The results show that most agents improve over default serving engines, but no agent clearly outperforms matched-budget hyperparameter search on the aggregate score.

Critical strengths

The benchmark addresses a practical problem. Including random search and Bayesian optimization provides a useful comparison for assessing the benefit of agents.

The evaluation is extensive, with 42 agent configurations, repeated runs, held-out evaluation, and useful ablations on prompts, engine choice, and starting servers.

Critical weaknesses

The practical validity of the decoding workload needs stronger support. The benchmark forces long outputs by ignoring EOS, and some of the largest gains come from n-gram speculative decoding. It is unclear whether these gains remain when responses stop naturally. The MMLU-Pro quality gate also does not directly assess the long outputs used for speed evaluation.

The main comparison uses one base model and one GPU type. Although additional models and hardware are tested, these experiments use only one agent configuration. They therefore do not establish whether the main rankings and conclusions generalize.

Questions for authors
  1. Do the decoding gains remain when responses stop naturally? How much of the measured output occurs after the first EOS?
  2. How do the results change when the search baseline allows speculative decoding lengths beyond seven tokens, matching the larger values used by the strongest agent?
  3. Are the rankings of the leading agents consistent across other base models and hardware?
  4. Could the quality evaluation include long-context and long-generation tasks that match the speed workloads?
2 comments

We thank the reviewer for recognizing the practical importance of the benchmark task and the breadth of the evaluations. We address the remaining concerns below.

Regarding decoding gains after accounting for EOS

We reran the held-out Scenario B requests with EOS ignored and with natural stopping.

With EOS ignored, around 75% of generated tokens came after the first EOS. This, as the reviewer suggests, encourages speculative decoding as a strong methodology to take advantage of the more predictable output tokens after EOS when the base model does not naturally produce long outputs. However, analysis over all agent transcripts and artifacts found that out of all of the agents evaluated, Opus 5.5 was the only agent to observe the output tokens after EOS and take advantage of this. No other agent made the decision to observe and analyze the output tokens.

We thus reran all scenario runs for Opus 5.5 with a cap on speculative decoding changes to see if removing this advantage affects the main claims of the benchmark, and got an aggregated score of 11.84x with a SEM of 0.72x. This still puts Opus 5.5 at the top of the benchmark and above the original search baseline results. The ordering and most of the gain remain under natural stopping, suggesting the forced-length workload did not affect the main rankings or claims of the benchmark.

In addition, as the reviewer suggested, we also ran a separate analysis for the quality gate that more explicitly tests for this longer decoding scenario using accuracy on LongBench v2 questions as a second quality gate along with token agreement with the unmodified FP16 reference as a third quality gate and found no shift in the rankings presented by the leaderboard.

We thank the reviewer for pointing this out, show that it does not significantly affect the current results of the benchmark or the analysis presented in the paper, and have added this point and the recommended fix of using a base model that can naturally produce longer token outputs in the Limitations section for future adoption of the benchmark.

Regarding search with speculative lengths beyond seven tokens

As requested by the reviewer, we have rerun the search baseline on vLLM hyperparameters with lengths from 0 to 47. The best non-agentic search results reached 20.17x on Scenario B, compared with 15.23x from before, thus 12.37x on the aggregate. However, the SEM for the aggregate remained at around 0.83x, meaning the results still remain comparable with the Opus 5.5 results when taking into account the variability of results. We have updated Section Appendix G.5 with these values.

Regarding generalization to other base models and hardware

There are existing ablations on other base models such as Qwen3-8B, DeepSeek-V2-Lite, and other hardware settings such as A100. They show that pass rates and speedups can change notably depending on the combination of hardware and base model, from 10/12 passing runs on Mistral-7B to 5/12 on DeepSeek-V2-Lite. In terms of general trends, we have additionally run a subset of 8 agents spread throughout the rankings shown in Table 2 and observe a high Kendall’s tau correlation of 0.86 and 0.71 for the base model of Qwen3-8B and the A100 ablation experiments respectively. This shows that while different configurations of the task can require different expertise, general trends of relative capability hold between agents. We have added this analysis to the respective Appendix sections G.2 and G.4.

We hope these clarifications and the additional experiments address the reviewer’s remaining concerns, and we would be grateful if the reviewer would consider raising their score.

Thanks for the response. I will keep my score.

Reviewer_e98oOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

InferenceBench evaluates agents that deploy and optimize an LLM inference server under a two-hour budget on one H100. Four scenarios measure prefill latency, decode latency, throughput, and a combined objective, subject to quality and integrity checks. Across 42 configurations, most agents outperform default serving engines, but nearly all remain below matched-budget hyperparameter search.

Critical strengths

The benchmark addresses a useful end-to-end engineering task with clearly specified objectives. Separate development and evaluation seeds, 504 scored runs across 42 configurations, and integrity checks validated through cross-judge agreement and manual auditing support the credibility of the evaluation.

Matched-budget search provides a valuable reference, while warm-start, structured-iteration, and forced-engine ablations help distinguish deployment failures from ineffective optimization (Appendix G). The authors appropriately acknowledge that the leading agent's 12.08x aggregate versus 11.53x for search does not establish a convincing advantage given the reported uncertainty (Section 3.1).

Critical weaknesses

Section 5.3 compares the best intermediate results obtained through the development evaluator with final held-out scores, without describing held-out reevaluation of those intermediate servers (Table 4; Appendix A.6). Selecting maxima from repeated development measurements can inflate the apparent loss from failing to preserve a good server. A comparison using the same held-out workload and gates is needed to isolate this explanation.

The original prompt does not clearly forbid third-party quantized checkpoints of the same model, yet Appendix G.1 treats their use as rule violations. The current prompt explicitly adds this restriction (Appendix B.1). This complicates the interpretation of the prompt ablation and pooled violation rate, because clarification and changes in effective task constraints are not clearly separated. Table 2 uses only current-prompt configurations, limiting the impact on the main ranking.

This is a useful and informative benchmark, and I lean toward weak acceptance. The strong baselines and targeted ablations support its contribution, although the intermediate-server comparison and interpretation of rule compliance across prompt versions need clarification.

Questions for authors
  1. Were the intermediate servers underlying Table 4 independently re-evaluated on the same held-out requests with the same quality and integrity gates? If not, how much of the reported improvement remains under that comparison?

  2. Was the prohibition on third-party quantized checkpoints communicated elsewhere to agents receiving the original prompt? How would the violation rate and prompt-ablation conclusions change if runs were judged only against explicitly communicated restrictions?

3 comments

We thank the reviewer for recognizing the value and breadth of the evaluation and the matched-budget search baseline. We address the remaining concerns below.

Regarding the evaluation of the intermediate servers in Table 4

Yes, Table 4 did reevaluate these servers using the same workload and quality/integrity gates. We have made the held out evaluation procedure more explicit in the main text to highlight this.

Regarding third-party quantized checkpoints

Yes, the original prompt states in the operational constraints section that agents must use the given model and they (the agent) may quantize the model. “Base Model: You must use {model}. You may quantize it…”

However, the results show that the more capable frontier agents found ways to interpret the prompt in a way that does not follow this constraint, hence the update to the current prompt. Table 2 and much of the main analysis and contributions of the paper utilize this current prompt.

As the reviewer noted, Table 3 and the respective analysis in Section 5.4 could be made more clear with a separation as well, so we have run this analysis. If we do not penalize third-party checkpoints under the original prompt, the integrity gate violations become 1.09% for the original prompt.

We hope these clarifications and the additional analysis regarding the integrity gate behavior address the reviewer’s remaining concerns, and we would be grateful if the reviewer would consider raising their score.

We thank the reviewer again for their constructive feedback. We have addressed both concerns raised in the initial review, clarified the held-out evaluation procedure, and added the requested analysis of integrity violations under the original prompt.

We would be grateful if the reviewer could take another look at our response and updated manuscript, and let us know whether these revisions address their remaining concerns. We would also appreciate reconsideration of the initial score in light of these clarifications.

I thank the authors for the clarification. I will maintain my positive score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary for the Area Chair

We thank all three reviewers for their thoughtful feedback and constructive suggestions.

We are grateful for the two of the three reviewers (e98o and uPha) gave us an initial rating leaning acceptance, recognizing the value of InferenceBench as a practical and important benchmark for autonomous AI engineering.

  • Reviewer e98o highlighted the clearly specified end-to-end task, extensive evaluation across 504 runs, independent development and evaluation seeds, validated integrity checks, and strong matched-budget baselines and ablations.

  • Reviewer uPha similarly recognized the practical importance of the benchmark, the inclusion of random search and Bayesian optimization baselines, and the breadth of the evaluation across 42 agent configurations.

  • Reviewer uPha also reported no substantive concerns about the paper's soundness.

We have addressed the reviewers' remaining concerns through additional experiments and manuscript revisions:

  • For Reviewer e98o, we have clarified the held-out reevaluation of intermediate servers and added a prompt-specific integrity analysis. These changes establish that intermediate and final servers are evaluated under consistent conditions and clarify how prompt wording affected the interpretation of integrity violations.

  • For Reviewer uPha, we have conducted additional experiments on natural stopping and speculative decoding, extended the search baseline, added LongBench v2 and token-agreement quality checks, and evaluated ranking consistency across base models and hardware (Kendall's τ = 0.86 and 0.71). These results demonstrate that the main rankings and overall findings of the benchmark are robust to additional quality checks, and that relative performance largely transfers across models and hardware.

  • For Reviewer nKxU, we have clarified the benchmark's scope and elaborated in detail its distinction from existing work, expanded the discussion of workload limitations and future extensions, and conducted additional SGLang experiments initialized from its default configuration. These clarifications establish the distinct capabilities measured by InferenceBench, explain the observed differences between SGLang search and default performance, and make the scope of the current evaluation explicit.

We have incorporated the results into the revised manuscript in blue and provided detailed explanations in our individual reviewer responses.

We hope these revisions address the remaining concerns noted by the reviewers and would be grateful if the reviewers would reconsider their assessments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.