- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper suggests InferenceBench, an evaluation toolkit which is used to see how good different LLM models are at improving an inference setup for certain workloads compared to a common PyTorch baseline.
Critical strengths
The paper present a study on a field that is very of interests in the system and efficient machine learning community.
Critical weaknesses
- The setting is too small to be practical. The scenarios use at most 8K input and 8K output tokens. Real long-context prefill and long-generation decode now can easily exceed 100K tokens. With a 7B model on an 96 GB H100… this setup is quite impractical as weight and activation memory is never the binding constraint: Mistral-7B’s KV cache is roughly 1 GB for an 8K request. As a result, key systems decisions such as KV-cache offloading to CPU or NVMe, paged attention, KV compression, and context or sequence parallelism are never exercised. This sits poorly with the claim of evaluating “end-to-end inference systems engineering.” The Limitations section mentions only multi-GPU, not context length or memory pressure.
- Weakness: Limited novelty. The paper claims to be the first open-ended benchmark for end-to-end inference systems engineering, but prior work already evaluates agents on closely related tasks: ISO-Bench[1] on optimizing vLLM and SGLang, and KernelBench[2] and FlashInfer-Bench on inference kernels. The open-ended action space also collapses in practice: 91.5% of runs ship vLLM and the median run tries one configuration, so the task reduces largely to serving-flag tuning. The main findings (shallow exploration, early convergence, weak experimental control) echo prior agent benchmarks. And also people are already doing inference serving improvements using different LLMs and harnesses.
- Prefix caching is presented as an optimization but never tested. The paper names prefix caching as one of the main optimization levers for Scenario A (Section 2.2), includes it in the vLLM search space (Appendix B.4), and describes an agent enabling it (Example 5). However, no scenario contains shared prefixes: every request is a distinct LongBench document, so prefix caching cannot meaningfully help in Scenarios A, B, or D. The only possible reuse is an artifact of Scenario C replaying the same 256 requests across its three traffic profiles. This is not a deliberate test of prefix caching but an uncontrolled confound, since the paper does not say whether the cache is cleared between profiles. Prefix reuse is central to real deployments, such as multi-turn chat, shared system prompts, RAG, and agent workloads, so its absence further limits how practical the benchmark is. A scenario with deliberately shared prefixes would test this lever properly.
- vLLM hyperparameter search beats most of their tested setups.
- Scenarios do not reflect modern agentic workloads. All four scenarios in Table 1 send single-turn, independent requests built from LongBench documents, with fixed input and output lengths of at most 8K tokens. Modern agentic workloads look very different. They are multi-turn sessions whose context grows to tens or hundreds of thousands of tokens. They share heavy prefixes (system prompts, tool definitions, conversation history), so prefix-cache reuse dominates. Model calls alternate with idle gaps during tool execution, and outputs per turn are often short and structured. None of this is captured, so the benchmark cannot show whether an agent can optimize for the deployments where inference cost now matters most. This is especially notable for a paper about agents, whose own workloads (like the agentic coding tools it evaluates) have exactly this profile.
- The SGLang search baseline is inconsistent with its default. On Scenario C, all three SGLang searches (31.95–33.05×) score about 35% below SGLang with default settings (51.12×), far outside the reported standard errors. A search whose space includes the default configuration should not end well below it. The SGLang default also exceeds the best vLLM search (46.70×), which is the main search baseline in Table 2. Since “search beats nearly all agents” is a central claim, the paper should explain how the search spaces and evaluation differ from the default runs.
Questions for authors
See weakness
6 comments
Regarding limited novelty
Despite there being an increasing interest in building benchmarks related to inference optimization, there are a number of factors that set InferenceBench apart from existing works. For instance, InferenceBench is open-ended by design. It is not limited to kernel optimization or to any single part of the inference stack. The agent must decide what to optimize. In addition, due to the benchmark score being based on the speedup on the PyTorch baseline, there is effectively no upper limit in how the agents can score beyond what is left theoretically available by the hardware that the model is running on. This design lets the benchmark keep measuring progress when agents begin exploring more widely and discover more ambitious optimization techniques than they are currently.
With ISO-Bench, the agent is given an existing code repository and agent submitted patches are compared against a future patch made by humans. While this a valuable contribution that is complementary to InferenceBench’s results, InferenceBench is less dependent on prior knowledge of one codebase and more on the research and development process as a whole.
Another important distinction is the coverage of current frontier capabilities. ISO-Bench's published agent evaluations include models such as Claude Sonnet 4.5 and GPT-5, whereas InferenceBench evaluates substantially newer models including Claude Opus 5.5 and GPT-6. This allows us to study the capabilities and limitations of the current generation of AI R&D agents, whose behavior may differ substantially from that of earlier models.
KernelBench and FlashInfer-Bench test single GPU kernels, which are smaller scoped and one level below a complete server. These are substantially different capabilities being tested than in InferenceBench.
These differences can also be seen empirically in the performance of frontier agents. While Claude and GPT models are much closer in performance on adjacent works that measure automated AI R&D such as PostTrainBench, Claude Opus 5.5 substantially outperforms GPT-6 configurations on InferenceBench. Similarly, newer open-weight models such as Muse Spark 1.3 fall considerably further behind the leading Claude and GPT models on InferenceBench compared to PostTrainBench. These differences in relative performance suggest that InferenceBench captures capabilities that are not fully reflected in existing AI R&D benchmarks, rather than simply reproducing their findings in a different environment.
Regarding the agents favoring vLLM, this is a failure mode of the agents and is not caused by the evaluation design in InferenceBench. The agents are not prompted to converge to vLLM and the environment clearly allows for other engines to be used as well. When forced to use SGLang in an otherwise identical environment as the main experiments, GPT-5.4 (High) even improved its Scenario A score by 28%, from 3.60x to 4.62x (Table 16), ruling out the notion that vLLM is the most optimal engine choice. This shows that the lack of exploration of agents, as mentioned by the reviewer to be externally validated in contemporary works, is due to the capabilities of the agents themselves. We observe this in our own benchmark, with Claude Opus 5.5 beating search on Scenario A by calibrating its own FP8 weights and patching vLLM's attention path (Section 3.2), something we haven’t seen previous agents explore.
Regarding prefix caching
Prefix caching is one of many levers that the open-ended setting leaves available to the agent. We do not claim that agents should use it, or that it is the correct choice for any scenario. We have revised the Section 2.2 to avoid such implications. We do agree a multi-turn scenario would be an exciting task to add to the benchmark that could require the agent to consider even more factors, and have added this to future work.
Regarding search beating most agents
This is one of the primary findings of the benchmark. Our contribution is not a state-of-the-art agent. It is a benchmark that measures how agent capability in this task grows. We noticed that many earlier agents converged to manually tuning hyperparameters, which we believed to be an unoptimal use of its time budget. This is why we added a simple non-agentic baseline that compares each agent with a cheap automated method under the same budget. A benchmark that all agents easily beat would be close to saturation and would not be as useful for tracking future progress. Here, the score has no upper limit, and the gap to search shows how far agents still have to go. The search baseline is just one simple method that agents are meant to beat as agent capability goes up.
Regarding modern agentic workloads
We agree that the current list of scenarios do not model multi-turn sessions. We believe this to be an exciting addition for future work. However, we would like to emphasize that the question the benchmark asks is whether an agent can optimize an inference server for a given workload, in an open-ended setting. The absence of a single scenario does not reduce the value of the four current tasks, each of which targets a distinct bottleneck of inference.
Regarding the SGLang search baseline
The default configuration is the one that is set by the framework developers, meaning it is already tuned for general purpose already. The search baseline initializes its search from configurations sampled from its space which are not necessarily the default configurations, and thus the search baseline has no guarantee of reaching it or something better than the default. We have expanded on the details as requested by the reviewer in Appendix B.4.
In addition, we have also run a separate set of baselines seeded from the default configuration of SGLang and have relatively minimal gains (51~53x vs 51.12x) beyond the default scores initially reported, suggesting that optimizations from tuning hyperparameters specifically for SGLang may have diminishing returns compared to the plethora of other optimization techniques the agents have access to. We have added this additional analysis in the main text.
We hope these clarifications and the additional analysis address the reviewer’s remaining concerns, and we would be grateful if the reviewer would consider raising their score.
We thank the reviewer for the detailed and careful review. Several of the reviewer's concerns relate to what the benchmark is meant to cover. So we first state more precisely what InferenceBench is designed to measure: whether an agent can autonomously set up and optimize an inference server, with no prescribed solution.
The main contribution that we present is the benchmark framework along with an analysis of the different runs of the benchmark showing its validity and usefulness as a measurement framework. We have revised the manuscript in order to make this more clear. We address the remaining concerns below.
By design, the benchmark does not forbid any of the techniques that the reviewer lists. The agent has root access, internet access, and a free choice of engine and implementation. It is free to offload or compress the KV cache or change the attention implementation, and the prompt even names many of the options it has in the open ended task (Appendix B.1).
However, the current scenarios already contain large gains that agents do not take. For instance, search on SGLang on scenario A beats search on vLLM under all three optimizers (Table 14), but 91.5% of runs ship vLLM. On Scenario B, GPT-6 Sol (Ultra) never speculative decoding and reached 5.99x, against 22.56x for the best agent which did. Due to not utilizing scheduler and memory settings no agent configuration reaches the vLLM or SGLang default on scenario C.
Even when using a more complex model such as DeepSeek-V2-Lite, the agent attempted to apply the same optimization techniques that it used on the dense model, and as a result, only 5 of 12 runs passed (Appendix G.2). On a separate experiment with three GPUs, GPT-5.4 (High) failed from 6.16x to 3.04x, and its pass rate fell from 10/12 to 5/12 (Appendix G.8). Results thus show that the current failure modes of the agents identified by our benchmark are not due to a problem of the scale of the evaluations.