acceptodds
Withdrawn from ICLR 2027

AccelMark-Agent: Benchmarking Inference-Optimization Agents Across Accelerators and Workloads

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Which serving plan is fastest depends on the deployment it runs in, so measuring how well an agent finds one requires evaluating across environments, and measuring it reliably requires a throughput number the agent did not produce for itself. We introduce \sysname, a benchmark for inference-optimization agents that supplies both. Five workloads stress different stages of the serving pipeline and run on three accelerators spanning three architectural generations, with the agent's backbone and its harness as two further axes. A trusted controller holds the measurement channel: the agent submits one atomic candidate per round and never measures, every reported gain is the median of three independent re-measurements behind an accuracy gate, failed and no-op rounds stay in every denominator, and delivered artifacts are fingerprinted so an improvement can be told apart from an unchanged baseline. Ten repeats of a fixed runner place the measurement band at . Over recorded rounds the benchmark separates agents on every factor and saturates on none: task medians run from to , backbones from zero to seven successful cells out of eight, and harness preference reverses between backbones on the same task. Because the protocol records what was delivered and not only what it scored, it also makes legible a regularity its axes were not designed to find: the size of what an agent returns tracks the configuration headroom it was handed. Holding the workload, the device and the stage-restriction prompts fixed and moving the starting configuration turns a stage that returns nothing into one that returns . The largest result in the study, , comes from a different mechanism: an agent detecting that the baseline runner had no sliding-window-attention operator and implementing one. An evaluation sampling one workload at one starting point cannot separate these regimes.

Primary area:
datasets and benchmarks
Keywords:
AI agents, LLM inference optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_jCiYOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper provides a novel approach to enabling robust agentic optimization for serving engines, and provides benchmark and its results for optimizing off-the-shelf inference engines using different inference workloads, models, and workloads using agents composed of different LLMs and agentic harnesses.

Critical strengths

The paper provides an interesting agentic workflow design for optimizing inference engines and provides a wide range of results across various inference workloads, hardware, and agent configurations.

Critical weaknesses
  1. It is not clear what it is focusing on. Is it the benchmark results? Is it the agentic serving engine optimization protocol design? Is it the optimization techniques and the findings from the benchmarks? Or the problems that arise while running agentic optimizations? I assume the main contribution is the benchmark results, following the title.

  2. The benchmark focuses on outdated hardware (A100) and models (Llama 3, Qwen 2.5). This setup lacks state-of-the-art optimization opportunities (native FP8 tensor cores, NVFP4 tensor cores, Tensor Memory Accelerator, Warp-specialization, etc.) and model architecture design (MoE, hybrid attention, etc.). This makes the benchmark results less relevant for evaluating the performance of the suggested agent configurations on modern inference deployment scenarios.

  3. The benchmark results are weak. The presented results are from single-shot search results where many of the agent configurations failed to produce a meaningful improvement. Also, the benchmark provides no real information on how inference engines can be optimized. Case study results seem to suggest that the improvements are mostly from hyperparameter search.

Questions for authors
  1. Why did you select such outdated hardware and models for benchmarking? It nullifies the relevance of the benchmark results.

  2. The design of the agentic optimization infrastructure for the inference engine is quite interesting. However, despite such a sophisticated design, the improvements that the agents found seem to be mostly from hyperparameter tuning. Were there any other architectural or algorithmic improvements found by the agents?

Reviewer_FJYoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AccelMark-Agent, a benchmark for testing how well AI coding agents can make LLM inference systems faster.

The authors test five workloads, five AI models, and two agent tools, mainly on A100 GPUs. Instead of trusting the fastest result found during a search, they check the final program for accuracy and measure its speed three more times.

Out of 50 experiments, 26 produced a changed program that was more than 2% faster than the starting version.

The paper also studies why some agents fail, how results depend on the starting system, and why some promising improvements do not survive final checks. The main contribution is a benchmark that measures not only how much faster an agent makes a system, but also whether it actually delivers a working improvement.

Critical strengths

1. The paper carefully checks whether an improvement is real.

I like that the authors do not simply report the fastest result an agent finds. They check the final program, measure it again, and keep track of failed attempts.

Table 6 gives a good example of why this matters. One candidate appeared to be about 255 times faster and passed the accuracy check, but it had changed the workload by caching and reusing results. The authors correctly excluded it.

This is an important strength because a benchmark should reward agents for making the original task faster, not for finding ways around the test.

2. The experiments show useful differences between agents and workloads.

The paper tests several models and agent tools across five different serving tasks. It also reports failed attempts instead of hiding them.

I found the distinction between agents that cannot produce working changes and agents that produce working but slower changes particularly useful.

The authors are also careful about their conclusions. They acknowledge that most experiments were run only once, that the starting setup affects the size of improvements, and that their results do not establish a reliable ranking of agents. This makes the analysis more convincing.

Critical weaknesses

The main weakness is that it is not clear enough how the benchmark checks that a faster program is still doing the same work.

The authors describe several useful checks in Appendix C, but some important details are missing.

For example, the accuracy test uses a fixed subset of MMLU questions, but the paper does not clearly specify which questions are used or the full evaluation settings. The PRECISION task requires five numerical formats, but I could not find a clear explanation of how their results are combined into the single speed score. Table 1 also gives ranges of request concurrency without fully explaining which settings determine the main scores.

My bigger concern is how the authors check that an optimized program still runs the original workload correctly.

An agent can change the program, its configuration, and its dependencies. Checking accuracy on a small set of questions does not necessarily prove that the program still processes the same requests, handles the same context, or performs the intended attention computation.

The paper itself shows why this matters. In the V100 example in Table 6, a candidate passed the accuracy test but was later rejected because it changed the workload. However, the paper does not explain the checks behind this decision clearly enough for someone else to repeat them.

This is especially important for the two largest reported improvements, +211.72% and +137.82%. Section 4.3 and Appendix B.4 say that the final programs are missing from the archive. The explanation of these improvements therefore relies mainly on recorded logs and agent notes.

I am not suggesting that these speedups are incorrect. The authors report that they passed validation. However, without the final programs or enough information to reconstruct them, it is difficult to independently check these important results.

This matters because the main purpose of the benchmark is to measure real, valid improvements. If the rules for deciding what counts as a valid improvement are not clear enough, different people may get different results when using the benchmark.

I think the paper makes a useful contribution, and the reported numbers are internally consistent. However, I lean toward weak rejection because the evaluation rules and supporting evidence need to be easier to verify. A clearer description of the checks and access to the relevant results could change my recommendation.

Questions for authors

1. How do you check that an optimized program still performs the original task?

Could you explain the exact checks used to make sure the program processes the same requests, preserves the required attention behavior, and counts output tokens correctly?

I would also appreciate clarification of which MMLU questions are used, how the five PRECISION formats contribute to the final score, and which concurrency settings determine the reported throughput.

In particular, what checks led you to reject the V100 caching example, and were the same checks applied to every accepted result?

2. Can the two largest speedups be independently verified?

The improvements of +211.72% and +137.82% are among the most interesting results in the paper, but the final programs are not included in the archive.

Would it be possible to provide these programs or reconstruct them from the saved changes? It would also help to see their accuracy checks and the measurements used to calculate the final improvements.

If the programs cannot be recovered, what other evidence is available to confirm that they performed the original workload correctly?

Clear answers to these two questions would have a meaningful effect on my overall assessment.

Reviewer_kCjwOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper addresses the challenge of evaluating coding agents for optimizing end-to-end LLM inference deployments, where speedups may actually just reflect measurement noise, reduced accuracy, or changes to the intended workload. It introduces AccelMark-Agent, a benchmark covering five inference workloads and multiple accelerators, backbones, and harnesses.

Critical strengths
  • It provides examples of interactions between optimizations across the stack in LLM inference that may make performance difficult to predict
  • Stresses that the highest throughput during search is not reliable with examples
Critical weaknesses

My major concern with this work is that the writing and presentation is difficult to follow. This makes it hard to assess both the need for the benchmark and the significance of its results. In particular, I have the following concerns.

First, the motivation for this benchmark is insufficiently developed and presented. As a reader who is not an expert in inference-serving benchmarks, I found it difficult to understand precisely what capability is missing from existing evaluations and why addressing that gap is significant. The paper discusses related benchmarks, including InferenceBench and ISO-Bench, but does not convincingly establish why its particular combination of workloads, accelerators, and evaluation controls provides a substantially more useful assessment. The intended users and use cases should also be articulated more concretely: what would an agent developer or deployment engineer learn from this benchmark, and what decisions could its results reliably support?

Second, there is insufficient justification for how the benchmark design addresses its stated goals. Table 1 identifies different bottlenecks and levels of implementation maturity, but this does not establish why these five selected workload configurations provide a representative or discriminative assessment of deployment optimization. Similarly, the choice of two agent harnesses.

Third, the relevance of the evaluation matrix to the benchmark’s intended use is not sufficiently justified. For example, is the matrix intended to identify suitable agents for particular workloads, reveal limitations of agent harnesses, or characterize optimization opportunities across serving systems? What are the intended conclusions

Questions for authors

Please address the points above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.