- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Accelerate Bench, a benchmark for evaluating coding agents on application-level performance optimization. Unlike commit-derived benchmarks such as GSO and SWE-Perf, Accelerate Bench defines tasks through complete application workloads and correctness constraints, allowing agents to identify bottlenecks and optimize algorithms, source code, build configurations, and runtime settings. The benchmark contains 32 CPU optimization tasks adapted from scientific applications and supercomputing competitions, with public testing and private verification.
The authors evaluate 12 model–harness combinations and analyze 480 additional runs from five agents. Among 384 main evaluation outcomes, 325 pass correctness checks, but 49.5% of correct submissions achieve less than 1.25× speedup. Further trajectory analyses and controlled experiments reveal limitations in agents' optimization strategies, including underuse of compiler tuning, inconsistent candidate selection, and dependencies between optimization decisions.
Critical strengths
-
Realistic application-level benchmark. Accelerate Bench evaluates end-to-end optimization rather than isolated code edits. Its workload-defined tasks, flexible optimization scope, and correctness verification provide a meaningful complement to GSO, SWE-Perf, and SWE-fficiency.
-
Comprehensive empirical evaluation. The benchmark evaluates 12 coding agents across 32 tasks, with an additional 480-run trajectory analysis. The finding that 49.5% of correct submissions achieve less than 1.25× speedup highlights the gap between functional correctness and effective optimization.
-
Insightful cross-layer analysis. Controlled experiments demonstrate how build configurations, runtime settings, and code changes interact. For example, identical GROMACS source code achieves 0.289×–2.123× speedup under different configurations, highlighting limitations of local optimization strategies.
Critical weaknesses
-
Limited workload representativeness. The 32 tasks focus on CPU workloads scaled down from supercomputing applications. The paper provides limited evidence that these adaptations preserve the original performance bottlenecks and optimization opportunities, potentially restricting generalizability.
-
Uncertain ranking stability. The main evaluation reports 384 outcomes across 12 agents and 32 tasks, suggesting one run per agent–task pair. Although Section 5 includes repeated runs for five agents, the paper does not establish the statistical stability of the full agent rankings, particularly the 40.3% versus 36.5% Score@1 difference between the top two agents.
-
Limited generality of behavioral findings. The trajectory analysis covers only five selected agents, and candidate-selection conclusions rely on 31 version histories across seven applications. Some findings are therefore better interpreted as illustrative observations than generalizable agent behaviors.
Questions for authors
-
Workload fidelity: How do the authors verify that scaled-down workloads preserve the dominant performance bottlenecks and optimization opportunities of the original applications?
-
Ranking robustness: How stable are agent rankings across independent runs? Could the authors provide uncertainty estimates, particularly for the leading agents?
-
Behavioral generalization: How representative are the 31 selected version histories of the full evaluation? Could selection effects or timing noise influence the candidate-selection findings?
-
Benchmark differentiation: What specific optimization capabilities does Accelerate Bench evaluate that existing benchmarks such as SWE-fficiency and RE-Bench cannot adequately capture?
6 comments
Response to Reviewer MJHa (2/2)
W3 / Q3: How general are the behavioral findings, and how representative are the 31 version histories?
The reviewer is right on both counts. The trajectory analysis covers five agents, which do not include the strongest agents of Table 1, and the Section 5 findings on how agents test, choose and keep candidate programs are best read as illustrative observations about these runs. The revision presents them this way and states the evidence each rests on:
- Final programs support counts with stated denominators. Of the 480 runs (5 agents x 32 tasks x 3 runs), 476 have technique labels; computation or reuse changes appear in 225 of them, and compiler and build tuning in 40.
- The 31 version histories show how candidate programs evolve within a run. A run is included when it has at least two correct candidates timed the same way and its candidate source code was kept; no outcome measure is used. The histories cover 7 applications and do not represent all tasks. The finding that the last correct candidate is more than 5% slower than the best in 11 of the 31 describes only these histories; it does not estimate how often this happens across runs.
- Controlled replays confirm specific mechanisms. With identical GROMACS source, build and runtime settings alone change the speedup from 0.289x to 2.123x; QiboTN's deferred loading gives 1.01x on its own and 6.3–6.4x combined with another change. Reviewer qq5K found that these experiments make the explanations more credible, and Reviewer pZJZ highlighted the non-intuitive tradeoffs from the order and combination of optimizations.
Timing noise can indeed affect candidate comparisons. Appendix L replays 7 HPCG-derived runs. In 6 of them, the candidate with the best public-test time is not the final submission. Over the replay rounds, the final submission is faster than that candidate in every round in 2 runs, slower in every round in 2, and the order changes between rounds in the other 2. A single timing can therefore pick the wrong candidate, which is why we present the 11-of-31 finding as an illustration, not as an estimate of how often agents submit a slower program.
Q4: What does Accelerate Bench evaluate that SWE-fficiency and RE-Bench do not?
These are the closest evaluations; here is what each measures.
- SWE-fficiency builds tasks from 498 performance pull requests in 9 Python libraries. The agent receives the repository and the slow workload used by the pull-request author, mainly edits Python/Cython, and is checked by existing unit tests that cover the expert patch; the score is speedup relative to the expert pull request. Its authors list C/C++/Rust and longer-running workloads as future extensions.
- RE-Bench has 7 machine-learning research-engineering environments with an 8-hour budget and human-expert comparisons; 2 are runtime-optimization tasks, an LLM Foundry fine-tuning script and a Triton prefix-sum kernel on GPU. Scores are normalized between the starting and reference solutions.
Repository access, long budgets and public/private evaluation are not new; prior work provides them. Accelerate Bench differs in the following combination:
- Compiled scientific applications with their full build process. Tasks span mixed C/C++/Fortran/Python applications. Agents can change source code, compiler options, SIMD and threading settings, and libraries, and these choices interact (GROMACS: 0.289x–2.123x; QiboTN: 1.01x versus 6.4x).
- Correctness checked on the whole application. Beyond numerical tolerances, tasks require complete outputs, recovery from a checkpoint in a new process, and consistent outputs when a program is run again. In Appendix E, four deliberately invalid variants pass component tests but fail the application checks; Reviewer qq5K also noted that these checks catch problems that simpler tests miss.
- Scores are not tied to an expert patch. Speedups are measured against the provided program on unseen, often larger private inputs.
- CPU scientific computing. RE-Bench's runtime tasks are ML on GPU; SWE-fficiency targets Python data-science libraries.
The capabilities these tasks require include tracing which code the application actually executes, keeping compiler optimizations within numerical tolerances, and coordinating build, runtime and code changes; Table R4 (in our response to Reviewer pZJZ) gives an example of each.
Thank you again for the careful review. If our response has addressed your concerns, we kindly ask you to consider raising your score.
The Authors
Dear Reviewer MJHa,
As the discussion phase comes to an end, please acknowledge the author's response and consider leaving them a comment on whether it has resolved some or all of your concerns.
Thanks!
Response to Reviewer MJHa (1/2)
We thank the reviewer for the careful reading and for recognizing the task design and the cross-layer analysis.
W1 / Q1: Do scaled workloads keep the original bottlenecks and optimization opportunities?
The submission does not show that the scaled tasks keep the bottlenecks of competition-scale runs, and it does not claim to: Section 2.2 states that they keep the original computational stages, inter-stage dependencies and correctness criteria, and Section 7 limits the results to the evaluated scales. Competition workloads run for hours on multi-node clusters; scaling them to one node is what lets an agent profile, edit, build and test repeatedly within 8 hours. The question for a scaled task is therefore whether it still requires the same kind of optimization work: finding where the time goes and changing the program without changing its results. Three pieces of evidence bear on this:
W2 / Q2: Are rankings stable across independent runs?
The reviewer is right that the submission does not establish the order of its top two agents. The submission also omitted how its runs were chosen: eight of its twelve agents had three runs per task, and Table 1 reported the median-reward run (now stated in Appendix B.1); GPT-6 Astra, Gemini-3.8-Flash, GLM-5.3 and Kimi K3 had one.
These repeated runs show how much an agent's Score@1 changes when it runs again (Table R2): the standard deviation across three runs is 0.6–3.9 points, and Hy4-Preview scores 34.61%, 32.74% and 40.18%, a 7.4-point range. The ranking of these eight agents changes little: Hy4-Preview is first, DeepSeek-V4.1-Flash second and Hy3 last in every round, and the only changes are swaps between Grok 4.6 and GLM-5.3-Flash (third and fourth) and among DeepSeek-V4-Pro, DeepSeek-V4-Flash and GLM-5.2 (fifth to seventh); Kendall's τ between two rounds is 0.79–0.86. GPT-6 Astra, however, ran only once, and its 3.8-point lead over Hy4-Preview is smaller than Hy4-Preview's range, so we do not use v1.0 to order the top two agents.
On v1.1, both agents ran three times per task (Table R1). GPT-6 Astra scores 53.06%, 50.62% and 48.10% (50.59 ± 2.48), and Hy4-Preview 43.24%, 40.62% and 34.63% (39.50 ± 4.42). A paired bootstrap over tasks puts GPT-6 Astra 11.1 points higher (95% interval 5.2–17.7), more than either agent's run-to-run standard deviation. Re-scoring GPT-6 Astra's unchanged programs moves its Score@1 by 1.45 points (standard deviation; our response to qq5K, Q1e).
The revised Table 1 has a new second place: GPT-6.1 Sol, 2.0 points below GPT-6 Astra with one run each. One run does not order these two: re-scoring the same programs moves a single run's Score@1 by about 1.5 points, and GPT-6 Astra's own three runs span 48.1–53.1 (Section 4 states this). Likewise, Kimi K3, Hy4-Preview and GPT-6.1 Terra are within 4 points of one another in one run. Second and third runs of the other agents are under way; Table 1 of the final version will report three-run means and intervals, and we will post the comparison of GPT-6 Astra and GPT-6.1 Sol here when Sol's runs finish.