acceptodds
Under review as a conference paper at ICLR 2027

Accelerate Bench: Evaluating Agents for Application-Level Program Optimization

Abstract

Application optimization requires coding agents to discover useful changes and coordinate them across a complete program while preserving the required computation. We introduce Accelerate Bench, a benchmark of 32 CPU optimization tasks adapted from scientific applications and supercomputing competitions. Each task provides a working program and specifies the workload, correctness requirements, timing boundaries, and permitted edits, leaving agents to choose where and how to optimize. Public testing supports iterative development, and a separate verifier scores the final submission. We evaluate 12 coding agents, each pairing a model with a harness. Of their 325 correct outcomes, 161 (49.5%) stay below 1.25× speedup. Controlled replays show why coordination across an application matters: identical source code runs between 0.29× and 2.12× depending only on build and runtime settings, and one code change is worth 1.01× or 6.4× depending on another. Accelerate Bench pairs executable workloads with agents' programs and trajectories to evaluate both their gains and the process behind them.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Agent Evaluation, Long-Horizon Tasks, Coding Agents, Application-Level Program Optimization

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_MJHaOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Accelerate Bench, a benchmark for evaluating coding agents on application-level performance optimization. Unlike commit-derived benchmarks such as GSO and SWE-Perf, Accelerate Bench defines tasks through complete application workloads and correctness constraints, allowing agents to identify bottlenecks and optimize algorithms, source code, build configurations, and runtime settings. The benchmark contains 32 CPU optimization tasks adapted from scientific applications and supercomputing competitions, with public testing and private verification.

The authors evaluate 12 model–harness combinations and analyze 480 additional runs from five agents. Among 384 main evaluation outcomes, 325 pass correctness checks, but 49.5% of correct submissions achieve less than 1.25× speedup. Further trajectory analyses and controlled experiments reveal limitations in agents' optimization strategies, including underuse of compiler tuning, inconsistent candidate selection, and dependencies between optimization decisions.

Critical strengths
  1. Realistic application-level benchmark. Accelerate Bench evaluates end-to-end optimization rather than isolated code edits. Its workload-defined tasks, flexible optimization scope, and correctness verification provide a meaningful complement to GSO, SWE-Perf, and SWE-fficiency.

  2. Comprehensive empirical evaluation. The benchmark evaluates 12 coding agents across 32 tasks, with an additional 480-run trajectory analysis. The finding that 49.5% of correct submissions achieve less than 1.25× speedup highlights the gap between functional correctness and effective optimization.

  3. Insightful cross-layer analysis. Controlled experiments demonstrate how build configurations, runtime settings, and code changes interact. For example, identical GROMACS source code achieves 0.289×–2.123× speedup under different configurations, highlighting limitations of local optimization strategies.

Critical weaknesses
  1. Limited workload representativeness. The 32 tasks focus on CPU workloads scaled down from supercomputing applications. The paper provides limited evidence that these adaptations preserve the original performance bottlenecks and optimization opportunities, potentially restricting generalizability.

  2. Uncertain ranking stability. The main evaluation reports 384 outcomes across 12 agents and 32 tasks, suggesting one run per agent–task pair. Although Section 5 includes repeated runs for five agents, the paper does not establish the statistical stability of the full agent rankings, particularly the 40.3% versus 36.5% Score@1 difference between the top two agents.

  3. Limited generality of behavioral findings. The trajectory analysis covers only five selected agents, and candidate-selection conclusions rely on 31 version histories across seven applications. Some findings are therefore better interpreted as illustrative observations than generalizable agent behaviors.

Questions for authors
  1. Workload fidelity: How do the authors verify that scaled-down workloads preserve the dominant performance bottlenecks and optimization opportunities of the original applications?

  2. Ranking robustness: How stable are agent rankings across independent runs? Could the authors provide uncertainty estimates, particularly for the leading agents?

  3. Behavioral generalization: How representative are the 31 selected version histories of the full evaluation? Could selection effects or timing noise influence the candidate-selection findings?

  4. Benchmark differentiation: What specific optimization capabilities does Accelerate Bench evaluate that existing benchmarks such as SWE-fficiency and RE-Bench cannot adequately capture?

6 comments
AuthorsOpenReview ↗

Response to Reviewer MJHa (1/2)

We thank the reviewer for the careful reading and for recognizing the task design and the cross-layer analysis.

W1 / Q1: Do scaled workloads keep the original bottlenecks and optimization opportunities?

The submission does not show that the scaled tasks keep the bottlenecks of competition-scale runs, and it does not claim to: Section 2.2 states that they keep the original computational stages, inter-stage dependencies and correctness criteria, and Section 7 limits the results to the evaluated scales. Competition workloads run for hours on multi-node clusters; scaling them to one node is what lets an agent profile, edit, build and test repeatedly within 8 hours. The question for a scaled task is therefore whether it still requires the same kind of optimization work: finding where the time goes and changing the program without changing its results. Three pieces of evidence bear on this:

  1. Scores already require gains to carry over to larger inputs. On 16 tasks, the private inputs used for scoring are larger than the public inputs used during development: miniVite grows from at most 4,096 to at most 32,768 vertices, OpenFOAM from a 48x48 grid and 300 steps to as much as 96x96 and 1,200 steps, and PRESTO from at most 524,288 to 1,048,576 samples. A change that helps only on small inputs earns less. Some gains do come from fixed costs such as startup, library imports and hardware detection (for example, QiboTN's deferred loading); these are part of the timed workflow but shrink as inputs grow, which motivates the next test.
  2. The largest speedups, re-scored at larger sizes. On HPCG-derived, QiboTN and Paged LM generation, agents obtained their largest speedups by changing how the result is computed, for example a direct solver instead of an iterative one. Before seeing any result, we fixed the programs to test: all nine of GPT-6 Astra's v1.1 final programs on these tasks (three runs each). The verifier re-runs them unchanged, without rerunning the agent, on HPCG-derived grids up to 128³, QiboTN circuits of up to 16 qubits and depth 16, and Paged LM prompts of up to 512 tokens with 64 generated tokens, after a control run of the unmodified program at the same sizes. We will post the correctness and speedup of every program at every size here, including failures.
  3. Review by competition finalists. Before submission, student supercomputing competition finalists reviewed the task designs; after submission, they reviewed the v1.1 statements, which now specify the permitted edits and the computation that must be preserved.

W2 / Q2: Are rankings stable across independent runs?

The reviewer is right that the submission does not establish the order of its top two agents. The submission also omitted how its runs were chosen: eight of its twelve agents had three runs per task, and Table 1 reported the median-reward run (now stated in Appendix B.1); GPT-6 Astra, Gemini-3.8-Flash, GLM-5.3 and Kimi K3 had one.

These repeated runs show how much an agent's Score@1 changes when it runs again (Table R2): the standard deviation across three runs is 0.6–3.9 points, and Hy4-Preview scores 34.61%, 32.74% and 40.18%, a 7.4-point range. The ranking of these eight agents changes little: Hy4-Preview is first, DeepSeek-V4.1-Flash second and Hy3 last in every round, and the only changes are swaps between Grok 4.6 and GLM-5.3-Flash (third and fourth) and among DeepSeek-V4-Pro, DeepSeek-V4-Flash and GLM-5.2 (fifth to seventh); Kendall's τ between two rounds is 0.79–0.86. GPT-6 Astra, however, ran only once, and its 3.8-point lead over Hy4-Preview is smaller than Hy4-Preview's range, so we do not use v1.0 to order the top two agents.

On v1.1, both agents ran three times per task (Table R1). GPT-6 Astra scores 53.06%, 50.62% and 48.10% (50.59 ± 2.48), and Hy4-Preview 43.24%, 40.62% and 34.63% (39.50 ± 4.42). A paired bootstrap over tasks puts GPT-6 Astra 11.1 points higher (95% interval 5.2–17.7), more than either agent's run-to-run standard deviation. Re-scoring GPT-6 Astra's unchanged programs moves its Score@1 by 1.45 points (standard deviation; our response to qq5K, Q1e).

The revised Table 1 has a new second place: GPT-6.1 Sol, 2.0 points below GPT-6 Astra with one run each. One run does not order these two: re-scoring the same programs moves a single run's Score@1 by about 1.5 points, and GPT-6 Astra's own three runs span 48.1–53.1 (Section 4 states this). Likewise, Kimi K3, Hy4-Preview and GPT-6.1 Terra are within 4 points of one another in one run. Second and third runs of the other agents are under way; Table 1 of the final version will report three-run means and intervals, and we will post the comparison of GPT-6 Astra and GPT-6.1 Sol here when Sol's runs finish.

AuthorsOpenReview ↗

Response to Reviewer MJHa (2/2)

W3 / Q3: How general are the behavioral findings, and how representative are the 31 version histories?

The reviewer is right on both counts. The trajectory analysis covers five agents, which do not include the strongest agents of Table 1, and the Section 5 findings on how agents test, choose and keep candidate programs are best read as illustrative observations about these runs. The revision presents them this way and states the evidence each rests on:

  • Final programs support counts with stated denominators. Of the 480 runs (5 agents x 32 tasks x 3 runs), 476 have technique labels; computation or reuse changes appear in 225 of them, and compiler and build tuning in 40.
  • The 31 version histories show how candidate programs evolve within a run. A run is included when it has at least two correct candidates timed the same way and its candidate source code was kept; no outcome measure is used. The histories cover 7 applications and do not represent all tasks. The finding that the last correct candidate is more than 5% slower than the best in 11 of the 31 describes only these histories; it does not estimate how often this happens across runs.
  • Controlled replays confirm specific mechanisms. With identical GROMACS source, build and runtime settings alone change the speedup from 0.289x to 2.123x; QiboTN's deferred loading gives 1.01x on its own and 6.3–6.4x combined with another change. Reviewer qq5K found that these experiments make the explanations more credible, and Reviewer pZJZ highlighted the non-intuitive tradeoffs from the order and combination of optimizations.

Timing noise can indeed affect candidate comparisons. Appendix L replays 7 HPCG-derived runs. In 6 of them, the candidate with the best public-test time is not the final submission. Over the replay rounds, the final submission is faster than that candidate in every round in 2 runs, slower in every round in 2, and the order changes between rounds in the other 2. A single timing can therefore pick the wrong candidate, which is why we present the 11-of-31 finding as an illustration, not as an estimate of how often agents submit a slower program.

Q4: What does Accelerate Bench evaluate that SWE-fficiency and RE-Bench do not?

These are the closest evaluations; here is what each measures.

  • SWE-fficiency builds tasks from 498 performance pull requests in 9 Python libraries. The agent receives the repository and the slow workload used by the pull-request author, mainly edits Python/Cython, and is checked by existing unit tests that cover the expert patch; the score is speedup relative to the expert pull request. Its authors list C/C++/Rust and longer-running workloads as future extensions.
  • RE-Bench has 7 machine-learning research-engineering environments with an 8-hour budget and human-expert comparisons; 2 are runtime-optimization tasks, an LLM Foundry fine-tuning script and a Triton prefix-sum kernel on GPU. Scores are normalized between the starting and reference solutions.

Repository access, long budgets and public/private evaluation are not new; prior work provides them. Accelerate Bench differs in the following combination:

  1. Compiled scientific applications with their full build process. Tasks span mixed C/C++/Fortran/Python applications. Agents can change source code, compiler options, SIMD and threading settings, and libraries, and these choices interact (GROMACS: 0.289x–2.123x; QiboTN: 1.01x versus 6.4x).
  2. Correctness checked on the whole application. Beyond numerical tolerances, tasks require complete outputs, recovery from a checkpoint in a new process, and consistent outputs when a program is run again. In Appendix E, four deliberately invalid variants pass component tests but fail the application checks; Reviewer qq5K also noted that these checks catch problems that simpler tests miss.
  3. Scores are not tied to an expert patch. Speedups are measured against the provided program on unseen, often larger private inputs.
  4. CPU scientific computing. RE-Bench's runtime tasks are ML on GPU; SWE-fficiency targets Python data-science libraries.

The capabilities these tasks require include tracing which code the application actually executes, keeping compiler optimizations within numerical tolerances, and coordinating build, runtime and code changes; Table R4 (in our response to Reviewer pZJZ) gives an example of each.


Thank you again for the careful review. If our response has addressed your concerns, we kindly ask you to consider raising your score.

The Authors

Dear Reviewer MJHa,

As the discussion phase comes to an end, please acknowledge the author's response and consider leaving them a comment on whether it has resolved some or all of your concerns.

Thanks!

Reviewer_qq5KOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Accelerate Bench tests whether coding agents can make complete applications faster while preserving their required behavior. It contains 32 CPU tasks adapted from scientific applications and competition themes. Agents have eight hours to change code, build options, and runtime settings using public tests. A separate verifier checks the final submission on private workloads. The main score measures the share of baseline runtime saved; it is not a task success rate. Across 12 systems, the best score is 40.3%, and 161 of 325 correct outcomes remain below 1.25× speedup. A separate 480-run analysis and controlled replays examine how agents find, combine, and retain improvements.

Critical strengths
  1. The correctness checks cover the application as a whole. Tasks require complete outputs and, where relevant, correct checkpoint recovery. This helps prevent speedups from skipping required work. In the two-task audit, all eight deliberately invalid variants are rejected. Four paged-generation variants pass component checks but fail the application checks (Appendix C). That is useful evidence that the broader checks catch problems simpler tests miss.

  2. The paper explains mechanisms, not just scores. With identical GROMACS source, changing build and thread settings produces speedups from 0.289× to 2.123×. In QiboTN, delaying library imports helps little on its own but gives a 6.3–6.4× gain when another change makes those imports unnecessary (Appendix J). The paper also checks sampled annotations and adjusts some comparisons for task and model differences. These steps make its explanations more credible.

Critical weaknesses

The stability of the ranking is unclear. Table 1 uses one run per agent and task. The separate repeated-run panel does not include the two leading systems. The paper also does not state how it chooses one run when several are eligible. Timing can matter: one optimized HPCG program has a 14.54% coefficient of variation in runtime across repeated rounds (Appendix E.1). This is one example, rather than a noise estimate for the whole benchmark. It also means that claims about submitting a slower candidate need care when most candidate timings were measured only once.

Some gains depend on how much tuning the starting program already has. For example, the GROMACS reference disables SIMD, a standard way to use CPU hardware efficiently. Some tasks are also newly constructed, reduced workloads. Improving these programs is useful, but the score alone cannot tell us how much comes from straightforward configuration changes or how much improvement remains possible. A broader reference using sensible configuration tuning would make the benchmark's difficulty easier to interpret. The paper acknowledges this limitation and provides a useful partial check for HPCG.

Questions for authors

How was the reported run selected, and how many runs were excluded for infrastructure problems? What variation appears in the existing repeated-run panel, and are repeated results available for the leading systems? Please separate variation in agent search from variation in timing a fixed program.

How much speedup does a fixed configuration-only tuning procedure achieve under the same task rules? How much additional improvement do the agents provide?

Were the main verifier runs given dedicated physical cores within one NUMA node, as in the controlled replays (App. J), and was background load recorded, as in the App. C controls? How were baseline and candidate execution orders handled?

7 comments
AuthorsOpenReview ↗

Response to Reviewer qq5K (1/3)

We thank the reviewer for these specific questions.

Q1a: How was the reported run selected?

In v1.0, eight agents ran three times per task: Hy4-Preview, DeepSeek-V4.1-Flash, GLM-5.3-Flash, Grok 4.6, DeepSeek-V4-Pro, GLM-5.2, DeepSeek-V4-Flash and Hy3. For each task, these are the three most recent eligible runs by completion time; a run is eligible if it finished with a verifier result and was not stopped by an infrastructure or model-API failure. The submitted Table 1 reports the run with the median reward, breaking ties by completion time and then trial identifier; correctness and usage play no part. GPT-6 Astra, Gemini-3.8-Flash, GLM-5.3 and Kimi K3 ran once per task. The submission omitted this rule; Appendix B.1 now states it.

The reviewer is right that the 480-run panel analyzed in Section 5 does not include the two leading agents. Hy4-Preview, second in the submitted Table 1, nonetheless had three runs per task in the main evaluation (Table R2), and both leading agents now have three v1.1 runs (Table R1).

Q1b: How many runs were excluded for infrastructure problems?

No agent–task pair is missing or scored zero because of infrastructure. A run stopped by an infrastructure or model-API failure receives no score; its slot is filled by a continuation, which resumes the interrupted run from its conversation history, or by a new run. Failures caused by the agent, including incorrect output, out-of-memory termination (exit 137) and timeout, are scored. Table R3 (Table 7 of the revision) accounts for all runs of the eight three-run agents. Each one-run agent has exactly one scored run per task; for Kimi K3 on 16 tasks and GLM-5.3 on 1, that run is a continuation and the interrupted run is not scored.

Table R3: Runs of the eight v1.0 agents with three runs per task (32 tasks)

CategoryRunsScored?
Runs behind the submitted Table 1 (8 x 32 x 3)768Yes
Model-API failure after retries were exhausted215No; a continuation or new run fills the slot
Infrastructure cancellation108No; same
Infrastructure runtime error18No; same
Error at termination that could not be attributed4No; same
Verifier result missing or verifier fault, to be re-scored4No
Eligible but not among the three most recent65 (in 49 agent–task pairs)No
Interrupted run replaced by its continuation13No; the continuation fills the slot

The only out-of-memory run left unscored is one whose verifier result is missing. v1.1 follows the same rules: a run stopped by a model-API, sandbox, image-build or verifier-environment failure without a valid result is not scored and is replaced by a new run with the same settings, whose result is used whatever it is. Failures caused by the agent are scored, including out-of-memory termination and timeout; after a timeout, the verifier still scores the program the agent left.

Which three runs count depends only on completion time and eligibility, never on scores. We scheduled replacement runs by which agent–task pairs still lacked runs and when the model API was available. A continuation keeps the interrupted run's conversation and gets the same 8-hour limit as any run, not the time the interrupted run had left, so the two together can exceed 8 hours: among 91 continuations in an earlier set of 842 runs, the median combined time was 2.92 hours, and 10 exceeded 8 hours. This applies only to v1.0: v1.1 replaces every interrupted run with a new run, so every scored v1.1 run had the same 8-hour budget. For the final version, Appendix B will compare results with and without continuations, and we will re-score the 65 other eligible runs with the corrected verifiers and report scores and rankings using all eligible runs.

Q1c: How much do the existing repeated runs vary?

See Table R2 in the general response. On a single task, the same agent's reward varies across its three runs with a standard deviation of 6.9–13.2 points on average, depending on the agent; averaged over 32 tasks, Score@1 varies by 0.6–3.9 points.

Q1d: Are repeated results available for the leading agents?

For the two leading agents of the submitted Table 1, yes: GPT-6 Astra and Hy4-Preview each have three independent v1.1 runs per task (Table R1). In the revised Table 1, GPT-6.1 Sol is second, 2.0 points below GPT-6 Astra with one run each, which does not order them (Section 4). Sol's second and third runs are under way, and we will post the comparison here.

AuthorsOpenReview ↗

Response to Reviewer qq5K (2/3)

Q1e: Can variation in agent search be separated from timing variation of a fixed program?

To separate them, we re-scored the v1.1 final programs of GPT-6 Astra and Hy4-Preview twice more with the unchanged private verifier, without changing the programs or calling a model. This covers all 96 of GPT-6 Astra's programs and 77 of Hy4-Preview's 96; the other 19 were still running or not yet scored when this experiment began.

MeasureGPT-6 Astra (96 programs)Hy4-Preview (77 programs)
SD of Score@1 when the same programs are re-scored1.45 points (95% interval over tasks 0.70–2.04)1.57 points
SD of Score@1 across three independent runs2.48 points4.42 points
Median coefficient of variation of speedup, programs correct in both re-scorings3.0%3.2%
Programs whose correctness changed between re-scorings3/967/77

Independent runs vary more than re-scored programs: 2.48 versus 1.45 points for GPT-6 Astra, and 4.42 versus 1.57 for Hy4-Preview. By point estimate, re-scoring accounts for about one third of GPT-6 Astra's run-to-run variance and the agent's search for the rest; with three runs, this split is imprecise (95% interval of the ratio 0.07–3.30).

The re-scoring figure has two parts. Timing is stable for most programs: the median program's speedup varies by about 3% between scorings. In addition, 10 of the 173 programs changed correctness: 7 had outputs within the numerical tolerance in one scoring and outside it in the other, and 3 failed to run once. The verifier's checks and tolerances were the same in both scorings; on many tasks it also draws fresh private inputs each time (Section 2.3). The 1.45- and 1.57-point figures include these changes, and the revision reports them separately. For the final version, we will examine each of the 10 programs and report the cause.

A single run's Score@1 therefore carries about 1.5 points of measurement uncertainty, and independent runs vary more (standard deviation 2.5–4.4 points), so differences of a few points between single runs do not order agents. The final version reports three-run means and intervals for every agent.

Unmodified programs time close to 1x. In the configuration-baseline experiment (Q2), each task's unmodified program was scored once by the v1.1 private verifier. The median speedup is 0.999x; 27 of 32 tasks fall within 0.98x–1.02x, and all within 0.923x–1.017x, the largest deviations being ABySS (0.923x), QuEST (0.940x) and DataLife (0.943x). Together they give a Score@1 of 0.40%. Nine earlier replays of the unmodified HPCG-derived program have a median of 0.9997x. Optimized, short-running programs can vary more: one HPCG-derived final submission has a 14.54% coefficient of variation across rounds (Appendix G.1). The appendix therefore calibrates the timing of the baseline and of optimized programs separately.

Q2: How much does fixed configuration-only tuning achieve, and how much do agents add?

The configuration-only baseline exceeds 1.25x on only 1 of 32 tasks, GROMACS. Measured against it, GPT-6 Astra's Score@1 falls from 50.59% to 48.47%, mostly because of GROMACS. The baseline shows what standard configuration achieves without source changes; it is not an upper bound on configuration tuning.

The baseline was fixed before it was run. For each task it changes build and runtime settings only, never source code, tolerances, solvers, block sizes or other numerical or algorithmic parameters. It tries three cumulative settings: (1) the highest optimization level with native instructions, such as -O3 -march=native, without options such as -ffast-math that change floating-point semantics; (2) the application's own SIMD/vectorization option at the highest level the evaluation CPU supports and the task permits; (3) thread or process counts equal to the allocated cores, with thread pinning, where the task permits and the submitted files control the timed run. The public tests select the fastest setting that passes, preferring a simpler one within 2% of the fastest. The private verifier scores the selected setting once, exactly as it scores an agent's final submission, with no model calls.

Graph sparsification (FeGRASS), Paged LM generation and QiboTN are pure Python programs whose verifiers do not compile submitted code and fix the thread count, so there is nothing to configure; their baseline is the unmodified program. Each task's unmodified program is also scored once as a control.

AuthorsOpenReview ↗

Response to Reviewer qq5K (3/3)

v1.1, 32 tasksUnmodifiedConfiguration baselineGPT-6 AstraGPT-6 Astra against the baseline
Score@10.40%3.85%50.59 ± 2.4848.47 ± 2.41
A1.250%3.1% (1 task)84.4%81.2%
A20%3.1% (1 task)51.0%47.9%

The unmodified and baseline columns are single private scorings per task; GPT-6 Astra shows the three-run mean ± standard deviation.

Baseline results. All 32 configured programs are correct, but only GROMACS exceeds 1.25x: changing its SIMD option from the default None to the highest permitted level, AVX2_256, and adding -march=native gives 2.23x. The next highest are ThunderSVM at 1.249x and Palabos at 1.16x.

On 23 tasks the baseline is the unmodified program: the 3 pure Python tasks; SU2, whose verifier build already uses -O3 and native instructions; wtdbg2, whose public tests report no runtime, so no setting can be selected as faster; and 18 tasks where no setting both passed the public tests and ran more than 2% faster than the unmodified program. On 8 of these 18, setting (1), -O3 -march=native, changed the numerical output and failed the full output comparison.

Agent contribution. On the 9 tasks where a setting was selected, we re-score agents against the baseline's private speedup (counting speedups below 1x as 1x); the other 23 tasks are unchanged. GPT-6 Astra's Score@1 falls from 50.59% to 48.47% (50.87%, 48.49% and 46.04% by run), and A1.25 and A2 each lose one task in every run. Of the 2.12-point drop, 1.70 points come from GROMACS, where GPT-6 Astra reaches 2.18x–2.21x, about the baseline's 2.23x, and adds nothing beyond it. The reviewer is right about GROMACS: its score reflects configuration gains. This baseline also qualifies our observation that agents rarely tune the build: standard build and runtime settings exceed 1.25x only on GROMACS, where the gain from the SIMD build option depends on the thread count (Section 5.3). The final version will revise the introduction accordingly. Because the baseline is computed once per task, the revised Table 1 reports Score@1 against it for every listed agent, and Appendix A lists the selected setting and private result for each task.

The largest speedups do not come from configuration. On HPCG-derived, QiboTN and Paged LM generation, the baseline gives 0.995x, 1.009x and 0.990x (the last two are unmodified programs), while GPT-6 Astra's three runs reach 28.9x–36.7x, 7.9x–11.5x and 11.1x–11.9x. On HPCG-derived it replaces the iterative solver with a sine-transform direct solver; on QiboTN it switches to a faster exact-simulation path and defers library loading; on Paged LM generation it rewrites the inference path.

Finding a wrong thread setting, enabling suitable vectorization and understanding how build and runtime settings interact are themselves part of application performance engineering.

Q3: Were main verifier runs given dedicated cores, was background load recorded, and how was execution order handled?

Neither v1.0 nor v1.1 ran private tests on dedicated physical cores, and neither recorded host load; Q1e measures the resulting measurement error. Baseline and candidate are timed alternately within each evaluation:

  • Execution order. Each private evaluation re-runs the unmodified and candidate programs and times them as a pair; the reward comes from their runtime ratio within that evaluation. On 30 tasks, the program that runs first alternates by round, for example the baseline first in round one and the candidate first in round two. SU2 and OpenFOAM run candidate–baseline–baseline–candidate within each case and reverse the order for the next case. Each side runs twice on 26 tasks and three times on 6; times are combined by median or mean per case and summed over cases, except that QuEST and ThunderSVM sum all timed runs. In v1.1, HPCG-derived runs one warm-up and then seven timed runs per side.
  • v1.0. Runs used Docker containers with task-specific quotas of 4 or 8 CPUs and memory limits. When each of the 384 runs in the submitted Table 1 finished, its host was running a median of 3 other evaluations of this benchmark (maximum 24). The controlled experiments in Appendices E and L instead used physical cores on one NUMA node and recorded load.
  • v1.1. Private evaluation runs in a sandbox separate from the agent and the public tests; baseline and candidate share it and alternate as above. Each sandbox requests 4 or 8 CPUs, depending on the task.
Reviewer_pZJZOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a new benchmark suite inspired by HPC workloads to test CPU performance optimization tasks across codebases. Correctness constraints are provided but the benchmark is set up to enable higher-level codesign optimizations across not just algorithm but also source code, build/flags, cross-code optimizations, etc. Paper presents results across multiple frontier coding agents showing the value from this benchmark.

Critical strengths

the paper addresses an important problem. it both creates a strong benchmark suite which when optimized advances the state of the art for an important class of workloads (HPC) but given the current scores, it also serves as a training suite to help improve complex reasoning capabilities of future models.

the paper presents several interesting insights on how agents perform optimization and how they should perform optimization. the insight about source/build parameters providing higher benefits and the sequencing and combination of optimizations having non-intuitive tradeoffs are both quite interesting.

Critical weaknesses

I was surprised by the relatively small speedups. I would have preferred maybe even a synthetic task where the authors knew about a significant improvement to test whether the system can figure out larger speedups. Right now, I am left wondering whether the results represent the "limits" of speedup for these tasks or limitations of the models.

I would have liked a bit more discussion on what the suite tries to target as its primary use case: help improve future HPC workloads or help improve future models. If the latter then I would have liked a bit more systematic discussion on how the tasks were chosen and some of the key reasoning tasks involved in how these workloads were identified as top candidates.

Questions for authors

See comments above

4 comments
AuthorsOpenReview ↗

Response to Reviewer pZJZ (1/2)

We deliver strong thanks the reviewer for the positive assessment.

W1: Do the speedups reflect the limits of these tasks or the limitations of the models?

Speedups are not uniformly small; they vary sharply by task. The average looks moderate partly because Score@1 is the fraction of runtime saved, r = 1 - 1/s: 2x earns 50 points, 10x earns 90 and 100x earns 99, so large speedups are compressed. Across GPT-6 Astra's three v1.1 runs:

SpeedupShare of tasks reaching it (three runs)
1.25x90.6% / 84.4% / 78.1%
2x56.2% / 53.1% / 43.8%
5x15.6% / 12.5% / 15.6%
10x9.4% / 6.2% / 6.2%

GPT-6 Astra reaches at least 2x on 14–18 of the 32 tasks in each run (19 in at least one run, 13 in all three) and at least 10x on 2–3.

These results are not the limits of the tasks. On at least 4 of the 32 tasks, another agent's run is at least 1.4x faster than GPT-6 Astra's best of three runs, for example 20.6x versus 11.9x on Paged LM generation (GPT-6.1 Terra) and 8.0x versus 3.3x on Tucker reconstruction (Kimi K3). These are correct programs on the same task version and platform, so even the strongest agent leaves speedups unrealized that another agent has shown to be possible.

We agree that a task with a known large improvement is a useful check. We did not add a synthetic one(really sorry for that); the closest case is HPCG-derived, whose task allows replacement solvers and is not the official HPCG benchmark (Appendix D). Its operator is diagonalized by sine transforms, so a direct solver can replace the iterative one. A direct solver we wrote after seeing earlier trajectories reaches 21.18x–22.22x on dedicated cores, and one agent's v1.0 program is 1.19–1.25x faster than ours in the same setup (Appendix G.4). GPT-6 Astra reaches 28.9x–36.7x in its three v1.1 runs, measured on the v1.1 platform and not directly comparable; the configuration baseline gives 0.995x. On Paged LM generation (11.1x–11.9x) and QiboTN (7.9x–11.5x), GPT-6 Astra also obtains large speedups that configuration does not explain (baseline 0.990x and 1.009x), but we had no reference solution for these tasks in advance.

On HPCG-derived, the large gain follows from a recognizable algorithmic structure; on all three tasks, the configuration baseline accounts for none of the gains, and outside GROMACS its highest value on any task is 1.249x (ThunderSVM). As the reviewer notes, the current scores are also what make the suite useful for improving future models. The revision reports the four thresholds for every listed agent (Appendix A.1); the final version will add per-task speedups of every agent beside the main results.

W2: Is the main goal to improve HPC workloads or models, and how were tasks chosen?

The main goal is to evaluate and diagnose coding agents, and thereby help improve future models and agents. We keep the agents' programs and trajectories to analyze how they optimize. Because the tasks come from real HPC applications, the optimizations agents find may also interest practitioners, but we do not claim that they transfer directly to production scale.

The task selection, now described in Appendix A:

  1. We took applications and optimization themes from six supercomputing competitions, whose organizers chose them as recognized optimization challenges.
  2. We kept tasks that run on one node with 4–8 cores and at most 32 GiB of memory, whose outputs can be checked against scientific tolerances and required workflow steps such as checkpoint recovery, and whose runtime the verifier can measure against the unmodified program.
  3. We covered distinct application domains and bottleneck types. Three candidate tasks were dropped for every agent (so we have more than 32 previously) because they could not be scored: ABINIT lacked an evaluation that runs and times the application, block15 lacked a usable baseline runtime, and SolverChallenge24 had an unresolved timing-resolution problem. Their verifiers gave every program zero reward by construction, so dropping them does not depend on any agent's result.
AuthorsOpenReview ↗

Response to Reviewer pZJZ (2/2)

Table R4: Capabilities the tasks require, with examples

CapabilityExample
Find the dominant stage by measuring419 of 480 runs used timing or profiling commands
Recognize mathematical structure and replace the algorithmHPCG-derived: a sine-transform direct solver replaces the iterative solver
Trace the code the application actually executesHPL: an agent optimized HPL_dgemm.c, but comparing binaries showed that the application calls cblas_dgemm; OpenFOAM: the key kernels sit in a precompiled shared library, so recompiling the solver did not change runtime
Understand how build and runtime settings interactGROMACS: identical source gives 0.289x–2.123x under different build and runtime settings
Keep compiler optimizations within numerical tolerancesSIESTA: building with -march=native made 662 of 868 output values differ from the reference; disabling FMA contraction restored them
Preserve required workflow behaviorPaged generation: a checkpoint must restore in a new process
Reviewer_pZJZOpenReview ↗

Acknowledging author response

Thanks for the additional comments.

Thank you so much for your support and thoughtful engagement throughout the rebuttal!

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response for Reviewers, ACs, SACs and PCs

Dear Reviewers, ACs, SACs and PCs:

We thank the three reviewers. All three rated the paper clear and easy to follow, reported gaining a substantially deeper understanding of the problem, and rated their soundness concerns as minor or fixable. Reviewer MJHa found that the workload-defined tasks, flexible optimization scope and correctness verification provide a meaningful complement to GSO, SWE-Perf and SWE-fficiency, and valued the cross-layer analysis. Reviewer qq5K noted that the application-level checks catch problems that simpler tests miss and that the paper explains mechanisms, not just scores. Reviewer pZJZ called it a strong benchmark suite for an important class of HPC workloads and a training suite for the complex reasoning of future models. The reviewers' questions led to the main additions in this revision: a configuration-only baseline, repeated runs and re-scoring of fixed programs, and an account of how runs were selected.

Benchmark versions. We call the task set of the submission v1.0. After submission, we rewrote all 32 statements in one common format that specifies the permitted edits, the computation that must be preserved, the timed stages, repetitions, reward formula, numerical tolerances and rejection conditions, and fixed several verifier, timing and environment problems. We call the result v1.1. The reward formula and task weights are unchanged. Student supercomputing competition finalists, who reviewed the task designs before submission, reviewed the v1.1 statements again. Per-task changes will be listed in the v1.1 release notes.

Each of the five agents with complete results on both versions scores higher on v1.1. Besides the statements and verifiers, v1.1 runs on a different evaluation platform, runs Grok 4.6 in Claude Code instead of Grok Build, and uses the same high reasoning effort for every agent (Kimi K3 and GLM-5.3 used max in v1.0); see Appendix A. We therefore report the two versions separately and do not rank agents across them. The submission's headline statistic changes accordingly: for the seven agents with complete v1.1 runs, 55 of 209 correct outcomes (26.3%) stay below 1.25x (161 of 325 for the twelve v1.0 agents).

The reviewers' questions center on three issues:

Shared concernRaised byResponse
Ranking stability and timing noiseMJHa, qq5KEight of the twelve agents in the submitted Table 1 had three runs per task; the table reported the median-reward run, a rule the submission omitted. We now report how much each agent varies across its three runs (Table R2). On v1.1, GPT-6 Astra and Hy4-Preview have three independent runs per task (Table R1); the other agents' repeat runs are under way. Re-scoring the same final programs estimates how much of the run-to-run variation is measurement noise.
How much improvement remains, and how much simple configuration explainsqq5K, pZJZA configuration-only baseline, fixed before it was run, exceeds 1.25x only on GROMACS; measured against it, GPT-6 Astra's three-run Score@1 falls by 2.1 points, mostly on GROMACS. The scores are not the limits of the tasks: on at least 4 of the 32 tasks, another agent's run is at least 1.4x faster than GPT-6 Astra's best of three runs.
Scaled-down workloads and the scope of behavioral findingsMJHa, qq5KPrivate inputs are larger than public inputs on 16 tasks. On the three tasks with the largest speedups, we will re-score GPT-6 Astra's unchanged final programs at larger input sizes. Section 5's findings on how agents choose candidate programs are now presented as illustrative observations, each with the evidence it rests on.

Table R1: Three independent v1.1 runs of GPT-6 Astra and Hy4-Preview. Score@1 averages the 32 tasks of one run; we report the mean ± standard deviation over three runs. A1.25 and A2, the shares of tasks with a correct program at least 1.25x or 2x faster, are three-run means.

AgentHarnessRuns per taskScore@1Score@1 by runA1.25A2
GPT-6 AstraCodex350.59 ± 2.4853.06 / 50.62 / 48.1084.451.0
Hy4-PreviewClaude Code339.50 ± 4.4243.24 / 40.62 / 34.6362.537.5

A paired bootstrap over tasks (20,000 resamples of each agent's three-run mean) puts GPT-6 Astra 11.1 points above Hy4-Preview (95% interval 5.2–17.7).

Continued in part 2.

AuthorsOpenReview ↗

General Comment (continued)

Table 1 of the revision reports v1.1: every agent runs once per task, and GPT-6 Astra and Hy4-Preview also show the three-run mean and standard deviation from Table R1. In this table, GPT-6.1 Sol is second, 2.0 points below GPT-6 Astra with one run each; one run does not order them (Section 4). Six agents that have not finished all 32 tasks are listed as in progress, and Gemini-3.8-Flash, whose v1.1 runs are not complete, is not yet listed; we will post their results here as they finish.

The v1.0 results move to Appendix B (the submitted Table 1 becomes Table 5), together with the run-selection rule, the variation across three runs (Table R2, Table 6 of the revision) and the run accounting (Table R3, Table 7). DeepSeek-V4.1-Flash and Hy3 have no v1.1 results yet; we are running Hy3 and a newer release of DeepSeek-V4.1-Flash on v1.1 under the same protocol and will add them here. Second and third runs of the other 11 agents in Table 1 and of Gemini-3.8-Flash are under way or scheduled (768 runs of up to 8 hours each). We will post their results here, and Table 1 of the final version will report three-run means and intervals for every agent. Every run, including failed and replaced ones, will be released with the benchmark.

Table R2: Score@1 of the v1.0 agents with three runs per task (Table 6 of the revision). Round i consists of the i-th run of every task by completion time.

AgentRound 1Round 2Round 3MeanSDSubmitted Table 1 (median-reward run)
Hy4-Preview34.6132.7440.1835.843.8736.46
DeepSeek-V4.1-Flash27.8729.2732.0729.742.1429.94
Grok 4.627.5326.0324.9326.161.3026.24
GLM-5.3-Flash25.4222.2429.0725.583.4127.34
DeepSeek-V4-Flash19.2320.0722.8420.721.8919.08
DeepSeek-V4-Pro19.4719.7821.7020.321.2120.63
GLM-5.219.2919.9214.2217.813.1219.92
Hy311.1612.0512.1711.800.559.76

GPT-6 Astra (40.30), Gemini-3.8-Flash (29.94), GLM-5.3 (28.10) and Kimi K3 (19.01) ran once per task in v1.0 because of cost. Since the leading GPT-6 Astra is among them, we compare agents only on v1.1.

Changes in the revised PDF (marked in blue):

  1. Table 1 reports v1.1: one run per task for every agent, plus three-run means and standard deviations for GPT-6 Astra and Hy4-Preview. The abstract, introduction and results are updated accordingly. The submitted Table 1 moves to Appendix B with the run-selection rule, the three-run variation, the run accounting and the rules for resuming interrupted runs.
  2. Table 1 adds Score@1 measured against the configuration-only baseline. Appendix A adds per-task baseline results, the re-scoring experiment on fixed programs, and the share of tasks reaching 1.25x, 2x, 5x and 10x.
  3. Appendix A summarizes the v1.1 revisions and the task-selection criteria.
  4. Section 5 presents its findings on how agents test, choose and keep candidate programs as illustrative observations, states the evidence each rests on, and gives the criteria used to select the 31 version histories.
  5. The new Appendices A (v1.1) and B (v1.0 results and run selection) come first; the submitted appendices follow in their original order, two letters later (A becomes C, ..., L becomes N).

Before the final version: When the remaining runs finish, Table 1 will report three-run means and intervals for every agent, and Appendix A will add the larger-input results. Appendix B will re-score, with the corrected verifiers, the 65 eligible v1.0 runs that were not among the three used per task (see Q1b in our response to Reviewer qq5K), report scores and rankings using all eligible runs, and compare results with and without resumed runs.

AuthorsOpenReview ↗

November 16 revision with complete v1.1 results for 16 agents

Dear Reviewers, ACs, SACs, and PCs:

We have uploaded a further revision. Changes relative to the submission remain in blue; this note summarizes what changed since our earlier responses.

1. Complete v1.1 results for 16 agents. The six agents listed as in progress have finished their first run on all 32 tasks, and we evaluated three more agents under the same protocol: Hy3, the newer 1017 release of DeepSeek-V4.1-Flash, and Hy4. Gemini-3.8-Flash is still not listed because the model API was unavailable for its last remaining run.

AgentHarnessScore@1 (%)
Claude Sonnet 5.5Claude Code67.4
Claude Opus 5.5(three tasks safe blocked)Claude Code59.4
GPT-6.1 SolCodex49.2
GPT-6 AstraCodex47.2
Hy4-PreviewClaude Code45.2
Hy4Claude Code45.0
Kimi K3Claude Code42.5
DeepSeek-V4.1-Flash-1017Claude Code40.1
GLM-5.3Claude Code39.7
GLM-5.3-FlashClaude Code39.3
DeepSeek-V4-ProClaude Code37.9
GPT-6.1 TerraCodex37.9
DeepSeek-V4-FlashClaude Code34.1
GLM-5.2Claude Code34.0
Grok 4.6Claude Code33.9
Hy3Claude Code25.5

2. Scoring on one dedicated host (Reviewer qq5K, Q3). Final programs are scored once, with no model calls, on a single host with two AMD EPYC 9K84 processors. The verifier container is pinned to the four or eight physical cores of one NUMA node that the task allocates, without their SMT siblings, with the task's memory limit and no network access (Appendix A). First-run scores therefore differ from the cloud scores in our earlier responses; for example, GPT-6 Astra's first run scores 47.2% instead of 53.1%. The repeated-run and re-judging statistics in Appendix A.3 are unchanged.

3. Runs stopped by a provider safety classifier. On Graph sparsification, PRESTO and VPIC, the provider's classifiers flagged the Claude Opus 5.5 session as cybersecurity or biology content and stopped the run. The previous PDF substituted Claude Sonnet 5.5's results on these tasks; these runs now count as failures, so Claude Opus 5.5 scores 59.4%. On its other 29 tasks, it scores 65.6%.

4. What a single run resolves (Reviewers MJHa, W2, and qq5K, Q1). New Appendix A.4 gives paired 95% task-bootstrap intervals for differences between agents. Claude Sonnet 5.5 leads every non-Claude agent by at least 13.4 points at the lower end of its interval, and Claude Opus 5.5 leads GPT-6.1 Sol by 0.1 to 18.2 points. Below the two Claude agents, no two neighbors are separated; Section 4 and Table 1 now state that single runs do not order agents a few points apart. These intervals cover task variation only, and Appendix A.3 reports the additional run-to-run variation.

5. Scope of the trajectory analysis (Reviewer MJHa, W3 and Q3). Section 5 now states that it analyzes annotated v1.0 runs of five agents to characterize how agents optimize, and that it does not explain the v1.1 ranking. We removed the candidate-selection observation from the main text, because single public timings cannot establish which of two candidates is faster; the version-history counts and the HPCG-derived replay remain in Appendices J.5 and L.3.

6. Task composition (Reviewers MJHa, W1, and pZJZ, W2). Section 2.2 now states that 29 tasks build on the competition's application, from full source trees to selected solver subtrees, and that three (Paged LM generation, Tucker reconstruction and Graph sparsification) are workflows we wrote around a competition's optimization theme. Without these three tasks, or also without HPCG-derived, the order of Table 1 barely changes (Kendall's tau-b of 0.95 and 0.97). The limitations state that the tasks measure whether agents find and realize speedups in the specified workloads.

7. Other clarifications. End-to-end runtime refers to the prescribed timed workflow, with build time reported separately (Section 2.1). The introduction states that the 14 verifier controls cover two tasks. In the QiboTN factorial replay, broad dispatch and deferred loading account for the full 9.6x to 9.7x gain, and state reuse adds nothing measurable (Section 5.3). Section 4 states that scores compare model and harness combinations rather than base models.

Thanks for your time on the paper.

The Authors

Dear Reviewers, ACs, SACs, and PCs,

Thank you for the care you put into reviewing Accelerate Bench. We appreciate the Area Chair’s help in keeping the discussion active and the time all three reviewers took to consider our responses. We are grateful that all three now recommend acceptance.

Your questions helped us improve the paper in three ways.

  • How reliable are the rankings? The revision reports first-run results for 16 agents on all 32 tasks in v1.1. We scored their final programs on a dedicated machine using fixed CPU cores. We also repeated agent runs and re-timed the same programs to show how much scores can vary and which differences between agents are large enough to interpret.
  • How much can simple tuning achieve? We added a baseline that changes only build and runtime settings. It reached a 1.25× speedup on just one of the 32 tasks, GROMACS. This gives readers a concrete reference for interpreting the agents’ gains. We also explain how we chose the tasks and what the scaled workloads measure.
  • What can we learn from the agents’ attempts? We clarified which observations come from individual runs and which we tested by replaying specific changes. In QiboTN, for example, delaying library loading made almost no difference on its own, but gave a 6.4× speedup when another code change made those libraries unnecessary.

We hope Accelerate Bench will help researchers study a practical question in both AI and systems research, even the self-evolution in Computer Science. Given a working application, can an agent find where the time goes, choose what to change, and make the program faster while preserving its results? The benchmark measures the speedup, and the programs and trajectories let researchers look closely at how an agent achieved it or where it missed an opportunity.

Thank you again for helping us make the paper clearer and more useful.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.