acceptodds
Under review as a conference paper at ICLR 2027

HPCAgent-Bench: A Framework for Sound Benchmarking of Code-Optimizing Agents, Harnesses, and Skills

Abstract

Coding agents are increasingly used to optimize code, but existing benchmarks fix one accelerator, programming model or harness, so they Coding agents are increasingly used to optimize code, but existing benchmarks fix one accelerator, programming model or harness, so they cannot isolate what each component contributes or show whether added skills and tools repay their cost. We present HPCAgent-Bench with 689 kernels in three tracks: loop-level optimization challenges, kernels extracted from HPC codes, and machine-learning models. The framework specifies each kernel once in NumPy, derives C and Fortran references, accepts submissions through a common C ABI or Python interface, and runs them on CPUs or GPUs, on one or multiple nodes. An external judge checks correctness on hidden fuzzed inputs and times each submission, closing paths agents use to exploit the testing system. We introduce two metrics for reporting results: intervention efficacy measures an intervention, such as adding a tool, through paired runs on speedup, success rate and cost, and weighted token cost reports input, cached-input and output tokens under explicit weights. Agents reach up to speedup over automatic parallelizers, and their optimizations transfer to another machine: – of answers without device-specific code remain correct, and per-kernel speedups keep a rank correlation of . Across 40 matched setups, interventions significantly change speedup only twice but token cost by up to . Because token weighting decides whether a saving is significant, we state the weights and release every token count as a standard for cost reporting. We provide the first comprehensive, broad and hardened benchmark for the sound evaluation of agentic optimization, with transparent cost metrics.

Primary area:
datasets and benchmarks
Keywords:
benchmark, coding agents, LLM agents, code optimization, high-performance computing, performance measurement, parallel programming, GPU kernels, agent harnesses, agent skills

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.20 over 5 reviews, above 49% of the others.
Reviewer_K5P9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose HPCAgent-Bench, a benchmark of 689 kernels for evaluating code-optimizing agents across CPUs, GPUs, and multiple nodes. The framework checks correctness using inputs from secret seeds, tests whether measured speedups are statistically significant, and flags suspicious implementations. It also provides paired comparisons of interventions like skills, tools, and harnesses, along with explicit token-cost accounting. Experiments with 3 open-weight models look at performance, portability, and intervention effects. The main reported finding is that interventions rarely change speed significantly but can substantially change token cost.

Critical strengths
  1. The grading protocol looks at concrete ways performance measurements can be misleading. Separate seeds for scoring and final grading, changing inputs between timed calls, and device checks are useful safeguards. The case studies show that one cached answer drops from 5309× to 20× on fresh inputs, and two CPU submissions are caught running their code on the GPU.

  2. Comparing setups through individual interventions looks at whether adding a skill, tool, or harness improves results enough to justify its cost. Reporting the token components separately is also valuable. Figure 5 shows that the choice of token weights can substantially change the saving. The framework also covers several languages, devices, and scientific domains, including multi-node execution, and the paper describes a detailed reproducibility package containing observations, submitted programs, and scripts to regenerate the results.

Critical weaknesses
  1. Some headline numbers don't match the figures. The abstract says interventions significantly changed speedup in only 2 of 40 comparisons, but Figures 3-4 show 5 significant changes, 3 of them from removing score feedback (Section 4.4). For tasks posed as git repositories, the text reports speedup ratios of 0.52-0.68, while Figure 4(i) shows almost no change for 2 of the 3 models. These results support the paper’s main conclusion, so they need to be reconciled.

  2. I’m not convinced by the conclusions that some interventions make no difference. Each comparison uses only 10-40 kernels, and the harness study runs each kernel once, so even pretty large effects might not reach significance. The git-repository framing is a good example. Going by the numbers in Section 4.4, it cuts speedup by 32-48%, yet the paper says it changes “nothing measurable”. The harness comparison also isn’t apples-to-apples. The same section says budgets are matched, but in Table 12 two of the harness runs get twice the tokens and time of the Claude Code baseline, so it’s hard to say how much of any difference comes from the harness itself.

  3. It's not clear what counts as correct, and how much of the benchmark was tested. In the Section 4.6 example, some agents first search the window where the test generator always places the match. This passes every test but gives the wrong answer if an earlier element also matches. The validity depends on whether the restriction is part of the task, which the paper doesn't mention. The experiments also cover only selected subsets, around 100 of the 689 kernels, with little explanation of how they were chosen, and loop-level GPU speedups are measured against a CPU baseline, which makes them hard to interpret.

Questions for authors
  1. Could you reconcile the “only twice” claim with the 5 significance markers in Figures 3-4, and the git-framing ratios in Section 4.4 with Figure 4(i)?

  2. How many episodes per kernel and arm does each experiment use, and could an A/A comparison show what effect size the design can detect? Do the harness results hold under matched token and time limits?

  3. Is the planted-window restriction part of ext_break_capture’s declared input contract? If not, have the other generators been checked for similar shortcuts?

  4. How were the evaluation subsets selected, and could you report at least one model on each full track?

2 comments
AuthorsOpenReview ↗

Response to K5P9

Please also see our general response (G1–G3).

Why 2 and 5 significant effects? The paper has 48 comparisons. Of these, 40 compare interventions and 8 compare the benchmark's own submission modes. The abstract's 2 counts speed effects among the 40 interventions. The 5 markers in Figures 3 and 4 add 3 speed effects among the submission modes. The revision will report all 48.

Why do the git ratios differ from Figure 4(i)? In the git experiment, each of the 10 kernels ran three episodes per setup and model. By each kernel's median, git agents reach 0.55 to 0.70 times the speedup of the kernel framing, while Figure 4(i) used each kernel's last run, which hides the slowdown for two models. The revision plots the medians with a paired interval (thank you for catching this), and the paired test finds a significant slowdown for two of the three models, which replaces "nothing measurable".

Is one run per kernel enough? For large effects, yes. To show that a single comparison has no effect, no. Every other experiment runs one episode per kernel and setup. The revision therefore reports non-significant results as inconclusive and claims "no meaningful difference" only where pooled data support it, for example for skill packets (G3).

To measure how much one setup varies from run to run, we ran an A/A comparison with 3 kernels and 20 episodes per kernel and model. Across episodes, the standard deviation ranges from near 0 to 1.7 for log speedup and from 0.41 to 0.71 for log token count. On 40 kernels with one episode each, the smallest difference detectable with 80% power is therefore about 1.3x in speedup for OSS-120B, 2.2x for Qwen3.8-27B, and 1.4x in tokens.

Do results hold under matched limits? Partly. The speed results hold, but two token savings lose significance. The release runs both setups under the same limits. In a first rerun under one limit for every harness (48M tokens), no harness reaches a significantly higher or lower speedup than Claude Code. The token savings of OpenHands for Qwen3.8-27B (2.14x) and mini-SWE-agent for OSS-120B (3.04x) shrink to 1.40x and 1.35x and are no longer significant, while OpenHands now uses 2.4x more tokens than Claude Code for OSS-120B (significant).

Is the window shortcut correct? It passed only because of our inputs. The paper's generator always placed the match between 40% and 70% of the input, a placement the task never requires: an answer must return the reference's result for every valid input. The revised generator adds an earlier match, which held-out inputs place anywhere in the first 40%, so searching the window first now fails. We are rerunning the experiment.

How were the subsets chosen? For each track, we fixed a quota per level and drew that many kernels at random from each level of the paper's corpus: 24 level-1 and 16 level-2 kernels of the 248 loop-level reasoning (LLR) kernels, and 3, 10, and 22 kernels of levels 1, 2, and 3 of the 171 scientific computing (SC) kernels. The SC subset had 35 in the paper, and the new release extends it to 40. Subsets of this size let one budget cover many ablations. The 6,800 episodes test 48 comparisons over three models and three languages, while the full corpus allows fewer than ten setups for the same compute budget. If the compute budget allows, we will run the full SC and LLR tracks with one model for the camera-ready version.

What is the GPU baseline? GPU speedups are measured against the CPU Numba build, and Figure 3(ii) will say so. No automatic GPU optimizer currently exists that covers the subset. Instead, the revision compares the agents' GPU speedups with Triton reference solutions that we built and tuned for the 40 LLR kernels and the release's 40 SC kernels.

AuthorsOpenReview ↗

Follow-up before the discussion closes

Thank you again for your careful reading. Following your questions, the revised PDF reports all 48 comparisons (Table 18), plots the git framing by each kernel's median with a paired interval (Figure 5a), and reports non-significant results as inconclusive. Could you let us know whether the revision resolves your concerns? If anything is still unclear, we would be glad to address it in the final revision.

Reviewer_F2uwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

HPCAgent-Bench provides 689 kernels spanning loop-level reasoning, scientific computing, and machine learning, with shared interfaces for CPU, GPU, and distributed execution. An external judge checks hidden inputs and measures runtime. Selected task sets evaluate agents, harnesses, and skills through paired changes in success, speed, and weighted token cost, alongside cross-machine and multi-node experiments.

Critical strengths
  1. Broad coverage under a common evaluation contract. The shared ABI links loop transformations, scientific kernels, and distributed ML tasks across programming models. Relative to ParEval (HPDC 2024) and KernelBench (ICML 2025), the contribution is the integration of this coverage with an external judge and controlled interventions.

  2. Transparent measurement and cost reporting. Separate hidden seeds, recorded exploitation cases, cross-machine regrading, paired statistics, and released token components make the evaluation more inspectable. Reporting the sensitivity to token weights also exposes how accounting choices change cost conclusions.

Critical weaknesses
  1. The core controlled-comparison claim is not established. Appendix C permits larger-budget reruns, Table 12 grants some harness treatments larger limits, and Appendix G.2 permits in-episode fallback grades and exclusion of unrun tasks. Their frequency and impact are not reported. These choices affect both resource matching and final-test independence, so the intervention results require a strictly matched, fully regraded analysis before they can support the framework’s central claim.

  2. The main intervention conclusions exceed the statistical evidence. Single-episode experiments cannot quantify agent-run variability, and nonsignificance is repeatedly used to suggest little effect. In the ten-task repository comparison, ratios of 0.52–0.68 remain inconclusive rather than showing equivalence. Across-kernel intervals and multiplicity correction do not resolve this issue; repeated runs and equivalence or power analysis are needed to substantiate the claimed lack of benefit.

Questions for authors
  1. How many paired episodes received budget extensions, and are all failed attempts included in token cost? Do the findings persist when both arms have identical hard limits?

  2. How many results use fallback in-episode grades or omit unrun tasks under Appendix G.2? Please recompute the main comparisons on an identical task set with completed hidden-seed final grading.

  3. With repeated agent episodes and a predefined equivalence margin, which claimed null effects are supported as equivalence, rather than remaining inconclusive?

  4. How is the generic reassociation tolerance validated for scientific kernels with cancellation or near-zero outputs? What false-acceptance rate does it yield on deliberately corrupted outputs?

4 comments
AuthorsOpenReview ↗

Response to F2uw

Please also see our general response (G1–G3).

How were limits and costs handled? In the paper's runs, both setups score each kernel on its latest launch with a valid answer and count only that launch's tokens. Adding the tokens of failed launches keeps the direction and significance of the token savings. Scoring each kernel's latest launch, even a failed one, leaves 5 of the paper's 20 significant effects non-significant and reverses none. The release separates agent failures from infrastructure failures. When the agent fails, the kernel is unsolved and every token counts. When the infrastructure fails, the episode resumes with its tokens counted, or restarts fresh with the lost tokens reported separately (configurable). Of the 6,800 episodes, 840 ran in separate rerun jobs: 291 after a limit stop, 238 after the agent ended without an answer, and 133 after an infrastructure failure or a voided submission. For the remaining 178, the earlier runs are no longer archived. Of the 840, 511 ran with larger limits. Our compute budget at the time did not allow repeating the runs under equal limits. In a first rerun under one limit, the speed results hold, while the harness token savings shrink from 2.14x and 3.04x to 1.40x and 1.35x and are no longer significant.

Were fallback grades used, and do task sets match? No fallback grades were used. Every reported result is the final grade, which the judge computes on hidden inputs when the agent submits its answer via /submit, never a grade from the /score tool. The final timing protocol times only fresh draws, never the repository's default seed. Of the paper's 2,206 agent answers, 1,524 were timed under it (1,611 counting its first version), and on these alone, no significant effect changed direction. Answers we reuse are regraded under the final protocol, and lost ones are rerun. The task sets match: both setups of every comparison ran the same kernels, and a kernel that ran and failed counts as unsolved.

Is one run enough to claim no effect? No. With one episode per kernel, 6,800 episodes cover every comparison, but on 40 kernels they reliably detect only speed differences of about 1.3x for OSS-120B and 2.2x for Qwen3.8-27B. A smaller change usually goes unnoticed. We estimated both from repeated runs of one setup against itself (an A/A comparison), with 20 episodes per model on each of 3 kernels. Measured on each comparison's own data, the smallest detectable speed difference ranges from 1.1x to 4.6x across the 40 intervention comparisons (median 1.7x). At the measured spread, detecting a 1.2x difference needs about 100 kernels. The full scientific computing (SC) track in the paper has 171 kernels, on which these values fall to about 1.15x and 1.5x. A single comparison with one episode per kernel therefore cannot show "no effect", and the revision no longer claims it can. Of the 80 speed and cost results of the 40 intervention comparisons, the 69 non-significant ones are now reported as inconclusive, each with the effect it could detect. Under the revised tests, the git framing significantly lowers speedups for two of three models (pooled 0.61x). Following your request, the revision fixes an equivalence margin for every comparison, 1.2x for ratios and 10 points for solve rates. Equivalence holds for pooled skill packets, whose speedup ratio is 1.01x with a 95% CI of 0.94 to 1.08, inside that margin.

Is the tolerance too loose? Not on loop kernels. On scientific kernels, it had room to improve, and the updated release tightens it. As you suggest, we corrupted every kernel's outputs with the usual mutation-testing errors (value perturbation, sign flips, zeroing, off-by-one shifts, constant outputs). The paper's tolerance accepts such an output in at most 2.1% of cases on loop kernels and 7.0% on scientific kernels. A new check on the relative L2 error of the whole output lowers that rate on scientific computing kernels to at most 3.2%, while correct -O3 builds of each kernel reach at most 3.8% of the check's threshold. A sign check on every element is available in an opt-in strict grading mode. It is off by default, because floating-point reassociation can flip the sign of near-zero elements, and the check would reject correct builds. The revision also extends the L2 check to distributed kernels. For every kernel with its own input generator, held-out inputs are now drawn away from zero, so a wrong element cannot hide in rounding noise. Of the 2,786 correct answers whose source we hold, 9 fail the L2 check, all on channel_flow, a fluid-dynamics kernel whose velocity is near zero. Eight fail only through rounding, and the ninth is a real error that the paper's tolerance had accepted.

AuthorsOpenReview ↗

Follow-up before the discussion closes

Thank you again for your detailed questions. We have uploaded a revised PDF (changes in blue) that follows them. Among other changes, it tests every comparison for equivalence within a fixed margin of 1.2× and reports the rest as inconclusive; for example, the Language skill packet leaves speed unchanged, with a pooled ratio of 1.01× (0.94–1.08) (Table 19). Could you let us know whether the revision and our responses resolves your concerns? If anything is still unclear, we would be glad to address it in the final revision.

Thanks to the authors for uploading the revised PDF and for the thorough audit of the logs, the whole-output L2 check on the scientific kernels, and the three-way statistical breakdown in Tables 17-19. These additions address my main technical concerns around numerical tolerances and separating equivalence from low statistical power, so I am raising my Soundness score from 2 to 3.

At the same time, checking the revised manuscript confirms why the empirical takeaways remain limited: in Table 18, none of the 48 individual intervention comparisons reaches equivalence within the 1.2x margin (equivalence only appears after pooling across setups in Table 19), enforcing a single limit on Harness20 removes the earlier harness token-saving claims, and Appendix C.1 notes that 435 episodes outside Harness20 and Blind vs. single still rely on 2-4x budget reruns. I will keep my overall rating at 2 for now and discuss the paper's overall contribution further with the AC and other reviewers.

AuthorsOpenReview ↗

Reply to F2uw

Thank you for checking the revision carefully and for raising the soundness score. Your questions made the protocol stronger. In the final PDF, every comparison except CPF runs under one limit, and no episode is relaunched with a larger budget; Appendix C.1 now states this. Five single comparisons now reach equivalence within 1.2x. If any further analysis would strengthen the contribution in your view, we would be glad to add it to the camera-ready version.

Reviewer_dHErOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Existing benchmarks for code-optimizing agents are often limited in workload coverage, evaluation reliability, and systematic comparisons. This paper introduces HPCAgent-Bench to provide a unified framework for evaluating code-optimizing agents and their components. It includes 689 kernels across Loop-Level Reasoning, Scientific Computing, and Machine Learning, with unified interfaces, hidden fuzzed inputs, and statistical performance validation. The authors also propose Intervention Efficacy and Weighted Token Cost metrics to assess optimization effectiveness and resource consumption. Experiments involving multiple models, hardware platforms, and interventions evaluate optimization performance, portability, and the impact of different agent configurations.

Critical strengths

S1. Broad Coverage of Heterogeneous Optimization Tasks

The framework supports three substantially different task categories: Loop-Level Reasoning, Scientific Computing, and Machine Learning. This broad coverage extends the applicability of the benchmark beyond a single optimization domain.

S2. Evaluation Across Multiple Settings

The paper evaluates selected subsets of the benchmark across multiple models, execution environments, and intervention settings. The experiments report optimization performance and token costs, along with analyses of cross-platform portability and distributed execution.

Critical weaknesses

W1. Insufficient Justification and Validation of the Unified Framework

Although the paper aims to address the limited scope of existing benchmarks, the advantages of integrating three substantially different optimization task categories into a unified benchmark are not sufficiently justified. These tasks involve different optimization objectives, workflows, and specialized tools. While a shared evaluation infrastructure may improve consistency, the experiments do not clearly demonstrate what additional insights or benefits arise from combining these tasks, compared with evaluating them separately using domain-specific benchmarks.

Furthermore, the framework relies on a unified NumPy-based task specification and a relatively fixed optimization and evaluation workflow (Section 2, Figure 1). Although this design facilitates standardized evaluation, the paper does not sufficiently discuss its applicability and limitations across different optimization approaches. For example, it remains unclear whether optimization tools requiring specialized profiling or iterative tuning workflows can be integrated without substantial modifications to their original processes.

Moreover, if the primary goal is to provide a general framework for heterogeneous code-optimization tasks, its extensibility should be further evaluated, particularly the effort and modifications required to integrate new tasks, tools, or execution environments.

Lastly, the paper emphasizes different aspects of its contributions, including heterogeneous task coverage, multi-language and multi-architecture support, and controlled evaluation of agent configurations. While these aspects are related, their relationship to the central research motivation could be more clearly articulated. The paper would benefit from clarifying its primary objective and organizing the specific contributions around it.

W2. Limited Methodological Novelty of the Benchmarking Framework

Existing benchmarks have already established a relatively mature workflow for code-optimizing agents, including separating the optimization process from independent validation, using hidden or fuzzed inputs to prevent overfitting to evaluation data, and performing repeated measurements to assess performance improvements. Similar mechanisms have been explored in prior work, such as KernelBench, robust-kbench, and KernelGenBench.

Building upon these established practices, the main additional contributions appear to be a unified task specification and interface for heterogeneous workloads, along with evaluation metrics such as Intervention Efficacy and Weighted Token Cost. However, the paper does not clearly distinguish the methodological novelty of these contributions or provide sufficient evidence of their advantages over existing approaches.

Questions for authors

Q1. Generality and Extensibility of the Unified Framework

What are the key advantages of unifying these three heterogeneous task categories, and how easily can new optimization tasks with different workflows be integrated into the framework?

Q2. Methodological Novelty and Improvements

What are the key methodological differences between HPCAgent-Bench and existing benchmarks, and what evidence demonstrates the advantages of the proposed framework and evaluation metrics?

3 comments
AuthorsOpenReview ↗

Response to dHEr

Please also see our general response (G1–G3).

What does unifying the three tracks add? A shared corpus and protocol make findings comparable, so a reader sees which findings generalize (G1).

Is the NumPy specification limiting? Only in scope. NumPy defines only what a kernel computes, and an answer may be written in any language or be a prebuilt shared library, as long as it follows the kernel's C ABI or Python interface. For optimizers that start from native code, the benchmark provides tooling that translates each NumPy specification into C, C++, Fortran, CUDA/HIP, and Numba, and the release ships Triton versions of the experiment kernels as GPU references. A whole application, such as ICON restructured across its modules, would cost node-hours instead of seconds to grade per answer, so we extract its hotspots as kernels instead.

How costly is integration? Exposing an optimization framework's code transformations to the agent required one plugin of about 1,100 lines and about 8 person-hours. Half of the lines are code that overrides the /score and /submit tools, and half are skills and prompt text. A plugin declares what it overrides, such as tools or prompt blocks. Everything else, including the judge, the limits, and the cost accounting, stays unchanged. In this plugin, /submit accepts only the framework's intermediate representation, so the agent must optimize through the framework's transformations. /submit compiles that representation into a shared library for the judge to grade, and a skill and a prompt block describe the transformations. For profiling, the container holds perf, PAPI, likwid, ncu, and rocprof, and a profile tool returns a candidate's call graph or counters. For tuning, /score can also return the build commands, the compiler output, and the timing spread.

How do other optimizers connect? Through a plugin. A setup combines a model, a harness (e.g., Claude Code, OpenHands), and plugins, which add tools, prompts, or an optimizer. An optimizer that is not an agent (e.g., an autotuner or evolutionary search) is a function that a plugin registers. It runs one episode against the same judge, with the same limits and the same single submission as an agent. It calls /score directly or through a wrapper. One wrapper is a Gymnasium environment for reinforcement learning (under 200 lines, about 4 person-hours), whose actions are sources with build flags and whose reward is the improvement in speedup of a correct /score call. An optimizer without a model spends no tokens, so it is compared on solve rate and speed alone, within the same wall-time limit.

What is the objective, and how does it differ? The objective is to compare agent setups soundly and to tell which results hold across tracks, languages, devices, and models. The revision states this in the introduction and presents each contribution as serving it. None of KernelBench, robust-kbench, and KernelGenBench has a held-out final grade or a tested comparison of methods (G3). As evidence, when re-analyzed with the protocol, our own paper's "up to 3.0x" token saving no longer holds (G3), and the audits found KernelBench tasks that accept an answer ignoring its input (G2). Our cost metric differs as well. The price assigned to cached-input tokens shifts cost ratios by up to a factor of 2.9, so we report input, cached-input, and output tokens separately and let a reader apply any price.

How are new tasks and machines added? A new task needs a NumPy reference and a manifest (§2). A new machine needs one entry in a small YAML system description: its Slurm partition, cores and GPUs per node, and time limit. A new architecture may also need a container with its toolchain and a skill that describes the hardware to the agent. On a GH200 system, 583 of 592 CPU and 493 of 512 GPU answers from the MI300A system without device-specific code stay correct, and their speedups largely keep their order (rank correlation 0.92 and 0.89).

Thank you for the clarifications. The response addresses my concerns to some extent. However, regarding the necessity of a unified benchmark, I remain cautious, as I prefer an expert agent dedicated to optimizing a specific domain. Since I was already leaning toward acceptance, I will maintain my current score.

AuthorsOpenReview ↗

Reply to dHEr: Domain-expert agents on a shared benchmark

Thank you for the follow-up and for your support. We agree that an expert agent may be the best optimizer for its domain. HPCAgent-Bench supports such specialization by design: a setup can add domain skill packets, tools, or its own harness, and the benchmark compares the specialized agent with a general one on the same kernels, under the same judge and protocol. Because one judge and one protocol cover all three tracks, an agent tuned for one domain can run unchanged on the other two, which tests whether its domain-specific optimizations transfer.

The same mechanism already shows, within one track, which effects of these additions hold across languages, devices, and models and which depend on them, for example:

  • Holds across settings: on the 40 loop-level reasoning (LLR) kernels, the Language skill packet, expert advice for each programming language, leaves speed unchanged across five languages, both devices, and three models (pooled 1.01x over 20 comparisons, equivalent within 1.2x).
  • Model-dependent: on the same kernels in C, starting from the canonical parallel form (CPF), which already exposes the parallelism, helps only the smaller models. Qwen3.8-27B becomes 2.05x cheaper and OSS-120B 1.23x faster, while for Kimi-K2.7 the effect is inconclusive.
Reviewer_D2KYOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces HPCAgent-Bench, a benchmark of 689 code tasks for evaluating AI agents that make programs faster across CPUs, GPUs, and multiple machines. It combines hidden correctness tests, reliable timing, and controlled comparisons to measure whether added tools or skills improve speed and reduce cost.

Overall, I find this to be a very interesting / valuable papers. I am happy to vote to accept as long as the authors can clearly articulate and defend their contributions (see weaknesses).

Critical strengths

This paper presents a substantial, unusually broad consolidated corpus, and an open-sourced Github codebase to facilitate others from running on it. A lot of the values is the set up -- instead of needing to onboard benchmarks separately, they can be unified under the same setup.

Critical weaknesses

Very unfortunately, the paper's novel aspect only comes from unifying a wide set of corpuses. I was hoping that the design of the dataset would offer some novelty (such as multi-node ML evaluation, multi-architecture evaluation, special design of the judge/scorer, etc.). However, it looks like prior works have covered a lot of these already.

I think unifying datasets and presenting a good framework has academic value and is worthy of publication. I am happy to vote for accept for this paper if the authors can provide enough signals that what they did is truly non-trivial -- maybe they describe what challenges they have faced when bringing these datasets together, or point out flaws that other benchmarks had but they fixed (even if the tasks are still the same).

(The following table is generated by an LLM)

  • Multi-node ML: CommBench evaluates inter-node GPU communication and compute–communication fusion on NVIDIA/AMD; CUCo demonstrates agentic optimization of inter-node MoE computation.
  • Multi-GPU ML: ParallelKernelBench provides 87 problems, scoped to intra-node NVLink.
  • Portability: KernelBench reruns the same generated kernels across six GPUs.
  • Validity: robust-kbench targets evaluation loopholes; KernelBench-Verified uses hidden multi-distribution tests; KernelGenBench adds static checks, no-op replay, and profiling.
  • Control/transparency: D2K-Bench uses matched guidance ablations; PerfCodeBench describes releasing generated-code logs.
Questions for authors

The score tool returns only a speedup, never the program’s output.

In some cases, such as hint-conditioned on-policy distillation, or RL from language feedback, having compilers or profilers return messages that describe the issues can help as well. Would you guys consider adding a side "info" into this benchmark that contain these information beyond a score? If you look at Atari gym env, it has: (next_state, reward, done, info) -- ideally info should contain those.

2 comments
AuthorsOpenReview ↗

Response to D2KY

Please also see our general response (G1–G3).

Is building the corpus non-trivial? Yes. It required solving three problems that no adopted suite had solved. (1) Adopted suites are not agent tasks, so we gave every adopted kernel a reference, an input generator, and an interface (G1, G2). (2) Fresh inputs must prevent caching, and the tolerance must cover numerically sound floating-point reorderings. On some of the original inputs, cancellation amplified harmless rounding past the tolerance, so we redesigned them: correct reordered builds now pass, while wrong variants still fail. (3) A submission must be able to choose each array's precision and data layout, on one node and across ranks. The judge converts the data outside the timed region, grades at the task's own precision, and grades distributed outputs from per-rank sums. Our audits also found tasks on which an answer that ignores its input passes, in KernelBench, TSVC-2, NPBench, and a proxy application. The new release of the benchmark fixes them, and continuous integration checks every new kernel for this defect.

What is novel? The measurement protocol (G3) and one corpus that brings loop-level reasoning, scientific computing, and ML together (G1, G2). Together, they close three gaps in existing benchmarks. (1) Inputs: of 20 related benchmark papers, 17 draw inputs from a fixed seed or fixed constants. On 14 KernelBench problems, the output changes too little with the input for KernelBench-Verified's tolerance to detect, so an answer that ignores its input passes that benchmark's own correctness check. KernelBench-Verified excludes only 3 of these problems from its metrics. Our judge grades every answer on hidden inputs drawn at several sizes. (2) Statistics: of the same 20, 15 report no interval, 9 report no cost, and none test a comparison with multiplicity correction. (3) Distributed data: of the seven papers that use more than one rank or GPU, four keep the data distribution fixed, two let the candidate program create its own inputs (CommBench, CUCo), and one publishes no harness (PCEBench). Only our judge generates the data and lets the submission choose its layout.

The nine works you list add inter-node communication (CommBench, CUCo), multi-GPU kernels (ParallelKernelBench), portability (KernelBench), validity checks (robust-kbench, KernelBench-Verified, KernelGenBench), expert-design guidance (D2K-Bench), and performance tasks in six languages (PerfCodeBench), yet none has all four parts of our protocol. (a) Inputs hidden in advance. Only D2K-Bench (a fresh seed per trial in its released code) and KernelBench-Verified (correctness inputs only) hide them. (b) A held-out final grade. Only KernelBench-Verified has one, for correctness only, and it, CUCo, and D2K-Bench report their fastest candidate with the timing that selected it, so their final score is not held out. (c) Intervals. Only KernelGenBench reports one, for a single comparison of two agents, and it computes each agent's interval on a different set of kernels, the ones that agent solved. (d) A tested comparison of methods. None of the nine has one. Appendix B lists the evidence for each work.

Can /score return more information? Yes. /score now optionally returns feedback beyond the speedup, in an optional info field with the build commands, the compiler output, and the timing spread. The field is off by default, costs no extra kernel run, and reveals nothing about hidden inputs. Plugins can extend what /score returns, so an approach that needs its own feedback doesn’t require changes to the benchmark or judge. For example, our Gymnasium environment took under 200 lines and about 4 person-hours to write and returns this feedback in the form you describe, with the info field in each observation (thank you for this suggestion). This feedback suits uses such as reinforcement learning from compiler feedback or hint-conditioned distillation.

AuthorsOpenReview ↗

Follow-up before the discussion closes

Thank you again for your encouraging review. The revised PDF (changes in blue) now includes the audit findings from our reply and the info field you suggested. Could you let us know whether the revision resolves your concerns? If anything is still unclear, we would be glad to address it in the final revision.

Reviewer_uGbpOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents HPCAgent-Bench, a framework for evaluating code-optimizing agents across loop-level optimization, scientific computing, and machine-learning workloads. It contains 689 kernels and supports CPU, GPU, and multi-node execution. The framework includes an external judge, hidden fuzzed inputs, repeated timing, and statistical tests. It also proposes an intervention-efficacy measure for comparing agent configurations and a weighted-token-cost scheme for reporting token usage.

Critical strengths

The paper addresses an important problem: reliably evaluating whether coding agents can optimize performance-critical code. Its focus on sound performance measurement and practical evaluation is valuable.

Critical weaknesses
  1. Unclear Scientific Value of Integration The core argument for a unified multi-domain benchmark is weak. The authors argue that existing benchmarks are fragmented, but simply lumping low-level loop kernels, HPC proxy apps, and ML models into one framework only proves that the infrastructure can support them. It does not prove why they need to be evaluated together. Without demonstrating concrete cross-domain benefits—such as predicting transferability or revealing generalizable agent behaviors across tracks—the integration feels like an engineering aggregation rather than a scientific necessity.
  2. Clarification on the "Intervention Efficacy" Metrics The proposed "intervention efficacy" metrics provide a structured way to quantify changes through paired runs. Since the core procedure closely mirrors a controlled paired ablation, it would be helpful if the authors could better highlight the methodological advancement over standard paired ablations. Additionally, it would strengthen the paper to discuss how these ratios translate into practical decision-making under real-world trade-offs (such as balancing speedups against token costs), thereby validating their utility beyond standard reporting conventions.
Questions for authors

1.What is the scientific advantage of a unified benchmark over separate domain-specific evaluations?

2.In what ways does "intervention efficacy" offer a methodological advance beyond standard paired ablation results (reporting success, speed, and cost separately)? Is it intended as a new metric or a standardized reporting convention?

5 comments
AuthorsOpenReview ↗

Response to uGbp

Please also see our general response (G1–G3).

Why one unified benchmark? Results often do not transfer from one track or device to the next, and only one corpus under one protocol can show which findings transfer and which do not (G1). For example, Qwen3.8-27B's lead over OSS-120B is 2.45x larger in HIP on the GPU than in C. In contrast, withdrawing score feedback lowered the speedup agents reached in all 8 setups we tested (G1). For the camera-ready version, we plan to run this comparison on 40 ML kernels, to see whether the finding holds on a second track.

New metric or reporting convention? You are right that efficacy is a reporting protocol, and the revision presents it this way throughout. Beyond a standard paired ablation, the protocol grades both setups on held-out inputs (always under equal limits in the new release of the benchmark), adjusts its p-values for the number of comparisons, and separates "equivalent" from "inconclusive" (G3). Interval reporting of this kind changed published deep RL comparisons (Agarwal et al., 2021, arXiv:2108.13264) and is urged for LLM evaluations (Miller, 2024, arXiv:2411.00640), yet among 20 related benchmark papers, only 5 report an interval and none tests a comparison with multiplicity correction (Appendix B).

How do the ratios guide decisions? Solve rate, speed, and cost each get their own verdict, and a reader decides from all three. If an intervention is equivalent in solve rate and speed but significantly cheaper, it can be adopted. If it improves one ratio and worsens another, it is a trade-off, and the two ratios show its size. For example, pooled over its five comparisons, the Terse prompt is equivalent in speed and 1.22x cheaper, but its solve rate drops by 8.3 points (Appendix D). Because input, cached-input, and output tokens are reported separately, a reader can recompute each cost ratio at their own provider's prices.

What changes for soundness? We re-analyzed all of the paper's data under the revised protocol, and none of its significant speed or cost results reverses. Every non-significant result that is not shown equivalent is now reported as inconclusive, with the smallest effect it could detect, and the paper no longer claims "no effect" without an equivalence test. Of the 6,800 episodes, 511 were reruns with larger limits (Appendix C). We are repeating them under equal limits. In a first rerun of the harness study, the speed results hold, while the two harness token savings are no longer significant.

Thank you for the detailed response. My remaining concern is as follows:

Representativeness. Could the authors clarify the selection criteria for the 8 model×language×device setups since three models × three languages × two devices would suggest more than 8 combinations? What dimensions of the agent/device/task space do they cover, and what dimensions are omitted? Could the authors at least discuss the coverage and limits of these choices, and whether any available ablations suggest that the reported patterns are not driven by a single setup?

Design of the planned 40-ML-kernel validation. Since the camera-ready validation is currently a plan, could the authors specify in the response what hypothesis it will test, what outcome would count as support for the unified-benchmark claim, and what outcome would count against it? This would help assess whether the current evidence is a proof of concept or a representative sample.

AuthorsOpenReview ↗

Reply to uGbp: Coverage of the 8 setups and design of the ML validation

Thank you for the follow-up. Because in these setups the device follows from the language (C and Fortran on the CPU, HIP on the GPU), the three models and three languages give 9 combinations, of which we ran 8.

1. Representativeness

Selection criteria. We chose the setups to get the most comparisons out of a fixed compute budget. One 40-kernel setup takes about 10 (OSS-120B), 40 (Qwen3.8-27B), or 110 (Kimi-K2.7) node-hours, and we left out the combinations we expected to add the least information. With four MI300A GPUs per node, 110 node-hours correspond to 440 GPU-hours. Kimi-K2.7 in HIP was left out due to the compute budget. We are adding it to the camera-ready runs, in blind and open-submission modes, which will complete the grid of 9. Besides C, Fortran, and HIP, the benchmark accepts C++, CUDA, and Triton source, OpenMP offloading from C, C++, or Fortran, and shared libraries behind the kernel's C ABI and Python interface (Appendix A). C++ was left out because we expected its answers to differ little from C. Triton and OpenMP offloading were left out because, in our other runs, HIP answers were at least as fast as Triton answers (1.00 to 1.24x) and 1.24 to 1.61x faster than OpenMP offloading answers (Section 4.5, Appendix C.3).

Coverage. The 8 setups cover three open-weight models, one harness (Claude Code), the CPU and the GPU, and 40 loop-level reasoning (LLR) kernels. A related comparison, blind against single-submission runs, also runs on 40 scientific computing (SC) kernels in C (below). The ML track, other harnesses, and proprietary models are not covered yet. Section 4.5 states this coverage and its limits.

Does one setup or one kernel decide the result? In a leave-one-out analysis, the pooled speed ratio of blind runs (no score tool, one submission) against open-submission runs (unlimited scores and submissions), 0.77x, stays between 0.75x and 0.80x whichever setup or kernel we drop, and every model and every language points the same way:

Speed ratio, blind against open submission
By modelKimi-K2.7 0.71x, OSS-120B 0.73x, Qwen3.8-27B 0.88x
By languageC 0.75x, Fortran 0.78x, HIP 0.80x

On the 40 scientific computing (SC) kernels under equal limits, blind runs against single-submission runs (score tool, one submission) give 0.83x (95% CI 0.71 to 0.95), significant after correction, and every model points the same way (OSS-120B 0.88x, Qwen3.8-27B 0.84x, Kimi-K2.7 0.77x). Without Kimi-K2.7, the pooled ratio is 0.86x, the same direction but inconclusive (Table 21 and its footnote).

2. Design of the 40-kernel ML validation

The validation runs 40 single-node ML kernels, drawn by the same quota per level and run under the same tests as LLR and SC. With the ML track, the two comparisons it repeats cover all three tracks instead of two. Its results will be in the camera-ready version.

Hypothesis 1: kernel difficulty is not a general property. Models agree on which LLR kernels are hard (rank correlation 0.83 to 0.89 in C and Fortran) but not on which SC kernels are (0.22 to 0.51). The ML runs test whether this agreement changes again on the third track.

Hypothesis 2: on ML kernels, blind runs are slower than single-submission runs, which differ only in the score tool, as they are on LLR and SC under equal limits (Table 21).

  • Support for the claim. The claim is that a shared protocol is needed to tell which findings transfer between tracks. The existing data already support it through the difference between LLR and SC above. The ML runs strengthen this evidence with a third track.
  • Against the claim. ML verdicts that match those of LLR or of SC for every repeated comparison. The ML track would then behave like an existing track and add no new information.
  • What can fail. The ML runs also test whether the score-feedback finding is general, and the result can go either way. A significant slowdown on ML would show that the finding transfers. Equivalence within 1.2x would show that it is specific to LLR and SC, and we will report it as such. Either outcome can be compared with LLR and SC only because all three tracks share the judge, the input generation, and the statistical tests.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response

We thank the reviewers for their constructive reviews. We are glad they find the focus on sound measurement valuable (uGbp), the corpus substantial (D2KY), the three task categories broad (dHEr), the cost reporting transparent (F2uw), and the grading safeguards useful (K5P9). Following feedback, we re-analyzed the paper's data under a stricter protocol (G3). Every effect the paper reported as significant has kept its trend in both speed and token costs. Only two findings change in significance: the git slowdown becomes significant for two models, and the harness token savings lose significance under equal limits (G3). We are rerunning the experiments where a change is possible with the new protocol under one limit for every setup. We will upload a first revised PDF by the end of November 16 (AoE) and the final revised PDF of this discussion, with the reruns completed, by the end of November 18 (AoE). Further experiments, and any that reviewers suggest, will be in the camera-ready version. All changes and updated data are in blue.

G1. General agents need a general benchmark. (uGbp, dHEr, D2KY). Coding agents are used to optimize any code, so a benchmark has to match this generality. Existing work falls short in three ways: (1) of 20 related benchmark papers, 12 test only GPU or ML kernels and none covers all three tracks, (2) benchmark suites for loop-level reasoning and scientific computing, such as TSVC-2 and PolyBench, run on fixed arrays and are not suitable as agent tasks, and (3) each benchmark hardcodes its own judge, input design, and statistical protocol, including cost reporting (Appendix B). We converted these suites into agent tasks and added 47 kernels extracted from production codes and proxy applications (G2). Ours is the only benchmark (Table 1) with HPC proxy applications on both CPU and GPU. All three tracks share one judge, one input design, and one statistical protocol, so a comparison can be repeated on any track and device. Repeating comparisons this way shows that some findings generalize and some don't. For example, withdrawing score feedback lowers the speedup agents reach in all 8 tested combinations of three models and three languages, on the CPU and the GPU (pooled 0.77x). In contrast, models agree on which loop-level reasoning kernels are hard (rank correlation 0.83 to 0.89) but not on which scientific computing kernels are (0.22 to 0.51). A single-track benchmark would report its own difficulty ranking as a fact about agents.

G2. No adopted suite could be used as it was, and a third of the corpus is new. (uGbp, D2KY). Every adopted kernel received a NumPy reference, a seeded input generator with fuzzed sizes, and a C ABI and a Python interface. These generators replaced the fixed inputs of all 150 TSVC-2 loops and all 32 PolyBench kernels. Audits repaired the inputs of 5 NPBench kernels, 3 proxy-application kernels, and several KernelBench tasks on which an answer that ignores its input passes, a defect KernelBench-Verified and robust-kbench partly catch. Of the 722 kernels in the new release (689 in the paper), 228 are our own: 181 written for the benchmark and 47 extracted from production codes (e.g., ICON, Quantum ESPRESSO) and proxy applications (e.g., CLOUDSC).

G3. Efficacy is a reporting protocol for comparing agent setups, not a new metric. (uGbp, dHEr, K5P9, F2uw). Success, speed, and cost are tested separately with multiplicity correction. Following F2uw and K5P9, we add equivalence tests and detectable effects to the paper's intervals and corrected tests, so each comparison ends with one of three verdicts: (1) different, (2) equivalent (within a default margin of 1.2x for ratios and 10 points for solve rates, configurable), or (3) inconclusive (the data cannot decide). Under the revised tests, 4 of the 40 intervention comparisons show a significant speed effect, and 7 a significant cost effect. The stricter protocol changes three of our paper's conclusions: pooled skill packets leave the speedup unchanged (1.01x, 95% CI 0.94 to 1.08, now a tested equivalence), the git framing does lower speedups (pooled 0.61x, 95% CI 0.42 to 0.88), and under equal limits, the harness token saving of "up to 3.0x" no longer holds.

AuthorsOpenReview ↗

First Revised PDF Uploaded

We have uploaded the revised PDF; every change is in blue. It addresses the reviews as follows:

  • Every comparison has a tested verdict (F2uw, K5P9, uGbp). Each result comes with its confidence interval and reads as different, equivalent, or inconclusive (§3, Table 17). Table 18 lists all 48 comparisons, of which 7 change speed and 12 change cost, and Table 19 pools each intervention; the abstract, the figure markers, and the text report these same verdicts.
  • Harnesses, and runs with and without the score tool, run under one limit (F2uw, K5P9, uGbp). No harness is significantly faster than Claude Code, and the token savings of OpenHands and mini-SWE-agent are not significant. Without the score tool, agents are slower: 0.86x on the loop-level kernels and 0.82x on the scientific-computing kernels (0.85x and inconclusive without Kimi-K2.7; Table 2, Table 19, Figure 5).
  • The detectable effect is measured (K5P9). An A/A study measures how much identical runs vary and, from that, how large an effect a comparison on 40 kernels can detect: about 1.3-2.2x in speed and 1.4x in cost (§4.1.1, Appendix C.2).
  • Results depend on the track (uGbp, dHEr). Models agree on which loop-level kernels are hard (rank correlation 0.83-0.89) more than on ML (0.45-0.61) or scientific-computing kernels (0.22-0.51), and the faster model changes between tracks (§4.5, Appendix C.3).
  • Selection, baselines, and inputs are stated (K5P9, uGbp). §4.1 states how the kernel subsets are drawn and that GPU speedups use the CPU baseline, with Pluto, PPCG, and hand-tuned Triton solutions as comparators (Appendix C.1); in the window-shortcut case study, a generator that plants an earlier match rejects the shortcut answers (§4.6, Appendix C.4).
  • Positioning and extensibility (D2KY, dHEr). Table 1 now also compares the benchmarks on ML kernels and on five properties of their evaluation protocol: hidden inputs, a held-out final grade, tested comparisons, intervals, and cost, with the rule for each column in Appendix B. It adds ParallelKernelBench, D2K-Bench, and KernelBench-Verified. The introduction states the objective, and the paper adds the input audits (Appendix A), the cost of writing a plugin (§2.2), and a limitations section (§4.7).

The final revised PDF (end of November 18, AoE) adds the runs that finish by then, under the same limits: the ML scaling reruns, a score-feedback study on three ML kernels, and the git framing and Triton references on the scientific-computing kernels. Where the PDF differs from our earlier replies, the PDF is current; its hidden-input definition (Appendix B) also corrects our reply to D2KY on PerfCodeBench, AlgoTune, and GSO.

AuthorsOpenReview ↗

Second Revised PDF Uploaded

Thank you all for the constructive discussion. Since the first revision, which gave every comparison a tested verdict (different, equivalent, or inconclusive), we have rerun every comparison except CPF under one limit (50M tokens, at most 12M output) and graded every answer under the final protocol. The final PDF reports only these runs; where it differs from our earlier replies, the PDF is current. Changes are in blue.

What the reruns confirm (uGbp, dHEr, D2KY). Under one limit, the main findings hold and gain a third track. Removing the score tool slows agents on both LLR and SC (0.86x and 0.81x against single submission). Submitting more than once leaves speed unchanged (equivalent). Kernel difficulty, in contrast, depends on the track: models agree on which kernels are hard on LLR (rank correlation 0.77 to 0.96), less on SC (0.58 to 0.71), and least on ML (0.10 to 0.71). A single-track benchmark would report either finding as a general fact. Agents also stay well ahead of automatic parallelizers (up to 3.0x over Pluto and 5.0x over PPCG), and on 16 GPUs, 8 of Qwen3.8-27B's 11 answers with RCCL hints beat PyTorch distributed (Figure 7).

Equal limits and tested verdicts (F2uw, K5P9). Every comparison except the three CPF comparisons comes from runs under one limit (Table 2); the CPF comparisons will follow for the camera-ready version. Equivalence is now reachable where the data support it: 5 of the 48 single comparisons are equivalent in speed within 1.2x, and the pooled Language skill packet is equivalent in speed, solve rate, and cost (1.05x, 95% CI 0.99 to 1.11; Table 19). The A/A study, now with the ML kernel gelu, shows why pooling is the designed route: one episode per kernel on 40 kernels detects speed effects of about 1.3x to 2.0x. The Kimi K2.7 A/A study will follow for the camera-ready version.

Decisions the verdicts support (uGbp). A skill earns its place by moving speed, solve rate, or cost past the margin. The Performance Toolkit, for example, keeps the solve rate (equivalent) and lowers cost 1.21x without a measurable change in speed, so it is worth adopting; the Language skill packet changes none of the three. None of the 20 related benchmark papers we could verify tests such a comparison for significance or equivalence, and authors who want a decisive verdict on their own toolkit can run the same protocol on more kernels or repeated episodes (about 33 paired runs settle equivalence within 1.2x for OSS-120B).

What the reruns refine (K5P9, F2uw, uGbp). The git framing can slow agents: Kimi-K2.7 is significantly slower (0.78x) on 27 SC kernels and no model becomes significantly faster, while the pooled change (0.94x) stays within the run-to-run noise, refining the two-model estimate of our earlier reply. Under one limit, no harness is significantly faster than Claude Code, and OpenHands costs OSS-120B 2.2x more tokens; the harness token savings we first reported do not hold under equal limits. The ML rerun with 20 models replaces the first-run numbers of our 17 November reply to uGbp (for example, ML difficulty agreement is now 0.10 to 0.71, was 0.45 to 0.61).

The camera-ready version extends these results. First, the score-feedback comparison runs on 40 single-node ML kernels (ML40), drawn by the same quota per level and tested as on LLR and SC; this is the validation uGbp asked for, and it completes the comparison on all three tracks. Second, Kimi-K2.7 joins the A/A study (K5P9). Third, the few runs that started under a lower limit are rerun at 50M tokens and 12M output, so every row of Table 2 shares one limit. Fourth, if compute allows, one model runs on a full track (K5P9).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.