- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose HPCAgent-Bench, a benchmark of 689 kernels for evaluating code-optimizing agents across CPUs, GPUs, and multiple nodes. The framework checks correctness using inputs from secret seeds, tests whether measured speedups are statistically significant, and flags suspicious implementations. It also provides paired comparisons of interventions like skills, tools, and harnesses, along with explicit token-cost accounting. Experiments with 3 open-weight models look at performance, portability, and intervention effects. The main reported finding is that interventions rarely change speed significantly but can substantially change token cost.
Critical strengths
-
The grading protocol looks at concrete ways performance measurements can be misleading. Separate seeds for scoring and final grading, changing inputs between timed calls, and device checks are useful safeguards. The case studies show that one cached answer drops from 5309× to 20× on fresh inputs, and two CPU submissions are caught running their code on the GPU.
-
Comparing setups through individual interventions looks at whether adding a skill, tool, or harness improves results enough to justify its cost. Reporting the token components separately is also valuable. Figure 5 shows that the choice of token weights can substantially change the saving. The framework also covers several languages, devices, and scientific domains, including multi-node execution, and the paper describes a detailed reproducibility package containing observations, submitted programs, and scripts to regenerate the results.
Critical weaknesses
-
Some headline numbers don't match the figures. The abstract says interventions significantly changed speedup in only 2 of 40 comparisons, but Figures 3-4 show 5 significant changes, 3 of them from removing score feedback (Section 4.4). For tasks posed as git repositories, the text reports speedup ratios of 0.52-0.68, while Figure 4(i) shows almost no change for 2 of the 3 models. These results support the paper’s main conclusion, so they need to be reconciled.
-
I’m not convinced by the conclusions that some interventions make no difference. Each comparison uses only 10-40 kernels, and the harness study runs each kernel once, so even pretty large effects might not reach significance. The git-repository framing is a good example. Going by the numbers in Section 4.4, it cuts speedup by 32-48%, yet the paper says it changes “nothing measurable”. The harness comparison also isn’t apples-to-apples. The same section says budgets are matched, but in Table 12 two of the harness runs get twice the tokens and time of the Claude Code baseline, so it’s hard to say how much of any difference comes from the harness itself.
-
It's not clear what counts as correct, and how much of the benchmark was tested. In the Section 4.6 example, some agents first search the window where the test generator always places the match. This passes every test but gives the wrong answer if an earlier element also matches. The validity depends on whether the restriction is part of the task, which the paper doesn't mention. The experiments also cover only selected subsets, around 100 of the 689 kernels, with little explanation of how they were chosen, and loop-level GPU speedups are measured against a CPU baseline, which makes them hard to interpret.
Questions for authors
-
Could you reconcile the “only twice” claim with the 5 significance markers in Figures 3-4, and the git-framing ratios in Section 4.4 with Figure 4(i)?
-
How many episodes per kernel and arm does each experiment use, and could an A/A comparison show what effect size the design can detect? Do the harness results hold under matched token and time limits?
-
Is the planted-window restriction part of ext_break_capture’s declared input contract? If not, have the other generators been checked for similar shortcuts?
-
How were the evaluation subsets selected, and could you report at least one model on each full track?
2 comments
Follow-up before the discussion closes
Thank you again for your careful reading. Following your questions, the revised PDF reports all 48 comparisons (Table 18), plots the git framing by each kernel's median with a paired interval (Figure 5a), and reports non-significant results as inconclusive. Could you let us know whether the revision resolves your concerns? If anything is still unclear, we would be glad to address it in the final revision.
Response to K5P9
Please also see our general response (G1–G3).
Why 2 and 5 significant effects? The paper has 48 comparisons. Of these, 40 compare interventions and 8 compare the benchmark's own submission modes. The abstract's 2 counts speed effects among the 40 interventions. The 5 markers in Figures 3 and 4 add 3 speed effects among the submission modes. The revision will report all 48.
Why do the git ratios differ from Figure 4(i)? In the git experiment, each of the 10 kernels ran three episodes per setup and model. By each kernel's median, git agents reach 0.55 to 0.70 times the speedup of the kernel framing, while Figure 4(i) used each kernel's last run, which hides the slowdown for two models. The revision plots the medians with a paired interval (thank you for catching this), and the paired test finds a significant slowdown for two of the three models, which replaces "nothing measurable".
Is one run per kernel enough? For large effects, yes. To show that a single comparison has no effect, no. Every other experiment runs one episode per kernel and setup. The revision therefore reports non-significant results as inconclusive and claims "no meaningful difference" only where pooled data support it, for example for skill packets (G3).
To measure how much one setup varies from run to run, we ran an A/A comparison with 3 kernels and 20 episodes per kernel and model. Across episodes, the standard deviation ranges from near 0 to 1.7 for log speedup and from 0.41 to 0.71 for log token count. On 40 kernels with one episode each, the smallest difference detectable with 80% power is therefore about 1.3x in speedup for OSS-120B, 2.2x for Qwen3.8-27B, and 1.4x in tokens.
Do results hold under matched limits? Partly. The speed results hold, but two token savings lose significance. The release runs both setups under the same limits. In a first rerun under one limit for every harness (48M tokens), no harness reaches a significantly higher or lower speedup than Claude Code. The token savings of OpenHands for Qwen3.8-27B (2.14x) and mini-SWE-agent for OSS-120B (3.04x) shrink to 1.40x and 1.35x and are no longer significant, while OpenHands now uses 2.4x more tokens than Claude Code for OSS-120B (significant).
Is the window shortcut correct? It passed only because of our inputs. The paper's generator always placed the match between 40% and 70% of the input, a placement the task never requires: an answer must return the reference's result for every valid input. The revised generator adds an earlier match, which held-out inputs place anywhere in the first 40%, so searching the window first now fails. We are rerunning the experiment.
How were the subsets chosen? For each track, we fixed a quota per level and drew that many kernels at random from each level of the paper's corpus: 24 level-1 and 16 level-2 kernels of the 248 loop-level reasoning (LLR) kernels, and 3, 10, and 22 kernels of levels 1, 2, and 3 of the 171 scientific computing (SC) kernels. The SC subset had 35 in the paper, and the new release extends it to 40. Subsets of this size let one budget cover many ablations. The 6,800 episodes test 48 comparisons over three models and three languages, while the full corpus allows fewer than ten setups for the same compute budget. If the compute budget allows, we will run the full SC and LLR tracks with one model for the camera-ready version.
What is the GPU baseline? GPU speedups are measured against the CPU Numba build, and Figure 3(ii) will say so. No automatic GPU optimizer currently exists that covers the subset. Instead, the revision compares the agents' GPU speedups with Triton reference solutions that we built and tuned for the 40 LLR kernels and the release's 40 SC kernels.