- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
The paper presents SWE-KERNEL-BENCH, a live benchmark for evaluating LLMs on GPU kernel generation tasks. These tasks are derived from publicly available pull requests. The authors build an automatic pipeline to find relevant PRs, and extract the tasks. These tasks contain original source code and the merged code as a golden implementation. The current benchmark includes 161 tasks from 10 production libraries, including both CUDA and Triton kernels. The authors evaluate eight models with the coding-agent harness and find that the models can produce correct results in most tasks, but only beat gold performance on about half of the tasks.
Critical strengths
- The benchmark is extracted from real-world PRs, which represent practical GPU kernel optimization tasks.
- The automated benchmark construction pipeline is potentially scalable, since it can update the benchmark according to real-world kernel changes in the open-source repos. This may enable the benchmark to adapt to new workloads and new hardware.
- The evaluation of multiple agentic systems with and without GPU feedback provides a useful comparison of their kernel optimization capabilities.
Critical weaknesses
W1. Correctness verification is fundamentally flawed.
The headline claim that agents produce correct kernels on up to 91% of tasks rests on an unreliable correctness check. The driver never clones inputs before running each implementation. For approximately 20% of the tasks involving in-place operations, the reference and candidate outputs can therefore be the same tensor, making the correctness check meaningless. This also affects gold admission. The released solutions demonstrate the problem: an audit identified at least 36 incorrect solutions marked as correct, including 15 counted as passes. The largest gold-relative speedup (218×) comes from a copy routine that very likely performs no copy. Other problems include silently dropping OOM workloads, applying a 1% relative tolerance to integer outputs, and failing to check output memory layouts. These flaws directly undermine the reported correctness and performance results. The authors should fix the verification procedure and re-evaluate the benchmark and agent solutions.
W2. The quality of extracted tasks is insufficiently validated.
Quality control relies primarily on an LLM reviewer and manual inspection of only 20 tasks. The released benchmark contains several questionable baselines. Some sources are inefficient Python loops with frequent host synchronization, while others execute on the CPU with host-device transfers included in the timed region. These can inflate the reported gold speedups without reflecting meaningful GPU kernel optimization. Some tasks also have incorrect source-type labels, resulting in inconsistent application of torch.compile. Moreover, the timing protocol includes host dispatch overhead, which can dominate small workloads. The corpus also contains duplicate tasks and gold implementations that can be outperformed by simply reusing the source. Per-task gold speedups and provenance records are not released, making the reported optimization headroom difficult to verify. More rigorous validation of extraction fidelity and benchmark quality is needed.
W3. The leaderboard comparison is insufficiently controlled.
The evaluation conditions differ across systems in ways that are not adequately disclosed. Different models use different harnesses and execution budgets, with time limits differing by up to 12× and additional token limits for some models. In particular, Gemini uses different harnesses for oneshot and GPU-tools evaluation, and an additional GPU-tools run is not reported. The Gemini results in Figure 14 also disagree with the released outcomes.json, raising concerns about the consistency of the reported results. Furthermore, each system is evaluated only once per task, and the difference between the two leading systems is not statistically significant (McNemar p = 0.30). The agents also run without enforced isolation from the gold implementations, while the contamination analysis provides limited evidence against data leakage. These issues make it difficult to attribute the reported differences to model capabilities or GPU feedback alone.
Questions for authors
- Can the authors fix the correctness verification issues and report the updated results?
- Can the authors provide stronger evidence of the task quality?
- Can the authors clarify the evaluation configurations, explain the Gemini results and assess how these issues affect the conclusions?
15 comments
Gemini evaluation and consistency of Figure 14
Reviewer comment:
In particular, Gemini uses different harnesses for oneshot and GPU-tools evaluation, and an additional GPU-tools run is not reported. The Gemini results in Figure 14 also disagree with the released outcomes.json, raising concerns about the consistency of the reported results.
Response:
Both reported Gemini modes used mini-SWE-agent, as verified against the published solution files. The additional, earlier GPU-tools campaign predominantly used Gemini CLI. We selected the mini-SWE-agent campaign as a whole because it provided usable token and cost telemetry, while the Gemini CLI did not.
Regarding the Figure 14 discrepancy: it was mistakenly produced from older results and a different pass criterion. We regenerated the figure, now Figure 15 in Appendix C.7. The refreshed figure and main table are now consistent.
We apologize for the inconsistent figure and the confusion it caused. Thank you for identifying this issue and helping us improve the consistency of the reported results.
Licensing and attribution corrections in the revised supplement
Reviewer concern (excerpt):
“…clarify how they ensure license compliance and provide the required notices, attribution, and modification information in the released benchmark.”
Response:
Thank you for identifying these omissions. We acknowledge that the original supplement lacked the licensing and attribution material described in Appendix D.
We selected the source repositories based in part on licenses permitting the extraction, adaptation, and redistribution involved in this benchmark, subject to their conditions. We recognize that this also requires preserving the applicable notices and attribution.
The revised supplement now includes copyright, license, contributor, and modification notices in task code; complete applicable license texts, required NOTICE files, and embedded-component terms under legal/third_party_licenses/; and per-repository legal/<repository>/ATTRIBUTION.md files with task-specific credits.
We have also updated our licensing workflow to preserve upstream notices and validate legal-document coverage and attribution references. Extracted code remains under its applicable upstream licenses, and the full public release will retain these notices and terms.
If these corrections satisfactorily address your concerns, we would be grateful if you would consider updating the ethics-review flag accordingly. Thank you again for bringing these omissions to our attention.
Thank you for your thoughtful review and for recognizing the benchmark’s grounding in practical GPU optimization tasks, the scalability of our construction pipeline, and the value of comparing systems with and without GPU feedback.
We will post responses to the issues you raised as we work through them.
Looking forward to a fruitful discussion!