acceptodds
Under review as a conference paper at ICLR 2027

SWE-KERNEL-BENCH: A Live Benchmark for GPU Kernel Generation

Abstract

We introduce SWE-KERNEL-BENCH, a live GPU kernel optimization benchmark that is periodically refreshed with new tasks derived from performance improvements merged into widely used open-source repositories. Its automated pipeline converts merged pull requests into tasks, each pairing an unoptimized source implementation with the corresponding maintainer-reviewed and shipped gold implementation. The benchmark evaluates whether coding agents can attain this performance target while preserving correctness. The first release contains 161 tasks from 10 production repositories and spans CUDA and Triton. Evaluating the benchmark with eight contemporary agentic systems shows that the best-performing system produces correct kernels for 91% of tasks, yet matches or exceeds gold performance on only 54%, revealing substantial headroom in real-world GPU kernel optimization.

Primary area:
datasets and benchmarks
Keywords:
GPU kernels, live benchmark, coding agents, LLM agents, performance optimization, CUDA, code generation, datasets and benchmarks

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_KDimOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper presents SWE-KERNEL-BENCH, a live benchmark for evaluating LLMs on GPU kernel generation tasks. These tasks are derived from publicly available pull requests. The authors build an automatic pipeline to find relevant PRs, and extract the tasks. These tasks contain original source code and the merged code as a golden implementation. The current benchmark includes 161 tasks from 10 production libraries, including both CUDA and Triton kernels. The authors evaluate eight models with the coding-agent harness and find that the models can produce correct results in most tasks, but only beat gold performance on about half of the tasks.

Critical strengths
  • The benchmark is extracted from real-world PRs, which represent practical GPU kernel optimization tasks.
  • The automated benchmark construction pipeline is potentially scalable, since it can update the benchmark according to real-world kernel changes in the open-source repos. This may enable the benchmark to adapt to new workloads and new hardware.
  • The evaluation of multiple agentic systems with and without GPU feedback provides a useful comparison of their kernel optimization capabilities.
Critical weaknesses

W1. Correctness verification is fundamentally flawed.

The headline claim that agents produce correct kernels on up to 91% of tasks rests on an unreliable correctness check. The driver never clones inputs before running each implementation. For approximately 20% of the tasks involving in-place operations, the reference and candidate outputs can therefore be the same tensor, making the correctness check meaningless. This also affects gold admission. The released solutions demonstrate the problem: an audit identified at least 36 incorrect solutions marked as correct, including 15 counted as passes. The largest gold-relative speedup (218×) comes from a copy routine that very likely performs no copy. Other problems include silently dropping OOM workloads, applying a 1% relative tolerance to integer outputs, and failing to check output memory layouts. These flaws directly undermine the reported correctness and performance results. The authors should fix the verification procedure and re-evaluate the benchmark and agent solutions.

W2. The quality of extracted tasks is insufficiently validated.

Quality control relies primarily on an LLM reviewer and manual inspection of only 20 tasks. The released benchmark contains several questionable baselines. Some sources are inefficient Python loops with frequent host synchronization, while others execute on the CPU with host-device transfers included in the timed region. These can inflate the reported gold speedups without reflecting meaningful GPU kernel optimization. Some tasks also have incorrect source-type labels, resulting in inconsistent application of torch.compile. Moreover, the timing protocol includes host dispatch overhead, which can dominate small workloads. The corpus also contains duplicate tasks and gold implementations that can be outperformed by simply reusing the source. Per-task gold speedups and provenance records are not released, making the reported optimization headroom difficult to verify. More rigorous validation of extraction fidelity and benchmark quality is needed.

W3. The leaderboard comparison is insufficiently controlled.

The evaluation conditions differ across systems in ways that are not adequately disclosed. Different models use different harnesses and execution budgets, with time limits differing by up to 12× and additional token limits for some models. In particular, Gemini uses different harnesses for oneshot and GPU-tools evaluation, and an additional GPU-tools run is not reported. The Gemini results in Figure 14 also disagree with the released outcomes.json, raising concerns about the consistency of the reported results. Furthermore, each system is evaluated only once per task, and the difference between the two leading systems is not statistically significant (McNemar p = 0.30). The agents also run without enforced isolation from the gold implementations, while the contamination analysis provides limited evidence against data leakage. These issues make it difficult to attribute the reported differences to model capabilities or GPU feedback alone.

Questions for authors
  • Can the authors fix the correctness verification issues and report the updated results?
  • Can the authors provide stronger evidence of the task quality?
  • Can the authors clarify the evaluation configurations, explain the Gemini results and assess how these issues affect the conclusions?
15 comments

Thank you for your thoughtful review and for recognizing the benchmark’s grounding in practical GPU optimization tasks, the scalability of our construction pipeline, and the value of comparing systems with and without GPU feedback.

We will post responses to the issues you raised as we work through them.

Looking forward to a fruitful discussion!

AuthorsOpenReview ↗

Gemini evaluation and consistency of Figure 14

Reviewer comment:

In particular, Gemini uses different harnesses for oneshot and GPU-tools evaluation, and an additional GPU-tools run is not reported. The Gemini results in Figure 14 also disagree with the released outcomes.json, raising concerns about the consistency of the reported results.

Response:

Both reported Gemini modes used mini-SWE-agent, as verified against the published solution files. The additional, earlier GPU-tools campaign predominantly used Gemini CLI. We selected the mini-SWE-agent campaign as a whole because it provided usable token and cost telemetry, while the Gemini CLI did not.

Regarding the Figure 14 discrepancy: it was mistakenly produced from older results and a different pass criterion. We regenerated the figure, now Figure 15 in Appendix C.7. The refreshed figure and main table are now consistent.

We apologize for the inconsistent figure and the confusion it caused. Thank you for identifying this issue and helping us improve the consistency of the reported results.

AuthorsOpenReview ↗

Licensing and attribution corrections in the revised supplement

Reviewer concern (excerpt):

“…clarify how they ensure license compliance and provide the required notices, attribution, and modification information in the released benchmark.”

Response:

Thank you for identifying these omissions. We acknowledge that the original supplement lacked the licensing and attribution material described in Appendix D.

We selected the source repositories based in part on licenses permitting the extraction, adaptation, and redistribution involved in this benchmark, subject to their conditions. We recognize that this also requires preserving the applicable notices and attribution.

The revised supplement now includes copyright, license, contributor, and modification notices in task code; complete applicable license texts, required NOTICE files, and embedded-component terms under legal/third_party_licenses/; and per-repository legal/<repository>/ATTRIBUTION.md files with task-specific credits.

We have also updated our licensing workflow to preserve upstream notices and validate legal-document coverage and attribution references. Extracted code remains under its applicable upstream licenses, and the full public release will retain these notices and terms.

If these corrections satisfactorily address your concerns, we would be grateful if you would consider updating the ethics-review flag accordingly. Thank you again for bringing these omissions to our attention.

Reviewer_hYUTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The work introduces SWE-KERNEL-BENCH, a live benchmark to evaluate the capabilities of a coding agent to generate and optimize GPU kernels. This work features mining from open-source performance-related pull requests from production repositories and extracts standalone optimization tasks. This constructed benchmark is live, meaning that it can evolve over time. The authors evaluate 8 agentic systems. Results show that the strongest system achieves 91% correctness but matches or exceeds gold performance on only 54% of tasks, highlighting the remaining challenges in production-level GPU kernel optimization.

Critical strengths
  1. This paper is of great practical relevance. It's a meaningful effort and something that researchers care in production.

  2. This benchmark creation is a scalable approach, different from prior works.

  3. The evaluation seems comprehensive and thorough.

Critical weaknesses
  1. The extraction is done automatically, but there could be tasks that are not extracted in a faithful manner, since it's automated by an agent. E.g., could a kernel task be oversimplified during the extraction phase? The authors did some manual inpsection, but source-gold equivalence does not necessarily establish correctness, or could the authors justify on why such inspection and current reviewing process is enough?

  2. The current paper, as far as I understand, is a single benchmark release. It's not evolving yet. During the rebuttal phase, could the authors provide more evidence on how the benchmark has evolved?

  3. The experiment results do not show anything related with the benchmark evolving as well. It would be good to know what evolution is happening and how results are changing as well. Otherwise it's just a benchmark automation, not live bench.

Questions for authors

Please see the weakness.

3 comments

Thank you for your review and for recognizing the practical relevance of our benchmark, the scalability of its construction approach, and the comprehensiveness of our evaluation.

We will post responses to the issues you raised as we work through them.

Looking forward to a fruitful discussion!

AuthorsOpenReview ↗

Independent Validation of Extraction Fidelity

Reviewer comment:

The extraction is done automatically, but there could be tasks that are not extracted in a faithful manner, since it's automated by an agent. E.g., could a kernel task be oversimplified during the extraction phase? The authors did some manual inpsection, but source-gold equivalence does not necessarily establish correctness, or could the authors justify on why such inspection and current reviewing process is enough?

Response:

Thank you for highlighting the risk of oversimplification during automated extraction. We agree that source–gold equivalence alone does not establish faithful extraction. We conducted a new structured annotation study, as kindly suggested by Reviewer oJsT, to provide a larger independent validation with a more systematic rubric than our preliminary author inspection.

Four experienced kernel developers outside the author team reviewed 30 tasks sampled uniformly without replacement. Each reviewer assessed between five and ten tasks using a new browser-based annotation tool developed for this study. The tool displayed extracted code alongside information already produced by the extraction pipeline: references linking individual code segments to their locations in the originating repository, along with extraction notes.

Reviewers separately rated four aspects on a five-point scale, comparing the extracted implementations and workloads against upstream evidence. The mean ratings across all annotated tasks were:

AspectMean rating
Source fidelity4.37
Gold fidelity4.80
Workload grounding and coverage4.23
Computation and measurement equivalence4.30

The overall mean was 4.42/5, with 110 of 120 ratings at 4 or 5. Gold fidelity received the highest ratings, with 25 of 30 tasks receiving the maximum score of 5. These results support high extraction quality, particularly for the gold implementations. Their fidelity is especially important because they set the performance target against which solutions are evaluated: the ratings support confidence that this reference bar faithfully preserves real upstream developer optimizations.

We have added the study and its results to Section 4 and Appendix A.2, including the evaluation questions, rating distributions, and follow-up findings. The supplementary material includes the full annotation results and anonymized review inputs.

AuthorsOpenReview ↗

Benchmark evolution and planned updates

Reviewer comments: (Q2 and Q3)

The current paper, as far as I understand, is a single benchmark release. It's not evolving yet. During the rebuttal phase, could the authors provide more evidence on how the benchmark has evolved?

The experiment results do not show anything related with the benchmark evolving as well. It would be good to know what evolution is happening and how results are changing as well. Otherwise it's just a benchmark automation, not live bench.

Response:

The paper presents the benchmark’s first release. During development, it evolved internally from an early snapshot of roughly 40 tasks to the current 160-task evaluated release, as our extraction effort expanded from kernel-generation PRs to broader generation coverage and then kernel-optimization PRs. Many additional PRs have already passed discovery and are awaiting extraction. We also plan to refresh the June–July 2026 discovery snapshot using the latest repository histories.

We are planning periodic releases, with the second targeted for January 2027 and at least 100 additional tasks. This update will extend coverage to categories identified during discovery but not included in the first release, such as kernel fusion and memory-related changes, and broaden target-hardware coverage. New repositories we plan to screen include ROCm libraries, including rocBLAS, OpenXLA, Qiskit’s SBD eigensolver, PhysX, LAMMPS, GROMACS, and Taichi. Each release will report system performance on existing and newly added task cohorts under a consistent evaluation protocol, with results broken out by target hardware, making both benchmark growth and its effect on measured performance visible.

Reviewer_oJsTOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SWE-KERNEL-BENCH, a live benchmark for evaluating the ability of coding agents to generate and optimize GPU kernels based on real-world software engineering tasks.SWE-KERNEL-BENCH automatically mines merged performance-related pull requests from production repositories and converts them into standalone optimization tasks.

The construction pipeline consists of task discovery, decomposition, extraction, and independent review, followed by correctness and performance-based admission.The authors evaluate eight contemporary coding-agent systems under two settings: one-shot generation without GPU feedback and an interactive setting with a budget of eight GPU tool calls. The strongest configuration achieves 91% correctness but matches or exceeds the performance of the shipped gold implementations on only 54% of tasks. The paper further investigates sensitivity to performance thresholds, potential training-data contamination, differences across source implementation families, and tool-use patterns.

Overall, the paper contributes an automated framework for constructing realistic GPU optimization benchmarks and provides an informative assessment of the gap between generating correct kernels and achieving production-level performance.

Critical strengths

1) Real-world task grounding and a scalable construction pipeline.

A major strength is the use of actual performance improvements merged into production repositories as the basis for benchmark construction. This gives the tasks a meaningful engineering motivation and avoids relying entirely on synthetic operators or manually designed optimization challenges. The use of provenance records, independent extraction review, and hardware-based verification is particularly valuable for maintaining task quality.

The automated pipeline also provides a potentially scalable alternative to manually curated benchmarks. If the proposed infrastructure is maintained as described, it could become a useful resource for tracking progress in GPU kernel optimization over time. I consider this the paper's strongest contribution.

2) A practically meaningful evaluation that distinguishes correctness from optimization quality.

The paper evaluates whether agents can reproduce performance improvements that were actually shipped in production, rather than merely checking whether generated kernels execute correctly. The distinction is important: the strongest system achieves 91% correctness but only a 54% pass rate against the gold performance target.

The evaluation covers eight agentic systems and includes both one-shot and GPU-feedback settings. The additional analyses of performance thresholds, task characteristics, and tool usage make the benchmark more informative than a single leaderboard score. In particular, the results identify a substantial gap between functional correctness and high-performance kernel generation, motivating further research in this area.

Critical weaknesses

1) Task selection may substantially limit the representativeness of the benchmark.

Although the paper motivates SWE-KERNEL-BENCH as a benchmark of real-world production GPU optimization, the released tasks are selected through several restrictive stages. In Section 3.3, only 363 of the 1,309 PRs that passed discovery were processed for extraction. Moreover, 317 of these were prioritized from the kernel-generate category, while the remaining 46 kernel-optimize PRs were selected using a best-first strategy that favors easier extraction and higher confidence.

This is understandable from an engineering perspective, but it introduces a potentially important selection bias. The resulting benchmark may overrepresent optimization tasks that are relatively easy to isolate and reproduce, while excluding more challenging production optimizations involving complex dependencies, stateful execution, or integration constraints.

Consequently, the reported performance may not fully reflect agents' capabilities on the broader population of production GPU optimization tasks. I would appreciate a more systematic characterization of the selection bias and a clearer discussion of the intended scope of the benchmark.

2) The evidence for the fidelity of automatically extracted tasks remains limited.

The paper relies heavily on LLM-based extraction and review to transform real pull requests into standalone benchmark programs. Although the proposed safeguards are thoughtful, the main independent human validation consists of two authors examining a random sample of 20 out of 161 released tasks, with all sampled tasks judged faithful to their originating PRs (Section 3.3).

This is encouraging, but the sample is relatively small, and agreement between the authors does not necessarily establish that the extracted workloads capture all performance-relevant aspects of the original production environment. In particular, numerical equivalence between the extracted source and gold implementations does not by itself guarantee that the extracted computation and workloads faithfully represent the original optimization.

Since the reliability of the automatically generated tasks is fundamental to the paper's contribution, stronger validation would increase confidence in the benchmark. A larger independent audit, ideally with a structured evaluation rubric and explicit reporting of discrepancies or borderline cases, would help substantiate the paper's central claims.

Questions for authors

Q1) Representativeness and selection bias (Sections 3.2–3.3).

Could the authors clarify how the 363 PRs selected for extraction differ from the remaining 946 PRs that passed discovery? In particular, how do their category distributions, estimated difficulties, and repository origins compare? Would it be possible to provide a small randomized extraction study (or at least a stratified analysis) to assess whether the current selection strategy systematically favors certain types of optimization tasks? This would help determine how broadly the benchmark results can be generalized.

Q2) Validation of extraction fidelity (Section 3.3)

Could the authors provide additional details about the manual validation of the 20 sampled tasks? Specifically, what criteria were used to assess faithfulness, and did the authors separately inspect source fidelity, gold fidelity, and workload fidelity? Was there any task-level disagreement between the two reviewers before reaching the final conclusions? A larger independent validation sample or a structured audit of particularly challenging extractions would strengthen the evidence for the automated pipeline's reliability.

Q3) Fairness of cross-system comparisons (Section 5.1).

The evaluated systems use different agent harnesses: Claude Code for Opus 5, Codex for GPT-5.6, and mini-SWE-agent for the remaining models. Could the authors clarify which aspects of the execution environment are standardized beyond the GPU tool-call budget and task prompt? To what extent might differences in agent harnesses, interaction protocols, or available coding capabilities influence the observed leaderboard rankings? A controlled comparison using a shared harness for at least a subset of models would help distinguish model-level performance from system-level effects.

5 comments

Thank you for your thoughtful review and for recognizing the real-world grounding and scalability of our construction pipeline, and the value of distinguishing correctness from optimization quality.

We will post responses to the issues you raised as we work through them.

Looking forward to a fruitful discussion!

AuthorsOpenReview ↗

Harness Comparison for Claude and GPT

Reviewer question (Q3):

The evaluated systems use different agent harnesses: Claude Code for Opus 5, Codex for GPT-5.6, and mini-SWE-agent for the remaining models. Could the authors clarify which aspects of the execution environment are standardized beyond the GPU tool-call budget and task prompt? To what extent might differences in agent harnesses, interaction protocols, or available coding capabilities influence the observed leaderboard rankings? A controlled comparison using a shared harness for at least a subset of models would help distinguish model-level performance from system-level effects.

Response:

Our original choice of harnesses was intended to reflect real-world usage: open-source models run with an open-source harness, while Claude and GPT run with their native harnesses, Claude Code and Codex. We agree that your suggested controlled comparison using a shared harness is an important control for distinguishing the effect of the model from that of the harness. We therefore used the same Claude Opus 5 and GPT-5.6 Sol models to generate new solutions with mini-SWE-agent on the same set of tasks in both interaction modes: oneshot (without GPU feedback) and gpu-tools (with GPU feedback). We benchmarked these solutions using the same experimental setup as the solutions generated with the native harnesses.

We now describe this comparison in Section 5.3, in the paragraph titled “Common-harness comparison,” and provide the full results in Appendix C.9.

Appendix C.9 includes Table 11, which summarizes the changes in the main leaderboard’s metrics. The table below reproduces these results.

  • Correct is the percentage of tasks solved correctly.
  • Pass is the percentage solved correctly while matching or exceeding the gold implementation’s speed.
  • Gold-relative speedup is the speedup of the agent’s solution over the gold implementation, aggregated geometrically over correctly solved tasks.

Each table cell shows the mini-SWE-agent metric minus the corresponding native harness metric, so positive deltas favor mini-SWE-agent. Correct and Pass changes are in percentage points (pp); the gold-relative speedup change is in ×.

ModelModeΔCorrect (pp)ΔPass (pp)ΔGold-relative speedup (×)
Claude Opus 5oneshot+12.5+15.6+0.23
GPT-5.6 Soloneshot+0.6+5.0+0.03
Claude Opus 5gpu-tools+0.6+10.0+0.15
GPT-5.6 Solgpu-tools−1.9+1.3−0.01

Compared with their native harnesses, GPT’s correctness is similar in both modes; its pass rate increases slightly in oneshot mode, while its gold-relative speedup remains nearly unchanged. For Claude, correctness improves most without GPU feedback (oneshot), while pass rate and gold-relative speedup improve in both modes. In particular, Claude oneshot correctness increases from 80% with its native harness to 92% with mini-SWE-agent (+12.5 pp), and its pass rate increases from 38% to 54% (+15.6 pp). With GPU feedback (gpu-tools), Claude’s pass rate also improves with mini-SWE-agent (+10.0 pp).

These results confirm that the choice of harness can materially affect performance in some scenarios. Overall, mini-SWE-agent performs comparably to or better than the native harnesses, suggesting that the other open models are not disadvantaged by using it.

AuthorsOpenReview ↗

Independent Validation of Extraction Fidelity

Reviewer question:

Could the authors provide additional details about the manual validation of the 20 sampled tasks? Specifically, what criteria were used to assess faithfulness, and did the authors separately inspect source fidelity, gold fidelity, and workload fidelity? Was there any task-level disagreement between the two reviewers before reaching the final conclusions? A larger independent validation sample or a structured audit of particularly challenging extractions would strengthen the evidence for the automated pipeline's reliability.

Response:

Thank you for emphasizing the importance of validating extraction fidelity. Following your suggestion, we conducted a structured annotation study to provide a larger independent validation with a more systematic rubric than our preliminary author inspection. Four experienced kernel developers outside the author team reviewed 30 tasks sampled uniformly without replacement.

Each reviewer assessed between five and ten tasks using a new browser-based annotation tool developed for this study. The tool displayed extracted code alongside information already produced by the extraction pipeline: references linking individual code segments to their locations in the originating repository, along with extraction notes.

Reviewers separately rated four aspects on a five-point scale. The mean ratings across all annotated tasks were:

AspectMean rating
Source fidelity4.37
Gold fidelity4.80
Workload grounding and coverage4.23
Computation and measurement equivalence4.30

The overall mean was 4.42/5, with 110 of 120 ratings at 4 or 5. Gold fidelity received the highest ratings, with 25 of 30 tasks receiving the maximum score of 5. These results support high extraction quality, particularly for the gold implementations. Their fidelity is especially important because they set the performance target against which solutions are evaluated: the ratings support confidence that this reference bar faithfully preserves real upstream developer optimizations.

Regarding task-level disagreement, both the original inspection and the new study assigned one reviewer per task to prioritize task coverage, given the substantial time required for each review. Reviewers in the new study reported 15–35 minutes per task. Inter-annotator agreement therefore cannot be estimated.

We have added the study and its results to Section 4 and Appendix A.2, including the evaluation questions, rating distributions, and follow-up findings. The supplementary material includes the full annotation results and anonymized review inputs.

Reviewer_KorLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SWE-KERNEL-BENCH, a live benchmark for evaluating the ability of large language models and coding agents to generate and optimize GPU kernels for real-world workloads. The authors develop an automated pipeline that extracts tasks from merged pull requests in widely used open-source repositories, enabling periodic benchmark updates. The first release comprises 161 tasks from 10 repositories, with gold implementations written primarily in CUDA and Triton. The authors evaluate eight LLMs with their respective coding-agent harnesses under both one-shot and GPU-tool-enabled settings. They also investigate potential training-data contamination and analyze the effects of access to GPU development tools on agent performance.

Critical strengths
  1. The authors present an automated pipeline that turns merged GPU performance PRs into executable benchmark tasks. By documenting each stage, how many candidates it retains, and what it costs, the paper provides a practical reference for building refreshable benchmarks and generating kernel programming tasks.

  2. The authors directly examine possible training-data contamination by comparing pass rates across task dates and relating them to model training cutoffs or release dates. This gives readers a concrete way to assess whether prior exposure to public PRs might influence the results.

  3. The paper compares model–agent configurations with and without GPU tools. Integrating such kind of tools into automated kernel programming systems has been shown to be effective in existing works on agent design and model optimization. However, the impact of these tools on overall system performance has received little systematic benchmarking in prior work. The comparison reveals interesting differences in how much different systems benefit from tool access.

  4. Using the geometric mean instead of the arithmetic mean in speedup statistics helps reduce extreme values that occur in individual operator test cases. This is a common situation when testing some operators that run very fast.

Critical weaknesses
  1. The automated pipeline for building the benchmark is valuable, but the paper could make a clearer case for what SWE-KERNEL-BENCH itself adds to existing suites such as KernelBench and TritonBench. For example, are its tasks more challenging or diverse, and how much overlap is there in their sources? A quantitative comparison would help clarify what new aspects of kernel programming the benchmark can evaluate and what it contributes as a research resource.

  2. The pipeline draws its tasks from merged PRs in open-source repositories. Given the use of AI-assisted coding and even automated research workflows, some of the original implementations may have been generated partly or entirely by LLMs. These implementations may reflect patterns characteristic of particular model families, which could bias the evaluation results.

  3. The benchmark uses NVIDIA H100 GPUs for both task validation and evaluation. Kernel performance and optimization strategies depend heavily on the hardware. The source PRs may target adding support for or optimizing kernels for specific devices. The paper does not report how tasks are distributed across the hardware originally targeted by their PRs, and the evaluation covers only the H100. This limits both the benchmark’s hardware coverage and how broadly the results can be interpreted.

Questions for authors
  1. As noted in W1, my main concern is what SWE-KERNEL-BENCH adds as a new benchmark. What aspects of kernel programming does it evaluate that existing benchmarks such as KernelBench and TritonBench cover less well? Could you quantify how the benchmarks differ in task difficulty, diversity, and overlap?

  2. The Claude and GPT models use Claude Code and Codex, respectively, while the other models use mini-SWE-agent. Since the harness can materially affect performance, could you also report results for Claude and GPT with mini-SWE-agent so that all models can be compared under the same harness?

5 comments
AuthorsOpenReview ↗

Authorship bias

Reviewer comment: (W2)

The pipeline draws its tasks from merged PRs in open-source repositories. Given the use of AI-assisted coding and even automated research workflows, some of the original implementations may have been generated partly or entirely by LLMs. These implementations may reflect patterns characteristic of particular model families, which could bias the evaluation results.

Response:

We agree that AI-assisted gold implementations could favor the model families involved in writing them. We have added a dedicated discussion in Appendix C.5 (“Authorship bias”), summarized in §5.3 (“Leaderboard”), under the “Authorship bias” paragraph.

We report two checks:

  1. If assistance from a particular family became more prevalent over time and favored its solutions, we would expect that family’s relative performance to improve on newer tasks. Our temporal analysis shows no sustained trend of this kind (Figure 12; Appendices C.4–C.5).

  2. We audited public AI-assistance disclosures in the source PRs underlying all tasks in the benchmark’s first release. Most tasks lack attribution to a specific model family. The largest explicitly attributed group contains 13 tasks with disclosed Claude implementation assistance; we compare model pass counts on this subset in Table 9. The results provide no strong evidence of an authorship-related advantage, but the small sample prevents ruling out authorship bias.

Together, these checks provide no strong evidence of authorship bias, yet we agree they cannot entirely rule it out. We welcome your feedback on how to strengthen this assessment.

AuthorsOpenReview ↗

Liveness and additional computational coverage

Reviewer question:

As noted in W1, my main concern is what SWE-KERNEL-BENCH adds as a new benchmark. What aspects of kernel programming does it evaluate that existing benchmarks such as KernelBench and TritonBench cover less well? Could you quantify how the benchmarks differ in task difficulty, diversity, and overlap?

Response:

SWE-KERNEL-BENCH’s main distinction is its liveness: automatic task extraction from real production optimization PRs enables a continuously refreshable benchmark (§2; §4). New releases can incorporate newly merged PRs, while repository and problem-category selection lets us steer task diversity and target different levels of optimization challenge (§3.2). The first release focuses on kernel generation and optimization. We characterize our tasks as diverse in their originating repositories and source implementation languages (§4, Table 1), and challenging for current agents (§5.3, Figure 4).

SWE-KERNEL-BENCH, KernelBench, and TritonBench serve complementary goals. KernelBench and TritonBench emphasize systematic coverage of operators and programming capabilities, while SWE-KERNEL-BENCH captures optimization challenges developers actually encountered and resolved.

To quantify computational overlap between SWE-KERNEL-BENCH and existing benchmarks, we conducted an exploratory, code-grounded comparison of all 161 released tasks against two reference benchmarks, KernelBench and TritonBench, using two independent matching agents. Each agent searched each reference benchmark for the most computationally similar task and assessed whether it qualified as a match. Shared components alone did not count as matches. For 98.1% of SWE-KERNEL-BENCH tasks, both agents found no match in KernelBench. For TritonBench, both agents found no match for 70.2% of tasks, suggesting substantial additional computational coverage. We added a summary of these findings to §4. A new paper section, Appendix A.5, details the experimental setup, results, and agreement statistics between the matching agents.

We would appreciate your feedback on whether these clarifications and the overlap analysis address your concern.

AuthorsOpenReview ↗

Harness Comparison for Claude and GPT

Reviewer question: (Q2)

The Claude and GPT models use Claude Code and Codex, respectively, while the other models use mini-SWE-agent. Since the harness can materially affect performance, could you also report results for Claude and GPT with mini-SWE-agent so that all models can be compared under the same harness?

Response:

Thank you for this suggestion. Our original choice of harnesses was intended to reflect real-world usage: open-source models run with an open-source harness, while Claude and GPT run with their native harnesses, Claude Code and Codex. We agree that your suggested comparison is also important for distinguishing the effect of the model from that of the harness. We therefore used the same Claude Opus 5 and GPT-5.6 Sol models to generate new solutions with mini-SWE-agent on the same set of tasks in both interaction modes: oneshot (without GPU feedback) and gpu-tools (with GPU feedback). We benchmarked these solutions using the same experimental setup as the solutions generated with the native harnesses.

We now describe this comparison in Section 5.3, in the paragraph titled "Common-harness comparison," and provide the full results in Appendix C.9.

Appendix C.9 includes Table 11, which summarizes the changes in the main leaderboard's metrics. The table below reproduces these results.

  • Correct is the percentage of tasks solved correctly.
  • Pass is the percentage solved correctly while matching or exceeding the gold implementation's speed.
  • Gold-relative speedup is the speedup of the agent's solution over the gold implementation, aggregated geometrically over correctly solved tasks.

Each table cell shows the delta between the mini-SWE-agent metric and the corresponding native harness metric, so positive deltas favor mini-SWE-agent. Correct and Pass changes are in percentage points (pp); the gold-relative speedup change is in ×.

ModelModeΔCorrect (pp)ΔPass (pp)ΔGold-relative speedup (×)
Claude Opus 5oneshot+12.5+15.6+0.23
GPT-5.6 Soloneshot+0.6+5.0+0.03
Claude Opus 5gpu-tools+0.6+10.0+0.15
GPT-5.6 Solgpu-tools-1.9+1.3-0.01

Compared with their native harnesses, GPT's correctness is similar in both modes; its pass rate increases slightly in oneshot mode, while its gold-relative speedup remains nearly unchanged. For Claude, correctness improves most in oneshot mode, while pass rate and gold-relative speedup improve in both modes. In particular, Claude oneshot correctness increases from 80% with its native harness to 92% with mini-SWE-agent (+12.5 pp), and its pass rate increases from 38% to 54% (+15.6 pp).

These results confirm that the choice of harness can materially affect performance in some scenarios. Overall, mini-SWE-agent performs comparably to or better than the native harnesses, suggesting that the other open models are not disadvantaged by using it.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Progress update on reviewer comments

We thank the reviewers for their detailed feedback. Below is a summary of addressed items, with links to the detailed responses.

This summary will be updated as the discussion and revisions progress.

  • Extraction validation

    • Structured annotation study of extraction fidelity, gold fidelity, workload grounding, and measurement equivalence (oJsT, hYUT, KDim)
  • Additional empirical results

    • NEW: Generation variability and statistical significance (KDim)
    • NEW: Hardware coverage analysis and additional A100 evaluation (KorL)
    • Common-harness comparison for Claude and GPT (KorL)
    • Code-grounded overlap comparison with KernelBench and TritonBench (KorL)
    • Authorship-bias analysis and qualifications (KorL)
  • Benchmarking improvements

    • NEW: Revised OOM handling (KDim)
    • NEW: Validation of required output layouts (KDim)
    • Independent-input correctness checks and exclusion of one task (KDim)
    • Exact integer-output checking (KDim)
    • Corrected Gemini evaluation figure (KDim)
    • Standardized generation limits (KDim)
    • Audit of solver access to gold implementations and sandboxing (KDim)
  • Ethics

    • Licensing and attribution made explicit in the supplementary materials (KDim)
AuthorsOpenReview ↗

Discussion concluding remarks

Dear Reviewers,

Following your feedback, we have revised the paper and supplement, improved the evaluation procedure, and added independent validation and additional experiments.

Our progress summary lists the addressed concerns and links to the detailed responses.

We hope you will find the revisions and additional evidence satisfactory.

Thank you for this opportunity to strengthen our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.