acceptodds
Under review as a conference paper at ICLR 2027

KernelGenBench: Can LLMs and Agents Write Efficient Kernels Across Operator Sources and Hardware Platforms?

Abstract

Modern AI systems depend on specialized accelerator kernels, whose development is complicated by increasingly diverse operators and hardware. LLMs and agentic systems promise to automate this work, but existing evaluations do not show how performance varies across operator sources and hardware platforms, or what that variation costs. We present KernelGenBench, the first unified multi-source × multi-chip infrastructure for evaluating LLM- and agent-generated Triton kernels. Unlike benchmarks that require platform-specific target languages, KernelGenBench uses Triton as a common programming target across six hardware platforms. We report two controlled analytical views: KernelGenBench-MS (Multi-Source) covers 210 operators from PyTorch ATen, production vLLM operators, and proprietary cuBLAS routines, while KernelGenBench-MC (Multi-Chip) evaluates a semantically stable 110-operator subset across six hardware platforms. Our evaluation consumed over 15 billion tokens. More interactive agentic workflows achieved higher overall correctness, but no method dominated across sources and platforms: vLLM posed the strongest correctness challenge, cuBLAS set the highest performance ceiling, and AutoKernel with GLM-5.0 fell from 87% on NVIDIA to 25% on Iluvatar CoreX. Higher correctness came at substantial cost: specialized agents averaged 4.99 million tokens per successful operator, rising to 6.25 million for CUDA Optimized Skill. The results establish operator source, hardware platform, and generation workflow as distinct dimensions of kernel-generation capability, and show that success in a familiar source–hardware setting is not a reliable proxy for deployment readiness.

Primary area:
datasets and benchmarks
Keywords:
Kernel Generation, Triton, Large Language Models, Agentic Systems, Heterogeneous Accelerators, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_PDFoOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents KernelGenBench, a benchmark for evaluating LLM- and agent-based Triton kernel generation across different operator sources and hardware platforms. The benchmark consists of two evaluation settings: KernelGenBench-MS evaluates 210 operators from PyTorch ATen, vLLM, and cuBLAS on NVIDIA A100, while KernelGenBench-MC evaluates 110 ATen operators across six hardware platforms.

The authors also develop a unified evaluation framework with parameterized correctness tests and multi-level anti-hack verification mechanisms. They evaluate multiple LLMs and agentic frameworks in terms of generation correctness, kernel speedup, and generation cost. The experiments show that kernel generation performance varies across operator sources and hardware platforms, while agentic workflows generally achieve higher correctness at greater computational cost.

Critical strengths
  • Multi-platform evaluation. The benchmark covers six hardware platforms, which is valuable for understanding how LLM-based kernel generation systems behave across different hardware and software environments. Most existing evaluations focus primarily on NVIDIA GPUs, so this broader hardware coverage is a useful contribution.
  • Anti-hack verification and evaluation infrastructure. The anti-hack mechanisms are interesting and help prevent generated implementations from bypassing Triton by directly calling existing framework operators. In particular, the combination of AST-based static checks, ghost replay, and hardware profiling provides stronger verification than correctness testing alone. The framework also evaluates correctness across multiple input configurations rather than relying on a single fixed shape.
Critical weaknesses
  • Limited novel insights beyond existing knowledge. The paper provides extensive experimental comparisons, but most of the findings are not particularly surprising. For example, specialized vLLM operators are harder to implement correctly, cuBLAS kernels are difficult to outperform, and agentic workflows improve correctness at the cost of additional tokens. These observations are largely consistent with existing knowledge about kernel generation. Although the experiments provide quantitative evidence for these behaviors, the paper offers limited deeper analysis of why different agents succeed or fail, or what new insights can be drawn from the additional evaluation dimensions. Given the substantial overlap with existing kernel generation benchmarks, I am not fully convinced that the empirical findings are sufficiently novel or informative.
  • Insufficient isolation of backend effects in the multi-hardware evaluation. One of the main contributions of the paper is its evaluation across multiple hardware platforms. However, all generated kernels use Triton, while the maturity and supported features of Triton compiler backends differ significantly across platforms. As discussed in Appendix P, some failures arise from unsupported Triton APIs, missing data type support, compiler errors, and runtime incompatibilities. The evaluation also counts operators with unavailable platform-native references as unsuccessful. Therefore, it is unclear how much of the observed correctness degradation is caused by the kernel-generation agents themselves, rather than limitations of the underlying software stacks. I understand that the authors explicitly characterize these results as end-to-end platform success rates, but without better isolation of backend effects, it is difficult to draw meaningful conclusions about the agents' actual cross-platform generation capabilities.
Questions for authors
  1. Can the authors provide more detailed insights into why agents behave differently across hardware platforms? Beyond reporting correctness and performance differences, it would be useful to understand which types of generated code or optimization strategies are particularly sensitive to the target platform, and whether these differences reveal any general limitations of existing kernel-generation agents.
  2. Can the authors better isolate the effects of Triton backend compatibility? For example, could the authors evaluate the same Triton kernels, already verified on NVIDIA, across different platforms to establish a baseline for code portability? A quantitative breakdown of failures caused by generated code, unsupported compiler features, runtime issues, and unavailable references would also help explain the reported portability gaps.
  3. Most of the evaluated models are not the latest frontier models. Could the authors include more recent LLMs, such as Claude Opus 5.5 or GPT-6 Astra, at least on a representative subset of operators? Given the rapid improvements in coding and agentic capabilities, this would help assess whether the main findings remain representative of current kernel-generation systems.
  4. Can the authors extend the multi-hardware evaluation to vLLM operators? These operators appear to be among the most challenging and practically relevant tasks in the benchmark. However, the main cross-platform evaluation only covers ATen operators, while Appendix D provides additional cross-platform results for cuBLAS-derived operators. Since the paper also describes validated vLLM runtime adapters for several hardware platforms, it would be valuable to evaluate at least a representative subset of vLLM operators across different hardware backends.
Reviewer_8or9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

KernelGenBench is a benchmark infrastructure for evaluating LLM- and agent-generated Triton kernels along two axes prior work treats separately: operator source and hardware platform. It uses Triton as a single portable target across six platforms and reports two controlled views — KernelGenBench-MS (210 operators from PyTorch ATen, vLLM, and cuBLAS, on NVIDIA) and KernelGenBench-MC (a 110-operator ATen subset across NVIDIA, Moore Threads, Hygon, Iluvatar CoreX, and two anonymized platforms). It adds a three-tier anti-hack pipeline (AST scan, ghost replay, NVIDIA-only hardware profiling) and measures correctness, successful-set speedup, and agentic cost.

Critical strengths

Genuinely new axis combination. First benchmark to cross distinct operator-reference ecosystems: framework-native ATen, production vLLM, closed-source cuBLAS, with controlled multi-platform evaluation under one pipeline. The source axis cleanly separates three capabilities (implement a standard op; recover specialized behavior from an API spec; match a vendor-optimized library) that PyTorch-only benchmarks conflate. Reporting tokens/success and cumulative solving time surfaces the "agentic portability tax" and the 4.99–6.25M tokens/success of specialized agent is very good

Critical weaknesses

The multi-source × multi-chip claim is only partially realized.

Questions for authors

Can you provide even a partial populated source×platform grid, or state which cells are blocked by missing references vs. not yet run?

Reviewer_fk3yOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

KernelGenBench (KGB) evaluates LLM and agent systems that generate Triton kernels. It provides two complementary evaluations. KGB-MS covers 210 operators on an NVIDIA A100 from PyTorch ATen, vLLM operators, and cuBLAS ones. The ATen subset has the 50 most frequent operators along with 60 sampled from the long-tail of operators for a balance. KGB-MC evaluates them across six hardware platforms: NVIDIA, Threads, Hygon, CoreX, and two anonymous platforms.

The paper compares direct sampling, general coding agents (Claude Code & OpenCode), and kernel-specialized agents (AKO4all, CUDA Optimized Skill, and AutoKernel). It checks for correctness by passing tests that vary semantic parameters, shapes, data types, and memory layouts. ATen implementations are registered through torch.library to test replacement through PyTorch’s dispatcher. Three additional checks detect prohibited calls, test whether outputs depend on Triton execution, and inspect profiler traces.

The paper primarily shows that LLM agents unsurprisingly achieve higher correctness and speedups than direct sampling. cuBLAS is similarly the hardest baseline to outperform (which seems straightforward). Results also vary substantially across hardware, especially on Platform A thingy. Specialized agents are expensive, costing about five million tokens per successful operator (technically not that expensive I believe).

Critical strengths
  1. The ATen, vLLM and cuBLAS comparison provides overview of different kinds of optimization. I found testing across semantic implementation (ATen), production-specific operator behavior (vLLM) and competing with a tuned library (cuBLAS) to be super useful

  2. The hardware coverage was impressive. Evaluating across six accelerator stacks requires substantial engineering.

  3. Additional verification checks were helpful. They tests provide stronger evidence of correctness, as KenrelBench-Verified and GPUMode competitions shows annoyingly rough failure modes in verification. The paper also shows concrete examples of intercepted shortcuts, the work seems thoroughly done.

  4. Cost reporting improves the evaluation. Tokens for each successful run and portability tax are interesting bits most kernel benchmarks ignore, only reporting final correctness and speedup.

Critical weaknesses

W1. The speedup results need a clearer breakdown by workload size and execution overhead. On Platform A, most reported speedups are approximately 0.1–0.3×, although AutoKernel achieves higher values. I noticed the test grids include very small tensors e.g. (), (1,), (1,2) and (1,32) -- does launch and dispatch overhead not dominate in these cases? If benchmarked mostly on large tensors, do the aggregate results improve? (I'm not super familiar with how evaluation works). Or the slowdowns reflect the current state accurately?

W2. Do tests sufficiently check correctness on unseen inputs? It seems like the LLM agents can repeatedly use the verifier during development. If not, can the benchmark test KernelBench-Verified like testing for hidden input-distribution tests additionally?

W3. Are we sure withholding the reference implementation does not establish that a task is unseen? e.g. FlagGems already provide many PyTorch-compatible operators across several backends. Maybe comparisons with existing implementations, including FlagGems and TorchInductor could serve as good non-agentic baselines?

W4. Is the vLLM speedup conclusion conditional on the tasks solved? Would higher speedup on successful vLLM tasks establish better optimization? Difficult operators may be excluded because generation fails making comparison far harder.

Note that I am very unfamiliar with this field, as noted in my confidence. Apologies to the authors, hope they still find it useful?

Questions for authors

Q1. Aside from the above questions -- could you provide analysis on failure modes -- what fraction were generated-code errors, budget exhaustion, platform limitations? Do you have false rejections (overly strict tests?)

Q2. Why were so old versions of vLLM (v0.13) and cuBLAS (12.4) selected? How sensitive are the conclusions to these versions? Did agents have internet access or did changes in newer releases explain some failures?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.