acceptodds
Under review as a conference paper at ICLR 2027

FastKernels: Benchmarking GPU Kernel Generation in Production

Abstract

LLM-based agents for GPU kernel generation are advancing rapidly, but the benchmarks they optimize against evaluate kernels in isolation, with synthetic inputs and weak baselines, rewarding sandbox speedups that break or vanish in real inference systems. We introduce FastKernels, a benchmark of 384 tasks drawn from 47 representative architectures across 8 categories, whose kernels suffice to reimplement 94.6% (472/499) of HuggingFace Transformers architectures with outputs matching the native implementations. Each task mirrors the interface of the corresponding production module and is scored against the kernels production frameworks ship, and tasks form a compositional hierarchy, from primitives to full models, in which higher-level modules import lower-level ones. Candidates are scored at the kernel level and end to end inside the models they come from, on the production execution path, and MacroEval aggregates calibrated correctness, coverage, and speedup into a leaderboard. Seeding it with five representative agents (6,900 agent-hours), we find that kernel-level speedups of 1.6-6.6x shrink to at most 1.25x end to end, only 20% of winning kernel sets run correctly as-is, and kernel-level scores mis-rank agents: Claude Code matches or beats KDA at every level in isolation, yet KDA scores 3x higher end to end. Anonymized code is included in the supplementary material and will be open-sourced upon publication.

Primary area:
datasets and benchmarks
Keywords:
gpu kernels, cuda kernels, kernel benchmarking, CUDA generation, inference

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_MY8rOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces FastKernels, a benchmark for evaluating the GPU kernel generation capabilities of large language models (LLMs). The authors observe that kernels generated on other benchmarks may fail to preserve correctness or improve actual performance when integrated into full-model inference pipelines. The benchmark comprises 384 production-aligned tasks derived from 47 architectures across eight categories. These tasks are organized into an L1-L4 hierarchy covering primitive operators, fused operators, network layers, and full models. The evaluation combines kernel-level tests on specific input configurations with end-to-end execution and supports integrated evaluation of calls across task levels. MacroEval measures calibrated correctness, coverage, and speedups in throughput and latency. The study evaluates five agents on an 11-model default subset. The authors further report composition failures during integration, substantially smaller end-to-end gains than those observed in isolated kernel tests, and differences between kernel-level and end-to-end rankings.

Critical strengths
  1. Strong motivation and practical value. FastKernels addresses an important gap between isolated kernel speedups and improvements that remain correct and useful in full-model inference. This provides a practically meaningful evaluation target for GPU-kernel generation and optimization agents. The L1-L4 hierarchy captures dependencies across task levels rather than evaluating them independently. It enables integrated evaluation of generated kernels and makes their interactions, composition failures, and performance effects directly measurable.
  2. Extensive and informative empirical analysis. The authors conduct fine-grained experiments with five agents on the production-aligned benchmark, comparing independent, composed, and sequential modes alongside end-to-end evaluation. The detailed results on correctness, speedups, and integration failures provide valuable empirical reference points for developing future kernel-generation and optimization agents
Critical weaknesses
  1. Ranking robustness. MacroEval averages speedups only over valid tasks and families. If a slow but correct task becomes incorrect, it is removed from the speedup average. This can increase the overall score despite the correctness and coverage penalties (Section 5, Equation 5). The impact of this scoring behavior on the reported rankings remains unclear
Questions for authors

1.Does MacroEval include safeguards to prevent the overall score from increasing when a slow but correct task becomes incorrect and is excluded from speedup aggregation? 2.Would the reported agent rankings remain stable under a scoring rule that prevents correctness degradation alone from increasing the score?

2 comments
AuthorsOpenReview ↗

Response to Reviewer MY8r

Thank you for your careful review and detailed feedback, our response is below.

W1 / Q1 / Q2. A slow correct model becoming invalid

We thank the reviewer for this observation. In the submitted version, a slow but correct model could raise the ranking by dropping out of a geometric mean taken only over valid models. The correctness and coverage factors did not always offset that increase.

We have revised our MacroEval metric to close this gap. The revised PDF reflects the new ranking score. Every evaluated model stays in a fixed average. A model contributes its speedup s_i when the run executes and C_i ≥ τ_i, and zero otherwise:

s_i, & \text{if the run executes and } C_i\ge\tau_i,\\ 0, & \text{otherwise,} \end{cases} \qquad \mathrm{Score} =\frac{1}{|\mathcal{F}|}\sum_{f\in\mathcal{F}}\frac{1}{|B_f|}\sum_{i\in B_f}q_i.$$ Making any valid model invalid replaces a positive term with zero, so the score decreases, including when that model is the last valid one in its family. S_macro, C_macro, and coverage stay in the table as separate diagnostics. We re-scored all ten agent entries from the saved end-to-end results. KDA Independent remains first (1.138). Dr. Kernel Independent was 4th of those ten and is 7th of the eleven rows, below the unchanged baseline. Claude Code Independent, which matches or beats KDA at the kernel level, scores 0.574. The submitted product put that gap at 3× (1.11 versus 0.35). Under this score it is 2× (1.138 versus 0.574), which is what the abstract, introduction, and conclusion now say. The 1.25× in those sentences is the end-to-end speedup, separate from this ranking score. Section 5 defines the score, and Table 2 reports the revised leaderboard.

Thank you for your response and clarification. I will maintain my original score and supported position as discussed.

Reviewer_byLYOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to evaluate GPU kernel-generation agents when their outputs must function together within complete inference models. FastKernels derives 384 tasks at 4 levels, from primitives to full models, from 47 real model architectures. Each task follows the interface of the corresponding module in vLLM, SGLang or similar libraries, whose kernels serve as baselines. Higher-level modules import lower-level modules, so a candidate can be evaluated in isolation, together with the agent's other kernels, or regenerated on top of them. Candidates are scored on shapes captured from real serving runs. End-to-end evaluation uses the benchmark's model runtime, including compilation, CUDA graphs and tensor parallelism. MacroEval aggregates calibrated correctness, coverage and end-to-end speedup. 5 agents were run for about 6,900 agent-hours. The authors report that kernel-level speedups of 1.6×1.6\times to 6.6×6.6\times shrink to at most 1.25×1.25\times end to end, that only 20% of winning kernel sets run correctly as-is, and that kernel-level scores rank Claude Code above KDA while KDA scores 3×3\times higher end to end.

Critical strengths

S1. The task hierarchy is well designed. Higher-level tasks import the lower-level kernels they depend on, the authors can test the same kernel over production lower-level kernels and over the agent's own lower-level winners, and see directly whether kernels that win alone still work together.

S2. I appreciate that the authors run the winning kernels as-is in full models to check whether kernel-level speedups carry over. They investigate why kernels fail and also report cases where the kernels make the model slower.

Critical weaknesses

Major

W1. The correctness threshold may be too permissive. A model counts as valid when its request-averaged correctness reaches Ci≥τi=0.9C_i \geq \tau_i = 0.9 (L313-314), but the paper does not show that such models keep their downstream quality. Under Equation 1, Ci,rC_{i,r} decays linearly from 1 at the noise-calibrated threshold gig_i to 0 at fi=0.5f_i = 0.5 (L292-304), so with τi=0.9\tau_i = 0.9 a model stays valid until its discrepancy reaches roughly 0.1fi+0.9gi0.1 f_i + 0.9 g_i. For models with no measurable discrepancy in the noise run, gig_i is set to 10−310^{-3} (L301). In these cases, the discrepancy cutoff is effectively fixed. Table 14 lists accepted deployable subsets with mean discrepancies of up to 0.040 on such models. The sensitivity analysis changes gig_i and fif_i (L304, L458-459), but not τi\tau_i.

W2. Invalid models appear to be penalized twice. The default score is defined as Scoredefault=Smacro⋅Cmacro⋅Covmacro\mathrm{Score_{default}} = S_{\mathrm{macro}} \cdot C_{\mathrm{macro}} \cdot \mathrm{Cov_{macro}} (L341-344), so an invalid model reduces both CmacroC_{\mathrm{macro}} and Covmacro\mathrm{Cov_{macro}}. In Table 2, KDA and Claude Code have similar speedups of 1.13×1.13\times and 1.09×1.09\times, but their Covmacro\mathrm{Cov_{macro}} (1.00 vs. 0.52) and CmacroC_{\mathrm{macro}} (0.98 vs. 0.61) differ substantially. Multiplying these 2 factors accounts for most of the 3×3\times score gap. It is also unclear how CmacroC_{\mathrm{macro}} is computed when a model has no deployable subset. This could substantially change the score gap. I would also like to see a baseline that simply reuses the original kernels. Appendix I states that an unchanged baseline that passes counts as correct at 1×1\times, which meets the winner rule at L403. Submitting the unchanged baselines would therefore score about 1.00 and rank third among 11 entries, above 8 of the 10 agent-mode entries in Table 2. Dr. Kernel's Independent entry already illustrates the issue: its deployable subsets are slower than the baseline (0.98×0.98\times), yet it ranks 4th, ahead of both Claude Code entries and AKO's Independent entry.

W3. The comparison does not isolate the effect of the harness. L460-461 attributes KDA's advantage to its planning and review harness, and Appendix I says that comparing KDA and AKO with Claude Code isolates the harness. However, KDA uses an additional GPT reviewer and substantially more compute: 1,751 agent-hours and 12.31B tokens, versus 1,171 hours and 4.35B tokens for Claude Code in the Independent campaign (Table 11). The results show that KDA performs better in this setup, but do not establish how much of the gain comes from the harness rather than the additional model or compute. I would like to see a comparison at the same compute budget, including reviewer calls, before attributing KDA's advantage to the harness.

Minor

W4. No kernel is tested inside vLLM or SGLang. L220-221 says the kernels can be copied into vLLM, SGLang or another production framework without interface refactoring, but no candidate kernel is run inside an actual vLLM or SGLang build. Matching module signatures does not guarantee compatibility with engine-level allocation, stream, state and CUDA-graph conventions. The end-to-end tier runs on the benchmark's own model runtime (L79-81). Figure 2 reports speedups of 1.45×1.45\times and 1.51×1.51\times over production implementations on linear attention and recommendation. Several other categories are also visibly faster, although L359-360 describes them as near parity. These results are measured on the authors' own runtime, with production implementations used as references. I think the paper should make this distinction clearer.

Questions for authors

Q1. Could the authors add an unchanged-baseline row to Table 2 and explain how CmacroC_{\mathrm{macro}} and Covmacro\mathrm{Cov}_{\mathrm{macro}} are computed for a model with no deployable subset? Have the authors checked downstream task accuracy for deployable subsets with nonzero discrepancy on models whose noise run shows none (Table 14)?

Q2. What attempt budget and stopping rule govern the procedure that drops failing kernels and re-runs the model to find deployable subsets (L417-419), and was this budget the same for every agent? When a hang or corruption involves several interacting kernels, how is the fault localized beyond the traceback and bisection described at L418?

Q3. Can the authors compare KDA and Claude Code at matched agent-hours or tokens, and share the agent prompts and agent configurations?

Q4. Does the kernel tier run candidates under torch.compile and CUDA-graph capture? If not, how many of the drops attributed to compilation-stack conflicts and to crashes on engine shapes the kernel bench never produced (L423-426) would a compile-aware kernel tier have caught?

Q5. Have any of the winning kernels been tested directly inside official vLLM or SGLang builds?

3 comments
AuthorsOpenReview ↗

Response to Reviewer byLY (part 1)

Thank you for your careful reading and detailed feedback. Our response is below.

W1 / Q1(b). Threshold and downstream quality

For each model with a standard task score, we compared the accepted deployment with the lowest C to the unchanged baseline on the same inputs. Qwen3-Next, Dr. Kernel independent, GSM8K (n = 1319): accuracy changes by -0.38 points (-1.14, +0.38). gpt-oss, Claude Code sequential, GSM8K with low reasoning effort in both arms (n = 1319): +0.68 (-0.38, +1.74). FLUX, KDA independent, CLIPScore (n = 200): -0.04 (-0.11, +0.04). YOLOv10, Codex sequential, COCO box AP (n = 5000): -0.03 (-0.12, +0.09). YOLOv10 is one of two accepted runs with mean discrepancy 0.040; the other is Oasis, KDA independent. Every 95% interval contains zero. Llama 3.1, Kimi-Linear, Qwen3-VL, and BGE-M3 already have C ≥ 0.99 on every accepted run. Oasis has no standard task metric for these outputs. OpenFold3's accepted runs have mean discrepancy 0.112-0.157, at or below its benign-numerics floor g_i = 0.157. GLA's accepted runs have mean discrepancy at most 0.009, and C ≥ 0.98 on the worst. At τ = 0.95, KDA Independent scores 0.997 (10/11; Oasis rejected) and at τ = 0.99 it scores 0.738 (7/11). The unchanged baseline scores 1.000, so at both thresholds KDA remains the top agent and does not exceed the baseline. Of the four runs above, only YOLOv10 is rejected at 0.95; all four are rejected at 0.99. Their task scores do not move, which is why the leaderboard keeps τ = 0.9. The revised PDF adds the comparison.

W2 / Q1(a). Double penalty and baseline rank

We thank the reviewer for pressing on this. The revised ranking score no longer multiplies S_macro, C_macro, and coverage, so an invalid model reduces the score once (see also our reply to Reviewer MY8r below).

Section 5 now states the diagnostic convention. A run that does not produce a scored output, including an empty deployable subset, has C_i = 0 and valid_i = 0. Those zeros are what C_macro and Cov_macro average in. They do not create a second penalty in the ranking.

Table 2 of the revised PDF adds the unchanged baseline, which reuses the original kernels. All 11 models run and match the reference, each at a speedup of 1×, so the score is 1.000 and the row ranks 4th. Only KDA Independent (1.138), AKO Sequential (1.082), and KDA Sequential (1.056) score higher. Dr. Kernel Independent, whose conditional speedup is 0.98×, was 4th of the ten agent entries under the submitted score. It now scores 0.739 and ranks 7th of the eleven rows, below that baseline. KDA Independent stays first. Separately, the kernel tier treats an unchanged copy of a baseline kernel as correct at 1× and does not record it as a win over itself. We use that definition only when deciding which kernel-level candidates count as winners. The end-to-end score is the Table 2 row.

W3 / Q3. Harness, compute, and prompts

We agree that this comparison does not isolate the harness. Our wording was imprecise. KDA adds a GPT reviewer and, on the independent campaign, spends 1,751 agent-hours and 12.31B tokens against Claude Code's 1,171 hours and 4.35B tokens (Table 11). The revised PDF corrects that sentence. Claude Code and Codex are commercial coding agents run out of the box. KDA and AKO are the full systems, run in the configurations their authors published.

We did not impose a shared token cap. KDA and AKO run in phases, and stopping a run mid-phase would not be the system as designed by its authors. Repeating complete rounds until every agent had spent as much as KDA's one pass was beyond this campaign. Each agent therefore ran in its default configuration, and Table 11 reports the hours and tokens, including the share that is cached context. The per-agent loops are in Table 10. The task prompts are in the new Agent Prompts appendix. Claude Code and Codex share one prompt. KDA adds three phase prompts, then calls Humanize's default RLCR review loop (/humanize:start-rlcr-loop) with no reviewer prompt of our own. AKO uses the short FastKernels master and sub-agent wrappers in that appendix, not its contest template. The result we report is a comparison of the two default runs: at the kernel level Claude Code matches or beats KDA, while KDA deploys 11 of 11 models and Claude Code deploys 6 of 11. It does not separate KDA's harness from its extra reviewer or its larger token budget.

AuthorsOpenReview ↗

Response to Reviewer byLY (part 2)

W4 / Q5. Engine transfer and Figure 2

Each task's interface matches the corresponding production module, so a winner is installed by substitution rather than by rewriting the kernel. We checked that for two frozen winner sets inside official vLLM 0.26.0, KDA Independent on BGE-M3 and AKO Independent on Llama-3.1-8B, against stock vLLM on the same workloads. Both pass on every run (Llama calibrated score about 0.999, BGE minimum cosine 0.999). Llama is 1.01-1.02× over stock vLLM, in line with its FastKernels speedups of 1.01-1.04×. BGE is 1.11× in FastKernels. The candidates are called through a small adapter: vLLM keeps the weights, the scheduler, and the caches, and the Llama cache-store kernel takes explicit destination strides for vLLM's interleaved K/V layout. These timings are the winner sets inside stock vLLM against stock vLLM. Figure 2 instead compares the benchmark runtime with each family's production reference, and Table 2 compares agents with the unchanged benchmark runtime. The revised PDF no longer calls the other categories near parity. It says those category means are closer to 1×, with the per-architecture results in the appendix.

Q2. Drop budget and fault localization

Every agent used the same procedure and the same budget. Each model gets 16 attempts. An attempt is one fresh process, killed after 3 hours if it hangs, so a crash or a hang does not leave state for the next try. Probes use 8 requests per workload, and a subset is kept only after the full workloads pass. The all-winners outcome is reported separately. A model that never yields a deployable subset scores 0.

A crash that names a candidate file drops that kernel, the innermost candidate frame in the last traceback. If the traceback names none, including a hang, the active kernels are split in half and each half is re-run. The half that still fails is split again, and these probes count toward the 16. If neither half fails alone, the failure depends on kernels in both halves. The search stops, and the model is recorded as not converged rather than blamed on one kernel. A run that finishes with mean discrepancy above 0.5 is bisected the same way, so the search continues. The revised PDF states this budget and the stopping rule.

Q4. Compile replay and engine-shape drops

The kernel numbers in the paper are from eager execution. We re-benched the frozen composed and sequential candidates under torch.compile and CUDA-graph replay on the same cases. Of 952 operators that passed eager, 90 fail, 59 of them while the operator is being compiled or a CUDA graph is being captured. The order is unchanged: AKO and Claude Code stay ahead of KDA, and Dr. Kernel stays last. On the composed and sequential drops, this catches 11 of 55 compilation-conflict drops and 13 of 63 engine-shape drops. On the deployed model's shapes, it catches 10 of 83 compilation-conflict drops and 36 of 127 engine-shape drops. One compilation drop has no row in its model's bench. The rest still pass in isolation, 73 of those 83 and 91 of the 127. Compiling the original cases cannot catch a shape they do not contain. The revised PDF reports both checks.

I thank the authors for their detailed response and revisions to the manuscript. The revisions address my main concerns about the correctness threshold, the MacroEval score, and transfer to vLLM. I also found the clarification about comparing the agents in their default configurations helpful.

Reviewer_nBrfOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

FastKernels introduces a benchmark for evaluating GPU kernel generation in production inference settings. It contains 384 tasks derived from 47 model architectures, organized into a compositional hierarchy from primitive operators to full models. The benchmark combines production-compatible interfaces, expert production baselines, and end-to-end evaluation. Experiments with five agents reveal substantial discrepancies between isolated kernel speedups and practical deployment outcomes.

Critical strengths
  1. Important and timely contribution. Evaluating whether generated kernels remain correct and beneficial inside real inference systems is essential. The paper moves evaluation beyond isolated operator speedups toward practical usability.

  2. Thoughtful benchmark design. The compositional hierarchy, production-compatible interfaces, and comparisons against production kernels make the benchmark more representative of deployment challenges.

  3. Substantial and informative evaluation. The agent experiments expose composition failures, integration issues, and ranking reversals that isolated evaluations miss. These findings provide useful guidance for future kernel-generation research.

Critical weaknesses
  1. Limited case sampling may miss important serving behavior. The default evaluation reduces a large number of captured cases to at most five per operator. Selecting cases by size and frequency is practical, but may omit uncommon shapes or execution states that trigger failures. The reported crashes on engine shapes absent from kernel-level testing reinforce this concern. A sensitivity analysis of the case-selection budget would strengthen the evaluation.

  2. Realistic input coverage remains incomplete at the kernel level. Although end-to-end evaluation uses real requests, kernel-level inputs are largely synthesized from captured shapes, with normally distributed values, all-one masks, and randomly initialized weights. Matching shapes and layouts does not preserve the value distributions and data-dependent behavior encountered in deployment, as the paper’s own MoE analysis demonstrates. Golden-input replay is a valuable mechanism, but its use in the reported experiments is unclear, particularly given the limitation stating that kernel-level results use synthesized inputs. Deployment-oriented tasks should include captured activations and representative masks, routing patterns, and other value-dependent inputs where applicable.

Questions for authors
  1. Which reported kernel-level results use golden-input replay? Can you provide captured real inputs for all deployment-relevant, data-dependent tasks?

  2. How sensitive are correctness, speedups, and agent rankings to the five-case-per-operator limit? Would additional or held-out cases expose more failures?

  3. How representative is the 11-model default set, and do the main ranking conclusions hold on another GPU architecture?

1 comment
AuthorsOpenReview ↗

Response to Reviewer nBrf

Thank you for your careful review and detailed feedback, our response is below.

W1 / Q2. Five-case limit

The five-case limit is the kernel-level screen. We re-benched the frozen candidates from all five agents and three modes at 5, 10, and 25 cases per operator (7,136, 11,896, and 25,070 scenarios). Correct operators go from 1,519 to 1,481. For operators that pass at both 5 and 25 cases, the median ratio of the 25-case speedup to the 5-case speedup is 1.00×. Independent and sequential order is the same at every budget: AKO, Claude Code, Codex, KDA, Dr. Kernel, with Claude Code ahead of KDA at every level. The only movement is a near-tie in composed mode, AKO 3.19× and Claude Code 3.18× at 5 cases, with Claude Code slightly ahead at 10 and 25. The added failures are shared across agents, not a new ranking. The deployment results come from full-model runs, not from this screen. The 25-case re-score took about 3 hours. The revised PDF reports this check in the appendix.

W2 / Q1. Synthesized inputs and golden replay

We support golden-input replay, but we did not enable it for the kernel-tier numbers in the paper. Those runs synthesize values from the captured shapes, so cases that share a shape can be deduplicated. Replay cannot combine them: the same shape can take a different path, as the MoE figure shows for expert routing, so each captured tensor stays its own case. That multiplies the cases on a 133-operator comparison, since an L2 or L3 task that calls one of these operators needs its own golden inputs as well. We kept the kernel tier lightweight by using synthetic inputs and leaving golden replay optional. The end-to-end tier runs the real requests. The appendix now lists the operators with data-dependent control flow and the L2-L3 tasks that call one, with how many input tensors select the work on each call. We will release the golden inputs for those tasks on the tested workloads upon publication/de-anonymization. We revised the PDF to clarify that the reported kernel results used synthesized inputs.

Q3. Default set and a second GPU

The default set is the cross-category set the leaderboard averages over: 11 models and 48 workloads, at least one model in each of the eight categories, tensor parallelism at 1, 2, and 4, BF16, FP16, FP8, and MXFP4, and a throughput and a latency workload for every model (133 L1-L3 tasks). The full suite, 47 models and 184 workloads, is where the runtime is checked against each family's production reference.

That check is already on a second GPU. On the 46 architectures that run on both, throughput is 1.19× on 8×H200 and 1.22× on 8×B200, and latency is 1.15× on both. GLM-5.2 is the one model omitted on H200, because its NVFP4 kernels require Blackwell. The agent order is a separate result. Kernel speed and what an agent is able to write both depend on the GPU, so that order can change. The paper reports it for the B200 campaign.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.