- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces FastKernels, a benchmark for evaluating the GPU kernel generation capabilities of large language models (LLMs). The authors observe that kernels generated on other benchmarks may fail to preserve correctness or improve actual performance when integrated into full-model inference pipelines. The benchmark comprises 384 production-aligned tasks derived from 47 architectures across eight categories. These tasks are organized into an L1-L4 hierarchy covering primitive operators, fused operators, network layers, and full models. The evaluation combines kernel-level tests on specific input configurations with end-to-end execution and supports integrated evaluation of calls across task levels. MacroEval measures calibrated correctness, coverage, and speedups in throughput and latency. The study evaluates five agents on an 11-model default subset. The authors further report composition failures during integration, substantially smaller end-to-end gains than those observed in isolated kernel tests, and differences between kernel-level and end-to-end rankings.
Critical strengths
- Strong motivation and practical value. FastKernels addresses an important gap between isolated kernel speedups and improvements that remain correct and useful in full-model inference. This provides a practically meaningful evaluation target for GPU-kernel generation and optimization agents. The L1-L4 hierarchy captures dependencies across task levels rather than evaluating them independently. It enables integrated evaluation of generated kernels and makes their interactions, composition failures, and performance effects directly measurable.
- Extensive and informative empirical analysis. The authors conduct fine-grained experiments with five agents on the production-aligned benchmark, comparing independent, composed, and sequential modes alongside end-to-end evaluation. The detailed results on correctness, speedups, and integration failures provide valuable empirical reference points for developing future kernel-generation and optimization agents
Critical weaknesses
- Ranking robustness. MacroEval averages speedups only over valid tasks and families. If a slow but correct task becomes incorrect, it is removed from the speedup average. This can increase the overall score despite the correctness and coverage penalties (Section 5, Equation 5). The impact of this scoring behavior on the reported rankings remains unclear
Questions for authors
1.Does MacroEval include safeguards to prevent the overall score from increasing when a slow but correct task becomes incorrect and is excluded from speedup aggregation? 2.Would the reported agent rankings remain stable under a scoring rule that prevents correctness degradation alone from increasing the score?
2 comments
Thank you for your response and clarification. I will maintain my original score and supported position as discussed.
Response to Reviewer MY8r
Thank you for your careful review and detailed feedback, our response is below.
W1 / Q1 / Q2. A slow correct model becoming invalid
We thank the reviewer for this observation. In the submitted version, a slow but correct model could raise the ranking by dropping out of a geometric mean taken only over valid models. The correctness and coverage factors did not always offset that increase.
We have revised our MacroEval metric to close this gap. The revised PDF reflects the new ranking score. Every evaluated model stays in a fixed average. A model contributes its speedup s_i when the run executes and C_i ≥ τ_i, and zero otherwise:
s_i, & \text{if the run executes and } C_i\ge\tau_i,\\ 0, & \text{otherwise,} \end{cases} \qquad \mathrm{Score} =\frac{1}{|\mathcal{F}|}\sum_{f\in\mathcal{F}}\frac{1}{|B_f|}\sum_{i\in B_f}q_i.$$ Making any valid model invalid replaces a positive term with zero, so the score decreases, including when that model is the last valid one in its family. S_macro, C_macro, and coverage stay in the table as separate diagnostics. We re-scored all ten agent entries from the saved end-to-end results. KDA Independent remains first (1.138). Dr. Kernel Independent was 4th of those ten and is 7th of the eleven rows, below the unchanged baseline. Claude Code Independent, which matches or beats KDA at the kernel level, scores 0.574. The submitted product put that gap at 3× (1.11 versus 0.35). Under this score it is 2× (1.138 versus 0.574), which is what the abstract, introduction, and conclusion now say. The 1.25× in those sentences is the end-to-end speedup, separate from this ranking score. Section 5 defines the score, and Table 2 reports the revised leaderboard.