- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a mutation-testing framework to evaluate the correctness checkers used in GPU kernel benchmarks such as KernelBench. The authors introduce 124 mutation rules to inject faults into CUDA implementations of 189 operator-level problems and 48 architectures. They use a validity gate to filter out test inputs that reject the original implementations and only include mutants detected by at least one valid input in the main mutation-score calculation. The experiments show that the official KernelBench checker misses 16.9% of detectable faults, especially precision and synchronization errors. The paper also evaluates existing checker improvements and uses set cover to select more effective test inputs.
Critical strengths
-
The paper provides a systematic way to measure the effectiveness of GPU kernel correctness checkers. This paper introduces mutation testing to measure which types of faults a checker can detect and which ones it misses. The validity gate and kill-witness mechanism are useful adaptations to GPU programs. This framework provides a useful tool for understanding and improving GPU kernel benchmarks.
-
The experimental findings are detailed and practically useful. The paper shows that different types of faults have very different detection rates. For example, the official checker misses 78.6% of precision faults but only 8.7% of arithmetic faults. These findings provide useful guidance for designing more effective correctness tests.
Critical weaknesses
-
Insufficient validation on real GPU kernel bugs and potential bias in the evaluation. The main experiments rely on artificially injected faults in the authors' own CUDA implementations. Although mutation testing provides a controlled way to measure checker effectiveness, these faults may not fully represent the errors produced by real LLM-generated or highly optimized GPU kernels. The paper includes a small experiment with 60 LLM-generated kernels in Appendix D.6, but it does not directly demonstrate whether the proposed tests can detect real bugs missed by the official checker. Another concern is how the evaluation set is constructed. Although Table 3 evaluates all methods on the same set of witnessed mutants, this set is constructed using the official inputs and the authors' own targeted and LLM-generated tests. As a result, the evaluation may favor the proposed method because its own test inputs help determine which faults are included in the comparison. Although the authors also report held-out results, the split is based on mutant IDs, meaning similar faults from the same code locations may appear in both sets. Therefore, it remains unclear how well the reported improvements generalize to real kernel bugs or different types of unseen faults.
-
Insufficient validation of the validity gate across different correct CUDA implementations. The paper uses a validity gate to prevent test inputs from incorrectly rejecting correct kernels. Specifically, a test input is considered valid if the authors' original CUDA implementation passes the comparison with the PyTorch reference. However, different CUDA implementations may use different accumulation orders, floating-point instructions, or optimization strategies, leading to legitimate numerical differences. An input that passes the validity gate for one implementation may therefore cause another otherwise correct implementation to fail the check. The zero false rejections reported in Table 3 are measured using the same substrate-based validity checks used to select the inputs. Therefore, this result does not establish whether the proposed tests would incorrectly reject other independently validated correct implementations. Since the proposed method aims to improve checker effectiveness without rejecting correct programs, additional validation across different optimized implementations is needed to support its practical reliability.
Questions for authors
Could the authors evaluate the proposed testing method on real LLM-generated or leaderboard CUDA kernels? In particular, it would be useful to compare how many real bugs are detected by the official KernelBench checker, KernelBench-Verified, and the proposed test suites. This would help establish whether the improvements observed on synthetic mutants transfer to real kernel errors.
Could the authors evaluate the proposed test inputs on multiple independently implemented and optimized CUDA kernels that have been validated as correct? For example, kernels using different reduction strategies, Tensor Cores, or floating-point instructions could help assess whether the validity gate generalizes beyond the authors' original implementations. Reporting false rejection rates on these kernels would strengthen the practical reliability claims.
4 comments
Response to Q1 and Q2
Q1. Real bugs: Official vs KernelBench-Verified vs ours.
(a) Three-way comparison on new LLM kernels.
We generated 397 new kernels with Codex and Claude Opus 5.5, each in an isolated workspace with no access to our mutation rules, tests or reference kernels. 137 kernels have confirmed defects (criterion below):
| Checker | Defective kernels detected (of 137) |
|---|---|
| Official KernelBench (5 random inputs) | 105 |
| KernelBench-Verified style (4 scaled inputs, ) | 106 |
| Ours (≤2 selected inputs) | 113 |
| Official + Ours | 133 |
25 bugs are caught only by our suite, on valid inputs whose sizes differ from the benchmark's fixed shape.
(b) The 25 ours-only bugs are genuine.
On the detecting input:
- the input is valid for the task. It keeps every Model constructor parameter (channels, kernel size, feature count) and every task precondition fixed, and varies only input sizes or values. Our reference CUDA kernel passes on it.
- the kernel genuinely fails. It crashes, returns a wrong shape, or, for wrong values, disagrees with an FP64 evaluation while PyTorch with TF32 enabled passes. A precision-mode effect is therefore ruled out.
- the failure reproduces in 3 fresh processes.
9 of the 25 crash, 2 return a wrong shape and 14 return FP64-confirmed wrong values. For example, 119relu drops the last elements, and a GEMM kernel launches batch // 64 row blocks, dropping any batch tail. Such a kernel implements the operator only for the checker's sampled shape, and this oracle gap is exactly what our framework measures.
(c) Public leaderboard kernels.
We ran our frozen suites, with the same defect criterion, on 492 published kernels at KernelBench v0.1 sizes. Our ≤2-input suite finds 52 defective kernels, 23 of which pass every official and KernelBench-Verified input:
| Kernel collection | Kernels run | Found by ours | Of these, missed by Official and KBV |
|---|---|---|---|
| CudaForge | 94 | 12 | 6 |
| CUDA-L1 | 115 | 7 | 2 |
| Sakana AI CUDA Engineer | 198 | 18 | 5 |
| AdaExplore | 85 | 15 | 10 |
For example:
-
a CUDA-L1 kernel (
L2p80) skips the computation, since the operator's output is all zeros on valid inputs. It returns a zero tensor cached from the first call (batch 1024), so a 13-row batch gets a output:batch_size = x.shape[0] self._zero_tensor = self._create_optimized_zero_tensor( batch_size, x.dtype, x.device, x.requires_grad ) self.forward = self._zero_tensor self._optimized = True ... def __call__(self, x): ... if self._optimized: return self._zero_tensor -
an AdaExplore kernel (
L2p70, GEMM–sigmoid–scaling–residual) casts the FP32 input and weight to fp16 before the GEMM. Any input value above the fp16 maximum (65,504) becomesinf, so a single large but valid input value makes every outputinf, while the FP32 reference is finite:self.register_buffer("weight_t_fp16", self.linear.weight.data.t().half(), persistent=False) ... x_fp16 = x.half().contiguous() ... return triton_gemm_sigmoid_residual(x_fp16, self.weight_t_fp16, self.bias_fp32, self.scaling_factor) -
a CudaForge GELU computes as , which overflows to NaN for large activations, and a CudaForge
argmaxnever selects a NaN; -
several Sakana GEMM and transposed-convolution kernels return
infor values up to on batch sizes such as 3, 9 or 13.
These third-party kernels played no role in selecting our suites.
(d) Evaluation-set bias and the ID split.
The real-kernel populations were fixed before any checker ran and defects are confirmed against FP64. For "similar faults from the same code locations may appear in both sets", holding out by source-edit site keeps detection at 92.9% (+11.2 pp), and holding out whole rules gives 91.0% (+10.4 pp).
Q2. Correct implementations beyond our substrate.
- Independent kernels. Our inputs produce 0 false rejections on 13 implementations of 6 tasks (different reduction trees, warp shuffles, FMA contraction, vectorized loads, shared-memory tiling; 132 implementation–input pairs).
- Defect-free LLM kernels. On the 232 defect-free LLM kernels, written independently of our substrates, our inputs produced 0 wrong-value or runtime failures.
- Tensor cores. Our inputs accept TF32 Tensor-Core PyTorch on all 70 convolution tasks. Wherever they reject it (9 of 40 GEMM tasks), TF32 itself deviates from FP64 by more than , so these are real precision losses rather than false alarms; at KernelBench's current default, even the official random inputs reject TF32 on 59% of inputs. We will state the FP32 contract and flag TF32-sensitive inputs.
I appreciate the authors' detailed rebuttal and additional experiments. The response addresses my main concerns, and I will maintain my positive rating.
Thank you very much for your time, constructive feedback and continued support! We will make sure all the new real-kernel and cross-implementation results are thoroughly integrated into the revised manuscript.
Response to Reviewer V6Xp
We thank the reviewer for recognizing the validity gate and kill-witness mechanism as useful adaptations of mutation testing to GPU programs, and the fault-family findings (78.6% of precision versus 8.7% of arithmetic faults missed) as practical guidance for test design. In response to your concerns, we:
We will add these results to the revision and hope they address your concerns.