- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This work introduces a new benchmark called KernelAudit for assessing the performance and reliability of LLM-generated GPU kernels beyond conventional correctness and speedup measurements. Its main contributions are a benchmark of 230 real-world kernel tasks across 14 domains, a five-level reliability verifier, and a secure evaluation framework that supports audit feedback during kernel generation. Evaluations across seven frontier LLMs show that while most generated kernels pass traditional correctness checks, only a few pass all reliability audits. Furthermore, providing audit feedback increases the reliability pass rate significantly from 42% to 80% on a 50-task subset, without significant speed degradation. These results reveal substantial reliability gaps in existing kernel evaluations and demonstrate the benefits of incorporating reliability checks into kernel generation.
Critical strengths
This work addresses a critical gap between benchmark performance and practical utility. Existing GPU kernel benchmarks often incentivize agents to maximize benchmark scores without ensuring that the reported speedups translate into reliable, usable implementations. This paper tackles this important problem by evaluating generated kernels against real-world requirements rather than relying solely on fixed correctness tests. This a significant strength because it addresses a fundamental limitation in how progress in automated kernel generation is currently measured.
Critical weaknesses
While the paper motivates KernelAudit through practical deployment reliability, its evaluation defines reliability primarily through behavioral equivalence to an expert-written kernel rather than explicit application-level requirements. This distinction is important because real deployments often permit specialized kernels with restricted input domains or reduced numerical precision, provided these choices satisfy the application's input and accuracy contracts. For instance, the L2 verifier can penalize certain mathematical intrinsics based on source-code patterns without demonstrating an unacceptable numerical error, while L3 penalizes kernels that reject alternative input layouts or dtypes supported by the expert reference. Such behaviors are not necessarily deployment failures. Conversely, passing these checks does not establish correctness or performance when integrated into the original application. Since the paper's central findings and model rankings depend heavily on these reliability criteria, the lack of validation against actual deployment requirements raises concerns about whether the reported reliability gap accurately reflects practical kernel usability.
Questions for authors
The paper constructs its five-level verifier from failure patterns observed in 505 generated kernels. However, it remains unclear how comprehensively these checks capture reliability issues beyond the cases used to develop them. Could the authors provide an independent assessment of kernels that pass all five levels, for example through manual inspection or additional adversarial testing of a representative sample, to determine whether significant failure modes remain undetected? This would help establish whether passing KernelAudit provides meaningful confidence in kernel reliability, rather than merely indicating the absence of known failure patterns.
3 comments
2. On an independent assessment of kernels that pass all five levels (Question)
Thank you for the suggestion. Your concern is whether the checks of the verifier, which come from failure modes observed during construction, can detect reliability issues beyond these modes. In other words, the question is whether passing KernelAudit provides meaningful confidence in kernel reliability, rather than merely indicating the absence of known failure patterns. In the original manuscript, the 22 sub-levels of the verifier were determined during construction, and the decision rules were fixed before the formal evaluation (Appendix B). To answer this question directly, we followed your suggestion and manually inspected kernels that pass all five levels, on tasks not used during construction. To also evaluate false positives, we inspected kernels that the verifier flags as well.
The results show that the labelers found no failure mode outside the 22 sub-levels. At the level of failure categories, the verifier has no false negative. Of the 20 kernels that pass all five levels, 15 are confirmed by the labels to have no violation. The violations of the other 5 all belong to existing sub-levels. They were missed because the probes of these sub-levels did not construct the specific inputs that trigger them, not because the verifier ignores these categories. We describe the sampling, labeling, and results below.
Sampling. The kernels used to refine the verifier come from 62 of the 230 tasks. We excluded these tasks. On the remaining 168 tasks, the seven models evaluated in the paper produced 1,152 kernels that pass the correctness check. To avoid concentrating the sample in a few levels, we grouped these kernels by the lowest audit level at which the verifier reports a violation. L0 is the most severe level. This gives six groups. One group passes all five levels, and the lowest violated level of the other five groups is L0, L1, L2, L3, or L4. Manual labeling is costly, so we sampled 50 kernels in total. We drew 20 kernels from the group that passes all five levels and 6 kernels from each of the other five groups. All kernels were drawn at random within their group, and the sample covers all seven models. The final 50 kernels come from 43 tasks.
Labeling. We invited experienced CUDA engineers to label the 50 kernels. The labelers do not know how the verifier is implemented, and they did not see its verdicts during labeling. For each kernel, we provided the generated code, the PyTorch reference, the reliable kernel of the task, and the definitions of the 22 sub-levels of L0–L4 (Appendix B). Sub-level x.y denotes the y-th sub-level of Lx. The labeling criteria match those of KernelAudit. The labelers judged whether the kernel violates each of the 22 sub-levels. They could also report any other problem outside the 22 sub-levels.
Results. A kernel may violate several sub-levels at once, so we compare the verifier with the labels in two ways. When counting violation entries, each (kernel, sub-level) pair is one entry. The labels contain 75 violations over the 50 kernels, and the verifier reports 65 (Table A). When counting kernels, we only consider whether a kernel passes all five levels (Table B).
Table A. Verifier vs. expert labels at each audit level. Each (kernel, sub-level) pair is one entry. Precision = TP / (TP + FP). Recall = TP / (TP + FN). New failure modes = problems reported by the labelers outside the 22 sub-levels.
| Audit level | TP | FP | FN | Precision | Recall | New failure modes |
|---|---|---|---|---|---|---|
| L0 | 7 | 1 | 0 | 87.5% | 100% | 0 |
| L1 | 8 | 0 | 2 | 100% | 80.0% | 0 |
| L2 | 14 | 0 | 0 | 100% | 100% | 0 |
| L3 | 25 | 0 | 9 | 100% | 73.5% | 0 |
| L4 | 10 | 0 | 0 | 100% | 100% | 0 |
| All | 64 | 1 | 11 | 98.5% | 85.3% | 0 |
Table B. Verifier vs. expert labels per kernel.
| Expert label: no violation | Expert label: ≥1 violation | Total | |
|---|---|---|---|
| Verifier: passes all five levels | 15 (TN) | 5 (FN) | 20 |
| Verifier: ≥1 violation | 0 (FP) | 30 (TP) | 30 |
| Total | 15 | 35 | 50 |
| FPR / FNR | FP/(FP+TN) = 0% | FN/(FN+TP) = 14.3% |
All 11 false negatives of the verifier fall within existing sub-levels (last column of Table A). The verifier has checks for these problems, but its probes did not construct the specific inputs that trigger them.
Of the 20 kernels that pass all five levels, 5 have violations. All of them fall into this case.
- S036 (DeepSpeed softmax), S069 (lmdeploy ban bad words), and S074 (Chrono fluid dynamics), sub-level 3.2 (non-contiguous layout). These 3 kernels read inputs with 16-byte vectorized instructions and assume that the start address of each input is 16-byte aligned. Suppose an input is a contiguous view cut from a larger buffer, with its start address offset by 1 or 2 elements. The kernel then raises a misaligned address error, while the reliable kernel gives correct results. The probes of 3.2 construct non-contiguous views. These kernels first copy a non-contiguous input into a contiguous tensor, so they pass. The probes do not include contiguous views with an unaligned start address.
- S006 (PyTorch3D face areas and normals), sub-level 1.3 (semantic tampering). When one vertex coordinate is NaN, the reference and the reliable kernel output NaN for all normal components of the adjacent faces. During normalization, this kernel replaces the NaN norm with a small constant. Some of the components therefore become finite values, up to about 2.6e6. The check of 1.3 compares the number of NaN/Inf values in the output. It reports a violation only when this number falls below one tenth of that of the reference. The kernel keeps 18 of 24, so it was not detected.
- S084 (OpenCV DNN concat), sub-level 3.3 (shape specialization). The kernel supports at most 32 input tensors and raises an error beyond 32. The reference and the reliable kernel both handle such inputs correctly. The probes of 3.3 vary the dimensions of the input tensors but not the number of input tensors.
The other 6 false negatives occur on kernels that the verifier already flags, and they have the same causes. Three are the same unaligned contiguous views (3.2). One is the same partial replacement of NaN/Inf with finite values (1.3). One arises because the probes of 3.3 do not vary the constructor argument dim. One arises because the probes of 3.1 (dtype lock-in) do not cover int32 indices. These false negatives do not change the overall verdict on the corresponding kernels. All of the above false negatives can be detected by adding the corresponding cases to the probes of existing sub-levels. No new category is needed, and such false negatives will decrease as the probe budget grows.
Of the 65 entries reported by the verifier, only one is a false positive. It is at sub-level 0.3 (precomputation outside timing). One probe of 0.3 checks whether a compute kernel runs only on the first call of a process. S082 (Kaolin sided distance) runs reset_kernel once on its first call to set its cached workspace to an initial value, so the verifier reported 0.3. This step only initializes the workspace and contains none of the computation of the reference. Every later call performs the full distance computation and resets the workspace after reading out the result. The first call and a steady call also take similar time (0.16 ms and 0.14 ms). The labelers therefore judged that the kernel does not violate 0.3. Although this verdict is a false positive at 0.3, the kernel has other confirmed violations, such as a race on the same workspace under concurrent calls (4.3). The per-kernel FPR therefore remains 0.
The full labeling results are given in Appendix L of the revision.
Thank you for the positive assessment and the constructive comments. You point out the gap between benchmark performance and practical utility in existing kernel evaluations. This is also a topic we care about and try to explore and address in our work. We respond to two points below. They are the relation between the reliability criteria and deployment requirements, and an independent assessment of kernels that pass all five levels.
1. On the reliability criteria and deployment requirements (Weakness)
Thank you for the suggestion. Your concern is that the reliability criteria of KernelAudit may be stricter than real deployments require. Real deployments often permit specialized kernels with restricted input domains or reduced numerical precision, provided these choices satisfy the input and accuracy contracts of the application. Some behaviors that KernelAudit penalizes may therefore not be deployment failures. We agree that reliability criteria are not fully equivalent to the deployment requirements of a specific application. KernelAudit was designed with the same concern in mind. It probes strictly, but it calibrates each verdict against the expert implementation used in real deployments. This keeps the evaluation as close to deployment requirements as possible.
Concretely, the verdict has two steps. In the first step, the audit probes the generated kernel under strict reliability requirements. Examples are other dtypes and memory layouts that the reference supports, and the precision in specific numerical ranges. In the second step, the audit decides whether a probe result counts as a violation. It probes the reliable kernel of the task with the same inputs. The reliable kernel is the expert implementation that the original application uses in deployment. If the reliable kernel also fails this probe, real deployment does not need this requirement, and the same problem in the generated kernel does not count (§2, Appendix H). Take the two cases you mention as examples. Suppose the expert implementation supports only specific input layouts or dtypes, or uses the same low-precision approximation. A generated kernel that makes the same choice is then not penalized. Every violation that counts is therefore a requirement that the expert implementation meets in real deployment but the generated kernel does not.
We did not adopt two other approaches. One is to state in the task prompt which requirements may be relaxed. The other is to relax requirements uniformly across all tasks in the audit. Relaxations in real deployments are usually limited to specific intermediate steps. For example, one step may use lower precision, or one input may always be contiguous in the original application. If requirements were relaxed uniformly, for example with a looser tolerance or a wider input domain for every task, models could easily exploit these loose conditions to hack the evaluation. Appendix K records such cases, for example using a loose tolerance to skip computation that should be performed. If the task prompt listed the allowed relaxations for each task, it would reveal the interface and the solution approach of the expert implementation to the model. Using the expert implementation as a post-hoc reference is therefore a trade-off between high reliability criteria and deployment requirements. It keeps the requirements on generated kernels strict. It also does not count the choices that the expert implementation already makes in deployment as violations.
A more direct validation would integrate the generated kernel into the original application and check end-to-end correctness and performance. However, the upstream and downstream code of each kernel in its original application is highly complex. Building such integration tests for all 230 tasks would be too costly. KernelAudit uses the expert implementation deployed in the original application as its reference. It can be seen as a simplified version of such integration validation. In future work, we hope to add real deployment cases and further check how well audit results agree with deployment outcomes.