acceptodds
Under review as a conference paper at ICLR 2027

KernelAudit: Can LLM-Generated GPU Kernels Be Both Reliable and Competitive on Real-World Workloads?

Abstract

Existing evaluations of LLM-generated GPU kernels report speedups over PyTorch and accept a kernel once it passes a functional check with a fixed error tolerance. Kernels that exploit the measurement or ignore real-world requirements still pass this check, so the reported speedups overstate what the kernels deliver. We present KernelAudit, a benchmark that asks whether kernels generated by frontier LLMs can be both reliable and competitive on real-world workloads. First, after correctness and validity audits, we construct 230 tasks across 14 domains from expert-written, community-maintained kernels collected from 69 high-star open-source repositories. Each expert kernel serves as both the speed baseline and the reliability baseline. Second, from an analysis of 505 generated kernels that passed functional checks, we build a verifier that audits every kernel at five levels, ordered by how strong a condition is needed to expose a defect. Third, we build a sealed evaluation environment that withholds all evaluation assets from the agent and times generated and expert kernels under one protocol on an exclusive GPU. Across seven frontier LLMs on all 230 tasks, only 25.3% of the generated kernels pass all five audit levels, and no model exceeds 40.9%. Giving the agent a fast audit tool built from the verifier during generation eliminates measurement bypass and algorithmic mismatch, raises the fraction of kernels that pass all five levels from 42% under an empty audit to 80%, and leaves speed unchanged. Agents optimize what is measured, and KernelAudit makes reliability measurable. Code and data are available at https://anonymous.4open.science/r/kernelaudit-BD8D.

Primary area:
datasets and benchmarks
Keywords:
benchmark, large language models, code generation, GPU kernel, CUDA, reliability, performance evaluation, code agent

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hKNrOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work introduces a new benchmark called KernelAudit for assessing the performance and reliability of LLM-generated GPU kernels beyond conventional correctness and speedup measurements. Its main contributions are a benchmark of 230 real-world kernel tasks across 14 domains, a five-level reliability verifier, and a secure evaluation framework that supports audit feedback during kernel generation. Evaluations across seven frontier LLMs show that while most generated kernels pass traditional correctness checks, only a few pass all reliability audits. Furthermore, providing audit feedback increases the reliability pass rate significantly from 42% to 80% on a 50-task subset, without significant speed degradation. These results reveal substantial reliability gaps in existing kernel evaluations and demonstrate the benefits of incorporating reliability checks into kernel generation.

Critical strengths

This work addresses a critical gap between benchmark performance and practical utility. Existing GPU kernel benchmarks often incentivize agents to maximize benchmark scores without ensuring that the reported speedups translate into reliable, usable implementations. This paper tackles this important problem by evaluating generated kernels against real-world requirements rather than relying solely on fixed correctness tests. This a significant strength because it addresses a fundamental limitation in how progress in automated kernel generation is currently measured.

Critical weaknesses

While the paper motivates KernelAudit through practical deployment reliability, its evaluation defines reliability primarily through behavioral equivalence to an expert-written kernel rather than explicit application-level requirements. This distinction is important because real deployments often permit specialized kernels with restricted input domains or reduced numerical precision, provided these choices satisfy the application's input and accuracy contracts. For instance, the L2 verifier can penalize certain mathematical intrinsics based on source-code patterns without demonstrating an unacceptable numerical error, while L3 penalizes kernels that reject alternative input layouts or dtypes supported by the expert reference. Such behaviors are not necessarily deployment failures. Conversely, passing these checks does not establish correctness or performance when integrated into the original application. Since the paper's central findings and model rankings depend heavily on these reliability criteria, the lack of validation against actual deployment requirements raises concerns about whether the reported reliability gap accurately reflects practical kernel usability.

Questions for authors

The paper constructs its five-level verifier from failure patterns observed in 505 generated kernels. However, it remains unclear how comprehensively these checks capture reliability issues beyond the cases used to develop them. Could the authors provide an independent assessment of kernels that pass all five levels, for example through manual inspection or additional adversarial testing of a representative sample, to determine whether significant failure modes remain undetected? This would help establish whether passing KernelAudit provides meaningful confidence in kernel reliability, rather than merely indicating the absence of known failure patterns.

3 comments

Thank you for the positive assessment and the constructive comments. You point out the gap between benchmark performance and practical utility in existing kernel evaluations. This is also a topic we care about and try to explore and address in our work. We respond to two points below. They are the relation between the reliability criteria and deployment requirements, and an independent assessment of kernels that pass all five levels.

1. On the reliability criteria and deployment requirements (Weakness)

Thank you for the suggestion. Your concern is that the reliability criteria of KernelAudit may be stricter than real deployments require. Real deployments often permit specialized kernels with restricted input domains or reduced numerical precision, provided these choices satisfy the input and accuracy contracts of the application. Some behaviors that KernelAudit penalizes may therefore not be deployment failures. We agree that reliability criteria are not fully equivalent to the deployment requirements of a specific application. KernelAudit was designed with the same concern in mind. It probes strictly, but it calibrates each verdict against the expert implementation used in real deployments. This keeps the evaluation as close to deployment requirements as possible.

Concretely, the verdict has two steps. In the first step, the audit probes the generated kernel under strict reliability requirements. Examples are other dtypes and memory layouts that the reference supports, and the precision in specific numerical ranges. In the second step, the audit decides whether a probe result counts as a violation. It probes the reliable kernel of the task with the same inputs. The reliable kernel is the expert implementation that the original application uses in deployment. If the reliable kernel also fails this probe, real deployment does not need this requirement, and the same problem in the generated kernel does not count (§2, Appendix H). Take the two cases you mention as examples. Suppose the expert implementation supports only specific input layouts or dtypes, or uses the same low-precision approximation. A generated kernel that makes the same choice is then not penalized. Every violation that counts is therefore a requirement that the expert implementation meets in real deployment but the generated kernel does not.

We did not adopt two other approaches. One is to state in the task prompt which requirements may be relaxed. The other is to relax requirements uniformly across all tasks in the audit. Relaxations in real deployments are usually limited to specific intermediate steps. For example, one step may use lower precision, or one input may always be contiguous in the original application. If requirements were relaxed uniformly, for example with a looser tolerance or a wider input domain for every task, models could easily exploit these loose conditions to hack the evaluation. Appendix K records such cases, for example using a loose tolerance to skip computation that should be performed. If the task prompt listed the allowed relaxations for each task, it would reveal the interface and the solution approach of the expert implementation to the model. Using the expert implementation as a post-hoc reference is therefore a trade-off between high reliability criteria and deployment requirements. It keeps the requirements on generated kernels strict. It also does not count the choices that the expert implementation already makes in deployment as violations.

A more direct validation would integrate the generated kernel into the original application and check end-to-end correctness and performance. However, the upstream and downstream code of each kernel in its original application is highly complex. Building such integration tests for all 230 tasks would be too costly. KernelAudit uses the expert implementation deployed in the original application as its reference. It can be seen as a simplified version of such integration validation. In future work, we hope to add real deployment cases and further check how well audit results agree with deployment outcomes.

2. On an independent assessment of kernels that pass all five levels (Question)

Thank you for the suggestion. Your concern is whether the checks of the verifier, which come from failure modes observed during construction, can detect reliability issues beyond these modes. In other words, the question is whether passing KernelAudit provides meaningful confidence in kernel reliability, rather than merely indicating the absence of known failure patterns. In the original manuscript, the 22 sub-levels of the verifier were determined during construction, and the decision rules were fixed before the formal evaluation (Appendix B). To answer this question directly, we followed your suggestion and manually inspected kernels that pass all five levels, on tasks not used during construction. To also evaluate false positives, we inspected kernels that the verifier flags as well.

The results show that the labelers found no failure mode outside the 22 sub-levels. At the level of failure categories, the verifier has no false negative. Of the 20 kernels that pass all five levels, 15 are confirmed by the labels to have no violation. The violations of the other 5 all belong to existing sub-levels. They were missed because the probes of these sub-levels did not construct the specific inputs that trigger them, not because the verifier ignores these categories. We describe the sampling, labeling, and results below.

Sampling. The kernels used to refine the verifier come from 62 of the 230 tasks. We excluded these tasks. On the remaining 168 tasks, the seven models evaluated in the paper produced 1,152 kernels that pass the correctness check. To avoid concentrating the sample in a few levels, we grouped these kernels by the lowest audit level at which the verifier reports a violation. L0 is the most severe level. This gives six groups. One group passes all five levels, and the lowest violated level of the other five groups is L0, L1, L2, L3, or L4. Manual labeling is costly, so we sampled 50 kernels in total. We drew 20 kernels from the group that passes all five levels and 6 kernels from each of the other five groups. All kernels were drawn at random within their group, and the sample covers all seven models. The final 50 kernels come from 43 tasks.

Labeling. We invited experienced CUDA engineers to label the 50 kernels. The labelers do not know how the verifier is implemented, and they did not see its verdicts during labeling. For each kernel, we provided the generated code, the PyTorch reference, the reliable kernel of the task, and the definitions of the 22 sub-levels of L0–L4 (Appendix B). Sub-level x.y denotes the y-th sub-level of Lx. The labeling criteria match those of KernelAudit. The labelers judged whether the kernel violates each of the 22 sub-levels. They could also report any other problem outside the 22 sub-levels.

Results. A kernel may violate several sub-levels at once, so we compare the verifier with the labels in two ways. When counting violation entries, each (kernel, sub-level) pair is one entry. The labels contain 75 violations over the 50 kernels, and the verifier reports 65 (Table A). When counting kernels, we only consider whether a kernel passes all five levels (Table B).

Table A. Verifier vs. expert labels at each audit level. Each (kernel, sub-level) pair is one entry. Precision = TP / (TP + FP). Recall = TP / (TP + FN). New failure modes = problems reported by the labelers outside the 22 sub-levels.

Audit levelTPFPFNPrecisionRecallNew failure modes
L071087.5%100%0
L1802100%80.0%0
L21400100%100%0
L32509100%73.5%0
L41000100%100%0
All6411198.5%85.3%0

Table B. Verifier vs. expert labels per kernel.

Expert label: no violationExpert label: ≥1 violationTotal
Verifier: passes all five levels15 (TN)5 (FN)20
Verifier: ≥1 violation0 (FP)30 (TP)30
Total153550
FPR / FNRFP/(FP+TN) = 0%FN/(FN+TP) = 14.3%

All 11 false negatives of the verifier fall within existing sub-levels (last column of Table A). The verifier has checks for these problems, but its probes did not construct the specific inputs that trigger them.

Of the 20 kernels that pass all five levels, 5 have violations. All of them fall into this case.

  • S036 (DeepSpeed softmax), S069 (lmdeploy ban bad words), and S074 (Chrono fluid dynamics), sub-level 3.2 (non-contiguous layout). These 3 kernels read inputs with 16-byte vectorized instructions and assume that the start address of each input is 16-byte aligned. Suppose an input is a contiguous view cut from a larger buffer, with its start address offset by 1 or 2 elements. The kernel then raises a misaligned address error, while the reliable kernel gives correct results. The probes of 3.2 construct non-contiguous views. These kernels first copy a non-contiguous input into a contiguous tensor, so they pass. The probes do not include contiguous views with an unaligned start address.
  • S006 (PyTorch3D face areas and normals), sub-level 1.3 (semantic tampering). When one vertex coordinate is NaN, the reference and the reliable kernel output NaN for all normal components of the adjacent faces. During normalization, this kernel replaces the NaN norm with a small constant. Some of the components therefore become finite values, up to about 2.6e6. The check of 1.3 compares the number of NaN/Inf values in the output. It reports a violation only when this number falls below one tenth of that of the reference. The kernel keeps 18 of 24, so it was not detected.
  • S084 (OpenCV DNN concat), sub-level 3.3 (shape specialization). The kernel supports at most 32 input tensors and raises an error beyond 32. The reference and the reliable kernel both handle such inputs correctly. The probes of 3.3 vary the dimensions of the input tensors but not the number of input tensors.

The other 6 false negatives occur on kernels that the verifier already flags, and they have the same causes. Three are the same unaligned contiguous views (3.2). One is the same partial replacement of NaN/Inf with finite values (1.3). One arises because the probes of 3.3 do not vary the constructor argument dim. One arises because the probes of 3.1 (dtype lock-in) do not cover int32 indices. These false negatives do not change the overall verdict on the corresponding kernels. All of the above false negatives can be detected by adding the corresponding cases to the probes of existing sub-levels. No new category is needed, and such false negatives will decrease as the probe budget grows.

Of the 65 entries reported by the verifier, only one is a false positive. It is at sub-level 0.3 (precomputation outside timing). One probe of 0.3 checks whether a compute kernel runs only on the first call of a process. S082 (Kaolin sided distance) runs reset_kernel once on its first call to set its cached workspace to an initial value, so the verifier reported 0.3. This step only initializes the workspace and contains none of the computation of the reference. Every later call performs the full distance computation and resets the workspace after reading out the result. The first call and a steady call also take similar time (0.16 ms and 0.14 ms). The labelers therefore judged that the kernel does not violate 0.3. Although this verdict is a false positive at 0.3, the kernel has other confirmed violations, such as a race on the same workspace under concurrent calls (4.3). The per-kernel FPR therefore remains 0.

The full labeling results are given in Appendix L of the revision.

Reviewer_DcksOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

KernelAudit constructs 230 tasks across 14 domains from expert kernels and audits generated implementations for measurement bypass, algorithmic mismatch, numerical mismatch, input-domain fragility, and environment fragility. On seven models, 97.8% of kernels pass the initial correctness check, but only 25.3% pass all audit levels. On a 50-task subset, audit feedback raises GPT-5.6-sol’s full-audit pass rate from 42% under an empty audit to 80%.

Critical strengths
  1. A detailed, executable reliability assessment. Expert kernels anchor both performance and defect attribution. The five-level verifier records reproducible evidence, and its checks are supported by 211 regression cases, including 150 expert-labeled cases and benchmark-wide negative controls.

  2. A controlled audit-feedback experiment. Empty-audit and gate-only controls help separate feedback content from tool presence. The main comparison uses held-out final checks, reports paired score statistics, and provides a speed-ratio confidence interval rather than relying solely on unpaired medians.

Critical weaknesses
  1. The headline reliability rate mixes different task obligations. The initial specification supplies the reference and public shapes, while some dtype, layout, device, and concurrency requirements are revealed only through audit feedback. Non-contiguous-input rejection alone affects 81% of GPT-5.6-sol kernels. Separate rates for explicit-contract errors and additional expert-envelope requirements are needed to distinguish incorrect computation from narrower supported interfaces.

  2. Verifier validation is substantial but not independent enough to establish general accuracy. The checks are revised against the same expert-labeled regression cases used to report agreement. The reliable-kernel self-tests do not replace a held-out evaluation on diverse, correct optimized alternatives. Independent per-level false-positive and false-negative estimates would strengthen the broad reliability conclusions.

Questions for authors
  1. What reliability rates remain when the full supported dtype/layout/device contract is stated explicitly before generation? Please distinguish hard rejection from silent wrong answers within and beyond the public task domain.

  2. On an independent, expert-labeled set of generated and correctly optimized kernels, what are the verifier’s per-level false-positive and false-negative rates?

  3. Does the 42%-to-80% feedback benefit persist across repeated runs or another model on tasks sampled independently of the expert-over-agent speed criterion used for the 50-task subset?

  4. The paired speed-ratio interval is [0.94, 1.01]. Should “without slowing” be weakened to “no statistically detected slowdown,” or can speed equivalence within a predefined margin be established?

5 comments
AuthorsOpenReview ↗

Response to W1&Q1

Thank you for the positive assessment and the detailed suggestions. We respond to four points below. They are the distinction between incorrect computation and narrower supported interfaces, the independent validation of the verifier, the effect of audit feedback on other models, and the wording about speed.

1. On distinguishing incorrect computation from narrower supported interfaces (Weakness1 & Question1)

Thank you for the suggestion. Your concern is that the pass rate mixes two kinds of problems with different natures. The first kind is explicit-contract errors, where the kernel computes a wrong result. The second kind is expert-envelope requirements, where the kernel supports a narrower interface than the expert implementation, for example by rejecting non-contiguous inputs. In addition, requirements such as dtype and memory layout are not stated in the task statement in advance. We understand this concern. Below we clarify how the scoring of KernelAudit separates these two kinds of problems and what role the expert implementation plays in the verdict. We also add an experiment that states these requirements in advance. It shows how high the pass rate becomes when the requirements are stated explicitly before generation.

The scoring of KernelAudit already separates these two kinds of problems. A silently wrong answer counts as an L1 violation, whether it occurs inside or outside the public task domain. Measurement bypass counts as an L0 violation. Both set the score to zero through the audit gate. An error or a hard rejection on an input outside the public task domain counts at L3. It only lowers the quality factor and does not set the score to zero (§2, Figure 3). The audit gate pass rate (Figure 4b) and the fraction of zero scores (Zero in Table 1) reflect the first kind. The pass rate (Pass in Table 1) also includes the second kind. Take GPT-5.6-sol, which you mention, as an example. 81% of its kernels reject non-contiguous inputs, so its quality factor is the lowest of all models (61.1). Yet its audit gate pass rate is the highest of all models (89.1, Figure 4b), and its fraction of zero scores is the lowest (10.9%, Table 1). The narrower interface thus only lowers its quality factor and is not treated as incorrect computation.

The expert envelope you mention is used in KernelAudit only to relax verdicts. It adds no requirement. The generated kernel must replace the reference. As defined in §2, the reference serves as both the problem statement and the correctness oracle. Dimensions such as dtype and memory layout are determined by the behavior of the reference itself. The role of the expert implementation is to exclude requirements that it cannot meet itself. A failed check counts as a violation of the generated kernel only if the reliable kernel passes the same check. If the reliable kernel also fails, the violation does not count (§2, Appendix H). Under this rule, the number of generated kernels scored zero for L0/L1 violations drops from 461 to 341.

The task statement does not list these dimensions one by one, because disclosing concrete test conditions would let models design hacks that target them (Appendix K). All of these behaviors can be determined by running the reference directly. To answer how high the pass rate becomes when the requirements are stated in advance, we compared three settings on GPT-5.6-luna (Table A). The first is the empty audit. The second adds a paragraph to the prompt of the empty audit. This paragraph asks the model to attend to the support of dtypes, memory layouts, numerical precision, and concurrency safety. It does not give concrete check instances. The third is audit feedback with the original prompt.

Table A. GPT-5.6-luna on the 50-task subset. Pass rate is the fraction of kernels that are correct and pass all five audit levels.

Empty auditEmpty audit + explicit requirementsAudit feedback
Pass rate34%58%74%
Mean score47.9853.5355.25
L0 violations2%0%0%
L1 violations2%0%2%
L2 violations14%8%12%
L3 violations58%30%14%
L4 violations2%2%0%
Median speedup1.16×1.14×1.21×

Under all three settings, silently wrong answers and measurement bypass (L0 and L1) stay at or below 2%. The gap concentrates at L3, which covers narrower interfaces. Stating the requirements in advance raises the pass rate from 34% to 58% and lowers L3 violations from 58% to 30%. Audit feedback reaches a pass rate of 74% and lowers L3 violations to 14%. Stating the requirements in advance thus removes about half of the L3 violations. The rest requires the diagnosis that audit feedback provides for the specific implementation. Full results are given in Appendix N of the revision.

AuthorsOpenReview ↗

Response to W2&Q2 (1/2)

2. On the independent validation of the verifier (Weakness2 & Question2)

Thank you for recognizing the regression cases and negative controls. Your concern is that the verifier was revised against labeled cases from the construction stage, and the same cases were used to report agreement. A held-out evaluation is therefore missing, especially one on diverse and correct alternative implementations. In the original manuscript, all sub-levels and their decision rules were determined during construction and fixed before the formal evaluation (Appendix B). To answer this question directly, we built a held-out labeled set on tasks that were not used to refine the verifier. We report false positives and false negatives at each level.

In short, the verifier passes all 15 kernels that the labels mark as having no violation. These are the correct alternative implementations you asked about. When counting violation entries, only 1 of the 65 violations reported by the verifier is a false positive. All 11 false negatives belong to the existing 22 sub-levels, and the labelers found no failure mode outside them. These false negatives arise because the probes of the corresponding sub-levels did not construct the specific inputs that trigger them. We describe the experiment below.

Sampling. The kernels used to refine the verifier come from 62 of the 230 tasks. We therefore drew an independent sample from the remaining 168 tasks. On these tasks, the seven models evaluated in the paper produced 1,152 kernels that pass the correctness check. To have samples at every level for estimating false positives and false negatives, we split these kernels into six groups by the lowest audit level at which the verifier reports a violation. L0 is the most severe level. One group passes all five levels, and the lowest violated level of the other five groups is L0, L1, L2, L3, or L4. Given the cost of manual labeling, we drew 20 kernels from the group that passes all five levels and 6 kernels from each of the other groups, 50 in total. Kernels were drawn at random within each group, and all seven models are represented. The 50 kernels span 43 tasks.

Labeling. The 50 kernels were labeled by experienced CUDA engineers we invited. The labelers do not know how the verifier is implemented, and they did not see its verdicts during labeling. Each kernel comes with the generated code, the PyTorch reference, the reliable kernel of the task, and the definitions of the 22 sub-levels of L0–L4 in Appendix B. Sub-level x.y denotes the y-th sub-level of Lx. Following the same criteria as KernelAudit, the labelers judged each of the 22 sub-levels in turn. They could report any problem outside the sub-levels separately.

Results. Because a kernel may violate several sub-levels at once, we compare the verifier with the labels at two granularities. When counting violation entries, each (kernel, sub-level) pair is one entry. The labels contain 75 violations, and the verifier reports 65. Table B gives the false positives and false negatives at each level. When counting kernels, we only consider whether a kernel passes all five levels (Table C).

Table B. Verifier vs. expert labels at each audit level. Each (kernel, sub-level) pair is one entry. Precision = TP / (TP + FP). Recall = TP / (TP + FN). New failure modes = problems reported by the labelers outside the 22 sub-levels.

Audit levelTPFPFNPrecisionRecallNew failure modes
L071087.5%100%0
L1802100%80.0%0
L21400100%100%0
L32509100%73.5%0
L41000100%100%0
All6411198.5%85.3%0

Table C. Verifier vs. expert labels per kernel.

Expert label: no violationExpert label: ≥1 violationTotal
Verifier: passes all five levels15 (TN)5 (FN)20
Verifier: ≥1 violation0 (FP)30 (TP)30
Total153550
FPR / FNRFP/(FP+TN) = 0%FN/(FN+TP) = 14.3%
AuthorsOpenReview ↗

Response to W2&Q2 (2/2)

All 11 false negatives fall within existing sub-levels (last column of Table B). They arise because the probes do not cover the corresponding input cases.

  • Sub-level 3.2 (non-contiguous layout), 6 entries. The kernel assumes that the start address of each input is aligned. It raises an error when an input is a contiguous view cut from a larger buffer with an unaligned start address, while the reliable kernel gives correct results. The probes of 3.2 construct only non-contiguous views. These kernels first copy a non-contiguous input into a contiguous tensor, so they pass.
  • Sub-level 1.3 (semantic tampering), 2 entries. The kernel replaces part of the NaN/Inf values in the reference output with finite values. The check of 1.3 reports a violation only when the number of NaN/Inf values in the kernel output falls below one tenth of that of the reference. These 2 kernels keep a fraction above this threshold.
  • Sub-level 3.3 (shape specialization), 2 entries. The kernel supports only a limited range of the constructor argument dim or of the number of input tensors. The probes of 3.3 vary the dimensions of the input tensors but not these two quantities.
  • Sub-level 3.1 (dtype lock-in), 1 entry. The kernel does not support int32 indices, and the probes of 3.1 do not cover the dtype of indices.

All of these false negatives can be detected by adding the corresponding cases to the probes of existing sub-levels. No new category is needed, and such false negatives will decrease as the probe budget grows.

The only false positive comes from a probe of sub-level 0.3 (precomputation outside timing). This probe checks whether a compute kernel runs only on the first call of a process. S082 (Kaolin sided distance) runs reset_kernel once on its first call to set its cached workspace to an initial value, so it was reported. The labelers judged that this does not violate 0.3. This step is only an initialization and contains none of the computation of the reference. Every later call performs the full distance computation and resets the workspace after reading out the result. The first call and a steady call also take similar time (0.16 ms and 0.14 ms). The kernel has other confirmed violations, such as a race on the same workspace under concurrent calls (4.3). Counted per kernel, there is therefore still no false positive. The full labeling results are given in Appendix L of the revision.

Reviewer_UAzmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces KernelAudit, a benchmark for evaluating the reliability and performance of LLM-generated GPU kernels. It constructs 230 tasks across 14 domains, using expert implementations as performance baselines and references for attributing reliability violations. Its five-level audit covers measurement integrity, algorithmic equivalence, numerical behavior, input-domain support, and execution-state robustness. Seven models are evaluated on the full benchmark, and audit feedback is studied for GPT-5.6-sol on a 50-task subset. The main findings are a substantial gap between ordinary functional correctness and audit acceptance, and improved acceptance when diagnostic feedback is available during generation.

Critical strengths

• The paper addresses an important problem and constructs a substantial benchmark covering diverse real-world operators. • Comparing generated and expert implementations under the same probes provides a useful basis for attributing violations while accounting for limitations shared by existing implementations. • The controlled timing protocol, empty-audit control, and held-out final checks are thoughtful experimental design choices.

Critical weaknesses

• The reported verifier validation does not sufficiently establish its accuracy on independent implementations. • The incremental detection benefit and computational cost of the full audit remain unclear relative to simpler strengthened testing strategies. • The feedback experiment combines disclosure of additional requirements with diagnosis of implementation-specific failures. • The broader behavioral contract communicated to the generation agent is not fully established in the manuscript.

Questions for authors

1 Independent validation of the verifier is limited

The verifier is central to the paper's conclusions, including the reported 25.3% audit acceptance rate. Section 2.2 states that checks are revised when they miss labeled regression cases. Reproducing all 150 expert-labeled outcomes therefore demonstrates regression coverage, rather than detection accuracy on an independent test set. Negative-case coverage is also uneven: 31 of the 42 labeled negatives concern shape specialization (Appendix B, Table 11).

Executable counterexamples, comparisons against expert kernels, and negative controls provide meaningful evidence for individual findings. However, the manuscript does not report an independent assessment covering both flagged and unflagged submissions and compliant alternative implementations. The overlap between the failure-mining kernels, regression cases, and final evaluation is not sufficiently documented. False-positive and false-negative rates consequently remain uncertain, especially for precomputation checks and source-based precision rules. This limits confidence in the aggregate reliability assessment without invalidating every reported counterexample.

2 The incremental value of the full audit is not isolated

The seven-model evaluation is substantial, but it characterizes the evaluated agents rather than the relative effectiveness of different validators. The contrast with ordinary functional checking establishes that additional audits identify more deviations. It does not establish how much benefit requires the full audit structure rather than a broader, simpler test suite.

The experiments do not compare KernelAudit with strengthened testing, such as additional seeds, diversified numerical inputs, or boundary and layout checks, on the same implementations under a common testing budget. Its incremental detection benefit, false-positive behavior, and cost relative to these alternatives are therefore not established. The common generation-time cap and controlled timing of generated versus expert kernels address different experimental questions; neither constitutes a resource-matched comparison between validators.

3 The feedback experiment does not separate two sources of improvement

The full-feedback and empty-audit conditions share the model, prompt, budget, and tool latency. The improvement from 42% to 80% audit acceptance is meaningful within this setting. However, Appendix E explicitly states that dtype, layout, precision, and concurrency requirements are omitted from the initial prompt and disclosed through feedback. The intervention therefore supplies both additional specification information and diagnoses of the current implementation.

There is no condition in which the full behavioral requirements are stated in advance without executable audit feedback. The experiment consequently does not separate the contribution of clearer requirements from that of implementation-specific diagnosis. This limits the claim that instructions alone do not close the gap. In addition, the empty audit always reports that no problems were found, which may influence further checking or repair. The evidence is also restricted to one model, a selected 50-task subset, and one generation run per condition.

4 The disclosed behavioral contract remains ambiguous

Assigning zero credit to a kernel that computes the wrong function on a valid input is reasonable. The ambiguity concerns which broader behaviors are part of the task contract. The agent receives a PyTorch reference and shape information, but the manuscript does not fully establish the required scope of state updates and non-finite input behavior.

For example, Appendix K penalizes a pyrUp implementation for not responding to changes in a buffer containing Gaussian coefficients. This is a valid failure under a mutable-module replacement contract, whereas folding fixed coefficients into the computation can be legitimate for a fixed operator. NaN/Inf propagation raises a similar specification question. Treating the reference module as the complete behavioral specification could justify these requirements, but that interpretation is not sufficiently explicit in the documented generation instructions. This leaves uncertainty about whether some reported failures concern a declared task requirement or an additional robustness criterion; it does not establish that the examples are misclassified.

7 comments
AuthorsOpenReview ↗

Response to W1&Q1 (1/2)

Thank you for the careful review and the constructive comments. We respond to four points below. They are the independent validation of the verifier and the incremental value of the full audit over strengthened testing. They also include the sources of improvement in the feedback experiment and the behavioral contract of the tasks.

1. On the independent validation of the verifier (Weakness1 & Question1)

Thank you for the suggestion. We first clarify that the samples used to refine the verifier are independent of the final evaluation. The regression cases used to refine the verifier come from the construction stage. None of them is a kernel generated in the full evaluation of the paper. We used these cases only during construction to revise the checks. All sub-levels and their decision rules were fixed before the formal evaluation (Appendix B) and were not modified during the evaluation. The labeled cases used in construction therefore do not affect the final evaluation results.

To further address your concern, we built an independent test set. Its tasks do not overlap with those of the regression cases.

Sampling. The kernels used to refine the verifier come from 62 of the 230 tasks. We excluded these tasks. On the remaining 168 tasks, the seven models evaluated in the paper produced 1,152 kernels that pass the correctness check. To avoid concentrating the sample in a few levels, we grouped these kernels by the lowest audit level at which the verifier reports a violation. L0 is the most severe level. This gives six groups. One group passes all five levels, and the lowest violated level of the other five groups is L0, L1, L2, L3, or L4. Manual labeling is costly, so we sampled 50 kernels in total. We drew 20 kernels from the group that passes all five levels and 6 kernels from each of the other five groups. All kernels were drawn at random within their group, and the sample covers all seven models. The final 50 kernels come from 43 tasks.

Labeling. We invited experienced CUDA engineers to label the 50 kernels. The labelers do not know how the verifier is implemented, and they did not see its verdicts during labeling. For each kernel, we provided the generated code, the PyTorch reference, the reliable kernel of the task, and the definitions of the 22 sub-levels of L0–L4 (Appendix B). Sub-level x.y denotes the y-th sub-level of Lx. The labeling criteria match those of KernelAudit. The labelers judged whether the kernel violates each of the 22 sub-levels. They could also report any other problem outside the 22 sub-levels.

Results. A kernel may violate several sub-levels at once, so we compare the verifier with the labels in two ways. When counting violation entries, each (kernel, sub-level) pair is one entry. The labels contain 75 violations over the 50 kernels, and the verifier reports 65 (Table A). When counting kernels, we only consider whether a kernel passes all five levels (Table B).

Table A. Verifier vs. expert labels at each audit level. Each (kernel, sub-level) pair is one entry. Precision = TP / (TP + FP). Recall = TP / (TP + FN).

Audit levelTPFPFNPrecisionRecall
L071087.5%100%
L1802100%80.0%
L21400100%100%
L32509100%73.5%
L41000100%100%
All6411198.5%85.3%

Table B. Verifier vs. expert labels per kernel.

Expert label: no violationExpert label: ≥1 violationTotal
Verifier: passes all five levels15 (TN)5 (FN)20
Verifier: ≥1 violation0 (FP)30 (TP)30
Total153550
FPR / FNRFP/(FP+TN) = 0%FN/(FN+TP) = 14.3%

Of the 65 entries reported by the verifier, only one is a false positive. It is at sub-level 0.3 (precomputation outside timing). One probe of 0.3 checks whether a compute kernel runs only on the first call of a process. S082 (Kaolin sided distance) runs reset_kernel once on its first call to set its cached workspace to an initial value, so the verifier reported 0.3. This step only initializes the workspace and contains none of the computation of the reference. Every later call performs the full distance computation and resets the workspace after reading out the result. The first call and a steady call also take similar time (0.16 ms and 0.14 ms). The labelers therefore judged that the kernel does not violate 0.3. Although this verdict is a false positive at 0.3, the kernel has other confirmed violations, such as a race on the same workspace under concurrent calls (4.3). The per-kernel FPR therefore remains 0.

AuthorsOpenReview ↗

Response to W1&Q1 (2/2)

There are 11 false negatives. All of them arise because the probes do not cover the corresponding case. Six are at sub-level 3.2 (non-contiguous layout), where the probes do not include unaligned contiguous views. Two are at sub-level 1.3 (semantic tampering), where the kernel suppresses only part of the NaN/Inf values and the retained fraction stays above the detection threshold. Two are at sub-level 3.3 (shape specialization), where the probes do not vary a constructor argument such as dim or the number of input tensors. One is at sub-level 3.1 (dtype lock-in), where the probes do not cover int32 indices. The labelers reported no problem outside the 22 sub-levels.

The verifier therefore produces almost no false positives, and nearly all of its errors are false negatives. The 25.3% pass rate reported in the paper is thus an optimistic estimate. The reliability gap revealed by the paper is in fact underestimated, and real generated kernels contain more problems than KernelAudit reports. The full labeling results are given in Appendix L of the revision.

AuthorsOpenReview ↗

Response to W2&Q2

2. On the incremental value of the full audit over strengthened testing (Weakness2 & Question2)

Thank you for the suggestion. KernelAudit checks generated kernels at five levels (L0–L4), namely measurement bypass, algorithmic mismatch, numerical mismatch, input-domain fragility, and environment fragility. Most problems at L0, L1, and L4 are hard to detect with strengthened testing, such as additional seeds, diversified numerical inputs, or shape and layout variations. In particular, measurement bypass at L0, such as moving computation outside the timed region, is a serious form of cheating that common simple tests rarely find. The full audit clearly costs more than strengthened testing. We nevertheless consider this cost necessary.

To further support this point, we compared three levels of increasingly strengthened testing with the full audit. We used the same 50 kernels and labels as in Section 1.

  • B1 uses 20 random seeds, including the 3 seeds of the original correctness check. Each seed generates new inputs at the original shapes of the task.
  • B2 adds diverse numerical inputs to B1. These include magnitude scaling, mean shifts, near-zero values, heavy-tailed distributions, rounding boundaries, all-zero and constant inputs, and NaN, +Inf, and −Inf injected at the start, middle, and end of each input.
  • B3 adds shape, dtype, and layout variations to B2. Each public shape dimension takes the base value ±1 and ±2, and the neighboring multiples of 16 ±1. Each tensor input is also given three non-contiguous views. They are a transposed view, a view that takes every other row along the first dimension, and a view with stride 2 in the last dimension.

All three tests compare only against the PyTorch reference, with rtol = atol = 1e-2. When the reference runs normally, a generated kernel is judged as a violation if, at any test point, it exceeds the tolerance, produces mismatched outputs, or raises an error. None of the three tests includes timing, repeated-call, or concurrency checks. Per-kernel results of each test are given in Appendix M of the revision.

Table C. Strengthened testing vs. the full audit on the 50 kernels of Table B. FPR = FP / (FP + TN). FNR = FN / (FN + TP).

TestTPFPTNFNFPRFNR
B1: 20 random seeds0015350%100%
B2: B1 + numerical inputs9015260%74.3%
B3: B2 + shapes, dtypes, and layouts233121220.0%34.3%
Full audit3001550%14.3%

B1 detects no problem. B3 detects 23 kernels but still misses 12, and it wrongly flags 3 kernels that have no violation. The full audit detects 30 kernels with no false positive. All three false positives of B3 come from non-contiguous views. On the same views, the reliable kernel of the task also fails to give a correct result. For example, S083 (Kaolin spc_conv) raises "must be contiguous" on a view with stride 2 in the last dimension, and the reliable kernel of this task also rejects such views. B3 compares only against the reference, so it counts such limitations shared with the expert implementation as violations. Neither the labelers nor the full audit count them as violations of the generated kernel.

The misses of B3 concentrate at L0 and L4. Two examples follow.

  • S078 (llm.c AdamW). The kernel returns its internal state buffer directly as the output, so the next call overwrites the previous output kept by the caller. B3 only compares the output of the current call, so the kernel passes. The full audit keeps the first output and calls the kernel again, and it detects 0.3 and 4.3.
  • S041 (Kaolin feature grids). The temporary index space is kept in a process-wide global cache. The lock protects only the cache lookup and not the asynchronous GPU computation that follows. When two instances run concurrently on two CUDA streams, 5 of 20 runs give wrong results, compared with 0 for the reliable kernel. B3 runs all tests sequentially and cannot trigger this conflict. The concurrency check of the full audit detects 4.3.

Overall, on the same kernels, the full audit detects 7 more faulty kernels than the strongest strengthened testing, with no false positive. The gain comes from two sources. First, the checks at L0 and L4 cover problems that only timing, repeated calls, or concurrent use can expose, and tests that only compare outputs cannot trigger them. Second, the comparison with the reliable kernel excludes limitations that the expert implementation shares, which avoids false positives.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers again for your time. On the one hand, we thank you for recognizing that our work addresses a critical gap between benchmark performance and practical utility of LLM-generated kernels, builds a substantial benchmark covering diverse real-world operators, and provides a detailed and executable five-level audit with a careful experimental design. At the same time, we thank you for your constructive comments and summarize the comments into four aspects below, along with our responses.

  1. Independent validation of the verifier
  2. Whether simpler methods can replace the audit
  3. Decision criteria anchored on the reliable kernel
  4. Audit feedback on more models

First, all reviewers (UAzm, Dcks, hKNr) ask whether the reliability of the verifier should be further evaluated with clearer independent tests. We therefore drew 50 kernels from the 168 tasks not used to refine the verifier, stratified by its verdicts. CUDA engineers labeled them blindly against all 22 sub-levels and could also report problems outside them. The labelers found no failure mode outside the 22 sub-levels, so the verifier misses no failure category. Of the 65 violations it reports, only one is a false positive. All misses fall within existing sub-levels. They occur because budget-limited probes did not construct the triggering inputs, not because the verifier ignores these problems. Adding probes to those sub-levels detects them without any new category. The 22 proposed sub-levels thus cover all failure categories observed in independent samples, the verdicts are reliable, and the verifier improves as the probe budget grows. See UAzm §1 (Tables A–B), Dcks §2 (Tables B–C), hKNr §2 (Tables A–B), and Appendix L.

Second, the reviewers ask whether simpler methods can replace the audit. UAzm asks about the incremental value of the full audit over merely repeated strengthened testing. UAzm and Dcks ask whether disclosing the audited problem types as requirements in advance would work similarly. We added two comparisons. On the same 50 kernels, we compared the full audit with strengthened testing that adds random seeds, diverse numerical inputs, and shape, dtype, and layout variations. The full audit detects 7 more faulty kernels, while strengthened testing wrongly flags 3 clean ones. The full audit has fewer misses because it adds anti-hacking checks on timing, repeated calls, and concurrency beyond more inputs. It has fewer false positives because it does not count limitations shared by the reliable kernel. On GPT-5.6-luna, stating the dtype, layout, precision, and concurrency requirements in the prompt raises the pass rate from 34% under the empty audit to 58%, while audit feedback reaches 74%. Therefore, given the cheating types summarized by our audit, more tests or upfront requirements can each prevent cheating to some extent. This indirectly confirms the effectiveness and soundness of the five-level audit itself. Neither, however, can replace the full audit's diagnosis of specific implementations. See UAzm §2 (Table C) and §3.3 (Table E), Dcks §1 (Table A), and Appendices M and N.

Third, the reviewers ask about anchoring decisions on the reliable kernel, the hand-written expert kernel deployed in the original application. UAzm and Dcks ask whether it adds requirements beyond the PyTorch reference. hKNr is concerned that the criteria are stricter than real deployment, which often accepts kernels with restricted input domains or lower precision. We clarify that all requirements come from the reference, which itself defines behaviors such as dtype, layout, module state, and NaN/Inf. The reliable kernel adds no requirement and only relaxes the judgment. We first probe the generated kernel under strict requirements, then probe the reliable kernel with the same inputs. A failure shared by the reliable kernel is not counted. Trade-offs accepted in real deployment, such as restricted input domains or low-precision approximations, are therefore not penalized. We do not relax requirements in the prompt, because uniform relaxation invites hacking and task-specific relaxation would reveal the expert solution. The anchor thus balances strict reliability with real deployment needs. See UAzm §4, Dcks §1, hKNr §1, and §2 of the revision.

Fourth, UAzm and Dcks ask whether the gain from audit feedback (42% to 80% for GPT-5.6-sol) holds beyond a single model. On DeepSeek V4.1 Flash and GPT-5.6-luna, audit feedback raises the pass rate from 64% to 78% and from 34% to 74%, respectively, with no clear drop in median speedup. The consistent direction across three models shows that the gain is not model-specific. See UAzm §3.2 (Table D), Dcks §3 (Table D), and Appendix N.

We thank all reviewers again for your time and constructive comments. We hope the above addresses your concerns, and we look forward to further discussion on these issues and our additional experiments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.