acceptodds
Under review as a conference paper at ICLR 2027

Measuring the Checker: Mutation Analysis for GPU-Kernel Benchmark Oracles

Abstract

Benchmarks for LLM-generated GPU kernels decide correctness with a few random inputs and a loose floating-point tolerance, and their verdicts now feed leaderboards and reinforcement-learning rewards. Recent work agrees these checkers are weak and patches them by hand—extra input distributions, fuzzing recipes, tighter tolerances—with no way to measure whether any patch suffices. We introduce mutation analysis as an adequacy metric for kernel-benchmark oracles: deterministic rules inject 10,303 compilable faults into verified CUDA implementations of 189 KernelBench problems, 7,384 of them with an independent kill witness; any test protocol is scored by the fraction it detects. The official check misses one in six witnessed faults (16.9%), deterministically, and the misses are skewed by family: 8.7% of arithmetic faults escape but 78.6% of precision faults do. The metric explains why (a tolerance blind band growing with reduction size; a measured ceiling on input aggressiveness set by legitimate floating-point variance), audits the strongest existing patch (KernelBench-Verified's gain splits into +4.0 points from hidden inputs and +4.5 from tighter tolerance, a split its authors could not compute), and exposes a published fuzzing recipe that rejects correct kernels 107 times. Optimizing suites over the kill matrix reaches 98.0% detection with two inputs per problem (94.8% held-out), and the measurement's fault taxonomy teaches a test generator more than the raw faults themselves. Across 48 whole architectures the blindness grows with scale, concentrating in deep homogeneous pipelines, and two problems prove unrefereeable: their official references violate the benchmark's own tolerance against fp64.

Primary area:
datasets and benchmarks
Keywords:
kernel generation, mutation testing, code generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_V6XpOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a mutation-testing framework to evaluate the correctness checkers used in GPU kernel benchmarks such as KernelBench. The authors introduce 124 mutation rules to inject faults into CUDA implementations of 189 operator-level problems and 48 architectures. They use a validity gate to filter out test inputs that reject the original implementations and only include mutants detected by at least one valid input in the main mutation-score calculation. The experiments show that the official KernelBench checker misses 16.9% of detectable faults, especially precision and synchronization errors. The paper also evaluates existing checker improvements and uses set cover to select more effective test inputs.

Critical strengths
  1. The paper provides a systematic way to measure the effectiveness of GPU kernel correctness checkers. This paper introduces mutation testing to measure which types of faults a checker can detect and which ones it misses. The validity gate and kill-witness mechanism are useful adaptations to GPU programs. This framework provides a useful tool for understanding and improving GPU kernel benchmarks.

  2. The experimental findings are detailed and practically useful. The paper shows that different types of faults have very different detection rates. For example, the official checker misses 78.6% of precision faults but only 8.7% of arithmetic faults. These findings provide useful guidance for designing more effective correctness tests.

Critical weaknesses
  1. Insufficient validation on real GPU kernel bugs and potential bias in the evaluation. The main experiments rely on artificially injected faults in the authors' own CUDA implementations. Although mutation testing provides a controlled way to measure checker effectiveness, these faults may not fully represent the errors produced by real LLM-generated or highly optimized GPU kernels. The paper includes a small experiment with 60 LLM-generated kernels in Appendix D.6, but it does not directly demonstrate whether the proposed tests can detect real bugs missed by the official checker. Another concern is how the evaluation set is constructed. Although Table 3 evaluates all methods on the same set of witnessed mutants, this set is constructed using the official inputs and the authors' own targeted and LLM-generated tests. As a result, the evaluation may favor the proposed method because its own test inputs help determine which faults are included in the comparison. Although the authors also report held-out results, the split is based on mutant IDs, meaning similar faults from the same code locations may appear in both sets. Therefore, it remains unclear how well the reported improvements generalize to real kernel bugs or different types of unseen faults.

  2. Insufficient validation of the validity gate across different correct CUDA implementations. The paper uses a validity gate to prevent test inputs from incorrectly rejecting correct kernels. Specifically, a test input is considered valid if the authors' original CUDA implementation passes the comparison with the PyTorch reference. However, different CUDA implementations may use different accumulation orders, floating-point instructions, or optimization strategies, leading to legitimate numerical differences. An input that passes the validity gate for one implementation may therefore cause another otherwise correct implementation to fail the check. The zero false rejections reported in Table 3 are measured using the same substrate-based validity checks used to select the inputs. Therefore, this result does not establish whether the proposed tests would incorrectly reject other independently validated correct implementations. Since the proposed method aims to improve checker effectiveness without rejecting correct programs, additional validation across different optimized implementations is needed to support its practical reliability.

Questions for authors

Could the authors evaluate the proposed testing method on real LLM-generated or leaderboard CUDA kernels? In particular, it would be useful to compare how many real bugs are detected by the official KernelBench checker, KernelBench-Verified, and the proposed test suites. This would help establish whether the improvements observed on synthetic mutants transfer to real kernel errors.

Could the authors evaluate the proposed test inputs on multiple independently implemented and optimized CUDA kernels that have been validated as correct? For example, kernels using different reduction strategies, Tensor Cores, or floating-point instructions could help assess whether the validity gate generalizes beyond the authors' original implementations. Reporting false rejection rates on these kernels would strengthen the practical reliability claims.

4 comments
AuthorsOpenReview ↗

Response to Reviewer V6Xp

We thank the reviewer for recognizing the validity gate and kill-witness mechanism as useful adaptations of mutation testing to GPU programs, and the fault-family findings (78.6% of precision versus 8.7% of arithmetic faults missed) as practical guidance for test design. In response to your concerns, we:

  • ran the requested three-way comparison on real bugs: on 137 confirmed defects in 397 new LLM kernels, our suite catches 113 (official 105, KBV 106), 25 of them caught only by ours (Q1);
  • evaluated leaderboard kernels: on 492 published kernels, our suite finds 52 defective kernels, 23 of which pass every official and KBV input (Q1);
  • removed evaluation-set bias: the real kernels were fixed before any checker ran, and site-level and rule-level holdouts keep detection above 91% (Q1);
  • validated the gate on other correct implementations: 0 false rejections on 13 independent kernels and 0 wrong-value or runtime failures on 232 defect-free LLM kernels (Q2).

We will add these results to the revision and hope they address your concerns.

AuthorsOpenReview ↗

Response to Q1 and Q2

Q1. Real bugs: Official vs KernelBench-Verified vs ours.

(a) Three-way comparison on new LLM kernels.

We generated 397 new kernels with Codex and Claude Opus 5.5, each in an isolated workspace with no access to our mutation rules, tests or reference kernels. 137 kernels have confirmed defects (criterion below):

CheckerDefective kernels detected (of 137)
Official KernelBench (5 random inputs)105
KernelBench-Verified style (4 scaled inputs, 10−310^{-3})106
Ours (≤2 selected inputs)113
Official + Ours133

25 bugs are caught only by our suite, on valid inputs whose sizes differ from the benchmark's fixed shape.

(b) The 25 ours-only bugs are genuine.

On the detecting input:

  • the input is valid for the task. It keeps every Model constructor parameter (channels, kernel size, feature count) and every task precondition fixed, and varies only input sizes or values. Our reference CUDA kernel passes on it.
  • the kernel genuinely fails. It crashes, returns a wrong shape, or, for wrong values, disagrees with an FP64 evaluation while PyTorch with TF32 enabled passes. A precision-mode effect is therefore ruled out.
  • the failure reproduces in 3 fresh processes.

9 of the 25 crash, 2 return a wrong shape and 14 return FP64-confirmed wrong values. For example, 119relu drops the last numel mod 4\mathit{numel} \bmod 4 elements, and a GEMM kernel launches batch // 64 row blocks, dropping any batch tail. Such a kernel implements the operator only for the checker's sampled shape, and this oracle gap is exactly what our framework measures.

(c) Public leaderboard kernels.

We ran our frozen suites, with the same defect criterion, on 492 published kernels at KernelBench v0.1 sizes. Our ≤2-input suite finds 52 defective kernels, 23 of which pass every official and KernelBench-Verified input:

Kernel collectionKernels runFound by oursOf these, missed by Official and KBV
CudaForge94126
CUDA-L111572
Sakana AI CUDA Engineer198185
AdaExplore851510

For example:

  • a CUDA-L1 kernel (L2p80) skips the computation, since the operator's output is all zeros on valid inputs. It returns a zero tensor cached from the first call (batch 1024), so a 13-row batch gets a [1024,1][1024,1] output:

    batch_size = x.shape[0]
    self._zero_tensor = self._create_optimized_zero_tensor(
        batch_size, x.dtype, x.device, x.requires_grad
    )
    self.forward = self._zero_tensor
    self._optimized = True
    ...
    def __call__(self, x):
        ...
        if self._optimized:
            return self._zero_tensor
    
  • an AdaExplore kernel (L2p70, GEMM–sigmoid–scaling–residual) casts the FP32 input and weight to fp16 before the GEMM. Any input value above the fp16 maximum (65,504) becomes inf, so a single large but valid input value makes every output inf, while the FP32 reference is finite:

    self.register_buffer("weight_t_fp16", self.linear.weight.data.t().half(), persistent=False)
    ...
    x_fp16 = x.half().contiguous()
    ...
    return triton_gemm_sigmoid_residual(x_fp16, self.weight_t_fp16, self.bias_fp32, self.scaling_factor)
    
  • a CudaForge GELU computes tanh⁡\tanh as (e2z−1)/(e2z+1)(e^{2z}-1)/(e^{2z}+1), which overflows to NaN for large activations, and a CudaForge argmax never selects a NaN;

  • several Sakana GEMM and transposed-convolution kernels return inf or values up to 103810^{38} on batch sizes such as 3, 9 or 13.

These third-party kernels played no role in selecting our suites.

(d) Evaluation-set bias and the ID split.

The real-kernel populations were fixed before any checker ran and defects are confirmed against FP64. For "similar faults from the same code locations may appear in both sets", holding out by source-edit site keeps detection at 92.9% (+11.2 pp), and holding out whole rules gives 91.0% (+10.4 pp).

Q2. Correct implementations beyond our substrate.

  • Independent kernels. Our inputs produce 0 false rejections on 13 implementations of 6 tasks (different reduction trees, warp shuffles, FMA contraction, vectorized loads, shared-memory tiling; 132 implementation–input pairs).
  • Defect-free LLM kernels. On the 232 defect-free LLM kernels, written independently of our substrates, our inputs produced 0 wrong-value or runtime failures.
  • Tensor cores. Our inputs accept TF32 Tensor-Core PyTorch on all 70 convolution tasks. Wherever they reject it (9 of 40 GEMM tasks), TF32 itself deviates from FP64 by more than 10−210^{-2}, so these are real precision losses rather than false alarms; at KernelBench's current 10−410^{-4} default, even the official random inputs reject TF32 on 59% of inputs. We will state the FP32 contract and flag TF32-sensitive inputs.

I appreciate the authors' detailed rebuttal and additional experiments. The response addresses my main concerns, and I will maintain my positive rating.

Thank you very much for your time, constructive feedback and continued support! We will make sure all the new real-kernel and cross-implementation results are thoroughly integrated into the revised manuscript.

Reviewer_XXKdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This article introduces KernelBench-M, a mutation-analysis framework for assessing correctness checkers for LLM-generated GPU kernels. It constructs CUDA mutants, identifies valid detection witnesses, and uses the resulting kill matrix to evaluate existing checking protocols and select compact test suites. The experiments reveal substantial differences in detection across fault families.

Critical strengths

• The article addresses an important problem in evaluating and training GPU-kernel generation systems. • The mutation campaign is substantial, and the fault-family analysis provides useful insights into checker weaknesses.

Critical weaknesses

• The contribution and applicability are overstated relative to prior work and the experimental scope. • The final witnessed-mutant population and several numerical results are not consistently accounted for. • Boundary-input coverage remains unclear and may affect the interpretation of unresolved mutants and the reported scores. • Differences in testing budgets, filtering, and access to the fault pool weaken the fairness and generalization claims

Questions for authors

1 Contribution claims and experimental scope

Mutation-based test assessment, GPU mutation testing, and empirical audits of kernel validators already exist. The claim that earlier work does not measure checkers is therefore too broad. All experiments originate from KernelBench and one GPU generation. Its levels vary in complexity but are not independent benchmarks, limiting the scope supported by the evidence.

2 Witnessed mutants and boundary input coverage

The article constructs targeted inputs, acknowledges unresolved mutants, and reports alternative accounting bounds. However, coverage of boundary conditions within each task's valid input domain remains unclear. A missing witness may reflect an untriggered boundary path rather than equivalence or an unobservable defect. This affects the interpretation of unresolved mutants and makes the witnessed population and scores dependent on input coverage.

Section 3 requires valid witnesses for inclusion, while Appendix D.2 describes excluding mutants whose only witnesses were subsequently invalidated. It is unclear whether these exclusions were propagated to all headline results.

3 Inconsistencies in numerical reporting

Section 6 states that five inputs detect every witnessed mutant, while Figure 5b reports a problem requiring six for full coverage. Rounding might explain Table 4's 100.0%, but not the claim of complete detection. Figure 6's depth groups total 12,025 mutants, whereas Table 1 reports 12,361 architecture-level distinct mutants. The exclusion of the remaining 336 is unexplained.

4 Comparison fairness and generalization

The relationship between inputs, suites, and trials is unclear, so "two inputs versus five" does not establish equal execution cost. Table 3 also places an ungated fuzzing score alongside gated results for the proposed method.

The proposed suites use fault-pool information, while the holdout splits related mutants within the same problems. This provides limited evidence of transfer to unseen source sites or implementations. The knowledge-ladder configurations also differ in testing budgets and evaluation denominators, so the observed difference does not establish white-box overfitting or a causal advantage of taxonomy-level information.

4 comments
AuthorsOpenReview ↗

Response to Reviewer XXKd

We thank the reviewer for recognizing the importance of checker evaluation for GPU-kernel generation and the insight from our mutation campaign and fault-family analysis. The constructive and detailed sugguestions tightened our accounting, and the headline results hold under the stricter version. In response to your concerns, we:

  • narrowed the contribution claim and validated our suites beyond the mutant pool: on held-out sites and rules, on 397 new LLM kernels, on 492 published kernels and at the current upstream tolerance (Q1);
  • audited boundary coverage: a 34,671-input boundary battery on all 841 unwitnessed mutants, a classification of the remainder, and a per-ID ledger that propagates every exclusion; the official checker still misses 16.62% (original: 16.9%) (Q2);
  • reconciled the numbers in Section 6 and Figure 6 (Q3);
  • equalized comparisons: a strict ≤2-input budget, all baselines gated on a common cohort, and an equal-budget knowledge ladder (Q4).

We will add these results to the revision and hope they address your concerns.

AuthorsOpenReview ↗

Response to Q1 / Q2 / Q3 / Q4

Q1. Contribution and scope.

We agree that the sentence claiming earlier work does not measure checkers is too broad, and we will replace it. Mutation-based test assessment (CLTestCheck, MuTGpu), checker hardening (KernelBench-Verified) and validator audits with injected faults (The Correctness Illusion) are the closest work. Our contribution is the combination that none of these provides:

  • a fixed population of GPU faults with a recorded witness for each mutant and a correct-implementation gate;
  • a checker × fault-family miss matrix for a widely used LLM-kernel benchmark;
  • compact suites derived from that matrix, validated on held-out source sites and rules (92.9% and 91.0% detection, +11.2 and +10.4 pp over the official checker), on 397 independently generated LLM kernels (113 of 137 confirmed defects caught, versus 105 for the official checker), on 492 published kernels (52 defective kernels found, 23 of which pass every official and KernelBench-Verified input), and at the current upstream tolerance 10−410^{-4} (the native checker still misses 11.0%, 392 of 3,558; our ≤2-input suite rejects 248 of them).
ValidationOfficialOurs
Held-out sites81.7%92.9%
Held-out rules80.6%91.0%
397 LLM kernels (137 defective)105113
492 published kernels–52 defective, 23 pass official and KBV
Tolerance 10−410^{-4} (3,558 mutants)misses 392rejects 248 of them

Beyond our substrates, the new evidence covers 889 independently written kernels (397 LLM-generated, 492 published).

Q2. Boundary coverage and what the unwitnessed mutants are.

The 841 compiled, distinct mutants without a valid witness (Figure 1's 869, minus 89 with only a disputed screening witness, plus 61 invalidated in App. D.2) are outside every reported denominator, so they do not enter any miss rate in the paper. Every exclusion in App. D.2 now propagates to all results through a per-ID ledger: on the resulting 7,142 mutants, the official checker misses 16.6% (95% CI 13.5–20.1%; paper: 16.9%), and the ordering of fault families is unchanged (from 8.7% for arithmetic to 81.3% for precision).

We ran all 841 against a boundary battery of 34,671 valid inputs (sizes around every block, tile and warp constant, plus edge-case values). 105 (12.5%) gained deterministic witnesses; adding them raises the official miss rate to 17.8%. We classified the remaining 736 from their records:

MutantsCategory
91out-of-bounds write that never reaches the output
44problem output is constant on valid inputs
178guard/bound edit never triggered by a valid input
10__syncthreads → __syncwarp in a 32-thread block (equivalent)
50other synchronization edits, race never manifested
120precision edits whose effect stays below 10−210^{-2}
243other edits with no deterministic witness on any valid input

A manual inspection of 50 finds most of them equivalent on valid inputs, e.g. a guard made dead by exact grid sizing, or an edit masked by the operator's mathematics (LogSumExp shift invariance, BatchNorm absorbing a bias sign flip). We will release the classification. Even if all 736 were real faults, the official miss rate would lie between 16.2% (official catches all of them) and 25.4% (official misses all), so the conclusion is unchanged.

Q3. Numerical consistency.

  • Five versus six inputs. On the paper's 7,384 mutants, five selected tests cover 7,382 and six cover all; we will state this precisely in Section 6.
  • The 336 records. Figure 6 excludes 179 mutants from two-kernel RNN architectures and 157 edits outside recognized layer kernels. We will state this grouping rule and its denominator.

Q4. Fairness and generalization.

  • Budgets. In the mutant analysis we run the official checker with 2 random inputs per task. Giving it 5 inputs raises its detection rate by at most 0.3 percentage points, so the smaller budget does not disadvantage it. Our suites use up to 3 inputs (2.3–2.5 on average on the held-out splits). Under a strict ≤2-input budget they still beat the official checker on every held-out split: 88.8 / 87.8 / 85.8% versus 81.7 / 81.7 / 80.6% for ID / site / rule. All real-kernel results (Q1) use at most 2 inputs.
  • Gating. With all baselines (official, KBV inputs and tolerance, gated fuzzing) gated on a common cohort of 4,858 mutants, our suites catch 409 mutants that every baseline misses. Table 3 will be reported on this cohort.
  • Transfer. See Q1: held-out source sites and rules, new LLM kernels and published kernels.
  • Knowledge ladder. With equal input budgets on a common set of 992 mutants, the taxonomy-informed arm beats naive prompting by +10.8 pp (95% CI [2.7, 21.9]) and matches the arm given additional mutant details, so taxonomy-level information is what matters. We will revise the interpretation accordingly.

Dear Reviewer XXKd,

As the author–reviewer discussion deadline approaches, we wanted to gently follow up and let you know that we have uploaded the revised manuscript incorporating all the changes and new experiments discussed in our response (highlighted in red for easy reading):

  • [Q1] Contribution scope & transfer to 889 real kernels (§1\S1, §6\S6, §7\S7, and Appendix E.1 / E.5 / E.6);
  • [Q2] Boundary-input battery (34,671 inputs), unwitnessed-mutant classification, and strict per-mutant ledger (§3\S3, §4\S4, and Appendix E.3 / E.4);
  • [Q3] Reconciled numerical reporting in §6\S6 and Figure 6 (Appendix D.1);
  • [Q4] Equal-budget and common-cohort comparisons across held-out sites/rules, gated baselines, and the knowledge ladder (§5\S5, §6\S6, Appendix C, and Appendix E.2 / E.3).

We hope our responses and the revised paper address your concerns. Please let us know if you have any further questions or if there is anything else we can clarify before the discussion period concludes. Thank you!

Dear Reviewer XXKd,

As the discussion period ends today, we would be grateful if you could take a look at our response and the revised manuscript, and let us know whether they address your concerns. We are happy to clarify anything further. Thank you!

Authors

Reviewer_5JUiOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents KernelBench-M, a mutation-analysis framework for evaluating the adequacy of correctness checkers in GPU-kernel generation benchmarks. The authors introduce a mutation-testing pipeline that injects deterministic faults into verified CUDA implementations and evaluates their detectability using a witnessed mutation score.

Critical strengths

1.The paper addresses an important and timely problem: the reliability of correctness oracles used to evaluate LLM-generated GPU kernels. While previous work has identified weaknesses in existing benchmarks, this study provides a systematic way to quantify oracle adequacy through mutation analysis. This contribution is consequential because benchmark correctness directly affects the reliability of model evaluations and learning signals.

2.The study provides a relatively large-scale mutation analysis, covering multiple GPU fault families and both operator-level and architecture-level tasks. The results reveal substantial differences in detection effectiveness across fault categories, especially the high escape rate of precision-related mutations. The analysis of existing benchmark-hardening strategies is also informative.

Critical weaknesses

1.A lthough the experiments are extensive, most mutants are generated through deterministic transformations of constructed, gate-verified CUDA substrates rather than being sampled from real erroneous submissions produced by LLM-based kernel generators. The paper includes a realism study involving 60 LLM-generated kernels, but only two runtime failures are available to support the correspondence between the mutation taxonomy and actual implementation errors.

2.The set-cover results are promising, but the main full-pool results are necessarily in-sample, since the selected tests are optimized against the same mutation matrix used for evaluation. The held-out experiment addresses this issue partially. However, the split is performed by mutant ID rather than mutation site or mutation operator. Mutants originating from the same source location may share detection conditions, potentially making the reported generalization performance optimistic.

Questions for authors
  1. Can the authors provide additional evidence that the proposed mutation taxonomy reflects the error distribution of real LLM-generated GPU kernels? 2.Could the authors report performance under a mutation-site-level or mutation-rule-level held-out split?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 5JUi

We thank the reviewer for recognizing that oracle adequacy, which decides model evaluations and learning signals, can now be quantified, and that precision faults escape far more often than other families. In response to your concerns, we:

  • tested the taxonomy on real LLM errors: 397 new kernels from Codex and Claude Opus 5.5 yield 137 confirmed defects that map onto our fault families, with source-level examples from generated and published kernels (Q1);
  • replaced the ID split with site-level and rule-level holdouts: detection stays at 92.9% on held-out source sites and 91.0% on held-out rules, +11.2 and +10.4 pp over the official checker (Q2).

We will add these results to the revision and hope they address your concerns.

AuthorsOpenReview ↗

Response to Q1 and Q2

Q1. Does the taxonomy reflect real LLM errors?

We generated 397 new kernels with Codex and Claude, each in an isolated workspace with no access to our mutation rules, tests or reference kernels. Beyond the paper's two runtime failures, 137 of them have confirmed defects, and the 135 with a recorded failure signature map onto the taxonomy:

Observed failure signatureKernelsFamilies
Illegal or misaligned memory access54boundary, indexing
Wrong values on most elements39arithmetic, semantic, indexing
Wrong values on a subset (partial tiles, tails)15boundary
Overflow, non-finite, or FP16 overflow21precision
Wrong output shape or dtype6semantic

Both our new kernels and the published kernels show classic instances of each family. Below are some examples, copied from the kernel sources (... marks omitted lines).

Example 1. Boundary (tail): 119relu, Codex

The kernel processes the tensor as float4 (CUDA kernel, first lines below), and its Python launcher sets n4 = numel // 4 threads with no scalar tail (last line), so the last numel mod 4\mathit{numel} \bmod 4 elements are never written. The benchmark shape is divisible by 4, which hides the bug.

long long i = (long long)blockIdx.x * blockDim.x + threadIdx.x;
if (i >= n4) return;
float4 v = reinterpret_cast<const float4*>(x)[i];
...
reinterpret_cast<float4*>(out)[i] = v;
...
n4 = x.numel() // 4

Example 2. Precision (overflow): L2p67 GELU, CudaForge

The kernel computes tanh⁡z\tanh z as (e2z−1)/(e2z+1)(e^{2z}-1)/(e^{2z}+1). For large activations e2ze^{2z} overflows to inf and the result is ∞/∞=NaN\infty/\infty=\mathrm{NaN}.

exp_2z = tl.exp(2.0 * z)
tanh_z = (exp_2z - 1.0) / (exp_2z + 1.0)
gelu_block = 0.5 * x_block * (1.0 + tanh_z)

Example 3. Semantic (NaN handling): argmax, CudaForge

The kernel keeps a running maximum with a strict comparison. Since NaN > best_val is false, a NaN never wins, whereas torch.argmax returns the index of the NaN.

chunk_max = tl.max(vals, axis=0)
chunk_arg = tl.argmax(vals, axis=0) + k
better = chunk_max > best_val
best_val = tl.where(better, chunk_max, best_val)

Several matmul kernels also raise misaligned-address faults on sizes such as 1027×5151027 \times 515. The revision will add per-kernel family labels.

Q2. Holdouts by mutation site and mutation rule.

We select suites on the training half only and evaluate them on held-out mutants (137 tasks, 6,201 mutants, 5 seeds):

Held-out unitOfficial checkerOur suiteGain
Mutant ID81.7%94.0%+12.4 pp
Source-edit site81.7%92.9%+11.2 pp
Mutation rule80.6%91.0%+10.4 pp

Detection stays at 92.9% on held-out source sites and 91.0% on held-out rules, close to 94.0% under the ID split. Generalization is therefore not driven by mutants that share a source location.

Dear Reviewer 5JUi,

As the author–reviewer discussion deadline approaches, we wanted to gently follow up and let you know that we have uploaded the revised manuscript incorporating all the new experiments and analyses discussed in our response (highlighted in red for easy reading):

  • [Q1] Correspondence between the mutation taxonomy and real LLM kernel bugs across 397 newly generated LLM kernels (137 confirmed defects) and 492 published leaderboard kernels, along with failure-signature mappings and source-level defect examples (§6\S6, Table 13 in Appendix E.1, and Table 16 / code examples in Appendix E.6);
  • [Q2] Holdout validation split by source-edit site and by mutation rule, where our suites maintain 92.9% and 91.0% detection (+11.2 and +10.4 pp over the official checker) (§6\S6 and Table 14 in Appendix E.2).

We hope our responses and the revised paper address your concerns. Please let us know if you have any further questions or if there is anything else we can clarify before the discussion period concludes. Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.