acceptodds
Under review as a conference paper at ICLR 2027

QKernelBench: Benchmarking LLMs for Quantized GPU Kernel Generation

Abstract

Quantized GPU kernels are central to efficient LLM inference, yet kernel benchmarks judge them as they judge dense kernels: by comparing a candidate's outputs with a reference's under a tolerance. A quantizer that computes its own scale returns quantized values (codes) together with that scale, and the codes must be judged together with it: the same codes can be correct under a scale one ULP from the reference's and wrong under the reference's own. A check that compares codes and scale with the reference's separately rejects correct kernels, accepts wrong ones, or both. We introduce QKernelBench, a benchmark of 84 CUDA kernel contracts for quantized inference. Each contract states its numerical semantics clause by clause, and its verdict follows from them: codes are accepted when the returned scale is admissible and a declared arithmetic path reproduces them bit for bit, other codes must be exact outside declared one-step masks, and floating-point outputs must lie within declared bounds: a number of representable steps, a bound derived from the declared accumulation, or a tolerance. Every verdict is calibrated against mutants of its own contract and reports the positions it rejects. We judge kernels written by eight LLMs in two rounds, the second generated after the contracts were frozen. Frontier models solve 71 to 80 of the 84 contracts in the first round, but their kernels reach only a fraction of the speed of the shipping kernels we timed. On the same kernels, the correctness rules of published benchmarks err in both directions: they reject kernels whose scale differs lawfully from the reference's and accept kernels that break a stated clause, and no distance threshold separates the two. Under KernelBench's allclose, each frontier model would be credited with 14 to 18 fewer contracts. After repairs that the shipping kernels of eight production libraries prompted, the contracts accept their scale paths, reassociation and tensor-core accumulation; the kernels still rejected follow another convention for a stated clause, such as rounding, saturation or the zero guard, compute in a lower precision than the contract admits, or return codes that change between calls.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, GPU Kernel Generation, Quantization, Benchmarking, Numerical Correctness

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_igUFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies the limitations of existing quantized GPU kernel benchmarks and proposes a new correctness contract for more reliable evaluation. The results show that while frontier models can generate correct kernels. But achieving high performance still remains challenging.

Critical strengths
  1. Strong Evaluation. The setting of this dataset is strong, including most recent LLMs and 84 kernels.
Critical weaknesses
  1. The practical significance of the observed checker disagreements needs clearer characterization. Appendix D shows severe perplexity degradation for a truncation variant, whereas the two defects explicitly observed in generated kernels have negligible effects on the tested model. Could the authors provide end-to-end case studies of generated kernels that pass a specific published benchmark but are rejected by QKernelBench and affect the model's final performance?

  2. The performance evaluation is limited to single-shot generation without execution feedback. While this is useful for measuring conformance to the stated specifications, it does not characterize how effectively models can optimize these kernels with compilation, correctness, and profiling feedback. An agentic evaluation under a fixed optimization budget, with multiple independent runs on frozen contracts, would make the benchmark more informative about optimization capability.

  3. The presentation could be clearer, particularly the final sentence of the abstract. It combines contract revisions, newly admitted implementation choices, and several categories of remaining rejections in one sentence. Please separate these points and make the distinction between a contract-convention mismatch and an implementation error more explicit.

Questions for authors

See above.

Reviewer_WsivOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces QKernelBench, a benchmark of 84 CUDA kernel contracts for quantized LLM inference. It argues that a quantizer’s codes must be judged at the scale the kernel returns, since the same codes can be correct under one lawful scale and wrong under another. The contracts accept codes only if the returned scale is admissible and a declared arithmetic path reproduces them bit for bit. They are calibrated against 225 defect mutants and 93 lawful variants. Eight LLMs are evaluated in two rounds, and five published correctness rules are shown to both wrongly reject and wrongly accept kernels relative to the contracts.

Critical strengths

The idea is interesting and the core insight is convincing. The three-element INT8 construction shows that comparing codes and scale separately cannot be right in both directions, and the membership check is cheap.

The contracts are carefully engineered, with constructed boundary cases for every input-conditioned clause, mutant-based calibration, and a receipt for every decision.

The paper is transparent about its own instrument. It documents every post-hoc change to the corpus and runs a second round after freezing the contracts. It also checks the verdict layer with an independent implementation and across devices, and measures how loose its own tolerances are.

Critical weaknesses

The contracts are tested much less for false acceptance than for false rejection. None of the 21 post-hoc revisions turned a recorded acceptance into a rejection, and the second round can only reveal over-strict contracts, as Section 5.4 notes. The independent checker validates the campaign version, not the repairs. This leaves the 225 self-derived mutants as the main test of false acceptance, which I find insufficient for the 26 tolerance-based outputs, ten of which admit more than 5% of the median output magnitude.

The comparison with published rules relies on the authors’ own conventions and adjudication (Appendix M). Some “wrong” kernels have little numerical consequence: the FP4 +0.0 decoder matches TensorRT-LLM and changes no sum, and the observed clamp and tie defects change perplexity by at most 0.003. Without two of the authors’ adaptation choices, the calibrated tolerance also rejects no correct kernel. The claim that published rules accept wrong kernels should be scoped accordingly.

As a benchmark for LLMs, the corpus has limited headroom. Frontier models solve 71 to 80 of 84 contracts, speed is measured mainly against a PyTorch reference, and production kernels are compared on only 6 contracts. I see the contribution as an evaluation methodology rather than a benchmark of kernel-generation ability.

Beyond these points, two contracts named and described as FP8 use INT8 tensors, as Appendix J acknowledges, and they should be fixed before release. In the first round, 45 calls from GLM-5.3 and DeepSeek-V4-Pro received a second draw with a larger budget, which should be kept in mind when comparing models. The main text is also very dense. It moves between four corpus versions and many interlocking counts, and several key conclusions can only be understood with the appendix in hand. Moving the version bookkeeping to the appendix and giving the main text a simpler narrative would make the paper much easier to follow.

Questions for authors

Please see the weakness.

Reviewer_JV3DOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

Current methods to verify a GPU kernel’s output is to comparing the output elementwise to a reference implementation within an error tolerance (torch.allclose). This verification method is unreliable for quantized kernels because legitimate ways of computing the scale can differ in the last bit (ULP bit). QKernelBench introduces a suite of 84 specifications (contracts) for quantized kernels. Each contract specifies the allowed range for the scale, and verifies the codes exactly at the scale the kernel returns.

Critical strengths
  1. Addresses a very consequential problem for production libraries.
  2. Clean solution
Critical weaknesses
  1. Need to list all legitimate computation paths in advance.
  2. Writing is not good. Paper is hard to read.
Questions for authors

Please address the comments below:

  1. The issue the paper identifies is that two valid implementations of the same scale formula may differ in the last bit. The paper handles this by accepting each declared variant. This only works if every legitimate source of variation is enumerated in advance, and this method does not scale well. For example, according to Appendix K.4, some paths were added only after correct kernels had been rejected. If each contract instead specified one exact computation, every correct kernel would produce the same output and a simple exact comparison would be enough. Why did the authors not choose to do this? How many production kernels would fail if they did?

  2. Are the wrong kernels that existing graders accept harmful in practice? Seems like the maximum perplexity loss is only 0.003.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.