- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper identifies the limitations of existing quantized GPU kernel benchmarks and proposes a new correctness contract for more reliable evaluation. The results show that while frontier models can generate correct kernels. But achieving high performance still remains challenging.
Critical strengths
- Strong Evaluation. The setting of this dataset is strong, including most recent LLMs and 84 kernels.
Critical weaknesses
-
The practical significance of the observed checker disagreements needs clearer characterization. Appendix D shows severe perplexity degradation for a truncation variant, whereas the two defects explicitly observed in generated kernels have negligible effects on the tested model. Could the authors provide end-to-end case studies of generated kernels that pass a specific published benchmark but are rejected by QKernelBench and affect the model's final performance?
-
The performance evaluation is limited to single-shot generation without execution feedback. While this is useful for measuring conformance to the stated specifications, it does not characterize how effectively models can optimize these kernels with compilation, correctness, and profiling feedback. An agentic evaluation under a fixed optimization budget, with multiple independent runs on frozen contracts, would make the benchmark more informative about optimization capability.
-
The presentation could be clearer, particularly the final sentence of the abstract. It combines contract revisions, newly admitted implementation choices, and several categories of remaining rejections in one sentence. Please separate these points and make the distinction between a contract-convention mismatch and an implementation error more explicit.
Questions for authors
See above.