- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper introduces SparseKernelBench, a benchmark for evaluating LLM generated GPU kernels on sparse computations. The Bench covers 41 tasks (from SpMV and SpMM over triangular solves, to sparse convolution, and GNN layers, sparse attention and MoE). The matrices are stored in 7 different sparse representations (typical sparse matrix formats). Each task comes with synthetic inputs according to different distributions (uniform, banded, etc). Also 78 SuiteSparse sparsity patterns are used. Each case compares against the fastest of up to 79 baselines. Input dependent preprocessing (format conversion etc) is timed in every call. The authors evaluate four LLM models for these tasks (claude-opus-5, claude-sonnet-5, claude-haiku-4-5 and gpt-6-astra). Each model generates one program per task, without execution feedback. Haiku did not produce any custom kernel as was thus excluded from the speedup analysis. The main findings are:
- the generated programs beat the fastest tested baseline on 35 - 55% of the cases;
- 3 of 103 programs that pass all performance input checks fail on correctness only inputs;
- sparse structure heavily influence the performance results, e.g., an SDDMM program takes close to 2x the baseline's runtime on uniform inputs but about 0.5x on skewed inputs;
- the choice of baseline formulation reverses up to 64 comparisons per model.
The paper builds directly on KernelBench (Ouyang et al., 2025), from which it borrows the metric and protocol. The paper’s novelty is a dedicated and broad sparse task suite. It has explicit structural inputs, per-task tolerances, correctness only edge cases and per input records of which baselines support which inputs. As far as I am aware, no previous LLM kernel benchmark covers sparse operations at this breadth. However, the three "consequential evaluation choices" that the paper presents as findings are well known:
- Input structure. That sparse kernel performance depends on the input structure, and that the best format or kernel changes with it, is the premise of the majority of sparse matrix work especially on format selection and autotuning (e.g., SMAT, Li et al., 2013; WACO, Won et al., 2023).
- Hidden inputs. That hidden or edge-case inputs reveal bugs in LLM generated kernels is the core message of robust-kbench and KernelBench-Verified.
- Baseline choice. That the choice of baseline changes the verdict is also made by KernelBench-Verified and, more recently, FastKernels (Oliaro et al., 2026). Several related benchmarks are not cited: TritonBench (Li et al., 2025); ParEval (Nichols et al., 2024), which includes sparse and graph problem types for CUDA/HIP; FlashInfer-Bench (Xing et al., 2026); KernelBenchX (Wang et al., 2026) and FastKernels (Oliaro et al., 2026), which appeared close to the deadline; work on reward hacking in kernel generation, e.g., Kevin (Baronio et al., 2025).
TritonBench: J. Li, S. Li, Z. Gao et al. TritonBench: Benchmarking Large Language Model Capabilities for Generating Triton Operators. arXiv 2502.14752, 2025. ParEval: D. Nichols, J. H. Davis, Z. Xie, A. Rajaram, A. Bhatele. Can Large Language Models Write Parallel Code? HPDC 2024. DOI 10.1145/3625549.3658689. https://arxiv.org/abs/2401.12554 FlashInfer-Bench: S. Xing et al. FlashInfer-Bench: Building the Virtuous Cycle for AI-driven LLM Systems. arXiv 2601.00227, 2026. Kevin: C. Baronio, P. Marsella, B. Pan, S. Guo, S. Alberti. Kevin: Multi-Turn RL for Generating CUDA Kernels. arXiv 2507.11948, 2025. SMAT: J. Li, G. Tan, M. Chen, N. Sun. SMAT: An Input Adaptive Auto-Tuner for Sparse Matrix-Vector Multiplication. PLDI 2013. WACO: J. Won, C. Mendis, J. S. Emer, S. Amarasinghe. WACO: Learning Workload-Aware Co-optimization of the Format and Schedule of a Sparse Tensor Program. ASPLOS 2023. Libraries for stronger baselines: dgSPARSE (https://github.com/dgSPARSE), NVIDIA cuSPARSELt.
Critical strengths
- The benchmark fills a real gap. Sparse and irregular computations are underrepresented in existing kernel benchmarks, but they matter for GNNs, MoE, sparse attention and recommender systems.
- Task contracts are explicit (index types, empty rows, stored zeros, tolerances that scale with reduction length). The suites can be reconstructed from hashes, and code is provided.
Critical weaknesses
- The baselines are weak. In my opinion this is the most important issue: the headline speedups mostly measure the strength of the baselines. Where a direct cuSPARSE adapter is the fastest baseline, the models win in very few cases. Where a compiled PyTorch composition is the fastest baseline, they win in a majority cases. Specialized state-of-the-art sparse kernels are not used as baselines, although the paper cites some of them: Sputnik, GE-SpMM, dgSPARSE, cuSPARSELt, and fused GNN kernels from DGL/PyG. torch.compile is only used in its default mode. So "faster than the fastest tested baseline" does not mean "faster than the state of the art. Even on the synthetic suite, the geometric mean speedup is below 1 for two of the three models (Opus 1.37x, Sonnet 0.67x, Astra 0.91x).
- The results on real matrices contradict the synthetic numbers. On the 78 SuiteSparse patterns the models are faster than the baseline in less than10% of SpMV cases and less than 6% of SpMM cases. No program is more than 2x faster, and all geometric mean speedups are low. The appendix shows that the real patterns differ substantially from the synthetic inputs. This raises the question of whether the synthetic suite is representative at all, and the real-matrix study should arguably be part of the main evaluation.
- Thin model study is narrow. Only three models produce usable programs. Each generates a single sample per task without execution or profiling feedback, which is far from how kernel generation is done today (agentic or evolutionary loops with feedback). That Haiku fails terribly raises the question of whether the prompt or contract was too demanding for weaker models.
- Timing does not consider amortization of preprocessing. Many real sparse workloads reuse a fixed sparsity pattern: iterative solvers, GNN training on a fixed graph, pruned weights. In these settings, format conversion and analysis are amortized Timing them on every call penalizes libraries and preprocessing-heavy algorithms.
- The evaluation scope is narrow. All measurements use one GPU (NVIDIA L40S) and FP32 only. Especially the fact that the 2:4 task is run in FP32 means that the sparse tensor-core path is never used and the fastest baseline is a dense GEMM.
- The text is very difficult to read and the comparisons are spread over more than 30 appendix pages.
Questions for authors
- How do the results change against state of the art sparse kernels (Sputnik, GE-SpMM/dgSPARSE, cuSPARSELt with FP16/BF16 for 2:4) and against torch.compile with max-autotune/CUDA graphs?
- Any idea why the synthetic suite and the SuiteSparse study diverge so strongly?
- Can you report an amortized regime (fixed pattern, changing values) and how the rankings would change?
- How do the models perform with execution and profiling feedback?
- Any idea why Haiku produce no custom kernel at all?