acceptodds
Under review as a conference paper at ICLR 2027

SparseKernelBench: A Benchmark for Sparse GPU Computation

Abstract

Sparse GPU performance depends on both the number and locations of nonzeros, requiring evaluation across input structures. We introduce SparseKernelBench, with 41 tasks across seven sparse representations, 533 performance cases and 637 correctness-only cases. Each task specifies input/output requirements and input families that a program must support. Four large language models generate programs without execution feedback for every task. Haiku produces no custom kernels. Programs from the other three models outperform the fastest tested library baseline, including compiled framework compositions, on 36.6–54.2% of the 533 performance cases. Requiring every task check to pass gives 36.0–44.3%, with the same denominator. Three of 103 programs that pass every performance-input check fail dedicated correctness-only inputs. At fixed dimensions, nonzero count and feature width, the fastest generated sampled dense–dense multiplication program takes 1.98× the fastest tested baseline’s runtime on uniform inputs but 0.54× on skewed inputs. Changing the baseline formulation also reverses whether programs outperform the library on 29–64 cases per model. These results show why sparse evaluation must establish the input domain, numerical requirements and tested comparator behind a performance claim.

Primary area:
datasets and benchmarks
Keywords:
sparse computation, GPU kernels, benchmarking, code generation, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_7oRxOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces SparseKernelBench, a benchmark for evaluating LLM generated GPU kernels on sparse computations. The Bench covers 41 tasks (from SpMV and SpMM over triangular solves, to sparse convolution, and GNN layers, sparse attention and MoE). The matrices are stored in 7 different sparse representations (typical sparse matrix formats). Each task comes with synthetic inputs according to different distributions (uniform, banded, etc). Also 78 SuiteSparse sparsity patterns are used. Each case compares against the fastest of up to 79 baselines. Input dependent preprocessing (format conversion etc) is timed in every call. The authors evaluate four LLM models for these tasks (claude-opus-5, claude-sonnet-5, claude-haiku-4-5 and gpt-6-astra). Each model generates one program per task, without execution feedback. Haiku did not produce any custom kernel as was thus excluded from the speedup analysis. The main findings are:

  • the generated programs beat the fastest tested baseline on 35 - 55% of the cases;
  • 3 of 103 programs that pass all performance input checks fail on correctness only inputs;
  • sparse structure heavily influence the performance results, e.g., an SDDMM program takes close to 2x the baseline's runtime on uniform inputs but about 0.5x on skewed inputs;
  • the choice of baseline formulation reverses up to 64 comparisons per model.

The paper builds directly on KernelBench (Ouyang et al., 2025), from which it borrows the metric and protocol. The paper’s novelty is a dedicated and broad sparse task suite. It has explicit structural inputs, per-task tolerances, correctness only edge cases and per input records of which baselines support which inputs. As far as I am aware, no previous LLM kernel benchmark covers sparse operations at this breadth. However, the three "consequential evaluation choices" that the paper presents as findings are well known:

  • Input structure. That sparse kernel performance depends on the input structure, and that the best format or kernel changes with it, is the premise of the majority of sparse matrix work especially on format selection and autotuning (e.g., SMAT, Li et al., 2013; WACO, Won et al., 2023).
  • Hidden inputs. That hidden or edge-case inputs reveal bugs in LLM generated kernels is the core message of robust-kbench and KernelBench-Verified.
  • Baseline choice. That the choice of baseline changes the verdict is also made by KernelBench-Verified and, more recently, FastKernels (Oliaro et al., 2026). Several related benchmarks are not cited: TritonBench (Li et al., 2025); ParEval (Nichols et al., 2024), which includes sparse and graph problem types for CUDA/HIP; FlashInfer-Bench (Xing et al., 2026); KernelBenchX (Wang et al., 2026) and FastKernels (Oliaro et al., 2026), which appeared close to the deadline; work on reward hacking in kernel generation, e.g., Kevin (Baronio et al., 2025).

TritonBench: J. Li, S. Li, Z. Gao et al. TritonBench: Benchmarking Large Language Model Capabilities for Generating Triton Operators. arXiv 2502.14752, 2025. ParEval: D. Nichols, J. H. Davis, Z. Xie, A. Rajaram, A. Bhatele. Can Large Language Models Write Parallel Code? HPDC 2024. DOI 10.1145/3625549.3658689. https://arxiv.org/abs/2401.12554 FlashInfer-Bench: S. Xing et al. FlashInfer-Bench: Building the Virtuous Cycle for AI-driven LLM Systems. arXiv 2601.00227, 2026. Kevin: C. Baronio, P. Marsella, B. Pan, S. Guo, S. Alberti. Kevin: Multi-Turn RL for Generating CUDA Kernels. arXiv 2507.11948, 2025. SMAT: J. Li, G. Tan, M. Chen, N. Sun. SMAT: An Input Adaptive Auto-Tuner for Sparse Matrix-Vector Multiplication. PLDI 2013. WACO: J. Won, C. Mendis, J. S. Emer, S. Amarasinghe. WACO: Learning Workload-Aware Co-optimization of the Format and Schedule of a Sparse Tensor Program. ASPLOS 2023. Libraries for stronger baselines: dgSPARSE (https://github.com/dgSPARSE), NVIDIA cuSPARSELt.

Critical strengths
  • The benchmark fills a real gap. Sparse and irregular computations are underrepresented in existing kernel benchmarks, but they matter for GNNs, MoE, sparse attention and recommender systems.
  • Task contracts are explicit (index types, empty rows, stored zeros, tolerances that scale with reduction length). The suites can be reconstructed from hashes, and code is provided.
Critical weaknesses
  • The baselines are weak. In my opinion this is the most important issue: the headline speedups mostly measure the strength of the baselines. Where a direct cuSPARSE adapter is the fastest baseline, the models win in very few cases. Where a compiled PyTorch composition is the fastest baseline, they win in a majority cases. Specialized state-of-the-art sparse kernels are not used as baselines, although the paper cites some of them: Sputnik, GE-SpMM, dgSPARSE, cuSPARSELt, and fused GNN kernels from DGL/PyG. torch.compile is only used in its default mode. So "faster than the fastest tested baseline" does not mean "faster than the state of the art. Even on the synthetic suite, the geometric mean speedup is below 1 for two of the three models (Opus 1.37x, Sonnet 0.67x, Astra 0.91x).
  • The results on real matrices contradict the synthetic numbers. On the 78 SuiteSparse patterns the models are faster than the baseline in less than10% of SpMV cases and less than 6% of SpMM cases. No program is more than 2x faster, and all geometric mean speedups are low. The appendix shows that the real patterns differ substantially from the synthetic inputs. This raises the question of whether the synthetic suite is representative at all, and the real-matrix study should arguably be part of the main evaluation.
  • Thin model study is narrow. Only three models produce usable programs. Each generates a single sample per task without execution or profiling feedback, which is far from how kernel generation is done today (agentic or evolutionary loops with feedback). That Haiku fails terribly raises the question of whether the prompt or contract was too demanding for weaker models.
  • Timing does not consider amortization of preprocessing. Many real sparse workloads reuse a fixed sparsity pattern: iterative solvers, GNN training on a fixed graph, pruned weights. In these settings, format conversion and analysis are amortized Timing them on every call penalizes libraries and preprocessing-heavy algorithms.
  • The evaluation scope is narrow. All measurements use one GPU (NVIDIA L40S) and FP32 only. Especially the fact that the 2:4 task is run in FP32 means that the sparse tensor-core path is never used and the fastest baseline is a dense GEMM.
  • The text is very difficult to read and the comparisons are spread over more than 30 appendix pages.
Questions for authors
  1. How do the results change against state of the art sparse kernels (Sputnik, GE-SpMM/dgSPARSE, cuSPARSELt with FP16/BF16 for 2:4) and against torch.compile with max-autotune/CUDA graphs?
  2. Any idea why the synthetic suite and the SuiteSparse study diverge so strongly?
  3. Can you report an amortized regime (fixed pattern, changing values) and how the rankings would change?
  4. How do the models perform with execution and profiling feedback?
  5. Any idea why Haiku produce no custom kernel at all?
Reviewer_ha4iOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SparseKernelBench, a benchmark for sparse GPU computation. The benchmark covers 41 computational tasks across seven sparse representations, including 533 performance test cases and 637 additional correctness test cases. The authors evaluate the correctness and runtime performance of GPU programs generated by four large language models and analyze how different sparsity patterns, correctness tests, and performance baselines affect the evaluation results.

Critical strengths
  • Broad benchmark coverage. The benchmark covers 41 computational tasks across seven sparse representations, incorporating diverse sparsity patterns, performance tests, and additional correctness tests to provide comprehensive evaluation scenarios.
  • Extensive experiments and detailed analysis. The authors evaluate GPU programs generated by multiple large language models and analyze how different sparsity patterns, performance baselines, and scoring criteria affect the evaluation results.
Critical weaknesses
  • Limited novelty. The paper primarily integrates and extends existing sparse GPU computational tasks, input data, and evaluation procedures, with limited methodological innovation. Its substantive differences from existing benchmarks are not sufficiently clear.
  • Lack of direct comparisons with existing benchmarks. Although the paper discusses related work, it does not systematically compare SparseKernelBench with existing benchmarks in terms of error detection, test coverage, and performance evaluation, making it difficult to establish the unique value of the proposed benchmark.
  • Insufficient justification for the representativeness of the benchmark. Although the authors explain the motivation behind different sparsity patterns, they do not sufficiently demonstrate that the selected 41 tasks and their test inputs are representative of real-world sparse GPU workloads.
Questions for authors
  1. What are the most significant and unique contributions of SparseKernelBench compared with existing benchmarks? Could the authors further clarify how its evaluation design substantively differs from prior work?
  2. What is the rationale behind the selection of benchmark tasks and test inputs? Could the authors explain why these 41 tasks were selected and how the proportions of different tasks and sparsity patterns were determined? How sensitive are the evaluation results to these choices?
  3. Can the benchmark generalize to other GPU platforms? Since the experiments are primarily conducted on NVIDIA L40S GPUs, could the authors validate the results on other GPU architectures?
Reviewer_RnshOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper uses a collection of sparse inputs and tasks to compare the performance of sparse kernels generated by different LLMs in terms of performance and correctness.

Critical strengths

The paper provides ample details regarding the testing harness, and the evaluation is an extensive 33-page appendix. I could only read parts of it, as it contained a considerable amount of information.

Critical weaknesses

I was unable to follow the paper's motivation from the abstract through the entire 10-page primary submission. Usually, this would call for the authors to provide a section to outline the contributions, but I think the primary contribution is comparing the ability of LLMs to generate sparse kernels that more closely meet the needs of the input application. Is that right? If that is the correct takeaway, I have to respectfully ask, so what? What was I supposed to learn from this exercise? Researchers have studied sparse inputs for evaluating different input domains and problem specifications for decades, and they have introduced many real and synthetic benchmarks to assess the utility of countless approaches. I can only imagine that LLMs are regurgitating sparse kernels built from prior work, and prompting seems to provide some benefit but nothing noteworthy.

I'll leave my weaknesses here, as I think I would be better served waiting for more clarification on the contributions and substantial effort to clean up the motivation throughout the entire paper.

Questions for authors

This text has no flow. For example, as I read the abstract it says: "Four large language models generate programs without execution feedback for every task. Haiku produces no custom kernels." At this point why would I care that Haiku produces no custom kernels? I have no idea what four LLMs were used or for what purpose. There are numerous examples of this same pattern throughout the entire text, with surprising statements with very little motivation. What should I take away from reading this paper?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.