- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies using LLMs as surrogate models to predict GPU kernel performance and reduce the number of expensive hardware evaluations during kernel optimization. The authors evaluate several off-the-shelf LLMs, integrate a surrogate into evolutionary search, and use RL to improve forecasting calibration. The proposed search matches or outperforms the GPU-only baseline on 4 out of 6 tasks.
Critical strengths
The paper addresses a practical problem in GPU kernel optimization. Reducing the cost of hardware evaluations could be useful for scaling LLM-based kernel search.
The idea of using LLMs to predict kernel performance is interesting, and the paper provides empirical evidence that LLMs can identify promising candidates without executing them.
Critical weaknesses
-
Limited complexity and representativeness of the evaluated kernels. My main concern is that the evaluated tasks do not sufficiently represent the complexity of modern GPU kernel optimization. In particular, the paper does not demonstrate performance forecasting for advanced optimization techniques such as warp specialization, producer-consumer role splitting, or complex pipeline scheduling. These optimizations involve nontrivial hardware interactions and performance tradeoffs that are difficult to predict from source code alone. It is unclear whether the proposed approach would work on these more challenging kernels.
-
Limited technical novelty. Using surrogate models to guide optimization is a well-established idea. The proposed approach mainly replaces traditional cost models with LLMs, without introducing substantial new modeling or optimization techniques.
Questions for authors
-
Can the authors evaluate the surrogate on more complex and realistic GPU kernels on Hopper and Blackwell, particularly those involving warp specialization, producer-consumer role splitting, and pipeline scheduling? For example, deciding whether 2 or 4 warp groups perform better can be highly workload-dependent. How well can the surrogate capture these subtle performance differences?
-
How does the LLM-based surrogate compare against traditional learned cost models? What advantages does an LLM provide, especially when generalizing to unseen kernel structures or GPU architectures?
7 comments
References
[1] Abramson et al. Accurate structure prediction of biomolecular interactions with AlphaFold 3. Nature, 2024. https://www.nature.com/articles/s41586-024-07487-w
[2] Yang, Kautz, Hatamizadeh. Gated Delta Networks: Improving Mamba2 with Delta Rule. ICLR 2025. https://arxiv.org/abs/2412.06464
[3] Qwen Team. Qwen3-Next. https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct
[4] Kimi Team. Kimi Linear: An Expressive, Efficient Attention Architecture. 2025. https://arxiv.org/abs/2510.26692
[5] Ouyang et al. KernelBench: Can LLMs Write Efficient GPU Kernels? 2025. https://arxiv.org/abs/2502.10517
[6] He et al. Deep Residual Learning for Image Recognition. CVPR 2016. https://arxiv.org/abs/1512.03385
[7] Simonyan and Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. ICLR 2015. https://arxiv.org/abs/1409.1556
[8] Dosovitskiy et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ICLR 2021. https://arxiv.org/abs/2010.11929
[9] Liu et al. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. ICCV 2021. https://arxiv.org/abs/2103.14030
[10] Dao and Gu. Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality. ICML 2024. https://arxiv.org/abs/2405.21060
[11] NVIDIA. Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models. 2025. https://arxiv.org/abs/2504.03624
[12] Air Street Capital. State of AI Compute Index, June 2026. https://press.airstreet.com/p/state-of-ai-compute-index-june-2026
[13] NVIDIA Q3 FY2026 earnings call, November 2025, as reported by Stocktwits. https://stocktwits.com/news-articles/markets/equity/nvidia-a100-gpu-shipped-6-years-ago-full-utilization-michael-burry-depreciation/cLPALD9RE9w
[14] GetDeploying, NVIDIA L40S cloud providers, accessed November 2026. https://getdeploying.com/reference/cloud-gpu/nvidia-l40s
[15] Mert Yuksekgonul, Daniel Koceja, Xinhao Li, Federico Bianchi, Jed McCaleb, Xiaolong Wang, Jan Kautz, Yejin Choi, James Zou, Carlos Guestrin, Yu Sun. Learning to Discover at Test Time. ICML 2026. arXiv:2601.16175. https://arxiv.org/abs/2601.16175
[16] Shiyi Cao, Ziming Mao, Joseph E. Gonzalez, Ion Stoica. K-Search: LLM Kernel Generation via Co-Evolving Intrinsic World Model. arXiv:2602.19128, 2026. https://arxiv.org/abs/2602.19128
[17[ Bolet, G., Georgakoudis, G., Menon, H., Parasyris, K., Hasabnis, N., Estes, H., ... & Oren, G. (2025, July). Can large language models predict parallel code performance?. In Proceedings of the 34th International Symposium on High-Performance Parallel and Distributed Computing (pp. 1-6).
Dear Reviewer cjso,
There are 3 days left in the discussion period. In our response we addressed your questions on kernel complexity, novelty, and the comparison with learned cost models. If there is anything preventing you from raising your score, please let us know and we would be happy to discuss things further.
We thank the reviewer for noting that the paper addresses a practical problem, and that it gives empirical evidence that LLMs can identify promising kernels without running them.
W1 / Q1: Kernel complexity and newer architectures
Our tasks and hardware match prior LLM kernel-search work (TTT-Discover [15], K-Search [16]). Evaluating the surrogate on warp-specialized Hopper and Blackwell kernels would require building a new set of such kernels and timing them on a different, more expensive set of GPUs. We leave this to future work and have stated it as a limitation in the conclusion (Section 5).
Our kernels come from widely used workloads and use common, standard GPU optimizations. The evaluation covers 21 tasks. Six come from the GPU Mode competition, including TriMul from AlphaFold 3 [1] and three Gated DeltaNet kernels. Gated DeltaNet [2] was developed at NVIDIA and is a core layer of recent efficient language models, including Qwen3-Next [3] and, in a refined form, Kimi Linear [4]. The other 15 are complete networks from KernelBench Level 3 [5], including highly-cited, widely used models such as ResNet [6], VGG [7], Vision Transformer [8], Swin Transformer [9] , and Mamba-2 [10], which NVIDIA uses in its Nemotron-H models [11]. The Triton kernels use tensor-core matrix multiplies and multi-stage software pipelining. The CUDA kernels use shared-memory tiling, warp-shuffle reductions, vectorized memory access, and kernel fusion.
A100 and L40S GPUs remain heavily used. In 2026, more AI research papers report using the A100 (15,327) than the H100 and H200 combined (9,823), and seventeen times as many as Blackwell GPUs (902) [12]. NVIDIA reported in November 2025 that A100s it shipped six years ago are still fully utilized [13]. The L40S is rented by 39 cloud providers, including AWS, Oracle, and CoreWeave [14].
W2: Novelty
In W1, you note that kernel performance is difficult to predict from source code alone. We agree with this, and believe it makes our empirical results particularly interesting as prior work has not shown that an LLM can predict kernel performance well enough to guide search. LLM kernel searches such as TTT-Discover and K-Search measure every candidate on the GPU. Learned cost models such as AutoTVM, Ansor, and TLP need training measurements for each operator and cannot score free-form Triton or CUDA code (see Q2). Recent work predicts selected properties of a single kernel from its source, such as performance counters (Omniwise) or whether it is compute- or memory-bound [17], but does not test whether these predictions can be used to improve the efficiency of LLM-driven search for kernels.
We also make the following contributions:
Q2: Comparison with learned cost models
AutoTVM, Ansor, and TLP are trained to rank schedules of a fixed operator, so they do not directly apply to our setting, which requires predictions about arbitrary Triton and CUDA kernels generated during LLM-driven kernel search. Learned cost models use features from the compiler's representation of a fixed operator. An LLM-written kernel frequently changes the algorithm, memory layout, and launch structure, which are not described by such features. The LLM also needs no training data for a new task or device. The off-the-shelf surrogates in Table 1 have never seen measurements from our tasks. The surrogate trained with correctness and Brier rewards was trained on one task and ranks kernels better on the five held-out tasks, from 84.1% to 88.7% (Table 2). We release the 12,388 measured kernels so that learned predictors for this setting can be trained and compared.