acceptodds
Under review as a conference paper at ICLR 2027

GPU Forecasters: Language Models as Speculative Surrogates for Kernel Runtime Optimization

Abstract

GPU kernels are the workhorse of modern deep learning, and optimizing them (via evolutionary search or coding agents) usually requires repeated measurement on target hardware. While these measurements provide the ground-truth signal necessary for kernel search, they are costly, because each evaluation of a kernel requires compilation and repeated execution on a GPU. As improvements in LLM inference reduce the cost of writing novel kernels and LLM-driven searches scale to large search budgets, on-device evaluation becomes a bottleneck. To address this, we study how LLMs can serve as calibrated, speculative surrogates of the GPU: probabilistic performance models that forecast the speedup of a candidate kernel from its source. During a kernel search algorithm’s generation phase, the surrogate forecasts runtimes for generated kernel, and only the top-ranked kernels are scored by the GPU. The search uses the speculative forecasts by the surrogate as scores for the remaining kernels. We first evaluate four off-the-shelf LLMs as surrogates. We find that LLMs can act as effective surrogates, helping searches find fast kernels while measuring only a fraction of generated candidates on the GPU. On 1,347 parent-child pairs from real searches, LLMs identify discovery moments, i.e., children that are much faster than their parents, at rates above chance. We then integrate a surrogate into evolutionary kernel search to prioritize candidates for GPU measurement. Measuring only the most promising generated candidates, this search matches or outperforms GPU-only search on most benchmark tasks. Finally, we train a surrogate with reinforcement learning and find that calibration rewards improve calibration without hurting ranking, while correctness rewards alone make confidence less reliable. We release the 12,388 measured LLM-generated kernels produced by these searches. These results suggest that LLMs can play a broader role in kernel optimization, by acting as virtual models of a GPU rather than solely as kernel generators for search.

Primary area:
foundation or frontier models, including LLMs
Keywords:
calibration, forecasting, kernels, evolutionary search, program synthesis, reinforcement learning, world modeling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cjsoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies using LLMs as surrogate models to predict GPU kernel performance and reduce the number of expensive hardware evaluations during kernel optimization. The authors evaluate several off-the-shelf LLMs, integrate a surrogate into evolutionary search, and use RL to improve forecasting calibration. The proposed search matches or outperforms the GPU-only baseline on 4 out of 6 tasks.

Critical strengths

The paper addresses a practical problem in GPU kernel optimization. Reducing the cost of hardware evaluations could be useful for scaling LLM-based kernel search.

The idea of using LLMs to predict kernel performance is interesting, and the paper provides empirical evidence that LLMs can identify promising candidates without executing them.

Critical weaknesses
  1. Limited complexity and representativeness of the evaluated kernels. My main concern is that the evaluated tasks do not sufficiently represent the complexity of modern GPU kernel optimization. In particular, the paper does not demonstrate performance forecasting for advanced optimization techniques such as warp specialization, producer-consumer role splitting, or complex pipeline scheduling. These optimizations involve nontrivial hardware interactions and performance tradeoffs that are difficult to predict from source code alone. It is unclear whether the proposed approach would work on these more challenging kernels.

  2. Limited technical novelty. Using surrogate models to guide optimization is a well-established idea. The proposed approach mainly replaces traditional cost models with LLMs, without introducing substantial new modeling or optimization techniques.

Questions for authors
  1. Can the authors evaluate the surrogate on more complex and realistic GPU kernels on Hopper and Blackwell, particularly those involving warp specialization, producer-consumer role splitting, and pipeline scheduling? For example, deciding whether 2 or 4 warp groups perform better can be highly workload-dependent. How well can the surrogate capture these subtle performance differences?

  2. How does the LLM-based surrogate compare against traditional learned cost models? What advantages does an LLM provide, especially when generalizing to unseen kernel structures or GPU architectures?

7 comments

We thank the reviewer for noting that the paper addresses a practical problem, and that it gives empirical evidence that LLMs can identify promising kernels without running them.

W1 / Q1: Kernel complexity and newer architectures

Our tasks and hardware match prior LLM kernel-search work (TTT-Discover [15], K-Search [16]). Evaluating the surrogate on warp-specialized Hopper and Blackwell kernels would require building a new set of such kernels and timing them on a different, more expensive set of GPUs. We leave this to future work and have stated it as a limitation in the conclusion (Section 5).

Our kernels come from widely used workloads and use common, standard GPU optimizations. The evaluation covers 21 tasks. Six come from the GPU Mode competition, including TriMul from AlphaFold 3 [1] and three Gated DeltaNet kernels. Gated DeltaNet [2] was developed at NVIDIA and is a core layer of recent efficient language models, including Qwen3-Next [3] and, in a refined form, Kimi Linear [4]. The other 15 are complete networks from KernelBench Level 3 [5], including highly-cited, widely used models such as ResNet [6], VGG [7], Vision Transformer [8], Swin Transformer [9] , and Mamba-2 [10], which NVIDIA uses in its Nemotron-H models [11]. The Triton kernels use tensor-core matrix multiplies and multi-stage software pipelining. The CUDA kernels use shared-memory tiling, warp-shuffle reductions, vectorized memory access, and kernel fusion.

A100 and L40S GPUs remain heavily used. In 2026, more AI research papers report using the A100 (15,327) than the H100 and H200 combined (9,823), and seventeen times as many as Blackwell GPUs (902) [12]. NVIDIA reported in November 2025 that A100s it shipped six years ago are still fully utilized [13]. The L40S is rented by 39 cloud providers, including AWS, Oracle, and CoreWeave [14].

W2: Novelty

In W1, you note that kernel performance is difficult to predict from source code alone. We agree with this, and believe it makes our empirical results particularly interesting as prior work has not shown that an LLM can predict kernel performance well enough to guide search. LLM kernel searches such as TTT-Discover and K-Search measure every candidate on the GPU. Learned cost models such as AutoTVM, Ansor, and TLP need training measurements for each operator and cannot score free-form Triton or CUDA code (see Q2). Recent work predicts selected properties of a single kernel from its source, such as performance counters (Omniwise) or whether it is compute- or memory-bound [17], but does not test whether these predictions can be used to improve the efficiency of LLM-driven search for kernels.

We also make the following contributions:

  1. We use an LLM surrogate inside LLM-driven evolutionary kernel search. Existing LLM kernel searches time every candidate on the GPU. For example, TTT-Discover uses 25,600 H100 measurements to optimize a kernel. Ansor uses a learned cost model inside evolutionary search, but only over schedules of a fixed operator, so it can’t score arbitrary programs written by LLMs.
  2. We build two benchmarks for kernel surrogates from real search runs. The first evaluates how close a surrogate's top picks come to the fastest of 480 kernels for six GPU Mode tasks. The second benchmark evaluates the surrogates ability to find the rare mutations which make a kernel much faster across 1,347 edits from evolutionary searches on 21 kernel tasks.
  3. We release the 12,388 LLM-written kernels from our searches, with runtimes measured across 118 kernel problems, CUDA and Triton, and three GPU types.
  4. We evaluate how RL training changes a kernel surrogate. We study how both correctness only GRPO and RL for calibration / forecasting (RLCR) affects the properties of the surrogate.

Q2: Comparison with learned cost models

AutoTVM, Ansor, and TLP are trained to rank schedules of a fixed operator, so they do not directly apply to our setting, which requires predictions about arbitrary Triton and CUDA kernels generated during LLM-driven kernel search. Learned cost models use features from the compiler's representation of a fixed operator. An LLM-written kernel frequently changes the algorithm, memory layout, and launch structure, which are not described by such features. The LLM also needs no training data for a new task or device. The off-the-shelf surrogates in Table 1 have never seen measurements from our tasks. The surrogate trained with correctness and Brier rewards was trained on one task and ranks kernels better on the five held-out tasks, from 84.1% to 88.7% (Table 2). We release the 12,388 measured kernels so that learned predictors for this setting can be trained and compared.

References

[1] Abramson et al. Accurate structure prediction of biomolecular interactions with AlphaFold 3. Nature, 2024. https://www.nature.com/articles/s41586-024-07487-w

[2] Yang, Kautz, Hatamizadeh. Gated Delta Networks: Improving Mamba2 with Delta Rule. ICLR 2025. https://arxiv.org/abs/2412.06464

[3] Qwen Team. Qwen3-Next. https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct

[4] Kimi Team. Kimi Linear: An Expressive, Efficient Attention Architecture. 2025. https://arxiv.org/abs/2510.26692

[5] Ouyang et al. KernelBench: Can LLMs Write Efficient GPU Kernels? 2025. https://arxiv.org/abs/2502.10517

[6] He et al. Deep Residual Learning for Image Recognition. CVPR 2016. https://arxiv.org/abs/1512.03385

[7] Simonyan and Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. ICLR 2015. https://arxiv.org/abs/1409.1556

[8] Dosovitskiy et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ICLR 2021. https://arxiv.org/abs/2010.11929

[9] Liu et al. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. ICCV 2021. https://arxiv.org/abs/2103.14030

[10] Dao and Gu. Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality. ICML 2024. https://arxiv.org/abs/2405.21060

[11] NVIDIA. Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models. 2025. https://arxiv.org/abs/2504.03624

[12] Air Street Capital. State of AI Compute Index, June 2026. https://press.airstreet.com/p/state-of-ai-compute-index-june-2026

[13] NVIDIA Q3 FY2026 earnings call, November 2025, as reported by Stocktwits. https://stocktwits.com/news-articles/markets/equity/nvidia-a100-gpu-shipped-6-years-ago-full-utilization-michael-burry-depreciation/cLPALD9RE9w

[14] GetDeploying, NVIDIA L40S cloud providers, accessed November 2026. https://getdeploying.com/reference/cloud-gpu/nvidia-l40s

[15] Mert Yuksekgonul, Daniel Koceja, Xinhao Li, Federico Bianchi, Jed McCaleb, Xiaolong Wang, Jan Kautz, Yejin Choi, James Zou, Carlos Guestrin, Yu Sun. Learning to Discover at Test Time. ICML 2026. arXiv:2601.16175. https://arxiv.org/abs/2601.16175

[16] Shiyi Cao, Ziming Mao, Joseph E. Gonzalez, Ion Stoica. K-Search: LLM Kernel Generation via Co-Evolving Intrinsic World Model. arXiv:2602.19128, 2026. https://arxiv.org/abs/2602.19128

[17[ Bolet, G., Georgakoudis, G., Menon, H., Parasyris, K., Hasabnis, N., Estes, H., ... & Oren, G. (2025, July). Can large language models predict parallel code performance?. In Proceedings of the 34th International Symposium on High-Performance Parallel and Distributed Computing (pp. 1-6).

Dear Reviewer cjso,

There are 3 days left in the discussion period. In our response we addressed your questions on kernel complexity, novelty, and the comparison with learned cost models. If there is anything preventing you from raising your score, please let us know and we would be happy to discuss things further.

Reviewer_dLmoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether LLMs can reduce the cost of GPU kernel search by predicting which candidates are worth measuring. The proposed method measures top-ranked candidates on the GPU and keeps the unmeasured candidates in the search archive with speculative scores. The authors also train GPT-OSS-20B with calibration rewards. Experiments evaluate forecast quality and compare search with and without the surrogate under a fixed GPU measurement budget.

Critical strengths
  1. The paper addresses a practical problem. Reducing evaluation cost is a complementary direction to improving kernel generation, especially when a search produces many candidates.

  2. Including calibration rewards makes sense. The search ranks candidates by expected speedup under the forecast distribution, while the correctness reward only checks whether the predicted bin is correct.

Critical weaknesses
  1. The forecasts look useful for screening implementations with large performance differences, but I am less sure they can guide optimization once the search has found a good kernel. In the evaluation of discovery moments, the off-the-shelf surrogates are near random at small speedup thresholds. Small improvements become important as optimization progresses, particularly when the search uses forecasts to select parents to expand. The paper does not show clearly whether the surrogate continues to help at this stage or mainly saves measurements on weak candidates early in search.

  2. The practical efficiency gain is still unclear. The proposed search generates four times as many candidates, whereas the main comparison fixes the GPU measurement budget. This may be a good trade-off, but it does not show whether the same measured speedup can be reached sooner or more cheaply. The per-kernel cost comparison does not resolve this, since the additional generation and forecasting need to be considered together over a complete search.

Questions for authors
  1. If the search starts from an already well-optimized kernel, does the surrogate still help the search? How does it behave when most useful edits give relatively small speedups?

  2. How much does keeping unmeasured candidates in the search archive contribute compared with using the same surrogate only for pre-screening?

  3. How do the methods compare in total wall-clock time and cost, including candidate generation and forecasting? Is on-device evaluation still the bottleneck in these runs?

  4. How does training with calibration rewards change the end-to-end search results compared with using the untrained surrogate? Among these questions, 1 and 3 are the biggest concerns I have about this paper.

3 comments

We thank the reviewer for noting that reducing evaluation cost complements work on kernel generation, and that calibration rewards make sense because the search ranks candidates by expected speedup.

W1 / Q1: Does the surrogate help once a good kernel is found?

Yes, surrogate is better than random at finding small improvements late in the search. A kernel that is only 5–25% faster than its parent is 47% more likely to be picked by the surrogate used in our searches than by random selection. We measured this on the same kernels as Figure 3. Note that random selection is a strong baseline as standard evolutionary search (non-surrogate assisted) uses random selection in practice. Generating 16 kernels and randomly choosing 4 (random selection baseline) is the same as generating 4 kernels and choosing 4 (non-surrogate assisted evolutionary search).

Small improvements make up most of each search. In five of the six baseline searches, the best kernel after 25 GPU measurements is within 11% of the final best, and the remaining 290 or so measurements go toward improving it. As the kernel searches progressed from 25 to 260 measurements, the surrogate search improved its best kernel by 12% on FP8 quantization, 40% on GDN ChunkFwd-o, and 11% on GDN Recompute W/U, compared with 4%, 12%, and 8% for the baseline.

In our kernel searches, forecasts never choose which kernel to expand next. Only kernels measured on the GPU enter the archive and can be expanded (point 2 of the general response).

W2 / Q3: Total wall clock and cost

Please see point 1 of the general response. The savings are larger for CUDA kernels, where compilation makes each measurement cost 7.5x more than for Triton (Table 3).

Q2: Keeping unmeasured candidates vs. pre-screening only

Please see point 2 of the general response. We thank the reviewer for this question, which helped us describe the search more clearly in Section 2.1.

Q4: Effect of RL training on search

We expect search with the untrained model to give similar results, because RL training changed how the surrogate ranks kernels very little. In Table 2, the untrained and trained models score 82.9 ± 3.9 and 84.9 ± 1.6, and the two are equally good at finding small improvements. We used the trained model because it ranked kernels best among the GPT-OSS-20B variants. The revised Section 3.4 describes RL training as an optional step that improves calibration.

Dear Reviewer dLmo,

There are 3 days left in the discussion period. In our response we addressed your two critical questions: whether the surrogate helps late in search (Q1) and total cost including generation and forecasting (Q3). If our answers resolve your concerns, we would be grateful if you would consider revisiting your score. If anything is still unresolved, we would be happy to discuss further.

Hi dLmo! The discussion period ends tomorrow. Have you had a chance to look at our general response and our specific response to you? Your engagement would be greatly appreciated!

Best, The authors of GPU Forecasters

Reviewer_J4yqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes adapting LLMs as probabilistic surrogates for GPU kernel runtime optimization. It evaluates their ability to rank candidate kernels, integrates speculative runtime predictions into evolutionary search, and investigates reinforcement learning for forecast calibration.

Critical strengths
  1. The paper focuses on a vital bottleneck in LLM-driven kernel optimization.

  2. The forecasting task is clearly described with analyses of ranking, calibration, and prediction errors.

  3. The release of a large collection of measured kernels is valuable for practical kernel generation.

Critical weaknesses
  1. Insufficient analysis for end-to-end search improvements. The main experiment includes only six tasks with one run per method and task. Given this, multiple runs and uncertainty estimates are needed to determine whether the reported 4/6 improvements are reliable.

  2. Incomplete resource accounting. The surrogate-assisted search generates four times as many candidates while keeping GPU measurement counts fixed. Although this matches the intended GPU-budget objective, the study does not sufficiently evaluate total search time, generation costs, and surrogate inference costs.

  3. Missing ablations for the speculative mechanism. The evaluation does not clearly discuss the contribution of retaining unmeasured candidates with speculative scores from existing surrogate pre-screening methods. Comparisons against random filtering and simpler learned cost models would further support the claim.

Questions for authors

Please refer to Weakness.

4 comments

We thank the reviewer for noting that the paper targets a vital bottleneck in LLM-driven kernel optimization, that the forecasting task is clearly described, and that the released kernels are valuable.

W1: One run per task

We have added a limitation (Section 5) noting that per-task differences of a few percent may fall within run-to-run variance. We have also updated Section 3.4 to state that the surrogate-assisted search is ahead of the search without the surrogate at every measurement budget from 75 GPU measurements onward, averaged over the six tasks. Additional evidence that the surrogate can choose good kernels comes from the offline evaluations, where we do three evaluation trials on 1,347 pairs of kernels. For example, we find that the surrogate is 2.4x better than random selection at picking child kernels that are 2x faster than the parent kernel they were mutated from.

W2: Resource accounting

Please see point 1 of the general response.

W3: Ablations and baselines

Please see points 2 and 3 of the general response for keeping unmeasured kernel candidates and for random selection. Our searches use the same selection rule as existing pre-screening methods and our novelty is in the LLM surrogate that predicts speedup from source code and does not require any training. Regarding learned cost models, see our response to cjso’s Q2:

AutoTVM, Ansor, and TLP are trained to rank schedules of a fixed operator, so they do not directly apply to our setting, which requires predictions about arbitrary Triton and CUDA kernels generated during LLM-driven kernel search. Learned cost models use features from the compiler's representation of a fixed operator. An LLM-written kernel frequently changes the algorithm, memory layout, and launch structure, which are not described by such features. The LLM also needs no training data for a new task or device. The off-the-shelf surrogates in Table 1 have never seen measurements from our tasks. The surrogate trained with correctness and Brier rewards was trained on one task and ranks kernels better on the five held-out tasks, from 84.1% to 88.7% (Table 2). We release the 12,388 measured kernels so that learned predictors for this setting can be trained and compared.

Dear Reviewer J4yq,

There are only 3 days left in the discussion period. In our response we have addressed your questions on run-to-run variance, total search cost, and the ablations against random selection and pre-screening. We would be grateful if you would consider revisiting your score in light of our responses. If anything is still unresolved, please let us know and we are happy to answer.

Hi J4yq! The discussion period ends tomorrow. Have you had a chance to look at our general response and our specific response to you? Your engagement would be greatly appreciated!

Best, The authors of GPU Forecasters

Thanks for the authors' detailed clarification. The additional offline trials support the surrogate’s ranking ability, but they do not resolve the run-to-run uncertainty of the end-to-end evolutionary search itself, which is still evaluated with only one run per task. I think my concerns still hold.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank all reviewers. All reviewers found that the paper addresses a practical, real problem: the cost of GPU measurements in LLM kernel search (J4yq, dLmo, cjso). Reviewer J4yq found our description of the forecasting task clear and the released kernels valuable. Reviewer dLmo felt our study of calibration rewards was well motivated. Reviewer cjso found the idea of using LLMs to predict kernel performance interesting, and noted our evidence that LLMs can identify promising kernels without running them.

1. Total cost of a search (J4yq, dLmo)

Counting both generation and forecasts, the surrogate-assisted search matches the non-surrogate assisted baseline with 2.6x fewer GPU measurements and 1.6x as many generator calls, summed over the six tasks. The GPU time saved is worth 4.8x the cost of the surrogate forecasts.

Without surrogateWith surrogate
GPU measurements1,434558
A100 hours of measurement8.03.1
Generator calls1,4492,312
Surrogate forecasts02,309

Compared with timing all 16 kernels generated at each step, the surrogate makes each step 3.0x cheaper for Triton kernels and 3.7x cheaper for CUDA kernels, with the same generation cost (Table 3).

Cost per stepTime all 16Forecast 16, time top 4Savings
Triton$0.224$0.0743.0x
CUDA$1.69$0.463.7x

Wall-clock time depends on how many generator calls run at once. Generator calls can run in parallel and benefit from greater parallelism due to batching efficiency, but each GPU timing needs an isolated GPU. Our experiments used a parallelism of 8 generator calls. Even so, the surrogate-assisted search explored more kernels per hour than the baseline search on every task. With a parallelism of 16 generator calls, it would explore about 3x as many kernels per hour as the baseline.

Kernels considered per hourSearch w/o SurrogateSurrogate, parallelism=8Surrogate, parallelism=16
Cross-entropy190257631
TriMul166206536
FP8 quantization204259665
GDN ChunkFwd-h182230601
GDN ChunkFwd-o185225602
GDN Recompute W/U184211550

2. Keeping unmeasured candidates vs. pre-screening only (J4yq, dLmo)

The surrogate-assisted search uses the surrogate as a pre-screen. At each step the surrogate scores all 16 generated candidates and the top 4 are measured on the GPU. Only measured kernels enter the search archive. The gains in Figure 4 come from pre-screening alone. We call the surrogate speculative by analogy to speculative decoding, where a small model drafts tokens, a large model checks them, and only checked tokens are kept. We have revised Section 2.1 to state this directly. Using forecasts as scores for unmeasured candidates is a natural extension that we leave to future work.

3. Comparison with random selection (J4yq, cjso)

The baseline search without a surrogate is equivalent to random selection. This is because generating 16 kernels and measuring 4 random ones (random selection) is the same as generating 4 and measuring all 4 (the selection rule used by standard evolutionary search without a surrogate). Note that this makes random selection a strong baseline, as evolutionary search is able to succeed with a random selection rule.

The surrogate is better than random at finding small improvements. A kernel that is 5–25% faster than its parent is 47% more likely to be picked by the surrogate used in our searches than by random selection.

For learned cost models, please see our reply to Reviewer cjso.

4. Contributions & Novelty

We reiterate our contributions below:

  1. We use an LLM surrogate inside LLM-driven evolutionary kernel search. Existing LLM kernel searches time every candidate on the GPU. For example, TTT-Discover uses 25,600 H100 measurements to optimize a kernel. Ansor uses a learned cost model inside evolutionary search, but only over schedules of a fixed operator, so it can’t score arbitrary programs written by LLMs.
  2. We build two benchmarks for kernel surrogates from real search runs. The first evaluates how close a surrogate's top picks come to the fastest of 480 kernels for six GPU Mode tasks. The second benchmark evaluates the surrogates ability to find the rare mutations which make a kernel much faster across 1,347 edits from evolutionary searches on 21 kernel tasks.
  3. We release the 12,388 LLM-written kernels from our searches, with runtimes measured across 118 kernel problems, CUDA and Triton, and three GPU types.
  4. We evaluate how RL training changes a kernel surrogate. We study how both correctness only GRPO and RL for calibration / forecasting (RLCR) affects the properties of the surrogate.
Area_Chair_29deOpenReview ↗

Reminder: Author–reviewer discussion ends today

Dear Reviewers,

As the author–reviewer discussion period ends today (November 18, AoE), please read the authors’ responses, any relevant manuscript updates, and the other reviewers’ comments, and engage with the authors before the deadline.

In particular, please clarify which of your concerns have been addressed and which remain unresolved, and raise any remaining questions promptly so the authors have an opportunity to respond. Even if your assessment is unchanged, a brief follow-up explaining your reasoning would be helpful.

Thank you for your time and thoughtful reviewing!

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.