- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper trains a small LLM (8B) to write optimization schedules for TVM. In TVM, one usually take a plain loop program and apply a schedule (tiling, thread binding, shared memory, tensor cores etc.) to get a fast GPU kernel, and normally you find a good schedule by running MetaSchedule, which is an evolutionary search that tries lots of candidates on the GPU. The authors run this search on a big pile of workloads, keep the best schedules, and turn them into training data for two tasks: write a schedule from scratch, or take a non-competitive schedule and rewrite it into a better one. They also design a compact format for schedules that converts back to the exact compiler trace, and attach an explanation to each example (written by Kimi-K3 and checked against the actual edits). Then it's the usual recipe, SFT first and then GRPO, where the reward gives some partial credit for parsing and compiling and only gives a speed bonus if the kernel is actually correct. On held-out tasks on an H100, the model gets correct schedules on the first try most of the time, and RL mostly makes the kernels faster. The authors report that it beats six frontier LLMs and also beats TVM's own search while being much faster to run.
Critical strengths
-
The method is concrete and carefully engineered. The schedule format is done with care, and the reward only pays for speed when the kernel is correct, which is a sensible guard against reward hacking.
-
They put real effort into the train/test split, separating tasks by workload and by a hash of the program, and they compare against the search that generated their own training data, including how long each one takes.
Critical weaknesses
Weakness #1 Practical Value and Comparisons with Kernel Optimization Agents
My biggest issue is that I'm not convinced this setup is useful in practice, and the paper fails to make a convincing case. To get this model you have to run TVM's search on tens of thousands of workloads, then train a model, and what you get is tied to one compiler, one GPU and even one TVM version (the paper says the format is only checked under the recorded version). Change any of those and you're collecting data and training again.
Meanwhile, the obvious alternative is just to take a strong LLM and put it in an agent loop that compiles, tests, profiles and fixes things. That needs no training and no data collection, and we already know agent harnesses boost LLMs a lot on kernel optimization. So it seems not an economic choice to distill TVM into a one-shot model.
Of course, this would be sensible if the proposed approach leads to much stronger kernels. Yet, while the paper cites LLM-guided search methods that work inside TVM on the same schedules (Reasoning Compiler, LiteCoOp) plus agent systems like AccelOpt and AlphaEvolve, it did not compare against any of them. Their own model has a refinement mode, so it could run in a loop too. A fair comparison would look at kernel quality vs attempts or wall-clock time, for their model refining its own outputs and for a frontier model inside an agent harness. Without that I honestly can't tell if all the training effort buys anything.
And the frontier LLM comparison they do have isn't fair. The frontier models get one shot, no compiler feedback, no retries, no examples. On top of that, for construction they're asked to write a whole TVM kernel from PyTorch code, which is a different and much harder task than what the trained model does (it only outputs schedule edits in its own format). A fine-tuned model beating zero-shot models in this setup is pretty much guaranteed, and I think the really low correctness of some frontier models is mostly about the interface, not about how good they are at optimization (under proper interface and harness).
Weakness #2. Lack of Strong Baselines
The rest of the evidence doesn't tell me how good the kernels really are or what part of the method matters. All the speedups are vs TVM's default schedule, which is super slow, so the huge speedup numbers don't mean much. *There's no comparison to popular and strong baselines, like cuBLAS, FlashAttention, Triton or torch.compile. They also say the model beats the same TVM search it learned from, but they don't say what search budget TVM got, so maybe the search just wasn't run long enough. Correctness is checked on one random input, which is pretty weak. And held-out tasks might just be the same operators with different shapes; there's no analysis of how close they are to the training tasks.
Also the method itself is a pretty standard recipe. SFT on compiler search data plus RL with an execution reward has been done before (TLM, LLM Compiler, Compiler-R1, CUDA Agent and so on). The new pieces are the compact format, the explanation graphs, the token weighting and the refinement data, but none of them is ablated, so we don't know if any of them matters. With one GPU, one base model and one training run, it's hard to see what this adds beyond applying a known approach to TVM schedules.
Questions for authors
- How does your model compare with a frontier model in an agent loop with compiler feedback, at matched time or compute? For example, KernelAgent and AccelOpt.
- What happens if you run your refinement mode in a loop on your own outputs?
- How would the frontier models do if they got the same schedule-edit interface as your model, plus a few examples and the compiler errors?
- What search budget did TVM's MetaSchedule get in your comparison, and how does your model do against a full-budget search?
- How do your kernels compare with cuBLAS, FlashAttention, Triton or torch.compile?
16 comments
Response to W1
W1
Our practical contribution is that our targeted training enables an 8B model to perform the complex task of compiler optimization, outperform frontier models on the evaluated workloads, and produce strong schedules without repeating an expensive search for each task. The scale difference is concrete: Kimi K3, one of our baselines, contains 2.8 trillion total parameters and activates 104 billion per token, which is 350 and 13 times our model’s parameter count, respectively. Achieving stronger task-specific performance with an 8B model is a significant result: our training approach makes a small model an effective alternative to invoking frontier models for these optimization tasks.
To illustrate the resource stakes, an adaptation run using 256–1,024 GPUs for two weeks would consume approximately 86,000–344,000 GPU-hours. At an assumed 3–5 per GPU-hour, that amounts to roughly 258,000–1.72 million for one run, before additional experiments and engineering. This budget shows why small model specialization matters. Our approach makes an accessible 8B model outperform available frontier models on TVM kernel optimization tasks.
That trained model then reuses compiler search experience across new workloads. Although its training labels come from approximately 9-minute searches, we evaluate it against TVM MetaSchedule searches with a 1,000-trial target and three seeded runs per workload, taking a median of 53.6 minutes per task. The model produces the reported high performance kernels with approximately 39× less optimization time compared with MetaSchedule. Thus, the benefit is both a small model capable of this demanding task and a reusable policy that reduces the search required at deployment; the upfront data generation and training costs are amortized across subsequent uses.
We would like to clarify the distinction between training the model to refine an existing schedule and running it in an agentic loop. The refinement task gives the model a starting schedule and asks the model to propose an improvement in one invocation; it does not repeatedly invoke the model with compiler feedback. Our contribution is a training method that teaches a small model compiler knowledge and improves its optimization ability, not an agentic workflow. The reported results demonstrate what the trained model achieves in a single shot.
Our trained model can also serve as the optimizer within an agentic loop, where repeated calls to frontier models can incur substantial inference costs. For example, Kimi K3 activates 104 billion parameters per token, compared with our model’s 8 billion. At equal generated token counts, this represents approximately 13× more parameter-dependent inference arithmetic per proposal. Using our smaller model therefore offers a substantially lower model compute requirement for repeated proposals, allowing an agentic workflow to spend that budget on more optimization attempts or reduce its inference time and cost.
At the reviewer’s request, we conducted additional experiments during the discussion period in which the model refines its first construction output once. This additional refinement step yields further speedups of 1.14× on attention kernels, 1.32× on quantization kernels, and 1.15× on convolution kernels. These results show that the same model can further improve its initial output, beyond the gains already achieved without an agentic loop.
Our comparison with frontier models has even made the task easier and more accessible to frontier models. We have two kinds of tasks, construction and refinement. In construction, the objective is to produce an optimized TVM implementation of the reference PyTorch kernel. Our trained model receives the set of information including initial TensorIR and compiler features and outputs the schedule edits which are lowered to TVM trace and applied to the initial TensorIR. Given the same input information and also the PyTorch reference code, asking the frontier baselines to generate TVM traces directly resulted in a lack of sufficient correct outputs. We therefore relaxed the output format requirement: the baselines still receive the same information and the PyTorch reference code but may generate complete TVMScript kernels rather than use our custom representation. Thus, this construction task does not require them to use our custom representation. For refinement, both the frontier baselines and Little Dragon receive the same prompt. Our SFT+RL model achieves 92.01% Correct@1, compared with 45.81% for the strongest frontier baseline, and a 59.28% correct-and-≥2× improvement rate, compared with 17.33% for the strongest frontier baseline on that metric. Little Dragon therefore achieves approximately twice the correctness rate and 3.4 times the rate of correct refinements that at least double performance. These results demonstrate that our targeted training enables an 8B model to outperform available frontier models on this demanding optimization task.
Response to W2 - Part 1
W2
We report additional baselines in our answer to Q5 and we achieve speedups over cuBLAS, FlashAttention, Triton and torch.compile. Every TVM datapoint in Section 3.5 and every search-best reference is a completed MetaSchedule evolutionary search with a 1,000-trial target per run, a median of 53.6 min per task on one H100, three seeded runs per workload. The training labels came from a much lighter 9-minute search, so the model learned from around 9-minute searches and is evaluated against search results after an average of 54 minutes. Some longest search runs include MoE grouped gate-up GEMM, which takes 1,138 minutes, LM-head GEMM with 655 minutes, and Sparse-MLA decode with 323 minutes.
Figure 4 of the paper compares optimization time and kernel quality across six workload families between first SFT+RL output and TVM MetaSchedule search run. Under the plotted timing convention, SFT+RL produces these faster kernels with approximately 39× less optimization time. These results show how compiler search experience can be reused through training: the 8B policy produces a complete schedule in a single proposal, reducing the per-task cost of obtaining an optimized kernel.
Our model generates compiler transformations, not kernels. It therefore operates within the bounds of TVM’s scheduling primitives rather than generating arbitrary code that replaces the original kernel. These primitives—including tiling, reordering, thread binding, caching, and tensorization—are applied to the original TensorIR kernel . TVM checks their applicability during trace replay, rejects transformations that violate the checked preconditions, and generates the resulting kernel.
We agree that a single random test is insufficient to establish correctness, and we do not use it as that basis. Our approach instead relies on TVM’s semantics-preserving scheduling primitives and their applicability checks. Because TVM relies on downstream code generation, including LLVM, we retain an additional numerical sanity check to catch potential compilation faults, not to establish the correctness of the schedule transformations themselves.
This sanity check compares the generated GPU kernel against the LLVM-compiled original kernel , element by element across the entire output tensor. “One randomly generated test case” refers to complete input tensors: the and matrices in Appendix A.1 contain 15,261,696 input values, and the check compares all output values. Passing this check is required before a candidate receives a latency bonus.
Our training set contains diverse workloads spanning six kernel families: GEMM, normalization and reduction, attention, quantization and irregular kernels, vision, and mixture-of-experts. These workloads come from models including DeepSeek-V3.2-Exp, Qwen3-Next-80B-A3B, Llama-3.1-8B, Mixtral-8x7B, Kimi-K3, Qwen3.5-9B, etc. Our split excludes exact workload identities and initial-IR hashes across training and evaluation. Sharing an operator family does not make workloads equivalent optimization problems: different shapes change the tiling, parallelization, and memory management decisions needed to produce an efficient schedule.
We also ran additional experiments to directly test shape extrapolation, creating 20 new construction cells across eight templates, with one axis extended to 2× and 4× beyond each template’s training range. The extrapolated configurations include GEMM rows up to 16,384, FP8-GEMM rows up to 8,192, MLA contexts up to 65,536, GQA KV lengths up to 131,072, softmax row widths of 16,384, RMSNorm sequences of 4,096 tokens, patch grids corresponding to 1,512-pixel images, and 256 experts.
| Factor beyond training range | Cells | SFT Correct@1 | SFT+RL Correct@1 | SFT+RL median over |
|---|---|---|---|---|
| 2× | 10 | 97.50% | 100.00% | 57.6× |
| 4× | 10 | 81.25% | 90.00% | 70.5× |
SFT+RL produces a correct candidate for every 2× cell and for seven of eight templates through 4×. Its median speedup is 57.6× at 2× and 70.5× at 4×. These new results establish substantial shape extrapolation within learned operator structures: extrapolation preserves correctness while continuing to produce high-performance schedules beyond the training range.
Our SFT/RL ablations are already reported in Table 1, which separately evaluates Seed-Coder-8B-Instruct (the base model), SFT, and SFT+RL. Adding RL increases construction from 64.98% to 77.69% and refinement correct and ≥ 2× over graph A improvement from 37.24% to 59.28%. These gains of 12.71 and 22.04 percentage points respectively directly establish the benefit of RL beyond supervised training.
We thank the reviewer for the reviews. We address the reviewer’s two weaknesses and five questions below: