- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents KernelOPT, an LLM agent that tries to speed up models after torch.compile. The main idea is to leave the cuBLAS/cuDNN calls alone and only rewrite the Triton kernels the compiler generates. A planner reads Nsight Compute profiles and proposes changes, an executor writes them, and a few search chains run in parallel with a small memory of past attempts. The best kernel gets stitched back into the model and is only kept if the whole model is still correct and actually faster, otherwise it falls back to torch.compile. On KernelBench (H200, Claude Sonnet), about a quarter of the problems get faster and the rest stay the same or fall back.
Critical strengths
The most useful part of the paper is the observation that kernel-level speedups often don't survive at the model level. In quite a few multi-kernel problems, a kernel that is clearly faster on its own leaves the whole model no faster or even slower, which is a good warning for work that only reports isolated kernel numbers.
I also appreciate that the reporting is candid. Fallbacks and unchanged cases are reported next to the wins and the implementation shortcuts are listed in the appendix, so the limitations are easy to check (though they are significant, see below).
Critical weaknesses
W1. Limited technical contribution.
To me this is a standard profile-guided kernel agent with an engineering wrapper. The planner/executor/summarizer loop, beam search and memory come straight from AccelOpt (the paper says so itself), and putting profiler output into the prompt is already common. The two new pieces are thin. "Respecting the compiler's dispatch decisions" is basically a regex that finds library calls plus telling the LLM not to touch them, and the model-level check with fallback is a keep-or-revert loop, similar to what AutoKernel does. The insights are either well known (cuBLAS is hard to beat, Amdahl's law dilutes kernel gains) or not really supported. The claim that compiler structure is a powerful inductive bias rests on one ablation where the LLM is allowed to rewrite library calls and fails, which just shows it shouldn't rewrite cuBLAS. I don't see a clear new idea here, and beyond a few hand-picked examples the paper doesn't explain why the system works.
W2. The evaluation doesn't support the claims.
There are no agent baselines, and more importantly the biggest gains seem to come from benchmark quirks and lower precision rather than the proposed ideas. The paper says other agents can't be compared because of hardware differences, but the related work says the authors ran KernelAgent themselves on the same H200 (Level 1 only). So running KernelAgent, AutoKernel or CudaForge on all levels was clearly doable, and since the method is mostly an agent plus a wrapper, "agent with vs without the wrapper" is the natural experiment.
The top speedups come from KernelBench problems that are written inefficiently (like multiplying by a full diagonal matrix) or from algebraic shortcuts such as precomputing weight sums outside the kernel. The showcase matmul speedup comes from casting to FP16, which goes against the paper's own planner prompt (it forbids TF32/BF16 casts for matmuls, and FP16 has the same mantissa as TF32), and the fallback correctness check has an absolute error bound designed to allow TF32-level error. Reduced-precision kernels vs an FP32 baseline is not a fair comparison, and we don't know how many accepted kernels do this.
The verification and ablation also feel weaker than claimed. During search, candidates are ranked by speed without filtering on correctness, and only the fastest one is checked at the end. In the ablation, dropping any single component (even the small memory) collapses success, and dropping the performance gate also reduces the number of optimized problems. The authors attribute this to retries but don't quantify it, and all of it is single runs with no variance. Finally, everything uses one LLM, one GPU and one benchmark, even though the paper itself argues kernel performance is hardware-sensitive, and most problems end up unchanged or falling back anyway.
Questions for authors
- Can you add agent baselines on the same hardware, e.g. KernelAgent on all levels, AutoKernel or CudaForge, ideally with and without your wrapper?
- How many accepted kernels lower precision compared to the FP32 baseline, and what do the results look like without them (or with torch.compile also allowed to use TF32)?
- Which candidate is actually reported? The setup says the best one passing all gates, but Algorithm 1 and Appendix B say only the fastest one is tested.
- How many of the full system's successes come from retries triggered by the performance gate, and why does removing the small memory wipe out all Level 2 and 3 optimizations?
- How much do the main results vary across runs?
- Do the results hold with another LLM (e.g. an open-weight one) and another GPU?
We thank the reviewers for the detailed reviews. We’ve uploaded a revision that clarifies the scope of dispatch preservation, explains how fallback cases enter the reported speedups, and states the limits of the precision comparison and ablations.