acceptodds
Withdrawn from ICLR 2027

KernelOPT: Dispatch-Aware Agentic Search for GPU Kernel Optimization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Deep learning inference and training performance depends critically on GPU kernel efficiency. Modern compilers such as PyTorch Inductor automatically generate GPU kernels from high-level model code, but frequently underperform expertwritten implementations by wide margins. Recent LLM-assisted kernel optimizers can close this gap for standalone kernels, yet treat compiled models as black boxes, generally optimizing individual standalone kernels without respecting the compiler’s structural decisions or verifying the model end-to-end. We present KernelOPT, a multi-agent system that treats compiled models as structured artifacts. It preserves vendor library calls (cuBLAS, cuDNN) and exclusively targets generated Triton sub-kernels using five profiling-guided LLM agents. A fourgate verification cascade applies static validation, multi-seed correctness checking, model-level float64-fallback verification, and performance gating (γ=1.03) to filter candidates and verify the re-stitched model end-to-end. When candidates fail verification, the system preserves the compiler baseline. The system accepts PyTorch nn.Modules, standalone Triton kernels, and Helion kernels. Evaluated on 250 KernelBench problems (100 Level 1, 100 Level 2 and 50 Level 3) on NVIDIA H200, KernelOPT achieves geometric mean speedups over torch.compile of 1.40× (L1), 1.15× (L2), and 1.07× (L3) across all kernels, including fallback cases. Optimized-only geomeans (excluding cases where verification gates preserve the compiler baseline) are substantially higher: 2.54× (L1: 36/100), 1.84×(L2: 23/100), and 1.37× (L3: 11/50), reflecting where the optimizer achieves meaningful leverage.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
GPU kernel optimization, LLM agents, multi-agent systems, agentic search, Triton, compiler optimization, performance verification

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_rf4EOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents KernelOPT, an LLM agent that tries to speed up models after torch.compile. The main idea is to leave the cuBLAS/cuDNN calls alone and only rewrite the Triton kernels the compiler generates. A planner reads Nsight Compute profiles and proposes changes, an executor writes them, and a few search chains run in parallel with a small memory of past attempts. The best kernel gets stitched back into the model and is only kept if the whole model is still correct and actually faster, otherwise it falls back to torch.compile. On KernelBench (H200, Claude Sonnet), about a quarter of the problems get faster and the rest stay the same or fall back.

Critical strengths

The most useful part of the paper is the observation that kernel-level speedups often don't survive at the model level. In quite a few multi-kernel problems, a kernel that is clearly faster on its own leaves the whole model no faster or even slower, which is a good warning for work that only reports isolated kernel numbers.

I also appreciate that the reporting is candid. Fallbacks and unchanged cases are reported next to the wins and the implementation shortcuts are listed in the appendix, so the limitations are easy to check (though they are significant, see below).

Critical weaknesses

W1. Limited technical contribution.

To me this is a standard profile-guided kernel agent with an engineering wrapper. The planner/executor/summarizer loop, beam search and memory come straight from AccelOpt (the paper says so itself), and putting profiler output into the prompt is already common. The two new pieces are thin. "Respecting the compiler's dispatch decisions" is basically a regex that finds library calls plus telling the LLM not to touch them, and the model-level check with fallback is a keep-or-revert loop, similar to what AutoKernel does. The insights are either well known (cuBLAS is hard to beat, Amdahl's law dilutes kernel gains) or not really supported. The claim that compiler structure is a powerful inductive bias rests on one ablation where the LLM is allowed to rewrite library calls and fails, which just shows it shouldn't rewrite cuBLAS. I don't see a clear new idea here, and beyond a few hand-picked examples the paper doesn't explain why the system works.

W2. The evaluation doesn't support the claims.

There are no agent baselines, and more importantly the biggest gains seem to come from benchmark quirks and lower precision rather than the proposed ideas. The paper says other agents can't be compared because of hardware differences, but the related work says the authors ran KernelAgent themselves on the same H200 (Level 1 only). So running KernelAgent, AutoKernel or CudaForge on all levels was clearly doable, and since the method is mostly an agent plus a wrapper, "agent with vs without the wrapper" is the natural experiment.

The top speedups come from KernelBench problems that are written inefficiently (like multiplying by a full diagonal matrix) or from algebraic shortcuts such as precomputing weight sums outside the kernel. The showcase matmul speedup comes from casting to FP16, which goes against the paper's own planner prompt (it forbids TF32/BF16 casts for matmuls, and FP16 has the same mantissa as TF32), and the fallback correctness check has an absolute error bound designed to allow TF32-level error. Reduced-precision kernels vs an FP32 baseline is not a fair comparison, and we don't know how many accepted kernels do this.

The verification and ablation also feel weaker than claimed. During search, candidates are ranked by speed without filtering on correctness, and only the fastest one is checked at the end. In the ablation, dropping any single component (even the small memory) collapses success, and dropping the performance gate also reduces the number of optimized problems. The authors attribute this to retries but don't quantify it, and all of it is single runs with no variance. Finally, everything uses one LLM, one GPU and one benchmark, even though the paper itself argues kernel performance is hardware-sensitive, and most problems end up unchanged or falling back anyway.

Questions for authors
  1. Can you add agent baselines on the same hardware, e.g. KernelAgent on all levels, AutoKernel or CudaForge, ideally with and without your wrapper?
  2. How many accepted kernels lower precision compared to the FP32 baseline, and what do the results look like without them (or with torch.compile also allowed to use TF32)?
  3. Which candidate is actually reported? The setup says the best one passing all gates, but Algorithm 1 and Appendix B say only the fastest one is tested.
  4. How many of the full system's successes come from retries triggered by the performance gate, and why does removing the small memory wipe out all Level 2 and 3 optimizations?
  5. How much do the main results vary across runs?
  6. Do the results hold with another LLM (e.g. an open-weight one) and another GPU?
Reviewer_sLn3OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

KernelOPT is an LLM-agent system that speeds up PyTorch models compiled with torch.compile. It traces the Inductor output, marks cuBLAS/cuDNN calls as fixed, has an LLM synthesize a standalone Triton kernel for each remaining fused group, and then improves these kernels with a five-agent loop (profiler, strategy analyst, planner, executor, summarizer) driven by Nsight Compute (NCU) metrics and a diversity-aware beam search (5 iterations, 4 plans per iteration, beam width 4). The optimized kernels are stitched back into the model and kept only if they pass four checks: a crash check, multi-seed correctness at 1e-3, model-level correctness at 1e-4 with an FP64-based fallback, and a 3% end-to-end performance margin; otherwise the compiler baseline is kept. On all 250 KernelBench problems (H200, Claude Sonnet 4.6), the paper reports geometric-mean speedups over torch.compile of 1.40× (L1), 1.15× (L2) and 1.07× (L3), with 70 problems optimized, 95 matched and 85 falling back. It also reports a 50-problem ablation, an L1 comparison with KernelAgent, and a total cost of about 1,100 H200 GPU-hours and 115M LLM tokens.

Critical strengths
  1. The paper optimizes the compiled model rather than isolated kernels: optimized kernels are stitched back, checked end to end for correctness and speed, and replaced by the compiler baseline when they fail these checks. The measurement behind this choice is useful in itself. In 37 of 126 multi-kernel problems, a sub-kernel's isolated speedup did not carry over to the end-to-end model, and 16 of these would have shipped as regressions without the model-level gate (§6.4). This is a concrete caution for work that reports standalone-kernel speedups.

  2. The evaluation is reported openly: per-problem results for all 250 problems, geomeans that include fallbacks, a frank list of implementation limitations (latency-only beam ranking, a single-candidate final check, unlogged FP64 error ratios for accepted candidates), and released code with per-problem logs and deployed models. This makes the work reproducible and auditable; several of the concerns below come from checking those artifacts.

Critical weaknesses

1. The evidence does not support the central claim. This weighs most in my rating: most of the reported speedup sits in problems where the deployed kernels lower arithmetic precision or remove the GEMM algebraically, gains that say little about dispatch awareness.

  • Precision. In the released deployed models, 17 of the 70 optimized problems, all of them GEMM problems, run their matrix products in FP16, BF16 or TF32 (one through Triton's TF32 default), while the torch.compile baseline runs with PyTorch's default matmul setting, which does not allow TF32. These 17 carry 57% of the summed log-speedup, and counted as 1.0× they leave geomeans of 1.14×/1.07×/1.04× instead of 1.40×/1.15×/1.07× (L1/L2/L3).
  • Dispatch. No optimized deployment leaves a GEMM to cuBLAS, and the graph tracer forces Triton GEMM templates; vendor calls survive only as cuDNN convolutions (25 problems, 1.13× geomean). The two largest wins remove the GEMM algebraically: L1-012 (88.63×) is the known diag(A) @ B rewrite (KernelBench, CUDA-L1), and L2-018 (33.11×) is a degenerate task that reduces over a singleton dimension.
  • Scope. KernelBench L1's 100 single-operator problems give dispatch awareness little to act on, and on L3, the full models the paper targets, the gain is 1.07× (11 of 50 models optimized).

2. The agent workflow is neither new nor shown to be needed.

  • Novelty. The planner–executor–summarizer loop, beam search and memory follow AccelOpt, as the paper states, and KernelAgent's public repository already combines multi-agent generation, NCU-guided optimization and beam search.
  • Baselines and cost. There is no general-purpose coding agent (e.g., Claude Code or Codex) with the same model, tools and budget, and the only external comparison, with KernelAgent, is on L1, where KernelOPT's 1.649× does not match Table 1's 1.40×. The evaluation takes about 1,100 H200 GPU-hours and 115M tokens, with no cost-matched baseline.
  • Search and ablation. DiverseSelect is compared only with B=1, not with plain beam search or independent sampling at the same budget, and the ablation is hard to read: removing the performance gate, which mainly rejects candidates, cuts optimized problems from 19 to 3.
Questions for authors
  1. Precision (experiment). What are the L1–L3 geomeans under matched precision, for example with the baseline at torch.set_float32_matmul_precision("high") or with candidates limited to FP32 or TF32x3?
  2. Equal-budget baselines (experiment, on a subset of L2 and L3). At the same per-problem GPU-hour and token budget, how do a single general-purpose coding agent with the same model and tools, and plain beam search or independent sampling in place of DiverseSelect, compare with KernelOPT?
  3. Reported results (clarification). Why does removing the performance gate cut optimized problems from 19 to 3, and why does the KernelAgent comparison report 1.649× for KernelOPT on L1 against 1.40× in Table 1?
Reviewer_8hTGOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces KernelOPT, a LangGraph-based multi-agent system for optimizing models compiled with torch.compile. It preserves traditional kernel library calls and only optimizes Triton sub-kernels. Then, these optimized sub-kernels are re-stitched into the model and verified through a four-gate cascade: static checks, multi-seed correctness, model-level float64-fallback checking, and a performance profiling. If the verification fails, the whole kernel falls back to the default compiler baseline. On all 250 KernelBench problems, it reports speedups over torch.compile of 1.40×, 1.15×, and 1.07× on L1, L2, and L3.

Critical strengths
  1. The composition gap analysis is informative and meaningful. In 29% of multi kernel problems, when sub-kernel get improvement, it do not lead to improvement after re-stitching, which directly motivates model level performance gating.
  2. The paper provides transparent details and make things clear: it gives full results and fallback root causes, and it openly acknowledges implementation deviations from the stated formulation.
Critical weaknesses
  1. First of all, the writing quality of this paper needs to be significantly improved. For example, it is very hard for me to understand symbols in eq5 and eq6.
  2. The paper states that "attempting to replace already-near-optimal vendor calls introduces more failures than gains. KernelOPT instead respects the compiler’s dispatch decisions, concentrates search on Triton sub-kernels,and verifies results at the model level rather than in isolation. " I respectfully disagree with this. I think only optimize a small fraction of Triton subkernels could constrain the searching space of optimization. For example, it excludes many possiblities of kernel fusion.
  3. The ablations are unconvincing and the reported numbers are inconsistent. Removing any single component drops the optimized rate from 38% to ≤10%. Removing the performance gate even reduces successes. All of this comes from single runs, which suggests confounds rather than component contributions. There are also numerous inconsistencies: (1) The KernelAgent comparison reports 1.649× on L1, versus 1.40× in the main results. (2) The Figure 3 panel titles give 52/31/12 optimized, versus 36/23/11 in the caption and main table. (3) Table 10 labels entries at 1.01–1.02× as OPT, despite the >1.03× threshold. (4) The rejection count is given as 167, 166, and 161 in different places. (5)Fallback counts differ between Figure 4 and Table 1 Panel D.
Questions for authors

Please see weaknesses, thanks.

Comments on the paper

1 comment

We thank the reviewers for the detailed reviews. We’ve uploaded a revision that clarifies the scope of dispatch preservation, explains how fallback cases enter the reported speedups, and states the limits of the precision comparison and ablations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.