acceptodds
Under review as a conference paper at ICLR 2027

KernelMaster: Agentic GPU Kernel Optimization from Semantics to Execution

Abstract

Agent systems can generate and refine GPU kernels, yet optimizing an unfamiliar operation often requires extensive trial and error. GPU kernel optimization requires coordinated choices about how to reorganize computation, distribute and schedule parallel work, and implement it using GPU instructions. However, runtime and profiler feedback reveal performance bottlenecks but do not directly prescribe effective program changes, making it difficult for an Agent to connect these observations to actionable revisions without repeated trial and error. We present KernelMaster, a system that guides Agent-driven GPU kernel optimization through mathematical dataflow search and hardware-calibrated execution synthesis. We expose mathematical transformations through an Agent-operable MathIR, validate candidates through source coverage and replay, and project selected dataflows into LoopIR. We then synthesize Warp pipelines and CTA programs using target-GPU measurements of service time, concurrency, and HBM bandwidth. The selected execution graph guides Agent-driven CUDA implementation and refinement. On five newly specified attention kernels, KernelMaster achieves – speedup over the strongest Agent baselines and – over PyTorch references. Across established kernels on Hopper and Blackwell, it approaches or exceeds expert implementations, with speedups up to .

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
GPU kernel optimization, LLM agents, mathematical dataflow optimization, execution graph synthesis, hardware-aware compilation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ecgrOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that LLM kernel agents are fine at local code fixes but bad at the big structural decisions, like which mathematical form of the computation to use and how to organize the work across warps and thread blocks. Profiler feedback tells you a kernel is slow, but not which of these levels to change. KernelMaster decides the structure before the agent writes any code. An LLM picks among legal math rewrites of the PyTorch reference (e.g. fusing ops, or folding a downstream multiply into a running reduction), each checked by replaying against the reference. Then the system searches how warps and thread blocks split the work, ranking plans with a cost model built from small calibration kernels timed on the target GPU. The best plan becomes a fixed contract the agent follows while writing and refining the CUDA code. On five new attention variants it beats four agent systems on every task, and on five established kernels it's faster than expert libraries like FlashInfer and FlashMLA in most settings.

Critical strengths

The central idea is really interesting. Splitting the problem into the math form, the work organization on the GPU and the low-level code, and handing the agent a checked structure instead of letting it figure out all three by trial and error, is a nice way to make agent-driven kernel optimization less blind. The premise that agents do well on instruction-level tweaks but struggle with structural choices also sounds right to me.

The evaluation setup is fair. They compare with four recent agent systems using the same LLM, use tasks designed to be unfamiliar to the model, and also compare with expert libraries on recent GPUs instead of just PyTorch eager. If the wins over hand-tuned libraries hold up, that's a solid result.

Critical weaknesses

W1. The central claim, that the searched structure is what makes the agent good, is never actually tested. There's no ablation at all, so I can't tell how much comes from the math rewrite search, how much from the execution planning, and how much from just giving a strong LLM a detailed plan. Removing each stage in turn would answer this. The execution planning also relies completely on the calibrated cost model, but there's no check that it ranks plans correctly (e.g. predicted vs measured time across candidates). The agent comparison needs more care too. Everyone gets five optimization steps, but KernelMaster does its whole rewrite search and calibration before step one, and KernelAgent is stopped at its first correct result, so it's comparing steps, not effort. It also looks like single runs with no variance, and some margins over the best baseline are small enough that noise could flip them.

W2. The scope is narrow. All five new tasks are attention variants and the established kernels are mostly attention plus RMSNorm. The flagship rewrite (folding the softmax-times-V product into a running reduction) is essentially the FlashAttention trick, so this might show the system is very good at one family of kernels, not that it generalizes. Matmul-heavy kernels, convolutions, MoE routing or scans would be much more convincing. Some wins over expert libraries also come from specialization, e.g. FlashMLA pads to more heads than the workload needs. That's a legit advantage, but a narrower one than "beats expert implementations".

W3. The paper is hard to follow. There are lots of new terms (MathIR, LoopIR, WrapIR, cuts, slots, channels, contracts), explained mostly in dense prose with one worked example, no algorithm box and no appendix. Claims that the projection and composition preserve semantics are stated, not shown. So it's hard to tell what's fully automatic, what relies on the LLM, and what needed hand engineering per task, and I don't think I could reproduce it from the paper.

Questions for authors
  1. Can you ablate the math rewrite search, the execution planning and the contract one at a time? To me this is the most important missing piece.
  2. How accurate is the cost model, i.e. how well do predicted times rank candidate plans vs measured latency?
  3. How much total effort does each system use (LLM calls, wall-clock time, including your search and calibration)? Would KernelAgent catch up if it kept refining after its first correct result?
  4. How much do results vary across repeated runs?
  5. How does it do outside attention and normalization, e.g. matmul-heavy ops, convolutions or MoE routing?
  6. Which parts are fully automatic, and which needed manual work per task (like defining legal rewrites or writing calibration kernels)?
5 comments

Q1. Component ablations

Experiment. We add KernelMaster's components one at a time to the same Agent on differential-gated MLA decode, on both GPUs, with three runs each. All systems use Codex CLI as the Agent harness with GPT-5.6-sol at high reasoning effort.

Results. Latency in µs, reported as min / max / avg:

SystemRTX 5090 DH20
Codex only18.43 / 36.86 / 26.6216.66 / 87.97 / 40.52
Codex + KM feedback18.43 / 28.67 / 25.2619.26 / 50.83 / 35.39
Codex + MathIR + KM feedback12.29 / 20.48 / 16.3822.59 / 35.12 / 28.20
KernelMaster12.29 / 14.34 / 12.9714.46 / 16.58 / 15.36

Conclusion. Each component brings a consistent gain:

  • The feedback loop improves the average and the worst run.
  • MathIR lowers the average further.
  • The searched, contract-bound structure gives the lowest average and the smallest spread on both GPUs.

Full results, including per-round trajectories, are in Appendix A.2 (Table 3, Figures 9–10).

Q2. Accuracy of the cost model

Experiment. We take candidates at rank percentiles P0–P100 of the 1,541-candidate ranking on the RTX 5090 D and compare KernelMaster's predicted device time with an analytic device roofline. The roofline uses no KernelMaster cost formula, only the candidate's geometry and bandwidth curves we measure with a stream kernel.

Results. Device time in µs:

PercentileRankGrid × blockKM predictionAnalytic roofline
P01256 × 1289.761.98
P25386256 × 12812.512.79
P50771512 × 12814.672.84
P75115664 × 12817.483.95
P90138764 × 12821.013.95
P10015411024 × 3248.707.32

Rank correlation between the prediction and the roofline:

SetSpearman ρKendall τ
79 candidates0.840.67
19 distinct structures0.810.65

Conclusion. The ranking follows an independent hardware bound from top to bottom: top-ranked structures have the lowest bounds, and the tail is slower because fewer CTAs and fewer threads per CTA lower the achievable bandwidth. The cost model therefore orders candidates by the same hardware limits that bound them, so the search selects good structures for principled reasons.

Full details, with all sampled percentiles, are in Appendix A.5 (Tables 8–9, Figure 13).

Q3. Total effort and KernelAgent

Experiment. We charge KernelMaster's full search to its first round and give AKO4X, the strongest baseline, the same total token budget. KernelMaster's search measures candidate structures with Agent-generated calibration kernels, which are stored in a calibration cache. Without the calibration cache, the Agent generates every calibration kernel anew. With the calibration cache, the search reuses the calibration kernels of an earlier search of the same task and makes no LLM calls.

Results. RTX 5090 D, reported as min / max / avg:

SystemTokens per runBest latency at KernelMaster's final budget (µs)
AKO4X4.44 / 16.12 / 9.16M over five rounds12.29 / 26.62 / 17.07, continued to 39.52M
KernelMaster without calibration cache38.71 / 41.06 / 39.52M12.29 / 14.34 / 12.97
KernelMaster with calibration cache20.87 / 23.22 / 21.68M12.29 / 14.34 / 12.97

In wall-clock time, a search takes about 86 minutes without the calibration cache and 28 minutes with it, and a final body takes 41–57 minutes. About two thirds of the search time is spent in Agent-driven stages, so higher API concurrency would shorten it.

Conclusion. KernelMaster leads at equal total tokens, while additional tokens help AKO4X little. KernelAgent prompts only for correctness and has no optimization loop of its own, so we keep its original single-pass design rather than altering the baseline.

Full details are in Appendix A.4 (Tables 6–7, Figure 12).

Q4. Variance across runs

Experiment. We repeat every system three times on both GPUs. We also run KernelMaster three times on RMSNorm, each with an independent search.

Results. Latency in µs, reported as min / max / avg:

TaskSystemRTX 5090 DH20
Differential-gated MLAKernelMaster12.29 / 14.34 / 12.9714.46 / 16.58 / 15.36
AKO4X12.29 / 30.72 / 19.8018.74 / 27.79 / 24.66
AutoKernel12.29 / 32.77 / 23.6239.36 / 138.11 / 72.90
AdaExplore48.99 / 49.15 / 49.1028.83 / 38.69 / 33.66
KernelAgent51.20 / 128.98 / 96.2464.67 / 100.37 / 86.46
RMSNorm (7168)KernelMaster4.172 / 4.264 / 4.2056.281 / 6.541 / 6.425

Conclusion. KernelMaster has the smallest spread on both GPUs and both tasks. The three independent RMSNorm searches select the same structure, so the search itself is reproducible. Fixing its result before implementation keeps run-to-run variation of the Agent small.

Full details, including per-round trajectories, are in Appendix A.3 (Tables 4–5, Figure 11).

Q5. Beyond attention and normalization

Experiment. We add Mamba2 selective-state-update decode, a state-space operator. We use the FlashInfer-Trace definition and reference verbatim, and compare against FlashInfer, the Agent baselines and a measured bandwidth roofline of 23.99 µs in the same timing scope.

Results. Latency in µs on the RTX 5090 D:

MethodLatencyFraction of roofline
PyTorch reference743.870.03
FlashInfer36.860.65
AutoKernel26.620.90
AdaExplore26.620.90
KernelAgent30.720.78
AKO4X37.280.64
KernelMaster feedback loop, min / max / avg24.58 / 30.78 / 27.330.78 / 0.98 / 0.89

Conclusion. SSU decode is a bandwidth-bound gather–update–scatter over a persistent state cache. Its only reduction is a 128-element dot product. The roofline shows that little room remains for a multi-stage pipeline or a different work decomposition: the best kernel already reaches 0.98 of it. We therefore evaluate this operator with KernelMaster's evaluation and feedback loop, without the structure search. KernelMaster's best run beats FlashInfer and every Agent baseline under the same contract.

Full details are in Appendix A.8 (Table 12, Figure 14).

Q6. Automatic versus manual

KernelMaster is fully automatic:

  • MathIR is derived automatically from the captured operator graph together with Agent-generated rewrites.
  • LoopIR, the calibration kernels and the final CUDA kernel are produced by Agents, with automatic checks.
  • The search, calibration and ranking are deterministic.

For each task, the user supplies only the PyTorch reference and its workload.

Thanks for the detailed response and the additional experiments. To be transparent, I see this paper as somewhere around a 2.5. I've decided to raise my score to 3 because I find the core idea interesting and think it could be valuable to the community, but several important concerns remain.

The main one is the scope of the evaluation. All the new tasks are attention variants and the established kernels are mostly attention plus RMSNorm, and the flagship rewrite is essentially the FlashAttention technique. The new Mamba2 task is run without the structure search, so it doesn't show whether MathIR and the execution search work beyond attention. I think results with the full pipeline on kernels with a different structure, such as matmul-heavy operators, convolutions or MoE, are needed to support the claim that the approach is general, and it would help to know which MathIR rewrite types are actually used across tasks.

Second, the cost model is compared with an analytic roofline rather than with measured latencies, so it's still unclear how accurately it ranks candidates in practice. Finally, the response says LoopIR is produced by Agents while the paper describes a deterministic projection, which should be clarified, and the correctness tolerances are still not stated.

I hope the authors can address these points in the revision.

Reviewer_D2tAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

KernelMaster tries to reduce trial-and-error CUDA optimization by making the main dataflow and scheduling decisions before code generation. It first searches for mathematical rewrites, then ranks execution graphs using a hardware-calibrated cost model. The selected graph constrains the CUDA agent, which gets 5 further optimization steps. The evaluation compares 5 new attention tasks with agent baselines, and 5 established kernels with expert implementations on Hopper and Blackwell GPUs. The authors report speedups of 1.08×1.08\times to 3.04×3.04\times over the strongest agent baselines and 4.91×4.91\times to 9.18×9.18\times over PyTorch, and lower latency than the fastest plotted baseline in 8 of 10 established kernel-GPU settings. The largest speedup over an expert implementation is 1.91×1.91\times against FlashMLA's padded 64-head representation.

Critical strengths

S1. The decomposition is well designed. KernelMaster settles the dataflow and execution structure before asking the agent to write CUDA. This resembles how kernel optimization is usually approached manually. The intermediate checks also seem useful, particularly coverage and replay for the rewrites.

S2. The agent baselines use the same base model, which makes the comparison more useful. I also appreciate that Figure 8 includes the cases where KernelMaster loses, and that the authors explain the padding difference behind the largest FlashMLA speedup.

Critical weaknesses

Major

W1. Unclear optimization budget. I could not tell whether the 5-step budget (L402-403) includes KernelMaster's MathIR search, calibration and execution-graph ranking. These stages precede CUDA refinement, so matching refinement steps alone may leave different total optimization budgets. This could affect the comparison: 3 of the 5 gains over agent baselines are at most 1.2×1.2\times, and KernelMaster only passes the strongest final baseline at step 4 on differential-gated and nonlinear-latent MLA and at step 5 on page-factorized MLA (L429-431). Please report total optimization time and model/GPU usage, along with the number of independent runs and variation across runs behind Figure 6.

W2. Small margins over expert implementations without a noise floor. I am not convinced that the 1.003×1.003\times and 1.008×1.008\times results should be counted as wins without reported measurement uncertainty. The paper does not report timing variance, so these differences may be noise. The platform split also deserves discussion: excluding the padded FlashMLA comparison, the H20 results range from 0.941×0.941\times to 1.033×1.033\times, while all other gains of 10% or more (1.177×1.177\times, 1.336×1.336\times and 1.429×1.429\times) are on the RTX 5090 D (L464-471).

W3. Unclear contribution of the proposed mechanisms. The paper does not show which part of the pipeline accounts for the gains. I would like to see the same CUDA agent run with and without the execution contract under a matched total budget. Using a contract from an unranked candidate graph would also help test whether the cost-model ranking is useful. The NCU comparison with AutoKernel (L432-440) does not resolve this, since the two systems differ in more than execution synthesis. I also could not find how contract compliance is checked. Passing correctness tests does not establish that the CUDA code follows the specified CTA partitioning or Warp roles. Is the realized-lowering manifest (L348-349) used for this?

Minor

W4. The abstract's 1.91×1.91\times omits the padding caveat. The abstract advertises the 1.91×1.91\times peak without noting that it compares a 16-head specialization with FlashMLA's padded 64-head representation (L469-470); that caveat belongs beside the number. The 5 new tasks behind the claim about unfamiliar operations are all attention variants (L395-400), close to the online fold the paper uses as its MathIR example (Figure 2), so generality beyond attention and normalization kernels is untested.

Questions for authors

Q1. Does the 5-step budget include MathIR search, calibration and graph ranking? Please report the total optimization cost and the number of independent runs for each system in Figure 6.

Q2. What numerical tolerance thresholds and how many test inputs were used for the reference correctness checks at L370? Does the finite-precision policy (L178, L189-190) permit algebraic reassociation or reduced-precision intermediates that were not available to the baseline implementations?

Q3. What does the "common memory-state timing protocol" at L449 specify about warm-up runs, timing iterations and cache handling? Please report measurement uncertainty for the 10 established kernel-GPU comparisons.

Q4. How much performance is lost if the same CUDA agent runs without a contract, or with a contract from an unranked graph, under the same total budget?

Q5. On the H20 sparse MLA benchmark, how does KernelMaster compare with an expert kernel on a native 64-head workload, or with a 16-head expert implementation if one exists?

4 comments

Q1. Budget and independent runs

Experiment. We charge KernelMaster's full search to its first round and give AKO4X the same total token budget. We also repeat KernelMaster and every Agent baseline three times on both GPUs. All systems use Codex CLI as the Agent harness with GPT-5.6-sol at high reasoning effort.

Results. KernelMaster's search measures candidate structures with Agent-generated calibration kernels, which are stored in a calibration cache. Here we run the search without the calibration cache, so the Agent generates every calibration kernel anew and its full token cost is included. Equal total tokens on the RTX 5090 D, latency in µs, reported as min / max / avg:

BudgetAKO4XKernelMaster
After round 2, 31.98M tokens12.29 / 26.62 / 17.5814.34 / 18.40 / 15.69
After round 5, 39.52M tokens12.29 / 26.62 / 17.0712.29 / 14.34 / 12.97

Repeated runs on differential-gated MLA decode, latency in µs, reported as min / max / avg:

SystemRTX 5090 DH20
KernelMaster12.29 / 14.34 / 12.9714.46 / 16.58 / 15.36
AKO4X12.29 / 30.72 / 19.8018.74 / 27.79 / 24.66
AutoKernel12.29 / 32.77 / 23.6239.36 / 138.11 / 72.90
AdaExplore48.99 / 49.15 / 49.1028.83 / 38.69 / 33.66
KernelAgent51.20 / 128.98 / 96.2464.67 / 100.37 / 86.46

Conclusion. With the search included in the budget, KernelMaster still leads on average. It also has the smallest spread on both GPUs. Because the Agent implements a fixed, searched structure, the result depends little on run-to-run variation of the Agent.

Full details are in Appendix A.4 for the equal-budget comparison and Appendix A.3 (Table 4, Figure 11) for the repeated runs.

Q2. Tolerances, test inputs and finite-precision policy

Protocol. Correctness tolerances per task, which apply to all systems:

TaskOutput dtypeatolrtol
Differential-gated MLA decodeBF1610−210^{-2}10−210^{-2}
RMSNorm, hidden size 7168BF163×10−33 \times 10^{-3}3×10−33 \times 10^{-3}
Mamba2 SSU decodeBF16 output and state cache10−210^{-2}10−210^{-2}

Conclusion.

  • Tolerances. They follow FlashInfer-Bench; RMSNorm uses a stricter one.
  • Test inputs. Each kernel is checked on the task's deterministic fixed-seed workload, the same for all systems. MathIR rewrites are additionally replayed on three seeds and on perturbed-metadata cases.
  • Rewrites. They are real-arithmetic identities, such as partitioned online-softmax merges and applying the linear projection per partition before the merge. They change only the rounding order.
  • Precision. The selected candidates keep FP32 accumulation for all states, and the final-body contract requires FP32 states and the reference's casts.

Every system may use any arithmetic, and all are judged by the same tolerance.

The full evaluation protocol is in Appendix A.1 (Table 1).

Q3. Timing protocol and measurement uncertainty

Protocol.

  • Every launch is timed with CUDA events.
  • L2 is flushed before every warm-up and timed launch, outside the timed region.
  • Each trial runs 10 warm-up and 50 timed launches.
  • We report the median of three trial medians.
  • Clocks are fixed and locked, with no thermal or power throttling.

Results. Speedup with its 95% bootstrap interval, from the per-trial statistics:

GPUKernelSpeedup [95% CI]
RTX 5090 DPaged GQA1.003 [1.001, 1.005]
RTX 5090 DRMSNorm1.099 [1.046, 1.448]
RTX 5090 DDense MLA1.429 [1.427, 1.429]
H20Dense MLA1.008 [1.002, 1.012]

Conclusion. KernelMaster's trial-to-trial spread stays below 0.2 µs, and the intervals for 1.003× and 1.008× exclude 1. These small margins are therefore not noise. The baselines are expert production kernels, and reaching them automatically is the substance of the claim.

The protocol is detailed in Appendix A.1, and the statistics for all comparisons are in Appendix A.6 (Table 10).

Q4. Running the same Agent without a contract

Experiment. We give the same Agent, Codex + KM feedback, a read-only text description of the structure that KernelMaster selects: launch geometry, partitioning, synchronization, merge and dataflow. It receives neither the contract itself nor its audits.

Results. Latency in µs on the RTX 5090 D, reported as min / max / avg:

SystemLatency
Codex + KM feedback18.43 / 28.67 / 25.26
Codex + structure text, no contract10.24 / 28.67 / 16.38
KernelMaster, with contract12.29 / 14.34 / 12.97

Conclusion. Knowing the structure is not enough. With the text description, the Agent can reach a fast best run, but its average is worse and its spread wide. All fifteen of its kernels follow the described launch geometry, synchronization and merge. Only one reuses each loaded K and latent row across heads and branches, as the contract specifies. In the slowest run, the Agent reloads them per head.

The contract fixes these decisions and makes them mechanically checkable, which is why KernelMaster's runs stay close to its best. For candidates lower in the ranking, Appendix A.5 shows that their structural bounds are higher, consistent with the ranking.

Full details are in Appendix A.7 (Table 11), and the ranking analysis is in Appendix A.5.

Q5. H20 sparse MLA

Experiment. We re-measured FlashMLA on the H20 in one session, both on a native 64-head workload and on the 16-head workload padded to 64 heads. While re-measuring, we found that converting the softmax-scale tensor on the host inside the timed call added CPU overhead and made the FlashMLA timing fluctuate; we moved the conversion to setup, as in our other adapters. We then compared FlashMLA with three independent KernelMaster runs in the same session, using the corrected timing.

Results. H20, latency in µs, mean ± 95% CI over three interleaved repeats:

PathLatency
FlashMLA, native 64-head workload20.99 ± 0.22
FlashMLA, 16 heads padded to 6421.17 ± 0.18
SystemLatency, min / max / avg over three runsSpeedup over FlashMLA
FlashMLA, padded, corrected timing21.171.00×
KernelMaster17.71 / 18.59 / 18.051.14–1.20×

Conclusion. Padding costs FlashMLA only 0.18 µs (0.86%), which is within the measurement interval, because MLA decode reads one latent KV shared by all heads and the extra heads add little work. Under the corrected timing, KernelMaster is 1.14–1.20× faster than FlashMLA across all three runs. The KernelMaster kernel reported here comes from the revised search space, which admits multi-token KV tiles and keeps the attention scores on tensor-core MMA fragments.

The per-trial statistics are in Appendix A.6 (Table 10).

I appreciate the additional comparisons provided by the authors. The response answers my questions about the search budget and execution contract on differential-gated MLA (Q1/Q4), the timing protocol (Q3), and the H20 FlashMLA comparison (Q5).

On graph selection (W3/Q4), I found Appendix A.5 helpful for understanding how the ranking reflects hardware constraints. What remains open for me is whether the same agent produces a faster kernel from a higher-ranked graph when both graphs receive the same refinement budget.

We thank the reviewer for the response and for the careful reading of our rebuttal and the appendix. We address the remaining question on graph selection below.

Experiment. On differential-gated MLA decode on the RTX 5090 D, we take the graphs at the 0th, 25th and 75th percentiles of the 1541-candidate ranking. Each graph is passed to the same final-body Agent with the same KM feedback and the same budget of five refinement rounds, and each is run three times.

Results. Average latency in µs over three runs:

Ranking percentileAfter round 1After round 5
P019.3712.97
P2524.9213.66
P7553.9324.59

Conclusion. Under the same Agent and the same refinement budget, higher-ranked graphs give faster kernels on average, both after the first round and after the full budget. The final Agent optimizes the concrete implementation of the body, but lower-ranked graphs tend to converge more slowly, and the P75 graph remains 1.9× slower than the top-ranked graph after five rounds.

Reviewer_oXXWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents KernelMaster, an agentic system for GPU kernel optimization. It combines three things: mathematical dataflow search, hardware-calibrated execution synthesis, and contract-guided CUDA refinement. Here's the flow: it first uses MathIR to expose and validate mathematical transformations, projects the selected candidates into LoopIR, and then searches hierarchical Warp and CTA execution structures based on measurements of local service time, concurrency, resource usage, and HBM bandwidth. The chosen execution graph then becomes an "execution contract" that constrains the subsequent Agent-driven CUDA implementation and profiler-guided refinement. For evaluation, the paper tests KernelMaster on five newly specified attention workloads and five established kernels, running on an RTX 5090D and an H20 GPU. On the new workloads, it reports 1.08–3.04× speedups over the strongest evaluated Agent baseline and 4.91–9.18× over PyTorch references. On established kernels, it shows lower latency in eight of ten settings, with speedups up to 1.91×.

Critical strengths

The paper gives Agent-driven kernel optimization a coherent multi-level structure. KernelMaster cleanly separates mathematical transformation, loop-level representation, Warp-level pipeline design, CTA-level task scheduling, and the actual CUDA implementation. On top of that, explicit contracts for dependencies, state versions, buffer lifetimes, ownership, and synchronization give a principled way to keep later Agent revisions in check. That's a meaningful systems contribution, because profiler feedback on its own often tells you where the bottleneck is but not which abstraction level you should actually go and change.

The reported performance is promising, and the hardware analysis is genuinely useful. KernelMaster gets strong results on the five newly specified workloads, and it stays competitive with established implementations on two GPU platforms. The xHC-wrapped GQA analysis ties the performance gain to concrete changes in CTA partitioning, occupancy, issue-active rate, and memory throughput, rather than just reporting a speedup number. All of this suggests the system can produce kernels that are actually useful in practice, though on their own these results don't pin down how much each proposed component contributes.

Critical weaknesses

Missing component ablations. Figure 7 and the xHC analysis only test the full pipeline: gains across refinement steps and a comparison of two final kernels cannot separate MathIR, CTA synthesis, calibrated ranking, the contract, or Agent refinement. Controlled ablations are needed, such as dropping MathIR rewriting, using random or heuristic ranking, removing Warp–CTA feedback, or removing the contract under equal budgets, plus a direct check of predicted versus measured graph rankings. Without these, the paper shows the full system works, not that the combination is necessary.

Budgets are not controlled and overhead is not accounted for. Equal step counts do not imply equal budgets: KernelMaster runs MathIR search, LoopIR projection, Warp/CTA search, and calibration before its first CUDA kernel, none of which is charged to the five-step budget, and the KernelAgent comparison is weaker still since it uses only that system's first correct result. So the finding that the first implementation beats the baselines' final results is not evidence of lower cost, and while excluding compile/setup from latency is fine for deployment claims, it cannot support efficiency claims; the authors should report LLM calls and tokens, candidate and compile counts, GPU, profiler and wall-clock time, and performance-versus-total-budget curves.

Questions for authors
  1. Can the authors provide component-level ablations for MathIR rewriting, hierarchical Warp–CTA synthesis, hardware-calibrated ranking, and contract-guided refinement? Which components account for the reported gains under a matched total budget?
  2. What exactly constitutes one optimization step for KernelMaster and each baseline? Does the five-step budget include all mathematical search, execution-graph search, calibration, compilation, profiling, and GPU measurements before the first CUDA implementation? Please report total LLM, GPU, compilation, and wall-clock costs, including the assumed amortization of reusable calibration.
  3. How accurately does the calibrated execution-graph model rank complete candidate implementations? Please report prediction-versus-measurement correlation, top-k recall, and selected-candidate regret, and compare against simple heuristic or random ranking under the same evaluation budget.
  4. Can the authors report repeated timing or confidence intervals for the established-kernel comparison? In particular, the reported 1.003× and 1.008× improvements may be within normal measurement noise.
2 comments

Q1. Component ablations

Experiment. On differential-gated MLA decode, the task of Figure 6, we add KernelMaster's components one at a time to the same Agent, on both GPUs:

  1. Codex only.
  2. Codex + KM feedback, which is KernelMaster's feedback loop without search.
  3. Codex + MathIR + KM feedback.
  4. Full KernelMaster: MathIR, the hierarchical execution search, and the contract-guided final body.

Every system uses Codex CLI as the Agent harness with GPT-5.6-sol at high reasoning effort, and runs three times. Each run's result is its best correct round.

Results. Latency in µs, reported as min / max / avg:

SystemRTX 5090 DH20
Codex only18.43 / 36.86 / 26.6216.66 / 87.97 / 40.52
Codex + KM feedback18.43 / 28.67 / 25.2619.26 / 50.83 / 35.39
Codex + MathIR + KM feedback12.29 / 20.48 / 16.3822.59 / 35.12 / 28.20
KernelMaster12.29 / 14.34 / 12.9714.46 / 16.58 / 15.36

Conclusion. Each component brings a consistent gain on both GPUs:

  • The feedback loop improves both the average and the worst run over the Agent alone.
  • Adding MathIR lowers the average further.
  • The searched, contract-bound execution structure gives the lowest average and by far the smallest spread.

Stability matters as much as the average. Every KernelMaster run lands close to its best result, while the other configurations vary by up to 5× between runs.

Full results, including per-round trajectories, are in Appendix A.2 (Table 3, Figures 9–10).

Q2. Definition of a step and total cost

Experiment. One step is one Agent round that produces, checks and times one kernel. We charge KernelMaster's full search to its first round. AKO4X, the strongest baseline, receives the same total token budget: we continue each of its three runs in its own thread. KernelMaster's search measures candidate structures with Agent-generated calibration kernels, which are stored in a calibration cache. We report two settings. Without the calibration cache, the Agent generates every calibration kernel anew. With the calibration cache, the search reuses the calibration kernels of an earlier search of the same task and makes no LLM calls.

Results. We compare at B(k), the total token budget KernelMaster has used after its final-body round k. Latency in µs on the RTX 5090 D, reported as min / max / avg:

Without calibration cache:

kB(k)AKO4XKernelMaster
129.75M12.29 / 26.62 / 17.7514.34 / 28.67 / 20.48
231.98M12.29 / 26.62 / 17.5814.34 / 18.40 / 15.69
333.75M12.29 / 26.62 / 17.0712.29 / 14.34 / 12.97
539.52M12.29 / 26.62 / 17.0712.29 / 14.34 / 12.97

With calibration cache:

kB(k)AKO4XKernelMaster
111.91M14.34 / 26.62 / 19.1114.34 / 28.67 / 20.48
214.14M14.34 / 26.62 / 19.1114.34 / 18.40 / 15.69
315.91M14.34 / 26.62 / 19.1112.29 / 14.34 / 12.97
521.68M12.29 / 26.62 / 17.7512.29 / 14.34 / 12.97

Tokens per run, min / max / avg:

SystemTokens per run
AKO4X, five rounds4.44 / 16.12 / 9.16M
KernelMaster without calibration cache38.71 / 41.06 / 39.52M
KernelMaster with calibration cache20.87 / 23.22 / 21.68M

Wall-clock time:

StageWall-clock time
KernelMaster search, without calibration cacheabout 86 minutes
KernelMaster search, with calibration cacheabout 28 minutes
KernelMaster five-round final body41–57 minutes, of which GPU measurement takes a few minutes
AKO4X five-round run17–32 minutes

About two thirds of the search time is spent in Agent-driven stages. These run at most four Agent calls in parallel through Codex CLI, so higher API concurrency would shorten the search.

Conclusion. With the search fully charged, KernelMaster's average is lower than AKO4X's at the same token budget from the second round on, with or without the calibration cache. Extra tokens help AKO4X little: beyond its original five rounds, one run improves and the others stay at their earlier results. Device calibration is a one-time cost per GPU, reused by every task.

Full details are in Appendix A.4 (Tables 6–7, Figure 12), and Appendix A.1 (Table 2) lists the rounds each system runs.

Q3. Accuracy of the calibrated ranking

Experiment. We take candidates at rank percentiles P0–P100 of the 1,541-candidate ranking on the RTX 5090 D. For each, we compare KernelMaster's predicted device time with an analytic device roofline. The roofline uses no KernelMaster cost formula, only the candidate's geometry and bandwidth curves we measure with a stream kernel.

Results. Device time in µs:

PercentileRankGrid × blockKM predictionAnalytic roofline
P01256 × 1289.761.98
P25386256 × 12812.512.79
P50771512 × 12814.672.84
P75115664 × 12817.483.95
P90138764 × 12821.013.95
P10015411024 × 3248.707.32

Rank correlation between the prediction and the roofline:

SetSpearman ρKendall τ
79 candidates0.840.67
19 distinct structures0.810.65

Conclusion. The ranking follows an independent hardware bound from top to bottom:

  • Top-ranked structures have the lowest bounds.
  • The tail is slower for identifiable reasons. Fewer CTAs lower the achievable DRAM bandwidth, and fewer threads per CTA lower the achievable L2 bandwidth.

The roofline lies below the prediction in absolute terms because it bounds only bandwidth and compute. The cost model also captures latency and synchronization.

Full details, with all sampled percentiles, are in Appendix A.5 (Tables 8–9, Figure 13).

Q4. Uncertainty of the established-kernel comparison

Experiment. For every established-kernel comparison, we report the per-trial values and a 95% bootstrap interval of the speedup, from 20,000 resamples of each side's trials.

Results. Latency in µs, reported as min / max / median of the trial values:

GPUKernelKernelMasterFastest baselineSpeedup [95% CI]
RTX 5090 DPaged GQA12.229 / 12.279 / 12.25012.288 / 12.288 / 12.2881.003 [1.001, 1.005]
RTX 5090 DRMSNorm4.220 / 4.254 / 4.2214.448 / 6.112 / 4.6401.099 [1.046, 1.448]
RTX 5090 DDense MLA14.336 / 14.336 / 14.33620.464 / 20.480 / 20.4801.429 [1.427, 1.429]
H20Dense MLA19.216 / 19.392 / 19.33619.424 / 19.488 / 19.4881.008 [1.002, 1.012]

Conclusion. KernelMaster's trial-to-trial spread stays below 0.2 µs. The intervals for the two small margins exclude 1, so 1.003× and 1.008× are small but consistent gains, not noise. The baselines are expert-written production kernels, mostly hand-written CUDA such as TensorRT-LLM's XQA and FlashInfer's kernels. KernelMaster reaches or exceeds them automatically, starting from a PyTorch reference, and that is the contribution we claim.

The statistics for all established-kernel comparisons are in Appendix A.6 (Table 10), and the timing protocol is in Appendix A.1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.