- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that LLM kernel agents are fine at local code fixes but bad at the big structural decisions, like which mathematical form of the computation to use and how to organize the work across warps and thread blocks. Profiler feedback tells you a kernel is slow, but not which of these levels to change. KernelMaster decides the structure before the agent writes any code. An LLM picks among legal math rewrites of the PyTorch reference (e.g. fusing ops, or folding a downstream multiply into a running reduction), each checked by replaying against the reference. Then the system searches how warps and thread blocks split the work, ranking plans with a cost model built from small calibration kernels timed on the target GPU. The best plan becomes a fixed contract the agent follows while writing and refining the CUDA code. On five new attention variants it beats four agent systems on every task, and on five established kernels it's faster than expert libraries like FlashInfer and FlashMLA in most settings.
Critical strengths
The central idea is really interesting. Splitting the problem into the math form, the work organization on the GPU and the low-level code, and handing the agent a checked structure instead of letting it figure out all three by trial and error, is a nice way to make agent-driven kernel optimization less blind. The premise that agents do well on instruction-level tweaks but struggle with structural choices also sounds right to me.
The evaluation setup is fair. They compare with four recent agent systems using the same LLM, use tasks designed to be unfamiliar to the model, and also compare with expert libraries on recent GPUs instead of just PyTorch eager. If the wins over hand-tuned libraries hold up, that's a solid result.
Critical weaknesses
W1. The central claim, that the searched structure is what makes the agent good, is never actually tested. There's no ablation at all, so I can't tell how much comes from the math rewrite search, how much from the execution planning, and how much from just giving a strong LLM a detailed plan. Removing each stage in turn would answer this. The execution planning also relies completely on the calibrated cost model, but there's no check that it ranks plans correctly (e.g. predicted vs measured time across candidates). The agent comparison needs more care too. Everyone gets five optimization steps, but KernelMaster does its whole rewrite search and calibration before step one, and KernelAgent is stopped at its first correct result, so it's comparing steps, not effort. It also looks like single runs with no variance, and some margins over the best baseline are small enough that noise could flip them.
W2. The scope is narrow. All five new tasks are attention variants and the established kernels are mostly attention plus RMSNorm. The flagship rewrite (folding the softmax-times-V product into a running reduction) is essentially the FlashAttention trick, so this might show the system is very good at one family of kernels, not that it generalizes. Matmul-heavy kernels, convolutions, MoE routing or scans would be much more convincing. Some wins over expert libraries also come from specialization, e.g. FlashMLA pads to more heads than the workload needs. That's a legit advantage, but a narrower one than "beats expert implementations".
W3. The paper is hard to follow. There are lots of new terms (MathIR, LoopIR, WrapIR, cuts, slots, channels, contracts), explained mostly in dense prose with one worked example, no algorithm box and no appendix. Claims that the projection and composition preserve semantics are stated, not shown. So it's hard to tell what's fully automatic, what relies on the LLM, and what needed hand engineering per task, and I don't think I could reproduce it from the paper.
Questions for authors
- Can you ablate the math rewrite search, the execution planning and the contract one at a time? To me this is the most important missing piece.
- How accurate is the cost model, i.e. how well do predicted times rank candidate plans vs measured latency?
- How much total effort does each system use (LLM calls, wall-clock time, including your search and calibration)? Would KernelAgent catch up if it kept refining after its first correct result?
- How much do results vary across repeated runs?
- How does it do outside attention and normalization, e.g. matmul-heavy ops, convolutions or MoE routing?
- Which parts are fully automatic, and which needed manual work per task (like defining legal rewrites or writing calibration kernels)?
5 comments
Q4. Variance across runs
Experiment. We repeat every system three times on both GPUs. We also run KernelMaster three times on RMSNorm, each with an independent search.
Results. Latency in µs, reported as min / max / avg:
| Task | System | RTX 5090 D | H20 |
|---|---|---|---|
| Differential-gated MLA | KernelMaster | 12.29 / 14.34 / 12.97 | 14.46 / 16.58 / 15.36 |
| AKO4X | 12.29 / 30.72 / 19.80 | 18.74 / 27.79 / 24.66 | |
| AutoKernel | 12.29 / 32.77 / 23.62 | 39.36 / 138.11 / 72.90 | |
| AdaExplore | 48.99 / 49.15 / 49.10 | 28.83 / 38.69 / 33.66 | |
| KernelAgent | 51.20 / 128.98 / 96.24 | 64.67 / 100.37 / 86.46 | |
| RMSNorm (7168) | KernelMaster | 4.172 / 4.264 / 4.205 | 6.281 / 6.541 / 6.425 |
Conclusion. KernelMaster has the smallest spread on both GPUs and both tasks. The three independent RMSNorm searches select the same structure, so the search itself is reproducible. Fixing its result before implementation keeps run-to-run variation of the Agent small.
Full details, including per-round trajectories, are in Appendix A.3 (Tables 4–5, Figure 11).
Q5. Beyond attention and normalization
Experiment. We add Mamba2 selective-state-update decode, a state-space operator. We use the FlashInfer-Trace definition and reference verbatim, and compare against FlashInfer, the Agent baselines and a measured bandwidth roofline of 23.99 µs in the same timing scope.
Results. Latency in µs on the RTX 5090 D:
| Method | Latency | Fraction of roofline |
|---|---|---|
| PyTorch reference | 743.87 | 0.03 |
| FlashInfer | 36.86 | 0.65 |
| AutoKernel | 26.62 | 0.90 |
| AdaExplore | 26.62 | 0.90 |
| KernelAgent | 30.72 | 0.78 |
| AKO4X | 37.28 | 0.64 |
| KernelMaster feedback loop, min / max / avg | 24.58 / 30.78 / 27.33 | 0.78 / 0.98 / 0.89 |
Conclusion. SSU decode is a bandwidth-bound gather–update–scatter over a persistent state cache. Its only reduction is a 128-element dot product. The roofline shows that little room remains for a multi-stage pipeline or a different work decomposition: the best kernel already reaches 0.98 of it. We therefore evaluate this operator with KernelMaster's evaluation and feedback loop, without the structure search. KernelMaster's best run beats FlashInfer and every Agent baseline under the same contract.
Full details are in Appendix A.8 (Table 12, Figure 14).
Q6. Automatic versus manual
KernelMaster is fully automatic:
- MathIR is derived automatically from the captured operator graph together with Agent-generated rewrites.
- LoopIR, the calibration kernels and the final CUDA kernel are produced by Agents, with automatic checks.
- The search, calibration and ranking are deterministic.
For each task, the user supplies only the PyTorch reference and its workload.
Thanks for the detailed response and the additional experiments. To be transparent, I see this paper as somewhere around a 2.5. I've decided to raise my score to 3 because I find the core idea interesting and think it could be valuable to the community, but several important concerns remain.
The main one is the scope of the evaluation. All the new tasks are attention variants and the established kernels are mostly attention plus RMSNorm, and the flagship rewrite is essentially the FlashAttention technique. The new Mamba2 task is run without the structure search, so it doesn't show whether MathIR and the execution search work beyond attention. I think results with the full pipeline on kernels with a different structure, such as matmul-heavy operators, convolutions or MoE, are needed to support the claim that the approach is general, and it would help to know which MathIR rewrite types are actually used across tasks.
Second, the cost model is compared with an analytic roofline rather than with measured latencies, so it's still unclear how accurately it ranks candidates in practice. Finally, the response says LoopIR is produced by Agents while the paper describes a deterministic projection, which should be clarified, and the correctness tolerances are still not stated.
I hope the authors can address these points in the revision.
Q1. Component ablations
Experiment. We add KernelMaster's components one at a time to the same Agent on differential-gated MLA decode, on both GPUs, with three runs each. All systems use Codex CLI as the Agent harness with GPT-5.6-sol at high reasoning effort.
Results. Latency in µs, reported as min / max / avg:
Conclusion. Each component brings a consistent gain:
Full results, including per-round trajectories, are in Appendix A.2 (Table 3, Figures 9–10).
Q2. Accuracy of the cost model
Experiment. We take candidates at rank percentiles P0–P100 of the 1,541-candidate ranking on the RTX 5090 D and compare KernelMaster's predicted device time with an analytic device roofline. The roofline uses no KernelMaster cost formula, only the candidate's geometry and bandwidth curves we measure with a stream kernel.
Results. Device time in µs:
Rank correlation between the prediction and the roofline:
Conclusion. The ranking follows an independent hardware bound from top to bottom: top-ranked structures have the lowest bounds, and the tail is slower because fewer CTAs and fewer threads per CTA lower the achievable bandwidth. The cost model therefore orders candidates by the same hardware limits that bound them, so the search selects good structures for principled reasons.
Full details, with all sampled percentiles, are in Appendix A.5 (Tables 8–9, Figure 13).
Q3. Total effort and KernelAgent
Experiment. We charge KernelMaster's full search to its first round and give AKO4X, the strongest baseline, the same total token budget. KernelMaster's search measures candidate structures with Agent-generated calibration kernels, which are stored in a calibration cache. Without the calibration cache, the Agent generates every calibration kernel anew. With the calibration cache, the search reuses the calibration kernels of an earlier search of the same task and makes no LLM calls.
Results. RTX 5090 D, reported as min / max / avg:
In wall-clock time, a search takes about 86 minutes without the calibration cache and 28 minutes with it, and a final body takes 41–57 minutes. About two thirds of the search time is spent in Agent-driven stages, so higher API concurrency would shorten it.
Conclusion. KernelMaster leads at equal total tokens, while additional tokens help AKO4X little. KernelAgent prompts only for correctness and has no optimization loop of its own, so we keep its original single-pass design rather than altering the baseline.
Full details are in Appendix A.4 (Tables 6–7, Figure 12).