- Rating
- 3
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces KernelCompass, an LLM-based agentic GPU kernel optimization framework that adapts to both hardware constraints and application-level objectives. It combines parameter tuning with structural rewriting, using measured performance and resource utilization to guide optimization decisions. At the model level, it introduces task contracts to select potentially promising operators and evaluate kernel improvements against end-to-end workload objectives rather than focusing purely on individual kernel's latency. The authors validate the approach through comparisons with existing kernel optimization frameworks across multiple GPUs and workloads, ablation studies examining the effectiveness of resource-aware feedback, and end-to-end LLM inference experiments demonstrating workload-specific performance improvements.
Critical strengths
None
Critical weaknesses
- Although the paper identifies six agents and provides their inputs, outputs, and search budgets, the actual decision-making process remains insufficiently specified. In particular, it is unclear how the Resource Analysis Agent interprets potentially conflicting hardware measurements, how these diagnoses are translated into concrete instructions for the Structural Rewrite Agent, and how the LLM selects among possible structural transformations. The paper also omits the actual agent prompts and LLM generation settings. Since these LLM-driven decisions constitute a central part of the proposed approach, the missing details make it difficult to reproduce the methodology or determine which components are responsible for the reported improvements.
- Although Section 3.3 mathematically defines performance and resource slopes, the practical mechanism for using them remains unclear. The paper does not sufficiently explain how the two parameter configurations used to estimate each slope are selected, how the resource measurements are obtained, or how different slope values lead to specific rewriting decisions. Given the variety of GPU resource measurements, a more precise description of the instrumentation and interpretation procedure is necessary. Overall, the connection between these measurements and concrete structural rewriting decisions remains insufficiently specified
Questions for authors
- Are the baseline comparisons conducted under comparable computational budgets? Section 4.1 reports different budget measures for KernelCompass and the competing frameworks. Could the authors clarify the actual wall-clock time, GPU usage, and LLM inference costs of each approach? An added bonus would be showing the input prompt, example kernel output, and different trajectories of the kernels being generated/rewritten.
4 comments
(Part 2/3) W1 (continued). Prompts and settings. We quote, verbatim, the three prompt passages that govern the decisions you ask about: how evidence selects a hypothesis, how a departure from a label is justified, and how a predicted resource change is checked against the generated child. The complete prompts of all agents will be released with the source code upon publication, as the paper's reproducibility statement says.
- Analysis agent: "A WALL or COST label is a hypothesis that continuing a direction that helps is stopped by the named dimension; it is not a proof of the full-task bottleneck. … If you depart from a label, write
override: <reason>at the start of that entry'sevidence. IMPROVE means the parameter has not converged at this configuration: it is not a resource target." - Transformation timings, given to A4 only: "A row that is slower than the parent at theta* AND at the probe points does not relieve the wall at an acceptable price on this kernel; a row that is faster at the probe points than the parent at theta* reopens the direction. The child's own retune and the unchanged correctness gate still decide."
- Rewrite agent: "
expectationsstates, PER RESOURCE DIMENSION, which way you expect the number to move … It is checked against the measurement afterwards and the result is shown to you next round, so it is how a structural claim becomes falsifiable rather than plausible."
| setting | value |
|---|---|
| base model | GLM-5.3-max, via an agent runtime with file and shell tools |
| candidates per generation / rewrite round | 4 / 2 |
| retune per candidate | 40 TPE trials, 10 random start-up |
| output limit per turn / time limit per call | 128k tokens / 3600 s |
| sampling | provider default |
The release also covers the configurations, the complete tuning records and the evaluation scripts.
How the LLM selects among transformations. A new experiment answers this by changing only what the agent is told. The agent, parent, prompt template, list of transformations and budget are fixed. A2 gives raw paired readings; A3 adds what is derived from them (slopes, labels, summary, cost estimates); A4 adds the transformation timings: each transformation applied to the parent and timed.
| A2 | A3 | A4 | |
|---|---|---|---|
| full runs: R = child/parent, 3 replicates, each regenerated and retuned | 0.897 / 0.744 / 0.891 | 0.742 / 0.752 / 0.812 | 0.751 / 0.760 / 0.768 |
| mean ± s.d. | 0.844 ± 0.086 | 0.768 ± 0.038 | 0.759 ± 0.009 |
| separate decision runs (5 analysis-and-rewrite runs per group, no retune): implemented the fastest transformation we tested | 0/5 | 1/5 | 5/5 |
The derived guidance lowers mean R from 0.844 to 0.768 and the standard deviation from 0.086 to 0.038; with the transformation timings, all five decision runs implement the fastest transformation we tested.
(The trajectory and the budget comparison follow in Part 3.)
(Part 3/3)
The trajectory you asked for (A4, replicate 2), from the measurements to the instruction, the generated code and its retune.
| min | step | content |
|---|---|---|
| 0 | input | parent 5.648 ms at θ*; paired readings, labels, summary, cost estimates, transformation timings |
| 2–11 | analysis agent | switch the MMA accumulation to fp16; cites its measured timing (−23.9% at BLOCK_M 256, 6 spills instead of 72); risk named: the correctness gate |
| 11–32 | rewrite agent | tl.dot(…, out_dtype=ACC_DT); expects registers 255, spills 6, shared 32768 → 49152 B |
| 32–42 | parameterization agent | adds the accumulator dtype as a tunable switch |
| 42–85 | retune + final | optimum at BLOCK_M 256 (costly on the parent); final 4.29 ms, R = 0.760 |
Lines from the generated child, verbatim: acc = tl.zeros((BLOCK_M, BLOCK_N), dtype=ACC_DT); d = tl.dot(a.to(CAST_DT), b.to(CAST_DT), out_dtype=ACC_DT); return acc + d.to(ACC_DT), with ACC_DT = fp16 at the chosen configuration.
A second trajectory for contrast, the median A3 replicate: it changes the weight precision instead, casting the weights once per call to the compute dtype, and keeps the fp32 accumulator.
Question. Comparable budgets. Section 4.1 reports a different budget measure for each framework, so we reran all five under one 12 h limit on ConvT2d-Mish. Same RTX 4090 machine, same API provider and plan, and the final candidates re-evaluated in the same environment. Ours is the fastest candidate at the fewest tokens and the least agent-call time:
| Ours | K-Search | ShinkaEvolve | KernelFoundry | KernelBand | |
|---|---|---|---|---|---|
| wall-clock (h) | 12.1 | 12.0 (limit) | 4.7 (128 gens done) | 12.0 (limit) | 12.0 (limit) |
| generated tokens (M) | 0.61 | 1.87 | 7.45 | 8.32 | 2.66 |
| cumulative agent-call time (h) | 6.5 | 7.1 | 30.7 | 35.6 | 10.8 |
| GPU time (h) | 7.1 | 4.2 | 3.0 | 3.5 | 0.5 |
| final latency (ms) | 2.954 | 3.070 | 3.019 | 3.848 | 4.602 |
ShinkaEvolve stops at 4.7 h because it finishes its own 128-generation budget, running 8 proposals at once; agent-call time adds up all calls, concurrent ones included, which is why its 4.7 h costs 4.7× ours. It measures execution time, not API charges. We find our candidate at 10.3 h and stop at 12.1 h, since the budget is checked only between candidates. Over the 18 task–GPU settings of Table 1, our median agent-call time is 10.6 h against 29.0–57.9 h for the baselines, and our cumulative agent-call time is lower than every baseline in 16 of the 18 settings. Our GPU time there is 7.8 h median.
The procedure, the prompt excerpts and the code trajectory make the path from measured feedback to a rewrite decision explicit. We welcome further questions on any of these steps.
Dear Reviewer,
Thank you again for the time you put into reviewing our paper.
The discussion period is about to close. We have replied to each point in your review above and uploaded the revised paper. We would be grateful if you could take a look. If anything is still unclear, please tell us while we can still reply, and we will be glad to explain further.
With thanks, The authors
(Part 1/3) Thank you for the review and for the detailed questions on how measurements become decisions. The base model (GLM-5.3-max), evaluation setup and gates are those of the paper throughout.
W2. How the two configurations are chosen and how a slope leads to a decision (answered first, since W1 builds on it). Seven steps, stated in §3.3 and Appendix B.9 of the revision:
near= the tuned configuration θ* with all other parameters fixed;probe= the next legal value along the direction;far= the first of the three nearest legal values the other way that compiles, passes the gate and can be timed.OutOfResourceswith required/limit, the shared-memory compile screen, or per-block register demand above the device limit.The slopes describe the latency–resource trade-off along each tested parameter move. The ranking of WALL and COST directions uses the measured relative latency gain.
Worked example, the L2:32 parent of our new experiment, direction BLOCK_M+:
Performance slope −0.0155 ms per unit of BLOCK_M, register slope −0.0152 ms per register: 65 more registers per thread come with a 0.99 ms gain. Spills and warps do not change between far and near, so they get no slope. Label: COST(registers), because the probe is valid but slower and spills rose from 0 to 72.
The feedback points to keeping the larger tile while reducing accumulator pressure. The NUM_WARPS+ measurements show that reducing per-thread work cuts registers 230→167 but costs 33.7%. A4 then measures two structural alternatives at θ*: splitting the accumulator costs 37.9%, while fp16 accumulation gains 10.6%. The agent implements fp16 accumulation.
W1. How the agents handle conflicting measurements. A fixed diagnostic procedure guides how conflicting measurements are interpreted. Its diagnosis order (launch overhead, CPU-side issue rate, saturation, capacity, latency) checks saturation before capacity pressure. When a direction has capacity events in several dimensions, the label is COST{d1, d2}, and the analysis agent selects a target and explains why. Any departure from a label must be written as
override:with a reason.The diagnosis becomes an instruction in a fixed form: the analysis agent writes hypotheses with
change,expected_effectandrisk, the rewrite agent implements one and states which way it expects every resource to move, and the child's measurements are checked against those expectations. Its retuned latency decides acceptance.(Prompts, settings and the experiment follow in Part 2; one trajectory and the budgets in Part 3.)