acceptodds
Under review as a conference paper at ICLR 2027

KernelCompass: Traversing the Feasible Resource Space of the Target GPU Toward a Task-Contracted Objective

Abstract

GPU kernel optimization must account for both the hardware and the user's goal: a kernel that is faster in isolation need not make the model faster under its intended workload. We present KernelCompass, an agent framework that combines hardware resource limits, measured bandwidth and compute throughput, and tuning records to adapt kernel resource demand to the target device. Within one algorithm structure, changing a single parameter while holding the others fixed yields paired performance and resource feedback that guides rewrites to relieve resource constraints while using available headroom. Rewritten structures are retuned, and the new measurements guide the next round. At the model level, a task contract fixes the metric, workload, quality conditions, and acceptance criterion before optimization starts, and drives both the choice of operator groups and the acceptance of candidates under the native workload. With GLM-5.3-max as the base model, KernelCompass is to faster than the strongest of four baseline frameworks in each of the 18 combinations of six KernelBench tasks and three GPUs. On Qwen3-4B on an RTX 4090, separate goal-specific optimizations reduce time to first token (TTFT) by 3.56%, raise single-stream throughput by 12.51%, and raise concurrent throughput by 9.29% under the respective tested workloads.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Agent Framework, Kernel Optimization, AI For Infra

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 5 reviews, above 27% of the others.
Reviewer_dq2QOpenReview ↗
Rating
3
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces KernelCompass, an LLM-based agentic GPU kernel optimization framework that adapts to both hardware constraints and application-level objectives. It combines parameter tuning with structural rewriting, using measured performance and resource utilization to guide optimization decisions. At the model level, it introduces task contracts to select potentially promising operators and evaluate kernel improvements against end-to-end workload objectives rather than focusing purely on individual kernel's latency. The authors validate the approach through comparisons with existing kernel optimization frameworks across multiple GPUs and workloads, ablation studies examining the effectiveness of resource-aware feedback, and end-to-end LLM inference experiments demonstrating workload-specific performance improvements.

Critical strengths

None

Critical weaknesses
  • Although the paper identifies six agents and provides their inputs, outputs, and search budgets, the actual decision-making process remains insufficiently specified. In particular, it is unclear how the Resource Analysis Agent interprets potentially conflicting hardware measurements, how these diagnoses are translated into concrete instructions for the Structural Rewrite Agent, and how the LLM selects among possible structural transformations. The paper also omits the actual agent prompts and LLM generation settings. Since these LLM-driven decisions constitute a central part of the proposed approach, the missing details make it difficult to reproduce the methodology or determine which components are responsible for the reported improvements.
  • Although Section 3.3 mathematically defines performance and resource slopes, the practical mechanism for using them remains unclear. The paper does not sufficiently explain how the two parameter configurations used to estimate each slope are selected, how the resource measurements are obtained, or how different slope values lead to specific rewriting decisions. Given the variety of GPU resource measurements, a more precise description of the instrumentation and interpretation procedure is necessary. Overall, the connection between these measurements and concrete structural rewriting decisions remains insufficiently specified
Questions for authors
  • Are the baseline comparisons conducted under comparable computational budgets? Section 4.1 reports different budget measures for KernelCompass and the competing frameworks. Could the authors clarify the actual wall-clock time, GPU usage, and LLM inference costs of each approach? An added bonus would be showing the input prompt, example kernel output, and different trajectories of the kernels being generated/rewritten.
4 comments

(Part 1/3) Thank you for the review and for the detailed questions on how measurements become decisions. The base model (GLM-5.3-max), evaluation setup and gates are those of the paper throughout.

W2. How the two configurations are chosen and how a slope leads to a decision (answered first, since W1 builds on it). Seven steps, stated in §3.3 and Appendix B.9 of the revision:

  1. Directions. Up to six numeric axes from the tuning record, those at a domain boundary first and then by effect size, in both directions; a categorical axis gets at most two alternative values.
  2. Endpoints. near = the tuned configuration θ* with all other parameters fixed; probe = the next legal value along the direction; far = the first of the three nearest legal values the other way that compiles, passes the gate and can be timed.
  3. Timing. 3 blocks × 100 launches. "X faster than Y" requires (T_Y − T_X)/T_near > δ, with δ = max(1%, block spread of X, block spread of Y).
  4. Resources. Compiler metadata (registers, spills, shared memory), analytical occupancy (resident blocks/SM and the limiting resource), SASS instruction counts.
  5. Wall evidence. A rejected probe counts as a wall only if the device named the dimension: Triton OutOfResources with required/limit, the shared-memory compile screen, or per-block register demand above the device limit.
  6. Capacity events between near and probe: spills rise, resident blocks/SM fall, or the limiter becomes dimension d.
  7. Labels, first match wins (below). WALL and COST directions are ranked by (T_far − T_near)/T_near. Each gets a short summary: the slopes with their zero-denominator markers, the binding dimension, the kernel the axis controls, and the applicable transformations with a cost estimate taken from the measured parameter moves.
labelrule
N.A.no legal probe or far point (domain edge or domain rule, not a device limit)
UNKNOWNa required point not measured or failing correctness, a refusal naming no device dimension, or block spread > 5% (a refusal with device evidence is WALL)
IMPROVEprobe or far valid and faster than near
TIEnear not faster than far, or probe within δ of near
WALL(d)near faster than far; probe rejected with device evidence naming d
COST(d)near faster than far; probe valid but slower, with a capacity event in d
UNEXPLAINEDnear faster than far; probe slower; no tracked capacity event (no resource target)

The slopes describe the latency–resource trade-off along each tested parameter move. The ranking of WALL and COST directions uses the measured relative latency gain.

Worked example, the L2:32 parent of our new experiment, direction BLOCK_M+:

endpointBLOCK_Mlatencyresources
far646.675 ms165 registers, 3 blocks/SM
near = θ*1285.684 ms230 registers, 2 blocks/SM, no spills
probe25611.68 ms72 spills

Performance slope −0.0155 ms per unit of BLOCK_M, register slope −0.0152 ms per register: 65 more registers per thread come with a 0.99 ms gain. Spills and warps do not change between far and near, so they get no slope. Label: COST(registers), because the probe is valid but slower and spills rose from 0 to 72.

The feedback points to keeping the larger tile while reducing accumulator pressure. The NUM_WARPS+ measurements show that reducing per-thread work cuts registers 230→167 but costs 33.7%. A4 then measures two structural alternatives at θ*: splitting the accumulator costs 37.9%, while fp16 accumulation gains 10.6%. The agent implements fp16 accumulation.

W1. How the agents handle conflicting measurements. A fixed diagnostic procedure guides how conflicting measurements are interpreted. Its diagnosis order (launch overhead, CPU-side issue rate, saturation, capacity, latency) checks saturation before capacity pressure. When a direction has capacity events in several dimensions, the label is COST{d1, d2}, and the analysis agent selects a target and explains why. Any departure from a label must be written as override: with a reason.

The diagnosis becomes an instruction in a fixed form: the analysis agent writes hypotheses with change, expected_effect and risk, the rewrite agent implements one and states which way it expects every resource to move, and the child's measurements are checked against those expectations. Its retuned latency decides acceptance.

(Prompts, settings and the experiment follow in Part 2; one trajectory and the budgets in Part 3.)

(Part 2/3) W1 (continued). Prompts and settings. We quote, verbatim, the three prompt passages that govern the decisions you ask about: how evidence selects a hypothesis, how a departure from a label is justified, and how a predicted resource change is checked against the generated child. The complete prompts of all agents will be released with the source code upon publication, as the paper's reproducibility statement says.

  • Analysis agent: "A WALL or COST label is a hypothesis that continuing a direction that helps is stopped by the named dimension; it is not a proof of the full-task bottleneck. … If you depart from a label, write override: <reason> at the start of that entry's evidence. IMPROVE means the parameter has not converged at this configuration: it is not a resource target."
  • Transformation timings, given to A4 only: "A row that is slower than the parent at theta* AND at the probe points does not relieve the wall at an acceptable price on this kernel; a row that is faster at the probe points than the parent at theta* reopens the direction. The child's own retune and the unchanged correctness gate still decide."
  • Rewrite agent: "expectations states, PER RESOURCE DIMENSION, which way you expect the number to move … It is checked against the measurement afterwards and the result is shown to you next round, so it is how a structural claim becomes falsifiable rather than plausible."
settingvalue
base modelGLM-5.3-max, via an agent runtime with file and shell tools
candidates per generation / rewrite round4 / 2
retune per candidate40 TPE trials, 10 random start-up
output limit per turn / time limit per call128k tokens / 3600 s
samplingprovider default

The release also covers the configurations, the complete tuning records and the evaluation scripts.

How the LLM selects among transformations. A new experiment answers this by changing only what the agent is told. The agent, parent, prompt template, list of transformations and budget are fixed. A2 gives raw paired readings; A3 adds what is derived from them (slopes, labels, summary, cost estimates); A4 adds the transformation timings: each transformation applied to the parent and timed.

A2A3A4
full runs: R = child/parent, 3 replicates, each regenerated and retuned0.897 / 0.744 / 0.8910.742 / 0.752 / 0.8120.751 / 0.760 / 0.768
mean ± s.d.0.844 ± 0.0860.768 ± 0.0380.759 ± 0.009
separate decision runs (5 analysis-and-rewrite runs per group, no retune): implemented the fastest transformation we tested0/51/55/5

The derived guidance lowers mean R from 0.844 to 0.768 and the standard deviation from 0.086 to 0.038; with the transformation timings, all five decision runs implement the fastest transformation we tested.

(The trajectory and the budget comparison follow in Part 3.)

(Part 3/3)

The trajectory you asked for (A4, replicate 2), from the measurements to the instruction, the generated code and its retune.

minstepcontent
0inputparent 5.648 ms at θ*; paired readings, labels, summary, cost estimates, transformation timings
2–11analysis agentswitch the MMA accumulation to fp16; cites its measured timing (−23.9% at BLOCK_M 256, 6 spills instead of 72); risk named: the correctness gate
11–32rewrite agenttl.dot(…, out_dtype=ACC_DT); expects registers 255, spills 6, shared 32768 → 49152 B
32–42parameterization agentadds the accumulator dtype as a tunable switch
42–85retune + finaloptimum at BLOCK_M 256 (costly on the parent); final 4.29 ms, R = 0.760

Lines from the generated child, verbatim: acc = tl.zeros((BLOCK_M, BLOCK_N), dtype=ACC_DT); d = tl.dot(a.to(CAST_DT), b.to(CAST_DT), out_dtype=ACC_DT); return acc + d.to(ACC_DT), with ACC_DT = fp16 at the chosen configuration.

A second trajectory for contrast, the median A3 replicate: it changes the weight precision instead, casting the weights once per call to the compute dtype, and keeps the fp32 accumulator.

Question. Comparable budgets. Section 4.1 reports a different budget measure for each framework, so we reran all five under one 12 h limit on ConvT2d-Mish. Same RTX 4090 machine, same API provider and plan, and the final candidates re-evaluated in the same environment. Ours is the fastest candidate at the fewest tokens and the least agent-call time:

OursK-SearchShinkaEvolveKernelFoundryKernelBand
wall-clock (h)12.112.0 (limit)4.7 (128 gens done)12.0 (limit)12.0 (limit)
generated tokens (M)0.611.877.458.322.66
cumulative agent-call time (h)6.57.130.735.610.8
GPU time (h)7.14.23.03.50.5
final latency (ms)2.9543.0703.0193.8484.602

ShinkaEvolve stops at 4.7 h because it finishes its own 128-generation budget, running 8 proposals at once; agent-call time adds up all calls, concurrent ones included, which is why its 4.7 h costs 4.7× ours. It measures execution time, not API charges. We find our candidate at 10.3 h and stop at 12.1 h, since the budget is checked only between candidates. Over the 18 task–GPU settings of Table 1, our median agent-call time is 10.6 h against 29.0–57.9 h for the baselines, and our cumulative agent-call time is lower than every baseline in 16 of the 18 settings. Our GPU time there is 7.8 h median.

The procedure, the prompt excerpts and the code trajectory make the path from measured feedback to a rewrite decision explicit. We welcome further questions on any of these steps.

Dear Reviewer,

Thank you again for the time you put into reviewing our paper.

The discussion period is about to close. We have replied to each point in your review above and uploaded the revised paper. We would be grateful if you could take a look. If anything is still unclear, please tell us while we can still reply, and we will be glad to explain further.

With thanks, The authors

Reviewer_RtD3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

KernelCompass uses paired performance/resource measurements to guide kernel rewrites within hardware limits, then retunes the resulting code. A fixed task contract governs model-level operator selection and acceptance. Four agent frameworks are compared on six KernelBench tasks across three NVIDIA GPUs; a four-start ablation examines the feedback.

Critical strengths
  • This work considered evaluation under the intended workload well. A locally slower candidate improves model TTFT, showing why isolated timing can mislead. Disjoint final prompts test the fixed task contracts beyond the search inputs, while logits/NLL checks assess quality as objectives change.
  • This work also considered evaluation under measured resource changes. The parameter contrasts connect rewrites to their resource footprint, allowing changes in spills and shared-memory use to be interpreted alongside residency. Performance gains survive cross-generation re-evaluation and independent full-K/K-loop retuning.
Critical weaknesses
  • The ablation has four starts on two tasks without repeated agent generations. Adding slopes also adds raw measurements and diagnostics, leaving their individual contributions unresolved. The method loses to performance-only feedback on CausalAttention/1, and history-only feedback already finds K-loop tiling. Three-seed retuning reuses fixed code, so the effect of feedback on code generation remains uncertain.
  • The eight-kernel RMSNorm reference leaves fusion headroom; comparison with a standard fused baseline would clarify the value of objective-specific search. Matmul–Swish stores weights at 8 bits, raising a further question about comparable optimization allowances. The conversion/preprocessing costs and baseline access to this change need clarification before the gains can be attributed to better search.
Questions for authors
  • For the 8-bit Matmul–Swish candidate in Appendix C.7, is weight conversion included in the reported latency? If conversion is performed beforehand, under what conditions can the converted weights be reused?
4 comments

(Part 1/2) Thank you for recognizing the workload-specific acceptance and the link between rewrites and their resource footprint. The added experiments keep the paper's setup: same base model (GLM-5.3-max), same evaluation and gates.

W1. The feedback ablation. A new experiment addresses the three gaps you name: it repeats the agent generation, separates the bundled signals, and lets each generation write its own code instead of retuning fixed code. Over three generations per group, the derived guidance lowers mean R from 0.844 to 0.768 and the standard deviation from 0.086 to 0.038; adding the transformation timings brings the standard deviation to 0.009.

Task: KernelBench L2:32 (Conv2d+scale+channel-min). Parent: a fused Triton kernel frozen at θ*, the configuration its 40-trial retune chose; its BLOCK_M+ direction is COST(registers): 72 spills and +105.5% at BLOCK_M 256. The agent, parent, prompt template, list of transformations and budget are fixed. A2: raw paired readings. A3: A2 plus what is derived from it (slopes, labels, ranked summary, cost estimates), no new measurement. A4: A3 plus transformation timings, each transformation applied to the parent as a small change and timed.

Full runs: each replicate generates its own code and retunes it for 40 trials. R = child/parent latency, same card.

groupR (rep 1/2/3)mean ± s.d.paired vs A2 (geom. mean)
A20.897 / 0.744 / 0.8910.844 ± 0.086—
A30.742 / 0.752 / 0.8120.768 ± 0.0380.913
A40.751 / 0.760 / 0.7680.759 ± 0.0090.903

Transformation timings, vs the parent at θ* (5.683 ms):

transformationat θ*at BLOCK_M 256at NUM_WARPS 2
none (parent)—+105.5%, 72 spills+111.8%, 70 spills
split accumulator (textbook)+37.9%+18.3%, 2 spills+21.4%, 6 spills
fp16 MMA accumulator−10.6%−23.9%, 6 spills−16.7%, 8 spills

Decision runs, a separate experiment with no retune: 5 analysis-and-rewrite runs per group, the implemented change classified with the group names hidden:

groupfp16 accumulator (fastest tested)split accumulatorfp16 weightsother
A20/50/53/52/5
A31/50/53/51/5
A45/50/50/50/5

All three A4 children retune to BLOCK_M 256, where the parent was at 128: with the fp16 accumulator that move goes from +105.5% to −14.9% (5.080→4.325 ms).

On fixed code, the three-seed retuning gives gains of 24.3 / 33.2 / 39.3%; the experiment above adds what it could not show, the effect on code the agent writes fresh.

The K-loop written under resource feedback has 12.2% lower latency than the one written under history-only (4.100 against 4.672 ms), and cuts its parent's registers from 255 to 140. Both groups choose K-loop tiling; the difference is the quality of the generated implementation.

Across starting points, Table 2a shows 4/4 over history-only and 3/4 over performance-only; at CausalAttn/1 performance-only is ahead, 0.8042 against 0.9735. Table 2a reports one result per group and starting point, which is why the experiment above repeats the generation three times per group.

(The RMSNorm comparison and the 8-bit candidate follow in Part 2.)

(Part 2/2)

W2a. RMSNorm against fused baselines. We added the comparison you ask for: Liger-Kernel 0.8.5, torch.compile(RMSNorm) and F.rms_norm, on the same 73 sites of Qwen3-4B under the paper's contracts, with CUDA-graph decode so no per-launch host overhead is counted.

Decode throughput gain over the 8-kernel reference:

replacementsingle-streamconcurrent
ours, single-stream objective+4.95%+3.78%
ours, concurrent objective+4.95%+3.89%
Liger-Kernel+4.91%+3.83%
torch.compile(RMSNorm)+4.93%+3.83%
F.rms_norm+3.82%+3.23%

Our searched decode kernels match Liger's 12.85 ms device time per token. On TTFT our candidate gains 3.19% over the reference while staying bit-identical to it on the held-out blocks; none of the three fused baselines is bit-identical there. Which implementation wins therefore depends on the objective, which is what the contract selects (§4.4).

W2b / Question. 8-bit Matmul–Swish (C.7). Weight conversion runs once in __init__, so the reported latency is the steady state; the conversion itself is in the startup row below. The converted weights are reused as long as the weights, device and configuration are unchanged. Every timed call includes the input quantization: with a new input each time the latency is 1.303 ms instead of 1.266 ms. Startup and steady state of the five RTX 4090 candidates on one card:

OursK-SearchShinkaEvolveKernelFoundryKernelBand
weightsint8, once in __init__int8, converted once, cachedint8, converted once, cachedfp32fp32, cached re-layout
steady state (ms)1.2662.3966.2755.7486.248
construct + upload + first call (s)14.645.19.99.037.5
calls until ours is ahead in total time—1≈950≈12601

On baseline access to the same change: lowering internal precision is permitted for every framework, with output precision and shape fixed to the reference (§4.1); K-Search and ShinkaEvolve both took it and cache int8 weights too, and all five candidates pass the same fp64 relative-error gate. (Revision: the experiment in §4.3, these tables in the appendix, each candidate's internal precision in Appendix C.8, Table 20.)

Thank you for the precise questions on the ablation and the baselines. Please let us know if any of these points needs further detail.

Thank you for the additional experiments and clarifications. These address some of my concerns, particularly regarding quantization and preprocessing costs. However, the added value of the proposed method over existing approaches remains insufficiently established, so I will retain my original rating.

Thank you for the reply, and for confirming that the quantization and preprocessing questions are addressed.

On existing approaches, Table 1 compares with K-Search, ShinkaEvolve, KernelFoundry and KernelBand under the same base model, and ours is the fastest in all 18 task–GPU settings. For the standard fused RMSNorm baseline your review asked for, we added three on the same 73 sites. Our searched kernels have the highest throughput gains over the reference in both single-stream and concurrent decoding. Under the TTFT objective, our candidate improves TTFT by 3.19% and is the only tested replacement that stays bit-identical to the reference on the held-out blocks.

To measure what the feedback itself contributes, we ran three independent generations for each setting, with new code written every time. With the raw measurements held fixed, adding the derived guidance lowered the mean child-to-parent latency ratio from 0.844 to 0.768. In a separate experiment, all five runs that were given the transformation timings implemented the fastest transformation we tested. None of the five given raw measurements alone did.

Which of these comparisons do you find insufficient to establish the added value over existing approaches, and what does it leave open? If you have another existing approach in mind, please name it.

Reviewer_8qiXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes an agent framework that alternates parameter tuning and structural rewriting to optimize GPU kernels. It uses paired performance and hardware-resource feedback to guide structural rewrites, followed by retuning to evaluate their benefits. It uses the task contracts which specify workloads, correctness requirements, and optimization objectives to guide kernel-group selection and candidate acceptance. The authors evaluate the framework on six KernelBench tasks across A800, RTX4090 and RTX5090 and compare against KernelFoundry, K-Search, KernelBand and ShinkaEvolve, the framework shows consistent kernel performance improvement.

Critical strengths

The evaluation shows encouraging results across tasks and hardware. It covers six KernelBench tasks on three GPU architectures and compares against four existing optimization frameworks. Under the main experimental setup, KernelCompass achieves lower final-kernel latency in all 18 task–device combinations, with final candidates re-evaluated in the same environment.

Critical weaknesses

This framework has limited modeling of execution dynamics and unclear attribution of gains. The proposed resource feedback loop primarily demonstrate the resource usage and capacity constraints, but does not capture temporal execution effects, like the computation-memory overhead and synchronization overhead. Although source-level rewriting and empirical timing may indirectly account for these effects, the paper does not sufficiently demonstrate how the proposed feedback guides such optimizations. Moreover, the resource-feedback ablation bundles multiple signals, making it difficult to isolate the contribution of resource slopes. It therefore remains unclear how much of the reported gain comes from the proposed guidance, rather than optimizations such as improved low-level assembly instruction selection, kernel fusion, or pipelining that a general-purpose agent might also discover.

Questions for authors

My concern is more fundamental than the absence of an isolated ablation. The proposed resource representation does not explicitly capture execution dependencies, synchronization, or computation–memory overlap, although these can substantially affect latency even when resource usage remains unchanged. Moreover, parameter changes can alter both resource usage and execution behavior, so the resulting resource slopes do not distinguish these effects. This leaves a central methodological gap: it is unclear why these local associations provide reliable guidance for structural rewrites. The reported speedups demonstrate that the overall optimization loop can find faster kernels, but do not establish that the proposed resource representation adequately supports the structural decisions attributed to it.

4 comments

(Part 1/3) Thank you for the review. How local measurements can support a structural decision is the central question for the method, and we answer it first, with a new controlled experiment. Every experiment added here uses the paper's base model (GLM-5.3-max), evaluation setup and gates.

KernelCompass makes three contributions (§1): an agent framework for task–hardware adaptation, paired resource and performance guidance for structural rewriting, and task-driven kernel selection and acceptance. At the kernel level, Table 1 shows lower latency in all 18 task–GPU settings after common re-evaluation, as you note. The added tests keep that lead in all 24 task–shape cases, including 18 new shapes, without retuning; beat the best passing torch.compile configuration on all six tasks; and cover three irregular workloads timed over the whole forward pass. At the model level, §4.4 reports gains under three objectives on Qwen3-4B: TTFT 3.56%, single-stream throughput 12.51%, concurrent throughput 9.29%.

Your question concerns the second contribution.

1. With the raw measurements fixed, the derived guidance improves the generated code. The experiment separates raw measurements, derived guidance and transformation timings across repeated agent generations. Parent: a fused Triton kernel for KernelBench L2:32 (Conv2d+scale+channel-min), frozen at θ*, the configuration its 40-trial retune chose (230 registers, 2 blocks/SM; BLOCK_M+ is COST(registers)). The agent, parent, prompt template, list of transformations and budget are fixed:

  • A2: raw paired readings;
  • A3: A2 plus everything derived from it (slopes, zero-denominator exclusions, labels, ranked summary, an estimated cost per transformation), with no new measurement;
  • A4: A3 plus transformation timings: each applicable transformation applied to the parent as a small change, timed at θ* and at two other configurations.

A3 against A2 tests the derived guidance; A4 against A3 tests what timing the transformations adds. Each replicate is a separate analysis, rewrite and parameterization session plus a 40-trial retune. R = child/parent latency on the same card; lower is better.

groupR (rep 1/2/3)mean ± s.d.worstpaired vs A2 (geom. mean)
A20.897 / 0.744 / 0.8910.844 ± 0.0860.897—
A30.742 / 0.752 / 0.8120.768 ± 0.0380.8120.913
A40.751 / 0.760 / 0.7680.759 ± 0.0090.7680.903

A3 adds no measurement to A2. On identical evidence, the derived guidance lowers mean R from 0.844 to 0.768, the worst replicate from 0.897 to 0.812, and the standard deviation from 0.086 to 0.038. Timing the transformations (A4) brings the standard deviation to 0.009.

2. What a local measurement is used for. The paired latency and resource measurements, and the slopes derived from them, identify which trade-off a rewrite should test. On this parent, raising BLOCK_M from 64 to 128 lowers latency while registers rise from 165 to 230. At 256 the kernel spills 72 registers and runs 105.5% slower. A larger NUM_WARPS cuts registers from 230 to 167 but costs 33.7%. Together, these measurements point to one rewrite hypothesis: keep the larger tile's gain and reduce the accumulator's register pressure, without paying the latency cost of raising NUM_WARPS. Retuning then checks whether the new structure makes that parameter direction pay. The same tile move before and after one transformation:

MMA accumulatorBLOCK_M 128BLOCK_M 256effect of the larger tile
fp32 (parent)5.683 ms11.680 ms, 72 spills+105.5%
fp16 (rewrite)5.080 ms4.325 ms, 6 spills−14.9%

fp16 accumulation keeps the larger tile's reuse and cuts its register pressure. The textbook alternative, splitting the accumulator, leaves only 2 spills at BLOCK_M 256 but takes 6.721 ms, 18.3% slower than the tuned parent. The smallest resource count is not the fastest structure, so the agent needs the latency cost of each option.

(Part 2: the transformation timings, which rewrite gets implemented, the labels, and the rewrites for dependencies and synchronization. Part 3: costly rewrites, one trajectory and the torch.compile comparison.)

(Part 2/3) 3. The feedback changes which rewrite is implemented. The transformation timings given to A4, relative to the parent at θ* (5.683 ms):

transformationat θ*at BLOCK_M 256at NUM_WARPS 2
none (parent)—+105.5%, 72 spills+111.8%, 70 spills
split accumulator (textbook)+37.9%+18.3%, 2 spills+21.4%, 6 spills
fp16 MMA accumulator−10.6%−23.9%, 6 spills−16.7%, 8 spills

In a separate experiment with no retune, each group runs 5 analysis-and-rewrite sessions, and the implemented change is classified with the group names hidden:

groupfp16 accumulator (fastest tested)split accumulatorfp16 weightsother
A20/50/53/52/5
A31/50/53/51/5
A45/50/50/50/5

A2 has the same model, the same list of transformations and the same raw measurements as the other groups, and implements the fastest tested transformation in none of its five runs. With the transformation timings, A4 implements it in all five. That transformation, fp16-accumulating MMA, is an instruction-level choice of the kind you mention. Everything else is fixed, so the feedback is what changes the implemented rewrite. After full generation and retuning, all three A4 children settle at BLOCK_M 256, the tile size that cost the parent 105.5%; the parent itself was tuned to 128.

4. Parameter changes affect resources and execution together. §3.3 therefore reads the latency and resource responses along the same tested parameter change, and two checks keep the guidance tied to what was measured:

  • zero denominator (Eq. 3): if a resource does not change along the direction, it gets no slope at all;
  • UNEXPLAINED: on the L2:32 parent, BLOCK_M 128→64 improves every resource we track (registers 230→165, resident blocks 2→3, no spills) and is still 17.4% slower. The label records that the latency change has no resource explanation, and no agent is told to reduce a resource here.

Capacity is also not the first thing checked. Each candidate goes through a fixed order: launch overhead, CPU-side issue rate, bandwidth or compute saturation, capacity limits, and finally latency. Results over all candidates of the 18 main runs:

capacity limitedbandwidth/compute saturatedmixedundeterminedCPU-side or launch overheadtotal
208794440335

Measured latency decides every acceptance, and each diagnosis lists the signals unavailable in our profiling setup: bank conflicts, divergence, cache hit rates, stall reasons.

5. The framework also produces rewrites for dependencies and synchronization. On block-sparse attention, one of the irregular tasks we added, no candidate was diagnosed capacity-limited (15 launch overhead, 5 compute saturated). Two rewrites the framework kept:

taskparent diagnosisrewritelatency
block-sparse attentionlaunch overhead dominates, 0 spills; with other parameters fixed, pipeline stages 1/2/5 give 0.268/0.276/0.270 msthe analysis agent attributes this to a data-dependent block-selection check in every loop iteration; the check is removed and block selection is separated from the loop over blocks0.268 → 0.175 ms (−34.6%); stages 1→2 now help (0.183 → 0.175 ms)
MoElaunch overhead dominates: device synchronizations caused by .item() on the routing countslaunch grids sized by an upper bound remove the synchronizations (CPU/GPU time 1.02 → 0.08)with fp16 weight copies, 3.874 → 3.108 ms (−19.8%)

The first case repeats point 2 on a dependency instead of a register limit: more pipeline stages do nothing for the parent, and after the check is removed stages 1→2 are worth 4.45%.

(Three rewrites flagged as costly, one trajectory and the torch.compile comparison follow in Part 3.)

(Part 3/3) 6. The cost estimates catch expensive rewrites. In three cases on different parents, the transformation the estimate flagged as costly stayed costly:

parenttransformation flagged as costlyoutcome
ConvT2d-Mish candidate, independent runsplit accumulator4.917 vs 3.739 ms after retune (+31.5%)
ConvT2d-Mish candidate, another runsplit accumulator4.970 vs 4.435 ms after retune (+12.1%)
L2:32 parent abovesplit over output channels (the rewrite agent's own child)+24% at its own parameters; retune reverted it

7. One trajectory (A4, replicate 2): from the measurements to the generated code and its retune.

minstepcontent
0inputparent 5.648 ms at θ*; readings, labels, summary, cost estimates, transformation timings
2–11analysis agentfp16 MMA accumulation, citing its measured timing (−23.9% at BLOCK_M 256, 6 spills instead of 72); risk named: the correctness gate
11–32rewrite agenttl.dot(…, out_dtype=ACC_DT); expects registers 255, spills 6, shared 32768 → 49152 B
32–42parameterization agentadds the accumulator dtype as a tunable switch
42–85retune + finaloptimum at BLOCK_M 256; final 4.29 ms, R = 0.760

8. System level. We ran torch.compile in three modes (default, max-autotune, max-autotune-no-cudagraphs) and three precisions (FP32, TF32, bf16) through the same gate. Ours is faster than each task's best passing configuration:

ConvT2d-MishMatmul-SwishConvT3d-GNMBConvCausalAttnMamba2-SSD
compile (ms)7.514.725.5410.435.8210.04
ours (ms)3.601.272.463.722.951.36
ours faster2.1×3.7×2.3×2.8×2.0×7.4×

From the same measurements, the derived guidance improves the mean and the consistency of the generated code. The transformation timings lead all five decision runs to the fastest tested rewrite, and all three retuned A4 children reach BLOCK_M 256. These results connect the feedback to structural decisions. The kernel-level results in Part 1 and the model-level results of §4.4 evaluate the rest of the system. (Revision: §3.2 diagnosis order; §3.3 UNEXPLAINED and endpoints; §4.3 the controlled experiment.)

Thank you for pressing on the methodological question; it led to the controlled experiment above. We would welcome any further questions on it.

Dear Reviewer,

Thank you again for the time you put into reviewing our paper.

The discussion period is about to close. We have replied to each point in your review above and uploaded the revised paper. We would be grateful if you could take a look. If anything is still unclear, please tell us while we can still reply, and we will be glad to explain further.

With thanks, The authors

Reviewer_U2quOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes KernelCompass, an agent framework for GPU kernel optimization that adapts kernels to both the target hardware and a model-level objective under target workload. At the kernel level, it alternates parameter search with LLM-driven structural rewrites. A rewrite is triggered when latency keeps improving along a parameter direction until the next value becomes infeasible because it exceeds a resource limit. Rewrites are guided by paired slopes: finite-difference ratios ΔT/Δθ (performance slope) and ΔT/Δr_d (resource slope), measured between two configurations that differ in a single parameter. Each rewritten structure is then retuned. At the model level, an agent translates the user's goal into a fixed task contract consist of metric, workload, quality gates, acceptance rule. The contract is used to group modules into operator groups, allocate search budget according to each group's measured effect on the end-to-end metric, and accept candidates under the native workload rather than by isolated operator speedup.

Core Experiments & Results:

  1. With GLM-5.3-max as the base model, KernelCompass beats four agentic frameworks (KernelFoundry, K-Search, KernelBand, ShinkaEvolve) on all 18 combinations of 6 KernelBench tasks × 3 GPUs (A800 / RTX 4090 / RTX 5090), by 1.014×–3.144×.
  2. A feedback-masking ablation on 4 starting points compares history-only, performance-slope, and resource-slope feedback.
  3. On Qwen3-4B (RTX 4090), RMSNorm candidates optimized separately for each objective improve TTFT, single-stream throughput, and concurrent throughput by 3.56%, 12.51%, and 9.29%. Further experiments show that candidates do not transfer well across objectives, and that an operator that is slower in isolation (0.98×) can still reduce TTFT end to end (−3.32%).
Critical strengths
  1. The paper recongnizes the model-level objective-driven acceptance, with an example that a RMSNorm candidates optimized for one objective can hurt another. These are clean counterexamples to the common practice of accepting kernels by isolated operator speedup, while many other kernel optimization or kernel agent papers skip such discussion.

  2. The experiment and evaluation is broad and consistent, covering multiple tasks with different bottleneck profiles (memory-bound or compute-bound or long-chain) on three GPU architectures, and providing detailed and reliable results of latency and speedup.

Critical weaknesses

The kernel-level advantage is not very attributable to the proposed method, and the comparison is not controlled. The direct evidence for the paper's specific contributions is thin.

  1. Fair comparision over search budget. The paper presents all results in table and numbers. It could be more convicing if a figure of performance (speedup vs baseline) - cost of API and physical time could be presented. The search budget difference might cause different end speedups.

  2. Fair comparison over low precision. The experiment allows changing internal precision of computing process. An obvious example is that in C7, the target kernel is Matmul-swish of FP32 lienar with a swish activation, which is a memory-bound case and the bottleneck is reading all the weights. The generated kernel makes an 8-bit quantization over all the weights to release the bottlenecks. And it can be inferred that weights are quantized but the quantization latency is excluded in the timing.

  3. For the paired-slope mechanism - Contribution 2, the only direct test is the feedback ablation in Table 2a. It uses 4 starting points from 2 tasks on one GPU, with apparently one LLM sample per group. The resource-slope group beats the performance-slope group on only 3 of 4 starting points, and on CausalAttention it is clearly worse (0.9735 vs. 0.8042). The history-only group independently found the same K-loop rewrite. Variance from resampling the LLM could plausibly exceed these differences, and the authors acknowledge the limited scale.

Questions for authors

Questions:

  1. Precision. What internal precision does the final candidate use and is it a fair comparison? Were the baselines held to the same correctness gate, including permission to lower internal precision?
  2. Budget. What are KernelCompass's total token cost, API time, and GPU-hours per task?
  3. Variance. Are all the trials repeated? How do the Table 1 results and the Table 2a ablation vary across seeds and repeated LLM samples (mean ± std)?
  4. Bseline. Is torch.compile baseline a strong candidate to beat?

Suggestions:

  1. Tests on more complicated and latest kernels: KDA, MoE.
  2. The resource dimensions are mostly static - registers, shared memory, spills. Profiler-based dynamic metrics like warp stall reasons from Nsight Compute might identify the binding resource more directly. They could also be compared against the current signals.
  3. Add stronger non-LLM baselines, such as torch.compile(mode="max-autotune"). This would separate the contribution of LLM-driven structural rewriting from conventional autotuning.
3 comments

(Part 1/2) We thank the reviewer for recognizing the model-level, objective-driven acceptance. The experiments added below keep the paper's base model (GLM-5.3-max), evaluation setup and gates.

Attribution and the Table 2a experiment. A new experiment measures what the feedback contributes, on code the agent generates and retunes from scratch. With the parent, agent and budget fixed, the derived guidance lowers mean child/parent latency from 0.844 to 0.768 and its standard deviation from 0.086 to 0.038, over three replicates per group.

Task: KernelBench L2:32 (Conv2d+scale+channel-min). Parent: a fused Triton kernel frozen at θ*, the configuration its 40-trial retune chose (230 registers, 2 blocks/SM). Its BLOCK_M+ direction is COST(registers): at BLOCK_M 256 the kernel spills 72 registers and runs 105.5% slower. The three groups:

  • A2: raw paired readings;
  • A3: A2 plus everything derived from it (slopes, labels, ranked summary, cost estimates), with no new measurement;
  • A4: A3 plus transformation timings: each applicable transformation applied to the parent as a small change and timed.

All three get the same list of transformations, so A3 differs from A2 only in the derived guidance (§4.3.1 of the revision), and A4 adds only the transformation timings.

Full runs. Each replicate is a separate analysis, rewrite and parameterization session plus a 40-trial retune. R = child/parent latency on the same card.

groupR (rep 1/2/3)mean ± s.d.worstpaired vs A2 (geom. mean)
A20.897 / 0.744 / 0.8910.844 ± 0.0860.897—
A30.742 / 0.752 / 0.8120.768 ± 0.0380.8120.913
A40.751 / 0.760 / 0.7680.759 ± 0.0090.7680.903

Transformation timings (what A4 adds), relative to the parent at θ* (5.683 ms):

transformationat θ*at BLOCK_M 256at NUM_WARPS 2
none (parent)—+105.5%, 72 spills+111.8%, 70 spills
split accumulator (textbook)+37.9%+18.3%, 2 spills+21.4%, 6 spills
fp16 MMA accumulator−10.6%−23.9%, 6 spills−16.7%, 8 spills

The split accumulator spills less but never beats the tuned parent; the fp16 accumulator does, so the choice has to be made on measured latency.

Decision runs, a separate experiment with no retune: 5 analysis-and-rewrite runs per group. We classified the change each one implemented with the group names hidden:

groupfp16 accumulator (fastest tested)split accumulatorfp16 weightsother
A20/50/53/52/5
A31/50/53/51/5
A45/50/50/50/5

All three A4 children end at BLOCK_M 256, where the parent was at 128. The rewrite is what makes the larger tile pay: with the parent's fp32 accumulator, 128→256 costs 105.5% (5.683→11.680 ms); with the fp16 accumulator the same move gains 14.9% (5.080→4.325 ms).

Table 2a. The K-loop written under resource feedback has 12.2% lower latency than the one written under history-only (4.100 against 4.672 ms), and cuts its parent's registers from 255 to 140. Both groups choose K-loop tiling; the difference is the quality of the generated implementation. The performance-feedback full-K implementation takes 5.076 ms.

Across the four starting points, resource feedback beats history-only at 4/4 and performance-only at 3/4; at the fourth, CausalAttn/1, performance-only is ahead (0.8042 against 0.9735). Table 2a reports one result per group and starting point, so the new experiment repeats independent generation within each feedback setting and measures both mean performance and variability.

Variance. Timing: each final candidate in 3 separate processes × 100 launches. Across the six tasks, the three process medians lie within 0.06–0.60% of each other. Search: independent searches with seeds 0 / 1 / 2, re-measured on one RTX 4090:

taskours, seeds 0 / 1 / 2 (ms)mean ± s.d.Table 1 baselines, re-measured (ms)
Mamba2-SSD1.313 / 1.326 / 1.3281.322 ± 0.008strongest 1.415
ConvT2d-Mish3.536 / 2.954 / 3.7733.421 ± 0.4213.84 / 4.08 / 4.18 / 6.04

On both tasks, all three independent searches are faster than every Table 1 baseline (same base model, re-measured on the same card). Generation: the three replicates per group above measure how much newly generated code varies, which the three-seed retuning of fixed code does not.

torch.compile baseline. We ran 6 tasks × {default, max-autotune, max-autotune-no-cudagraphs} × {FP32, TF32, bf16} through the same gate. Against the fastest configuration that passes, per task:

ConvT2d-MishMatmul-SwishConvT3d-GNMBConvCausalAttnMamba2-SSD
compile (ms)7.514.725.5410.435.8210.04
ours (ms)3.601.272.463.722.951.36
ours faster2.1×3.7×2.3×2.8×2.0×7.4×

(Budget, precision and suggestions in Part 2.)

(Part 2/2) Budget and cost. Under the same 12 h limit, KernelCompass produces the fastest final candidate (2.954 ms) with the fewest generated tokens (0.61 M) and the least cumulative agent-call time (6.5 h).

(a) All five frameworks, rerun under the same 12 h limit on ConvT2d-Mish: same RTX 4090 machine, same API provider and plan, final candidates re-evaluated in the same environment. Every quantity below is logged for every framework:

OursK-SearchShinkaEvolveKernelFoundryKernelBand
wall-clock (h)12.112.0 (limit)4.7 (128 gens done)12.0 (limit)12.0 (limit)
generated tokens (M)0.611.877.458.322.66
cumulative agent-call time (h)6.57.130.735.610.8
GPU time (h)7.14.23.03.50.5
final latency (ms)2.9543.0703.0193.8484.602

We find our candidate at 10.3 h and stop at 12.1 h, because the budget is checked only between candidates (Appendix B.2). ShinkaEvolve finishes its own 128-generation budget in 4.7 h by running eight proposals at once, which is why it spends 4.7× our agent-call time in a shorter wall-clock. Agent-call time adds up the duration of all calls, concurrent ones included, and ours also covers the compile-and-test steps inside each call. It measures execution time, not API charges.

(b) All 18 task–GPU settings of Table 1, agent-call time in hours:

OursKernelFoundryK-SearchKernelBandShinkaEvolve
median10.657.929.044.756.8
mean ± s.d.12.0 ± 5.259.1 ± 10.929.8 ± 12.045.0 ± 10.868.9 ± 28.3
range4.0–20.938.4–83.410.5–56.128.4–64.737.2–127.9
settings where ours is lower—18/1816/1818/1818/18

Our median is 10.6 h against 29.0–57.9 h for the baselines, and our cumulative agent-call time is lower than every baseline in 16 of the 18 settings. Our GPU time is 7.8 h median (3.1–14.6), almost all of it the 40-trial parameter searches; generated tokens 1.7 M median (0.8–3.1), uncached input 3.3 M. The revision plots latency against cost from these tables.

Precision and the 8-bit candidate. For every framework, §4.1 fixes the output precision and shape to the reference and leaves internal precision open to rewriting; every final candidate goes through the same gates. The Matmul-Swish candidates on one RTX 4090:

OursK-SearchShinkaEvolveKernelFoundryKernelBand
weightsint8, once in __init__int8, converted once, cachedint8, converted once, cachedfp32fp32, cached re-layout
steady state (ms)1.2662.3966.2755.7486.248
construct + upload + first call (s)14.645.19.99.037.5
calls until ours is ahead in total time—1≈950≈12601

Every timed call includes the input quantization; with a new input on every call the latency is 1.303 ms instead of 1.266 ms. K-Search and ShinkaEvolve also convert their weights to int8, and all five candidates pass the same fp64 relative-error gate. Appendix C.8 of the revision (Table 20) lists the internal precision of every final candidate in Table 1.

Suggestions. KDA and MoE: we ran one 12 h search on each, plus block-sparse attention. All three final candidates pass the gate; timing covers the whole forward pass (medians; eager / torch.compile TF32 / ours): MoE, 64 experts top-8 with unbalanced routing, 21.0 / 18.2 / 3.120 ms; block-sparse attention, seq 4096, 14.0 / 4.90 / 0.140 ms; KDA, seq 2048, 305 ms / compilation did not finish in 30 min / 0.415 ms.

Dynamic profiler metrics: stall reasons are the right signal to refine our labels, and these counters are unavailable in our profiling setup. The feedback therefore uses compiler metadata, analytical occupancy, SASS counts and measured timing, and marks a latency change that no tracked resource explains as UNEXPLAINED, with no resource target. These signals guided two execution-level rewrites on the new tasks: removing a data-dependent check in block-sparse attention (−34.6%) and device synchronization in MoE (−19.8%, together with fp16 weight copies).

We appreciate the concrete suggestions, several of which are now experiments in the revision. If any part of the comparison remains unclear, we would be happy to clarify.

Dear Reviewer,

Thank you again for the time you put into reviewing our paper.

The discussion period is about to close. We have replied to each point in your review above and uploaded the revised paper. We would be grateful if you could take a look. If anything is still unclear, please tell us while we can still reply, and we will be glad to explain further.

With thanks, The authors

Reviewer_PGGyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces KernelCompass, which improves agentic kernel optimization by letting the agent workflow tune parameters and rewrite kernels. KernelCompass guides the agent for higher performance gains by exploring the feasible kernel space through analyzing resource constraints and bottlenecks. KernelCompass is tested on 6 different kernel tasks with 3 different GPUs, and shows up to 3.14x speedup compared to the strongest tested baseline.

Critical strengths

S1. The paper addresses an important problem, which is GPU kernel optimization requires substantial effort, and recent advances in LLMs provide an opportunity to automate this process.

S2. In the main evaluation, the paper shows consistent performance improvements across six kernel tasks and three GPUs, providing evidence that the proposed method can adapt to different kernels and hardware configurations.

Critical weaknesses

W1. The bottleneck analysis focuses mainly on resource-capacity limits. It is unclear how KernelCompass identifies and addresses performance limitations that do not correspond to these limits, such as dependent memory accesses and shared-memory bank conflicts. Targeted experiments on kernels dominated by these issues are needed to show whether the proposed resource feedback provides useful guidance.

W2. The paper aims to optimize for a specific metric and workload, but this raises concerns about overfitting to the selected configurations and the generalizability of the resulting kernels. Although the evaluation includes complex kernels, it is concentrated on selected workload configurations. Evaluation with different inputs and shapes, or irregular kernels such as sparse attention and data-dependent MoE workloads, is needed to better support the claim.

Questions for authors
  1. In Section 3.4, it is unclear how the rewrite agent chooses which optimization techniques to apply to the kernel. Could you explain this process and how the parameters are selected after rewriting? Also, how do we know whether the 40 tuning trials are sufficient to decide that the lack of improvement is due to the kernel structure?

  2. Section 3.5 is a bit confusing. Could you explain how the budget is assigned to each operator group, and what indicates its potential for further optimization?

3 comments

(Part 1/2) Thank you for the review, and for noting the consistent improvements across the six tasks and three GPUs. All new experiments use the paper's base model (GLM-5.3-max), evaluation setup and correctness gates.

W1. Bottlenecks that are not capacity limits. The new searches in W2b produced the three rewrites below, none of them a capacity fix, and the framework kept all three. On block-sparse attention none of the 20 candidates was diagnosed capacity-limited at all (15 launch overhead, 5 compute saturated).

taskparent diagnosisrewritelatency
block-sparse attentionlaunch overhead dominates, 0 spills; with other parameters fixed, pipeline stages 1/2/5 give 0.268/0.276/0.270 msanalysis agent: a data-dependent block-selection check in every loop iteration prevents pipelining; the check is removed and block selection is separated from the loop over blocks0.268 → 0.175 ms (−34.6%)
block-sparse attentionlaunch overhead dominates: ~20 host operations around the kernelfused into 2 launches0.232 → 0.140 ms (−39.6%)
MoElaunch overhead dominates: device synchronizations caused by .item() on the routing countslaunch grids sized by an upper bound remove the synchronizations (CPU/GPU time 1.02 → 0.08), plus fp16 weight copies3.874 → 3.108 ms (−19.8%)

The first rewrite also changes which tuning directions are useful: more pipeline stages do not improve the parent, whereas stages 1→2 reduce the rewritten kernel's latency from 0.183 to 0.175 ms.

How the diagnosis works. Each candidate is checked in a fixed order: launch overhead, CPU-side issue rate, bandwidth or compute saturation, capacity limits, and finally latency. Eq. 2 covers the capacity step (registers, shared memory, threads, spills, resident blocks per SM); at the other steps the agents work from the source code and the measured timing. Results over all candidates of the 18 main runs:

capacity limitedbandwidth saturatedcompute saturatedmixedundetermined*CPU-side or launch overheadtotal
20841384440335

*10 without a valid timing; 34 whose working set fits in L2, so the bandwidth test does not apply.

Latency is the objective. Take BLOCK_M 128→64 on the parent of our new experiment (KernelBench L2:32). It improves every resource we track: registers 230→165, shared memory 32→24 KB, resident blocks 2→3, no spills. Latency still rises 17.4%, from 5.684 to 6.675 ms. The UNEXPLAINED label marks this case, and no agent is told to reduce a resource along such a direction. Each diagnosis also lists the signals unavailable in our profiling setup (bank conflicts, warp divergence, cache hit rates, stall reasons), so a signal we cannot read is never reported as clean; a bank-conflict-specific study is future work.

W2. Other shapes and irregular kernels. (a) New shapes. We ran each frozen Table 1 candidate on three new shapes per task, unchanged and without retuning, with new inputs. The new shapes include non-aligned ones: 125×125, 31×30×29, sequence length 500. On RTX 4090, over the 6 original and 18 new shapes, every candidate passes the gate and ours is fastest in all 24. Baseline latency / ours, against the fastest baseline of the same base model at each shape:

taskdefaultshape 0shape 1shape 2
ConvT2d-Mish1.09×1.09×1.15×1.03×
Matmul-Swish1.91×1.92×2.72×1.92×
ConvT3d-GN2.31×2.30×2.33×2.19×
MBConv1.22×1.29×1.19×1.06×
CausalAttn2.99×2.89×2.91×2.96×
Mamba2-SSD1.08×1.03×1.10×1.08×

A second, independent 12 h search on ConvT2d-Mish gives 1.30 / 1.31 / 1.15 / 1.27×.

(b) Irregular, data-dependent kernels. Three new tasks in KernelBench format, one 12 h search each with the Table 1 configuration. Timing covers the whole forward pass, so routing and block selection are included. All final candidates pass the gate (medians):

taskeagertorch.compile (TF32)ours
MoE: 4096 tokens, hidden 2048, 64 experts, top-8, unbalanced input-dependent routing21.0 ms18.2 ms3.120 ms (6.7× / 5.8×)
block-sparse attention: 2×8 heads, seq 4096, block 64, input-dependent top-8 blocks14.0 ms4.90 ms0.140 ms (100× / 34.9×)
Kimi Delta Attention: 2×16 heads, seq 2048, head dim 128, gated delta rule305 mscompile did not finish in 30 min0.415 ms (735×)

(Questions in Part 2.)

(Part 2/2)

Q1a. How the rewrite agent chooses a technique. The agent compares what each way of relieving a resource limit costs in latency, then retunes the new structure. On the L2:32 parent (Conv2d+scale+channel-min), where BLOCK_M+ is COST(registers), lowering the accumulator precision turns BLOCK_M 128→256 from a 105.5% slowdown into a 14.9% speedup:

MMA accumulatorBLOCK_M 128BLOCK_M 256effect of the larger tile
fp32 (parent)5.683 ms11.680 ms, 72 spills+105.5%
fp16 (rewrite)5.080 ms4.325 ms, 6 spills−14.9%

Splitting the accumulator, the textbook alternative, leaves only 2 spills at BLOCK_M 256 but takes 6.721 ms (18.3% slower than the tuned parent at θ*): the smallest resource count is not the fastest structure.

What the agent is given. After tuning, the framework measures configurations around the best one θ*, one axis moved and the rest fixed, and labels each direction: IMPROVE, TIE, WALL(d), COST(d), UNEXPLAINED. WALL and COST directions are ranked by the measured relative latency gain toward the boundary. For each, the analysis agent receives the slopes, the binding dimension and the applicable transformations with a cost estimate, and writes a hypothesis: change, expected effect, risk. The rewrite agent implements it and states the expected change for every resource.

What the feedback changes. A new experiment holds the agent, parent, prompt template, list of transformations and budget fixed. A2 = raw paired readings; A3 = A2 + derived slopes, labels, summary and cost estimates (no new measurement); A4 = A3 + transformation timings, measured on the parent.

A2A3A4
full runs: R = child/parent, 3 replicates, each regenerated and retuned for 40 trials0.897 / 0.744 / 0.8910.742 / 0.752 / 0.8120.751 / 0.760 / 0.768
mean ± s.d.0.844 ± 0.0860.768 ± 0.0380.759 ± 0.009
separate decision runs (5 analysis-and-rewrite runs per group, no retune): implemented the fastest transformation we tested0/51/55/5

Derived guidance lowers mean R from 0.844 to 0.768. With the transformation timings, all five decision runs pick the fastest transformation we tested, and all three A4 children retune to BLOCK_M 256, the direction that was costly on the parent.

Q1b. Parameters after rewriting. Each new structure gets its own 40-trial TPE search (10 random start-up trials); failed and infeasible trials count against the same budget. A child replaces the parent only if it is faster after this retune, so the comparison is always between tuned structures.

Q1c. Are 40 trials enough? The recorded tuning curves of four RTX 4090 Table 1 runs (ConvT3d-GN, MBConv, CausalAttn, Mamba2-SSD; 90 parameter spaces) support our 40-trial budget for comparing parent and rewritten structures:

tuning-curve statisticvalue
spaces where trials 31–40 added no improvement61 of 90
spaces where trials 31–40 added at most 2%80 of 90
share of the 40-trial gain already reached at trial 30 (median, the 58 spaces that improved)99.7%

We extended two late-improving spaces of the CausalAttn run, whose last improvements occurred at trials 37 and 38 of the original 40. Change in the best latency relative to trial 40:

spacetrial 60trial 80 (2× budget)trial 100 (2.5× budget)
10%0%0%
20%−1.8%−2.0%

Extending the tuning budget by 2.5× improves the best latency by at most 2.0% in these late-improving spaces.

Q2. §3.5, budget per operator group. Priority follows the measured gain in the declared objective, not CUDA time. Under TTFT, RMSNorm goes first although the projection group has 6.3× its CUDA time: replacing RMSNorm improves TTFT by 3.56%, while replacing the projections stays within noise. The procedure:

  1. Modules with identical source and weight shapes form one group; groups that cannot run standalone (e.g. only with a KV cache) are excluded.
  2. For each group we install a numerically equivalent replacement and measure its change ΔJ in the objective under the contract.
  3. The group with the largest ΔJ above noise gets the search budget first; groups within noise come after.
  4. An installed candidate is accepted on J and the contract's quality conditions.

Example (Qwen3-4B, TTFT objective, Appendix C.3):

groupCUDA timeΔJ of the equivalent replacement
RMSNorm14041 µs+3.56%
gate_proj / up_proj88822 µs (6.3×)within noise
SiLU5095 µsno faster replacement

Both the budget and the final acceptance therefore follow the model-level objective. The revised §3.5 states these four steps.

Thank you again for the careful questions. The added experiments show transfer across shapes and effective rewrites beyond capacity limits, and we are glad to provide more detail on any point.

Dear Reviewer,

Thank you again for the time you put into reviewing our paper.

The discussion period is about to close. We have replied to each point in your review above and uploaded the revised paper. We would be grateful if you could take a look. If anything is still unclear, please tell us while we can still reply, and we will be glad to explain further.

With thanks,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.