acceptodds
Withdrawn from ICLR 2027

KernelWeaver: A Shape-Aware Kernel Agent for GPU Kernel Optimization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language model (LLM)-based kernel agents automate GPU kernel development, but the kernels they generate are often optimized for only a few representative input shapes and struggle to maintain performance across the broader range encountered at deployment. This mismatch arises because shape changes alter the dominant hardware bottleneck, requiring different execution configurations. Although generating a dedicated kernel for every shape can address this mismatch, it increases compilation and code-size costs. We present KernelWeaver, a shape-aware kernel agent that optimizes cross-shape performance by composing a small set of complementary kernels under deployment budgets. To construct this portfolio, KernelWeaver first partitions the shape space using hardware resource constraints to guide candidate generation, then selects kernels according to their marginal performance gain per unit cost. The resulting portfolio is deployed through a shape-aware router, whose execution feedback identifies gaps in offline coverage and guides targeted refinement. On nine common GPU kernel workloads, KernelWeaver achieves geometric-mean speedups over PyTorch eager of on the NVIDIA RTX 4090 and on the NVIDIA A800, compared with and for the strongest evaluated baseline on the respective platforms. In our deployment cost evaluation, KernelWeaver reduces deployment compilation time by 91.71% compared with torch.compile.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Kernel agents, GPU kernel generation, shape-aware optimization, budgeted kernel portfolios, online feedback.

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_LYWYOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

KernelWeaver addresses a practical limitation of current kernel agents: optimizing one kernel for a few representative inputs often leads to poor performance on other shapes seen in deployment. The paper proposes generating candidates for hardware-derived shape regions, selecting a small portfolio based on marginal performance gain under deployment costs, and dispatching requests through a shape-aware router. Online feedback is then used to identify poorly served shapes and trigger further refinement. The evaluation covers nine operators on an RTX 4090 and an A800. The paper reports geometric-mean speedups of 1.59x and 1.44x over PyTorch eager, respectively, while deploying 28 kernels selected from a pool of 96 candidates.

Critical strengths

1.The paper identifies a real and underexplored issue in kernel-agent evaluation. Reporting performance on a few representative shapes can hide severe regressions elsewhere, and the motivation experiments make this point clearly.

2.Reframing the problem as portfolio construction is sensible. A kernel that is not the best on average can still be useful if it covers shapes where the current portfolio performs poorly.

3.The method considers deployment concerns beyond kernel latency, including compilation cost, code size, and routing overhead. This is more realistic than evaluating each shape independently.

4.The experimental section is reasonably transparent about some trade-offs. For example, the paper reports that portfolio selection slightly hurts Softmax performance, and that several held-out cases remain below eager performance.

Critical weaknesses

1.The core shape-partitioning procedure is not specified precisely enough to reproduce. The paper lists hardware checks involving shared memory, Tensor Core granularity, occupancy, and cache behavior, but does not explain how these checks become concrete shape boundaries. It is also unclear how tile candidates are chosen, how overlapping regions are resolved, or how “extreme-shape families” are defined.

2.The cost-aware selection objective is under-justified. Compilation time, code size, and routing-table complexity are normalized and summed without weights, but their relative importance depends heavily on the deployment setting. The paper does not provide the actual budget values, the normalization details, or sensitivity experiments for alternative cost weights.

3.The claimed compilation-cost advantage is incomplete. The reported 13.53 seconds only covers compilation of the final deployed portfolio. It excludes the cost of generating, compiling, validating, and profiling the full candidate pool, as well as the cost of LLM inference. Since KernelWeaver profiles every candidate across the target shape set, the total optimization cost may be substantially larger than the reported deployment compilation time.

4.The evidence for online refinement is weak. The ablation is conducted with fixed-shape offline timings, so it does not demonstrate that the feedback loop works under a real deployment trace. The paper does not explain how long feedback must be collected, how low-gain cases are detected online, how much refinement costs, or how regressions are prevented before updated kernels are deployed.

5.The baseline comparison is not broad enough. The paper compares against PyTorch eager, torch.compile, Starter Kernel, and AutoKernel, but omits direct comparisons with dynamic-shape optimization systems such as DietCode and Roller, or with a strong Triton autotuning baseline under the same search budget. PyTorch eager is a relatively weak reference for several of these operators.

6.The evaluation lacks variance and tail-performance analysis. Kernel generation is stochastic, yet results are reported from what appears to be a single generation run. There are no confidence intervals, repeated runs, or p95/p99 latency results. The held-out evaluation also lacks a clear sampling protocol and exact shape lists.

Questions for authors

1.Can the authors provide a complete algorithm or pseudocode for shape partitioning? How are tile configurations generated, how are hardware checks converted into boundaries, and how are overlapping regions handled?

2.What are the exact deployment budgets and normalization formulas used in portfolio selection? How do results change when the relative weights of compilation time, code size, and routing complexity vary?

3.Can the authors report the full end-to-end optimization cost, including LLM generation, candidate compilation, correctness checking, profiling, routing-table construction, and final deployment? At what request volume does this cost amortize against torch.compile?

4.Why are DietCode, Roller, or a budget-matched Triton autotuning baseline not included? Is the torch.compile compilation cost measured over exactly the same shape coverage as KernelWeaver?

5.What is the concrete protocol for online refinement? What workload trace is used, how is a low-gain shape identified, how many requests are needed before refinement, and how are correctness and performance regressions controlled before deployment?

6.How are held-out shapes sampled and kept separate from prompt design and method development? Please provide the exact shapes, dtypes, layouts, fallback rate, worst-case slowdown, and tail latency results.

7.Can the authors report results over multiple independent kernel-generation runs and repeated timing measurements? This is especially important because shape-aware candidate generation accounts for most of the reported gain.

Reviewer_eawzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

LLM kernel agents usually tune one kernel on a few input shapes, and that kernel can be really slow on shapes it never saw. Writing one kernel per shape fixes this but is wasteful. KernelWeaver instead keeps a small set of complementary kernels per operator and routes each shape to the best one. Simple hardware rules (does the tile fit in shared memory, is the input too narrow for tensor cores, is there enough parallelism) tell the agent which shape regions need their own kernel, a greedy step picks which candidates to keep under a budget, and a feedback loop uses runtime logs to fix shapes that are still slow. On nine operators and two GPUs they report good speedups over PyTorch eager and single-kernel baselines with only a few kernels per operator, plus a test on held-out shapes.

Critical strengths

Strength #1.

The paper points out a real blind spot in how kernel agents are evaluated: a kernel scored on a few shapes can be way slower than PyTorch on other realistic shapes. That said, IIUC this is pretty much common sense in hardware optimization.

Strength #2.

The design is sensible and practical. Generate diverse kernels, measure them all, keep a small set that covers the shape space and dispatch by shape. That's basically how production GPU libraries already work.

Critical weaknesses

Weakness #1.

Every piece is already known, and there seems to be no new insight.

Picking among several kernels by input shape is what cuBLAS, CUTLASS, Triton's autotuner and torch.compile already do, and DietCode and Roller study exactly this problem (it's really the classic algorithm selection problem). The hardware rules are textbook checks used as prompt hints, and for most of the nine operators the paper doesn't even say how the shape regions are formed. The greedy selection is a standard budgeted covering problem with known methods and guarantees, but the paper doesn't connect to any of that. The feedback loop is the usual generate, measure, revise agent loop, just applied per shape region. On top of that, the paper doesn't explain why it works. We only get aggregate speedups, nothing about which bottleneck shifts drive the gains, which kernels win where, or what the hardware hints actually change in the generated code.

Weakness #2. Only one real agent baseline.

Since the method is a combination of known pieces, its value should come from clearly beating existing agents, and the evaluation doesn't show that. KernelWeaver sits on top of a kernel agent, so the obvious experiment is "agent X alone vs agent X inside KernelWeaver" for a few different agents. Instead AutoKernel is the only agent compared (the other baselines are a hand-written starter kernel and torch.compile), and the hardware-aware agents the paper itself cites (KernelBand, KernelFoundry, CudaForge, Astra, CUDA Agent) are left out, even though hardware awareness is the main selling point. The simplest strong baseline is missing too: ask an existing agent for several diverse kernels without any hardware rules and route between them by shape. The ablation can't answer this because it changes two things at once (one kernel vs many kernels with hardware hints). I also have some reliability concerns: the evaluation shapes seem to overlap with the ones used for selection, offline and online numbers disagree on the same GPU, and there are no repeated runs (see questions).

Questions for authors
  1. Could you compare against more agents, ideally KernelWeaver on top of two or three different ones including the hardware-aware agents you cite? If some can't be run, why not?
  2. How does an existing agent do if you just ask it for several diverse kernels and route by shape, without the hardware rules?
  3. Why are the offline results so much better than the online ones on the same GPU for some operators, when routing overhead is small?
  4. What do the hardware hints actually change in the generated kernels? Which kernels win in which shape regions, and which bottleneck does each one address?
  5. Are the shapes used for selection and refinement the same as the ones in the main results, and does the candidate pool include the exact baseline kernels? If so, beating every baseline there is almost guaranteed by construction.
  6. How do you guarantee correctness when a shape is routed to a kernel that was never tested on it, or when the online loop produces a new kernel?
  7. Maybe try newer GPUs like Hopper, where shape effects tend to be stronger.
Reviewer_EMyYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

KernelWeaver generates shape-aware kernel candidates, selects a compact portfolio under deployment budgets, and uses routing feedback to refine poorly served shapes. On nine operators, it reports geometric-mean speedups of 1.59× and 1.44× over PyTorch eager on RTX 4090 and A800, respectively. A frozen portfolio is also evaluated on 63 held-out shapes.

Critical strengths
  1. An explicit cross-shape optimization objective. The method evaluates complementary kernels as a portfolio and accounts for compilation, code size, and routing, rather than assessing each candidate solely by its standalone speed.

  2. Evaluation beyond the profiled shapes. The paper includes two GPUs, routing-overhead measurements, compilation-protocol comparisons, and a frozen-portfolio test on 63 unseen shapes, where it reaches 1.58× over eager.

Critical weaknesses
  1. The methodological advance over existing shape-aware optimization is not established. DietCode (MLSys 2022) already addresses dynamic shapes. Here, hand-written starters and a multi-candidate portfolio are compared with AutoKernel’s single output after 25 iterations. Without a matched-budget portfolio baseline, the experiments cannot distinguish the proposed partitioning and selection from the benefit of more candidates and shape-based dispatch.

  2. The cost-aware formulation is not evaluated as a complete optimization procedure. Actual budget caps and default selection settings remain unclear, while 13.53 s measures deployment compilation rather than candidate acquisition and full-shape profiling. The sequential ablation also lacks an equal-budget continued-search control. These results establish a deployment trade-off, but do not establish the efficiency of obtaining the portfolio or isolate the value of feedback.

Questions for authors
  1. With the same generation budget and starter access, how does KernelWeaver compare with generating a candidate pool and selecting a simple profiled portfolio under the same deployment limits?

  2. What exact budget caps, cost-normalization references, admission margin, and stopping rule produced the reported portfolios? Please specify the marginal-gain calculation.

  3. Does feedback-directed refinement still improve end-to-end performance when compared with spending the same additional generation and profiling budget without residual-directed feedback?

  4. What does “copy-taxed torch.compile” mean in Table 3? Are input copying, graph replay, and compilation process/cache state aligned across the compared methods?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.