- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes StruTune, a two-stage framework for LLM-based GPU kernel generation and optimization. The key idea is to decouple structural search from local implementation optimization. In the first stage, the system generates and evaluates multiple structural plans without executing code. In the second stage, it implements the selected plans and independently optimizes each structure under a fixed structural contract. Experiments on selected operators demonstrate promising performance improvements over several existing kernel optimization agents on A800 and H100 GPUs.
Critical strengths
-
The paper identifies an interesting practical problem: existing LLM-based kernel optimization agents may overcommit to an initial implementation structure and spend excessive effort on local refinements.
-
The separation between structural exploration and local optimization is intuitive and potentially useful. The reported results across benchmarks and GPU architectures are promising.
Critical weaknesses
- Limited discussion of novelty relative to existing planning-based kernel optimization agents.
The central idea of separating planning from implementation and iteratively refining optimization strategies has been explored in prior work. In particular, AccelOPT (MLSys 2026, also missing citation) uses a Planner–Executor–Summarizer architecture, where the Planner proposes multiple optimization plans, the Executor realizes them, and profiling results are summarized into reusable optimization experiences that guide subsequent planning iterations. AccelOPT additionally uses beam search to maintain multiple candidate implementations and demonstrates the discovery of both local and nontrivial global optimizations. Another paper is https://arxiv.org/pdf/2606.26758 EGG, you may also want to clarify the distinction clearly. In general, "planning" then "implementation" in the area of coding has been widely explored before. I am a bit hesitant about the novelty here.
While StruTune explicitly distinguishes structural decisions from local implementation decisions and optimizes selected structures independently, it is unclear to me whether this represents a fundamental algorithmic advantage over existing iterative planning and search frameworks, rather than a different organization and presentation of the search process.
I would appreciate a more detailed discussion of the conceptual and algorithmic differences from AccelOPT and related approaches such as K-Search and QiMeng-Kernel. In particular, what structural transformations can StruTune systematically explore that existing planning-based agents cannot? Is the main contribution the explicit structural representation, the execution-free structural evaluation, or the independent allocation of optimization budgets? Can the authors provide empirical evidence isolating the advantage of these design choices?
- The current experiments do not sufficiently isolate the algorithmic advantage of structural search.
The ablation study seems too simple. It does not control for the same API budget.
Questions for authors
Please see the Critical Weaknesses section.
Also, a potentially distinguishing component of StruTune is its ability to evaluate and rank structural plans before generating or executing kernels. Does the ranking really make sense without actually executing the kernels? Some plans may seem bad at first glance, but a good kernel may still come out of such a plan.
Do you think about this structural lock-in problem as a model capability issue? If so, any way to mitigate it it via training?
4 comments
Rebuttal - Part 2
W2: Ablation with the same API budget
The current experiments do not sufficiently isolate the algorithmic advantage of structural search. The ablation study seems too simple. It does not control for the same API budget.
We appreciate the reviewer’s suggestion to better control the optimization budget and isolate the contribution of each component. Structural search and our local optimizer each improve performance, independently of dispatch. We redesigned the ablation on 10 FlashInfer-Bench operators on H100 to isolate these contributions.
Budget control. Every configuration uses two parallel trajectories with 10 local optimization rounds each. Ours Structural Plan initializes them from the top two of five scored structures; PyTorch Reference initializes them from reference code. Stage II uses either our action library (Ours Local Optimization) or direct LLM refinement (Direct LLM), keeping the number of trajectories and local rounds fixed.
Component contributions.
- Structural search improves initialization. With our local optimizer and dispatch disabled, structural initialization increases GM speedup from 1.21× to 1.92×, maintaining 100% correctness.
- Our local optimizer improves refinement. Starting from the same structural plans, our action library achieves 1.92× versus 1.42× with direct LLM refinement, both with 100% correctness.
- Dispatch adds no benefit in this run. The candidates showed no consistent complementary performance across workloads, so the dispatch-enabled configuration retained the single-structure result of 1.92×.
R_Table 2: Ablation with the same API budget on FlashInfer-Bench on H100. GM speedup is relative to the FlashInfer expert library
| # | Stage I: Ours structural plan | Stage I: PyTorch Reference | Stage II: Ours Local Optimization | Stage II: Direct LLM | Dispatch | Correct (%) | GM speedup |
|---|---|---|---|---|---|---|---|
| 1 | — | ✓ | — | ✓ | — | 80.00 | 1.61× |
| 2 | — | ✓ | ✓ | — | — | 100.00 | 1.21× |
| 3 | ✓ | — | — | ✓ | — | 100.00 | 1.42× |
| 4 | ✓ | — | ✓ | — | — | 100.00 | 1.92× |
| 5 | ✓ | — | ✓ | — | ✓ | 100.00 | 1.92× |
We will include the revised ablation and its budget-accounting details in Section 5.3 and Appendix E of our paper.
Q1: Effectiveness of Pre-execution Structural Scoring
Also, a potentially distinguishing component of StruTune is its ability to evaluate and rank structural plans before generating or executing kernels. Does the ranking really make sense without actually executing the kernels? Some plans may seem bad at first glance, but a good kernel may still come out of such a plan.
This is really an excellent question. Pre-execution scoring effectively identifies structures with high performance potential, even when their initial implementations are slow. To evaluate this, we implemented five structures for each of the 10 FlashInfer-Bench operators on H100 and optimized each for 10 rounds without dispatch.
Pre-execution structural scoring effectively identifies promising structures. We implemented all five candidates for each of the 10 FlashInfer-Bench operators on H100 and optimized each for 10 rounds without dispatch. The top two by score include the best-performing final structure for every operator (R_Figure 1), demonstrating the effectiveness of ranking structures before execution.
R_Figure 1: Structural scores versus final performance on 10 FlashInfer-Bench operators R_Figure 1 (https://anonymous.4open.science/r/ICLR_2027_Rebuttal-C98D/R_Figure_1.png)
Structural scoring also identifies potential that initial performance comparisons can miss. For DSA, the highest-scored structure, S₁, initially achieves approximately 5× speedup, below S₂’s 11×, but reaches approximately 38× after optimization, compared with S₂’s 20× (R_Figure 2). Thus, selecting by initial performance would favor S₂, while structural scoring correctly prioritizes S₁’s greater optimization potential.
R_Figure 2: Structural scores versus initial and optimized speedups for five DSA structures on H100. R_Figure 2 (https://anonymous.4open.science/r/ICLR_2027_Rebuttal-C98D/R_Figure_3.png)
We will include these analyses in Section 5.3 and Appendix E of our paper.
Rebuttal -Part 3
Q2: Model Capabilities and Trajectory Training for Mitigating Structural Lock-in
Do you think about this structural lock-in problem as a model capability issue? If so, any way to mitigate it it via training?
This is a practically important question. Structural lock-in may partly reflect a model capability limitation, and targeted trajectory training could help mitigate it.
(1) Recognizing a slow kernel does not necessarily mean identifying a structural bottleneck. A model may detect poor performance without determining whether it stems from structural design or implementation choices. One possible reason is limited training data that explicitly connects performance bottlenecks to structural causes and demonstrates how to address them.
(2) Training on bottleneck diagnosis and structural changes could improve this capability. Such trajectories should show how to identify a performance bottleneck, link it to structural design, and modify the structure, followed by correctness and performance evaluation. This supervision could help models learn both when structural changes are necessary and how to make them effectively.
We will discuss this model capability issue and training direction in Section 6 of our paper.
Thank you again for your valuable suggestions. As the rebuttal deadline approaches, we have revised the organization of our responses to clarify our main contribution and its relationship to prior work. We added a concise comparison with AccelOpt, K-Search, QiMeng-Kernel, and EGG, and made the supporting component ablations and pre-execution scoring results more prominent. We also clarified how trajectory training could help mitigate structural lock-in.
We would appreciate your feedback on whether the revised explanations and evidence address your concerns about novelty and experimental support. If further clarification would be helpful, we would be happy to provide it during the remaining discussion period.
Thank you again for your time and consideration.
Rebuttal - Part 1
We thank the reviewer for recognizing the practical importance of structural lock-in and the potential usefulness of separating structural exploration from local optimization. We also appreciate the constructive suggestions on novelty, budget-controlled comparisons, pre-execution ranking, and training-based mitigation. We hope the following clarifications and additional results address these concerns.
W1: Limited discussion of novelty relative to existing planning-based kernel optimization agents
We appreciate the reviewer’s suggestions on contribution clarity and related work. Our novelty lies in identifying structural lock-in and addressing it by separating structural search from local optimization. Existing methods tend to continue local refinement within an already implemented structure, with limited exploration of structural alternatives. So our distinction from “planning before implementation” is the separation of structural and local decision spaces, rather than simply assigning planning and coding to different steps.
(1) Clarifying AccelOpt's local and global optimizations. AccelOpt's local–global distinction differs from ours. Its local examples apply algebraic simplification and instruction substitution, while its global example introduces recomputation and reorganizes loops to reduce spilling and intermediate storage usage. These illustrated transformations fall within our local implementation optimization categories L2 and L5, and thus do not change the structure as we define it.
(2) Differences from related work. AccelOpt, K-Search, and QiMeng-Kernel interleave structural and local implementation decisions. EGG separates algorithmic design from hardware tuning, but defines and selects candidates differently from StruTune. We summarize this in R_Table 1.
R_Table 1: Differences in planning and search organization between StruTune and related methods.
We will add the missing AccelOpt and EGG citation, clarify these distinctions in Section 2 of our paper, and revise the contribution statement in the introduction.
(3) What structural transformations can StruTune systematically explore that existing planning-based agents cannot?
StruTune does not introduce structural transformations that existing agents cannot generate; it makes their exploration an explicit search stage. For DSA, these alternatives include fused online attention, partitioned attention with stable merging, and materialized-logit attention. Existing agents may discover them while modifying an implementation, whereas StruTune compares structural alternatives before code generation and independently optimizes the selected structures, reducing dependence on the initial implementation.
(4) Main contribution and empirical evidence. Our main contribution is the two-stage framework that separates structural search from local optimization. The explicit structural representation, execution-free evaluation, and independent optimization budgets work together to implement this separation, rather than constituting three standalone contributions. Our response to W2 provides detailed ablation results and analyzes the roles of structural search, local optimization, and workload dispatch under the same local optimization budget.
We will add the missing AccelOpt and EGG citation, expand these comparisons in Section 2 of our paper.