acceptodds
Under review as a conference paper at ICLR 2027

Compiler-Grounded Hierarchical Diagnosis for LLM-Based Triton Kernel Optimization

Abstract

Recent advances in large language models (LLMs) have enabled automated kernel generation and optimization, but most existing approaches rely on surface signals such as compilation feedback and profiling metrics, which reveal that a kernel is slow but not why the backend compiler fails to realize a profitable optimization, especially on emerging accelerators such as NPUs. We therefore formulate kernel optimization as a progressive cross-layer diagnosis problem that links runtime symptoms to IR structure and compiler behavior before rewriting source, and present a compiler-grounded, hierarchical optimization framework for Triton kernels that escalates from lightweight pattern triage and profiling diagnosis to IR attribution and compiler-grounded analysis only when deeper evidence is needed. Implemented on Triton for Ascend NPUs and evaluated on 37 successfully converted entries from a standardized NPUKernelBench-derived Ascend 950 benchmark, the system attains a geometric-mean speedup of and a median of from the initial to optimized Triton kernel; 22/37 exceed and 13/37 exceed . The complete distribution ranges from near-baseline entries to large wins, motivating transparent reporting of the current system's scope and limitations.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM agents, kernel optimization, Triton, NPU accelerators

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_pfnkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes hierarchical diagnosis for LLM-based Triton kernel optimization. It progresses from profiling to IR and compiler analysis to guide source rewrites, with evaluation on 37 Ascend NPU kernels.

Critical strengths

The real-hardware speedups are useful, and the case studies connect compiler behavior to concrete source changes.

Critical weaknesses
  1. There is no comparison with profiling-only or always-deep analysis. The benefit of the hierarchy itself is therefore unclear.

  2. Token use and total optimization time are not reported. It is unclear whether using deep analysis only when needed actually saves resources.

  3. The agent decides when to use deeper analysis, but the paper does not test how well these decisions work.

  4. The main speedup is measured on the same five input cases used during optimization. It is unclear whether these gains hold on new inputs.

Questions for authors
  1. With the same model, toolchain, and budget, does the hierarchy perform better than profiling-only and always-deep analysis?

  2. How many tokens and how much time does each method need to reach the same speedup?

  3. Does the decision rule work better than simply using deeper analysis after a few failed attempts?

  4. Do the gains hold on new input shapes and sizes? How often is the optimized kernel slower than the initial kernel on these inputs?

Reviewer_hZA5OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a hierarchical LLM-based framework for optimizing Triton kernels on Ascend NPUs. The approach progressively escalates from pattern-based optimization to profiling, IR attribution, and compiler-source analysis, while accumulating reusable optimization knowledge. Evaluated on 37 successfully converted operators, the framework achieves a 4.35× geometric-mean speedup over initial Triton implementations and a 2.60× speedup over Torch NPU on a separate set of 35 operators.

Critical strengths
  1. I like the idea of multi-stage bottleneck identification, progressively using profiling, IR, and compiler-level information to guide kernel optimization
Critical weaknesses

1.The paper does not compare against existing LLM-based optimization agents such as triton forge, alphaevolve or provide ablations isolating the contributions of different diagnostic stages. It is therefore unclear how much the proposed hierarchy contributes to the reported speedups.

  1. Only 3 and 4 trajectories reach L3 and L4, respectively, and none of the winning rounds is attributed to these stages. Most winning rounds are attributed to pattern triage. Furthermore, the paper does not distinguish the contribution of prebuilt optimization knowledge from newly learned patterns or demonstrate whether the latter improve subsequent optimizations.

  2. The paper would benefit from careful proofreading and clearer presentation. Figure 3 is distorted and difficult to read. There are also several grammatical and formatting issues, such as sentences beginning with lowercase "the system" instead of "The system." The approach section is dense and would benefit from a clear end-to-end example illustrating the agent's decisions and transitions between stages. There are also inconsistencies: Section 7 states that the agent must proceed through profiling, IR attribution, and compiler-grounded explanation before rewriting, whereas Section 4 allows rewrites at earlier stages.

  3. Since the framework heavily relies on agent-driven decisions, more transparency regarding prompts, skill definitions, tools, and escalation criteria is necessary for reproducibility. The paper does not report token consumption, monetary cost, or optimization time, despite efficiency being a key motivation for hierarchical diagnosis.

Questions for authors
  1. Given that most winning rounds are attributed to pattern triage, what evidence supports the effectiveness of L3/L4? How would the results compare against a pattern-only or profiling-guided agent?
  2. How much of the pattern knowledge is prebuilt versus learned during optimization? How frequently are newly learned patterns reused, and what measurable benefit do they provide?
  3. How are escalation decisions enforced? Could the authors provide the agent prompts, skill definitions, and a complete end-to-end optimization trace? Also, why do more trajectories reach L4 than L3 if escalation is sequential?
  4. What are the token, monetary, and wall-clock costs of optimization?
  5. Which aspects of the proposed framework are specific to Ascend NPUs and which would generalize to other backends, such as GPUs.
Reviewer_SyCsOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a Compiler-Grounded Hierarchical Diagnosis (CGHD) framework for LLM-based Triton kernel optimization on Ascend NPUs. The framework uses a four-layer diagnostic pipeline: pattern classification, profiling diagnosis, IR attribution, and compiler-source escalation, to progressively identify and fix performance bottlenecks. It is evaluated on 37 operators from NPUKernelBench on Ascend 950, reporting a 4.35× geometric mean kernel speedup and a 2.60× speedup over Torch NPU.

Critical strengths
  1. Progressive hierarchical design. The four-level escalation strategy: pattern classification, profiling diagnosis, IR attribution, and compiler-source escalation, provides a structured workflow for progressively deeper kernel optimization. By restricting deep compiler analysis to cases where lightweight evidence is insufficient, the framework avoids unconstrained search and offers a principled diagnostic pipeline. This design may be informative for future compiler-aware code optimization efforts.
  2. Compiler-grounded analysis. Integrating IR-level evidence, such as pattern matching, memory/layout analysis, fusion/scheduling checks, and compiler pass behavior into the optimization loop is a promising direction. Distinguishing source-level symptoms from compiler-mediated causal effects is well motivated and addresses a real limitation of existing LLM-driven kernel systems that rely heavily on execution results as reward signals.
Critical weaknesses
  1. The main experimental figure (Figure 3) has serious formatting issues that undermine result verification. Figure 3 is the primary visual evidence supporting the paper’s headline speedup claims. However, its current formatting problems make it difficult to independently verify per-operator results. A main results figure should be correctly formatted, clearly readable, and independently verifiable, especially when it underpins the paper’s central quantitative claims.

  2. The evaluation scope is narrow. The system is evaluated only on 37 operators from a single NPU platform (Ascend 950). The paper does not compare with Kernel Design Agent (KDA) or other closely related methods, nor does it evaluate on a more diverse set of operators or shapes. The paper also does not discuss whether the method generalizes beyond Ascend NPUs, which limits the broader impact and external validity of the proposed framework.

Questions for authors
  1. Could the authors provide a corrected version of Figure 3 with properly readable labels, axes, and legends, so that per-operator results can be independently verified?

  2. The evaluation is limited to Ascend NPUs. How well does the method generalize to GPU/CUDA environments? Would migrating the hierarchical diagnosis framework to other accelerators require substantial modifications?

  3. Could the experimental section include comparisons with additional methods, such as Kernel Design Agent (KDA), to better position the proposed approach relative to closely related work?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.