acceptodds
Under review as a conference paper at ICLR 2027

AI as a Compiler: Compiling triton kernels without the triton compiler

Abstract

Compiler backends are expensive to build and maintain as programming models, workloads, and accelerators evolve. We investigate whether large language models can replace the conventional optimizing and lowering pipeline, a process that we call AI lowering. We study AI lowering from Triton to NVIDIA PTX: an LLM agent translates Triton kernels directly into PTX, bypassing Triton’s intermediate representations and optimization passes while preserving the kernel ABI, and refines its output using randomized differential testing, benchmarking, and profiler feedback. Across twelve common kernels on Ada, Hopper, and Blackwell GPUs and ten kernels from recent ML papers, AI lowering achieves 0.83×-3.34× the performance of autotuned Triton. The largest gains come from transformations that Triton’s lowering pipeline does not perform, such as decoding packed binary weights directly into Tensor Core operands (3.34× on BitDelta), assigning each thread a complete softmax row in tensor memory (1.36× on FlashAttention), and reusing overlapping convolution windows (up to 2.23×). Getting correctness guarantees is now the bottleneck. The state-of-the-art PTX verifier does not model the Hopper and Blackwell features these kernels rely on, and restricting generation to its supported fragment erases the gains on compute-heavy kernels. With best- effort extensions, the verifier checks 16 of 22 kernels; atomics, input-dependent control flow, and bit-level floating-point manipulation remain out of reach, making verification the central open challenge for trusted AI compilation.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
compilation, kernels, verification

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vq8DOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents the Triton AI Compiler (TAIC), an agentic pipeline that lowers Triton kernels directly into NVIDIA PTX. An LLM generates and iteratively optimizes PTX according to feedback from Triton Compiler Environment (TCEnv), including numeric checks and performance results. The authors also extend an off-the-shelf PTX verifier to support additional GPU features, which checks the equivalence for 16 of 22 kernels. Evaluation shows that TAIC achieves 0.83x-3.34x performance of autotuned Triton on modern NVIDIA GPUs.

Critical strengths
  • Good performance on evaluated workloads over autotuned Triton compiler.
  • Useful investigation of verification limits. Integrating Volta and measuring the performance impact of its supported fragment exposes a concrete gap between current verification capabilities and modern GPU optimization.
Critical weaknesses
  • Weak verification. The claimed verification coverage relies on Volta extensions whose soundness is explicitly not established. Numeric check is vulnerable to reward hacking.
  • Unclear source of performance gains. Evidence of gains on newly constructed Triton kernels is limited. The frontier model may have seen the evaluated kernels during training. No ablation study isolates the harness’s contribution.
Questions for authors

The following comments and questions are related to the aforementioned weaknesses.

  • Appendix B explicitly declines to claim soundness or completeness for the Volta extensions and leaves their formalization to future work. Given this, what guarantees does acceptance by the extended verifier actually provide, and what justifies describing the 16 accepted kernels as “verified”? Please provide more details on that.
  • The correctness of the kernel is heavily depended on numeric check, which is vulnerable to reward hacking, such as exploiting loose tolerances, overfitting to the test inputs, or skipping computation the tests don’t exercise.
  • Async operations are critical to achieve state-of-the-art performance on modern NVIDIA GPUs, but they are out of the scope of Volta. Can you extend the verification based on async kernel verification work, such as Weft [1]?
  • Please provide more details on compiler lowering harness. How is the initial PTX generated, and how is an optimization plan translated into subagent tasks? Do subagents differ or just simply sample independently? How many subagents are used?
  • Triton has a known performance gap on Hopper and Blackwell GPUs. Did you try newer Triton-based systems for baselines, such as Tawa and Gluon?
  • Most evaluations still use common and publicly available workloads. The synthetic kernels (randomly generated and modification to familiar examples) seem relatively easy to optimize and provide limited evidence of generalization. Can you evaluate TAIC on widely used Triton kernels created after GPT-6 Astra’s knowledge cutoff?
  • Can you ablate the harness and compare it with other harnesses like codex? Including an open-weight model would also help make the experiments more controlled and compelling.
  • How is the agent sandboxed? Can it access Internet or modify the evaluation harness?
  • Traditional compilers are generally fast, cheap, and deterministic, which is important in production where compilation may be invoked frequently. How can TAIC’s agentic workflow be made faster, cheaper and more deterministic? For example, a more structured agentic workflow may improve reproducibility but restrict exploration.

[1] Rahul Sharma, Michael Bauer, and Alex Aiken. 2015. Verification of producer-consumer synchronization in GPU programs. In Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI '15). Association for Computing Machinery, New York, NY, USA, 88–98.

3 comments

Thank you for your detailed review and your questions. We appreciate your recognition of the performance results and of our analysis of Volta's coverage. Below, we address your questions on verification and generation reliability, reward hacking, the lowering harness, baselines, generalization, ablations, sandboxing, and compilation cost.

Q1 & Q3. Concerns Addressed in the Common Response

Our extension adds to Volta an event-graph analysis that checks memory safety and synchronization for asynchronous Blackwell kernels. “C1. Our Extension of Volta” in the common response separates what Volta already provided from what we added and lists what each check requires.

Across 120 GPT-6 Astra runs, 680 of 844 verification calls (80.6%) passed all executed checks, and no run needed a restart once the pipeline had stabilized. “C2. Generation Reliability” in the common response gives the breakdown by failure type, hardware, and kernel.

Q2. Numerical Checks and Reward Hacking

We agree that numerical testing alone is vulnerable to reward hacking, particularly through overfitting to sampled inputs or exploiting loose tolerances. This is precisely why TAIC combines numerical testing with symbolic verification—to the best of our knowledge, the first such approach in LLM-based GPU kernel compilation. For example, the ReLU test generator samples only nonnegative inputs, so a kernel implementing the identity function could pass all numerical tests despite being incorrect for negative inputs. In contrast, our symbolic verifier checks the kernel against the mathematical specification, max(x, 0), allowing it to detect this discrepancy independently of the sampled inputs.

Importantly, the two verification mechanisms are complementary. Our symbolic verifier reasons under infinite-precision arithmetic and therefore does not capture all floating-point behaviors. Numerical testing remains essential to detect incorrect results arising from reduced-precision arithmetic, which an LLM might otherwise exploit to achieve artificial speedups. Together, these mechanisms prevent reward hacking by verifying not only that the outputs are correct on a given set of test inputs, but also that the implemented function is mathematically equivalent to the target computation. This ensures that an LLM cannot achieve artificial speedups by substituting a simpler function that happens to produce the correct outputs on the test distribution but behaves incorrectly on other inputs.

Sigmoid illustrates this well. Its test generator samples inputs uniformly from [0, 1), where sigmoid is smooth and nearly linear. A kernel that replaces 1 / (1 + exp(-x)) with the fifth-order Taylor polynomial 1/2 + x/4 - x^3/48 + x^5/480 avoids both the exponential and the division. Its maximum error on [0, 1) is about 1.9 × 10⁻⁴, well within the 10⁻³ verification tolerance, so it passes every numerical test. Outside the sampled range, however, it is wrong: at x = 4, it returns 2.30 instead of 0.98. Our symbolic verifier checks the kernel against the specification 1 / (1 + exp(-x)), which the polynomial does not match, and therefore rejects it regardless of the sampled inputs.

This example also shows that the symbolic check is strict: Volta rejects the kernel outright, because the Taylor polynomial is symbolically different from the requested function, however small its numerical error. Deciding which approximations should be accepted is a question we leave to future work.

Q4. Compiler Lowering Harness and Subagent Design

In the current implementation, Triton compiles the reference kernel to establish its PTX signature, launch contract, and performance baseline. This is an engineering simplification that leverages the existing Triton infrastructure for kernel setup and benchmarking, rather than a fundamental requirement of our approach. By default, the LLM generates the initial candidate from the Triton source and this context; optional flags load saved Triton PTX or a previous candidate. Each optimization round uses the current candidate, benchmark results, and available Nsight Compute metrics to propose one to three ordered improvements, each with a name, rationale, and concrete instruction. Each improvement becomes a separate task for the same model, which patches, verifies, and benchmarks candidate variants. The tasks run sequentially with distinct optimization instructions, and accepted speedups become the starting point for subsequent tasks. Each round therefore comprises one to three sequential candidate-generation tasks; there are no parallel subagents.

Q5. Newer Triton-Based Baselines (Tawa and Gluon)

The gap between the performance Triton achieves and what newer GPU architectures can deliver is precisely what motivates our work. Gluon addresses this gap by exposing more explicit control over hardware-level details, at the cost of additional programming effort. Tawa, in contrast, preserves the Triton programming interface but introduces specialized compiler transformations for automatic warp specialization.

Our evaluation explicitly focuses on the standard upstream Triton compilation pipeline, rather than alternative programming interfaces or research compiler extensions. Our objective is to measure how well an LLM-based compiler performs against a modern production compiler, using the same high-level Triton source programs. This allows us to directly assess the performance improvements achievable through AI-based lowering over conventional compiler optimization.

Q6. Generalization to Recent Real-World Kernels

The randomly generated kernels contain long, dense sequences of shifts, bitmasks, and other operations designed to force the LLM to reason about the computation. To address your request for recent practical workloads, we additionally compiled the selective-logprob forward and backward kernels introduced in Liger-Kernel PR #1548, opened on October 20, 2026. The kernel bodies and arguments in our recorded prompts match the corresponding functions in commit 5567d8c. These kernels compute selected-token log-probabilities and their gradients for the PR's PPO/GRPO implementation.

On NVIDIA B200, we evaluated BF16 logits with bias enabled and ignored targets included. The forward kernel produces both log-probabilities and log-normalizers; the backward kernel writes logit gradients in place. The Triton baseline was autotuned over 12 configurations for each kernel, selecting a block size of 4096 and four warps. TAIC retained the source ABI and the selected 128-thread launch configuration.

KernelAutotuned Triton p50 (ms)TAIC p50 (ms)Speedup
Selective-logprob forward0.135140.064652.09×
Selective-logprob backward0.363470.287711.26×

TAIC achieved speedups on both kernels in a single run each: the forward kernel reached its final speedup on the fourth verification call, after ptxas rejected the first candidate, and the backward kernel on the third.

Q7. Harness and Model Ablations

Regarding model ablations, we experimented with earlier-generation and open-weight LLMs, but found them substantially less reliable at generating valid, performant PTX. At submission time, GPT-6 Astra was the only model that consistently produced competitive results.

Our primary objective is to investigate the feasibility of AI-based compilation with a strong reasoning model and a verification-guided optimization loop, rather than to benchmark general-purpose coding agents. We acknowledge that systematic comparisons across agentic harnesses would be valuable future work. We likewise leave post-training of a large language model to future work.

Q8. Agent Sandboxing

The agent operates in a restricted environment with no internet access, limiting opportunities for reward hacking. It receives the kernel source, relevant PTX documentation, and hardware-specific optimization guidelines. It can execute shell commands and modify files within a dedicated workspace, but it cannot inspect the separate evaluation codebase or modify the evaluation harness. It can also invoke the verification and benchmarking pipeline through predefined tool calls, but it cannot modify the underlying evaluation procedures.

Q9. Compilation Speed, Cost, and Determinism

For kernels that remain stable and execute repeatedly, longer compilation times can be worthwhile when the resulting speedups amortize the optimization cost. Substantial search costs also arise in existing compiler systems: TVM reports that tuning a single neural network can take several hours (TVM/Ansor), while Mirage reports optimization times of up to four hours per LAX tensor program, incurred once before deployment (Mirage, OSDI 2025, §8.4).

To improve efficiency and reproducibility, future implementations could cache previously optimized kernels, reuse successful optimization strategies, and impose fixed search budgets. A more structured optimization process could also reduce variability, although this introduces a trade-off with exploration. Our current work focuses on demonstrating the feasibility of verification-guided AI compilation; systematically optimizing compilation cost, latency, and reproducibility remains an important direction for future work. “C3. Search Overhead and Compilation Costs” in the common response reports measured wall-clock times, evaluation calls, LLM API costs, and a stage breakdown for three instrumented runs.

Reviewer_HWGFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes AI lowering, an alternative compilation approach in which LLMs directly translate Triton kernels into NVIDIA PTX, bypassing the conventional compiler lowering and optimization pipeline. It introduces the Triton AI Compiler and Triton Compiler Environment , which enable iterative PTX generation and optimization using compilation feedback, numerical correctness testing, benchmarking, and profiling. The approach also integrates formal verification into the optimization loop.

Critical strengths
  1. The paper is well written and easy to follow. The experimental evaluation covers diverse GPU kernels and hardware architectures, providing a thorough assessment of the proposed approach.
  2. I like the idea of incorporating formal verification into the optimization loop, moving beyond numerical testing toward stronger correctness guarantees for AI-generated GPU kernels.
Critical weaknesses
  1. Verifier extensions are unclear: How are instructions such as tcgen05, wgmma, and mbarrier modeled in the symbolic executor, particularly their memory-ordering and synchronization semantics.
  2. How stable are the AI-generated kernels when integrated into real inference? Can we evaluate their impact on end-to-end model accuracy, numerical error accumulation, compared with Triton-generated kernels?
  3. CAKE reports the opposite result: in clean-start Flash-KMeans experiments on B200, agents writing CAKE IR reach 1.144× over a tuned baseline, while a direct CUDA/PTX arm reaches only 0.928×. Could the authors comment on this? For example, is it explained by the stronger model used here, the starting point (a fixed Triton kernel versus a high-level description), the feedback loop, or the workloads? If feasible, a comparison on overlapping kernels under a matched model and API budget, reporting both best speedup and search efficiency (API cost, number of candidates, and wall-clock time to reach parity), would help resolve this. CAKE also argues that a typed IR enables verification before code generation. Given that the paper identifies verification as the central open challenge, could the authors discuss whether an agent-facing IR would be a more practical route to closing that gap than extending a PTX-level verifier?
Questions for authors
  1. Can you provide a detailed breakdown of how many generated candidates are rejected due to actual inequivalence, unsupported PTX instructions, or verification timeouts?
  2. Although the cumulative LLM API costs are reported, the experiment lacks a detailed analysis of search overhead, including wall-clock compilation time, number of evaluated candidates, and costs associated with testing, profiling, and formal verification. How does this overhead compare with conventional Triton compilation?
  3. Can you provide a deeper analysis of the performance regressions, including the underlying hardware bottlenecks and whether they stem from limitations in PTX generation or insufficient search?
3 comments

Thank you for your thoughtful and constructive feedback. We appreciate your recognition of the breadth of our evaluation and the importance of integrating formal verification into AI-based compiler optimization. Below, we address your concerns regarding verifier semantics, end-to-end numerical stability, comparisons with CAKE, verification failures, search overhead, and performance regressions.

W1. Modeling of GPU Instructions and Synchronization Semantics

On top of Volta's symbolic execution, our extension records an event graph in which each asynchronous operation, such as a TMA transfer or a tcgen05 operation, appears as separate issuance, memory-effect, and completion events, and in which mbarrier phases and counters are tracked explicitly. Memory ordering and synchronization are never inferred from issue order: they are established by five sets of checks on that graph.

The details are in the common response, under “C1. Our Extension of Volta,” which separates what Volta already provided from what we added and lists what each check requires. We will add this description to the paper.

W2. End-to-End Inference Stability and Numerical Accuracy

To investigate numerical stability beyond isolated kernel-level tests, we conducted an end-to-end inference experiment in which selected kernels from two open-source LLMs, Apertus and Gemma 4B, were replaced with TAIC-generated implementations. In the configurations we evaluated, the generated text was identical whether the models used the original implementations or the TAIC-generated kernels.

We also compared numerical discrepancies across three implementations: PyTorch, Triton, and TAIC. In our experiments, the numerical differences between Triton and TAIC were smaller than those between PyTorch and Triton. Although TAIC uses Volta, the verifier reasons over idealized real-number semantics rather than finite-precision floating-point arithmetic. Consequently, the verification does not account for rounding errors or differences arising from the reordering of floating-point operations, which may lead to different numerical results in practice.

W3. Comparison with CAKE and the Role of an Agent-Facing IR

Thank you for highlighting CAKE, which provides complementary evidence for the benefits of agent-facing IRs. We believe the two results are consistent rather than opposite, and the model is the most likely explanation. CAKE's Flash-KMeans experiments, which include the direct CUDA/PTX arm that reaches 0.928×, use GPT-5.6 Sol. With that model, our own direct lowering was also below parity (0.97× on FP16 GEMM); it is with GPT-6 Astra that TAIC exceeds parity (1.02×). We return to this point below.

CAKE introduces a typed, hardware-explicit intermediate representation. The two methods differ in several ways, the most significant of which concerns verification. Unlike TAIC, CAKE does not formally verify the generated code. Synchronization and memory-ordering errors (e.g., missing fences or mbarrier misuse) can therefore survive output-based tests. This is exactly the class of bugs that our verifier caught in two kernels (a Mamba kernel and a warp-specialized matrix-multiplication kernel on B200).

CAKE performs numerical correctness testing against reference implementations and reports end-to-end validation for applicable workloads. However, its IR-level static analyses do not formally guarantee the correctness of the generated GPU code, a limitation explicitly acknowledged by CAKE's authors. These analyses can reject invalid candidates early, but they do not eliminate the need to validate the final implementation. We therefore view IR-level and PTX-level verification as complementary rather than competing approaches. Verification techniques developed for generated PTX could also be applied to CAKE's compiler output, while structural checks could benefit our approach.

Unlike TAIC, CAKE still relies on a conventional compiler backend to translate its IR into CUDA/PTX. Even its direct CUDA/PTX baseline generates CUDA C++ with inline PTX rather than complete PTX kernels. Consequently, CAKE is limited by what its IR and compiler can express, whereas TAIC directly emits PTX and can exploit architecture-specific features (e.g., TMA, wgmma, tensor memory) without compiler support. This is particularly valuable for emerging hardware architectures.

As LLMs improve, CAKE's reliance on compiler infrastructure may become an increasingly significant limitation, and we believe the model also explains why CAKE's direct CUDA/PTX arm reaches only 0.928× whereas TAIC exceeds parity. CAKE's experiments use GPT-5.6 Sol. When we ran TAIC with that model, FP16 GEMM reached only 0.97× over autotuned Triton and the required budget was 150 dollars, which is comparable to CAKE's result for its direct arm; with GPT-6 Astra, the same kernel reaches 1.02× while API cost drops by over 90%.

TAIC delivers high performance across a diverse range of kernels with a lightweight, generalizable approach. CAKE uses GPT-5.6 Sol and an 80-million-token budget for its Flash-KMeans experiments, whereas we evaluate 22 kernels across three GPU architectures, generally spending less than 15 dollars per kernel. Reproducing CAKE's search budget is beyond our available resources. Moreover, differences in models, workloads, and starting representations prevent us from drawing conclusions about the relative efficiency of the two approaches. We agree that a matched-budget comparison would be valuable future work.

We will clarify that our contribution is demonstrating competitive direct AI lowering without a conventional compiler backend.

Q1. Breakdown of Verification Failures

Among candidates evaluated with our extended Volta verifier, two were rejected for synchronization errors (the missing fences described in C1), none for arithmetic inequivalence, and none for timeouts; all other failures stemmed from unsupported PTX constructs, mainly atomics, input-dependent control flow, and bit-level manipulation (Section 4.5, Appendix B). For further context, see the common response: “C2. Generation Reliability” discusses candidate revisions, and “C1. Our Extension of Volta” describes the scope of the verifier's guarantees.

Q2. Search Overhead and Compilation Costs

In three instrumented B200 runs, the full workflow took 5 to 51 minutes of wall-clock time, 3 to 14 evaluation calls, and 0.35 to 12.41 dollars in API charges, against 8 to 99 seconds for Triton compilation with autotuning. LLM inference dominates for GEMM and FlashAttention (56% and 73% of wall-clock time), repeated benchmarking dominates for sigmoid, and assembling the PTX takes under one second.

The full stage breakdown, including testing and profiling time, is in the common response, under “C3. Search Overhead and Compilation Costs.”

Q3. Analysis of Performance Regressions

On some kernels, we observe a remaining performance gap between LLM-generated PTX and highly optimized implementations such as CUTLASS and cuBLAS. FP16 matrix multiplication on the NVIDIA L40S is a particularly challenging example. The profiled PTX candidate achieves a speedup of approximately 0.99× relative to Triton. Approaching Triton performance, however, does not imply matching these specialized libraries.

Matching CUTLASS requires jointly optimizing threadblock and warp tiling, register allocation, shared-memory layouts, software pipelining, and the output epilogue. These decisions interact: increasing data reuse can increase register pressure, while deeper buffering consumes additional shared memory. CUTLASS explicitly coordinates these mechanisms rather than relying on Tensor Core instructions alone. Our generated PTX already incorporates mma.sync, asynchronous copies with cp.async, double-buffered shared memory, and vectorized output stores. The remaining challenge is therefore not simply generating the appropriate instructions, but organizing them into an implementation that matches the efficiency of a tuned library kernel.

Example hardware bottleneck: The L40S profile points primarily to a compute-pipeline bottleneck rather than DRAM-bandwidth saturation: the reported Tensor-pipeline activity is approximately 87.5%, whereas DRAM throughput reaches only 9.75% of its reported peak. The dominant sampled stall reason is math_pipe_throttle, indicating that warps are waiting for an execution pipeline to become available; the profile also records substantial wait stalls associated with fixed-latency execution dependencies. Together, these observations suggest pressure on the arithmetic pipeline and instruction dependencies, rather than a failure to use Tensor Cores. The kernel additionally uses 222 registers per thread and 48 KiB of shared memory per block, with the profile reporting both register and shared-memory residency limits of one block per multiprocessor and 16.6% achieved occupancy, without register spilling. This limits the ability to hide latency, although low occupancy alone does not establish the cause of the performance gap, particularly given the already-high Tensor-pipeline activity.

We present a detailed profile only for the L40S GEMM, but our analyses of other kernels where TAIC regresses relative to Triton suggest the same bottleneck: the generated PTX uses the appropriate instructions, and the gap appears to stem from the joint organization of tiling, register allocation, and pipelining.

We tried improving the optimization harness and allocating additional search budget, but reached a performance plateau on this workload and on other kernels: the LLM did not discover an implementation that closed the gap to the optimized library kernels. Consequently, we do not attribute this result solely to the original search budget or assume that additional iterations would necessarily resolve it. We view it as a practical limitation of the current model and optimization procedure, rather than an intrinsic limitation of PTX or evidence that direct AI lowering cannot improve. These experiments do not conclusively separate model capability from search effectiveness. The profiling does, however, identify the limiting resources in the generated implementation.

We expect future generations of LLMs to further narrow this gap. For instance, GPT-6.1 Sol achieves the same speedup over standard matrix multiplication with a budget of just 2 dollars, compared with 6 dollars at the time of our ICLR submission—a 67% reduction in cost. We anticipate that increasingly capable models will drive further improvements in cost efficiency.

Reviewer_a51nOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the Triton AI Compiler, TAIC, an LLM-based system that directly lowers Triton kernels to NVIDIA PTX while preserving the kernel ABI, launch configuration, compile-time constants, and target-GPU contract. The evaluation covers twelve common kernels and ten kernels from recent ML papers on Ada, Hopper, and Blackwell GPUs. The reported speedups over autotuned Triton range from 0.83× to 3.34×, with particularly large gains on BitDelta, convolution, FlashAttention, FlashSinkhorn, and SageAttention.

Critical strengths
  1. The paper evaluates the method on L40S, H100, and B200 and shows that generated PTX can use architecture-specific mechanisms such as mma.sync, wgmma, TMA, tcgen05, and tensor-memory accumulators. This is consequential because it provides concrete evidence that an LLM can sometimes perform low-level lowering and instruction selection at a level competitive with a mature compiler backend, rather than merely producing high-level CUDA-like code.

  2. The analysis of Volta’s coverage and the performance cost of restricting generation to verifier-supported subsets is especially valuable. It highlights a concrete gap between the hardware features needed for high performance and the semantics that current formal tools can reliably check.

Critical weaknesses
  1. Although the reported speedups are impressive, the main grid reports one run per experiment and the repeatability study covers only four B200 kernels. The evaluation also uses selected problem sizes and does not compare against hand-tuned CUDA/CUTLASS or vendor-optimized implementations under matched budgets.

  2. Correctness assurance remains substantially weaker than the performance results. The paper acknowledges these limitations, which is helpful, but the combination of single-run performance results and incomplete verification makes it difficult to assess the reliability of the generated kernels in production-like settings.

Questions for authors
  1. What is the exact randomized differential-testing protocol: number of inputs, input distributions, tolerances, and coverage of edge cases such as NaNs, infinities, denormals, negative zero, unusual strides, and boundary shapes?

  2. How do the generated kernels compare with state-of-the-art CUDA/CUTLASS or vendor-optimized implementations, especially for GEMM, FlashAttention, BitDelta, and convolution?

3 comments

Thank you for your thoughtful feedback and positive assessment of our work. We particularly appreciate your recognition of the significance of AI lowering on modern GPU architectures and the challenges of formally verifying performance-critical PTX.

We address your concerns regarding performance reproducibility, comparisons with optimized implementations, and correctness guarantees below.

W1. Performance Reproducibility and Statistical Reliability

We agree that reporting a single measurement per experiment limits the assessment of performance stability. We report a single run per experiment because of cost, and we acknowledge that this leaves uncertainty about run-to-run variability. To assess reliability, we nevertheless reproduced four kernels five times each, and compilation did not fail in any of these runs. We invite you to look at the repeatability study in the paper for the variation in performance across these runs.

We chose tensor sizes representative of neural network workloads. For example, consider a prefill pass with batch size 1 and a sequence length of 2,048 tokens. The official model configurations imply the following kernel shapes:

  • Feed-forward GEMM: Mistral-7B-v0.3's up projection multiplies (2048, 4096) × (4096, 14336) to produce (2048, 14336); its down projection multiplies (2048, 14336) × (14336, 4096) to produce (2048, 4096). Mistral configuration.
  • RMSNorm: Mistral-7B-v0.3 normalizes an activation tensor of shape (2048, 4096), with 2,048 independent rows and a reduction over 4,096 features per row. Mistral configuration.

These sizes are static multiples of 32, so we can use static shapes and compare implementations without masking.

For transparency, our reported speedups are relative to the autotuned Triton baseline. Comparisons with hand-tuned CUDA, CUTLASS, and vendor-optimized implementations provide additional context about absolute performance; we report them in our response to Q2. Developing and maintaining such optimized implementations requires substantial specialist effort: GPU application tuning for new architectures can take months, as discussed in Record-Remix-Replay. A research group that needs to iterate rapidly generally cannot dedicate full-time GPU engineers to writing and retuning a custom kernel for every new operator, fusion, or hardware target. AI as a compiler is intended to help address this constraint by automating the generation and optimization of low-level code.

This motivation is even stronger for emerging accelerators, whose software ecosystems may not yet offer a mature equivalent of CUTLASS or sufficient library coverage for the desired operations.

Our primary objective is to compare AI-based lowering with a mature compiler while preserving the source kernel's ABI and launch configuration. This is why we used autotuned Triton as the baseline and did not include CUTLASS in the main evaluation.

W2. Correctness Assurance and Verification Limitations

To our knowledge, TAIC's correctness checking is the state of the art for LLM-generated GPU kernels, because it combines differential testing and Compute Sanitizer with formal verification, and its coverage is now broader than at submission. Our extension of Volta checks memory safety and synchronization for asynchronous Blackwell kernels and found two missing fences that output-based tests had missed, and the differential tests now cover the edge cases you raised (see Q1).

The details are in the common response: “C1. Our Extension of Volta” describes what the verifier checks and what it guarantees, and “C2. Generation Reliability” reports how often candidates fail each check.

Q1. Randomized Differential-Testing Protocol

We describe the randomized differential-testing protocol below.

Each candidate PTX kernel is compared against the Triton reference on 1,000 generated input sets, stopping at the first mismatch. Both implementations receive independent copies of the same inputs, and CUDA is synchronized before comparison. The checker resets all RNGs to seed 42 before each verification call, so all candidates are evaluated on the same reproducible sequence of inputs. Inputs follow workload-specific distributions: uniform [0, 1) for most elementwise and normalization kernels, Gaussian for GEMM and attention, and discrete values for integer kernels. FP8 inputs are quantized where applicable. Shapes and launch configurations are fixed per kernel (e.g., 4096 × 4096 GEMM and 4096 × 256 softmax); only input values change across iterations.

Numerical comparison: Correctness requires matching output shapes and torch.allclose agreement, with the reference cast to the candidate's dtype. The default tolerance is atol = rtol = 1e-3, meaning each element must satisfy |candidate − reference| ≤ atol + rtol × |reference|. More numerically sensitive kernels use relaxed tolerances: 1e-2 for many FP8 and fused operations, 1e-1 for selected FP8 operations and FlashAttention backward, and 2e-2 to 5e-2 for Mamba variants. Thus, we check numerical agreement rather than bitwise equality. Failures report the iteration, mismatching indices, and error statistics.

Coverage and limitations: The original generators did not explicitly test NaNs, infinities, signed zeros, or subnormals. Moreover, nonnegative inputs may leave the negative branches of activation functions untested. The default evaluator additionally runs Compute Sanitizer (memcheck, racecheck, synccheck, initcheck) to detect memory and synchronization errors. These checks strengthen validation but do not guarantee correctness for all inputs or execution schedules.

Independence from optimization: Test inputs are generated outside the candidate PTX and are not included in the solver prompt. However, the generators and seed are public, and there is no secret held-out test set. We therefore characterize this as reproducible differential testing, not formal verification.

Following your suggestions, we extended the tests to cover the edge cases you raised: NaNs, infinities, subnormals, negative zero, negative values, unusual strides, and boundary shapes. During testing, we identified some discrepancies in NaN propagation between PyTorch and Triton. In every such case, however, our implementations matched the behavior of at least one of the two frameworks. All current generated kernels pass these extended tests, improving empirical confidence without changing the verification guarantees discussed in the common response.

Q2. Comparison with CUDA/CUTLASS and Vendor-Optimized Implementations

Our primary comparison against autotuned Triton is motivated by the central question of our paper: whether an LLM can replace the traditional Triton lowering pipeline while preserving the source kernel's ABI and launch configuration. We agree, however, that comparisons with specialized implementations provide valuable context for evaluating absolute performance.

We therefore evaluated TAIC against CUTLASS v4.7.1, cuBLAS, and cuDNN on NVIDIA B200 (sm_100a). We used the paper's PTX Gym timing protocol, with identical seeded inputs, L2-cache flushing, and ten rounds with rotated implementation order. All implementations passed correctness checks at the benchmark's prescribed tolerances.

For each workload, the table reports the fastest measured specialized baseline that matches its input/output requirements. Latencies are means of per-round medians; speedup is baseline latency divided by TAIC latency.

WorkloadSpecialized baselineBaseline (ms)TAIC (ms)TAIC speedup
FP16 GEMMcuBLAS0.10350.11160.93×
Convolution, NCHWcuDNN0.05450.05261.04×
FlashAttention, O and LSE outputsCUTLASS SM100 FMHA0.06800.06451.05×

For GEMM and FlashAttention, the TAIC results use the best of five independent generation runs; the median-run latencies were 0.1117 and 0.0680 ms, respectively. These results show that TAIC approaches cuBLAS performance on GEMM, achieves slightly lower measured latency than cuDNN on NCHW convolution, and outperforms the tested CUTLASS attention implementation. The attention result is specific to this workload.

We do not include the BitDelta workloads in this comparison because no CUTLASS implementation exists for them: these custom kernels fall outside the operators covered by vendor libraries, and a specialized baseline would have to be engineered by hand. This is precisely where treating AI as a compiler is most valuable. Specialized libraries cover a fixed set of well-studied operators, whereas TAIC lowers any Triton kernel, including custom kernels for which no hand-optimized implementation exists.

Overall, these comparisons demonstrate competitive performance on the tested workloads for which specialized implementations are available. They support our claim that TAIC can improve Triton lowering under fixed ABI and launch constraints, without implying that it consistently surpasses vendor-optimized libraries.

Comments on the paper

5 comments

C3. Search Overhead and Compilation Costs

We agree that LLM API charges are not the only bottleneck of AI-based compilation: compilation time also grows relative to a traditional compiler. We therefore instrumented three additional runs on NVIDIA B200, recording end-to-end wall-clock time, evaluation calls, LLM API charges, and time spent in individual stages. The following results use GPT-6 Astra, with medium reasoning for sigmoid and maximum reasoning for GEMM and FlashAttention.

WorkloadEvaluation callsCalls reaching benchmarkingWall-clock time (min)LLM API cost (USD)Final speedup over autotuned Triton
FP16 sigmoid335.320.351.0001×
FP16 GEMM141351.1211.431.0204×
FlashAttention forward12847.7212.411.2594×

Evaluation calls include candidate attempts made through the agent's tools and the harness's reevaluation of the returned candidate. The counts therefore measure evaluation work, rather than independent runs or outer optimization rounds.

The stage breakdown below reports accumulated exclusive times in seconds, with minutes in parentheses, so that nested tool calls are not double-counted. The source measurements are the archived overhead profiles for sigmoid, GEMM, and FlashAttention.

Stage: seconds (minutes)SigmoidGEMMFlashAttention
Triton baseline compilation/autotuning8.14 (0.14)99.44 (1.66)31.74 (0.53)
LLM API and client-side overhead56.18 (0.94)1728.58 (28.81)2097.50 (34.96)
Candidate PTX assembly (ptxas)0.05 (<0.01)0.93 (0.02)0.51 (0.01)
Compute Sanitizer38.44 (0.64)289.87 (4.83)177.21 (2.95)
Numerical correctness/JIT loading0.80 (0.01)17.74 (0.30)13.71 (0.23)
Benchmarking199.68 (3.33)860.94 (14.35)498.88 (8.31)
Nsight Compute invocations13.95 (0.23)67.00 (1.12)41.37 (0.69)
Other setup and bookkeeping1.77 (0.03)2.42 (0.04)2.03 (0.03)
Total wall-clock time319.00 (5.32)3066.90 (51.12)2862.93 (47.72)

LLM calls account for 56.4% of GEMM's wall-clock time and 73.3% of FlashAttention's. For sigmoid, candidate evaluation dominates at 79.3%, principally because of repeated benchmarking. PTX assembly itself contributes less than one second in each run. Thus, the observed overhead comes primarily from model inference and repeated evaluation, rather than from assembling the generated PTX.

The Triton baseline stage took 8.14–99.44 seconds, compared with 5.32–51.12 minutes for the full AI workflow, which includes that baseline stage. This reference includes autotuning, so it does not measure pure Triton JIT compilation. The additional cost motivates using TAIC as an offline optimization mode for frequently reused kernels. Caching accepted kernels and using explicit search budgets can amortize or bound that cost, but the current measurements do not establish a deployment-wide break-even point or deterministic compilation behavior.

To quantify this across all experiments, we classified every verification call (launch_verifier and verify_current_ptx) made in the 120 GPT-6 Astra runs of our PTX agent into exactly one of five outcomes: success (the candidate compiled, passed the correctness checks, and was benchmarked); syntax failure (ptxas rejected the PTX); numerical failure (the candidate compiled but produced incorrect outputs); sanitizer failure (the candidate passed the correctness checks, but the Compute Sanitizer run reported a memory error, race, or illegal instruction, hit a harness error, or timed out after 120 s); and other (a CUDA “invalid argument” error during the correctness check, Compute Sanitizer failing to start, or a Volta equivalence-check failure).

HardwareRunsCallsSuccessSyntaxNumericalSanitizer (real / harness / timeout)Other
L40S3020516911421 (4 / 5 / 12)0
H100 PCIe97160515 (1 / 4 / 0)0
H100 SXM211361176112 (3 / 9 / 0)0
B20060432334391235 (30 / 5 / 0)12
Total120844680611873 (38 / 23 / 12)12

Overall, 680 of 844 calls (80.6%) passed all executed checks and reached benchmarking. Syntax failures account for 61 calls (7.2%) and numerical failures for only 18 (2.1%). Of the 73 sanitizer failures, 23 are harness errors rather than kernel defects: in 21 cases, the line-info directives (.loc) inserted by the sanitizer instrumentation caused ptxas to fail, and the remaining two were internal Compute Sanitizer errors. A further 12, all on L40S, are 120 s timeouts rather than reported memory errors. We conservatively count all of these calls as failures; only 38 calls (4.5%) were rejected for genuine memory errors or races, 30 of them on B200.

The following tables break these results down per kernel for the paper's B200 problems and for the GEMM and convolution kernels across hardware. Best speedup is the highest speedup over Triton among the verified calls of the corresponding runs.

Kernel (B200)RunsCallsSuccessSyntaxNumericalSanitizerOtherBest speedup
BitDeltaNeurIPS2024BatchedMatmul18800002.51×
BitDeltaNeurIPS2024Matmul1131110103.34×
Dion2TritonPostOrthogonalize18610100.97×
FlashAttentionNeurIPS2022Forward5412876001.37×
FlashSinkhornFusedSchurMatvec18700101.53×
ForgettingAttentionICLR2025Forward111411231.01×
LionNeurIPS2023Optimizer15410001.00×
Mamba2ChunkScanForward18700101.07×
Mamba2ChunkStateForward110910001.10×
SageAttentionICLR202515120021.41×
Total (10 kernels)141178514765—
KernelHardwareRunsCallsSuccessSyntaxNumericalSanitizerBest speedup
GEMM FP16L40S11090011.00×
GEMM FP16H100 SXM22826002†0.83×
GEMM FP16B200563513091.03×
GEMM FP8L40S1550001.14×
GEMM FP8H100 SXM176001†1.61×
GEMM FP8B2001861011.00×
Conv2D FP16L40S1970021.17×
Conv2D FP16H100 SXM165001†1.90×
Conv2D FP16B200114101121.36×
Conv2D FP8L40S164101‡1.17×
Conv2D FP8H100 SXM198001†1.55×
Conv2D FP8B200550434212.23×

† Harness error (.loc parse failure) on the first call; no real memory errors were reported for these kernels on H100 SXM. ‡ 120 s sanitizer timeout. None of these kernels produced an “other” outcome, and none was run on H100 PCIe.

Two patterns stand out in the per-call sequences. First, syntax failures are dominated by two recurring errors: a mov from %tid.x written in a form that ptxas parses as a video selector (.x), and incorrect operands for tcgen05.mma on B200. In most runs, the error occurs on the first call and is fixed on the next. Second, real sanitizer failures are concentrated on B200, where they mainly involve out-of-bounds shared-memory accesses, races between cp.async writes and shared-memory reads, uninitialized tensor-map reads, and mbarrier misuse; these are precisely the defects that output comparison alone would miss.

C2. Generation Reliability

GPT-6 Astra was substantially more reliable than GPT-5.6 Sol in our experiments. This model difference also matters for the comparison with CAKE, whose experiments use GPT-5.6 Sol (see our response to HWGF, W3). Once the pipeline had stabilized and environment bugs had been resolved, none of our GPT-6 Astra experiments required restarting the optimization run. Each run nevertheless involved multiple candidate revisions. When failures occurred, they followed a consistent order: syntax errors in the first one or two attempts, numerical mismatches around attempts three to five, and, from then on, candidates that passed the correctness checks and were optimized further for performance. These observations describe convergence within our evaluated experiments; they do not imply that initial candidates were correct or that later revisions could never introduce regressions. We found two kernels to be rejected by Volta for missing fences.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.