- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper presents the Triton AI Compiler (TAIC), an agentic pipeline that lowers Triton kernels directly into NVIDIA PTX. An LLM generates and iteratively optimizes PTX according to feedback from Triton Compiler Environment (TCEnv), including numeric checks and performance results. The authors also extend an off-the-shelf PTX verifier to support additional GPU features, which checks the equivalence for 16 of 22 kernels. Evaluation shows that TAIC achieves 0.83x-3.34x performance of autotuned Triton on modern NVIDIA GPUs.
Critical strengths
- Good performance on evaluated workloads over autotuned Triton compiler.
- Useful investigation of verification limits. Integrating Volta and measuring the performance impact of its supported fragment exposes a concrete gap between current verification capabilities and modern GPU optimization.
Critical weaknesses
- Weak verification. The claimed verification coverage relies on Volta extensions whose soundness is explicitly not established. Numeric check is vulnerable to reward hacking.
- Unclear source of performance gains. Evidence of gains on newly constructed Triton kernels is limited. The frontier model may have seen the evaluated kernels during training. No ablation study isolates the harness’s contribution.
Questions for authors
The following comments and questions are related to the aforementioned weaknesses.
- Appendix B explicitly declines to claim soundness or completeness for the Volta extensions and leaves their formalization to future work. Given this, what guarantees does acceptance by the extended verifier actually provide, and what justifies describing the 16 accepted kernels as “verified”? Please provide more details on that.
- The correctness of the kernel is heavily depended on numeric check, which is vulnerable to reward hacking, such as exploiting loose tolerances, overfitting to the test inputs, or skipping computation the tests don’t exercise.
- Async operations are critical to achieve state-of-the-art performance on modern NVIDIA GPUs, but they are out of the scope of Volta. Can you extend the verification based on async kernel verification work, such as Weft [1]?
- Please provide more details on compiler lowering harness. How is the initial PTX generated, and how is an optimization plan translated into subagent tasks? Do subagents differ or just simply sample independently? How many subagents are used?
- Triton has a known performance gap on Hopper and Blackwell GPUs. Did you try newer Triton-based systems for baselines, such as Tawa and Gluon?
- Most evaluations still use common and publicly available workloads. The synthetic kernels (randomly generated and modification to familiar examples) seem relatively easy to optimize and provide limited evidence of generalization. Can you evaluate TAIC on widely used Triton kernels created after GPT-6 Astra’s knowledge cutoff?
- Can you ablate the harness and compare it with other harnesses like codex? Including an open-weight model would also help make the experiments more controlled and compelling.
- How is the agent sandboxed? Can it access Internet or modify the evaluation harness?
- Traditional compilers are generally fast, cheap, and deterministic, which is important in production where compilation may be invoked frequently. How can TAIC’s agentic workflow be made faster, cheaper and more deterministic? For example, a more structured agentic workflow may improve reproducibility but restrict exploration.
[1] Rahul Sharma, Michael Bauer, and Alex Aiken. 2015. Verification of producer-consumer synchronization in GPU programs. In Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI '15). Association for Computing Machinery, New York, NY, USA, 88–98.
3 comments
Q5. Newer Triton-Based Baselines (Tawa and Gluon)
The gap between the performance Triton achieves and what newer GPU architectures can deliver is precisely what motivates our work. Gluon addresses this gap by exposing more explicit control over hardware-level details, at the cost of additional programming effort. Tawa, in contrast, preserves the Triton programming interface but introduces specialized compiler transformations for automatic warp specialization.
Our evaluation explicitly focuses on the standard upstream Triton compilation pipeline, rather than alternative programming interfaces or research compiler extensions. Our objective is to measure how well an LLM-based compiler performs against a modern production compiler, using the same high-level Triton source programs. This allows us to directly assess the performance improvements achievable through AI-based lowering over conventional compiler optimization.
Q6. Generalization to Recent Real-World Kernels
The randomly generated kernels contain long, dense sequences of shifts, bitmasks, and other operations designed to force the LLM to reason about the computation. To address your request for recent practical workloads, we additionally compiled the selective-logprob forward and backward kernels introduced in Liger-Kernel PR #1548, opened on October 20, 2026. The kernel bodies and arguments in our recorded prompts match the corresponding functions in commit 5567d8c. These kernels compute selected-token log-probabilities and their gradients for the PR's PPO/GRPO implementation.
On NVIDIA B200, we evaluated BF16 logits with bias enabled and ignored targets included. The forward kernel produces both log-probabilities and log-normalizers; the backward kernel writes logit gradients in place. The Triton baseline was autotuned over 12 configurations for each kernel, selecting a block size of 4096 and four warps. TAIC retained the source ABI and the selected 128-thread launch configuration.
| Kernel | Autotuned Triton p50 (ms) | TAIC p50 (ms) | Speedup |
|---|---|---|---|
| Selective-logprob forward | 0.13514 | 0.06465 | 2.09× |
| Selective-logprob backward | 0.36347 | 0.28771 | 1.26× |
TAIC achieved speedups on both kernels in a single run each: the forward kernel reached its final speedup on the fourth verification call, after ptxas rejected the first candidate, and the backward kernel on the third.
Q7. Harness and Model Ablations
Regarding model ablations, we experimented with earlier-generation and open-weight LLMs, but found them substantially less reliable at generating valid, performant PTX. At submission time, GPT-6 Astra was the only model that consistently produced competitive results.
Our primary objective is to investigate the feasibility of AI-based compilation with a strong reasoning model and a verification-guided optimization loop, rather than to benchmark general-purpose coding agents. We acknowledge that systematic comparisons across agentic harnesses would be valuable future work. We likewise leave post-training of a large language model to future work.
Q8. Agent Sandboxing
The agent operates in a restricted environment with no internet access, limiting opportunities for reward hacking. It receives the kernel source, relevant PTX documentation, and hardware-specific optimization guidelines. It can execute shell commands and modify files within a dedicated workspace, but it cannot inspect the separate evaluation codebase or modify the evaluation harness. It can also invoke the verification and benchmarking pipeline through predefined tool calls, but it cannot modify the underlying evaluation procedures.
Q9. Compilation Speed, Cost, and Determinism
For kernels that remain stable and execute repeatedly, longer compilation times can be worthwhile when the resulting speedups amortize the optimization cost. Substantial search costs also arise in existing compiler systems: TVM reports that tuning a single neural network can take several hours (TVM/Ansor), while Mirage reports optimization times of up to four hours per LAX tensor program, incurred once before deployment (Mirage, OSDI 2025, §8.4).
To improve efficiency and reproducibility, future implementations could cache previously optimized kernels, reuse successful optimization strategies, and impose fixed search budgets. A more structured optimization process could also reduce variability, although this introduces a trade-off with exploration. Our current work focuses on demonstrating the feasibility of verification-guided AI compilation; systematically optimizing compilation cost, latency, and reproducibility remains an important direction for future work. “C3. Search Overhead and Compilation Costs” in the common response reports measured wall-clock times, evaluation calls, LLM API costs, and a stage breakdown for three instrumented runs.
Thank you for your detailed review and your questions. We appreciate your recognition of the performance results and of our analysis of Volta's coverage. Below, we address your questions on verification and generation reliability, reward hacking, the lowering harness, baselines, generalization, ablations, sandboxing, and compilation cost.
Q1 & Q3. Concerns Addressed in the Common Response
Our extension adds to Volta an event-graph analysis that checks memory safety and synchronization for asynchronous Blackwell kernels. “C1. Our Extension of Volta” in the common response separates what Volta already provided from what we added and lists what each check requires.
Across 120 GPT-6 Astra runs, 680 of 844 verification calls (80.6%) passed all executed checks, and no run needed a restart once the pipeline had stabilized. “C2. Generation Reliability” in the common response gives the breakdown by failure type, hardware, and kernel.
Q2. Numerical Checks and Reward Hacking
We agree that numerical testing alone is vulnerable to reward hacking, particularly through overfitting to sampled inputs or exploiting loose tolerances. This is precisely why TAIC combines numerical testing with symbolic verification—to the best of our knowledge, the first such approach in LLM-based GPU kernel compilation. For example, the ReLU test generator samples only nonnegative inputs, so a kernel implementing the identity function could pass all numerical tests despite being incorrect for negative inputs. In contrast, our symbolic verifier checks the kernel against the mathematical specification, max(x, 0), allowing it to detect this discrepancy independently of the sampled inputs.
Importantly, the two verification mechanisms are complementary. Our symbolic verifier reasons under infinite-precision arithmetic and therefore does not capture all floating-point behaviors. Numerical testing remains essential to detect incorrect results arising from reduced-precision arithmetic, which an LLM might otherwise exploit to achieve artificial speedups. Together, these mechanisms prevent reward hacking by verifying not only that the outputs are correct on a given set of test inputs, but also that the implemented function is mathematically equivalent to the target computation. This ensures that an LLM cannot achieve artificial speedups by substituting a simpler function that happens to produce the correct outputs on the test distribution but behaves incorrectly on other inputs.
Sigmoid illustrates this well. Its test generator samples inputs uniformly from [0, 1), where sigmoid is smooth and nearly linear. A kernel that replaces 1 / (1 + exp(-x)) with the fifth-order Taylor polynomial 1/2 + x/4 - x^3/48 + x^5/480 avoids both the exponential and the division. Its maximum error on [0, 1) is about 1.9 × 10⁻⁴, well within the 10⁻³ verification tolerance, so it passes every numerical test. Outside the sampled range, however, it is wrong: at x = 4, it returns 2.30 instead of 0.98. Our symbolic verifier checks the kernel against the specification 1 / (1 + exp(-x)), which the polynomial does not match, and therefore rejects it regardless of the sampled inputs.
This example also shows that the symbolic check is strict: Volta rejects the kernel outright, because the Taylor polynomial is symbolically different from the requested function, however small its numerical error. Deciding which approximations should be accepted is a question we leave to future work.
Q4. Compiler Lowering Harness and Subagent Design
In the current implementation, Triton compiles the reference kernel to establish its PTX signature, launch contract, and performance baseline. This is an engineering simplification that leverages the existing Triton infrastructure for kernel setup and benchmarking, rather than a fundamental requirement of our approach. By default, the LLM generates the initial candidate from the Triton source and this context; optional flags load saved Triton PTX or a previous candidate. Each optimization round uses the current candidate, benchmark results, and available Nsight Compute metrics to propose one to three ordered improvements, each with a name, rationale, and concrete instruction. Each improvement becomes a separate task for the same model, which patches, verifies, and benchmarks candidate variants. The tasks run sequentially with distinct optimization instructions, and accepted speedups become the starting point for subsequent tasks. Each round therefore comprises one to three sequential candidate-generation tasks; there are no parallel subagents.