- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Glueball, which equips LLM agents with theorem-proving tools to reason about correctness during GPU kernel generation and optimization for Gluon. Glueball translates reference Triton kernels into initially unoptimized Gluon implementations and iteratively improves them using a multi-agent pipeline that combines profiling, optimization proposal generation, implementation, correctness checking, and end-to-end benchmarking. The evaluation results show that improved kernel bug detection rate after applying Glueball's pipeline into current LLMs.
Critical strengths
- Nice idea of intergating formal methods with compiler infrastructure for gpu kernel agents. The paper presents a compelling idea of exposing compiler IR and hardware-specific semantics to theorem-proving tools, allowing agents to reason about issues such as races, synchronization, and layout constraints rather than relying only on testing.
- Evaluation on realistic workloads. The system is tested on real model workloads and standalone kernels, with end-to-end performance measurements and a dedicated correctness suite, making the evaluation more practical.
Critical weaknesses
I appreciate the opportunity to review this paper, and I find the overall direction interesting. However, I believe the work still has substantial room for improvement as an ICLR paper.
-
The novelty of introducing theorem proving into a tile-based GPU DSL/compiler stack is somewhat limited. To my knowledge, TileLang has already been progressively integrating Z3 since late 2025 to prove properties of symbolic expressions and check hardware-related constraints such as TMA legality. Therefore, I think the paper should more clearly articulate what is fundamentally new about Glueball beyond exposing existing compiler/hardware constraints to an LLM-accessible theorem prover. In my view, the more interesting novelty may lie in how theorem proving is integrated into the agentic optimization loop, rather than in the use of Z3 itself.
-
The presentation, particularly in the evaluation section, needs significant improvement. One concrete issue is the use of the term baseline. Figure 3 uses the reference Triton implementation as the performance baseline, whereas Figure 4 uses “Baseline” to refer to the same LLM without the full Glueball correctness pipeline. These are very different notions of baseline, but the distinction is not made sufficiently explicit, which made the evaluation unnecessarily difficult to follow.
-
The evaluation does not sufficiently compare against strong contemporary coding-agent workflows. Modern coding agents can already generate and iteratively debug reasonably strong GPU kernels when equipped with execution, profiling, and compiler feedback. While adding theorem-prover feedback is intuitively well motivated and the paper demonstrates that it can help, the current evaluation mainly compares Glueball against variants of the authors’ own workflow with parts of the correctness pipeline removed. I would like to see a comparison against a strong general-purpose coding-agent harness, e.g., GPT-6 Astra with Codex, in terms of both final performance/correctness and total inference or api cost. This would help establish the incremental value of theorem proving beyond what a strong execution-driven agent can already achieve.
-
The system overhead of formal reasoning is not sufficiently characterized. Introducing symbolic execution, Z3 queries, additional verification agents, and counterexample synthesis may significantly increase the time and token cost required to obtain a valid optimized kernel. Although the appendix reports overall wall-clock time and token consumption, it does not isolate the overhead attributable to the verification machinery. A useful evaluation would report metrics such as the fraction of runtime/token cost spent on theorem proving/LLM response/compilation.
-
The performance results are not yet compelling enough. For example, all tested agents fail to outperform the Triton baseline on Mamba-2, with the best results remaining around parity or slightly below it. The paper itself attributes this to the difficulty of improving an already strong software-pipelined Triton schedule. I would encourage the authors to further strengthen the optimization results and evaluate on a broader range of GPU architectures. Currently, all experiments are conducted on a single NVIDIA DGX Spark / GB10 platform.
Questions for authors
- Are all formal proofs currently restricted to individual symbolic regions? How are these regions partitioned in practice, and approximately how many symbolic regions does a typical kernel contain?
- In the evaluated cases, what evidence do you have that it is sufficient to establish whole-kernel correctness rather than only local correctness within each symbolic region?
6 comments
4. System overhead. We agree that the overhead of formal reasoning should be characterized separately from the overall optimization cost. We have therefore broken down both token consumption by agent stage and tool usage by category across the four evaluated models.
The correctness pipeline, which includes the symbolic execution and Z3 tools, accounts for between 31.8% and 68.2% of total token consumption, depending on the model.
| Model | Correctness pipeline | Optimizer | Documentation agent | Setup, profiling, translation |
|---|---|---|---|---|
| DeepSeek V4 Pro | 33.1% | 33.9% | 13.8% | 19.2% |
| DeepSeek V4 Flash | 56.5% | 27.6% | 5.4% | 10.5% |
| Meta Muse Spark | 31.8% | 31.1% | 9.1% | 28.0% |
| GPT-6 Sol | 68.2% | 22.0% | 0.0% | 9.8% |
Theorem-prover tools account for a relatively small proportion of total tool calls. The following table breaks down tool usage between symbolic execution/Z3, shell and CLI operations, and other general tools:
| Model | Total calls | Symbolization / Z3 | Shell / CLI | Other general tools |
|---|---|---|---|---|
| DeepSeek V4 Pro | 17,311 | 136 (0.8%) | 9,624 (55.6%) | 7,551 (43.6%) |
| DeepSeek V4 Flash | 34,338 | 411 (1.2%) | 23,682 (69.0%) | 10,245 (29.8%) |
| Meta Muse Spark | 6,599 | 215 (3.3%) | 3,315 (50.2%) | 3,069 (46.5%) |
| GPT-6 Sol | 3,191 | 653 (20.5%) | 899 (28.2%) | 1,639 (51.4%) |
These results show that, although the correctness pipeline accounts for a substantial proportion of LLM inference, the actual theorem-prover interactions require relatively few tool calls for most models. We will include these breakdowns in the revised appendix to provide a clearer characterization of the system's inference and tool-use overhead. Percentages may differ slightly from 100% due to rounding.
5. Performance results and hardware coverage. We believe there are two important aspects of our evaluation that should be considered when interpreting the performance results. Firstly, while much of the existing kernel-generation literature evaluates against PyTorch eager or Inductor implementations, Glueball is evaluated against human-written and expert-tuned Triton kernels, a substantially stronger baseline. Secondly, our objective is to optimize entire model architectures rather than isolated kernels. Consequently, some seemingly modest end-to-end improvements are actually driven by much larger speedups in individual kernels, whose contributions are diluted at the model level because they account for only a fraction of total execution time. Regarding the Mamba-2 experiment, we acknowledge that none of the evaluated agents, including GPT-6 Astra with Codex, substantially outperforms the Triton reference.
We acknowledge that our current evaluation is limited to a single NVIDIA DGX Spark / GB10 platform, and will make this limitation explicit, identifying evaluation across additional GPU architectures as a direction for future work.
6. Symbolic-region partitioning. The equivalence checks operate over individual symbolic regions rather than a single monolithic representation of the entire kernel. The number of regions depends on kernel complexity: across our evaluations, this ranges from just 1 symbolic region for very simple kernels within models to over 15 regions for complex kernels like those involved in Mamba-2. The median number of regions across all experiments and models was 8.
7. Whole-kernel correctness. We would like to clarify that Glueball does not attempt to construct a single formal proof of whole-kernel equivalence by composing the individual symbolic-region proofs. Rather, our objective is to find a concrete execution or input that demonstrates inequivalence, with region-level checks forming part of a broader counterexample-search process.
Thank you for clarifying the positioning and contributions of Glueball. I now have a better understanding of the intended scope of the work, and I will raise my rating to Weak Reject.
My remaining suggestions are mainly about strengthening the evaluation:
-
Support for more advanced asynchronous architectures. GB10 is a relatively simple, edge-oriented SM120 architecture. Compared with SM80, its main additional asynchronous capability relevant to this work is TMA. I encourage the authors to further strengthen the paper by evaluating Glueball on at least one architecture that supports asynchronous MMA instructions, such as Hopper. This would help demonstrate the generality of the proposed approach across more sophisticated asynchronous execution mechanisms.
-
Per-case API cost for coding-agent baselines. For the comparison with baseline coding agents, I suggest reporting the API cost for each test case. This information would make the comparisons more informative.
-
Model workloads performance. I do not consider the current model workloads experiment to be well suited to substantiating the paper's claims. I nevertheless appreciate the authors' transparency in reporting and explaining the lack of performance improvements. I suspect that the main reason may be the choice of GB10 as the evaluation platform, where the optimization opportunities are relatively limited. Therefore, I would again encourage the authors to strengthen the evaluation by considering architectures with more exloring space for agents, as suggested in Point 1.
I hope these suggestions will be helpful in further strengthening the paper.
We sincerely thank you for your careful review.
1. Novelty and relationship to TileLang. We respectfully disagree that TileLang's integration of Z3 is directly comparable to Glueball. The use of Z3 in TileLang is primarily for compiler-side reasoning about symbolic expressions and hardware constraints, supporting specific compiler transformations and legality checks. Glueball uses theorem provers for a fundamentally different purpose: providing an LLM agent with an interactive means of finding counterexamples to the correctness of its generated optimizations. As illustrated in Figure 2 of the paper, our correctness pipeline explicitly searches for inequivalence between reference and candidate kernels, while Figure 5 demonstrates how the agent uses Z3 to identify an asynchronous race and derive a concrete execution witness. Crucially, this verification feedback is integrated into the iterative optimization loop (Figure 1), allowing counterexamples to guide subsequent kernel revisions. We believe this integration, rather than simply exposing Z3 to an LLM, is a central contribution of Glueball.
We'd also like to draw attention to the fact that verification can be approached from two complementary perspectives: attempting to prove that a program is correct for all possible executions, or attempting to demonstrate that it is incorrect by finding a counterexample. Glueball deliberately adopts the latter. Rather than requiring a complete formal model of every aspect of the compiler and hardware, we use targeted formal models to search for concrete violations of equivalence and hardware properties. If no counterexample is found by the verification pipeline, Glueball accepts the candidate as correct within the scope of its checks. We believe this approach is particularly valuable for GPU optimization, where hardware evolves rapidly and constructing complete formal models of new features is both difficult and sometimes impossible given the limited availability of hardware specifications.
2. Presentation and baseline terminology. We agree that the use of the term baseline in Figures 3 and 4 is ambiguous. In Figure 3, the reference Triton implementation is the performance baseline against which speedups are measured. In Figure 4, the baseline refers to the same LLM agent operating without Glueball's correctness pipeline. We will distinguish these explicitly throughout the paper, referring to them as the Triton reference and LLM-only (without correctness pipeline), respectively. We will also revise the figure captions and evaluation text to make the experimental comparisons clearer.
3. Comparison against contemporary coding agents. We agree that comparison against strong general-purpose coding agents is important to establish the incremental value of Glueball. We have therefore conducted an additional comparison against GPT-6 Astra with Codex, allowing it to use the available command-line tools for compilation, execution, profiling, and debugging, without Glueball's formal verification tools. We evaluated both approaches on the same six optimization workloads. The following table reports speedups relative to the reference Triton implementations:
Notably, even a substantially cheaper model such as DeepSeek V4 Pro, when equipped with Glueball, achieves greater or comparable overall performance to GPT-6 Astra with Codex. These results demonstrate the value of the specialized optimization workflow, rather than relying solely on the capabilities of the underlying foundation model. We will include these comparison results in the final version of our paper.