acceptodds
Desk rejected at ICLR 2027

PTXBench: Interpretable Evaluation of LLMs for Architecture-Specific GPU Programming

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Reaching peak performance on modern GPUs requires specialized hardware-accelerated mechanisms. Writing kernels that use them is therefore a key capability for LLMs. Yet correctness and speedup alone cannot show whether a generated kernel uses the intended hardware mechanisms. We propose instruction-qualified speedup, which counts a kernel's speedup only if the kernel is correct and executes selected tensor-compute or data-movement instructions. Our framework, PTXBench, measures this metric while controlling four conditions: supplied architectural knowledge, execution feedback, harness capabilities, and generation budgets. Ablations show that supplied architectural knowledge and execution feedback improve instruction-qualified speedup. Using a shared multi-turn refinement harness, we evaluate five recent LLMs on one GEMM and four attention workloads across NVIDIA H100 and B200. Correctness is not enough. Even with full knowledge and execution feedback, 25.9% of trajectories that produce a correct kernel never produce an instruction-qualified one. In all ten GPU-workload settings, the fastest CUDA-PTX kernels execute both accelerated compute and data movement. Still, no model’s median instruction-qualified speedup reaches the vendor-library reference. Under the same budget, evaluated LLMs reach higher median instruction-qualified speedups with Triton than with CUDA-PTX, even without Triton's autotuning.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
GPU kernel agents, environment and benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JBRdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces PTXBench, a framework for evaluating LLMs on architecture-specific GPU programming in CUDA with inline PTX. It proposes instruction-qualified speedup: a kernel's speedup counts only if the kernel is correct and executes targeted hardware-accelerated instructions at runtime (e.g., wgmma/TMA on H100, tcgen05/TMEM on B200), verified via SASS execution counts from Nsight Compute. The framework also fixes the generation conditions: supplied architectural knowledge, execution feedback, harness, and budget. Experiments cover one BF16 GEMM and four attention workloads on H100 and B200. Ablations with Gemini show that supplied knowledge and execution feedback raise instruction-qualified speedup, and that simple multi-turn refinement is competitive with coding agents. Across five LLMs, 25.9% of trajectories with a correct kernel never use the target instructions, no model's median reaches cuBLAS/cuDNN, and Triton outperforms CUDA–PTX in all settings.

Critical strengths

1. Rigorous and reproducible measurement. Instruction execution is verified from runtime SASS counters rather than source matching. This makes the check hard to game and applicable across languages, and the released artifacts regenerate all results. Even though I question the metric's necessity, the reported numbers are trustworthy and reusable.

2. Useful findings on evaluation conditions. Supplied knowledge strongly changes model behavior (models almost never use wgmma without PTX wrappers), and model rankings vary sharply across workloads. Kernel-generation evaluations should therefore report and control these conditions.

Critical weaknesses

1. The metric is ill-founded in principle. Using specific instructions is a means, not the goal. If instruction use correlates with performance, the gate is redundant, and the paper's own data shows this (leaders unchanged, L356–357). If it does not correlate, the gate penalizes better solutions. This is common in practice: in low-latency decoding kernels (small batch, memory-bound), legacy mma or CUDA-core paths can outperform wgmma/tcgen05, yet such kernels would score zero. Even within this paper, a correct 0.41× kernel without target instructions scores 0 (Fig. 4, H100 MHA-Fwd), while a 0.02× kernel that executes a single wgmma scores higher. Because the target instructions are fixed in advance by experts, the metric rewards imitating a known solution rather than optimizing. Instruction use would be useful as a separate diagnostic, but folding it into the score is not justified.

2. The metric adds little beyond plain speedup. In my experience, agents given sufficient spec knowledge find the fast instructions on their own. The paper's own results agree: the leaders are unchanged when non-qualified kernels are allowed (L356–357), and all kernels ≥0.5× already qualify (Table 4). Combined with the small scope (five workloads, one shape each, few repeats per setting), the main contribution is not substantial enough for ICLR.

Questions for authors

See my concerns on the motivation, that's the major question I have for this paper.

4 comments

We thank the reviewer for the detailed critique, and for noting that the measurement is rigorous and reusable. New workloads and results are in the shared comment.

Justification for the metric

  1. "Using specific instructions is a means, not the goal."

Instruction qualification measures a specific capability: whether an LLM uses the hardware mechanisms that raise the performance ceiling on new GPUs, as expert kernels do. Speedup only tells how fast a kernel is now; qualification tells whether it is on the path to the higher ceiling. In compute-bound workloads (GEMM, FP8-GEMM, Conv2D and attention have arithmetic intensity 3.5–9× the GPUs' roofline ridge point), the target instructions set that ceiling:

  • Compute (C). Without wgmma/tcgen05.mma, a kernel must use legacy mma.sync, which we measured at 0.67× the wgmma rate per SM clock on H100 and 0.25× the tcgen05.mma rate on B200. However well tuned, a BF16 kernel without C cannot exceed 782 TFLOPS on H100 or 675 TFLOPS on B200, even adding full FFMA throughput at peak boost clock; with C the ceilings are 989 and 2,250 TFLOPS. FP8 has no legacy tensor-core path; the compiler emulates FP8 mma.sync with FP16 instructions. mma.sync is also synchronous, whereas wgmma is asynchronous, which lets CUDA core execution overlap with the matmul.
  • Data movement (D). TMA is a dedicated unit for asynchronous global-to-shared copies, issued by a single thread, so the other warps and most registers stay with the matmul.
  • Experts use both. Every compute-bound reference (cuBLAS, cuDNN, DeepGEMM) runs all required matmul FLOPs on C, executes zero legacy HMMA (mma.sync), and moves 88–100% of its L2→SM load bytes with TMA on both GPUs.
  • Experts without them fall short. FlashAttention-2 (mma.sync + cp.async) reaches 0.60× (forward), 0.62× (causal forward) and 0.62× (backward) of cuDNN in FlashAttention-3's H100 benchmark (batch 4, sequence length 4k, head dim 128, as in our workloads; Figs. 5–6). FlashAttention-3, using both, reaches 1.05–1.12× across the same three settings.

2. The correlation dilemma. Instruction qualification and performance can align or diverge. Our objective is to measure LLM’s ability to use target mechanisms and the performance achieved while meeting that requirement. Instruction-qualified speedup is therefore a capability-specific measurement, while ordinary speedup remains necessary for comparing performance. In the paper's compute-bound workloads, correct non-qualified trajectories drop from 116 to 8 at 0.25× and to 0 at 0.50× (Table 4). In the HBM-bound RMSNorm, non-qualified kernels come close, reaching 0.93× (H100) and 0.97× (B200) of QuACK, while the fastest kernels are qualified (0.97× and 1.06×), using distributed shared memory (DSMEM) as QuACK does. For Gemini 3.1 Pro on B200, the non-qualified CUDA–PTX median is even higher (0.86× vs. 0.77×). Qualification thus does not always agree with speedup, so it is not redundant. Where they disagree, we report both speedups side by side, as the reviewer suggests, so no faster kernel is hidden.

3. Low-latency decoding. Qualification is C ∨ D, not C alone. A memory-bound decode kernel that computes with mma.sync or CUDA cores but moves data through TMA or DSMEM qualifies. One using neither is the RMSNorm case above: we report its speedup next to the qualified one, and the gap between them.

4. The 0.41× vs. 0.02× example. The two kernels come from different settings: the 0.41× kernel is H100 MHA-Fwd, and the 0.02× kernel is B200 GEMM. The 0.41× kernel executes ¬C ∧ ¬D, and even FlashAttention-2, an expert ¬C ∧ ¬D design, reaches only 0.60× of an cuDNN at our shape (Point 1). Instruction-qualified speedup answers a deeper question than plain speedup: what performance does the model achieve while correctly exercising at least one targeted mechanism? The 0.02× kernel is an early step on the higher-ceiling path. In the same trajectory, the next turn reaches 0.49× with C ∧ D, and the final turn reaches 0.52× with C ∧ D, 2.5× the best LLM-generated B200 GEMM kernel with ¬C ∧ ¬D (0.21×). Figure 4 shows ordinary speedup next to the qualified score, so both readings are visible.

  1. "...the metric rewards imitating a known solution rather than optimizing."

Qualification does not fix the algorithm, schedule, tiling or pipelining. Expert baselines converge on these instruction classes because they are the hardware's fast paths, not because one expert design is copied. The refinement loop does not directly reward instruction use: its feedback reports correctness and latency.

What does our metric add beyond plain speedup?

That leaders are unchanged (L356–357) and all kernels ≥0.50× qualify (Table 4) is what Point 1 predicts: in compute-bound workloads, fast kernels need the target instructions. Qualification adds the diagnosis behind it: 25.9% of correct trajectories never use them and stay below 0.50×. It also makes "sufficient spec knowledge" measurable: as the reviewer notes (Strength 2), models almost never execute wgmma without supplied PTX wrappers, which speedup alone cannot attribute.

On the number of repeats: each Table 5 cell has 12 trajectories, and our reply to Reviewer AzfQ (Q1) shows every Table 5 conclusion survives leave-one-trajectory-out resampling. The new workloads in the shared comment also use 12 per cell, and the paper's conclusions hold on them.

I thank the authors for the detailed response and the new workloads. However, my main concern is not addressed. The ceiling argument shows that the target instructions are necessary for peak performance on compute-bound workloads, but this is exactly why plain speedup already captures their benefit there, consistent with the leaders being unchanged. Where instruction use and performance diverge, the new RMSNorm results confirm my concern. For Gemini on B200, the non-qualified CUDA–PTX median (0.86×) exceeds the qualified one (0.77×). Triton kernels reaching 0.67–0.94× of QuACK score zero in most cells because Triton cannot emit DSMEM accesses. The qualification set also had to be extended post hoc to match the expert baseline. This suggests that the metric rewards matching expert instruction choices rather than performance.

I agree that "ordinary speedup remains necessary for comparing performance", and would encourage the authors to report speedup as the primary metric, with instruction qualification as a separate diagnostic. I keep my score.

We thank the reviewer for the careful follow-up. The reviewer finds the measurement rigorous and instruction qualification useful as a separate diagnostic; the remaining concern is whether it should be folded into the headline metric. The RMSNorm cases the reviewer cites, the Gemini B200 median and Triton's lack of DSMEM, are where reporting both matters, which is why we included them in our response. Even there, the fastest kernels on both GPUs use DSMEM (0.97× and 1.06× of QuACK, vs. 0.93× and 0.97× without it): qualification marks what reaches the ceiling, while ordinary speedup shows how far kernels get without it. On Table 1, the qualification set follows one fixed rule: the hardware mechanisms the expert baseline relies on. We added the new rows before scoring any LLM kernel, and they change no result in the paper. We are grateful to the reviewer's suggestion. In the revision, every main table will report ordinary speedup and instruction-qualified speedup as co-primary metrics, with the qualification rate beside them.

Reviewer_cWPXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PTXBench, a benchmark for evaluating LLMs’ architecture-specific GPU programming capabilities. It proposes instruction-qualified speedup and investigates the effects of architectural knowledge, execution feedback, and evaluation harnesses.

Critical strengths
  1. Studying the correctness of the generated kernels on specific architectures is reasonable.

  2. The evaluation protocal is clearly established.

  3. The experiments include multiple LLMs, two GPU architectures, and different kernel languages, with detailed reproducibility information.

Critical weaknesses
  1. The significance of the proposed metric is discussed insufficiently. Although instruction qualification provides additional diagnostic information, the paper reports that it does not change the leading model in any of the ten settings. The authors should better clarify the practical value of qualification beyond existing speedup metrics.

  2. Restricted benchmark coverage. The evaluation focuses on one GEMM configuration and four attention variants, all using BF16 on NVIDIA GPUs. It remains unclear how well the conclusions generalize across operators, tensor shapes, numerical precisions, and hardware platforms.

Questions for authors

Please refer to Weakness.

2 comments

We thank the reviewer for finding the protocol clear and the experiments reproducible. New workloads and results are in the shared comment.

The practical value of qualification beyond existing speedup metrics

1. What does qualification tell us beyond speedup? Instruction qualification measures a specific capability: whether an LLM uses the hardware mechanisms that raise the performance ceiling on new GPUs, as expert kernels do. Speedup only tells how fast a kernel is now; qualification tells whether it is on the path to the higher ceiling. In compute-bound workloads (GEMM, FP8-GEMM, Conv2D and attention have arithmetic intensity 3.5–9× the GPUs' roofline ridge point), the target instructions set that ceiling:

  • Compute (C). Without wgmma/tcgen05.mma, a kernel must use legacy mma.sync, which we measured at 0.67× the wgmma rate per SM clock on H100 and 0.25× the tcgen05.mma rate on B200. However well tuned, a BF16 kernel without C cannot exceed 782 TFLOPS on H100 or 675 TFLOPS on B200, even adding full FFMA throughput at peak boost clock; with C the ceilings are 989 and 2,250 TFLOPS. FP8 has no legacy tensor-core path; the compiler emulates FP8 mma.sync with FP16 instructions. mma.sync is also synchronous, whereas wgmma is asynchronous, which lets CUDA core execution overlap with the matmul.
  • Data movement (D). TMA is a dedicated unit for asynchronous global-to-shared copies, issued by a single thread, so the other warps and most registers stay with the matmul.
  • Experts use both. Every compute-bound reference (cuBLAS, cuDNN, DeepGEMM) runs all required matmul FLOPs on C, executes zero legacy HMMA (mma.sync), and moves 88–100% of its L2→SM load bytes with TMA on both GPUs.
  • Experts without them fall short. FlashAttention-2 (mma.sync + cp.async) reaches 0.60× (forward), 0.62× (causal forward) and 0.62× (backward) of cuDNN in FlashAttention-3's H100 benchmark (batch 4, sequence length 4k, head dim 128, as in our workloads; their cuDNN version 9.1.1.17 is older than our v9.20.0 Figs. 5–6). FlashAttention-3, using both, reaches 1.05–1.12× across the same three settings.

2. Why are the leaders unchanged? Since fast compute-bound kernels need these instructions, the top of each ranking must agree with plain speedup; we read this as a consequence of the choice of targets. Instruction qualification adds information for kernels below the leaders, where speedup cannot tell kernels apart:

  1. Diagnosis. Even with full knowledge and execution feedback, 25.9% of trajectories that produce a correct kernel never produce a qualified one, and none of their kernels reaches 0.50× (Table 4). Speedup alone scoring would count them as partial successes, although their path has a lower ceiling.
  2. Refine or change approach. It separates kernels on the right path but not yet tuned from those on a lower-ceiling path, e.g., the correct 0.41× H100 MHA-Fwd kernel in Fig. 4 uses neither C nor D, the FlashAttention-2 design, which reaches only 0.60× of cuDNN at our shape in FlashAttention-3's benchmark. Instruction qualification can tell an agent whether to keep refining or switch.
  3. Attribution. It shows which conditions unlock the capability: without supplied PTX wrappers, models almost never execute wgmma, a difference invisible in speedup alone.
  4. Per-workload gap. It quantifies how much the target instructions matter. In the new HBM-bound RMSNorm, non-qualified kernels reach 0.93× (H100) and 0.97× (B200) of QuACK, close to the fastest qualified kernels (0.97× and 1.06×); for Gemini 3.1 Pro on B200 the non-qualified CUDA–PTX median is even higher (0.86× vs. 0.77×). Qualification therefore does not always agree with speedup, and we report both side by side.

Benchmark Coverage

We added two operators (Conv2D, RMSNorm), a precision (FP8-GEMM), and a second attention shape (MHA-Fwd/Bwd at head dim 96), each against the fastest expert baseline we found (DeepGEMM, cuDNN, QuACK). Details are in the shared comments.

On hardware, H100 and B200 have different C (wgmma vs. tcgen05.mma + TMEM), and the conclusions hold on both. Beyond NVIDIA, PTXBench needs runtime instruction-level evidence and an interface exposing the target instructions. Porting means defining C and D, a profiler backend, and the supplied knowledge; execution feedback, harness, and budget carry over unchanged.

Thanks for authors' response. I have no other questions.

Reviewer_AzfQOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

Correctness and speedup alone cannot show whether a generated kernel uses the intended hardware mechanisms. PTXBench counts a kernel's speedup only if the kernel is correct and executes the selected instructions at runtime. The authors map PTX interfaces to SASS instructions that must execute, and replay each correct kernel on the original workload under Nsight Compute. The authors checked the output targeted instructions with the LLM source inspection and targeted human review. The paper measures the speedup while controlling four conditions: supplied architectural knowledge (architecture parameters, plus PTX instruction wrappers, plus architecture contracts), execution feedback, harness capabilities (multi-turn refinement vs. coding agents), and generation or evaluation budgets. The paper evaluates five recent LLMs on one BF16 GEMM and four BF16 attention workloads across H100 and B200 against well-optimized cuBLAS and cuDNN implementations.

Critical strengths

The paper has the following strength:

This paper examines SASS (the actual executed code), not PTX source code, and uses a uniform approach to verify all languages ​​that can be reduced to PTX, thus translating an assumption into actual measurement results. Table 4 shows its importance: a quarter of the correct trajectories never touched any mechanism, and these paths were not fast. If judged solely by correctness+speedup, they would all be counted as success cases.

Critical weaknesses

The paper has the following weaknesses:

  1. Breadth. It only supports one precision, one GEMM shape, one attention family, and one vendor. The evidence supports a narrower conclusion than the framework suggests: current LLMs fall short on these five workloads under these conditions. Since the predicates are manually derived for each architecture, I don't know how this benchmark can be extended to convolution, reduction, or other vendors. And this is what the field mostly needs.

  2. Contribution Gap. The runtime counters certify execution, not usefulness, and according to the authors, speedup attribution for each instruction is future work. So the metric can still be gamed by a kernel that executes a target instruction, but the performance was not actually improved. This is a real gap for a paper aiming for interpretable evaluation, and I think a coarse stub-and-retime ablation could make up most of it.

Questions for authors
  1. Please confirm the per-cell count for Table 5 (I count 12). How stable are the conclusions under leave-one-trajectory-out resampling, especially on the B200 attention cells where several medians are zero?
  2. Autotuning searches many configurations inside one program, and the generation budget does not cap it. What were the effective GPU-side evaluation budgets across the Triton, Triton-without-autotune, and CUDA-PTX arms? Is the comparison compute-matched on the GPU side?
  3. Why accept C ∨ D? A kernel that uses TMA but computes with FFMA loops would qualify. Did such cases appear, and how were they scored? (Table 4 suggests D-only was rare, but the choice deserves a sentence.)
  4. Would a stub-and-retime ablation be feasible from the recorded NCU artifacts? That would turn the contribution caveat into a measurement.
8 comments

We thank the reviewer for the careful reading and for noting that checking SASS turns an assumption into a measurement. On breadth, we added Conv2D, RMSNorm (a reduction), FP8-GEMM, and head-dim-96 attention (see shared comment). Beyond NVIDIA, PTXBench needs runtime instruction-level evidence and an interface exposing the target instructions. Porting means defining C and D, a profiler backend, and the supplied knowledge; execution feedback, harness, and budget carry over unchanged. We also take your presentation score seriously and will improve the paper for readability, defining notation and the qualification rule.

Q1: Leave-one-trajectory-out resampling

Yes, we confirm that each Table 5 cell has 12 trajectories (3 scheduling-guidance variants × 4 repeats). Every Section 4.4 conclusion drawn from Table 5 survives leave-one-trajectory-out (LOTO) resampling; one numerical bound changes: the best B200 attention median goes from 0.28× (paper) to ~0.30× (LOTO).

LOTO Method. Sort a cell's 12 endpoints as x1≤⋯≤x12x_1 \le \dots \le x_{12}; the full-data median is (x6+x7)/2(x_6 + x_7)/2. Dropping one trajectory leaves 11, whose median is x7x_7 if the dropped one is among x1x_1–x6x_6 and x6x_6 otherwise. So every LOTO median is x6x_6 or x7x_7, and checking a conclusion at these two endpoints is exhaustive. We check each conclusion against the worst-case endpoint of all models simultaneously, which also covers any combination of single-trajectory removals across models.

Below C1-C7 are Section 4.4 conclusions drawn from Table 5.

C1: "Claude Opus 4.8 has the highest median on GEMM for both GPUs and both H100 forward-attention workloads; GPT-5.6 Sol leads the remaining settings." Holds. A leader can change only if its x6x_6 falls below another model's x7x_7. In all 10 settings it does not; the smallest LOTO margin is 0.021× (H100 MHA-Fwd). Δ = LOTO margin − Table 5 margin. Margins are computed before rounding.

GPUWorkloadLeaderLeader x6x_6Best other x7x_7LOTO margin (Δ)
H100GEMMClaude0.9140.758 (Gemini)0.156 (−0.008)
MHA-FwdClaude0.7470.725 (GPT)0.021 (−0.017)
MHA-Fwd-CausalClaude0.7030.662 (GPT)0.041 (−0.010)
MHA-BwdGPT0.5470.274 (Gemini)0.274 (−0.050)
MHA-Bwd-CausalGPT0.3730.332 (Gemini)0.040 (−0.074)
B200GEMMClaude0.9390.670 (GPT)0.269 (−0.031)
MHA-FwdGPT0.2490.104 (Claude)0.144 (−0.007)
MHA-Fwd-CausalGPT0.2610.121 (Claude)0.140 (−0.020)
MHA-BwdGPT0.1540.0000.154 (−0.004)
MHA-Bwd-CausalGPT0.1520.0000.152 (−0.014)

C2: "On H100 MHA-Bwd-Causal, Claude is far below GPT (0.03 vs. 0.41)." Holds. Claude's median stays in [0.000, 0.052] and GPT's in [0.373, 0.454], a gap of at least 0.32×.

C3: "No model's median instruction-qualified speedup reaches its vendor-library reference; the highest median is 0.94× (Claude, B200 GEMM)." Holds. The largest x7x_7 across all 50 cells is 0.946 (Claude, B200 GEMM), below 1.0×, and that cell's median stays in [0.939, 0.946], above every other cell's x7x_7.

C4: "The best median on every B200 attention workload is at most 0.28×." Holds after rounding up. The largest x7x_7 among B200 attention cells is 0.299 (GPT, MHA-Fwd-Causal; full-data 0.280). We will write "at most 0.30×".

C5: "Gemini's GEMM median drops from 0.76× on H100 to 0.51× on B200." Holds. The ranges are [0.753, 0.758] on H100 and [0.508, 0.513] on B200, so the drop is at least 0.240×.

C6: "Gemini beats GLM-5.2 and Qwen3.8-27B on every H100 workload." Holds. On every H100 workload Gemini's x6x_6 exceeds both models' x7x_7; the smallest margin is 0.139× (GEMM, over Qwen). On attention, GLM and Qwen are 0 at both endpoints.

WorkloadGemini x6x_6Best GLM/Qwen x7x_7LOTO margin (Δ)
GEMM0.7530.614 (Qwen)0.139 (−0.013)
MHA-Fwd0.5090.0000.509 (−0.031)
MHA-Fwd-Causal0.4620.0000.462 (−0.011)
MHA-Bwd0.2640.0000.264 (−0.005)
MHA-Bwd-Causal0.2650.0000.265 (−0.034)

C7: zero-median cells, including the B200 attention cells you highlight. All 21 zero-median cells in Table 5 stay at zero under LOTO: each has at least 7 zero trajectories, so x6=x7=0x_6 = x_7 = 0. A 12-trajectory median is positive only if at most 6 are zero, so a cell with k zeros would need k − 6 of them to succeed. Of the 13 zero-median B200 attention cells, 12 have 10–12 zeros (4–6 more successes needed); only Gemini 3.1 Pro on MHA-Fwd-Causal has 7 (one needed). The other 8 are GLM-5.2 and Qwen3.8-27B on H100 attention, with 8–12 zeros (2–6 needed). The B200 attention zeros thus reflect consistent failure across trajectories, not one or two unlucky runs.

Q2: GPU-side evaluation budget

Triton without autotuning and CUDA–PTX share the same budget and use one config per request on the GPU side; standard Triton may search more configurations per request. but its extra search is small in wall-clock time compared to LLM API calls.

  1. Budgets per arm. Every turn sends one evaluation request. In CUDA–PTX and Triton without autotuning, that request compiles and times exactly one kernel configuration. In standard Triton, a correct turn in Table 6 evaluates 6.7 configurations averaged over both GPUs.
  2. The search adds little time. On H100, autotuning adds an estimated 15.1 s per correct turn, 7.7% of the 197 s average LLM API time per turn. The configuration search overhead depends on how many configurations a model writes: Gemini 3.1 Pro adds 23.6 s (9.5 configurations per turn; 11.2% of its 212 s), GPT-5.6 Sol adds 5.8 s (3.3 configurations; 3.2% of its 181 s) on average.
  3. Autotuning search does not affect kernel latency measurement. The search finishes before the CUPTI timing window opens.
  4. The matched comparison supports the claim. Triton without autotuning has a higher median instruction-qualified speedup than CUDA–PTX in all 20 model-GPU-workload settings of Table 6, so Triton's advantage does not come from extra GPU-side search.

Counting configurations

A turn's configuration count is the length of the configs list passed to each @triton.autotune, summed over the autotuned kernels the program launches. Many kernels build this list programmatically (e.g., nested loops over tile sizes and stages), so we execute each module with a recording stub instead of counting triton.Config occurrences in the source.

Estimating the search time

We replayed 62 Triton kernels from Table 6 through the original evaluation flow twice: once as recorded (A), and once with every autotuner pinned to the configuration A selected (B). A − B is the cost of the search alone.

Within a workload, search time grows roughly linearly with the configuration count (Pearson r = 0.83–0.94), so we summarize each workload by its cost per configuration (total search time / total configurations over its sampled kernels): 1.0 s for GEMM, 2.0 s for MHA-Fwd-Causal, 2.8 s for MHA-Fwd, 5.5 s for MHA-Bwd-Causal, and 7.6 s for MHA-Bwd. For each of the 641 correct H100 standard-Triton turns, the added time is this cost times the turn's configuration count, or 0 s without autotuning. This gives the averages in Item 2 above, which we divide by the average LLM API time per correct turn in the same setting.

We stratified the sample by configuration count with a fixed seed: 10 kernels per bin up to 32 configurations, and every kernel above 32. A proportional draw would concentrate in the 5–12 bins and leave the expensive tail almost empty. The replayed kernels are listed here. Distribution over all correct H100 standard-Triton kernels:

ConfigsKernels (Gemini / GPT)Sampled (Gemini / GPT)
1 (no autotuning)64 / 2270 / 0
2–422 / 37 / 3
5–8129 / 276 / 4
9–1284 / 385 / 5
13–1610 / 124 / 6
17–3212 / 19 / 1
33–647 / 07 / 0
65–2565 / 05 / 0
Total333 / 30843 / 19

Q3: Why accept C ∨ D?

Some workloads need only one target family, so requiring C ∧ D would reject even expert kernels. RMSNorm, added in the rebuttal, has no matmul work for tensor cores; its expert reference QuACK, the fastest library kernel we measured, qualifies only through D. C ∨ D lets one rule cover both compute-bound and memory-bound workloads.

A TMA + FFMA kernel would indeed qualify, but qualification is only a gate: the score is still speedup, which is capped by CUDA-core throughput for such a kernel, and Figure 4 reports correct kernels' C/D category, so a partly qualified kernel stays visible. Moreover, no such case appeared in the paper's corpus: we observed no LLM-authored correct kernel pairs TMA with FFMA-only compute. The corpus's ¬C∧D kernel uses TMA + mma.sync, which reaches 0.18× (Figure 4). Its TMA does real work: it moves 100% of the kernel's global-load bytes, and replacing it with cp.async slows the kernel by 2.32× (see our Q4 reply). It is scored as instruction-qualified like any C ∨ D kernel, but a faster kernel is best in its trajectory, so it changes no number in Tables 4 and 5. We will add this justification after the qualification rule.

Comments on the paper

3 comments

We thank the reviewers for recognizing PTXBench's rigor and reproducibility. Following the suggestions to cover more precisions, shapes, convolution, and reduction, we added FP8-GEMM, MHA-Fwd-d96, MHA-Bwd-d96, Conv2D, and RMSNorm, tested on Gemini 3.1 Pro and GPT-5.6 Sol, the two models in the paper's language comparison. Main conclusions:

  1. These new experiments strengthen the paper's point that using the right instructions is necessary but not sufficient to match experts: no model's median instruction-qualified speedup reaches its expert baseline.
  2. As argued in the paper, instruction qualification reveals more than speedup alone: it quantifies how much the target instructions matter and how well LLMs use them. In compute-bound workloads, no LLM-authored non-qualified kernel reaches 0.50× as in the paper. In the HBM-bound RMSNorm, the gap is smaller, but the fastest kernels are still qualified.

As in the paper, we compare against well-optimized expert implementations: DeepGEMM for FP8-GEMM (SGLang's default block-scaled FP8 backend), cuDNN for Conv2D and MHA-d96 (cuDNN's flagship operators), and QuACK for RMSNorm (fastest we measured; the next best library reaches only 0.65×/0.69× on H100/B200).

We follow the protocol of Table 6: full architectural knowledge, execution feedback, and multi-turn refinement with up to 8 attempts. We run 12 trajectories per cell and report the per-trajectory best Inst. Speedup as median (min–max):

GPUModelLanguageFP8-GEMMRMSNormConv2DMHA-Fwd-d96MHA-Bwd-d96
H100Gemini 3.1 ProCUDA–PTX0.39 (0.31–0.61)0.86 (0.73–0.89)0.64 (0.49–0.94)0.59 (0.00–0.87)0.22 (0.00–0.54)
Triton0.71 (0.64–0.79)-- (0.00–0.63)0.84 (0.74–0.98)0.83 (0.76–0.86)0.60 (0.47–0.76)
GPT-5.6 SolCUDA–PTX0.57 (0.44–0.68)0.88 (0.76–0.97)0.79 (0.61–0.88)0.84 (0.63–1.02)0.50 (0.28–0.81)
Triton0.70 (0.64–0.73)--0.95 (0.65–0.98)0.98 (0.90–1.04)0.52 (0.43–0.79)
B200Gemini 3.1 ProCUDA–PTX0.42 (0.30–0.56)0.77 (0.53–0.97)0.68 (0.45–0.81)-- (0.00–0.15)--
Triton0.47 (0.00–0.54)-- (0.00–0.77)0.70 (0.60–0.84)0.47 (0.38–0.51)0.37 (0.26–0.45)
GPT-5.6 SolCUDA–PTX0.46 (0.28–0.71)0.91 (0.52–1.06)0.88 (0.50–0.95)0.24 (0.10–0.40)0.16 (0.00–0.38)
Triton0.53 (0.45–0.54)--0.83 (0.73–0.85)0.43 (0.33–0.65)0.36 (0.26–0.48)

Bold: better language per cell in the above table; best median and max per GPU–model below. --: median 0 (no range: all 12 trajectories are 0). Triton's RMSNorm entries are mostly 0 because Triton never emits DSMEM accesses; only three Gemini Triton kernels qualify, all through TMA.

On RMSNorm, non-qualified kernels reach 0.93× (H100) and 0.97× (B200), while qualified kernels remain the fastest (0.97× and 1.06×). Per trajectory, we report the best correct qualified (Inst.), non-qualified (Non-Inst.), and overall kernel:

GPUModelLanguageInst. SpeedupNon-Inst. SpeedupOverall Speedup
H100Gemini 3.1 ProCUDA–PTX0.86 (0.73–0.89)0.67 (0.00–0.68)0.86 (0.73–0.89)
Triton-- (0.00–0.63)0.74 (0.66–0.75)0.74 (0.66–0.75)
GPT-5.6 SolCUDA–PTX0.88 (0.76–0.97)0.39 (0.00–0.66)0.88 (0.76–0.97)
Triton--0.67 (0.64–0.93)0.67 (0.64–0.93)
B200Gemini 3.1 ProCUDA–PTX0.77 (0.53–0.97)0.86 (0.80–0.94)0.88 (0.81–0.97)
Triton-- (0.00–0.77)0.94 (0.82–0.96)0.94 (0.82–0.96)
GPT-5.6 SolCUDA–PTX0.91 (0.52–1.06)0.78 (0.00–0.97)0.92 (0.83–1.06)
Triton--0.80 (0.59–0.89)0.80 (0.59–0.89)

Since the expert baselines use new instructions, we add rows to Table 1; a kernel qualifies for a class if it matches any of its rows, existing or new:

GPUClassPTX interfaceExecuted SASS
H100Cwgmma.mma_async (FP8, INT8, binary)QGMMA, IGMMA or BGMMA
B200Ctcgen05.mma (FP8 / block-scaled, INT8, FP4) + tcgen05.ld(UTCQMMA, UTCIMMA or UTCOMMA) ∧ (LDTM or LDT)
H100, B200DDSMEM access (barrier.cluster)UCGABAR

We add UCGABAR to D to cover distributed shared memory (DSMEM), which RMSNorm's baseline uses. The CUDA Programming Guide prescribes a cluster barrier (barrier.cluster) before a CTA accesses a peer CTA's shared memory, to ensure the peer is running. After inspection, we confirm that all correct RMSNorm kernels that execute UCGABAR transfer useful data through DSMEM. No correct non-qualified kernel in the paper executes a newly added SASS instruction, so no paper result changes.

All LLM-generated kernels for the five new workloads, along with the workload definitions, per-response results, and the script that reproduces the tables above, are available here and in the supplementary material, in case Anonymous GitHub's server is down.

Dear Area Chairs and Reviewers,

We thank all reviewers for their careful and constructive feedback, and Reviewer AzfQ for raising the score. We addressed the reviewers' concerns in the following ways.

  1. Benchmark coverage (AzfQ, cWPX, JBRd). We added five workloads: FP8-GEMM, Conv2D, RMSNorm, and MHA-Fwd/Bwd at head dim 96, evaluated on Gemini 3.1 Pro and GPT-5.6 Sol against DeepGEMM, cuDNN, and QuACK. PTXBench now spans GEMM, attention, convolution, and reduction in BF16 and FP8. The results support the paper's main finding: no model's median instruction-qualified speedup reaches its expert baseline, and in compute-bound workloads no non-qualified kernel reaches 0.50×. For the new baselines, we added Table 1 rows (FP8 tensor-core instructions and DSMEM) before scoring any LLM kernel; they change no result in the paper. Porting to other hardware requires defining C and D, a profiler backend, and the supplied knowledge, while harness and budget carry over unchanged.

  2. What instruction-qualified speedup adds beyond plain speedup (cWPX, JBRd). Instruction qualification measures whether an LLM uses the hardware mechanisms that raise the performance ceiling on new GPUs: speedup tells how fast a kernel is now; qualification tells whether it is on the path to the higher ceiling. The leaders match plain speedup because fast compute-bound kernels need these instructions; below them, 25.9% of trajectories with a correct kernel never produce a qualified one, and none reaches 0.50×. Without supplied PTX wrappers, models almost never execute wgmma, a difference invisible in speedup alone. Where the two metrics diverge (in RMSNorm), the fastest kernels still qualify (0.97×/1.06× vs. 0.93×/0.97× of QuACK). Following JBRd's suggestion, every main table will report both as co-primary metrics.

  3. Whether the target instructions drive performance (AzfQ). As the reviewer suggested, we ran a stub-and-retime ablation on the fastest kernel of each C/D category in Figure 4, with NCU workload accounting. (1) They drive performance. Replacing TMA with cp.async slows the leaders by 2.61× (H100) and 2.77× (B200) in geometric mean, and replacing wgmma with mma.sync slows H100 leaders by 1.98× (C∧D) and 1.57× (C∧¬D). Every stubbed leader slows by at least 1.50×. (2) They do the compulsory work. The C path executes at least 100% of the compulsory matmul FLOPs, and TMA moves 96–100% of the global-load bytes, so they are not placeholders.

  4. Stability under resampling (AzfQ). Each Table 5 cell has 12 trajectories (3 scheduling-guidance variants × 4 repeats). Under leave-one-trajectory-out resampling, all seven Section 4.4 conclusions hold, with the best B200 attention median bound moving from 0.28× to 0.30×. Under the stricter leave-one-variant-out resampling the reviewer requested, 8 of 10 leaders hold, two H100 leaders flip (MHA-Fwd-Causal, MHA-Bwd-Causal), H100 MHA-Fwd becomes a near-tie, 19 of 21 zero-median cells stay zero, and the remaining conclusions hold. The revision will report these three H100 settings as ties, state the B200 attention bound as at most 0.34×, and include both analyses.

  5. GPU-side evaluation budget (AzfQ). Every turn sends one evaluation request. CUDA–PTX and Triton without autotuning compile and time exactly one configuration per request, and the latter still beats CUDA–PTX in median instruction-qualified speedup in all 20 settings of Table 6, so Triton's advantage does not come from extra configuration search. For standard Triton, we replayed a stratified sample of 62 kernels with and without the search: on H100, autotuning adds an estimated 15.1 s per correct turn, 7.7% of the average LLM API time per turn. The search finishes before timing starts, so it does not affect measured latency.

  6. Why qualification accepts C ∨ D (AzfQ). Some workloads need only one target family: RMSNorm has no matmul work, and its expert baseline qualifies only through D, so requiring C ∧ D would reject even expert kernels. No TMA + FFMA kernel appeared in the corpus. The corpus's ¬C∧D kernel uses TMA + mma.sync; its TMA moves 100% of the global-load bytes, and replacing it slows the kernel by 2.32×.

We are fully committed to including the following in the final version: (1) the five new workloads and the added Table 1 rows; (2) ordinary and instruction-qualified speedup as co-primary metrics, with qualification rates; (3) the stub-and-retime ablation and workload accounting; (4) the resampling analyses and revised Section 4.4 statements; (5) the GPU-side budget analysis; (6) the C ∨ D justification after the qualification rule; and (7) clearer notation and definitions. All new LLM-generated kernels and reproduction scripts are in the anonymous repository and supplementary material.

We are sincerely grateful for the reviewers' detailed feedback, which has significantly strengthened the paper.

Thank you for your time and consideration.

Sincerely, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.