- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces PTXBench, a framework for evaluating LLMs on architecture-specific GPU programming in CUDA with inline PTX. It proposes instruction-qualified speedup: a kernel's speedup counts only if the kernel is correct and executes targeted hardware-accelerated instructions at runtime (e.g., wgmma/TMA on H100, tcgen05/TMEM on B200), verified via SASS execution counts from Nsight Compute. The framework also fixes the generation conditions: supplied architectural knowledge, execution feedback, harness, and budget. Experiments cover one BF16 GEMM and four attention workloads on H100 and B200. Ablations with Gemini show that supplied knowledge and execution feedback raise instruction-qualified speedup, and that simple multi-turn refinement is competitive with coding agents. Across five LLMs, 25.9% of trajectories with a correct kernel never use the target instructions, no model's median reaches cuBLAS/cuDNN, and Triton outperforms CUDA–PTX in all settings.
Critical strengths
1. Rigorous and reproducible measurement. Instruction execution is verified from runtime SASS counters rather than source matching. This makes the check hard to game and applicable across languages, and the released artifacts regenerate all results. Even though I question the metric's necessity, the reported numbers are trustworthy and reusable.
2. Useful findings on evaluation conditions. Supplied knowledge strongly changes model behavior (models almost never use wgmma without PTX wrappers), and model rankings vary sharply across workloads. Kernel-generation evaluations should therefore report and control these conditions.
Critical weaknesses
1. The metric is ill-founded in principle. Using specific instructions is a means, not the goal. If instruction use correlates with performance, the gate is redundant, and the paper's own data shows this (leaders unchanged, L356–357). If it does not correlate, the gate penalizes better solutions. This is common in practice: in low-latency decoding kernels (small batch, memory-bound), legacy mma or CUDA-core paths can outperform wgmma/tcgen05, yet such kernels would score zero. Even within this paper, a correct 0.41× kernel without target instructions scores 0 (Fig. 4, H100 MHA-Fwd), while a 0.02× kernel that executes a single wgmma scores higher. Because the target instructions are fixed in advance by experts, the metric rewards imitating a known solution rather than optimizing. Instruction use would be useful as a separate diagnostic, but folding it into the score is not justified.
2. The metric adds little beyond plain speedup. In my experience, agents given sufficient spec knowledge find the fast instructions on their own. The paper's own results agree: the leaders are unchanged when non-qualified kernels are allowed (L356–357), and all kernels ≥0.5× already qualify (Table 4). Combined with the small scope (five workloads, one shape each, few repeats per setting), the main contribution is not substantial enough for ICLR.
Questions for authors
See my concerns on the motivation, that's the major question I have for this paper.
4 comments
What does our metric add beyond plain speedup?
That leaders are unchanged (L356–357) and all kernels ≥0.50× qualify (Table 4) is what Point 1 predicts: in compute-bound workloads, fast kernels need the target instructions. Qualification adds the diagnosis behind it: 25.9% of correct trajectories never use them and stay below 0.50×. It also makes "sufficient spec knowledge" measurable: as the reviewer notes (Strength 2), models almost never execute wgmma without supplied PTX wrappers, which speedup alone cannot attribute.
On the number of repeats: each Table 5 cell has 12 trajectories, and our reply to Reviewer AzfQ (Q1) shows every Table 5 conclusion survives leave-one-trajectory-out resampling. The new workloads in the shared comment also use 12 per cell, and the paper's conclusions hold on them.
I thank the authors for the detailed response and the new workloads. However, my main concern is not addressed. The ceiling argument shows that the target instructions are necessary for peak performance on compute-bound workloads, but this is exactly why plain speedup already captures their benefit there, consistent with the leaders being unchanged. Where instruction use and performance diverge, the new RMSNorm results confirm my concern. For Gemini on B200, the non-qualified CUDA–PTX median (0.86×) exceeds the qualified one (0.77×). Triton kernels reaching 0.67–0.94× of QuACK score zero in most cells because Triton cannot emit DSMEM accesses. The qualification set also had to be extended post hoc to match the expert baseline. This suggests that the metric rewards matching expert instruction choices rather than performance.
I agree that "ordinary speedup remains necessary for comparing performance", and would encourage the authors to report speedup as the primary metric, with instruction qualification as a separate diagnostic. I keep my score.
We thank the reviewer for the careful follow-up. The reviewer finds the measurement rigorous and instruction qualification useful as a separate diagnostic; the remaining concern is whether it should be folded into the headline metric. The RMSNorm cases the reviewer cites, the Gemini B200 median and Triton's lack of DSMEM, are where reporting both matters, which is why we included them in our response. Even there, the fastest kernels on both GPUs use DSMEM (0.97× and 1.06× of QuACK, vs. 0.93× and 0.97× without it): qualification marks what reaches the ceiling, while ordinary speedup shows how far kernels get without it. On Table 1, the qualification set follows one fixed rule: the hardware mechanisms the expert baseline relies on. We added the new rows before scoring any LLM kernel, and they change no result in the paper. We are grateful to the reviewer's suggestion. In the revision, every main table will report ordinary speedup and instruction-qualified speedup as co-primary metrics, with the qualification rate beside them.
We thank the reviewer for the detailed critique, and for noting that the measurement is rigorous and reusable. New workloads and results are in the shared comment.
Justification for the metric
Instruction qualification measures a specific capability: whether an LLM uses the hardware mechanisms that raise the performance ceiling on new GPUs, as expert kernels do. Speedup only tells how fast a kernel is now; qualification tells whether it is on the path to the higher ceiling. In compute-bound workloads (GEMM, FP8-GEMM, Conv2D and attention have arithmetic intensity 3.5–9× the GPUs' roofline ridge point), the target instructions set that ceiling:
wgmma/tcgen05.mma, a kernel must use legacymma.sync, which we measured at 0.67× thewgmmarate per SM clock on H100 and 0.25× thetcgen05.mmarate on B200. However well tuned, a BF16 kernel without C cannot exceed 782 TFLOPS on H100 or 675 TFLOPS on B200, even adding full FFMA throughput at peak boost clock; with C the ceilings are 989 and 2,250 TFLOPS. FP8 has no legacy tensor-core path; the compiler emulates FP8mma.syncwith FP16 instructions.mma.syncis also synchronous, whereaswgmmais asynchronous, which lets CUDA core execution overlap with the matmul.mma.sync), and moves 88–100% of its L2→SM load bytes with TMA on both GPUs.mma.sync+cp.async) reaches 0.60× (forward), 0.62× (causal forward) and 0.62× (backward) of cuDNN in FlashAttention-3's H100 benchmark (batch 4, sequence length 4k, head dim 128, as in our workloads; Figs. 5–6). FlashAttention-3, using both, reaches 1.05–1.12× across the same three settings.2. The correlation dilemma. Instruction qualification and performance can align or diverge. Our objective is to measure LLM’s ability to use target mechanisms and the performance achieved while meeting that requirement. Instruction-qualified speedup is therefore a capability-specific measurement, while ordinary speedup remains necessary for comparing performance. In the paper's compute-bound workloads, correct non-qualified trajectories drop from 116 to 8 at 0.25× and to 0 at 0.50× (Table 4). In the HBM-bound RMSNorm, non-qualified kernels come close, reaching 0.93× (H100) and 0.97× (B200) of QuACK, while the fastest kernels are qualified (0.97× and 1.06×), using distributed shared memory (DSMEM) as QuACK does. For Gemini 3.1 Pro on B200, the non-qualified CUDA–PTX median is even higher (0.86× vs. 0.77×). Qualification thus does not always agree with speedup, so it is not redundant. Where they disagree, we report both speedups side by side, as the reviewer suggests, so no faster kernel is hidden.
3. Low-latency decoding. Qualification is C ∨ D, not C alone. A memory-bound decode kernel that computes with
mma.syncor CUDA cores but moves data through TMA or DSMEM qualifies. One using neither is the RMSNorm case above: we report its speedup next to the qualified one, and the gap between them.4. The 0.41× vs. 0.02× example. The two kernels come from different settings: the 0.41× kernel is H100 MHA-Fwd, and the 0.02× kernel is B200 GEMM. The 0.41× kernel executes ¬C ∧ ¬D, and even FlashAttention-2, an expert ¬C ∧ ¬D design, reaches only 0.60× of an cuDNN at our shape (Point 1). Instruction-qualified speedup answers a deeper question than plain speedup: what performance does the model achieve while correctly exercising at least one targeted mechanism? The 0.02× kernel is an early step on the higher-ceiling path. In the same trajectory, the next turn reaches 0.49× with C ∧ D, and the final turn reaches 0.52× with C ∧ D, 2.5× the best LLM-generated B200 GEMM kernel with ¬C ∧ ¬D (0.21×). Figure 4 shows ordinary speedup next to the qualified score, so both readings are visible.
Qualification does not fix the algorithm, schedule, tiling or pipelining. Expert baselines converge on these instruction classes because they are the hardware's fast paths, not because one expert design is copied. The refinement loop does not directly reward instruction use: its feedback reports correctness and latency.