- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CUDAHercules, a benchmark for evaluating LLMs on hardware-aware CUDA code generation and optimization.
Existing GPU code benchmarks like KernelBench, CUDABench, and ComputeEval mainly compare generated code against PyTorch baselines on isolated kernels, without testing architecture-specific features on newer GPUs or comparing against human-expert implementations and full applications.
To address this, the authors collect 195 tasks across four classes (63 single kernels, 119 module-level kernels, 10 full applications, and 3 open challenges), and evaluate 4 models (GPT-5.4, Claude-Opus-4.6, Qwen3.5-122B-A10B, and Kevin32B) on general, Hopper, and Blackwell GPUs under single-shot, self-refine, and tool-augmented settings.
Critical strengths
-
Comparing generated CUDA kernels directly against human-expert code (like CUTLASS, FlashAttention-3, and cuFFT) across different GPU generations is a practical and relevant problem.
-
The task coverage is broad, ranging from single kernels and fused modules to full applications in compression, scientific computing, SAT solving, and zero-knowledge proving.
Critical weaknesses
W1. Bugs in the evaluation scripts and task definitions in the anonymous repository invalidate the model evaluation.
Code bugs:
- Explicitly passing the "a"-suffixed architecture flags (sm_90a / sm_120a) when compiling the reference code, but omitting them when compiling model solutions (so PyTorch defaults to compiling with standard sm_90 / sm_120 without the "a" suffix): src/cuda_hercules/eval.py, tasks/class2/hopper/fa3_/def.py, tasks/class2/blackwell/sageattn3_fp4_/def.py
- Checking two internal temporary variables from the reference code (dsoftmax_sum and dq_accum) as required outputs in all 20 Hopper FlashAttention-3 backward tasks, even though the task prompt never asks the model to output them: src/cuda_hercules/eval.py, tasks/class2/hopper/fa3_bwd_*/def.py
- Setting the test input weight gamma to all 1s and bias beta to all 0s in all 16 LayerNorm tasks: tasks/class2/general/layernorm_*/def.py
- Telling the model in the Class 2 tool-augmented prompt that CUTLASS (#include <cutlass/...>) is allowed, and then blocking CUTLASS as cheating in the anti-cheat checker: scripts/eval_agent.py, src/cuda_hercules/static_checker.py
Consequences:
- In nvcc, Hopper wgmma and Blackwell FP4 instructions require the "a"-suffixed targets (sm_90a / sm_120a) and fail to compile under PyTorch's default auto-detected sm_90 / sm_120 flags (which is why the authors had to explicitly add sm_90a / sm_120a for the reference code).
- In the 20 fa3_bwd_* tasks (31% of Hopper Class 2), even if a model computes the exact gradients dQ, dK, and dV, it is marked wrong unless those two unmentioned temporary variables also match the reference code's internal values.
- Section 4.3 blames GPT-5.4's drop from 72% (1-sample) to 2% (Tool-augmented) on Class 2 on multi-round edits breaking working code. In reality, eval_agent.py keeps the best passing solution across all rounds (so later bad edits never overwrite an earlier passing result), and Figure 5(a) shows that accuracy is already ~2% at Round 0 before any tool is called (because the prompt tells the model to use CUTLASS and the checker immediately blocks it).
W2. Heavy task duplication in Class 2 by changing parameter constants. The paper claims 119 tasks in Class 2, but most of them in the repo are just copies of the same kernel with different dimension or dtype constants. For example, 60 of the 64 Hopper Class 2 tasks are the same 3 FlashAttention-3 templates copied 20 times across different head dimensions, precisions, and causal flags.
Questions for authors
Plz solve these bugs and I am happy to raise my scores.
-
Why does eval.py omit the sm_90a / sm_120a flags when compiling model code?
-
Why do the 20 fa3_bwd_* tasks check unmentioned internal temporary variables (dsoftmax_sum and dq_accum)?
-
Why does eval_agent.py allow CUTLASS in the Class 2 prompt and then block it as cheating in static_checker.py, and why is GPT-5.4 Tool-augmented in Figure 5(a) already at ~2% at Round 0 before any tool is called?
2 comments
I am glad to see that all the bugs are fixed and the author's promise to make a long-term effort for maintaining and improving this benchmark. Therefore, I have raised my score to 4.
Thank you for the very careful review and for inspecting our code in such detail. We apologize that the anonymous repository had not been synchronized with our latest code. We have since audited all evaluation scripts and task definitions, and every bug you identified has been fixed. The anonymous repository is now fully synchronized with our latest code. We treat maintaining this benchmark as a long-term effort and have kept improving and updating it until very recently, including adding new tasks. Throughout the review period, we will keep the anonymous repository synchronized with our latest code. The current behavior is as follows.
1. Architecture flags for model solutions (eval.py, fa3_*, sageattn3_fp4_*). Model solutions are now compiled for exactly the same target as the reference. For Hopper and Blackwell tasks, the solution reuses the reference's
-gencodepairs (compute_90a/sm_90a,compute_120a/sm_120a,sm_100a), sowgmma,setmaxnregand block-scaled FP4 MMA instructions are available to models exactly as they are to the reference. For general tasks, both the reference and the solution are compiled for the architecture of the GPU in use, and flags that affect code generation (e.g.,-DNDEBUG) are shared by both sides.2. Outputs checked in the FA3 backward tasks (fa3_bwd_*). All 20 tasks now check only the gradients
dQ,dKanddV, which is exactly what the prompt asks for. The reference's internal buffers (dsoftmax_sum,dq_accum) are treated as scratch space and are never compared.3. LayerNorm inputs (layernorm_*). In all 16 tasks,
gammaandbetaare now drawn at random instead of being all ones and zeros, so a solution that ignores the affine parameters fails.4. CUTLASS policy (eval_agent.py, static_checker.py). The library rules are now defined once in the static checker, and the same text is inserted into every prompt, including the tool-augmented prompts, so the prompt and the checker can no longer disagree. Prebuilt CUTLASS device-, kernel-, collective- and threadblock-level templates are disallowed in both. CuTe and CUTLASS's low-level building blocks (numeric types,
cutlass::arch, layouts, warp-level MMA) are allowed in both and are not blocked.5. Round 0 in Figure 5(a). The tool-augmented setting has 10 rounds, which we numbered 0–9 in the figure rather than 1–10. Round 0 is therefore the first round of the tool-augmented loop, not a state before the loop starts. We will relabel the axis 1–10 in the revision.