- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper presents D2K-Bench, a benchmark designed to evaluate agents' ability to turn expert designs into efficient GPU kernels. Specifically, the benchmark provides 26 tasks and 85workloads with three levels of expert guidance for each task: the L1 guidance for high-level algorithmic insights, the L2 guidance for dataflow design and the L3 guidance for low-level optimization tricks. The authors evaluate 5 models on B200GPUs and find that the tree levels of guidance can steadily raise the performance of LLM agents. The authors also use LLM as a judge to investigate the gap between design and implementation ability of LLM agents.
Critical strengths
- The evaluation of models under different levels of guidance provides useful evidence on how expert guidance can affect agentic kernel optimizations.
- The benchmark includes several non-trivial GPU kernel targets, including recent workloads from frontier open-source LLMs. Optimizing on modern GPUs such as the B200 is a practical workload.
- The construction of the benchmark involves much engineering effort to collect expert guidance for the LLM.
Critical weaknesses
1. The core measurement tool (the LLM judge) is not validated.
One of the paper's central contributions is to use the LLM judge scores to compare the design and implementation capabilities of the agents. However, the paper does not provide any human expert validation for these scores. Moreover, the design and implementation comparison uses different evaluation protocols, which may introduce bias and make their scores difficult to compare with each other. There are two examples of the different evaluation protocols:
- The implementation assessment assigns a score of zero to incorrect submissions, while the design assessment evaluates all proposals without a corresponding correctness penalty. This asymmetric behavior may introduce a bias when comparing the two scores.
- The two evaluations use different design documents. The design is scored against the design rubric, while the implementation is scored against the expert guidance, and these documents can impose opposite requirements. For example, in DeepEP v2, the design judge classifies rank deduplication as an "invalid partial state" (Claude-Opus-4.8), whereas most code-judge records treat it as the required L1 transformation.
2. The expert guidance and rubrics contain internal contradictions and incorrect constraints.
There are several inconsistencies in the expert guidance and scoring rubrics, some of which have already affected the judge scores.
-
Self-contradiction. In DeepEP v2,
expert_design_guidance.jsonpermits rank deduplication at L1 (line 31) but forbids it at L2/L3 (lines 122–123, 138). It also claims deduplication "is rejected by the runtime record-count gate" (line 139), whereas the evaluator explicitly accepts it as "the expert's own strategy" (prog_dist_helper.py:94-99). Guided agents therefore received incorrect instructions. In MiniMax, L1 states that unsplit rows "publish directly" (line 31), while L2 requires that "no row bypasses [the combine pass]" (line 150). -
Implementation choices treated as requirements. In the vLLM router, the guidance acknowledges that selected renormalization equals a softmax over the selected logits (line 16), yet requires exponentials for all 128 experts (lines 15, 118). The rubric declares the cheaper, equivalent alternative invalid (design_rubric.json:4,18), even though the task description permits algebraically equivalent lowerings.
3. In some examples, L1/L2/L3 levels are not substantively distinct.
- fouroversix_nvfp4_quant. L1 (line 46) already specifies the design: "reduce one maximum per 16-value block and the scalar tensor maximum, but do not carry normalized … values across the barrier … reread each source block with its saved local maximum". The L3 item (line 152) restates it: "Publish exactly one FP32 maximum per quantization block and the scalar amax, then pair those completed statistics with one reread of the original block. Do not preserve normalized values across the barrier". Its constraints then state that the "synchronization mechanism, and launch decomposition are not prescribed", so the L3 item specifies no execution mechanism at all.
- flash_kmeans_exact. L1 (line 30) already specifies the contention-based dispatch on N/K, the sorted equal-id runs with one contribution per run, and the direct scatter for large K. The L3 item (line 147) repeats all three, adding only "balancing parallel work over runs and feature regions", and its constraints are nearly verbatim copies of L1's.
Conversely, L1 items already contain dataflow and execution detail, e.g., "publish the winner directly in packed-value and blocked-scale coordinates" (fouroversix, line 46).
These confusion will make the results of the paper less convincing.
4. All the experiments use only one independent agent run per configure.
As the agent's behavior can vary dramatically between different runs, and some gaps (LLM judge scores, and the speed-up gains) presented in the paper are small, several of the paper's conclusions are based on the differences that may stem from the run-to-run variability.
5. The Triton-only requirements is questionable for L3, and is not consistently enforced.
The agents are required to use Triton to write kernels, while the L3 guidance focuses on the low-level implementation details derived from CUDA implementations. So it is unclear whether the agents' behaviors are affected by the constraint of using Triton. The authors' own agent instructions acknowledge this: "If a required Level 3 mechanism is unavailable in the required framework, record the route limitation…" (design_guidance/CLAUDE.md:238-239). Therefore, the scores can be a gap induced by the usage of the triton language.
Moreover, the requirement is not enforced for all evaluations. Several non-Triton solutions are counted as correct (e.g. FA4 solution from GPT-5.6-Sol), and GLM-5.3's solution directly wraps CUTLASS's own FMHA backward kernel, which is an existing implementation.
Questions for authors
- Has the LLM judge been validated by human experts? How can we compare the design and implementation scores given that they are using different evaluation protocols?
- How was the dataset checked for consistency, and how do these inconsistencies in datasets affect the experimental results?
- Can the authors clarify how L2 and L3 add more content beyond L1, and how do the L1/L2/L3 meet their definenation?
- How will the results change with more independent runs per configuration?
- Can the authors justify the choice of using Triton instead of CUDA? Why are the non-Triton FA4 submissions counted as correct? How would the results change if Triton were not enforced?
7 comments
Response to Q2 about Dataset Consistency and Correction Effects
Q2: How did you check the consistency of the dataset? How did the inconsistencies in DeepEP v2, MiniMax, and the vLLM router affect the results?
We clarify that the task requirements for DeepEP v2, MiniMax, and the vLLM router, reran the affected experiments, and updated the results in the paper (Q2). Table 2 reports the guided GPT-6-Astra speedups over the Fable-5 baseline.
Table 2. Corrected task requirements and measured speedups (Q2)
| Task | Corrected requirement | Speedup |
|---|---|---|
| DeepEP v2 | Allow deduplication by rank while preserving all logical expert assignments and their contributions to the inverse aggregation | 1.43× |
| MiniMax | Write the output for a query after combining all contributions to that query | 2.18× |
| vLLM router | Preserve exact top- selection and allow normalization over the selected experts alone | 3.33× |
Consistency checks (Q2). For each task, we checked the task description, individual and cumulative guidance, and scoring criteria against a fixed revision of the expert implementation and task evaluator. We looked for conflicting requirements and checked that the guidance and rubric allowed equivalent implementations that satisfy the task.
Results after correction (Q2). All three revised submissions passed the task checks. On all three tasks, the expert scored the revised guided implementation higher than its unguided counterpart (Appendix D.2). Across all 26 tasks, GPT-6-Astra’s went from 2.68 without guidance to 3.33 with guidance in the submitted evaluation, and from 2.43 to 3.16 in the revised evaluation. Appendix D.5 presents code evidence for the corrected tasks.
Response to Q3 about the Distinct Content of Guidance Levels
Q3: What do L2 and L3 add for the NVFP4 and k-means tasks, and how does each of the three levels fit its definition?
We clarify that the overlap between guidance levels for NVFP4 quantization and exact k-means by moving data representation and output writeback from to and specifying the execution mechanisms at (Q3). The revised scoring criteria give credit for each design property once.
(high-level algorithmic insights) describes the algorithmic change and the computation or data movement it removes. (dataflow design) specifies where variables are produced, stored, and consumed, including their layout, lifetime, and reuse. (low-level optimization tricks) describes how to execute that dataflow, including mechanisms such as pipelining and warp specialization and their ordering and synchronization requirements.
Table 3. Information added at each guidance level (Q3)
| Task | |||
|---|---|---|---|
| fouroversix_nvfp4_quant | Evaluate both specified quantization candidates locally to avoid generating candidates for the full tensor and selecting among them in a later stage | Carry block and tensor maxima across dependency boundaries. Retain the local candidate encodings and scaling factors until the winner is determined and written out. | Reduce maxima within warps and blocks. Convert two fragments of 8 values each and accumulate errors locally in FP32. |
| flash_kmeans_exact | Aggregate assignments that belong to the same cluster before updating the global centroids | Keep sorted IDs and index metadata, read the original rows, and finish computing the cluster sums and counts before reuse. | Use bounded point and feature tiles, FP32 segmented sums, and atomic accumulation across blocks. Update counts outside the feature loop. |
Rerun results (Q3). We reran both tasks after revising the guidance and scoring criteria. The guided NVFP4 implementation uses packed conversions and scale-pair instructions, and the measured instruction count fell from 110.82 million without guidance to 62.29 million with guidance. The guided k-means implementation groups assignments and uses segmented centroid updates. We verified these mechanisms in the code and examined the hardware counters. Appendix D.5 reports the code observations and execution metrics for both tasks.
Response to Q1 about Score Comparability and Human Validation
Thank you for reviewing the released materials in detail. JSdesign measures the quality of an independent design proposal, and JSimpl measures which design properties the agent realizes in executable code (Q1). We compare the properties found in proposals and code for the same fully correct tasks.
Why the two scores use different penalties (Q1). The agent writes the proposal without expert guidance or kernel execution. Missing properties, incorrect statements, and contradictions lower its score at the relevant design level. During kernel development, the agent can fix design flaws as it writes, tests, profiles, and revises the code. We report correctness separately and set JSimpl=0 when the final code fails the independent correctness checks.
Resolving conflicting requirements (Q1/Q2). We aligned the rank deduplication requirements in the DeepEP v2 expert design guidance and scoring criteria and reran the affected experiments.
Agreement with human evaluation (Q1). A domain expert assessed the guided and unguided GPT-6-Astra implementations for all 26 tasks using the task requirements, source code, and scoring rubric, with the expert design and reference implementation withheld. We also reviewed the judges’ rationales and the code they cited. With expert materials provided to the judges, Qwen3.8-Max-0902 and GPT-6.1-Sol each agreed with the expert’s ordering on 23/26 tasks (88.5%). For FlashQLA prefill, for example, Qwen’s score rose from 47.7 to 79.3 and the expert’s score rose from 51.7 to 85.0. The guided code fused the resident scan with readout. Appendices D.3 and D.6 report the full scores and agreement results, and Appendix D.5 gives the code evidence.