acceptodds
Under review as a conference paper at ICLR 2027

D2K-BENCH: Can LLM Agents Turn Expert Designs into Efficient GPU Kernels?

Abstract

GPU kernels generated by large language model (LLM) agents can remain less efficient than expert implementations, but runtime alone does not reveal how the gap relates to design discovery and implementation. We introduce D2K-Bench, a diagnostic benchmark of 26 tasks and 85 workloads that measures how effectively agents translate expert design guidance into efficient GPU kernels. The guidance covers L1: high-level algorithmic insights, L2: dataflow design, and L3: low-level optimization tricks, including dependencies among these levels. Pairwise runs with and without guidance share task descriptions, workloads, tools, hardware, and a 350-turn budget. Complementary assessments examine independently proposed designs and the design properties implemented in generated code. Across five models on NVIDIA B200 GPUs, guidance raises correctness over 130 model–task pairs from 93.1% to 98.5% and increases the Performance Score over all 26 tasks from 1.46 to 1.95. For the three frontier models with correct submissions on all 26 tasks in both runs (GPT-6-Astra, Claude-Opus-4.8, and GPT-5.6-Sol), geometric mean speedup increases from to . Across all five models, the mean combined implementation score increases from 57 to 70 out of 100. These results show the value of expert design guidance while identifying design properties that remain unimplemented.

Primary area:
datasets and benchmarks
Keywords:
GPU kernel generation, LLM agents, expert design guidance, benchmark evaluation, GPU optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Ej1MOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents D2K-Bench, a benchmark designed to evaluate agents' ability to turn expert designs into efficient GPU kernels. Specifically, the benchmark provides 26 tasks and 85workloads with three levels of expert guidance for each task: the L1 guidance for high-level algorithmic insights, the L2 guidance for dataflow design and the L3 guidance for low-level optimization tricks. The authors evaluate 5 models on B200GPUs and find that the tree levels of guidance can steadily raise the performance of LLM agents. The authors also use LLM as a judge to investigate the gap between design and implementation ability of LLM agents.

Critical strengths
  • The evaluation of models under different levels of guidance provides useful evidence on how expert guidance can affect agentic kernel optimizations.
  • The benchmark includes several non-trivial GPU kernel targets, including recent workloads from frontier open-source LLMs. Optimizing on modern GPUs such as the B200 is a practical workload.
  • The construction of the benchmark involves much engineering effort to collect expert guidance for the LLM.
Critical weaknesses

1. The core measurement tool (the LLM judge) is not validated.

One of the paper's central contributions is to use the LLM judge scores to compare the design and implementation capabilities of the agents. However, the paper does not provide any human expert validation for these scores. Moreover, the design and implementation comparison uses different evaluation protocols, which may introduce bias and make their scores difficult to compare with each other. There are two examples of the different evaluation protocols:

  • The implementation assessment assigns a score of zero to incorrect submissions, while the design assessment evaluates all proposals without a corresponding correctness penalty. This asymmetric behavior may introduce a bias when comparing the two scores.
  • The two evaluations use different design documents. The design is scored against the design rubric, while the implementation is scored against the expert guidance, and these documents can impose opposite requirements. For example, in DeepEP v2, the design judge classifies rank deduplication as an "invalid partial state" (Claude-Opus-4.8), whereas most code-judge records treat it as the required L1 transformation.

2. The expert guidance and rubrics contain internal contradictions and incorrect constraints.

There are several inconsistencies in the expert guidance and scoring rubrics, some of which have already affected the judge scores.

  • Self-contradiction. In DeepEP v2, expert_design_guidance.json permits rank deduplication at L1 (line 31) but forbids it at L2/L3 (lines 122–123, 138). It also claims deduplication "is rejected by the runtime record-count gate" (line 139), whereas the evaluator explicitly accepts it as "the expert's own strategy" (prog_dist_helper.py:94-99). Guided agents therefore received incorrect instructions. In MiniMax, L1 states that unsplit rows "publish directly" (line 31), while L2 requires that "no row bypasses [the combine pass]" (line 150).

  • Implementation choices treated as requirements. In the vLLM router, the guidance acknowledges that selected renormalization equals a softmax over the selected logits (line 16), yet requires exponentials for all 128 experts (lines 15, 118). The rubric declares the cheaper, equivalent alternative invalid (design_rubric.json:4,18), even though the task description permits algebraically equivalent lowerings.

3. In some examples, L1/L2/L3 levels are not substantively distinct.

  • fouroversix_nvfp4_quant. L1 (line 46) already specifies the design: "reduce one maximum per 16-value block and the scalar tensor maximum, but do not carry normalized … values across the barrier … reread each source block with its saved local maximum". The L3 item (line 152) restates it: "Publish exactly one FP32 maximum per quantization block and the scalar amax, then pair those completed statistics with one reread of the original block. Do not preserve normalized values across the barrier". Its constraints then state that the "synchronization mechanism, and launch decomposition are not prescribed", so the L3 item specifies no execution mechanism at all.
  • flash_kmeans_exact. L1 (line 30) already specifies the contention-based dispatch on N/K, the sorted equal-id runs with one contribution per run, and the direct scatter for large K. The L3 item (line 147) repeats all three, adding only "balancing parallel work over runs and feature regions", and its constraints are nearly verbatim copies of L1's.

Conversely, L1 items already contain dataflow and execution detail, e.g., "publish the winner directly in packed-value and blocked-scale coordinates" (fouroversix, line 46).

These confusion will make the results of the paper less convincing.

4. All the experiments use only one independent agent run per configure.

As the agent's behavior can vary dramatically between different runs, and some gaps (LLM judge scores, and the speed-up gains) presented in the paper are small, several of the paper's conclusions are based on the differences that may stem from the run-to-run variability.

5. The Triton-only requirements is questionable for L3, and is not consistently enforced.

The agents are required to use Triton to write kernels, while the L3 guidance focuses on the low-level implementation details derived from CUDA implementations. So it is unclear whether the agents' behaviors are affected by the constraint of using Triton. The authors' own agent instructions acknowledge this: "If a required Level 3 mechanism is unavailable in the required framework, record the route limitation…" (design_guidance/CLAUDE.md:238-239). Therefore, the scores can be a gap induced by the usage of the triton language.

Moreover, the requirement is not enforced for all evaluations. Several non-Triton solutions are counted as correct (e.g. FA4 solution from GPT-5.6-Sol), and GLM-5.3's solution directly wraps CUTLASS's own FMHA backward kernel, which is an existing implementation.

Questions for authors
  • Has the LLM judge been validated by human experts? How can we compare the design and implementation scores given that they are using different evaluation protocols?
  • How was the dataset checked for consistency, and how do these inconsistencies in datasets affect the experimental results?
  • Can the authors clarify how L2 and L3 add more content beyond L1, and how do the L1/L2/L3 meet their definenation?
  • How will the results change with more independent runs per configuration?
  • Can the authors justify the choice of using Triton instead of CUDA? Why are the non-Triton FA4 submissions counted as correct? How would the results change if Triton were not enforced?
7 comments
AuthorsOpenReview ↗

Response to Q1 about Score Comparability and Human Validation

Q1: How can the design and implementation scores be compared when their evaluation protocols differ? Has the LLM judge been validated against human experts?

Thank you for reviewing the released materials in detail. JSdesign\mathrm{JS}_{\text{design}} measures the quality of an independent design proposal, and JSimpl\mathrm{JS}_{\text{impl}} measures which design properties the agent realizes in executable code (Q1). We compare the properties found in proposals and code for the same fully correct tasks.

Why the two scores use different penalties (Q1). The agent writes the proposal without expert guidance or kernel execution. Missing properties, incorrect statements, and contradictions lower its score at the relevant design level. During kernel development, the agent can fix design flaws as it writes, tests, profiles, and revises the code. We report correctness separately and set JSimpl=0\mathrm{JS}_{\text{impl}}=0 when the final code fails the independent correctness checks.

Resolving conflicting requirements (Q1/Q2). We aligned the rank deduplication requirements in the DeepEP v2 expert design guidance and scoring criteria and reran the affected experiments.

Agreement with human evaluation (Q1). A domain expert assessed the guided and unguided GPT-6-Astra implementations for all 26 tasks using the task requirements, source code, and scoring rubric, with the expert design and reference implementation withheld. We also reviewed the judges’ rationales and the code they cited. With expert materials provided to the judges, Qwen3.8-Max-0902 and GPT-6.1-Sol each agreed with the expert’s ordering on 23/26 tasks (88.5%). For FlashQLA prefill, for example, Qwen’s score rose from 47.7 to 79.3 and the expert’s score rose from 51.7 to 85.0. The guided code fused the resident scan with readout. Appendices D.3 and D.6 report the full scores and agreement results, and Appendix D.5 gives the code evidence.

AuthorsOpenReview ↗

Response to Q2 about Dataset Consistency and Correction Effects

Q2: How did you check the consistency of the dataset? How did the inconsistencies in DeepEP v2, MiniMax, and the vLLM router affect the results?

We clarify that the task requirements for DeepEP v2, MiniMax, and the vLLM router, reran the affected experiments, and updated the results in the paper (Q2). Table 2 reports the guided GPT-6-Astra speedups over the Fable-5 baseline.

Table 2. Corrected task requirements and measured speedups (Q2)

TaskCorrected requirementSpeedup
DeepEP v2Allow deduplication by rank while preserving all logical expert assignments and their contributions to the inverse aggregation1.43×
MiniMaxWrite the output for a query after combining all contributions to that query2.18×
vLLM routerPreserve exact top-kk selection and allow normalization over the selected experts alone3.33×

Consistency checks (Q2). For each task, we checked the task description, individual and cumulative guidance, and scoring criteria against a fixed revision of the expert implementation and task evaluator. We looked for conflicting requirements and checked that the guidance and rubric allowed equivalent implementations that satisfy the task.

Results after correction (Q2). All three revised submissions passed the task checks. On all three tasks, the expert scored the revised guided implementation higher than its unguided counterpart (Appendix D.2). Across all 26 tasks, GPT-6-Astra’s Sperf\mathrm{S}_{\text{perf}} went from 2.68 without guidance to 3.33 with guidance in the submitted evaluation, and from 2.43 to 3.16 in the revised evaluation. Appendix D.5 presents code evidence for the corrected tasks.

AuthorsOpenReview ↗

Response to Q3 about the Distinct Content of Guidance Levels

Q3: What do L2 and L3 add for the NVFP4 and k-means tasks, and how does each of the three levels fit its definition?

We clarify that the overlap between guidance levels for NVFP4 quantization and exact k-means by moving data representation and output writeback from L1‾\underline{\textit{L1}} to L2‾\underline{\textit{L2}} and specifying the execution mechanisms at L3‾\underline{\textit{L3}} (Q3). The revised scoring criteria give credit for each design property once.

L1‾\underline{\textit{L1}} (high-level algorithmic insights) describes the algorithmic change and the computation or data movement it removes. L2‾\underline{\textit{L2}} (dataflow design) specifies where variables are produced, stored, and consumed, including their layout, lifetime, and reuse. L3‾\underline{\textit{L3}} (low-level optimization tricks) describes how to execute that dataflow, including mechanisms such as pipelining and warp specialization and their ordering and synchronization requirements.

Table 3. Information added at each guidance level (Q3)

TaskL1‾\underline{\textit{L1}}L2‾\underline{\textit{L2}}L3‾\underline{\textit{L3}}
fouroversix_nvfp4_quantEvaluate both specified quantization candidates locally to avoid generating candidates for the full tensor and selecting among them in a later stageCarry block and tensor maxima across dependency boundaries. Retain the local candidate encodings and scaling factors until the winner is determined and written out.Reduce maxima within warps and blocks. Convert two fragments of 8 values each and accumulate errors locally in FP32.
flash_kmeans_exactAggregate assignments that belong to the same cluster before updating the global centroidsKeep sorted IDs and index metadata, read the original rows, and finish computing the cluster sums and counts before reuse.Use bounded point and feature tiles, FP32 segmented sums, and atomic accumulation across blocks. Update counts outside the feature loop.

Rerun results (Q3). We reran both tasks after revising the guidance and scoring criteria. The guided NVFP4 implementation uses packed conversions and scale-pair instructions, and the measured instruction count fell from 110.82 million without guidance to 62.29 million with guidance. The guided k-means implementation groups assignments and uses segmented centroid updates. We verified these mechanisms in the code and examined the hardware counters. Appendix D.5 reports the code observations and execution metrics for both tasks.

Reviewer_KV9mOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces D2K-Bench, a benchmark asking whether the gap between LLM-written GPU kernels and expert kernels comes from not finding the right design or from not implementing it well. It has 26 tasks taken from real systems (attention, MoE, sampling, quantization etc.) with expert implementations on B200, and for each task the authors wrote expert design guidance at three levels: algorithm, dataflow, and low-level tricks. Five models run in Claude Code or Codex writing Triton kernels, once with the guidance and once without, and an LLM judge scores how much of the expert design shows up in the code (and in a separate design-only proposal). Guidance improves correctness and speed for all five models.

Critical strengths

S1. The tasks are realistic and current. They come from real serving and training systems, run on B200 with verified expert implementations, and the authors release the tasks, code and traces, so it's a reusable resource.

S2. The with/without guidance pairing is clean. Same task files, tools, budget and hardware, only the guidance changes, so the effect of guidance is easy to read. Splitting "can't find the design" from "can't implement it" is also a nice way to frame the question.

Critical weaknesses

W1. I'm not sure what new this benchmark offers. The main result, that giving agents the expert's design makes their kernels better, is pretty expected, and the paper's own related work already cites similar findings (Shui et al. show expert directions improve kernels in a B200 harness, and KernelBench and Hari et al. look at hardware and Speed-of-Light guidance). So the new value would have to be the diagnostic side, i.e. telling us which design properties agents fail to discover or implement. But the paper mostly reports averaged judge scores plus a single Muon example, and there's no systematic breakdown of what kinds of properties agents keep missing. I also worry about headroom. The strongest model already beats the expert implementations without any guidance, and by a lot with guidance, so it's unclear what gap this benchmark is supposed to track going forward. And since every new task needs experts to write three levels of guidance, it won't be easy to grow either.

W2. The evaluation doesn't test the agents that matter most here, and the diagnostic part leans on an unvalidated judge. The benchmark is supposed to tell us where kernel agents fall short, but it only runs general coding tools (Claude Code and Codex) with different models underneath. The related work cites a bunch of dedicated kernel agents (AutoKernel, TritonForge, Astra, CUDAMaster, KernelSkill, KernelFoundry, the Speed-of-Light guided agent from Hari et al.), and none of them is evaluated. These are exactly the systems people build for this problem, and several of them add profiling loops, hardware-aware guidance or optimization knowledge, which could close part of the "can't find the design" gap without any expert guidance. So right now we can't tell if the conclusions hold for actual kernel agents.

W3. On the judge side, it's one judge model, and its agreement with speed is only shown across five model-level averages, with no task-level correlation and no human agreement check. The judge also sees the expert code and the guidance when scoring, so guided runs could score higher just because their code looks like the guidance, not because they are better designs. I also don't fully see why a judge is needed for the implementations. The kernels are actually run on B200, and many of the design properties being scored are measurable, e.g. whether an intermediate is written to memory shows up in DRAM traffic, algorithmic savings show up in FLOP or instruction counts, and the kernel structure shows up in the launches. Profiler-based checks like these would be more reliable than an LLM reading the source, and wouldn't have the issue of the judge seeing the guidance.

W4. Beyond the judge, every model and setting is run once with no variance, the baseline is one model's first correct submission (hard to reproduce or interpret), the 0.5 penalty for failed tasks is arbitrary, and models run in different harnesses. The level ablation only adds levels in one fixed order, so we never see dataflow or low-level guidance on their own. Small thing but worth noting: in the Muon example the guided kernel skips the restart step the expert uses to limit numerical drift and still passes, which makes me wonder how tight the correctness checks are.

Questions for authors
  1. Why not evaluate dedicated kernel agents such as AutoKernel, Astra or KernelSkill? Do they close part of the gap between unguided and guided runs on their own?
  2. Can you give a systematic breakdown of which design properties agents miss most often, by level and by task type? That's what would make the benchmark diagnostic.
  3. How well does the judge agree with human experts, and how well do judge scores track speed at the task level rather than across model averages?
  4. Would guided runs still get higher implementation scores if the judge didn't see the guidance?
  5. How much do results vary across repeated runs?
  6. Who wrote the guidance and how was it checked for consistency across tasks? The AI-use statement mentions AI help with data synthesis, which part was that?
11 comments
AuthorsOpenReview ↗

Response to W1/Q2 about Diagnostic Value and Remaining Design Gaps

W1/Q2: What can the benchmark diagnose once the strongest model outperforms the expert implementation? Which design properties do agents most often miss across guidance levels and task categories?

We highlight that D2K-Bench identifies gaps in kernel design even when a model outperforms the expert implementations in aggregate (W1/Q2). KernelClaw lets agents retrieve reusable expert optimization knowledge, and KLineage derives reusable optimization skills from expert kernels. Sun et al., 2026 · Zhang et al., 2026

We vary the level of task-specific expert guidance to test which level improves performance the most. We assess design proposals and executable code separately across the algorithmic, dataflow, and execution levels. The proposals show whether models can independently identify and accurately describe designs like the experts’. The code shows how much of the supplied design the models implement and which properties they still miss even with guidance. Together with the performance measurements, these assessments help connect speed gains to specific changes in the code.

Design gaps by task category (Q2). The table reproduces the mean changes (in points) in Qwen3.8-Max-0902’s implementation scores from unguided to guided code across the 26 GPT-6-Astra tasks (Appendix D.4, Table 10). MoE compute and routing gained the most at L1‾\underline{\textit{L1}} and L2‾\underline{\textit{L2}}, and MoE communication gained the most at L3‾\underline{\textit{L3}}.

Task categoryTasksΔL1‾\Delta\underline{\textit{L1}}ΔL2‾\Delta\underline{\textit{L2}}ΔL3‾\Delta\underline{\textit{L3}}
Low precision / determinism3+5.3+6.0+11.3
MoE communication3+33.0+26.3+26.0
MoE compute / routing4+38.3+38.3+12.3
Attention6+14.3+11.2+2.7
Other fused operators4+13.0+10.3+4.3
Linear recurrence4+18.3+19.0+11.5
Sampling / verification2+18.5+19.5+4.5

With guidance, DeepEP v2 deduplicates transfers by destination rank, KDA context-parallel prefill replaces serial state passing with an exchange of affine summaries, and MiniMax prefill uses a warp-specialized pipeline. Appendix D.5 reports the code evidence and execution measurements.

Design guidance can improve a kernel that already outperforms the expert implementation (W1). On the 8192×81928192\times8192 workload for Adaptive NVFP4 quantization, GPT-6-Astra’s unguided implementation used a quantization path tailored to Blackwell and ran in 0.147 ms, compared with 0.339 ms for the expert implementation. With design guidance, the model used packed SIMD to compute the error of each quantization candidate and reduced latency to 0.090 ms on the same workload.

AuthorsOpenReview ↗

Response to W2/Q1 about Dedicated Kernel Agents

W2/Q1: Why not evaluate dedicated kernel agents such as AutoKernel, Astra, or KernelSkill? Can they close the gap between the guided and unguided runs on their own?

Design ability in everyday development (W2/Q1). We use Claude Code and Codex CLI in our main evaluation to reflect how experts use coding agents in practice. We test whether agents can help experts design kernels, follow expert design instructions, and propose better designs or improve performance as they iterate.

We also evaluated AutoKernel, driven by GPT-6-Astra, on all 26 tasks (Q1). We kept AutoKernel’s built-in skills and complete knowledge base in both conditions and set a budget of US\30pertask.Onthe25tasksbothtoolssolvedinbothconditions,geometricmeanspeedupoverFable−5’sfirstcorrectsubmissionrosefrom2.092×to2.449×forAutoKernelandfrom2.366×to3.050×forGPT−6−AstrainCodexCLI.Wetakethegeometricmeanacrossworkloadswithineachtaskandgiveeachtaskequalweight.Acrossall26tasks,AutoKernel’scorrectnessrosefrom25/26to26/26andits30 per task. On the 25 tasks both tools solved in both conditions, geometric mean speedup over Fable-5’s first correct submission rose from 2.092× to 2.449× for AutoKernel and from 2.366× to 3.050× for GPT-6-Astra in Codex CLI. We take the geometric mean across workloads within each task and give each task equal weight. Across all 26 tasks, AutoKernel’s correctness rose from 25/26 to 26/26 and its \mathrm{S}{\text{perf}}increasedfrom1.980to2.430.GPT−6−Astrasolvedall26tasksinbothconditions,anditsincreased from 1.980 to 2.430. GPT-6-Astra solved all 26 tasks in both conditions, and its\mathrm{S}{\text{perf}}$ increased from 2.426 to 3.155. The gains on AutoKernel show that task-specific expert design guidance can improve performance even when the agent already has general skills for implementation and hardware tuning.

AuthorsOpenReview ↗

Response to W3/Q3/Q4 about Judge Validation, Guidance Bias, and Profiling

W3/Q3/Q4: How closely do the judge scores agree with expert ratings and measured speed by task? Do guided implementations still score higher when the judge cannot see the expert guidance? Why use a judge when profiler measurements are available?

We confirmed that On the 26 GPT-6-Astra tasks, both judges agreed with the expert’s ordering on 23 tasks when given expert materials. Both judges also gave guided code higher mean scores with those materials withheld (Q3/Q4).

Agreement with human ratings and measured speed by task (Q3). With expert materials withheld, Qwen and GPT-6.1-Sol agreed with the expert’s within-task ordering on 23/26 and 24/26 tasks, respectively. With expert materials provided, the higher scoring implementation also ran faster on 19/26 tasks for Qwen and 21/26 for GPT-6.1-Sol. Appendix D.3 reports the human and Qwen scores by task.

Scoring with expert materials withheld (Q4). Both judges assessed all 26 tasks in both settings. In the second setting, we gave them the scoring criteria and available NCU profiling results and withheld the expert guidance and reference implementation. The two judges agreed on the direction of the score difference on 23/26 tasks, favoring the guided code on 19 and the unguided code on 4. With expert materials provided, the judges agreed with each other on 24/26 tasks. Appendix D.6 compares the two settings.

Table 2. Mean JSimpl\mathrm{JS}_{\text{impl}} (without → with design guidance, 0–100) (Q3/Q4)

Judgew/ expert materialsw/o expert materials
Qwen3.8-Max-090265.7 → 81.671.6 → 79.7
GPT-6.1-Sol59.3 → 77.059.5 → 70.0

Why we use an LLM judge (W3). We use an LLM judge to combine source code and execution evidence and assess which design properties each implementation realizes. DRAM traffic, tensor FLOPs, and kernel launch counts can corroborate specific mechanisms. For example, the drop in tensor FLOPs for Muon is consistent with replacing repeated rectangular matrix updates with a composition of smaller square matrices. The same counter can reflect different mechanisms and have different implications for speed depending on the task’s bottleneck. The judge interprets the evidence against each task’s requirements and credits alternative implementations that realize the same design properties. Appendix D.5 (Table 11) reports the code observations and execution metrics for all 26 tasks.

Reviewer_7UVXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

D2K-Bench studies whether kernel agents can discover and implement expert designs on 26 tasks comprising 85 workloads. It pairs unguided and guided runs, organizes guidance into algorithmic, dataflow, and low-level execution levels, and separately scores design proposals and implementations with an LLM judge. Across five models, guidance raises correctness from 93.1% to 98.5% and the aggregate performance score from 1.46 to 1.95.

Critical strengths
  1. A focused diagnostic design. Paired guidance interventions and separate proposal/code assessments extend correctness-and-speed evaluation in KernelBench (ICML 2025). The three guidance levels make the supplied optimization knowledge explicit, including dependencies between levels.

  2. Measured gains with correctness controls. The evaluation checks all workloads and relevant state updates. It also reports the three models that are correct on every task in both settings, whose geometric-mean speedup rises from 1.69× to 2.49×, separating this result from failure-penalized scores.

Critical weaknesses
  1. The claimed diagnostic advance is not established. JSdesign and JSimpl use the same expert-derived criteria, without validation against independent expert judgments of discovery and implementation failures. Alignment over five model averages cannot establish task-level diagnostic accuracy or reliable credit for alternative designs. This leaves the benchmark’s main contribution beyond correctness-and-speed evaluation insufficiently supported.

  2. The experiments establish a guidance benefit, but provide limited evidence about its source and stability. The cumulative ablation identifies only order-dependent increments; independent repeats and uncertainty estimates are absent. Equal 350-turn limits also leave inference cost unmatched, with a 29.9% increase for Claude-Opus-4.8. These gaps limit the stronger conclusions about capability bottlenecks and the relative value of guidance levels.

Questions for authors
  1. On a blinded, expert-labeled sample that includes correct alternative designs, how accurately do the judge scores identify implemented properties and discovery-versus-implementation errors?

  2. Do the guidance gains and model ordering persist across independent agent runs and at matched inference cost? Please report paired uncertainty estimates.

  3. Within the fully correct subset, do task-level design and implementation scores remain associated with measured performance, after accounting for task difficulty? The model-level averages alone cannot establish this.

  4. Which conclusions about L1, L2, and L3 are intended to be conditional on the fixed disclosure order? Figure 3 measures incremental gains after earlier levels are supplied, not each level’s independent contribution.

6 comments
AuthorsOpenReview ↗

Response to W1/Q1 about Judge Validation and Diagnostic Accuracy

W1/Q1: In a blind review that includes correct alternative designs, how accurately does the judge identify implemented design properties and distinguish design discovery errors from implementation errors?

Thank you for identifying the need to validate the diagnostic interpretation. We verified that the judges matched the human expert’s within-task implementation ordering on 23–24 of the 26 tasks (88.5–92.3%), and manual code checks support specific property assessments, including a correct alternative design.

Concretely, a domain expert assessed the 52 GPT-6-Astra implementations using task requirements, candidate code, and the scoring rubric, with the expert design and reference implementation withheld (Appendix D.1). With expert materials available to the judges, Qwen3.8-Max-0902 and GPT-6.1-Sol each matched the expert's within-task ordering on 23/26 tasks (88.5%). Without those materials and with available NCU evidence, agreement was 23/26 and 24/26, respectively (Appendix D.6). These percentages measure ranking agreement under the stated inputs, rather than accuracy on a fully blinded error-classification test.

The code checks also examine whether an alternative way could realize the assessed property. For example, Qwen credited the vLLM router's normalization over selected experts. Manual inspection confirmed that its denominator includes only those experts, giving the same weights as full softmax followed by selected-expert renormalization. This example shows that the judge credited an equivalent implementation.

JSdesign\mathrm{JS}_{\text{design}} assesses an independently generated proposal, whereas JSimpl\mathrm{JS}_{\text{impl}} assesses properties realized in code. When a proposal omits an assessed property, it indicates an omission in design formulation. When guided code still omits that property, it indicates incomplete realization of the supplied guidance. These are actionable diagnoses under the respective protocols. Because the proposal is not the blueprint for the evaluated code, their score difference cannot uniquely attribute a particular failure to discovery or implementation.

AuthorsOpenReview ↗

Response to W2/Q2 about Repeatability and Matched Inference Costs

W2/Q2: Do the gains from guidance and the model rankings hold across independent runs and at matched inference costs? What is the uncertainty in the paired estimates?

We confirmed that with design guidance, Claude-Opus-4.8 achieved 1.66–1.68× the Sperf\mathrm{S}_{\text{perf}} of its unguided runs at the same API budgets (Q2). For each of the 26 tasks and each budget, we selected the best eligible submission within that budget from the traces. Table 2 reports Sperf\mathrm{S}_{\text{perf}} relative to Fable-5’s first correct submission.

Table 2. Sperf\mathrm{S}_{\text{perf}} for Claude-Opus-4.8 (Claude Code) at matched API budgets (Q2)

API budget per task (US dollars)Without design guidanceWith design guidanceGuided-to-unguided performance ratio
201.2302.0421.661×
401.3752.3111.681×
801.3792.3131.678×

Performance and API cost at the same turn budget (W2/Q2). In the 350-turn runs, the guided trajectory is higher at every 50-turn checkpoint (Figure 4). The remeasured final submissions scored 1.50 without guidance and 2.04 with guidance on Sperf\mathrm{S}_{\text{perf}} (Table 1 of the paper). Using the same token prices for both runs, we estimate that total API cost rose from US\631.56toUS631.56 to US\\820.10 (+29.9%). Design guidance lengthened the inputs, increasing cache read costs by US\115.88andcachewritecostsbyUS115.88 and cache write costs by US\\67.64. Together, these increases accounted for 97.3% of the additional API cost. Output token usage rose by about 2.4% (Appendix A.4).

Independent repeat runs (W2/Q2). We conducted a small-scale repeat-run check to examine variability in measured kernel performance. On the tested configurations, performance varied little across runs, providing preliminary evidence of stability within this subset.

AuthorsOpenReview ↗

Response to W1/Q3 about Task-Level Score–Performance Associations

W1/Q3: On the subset of fully correct submissions, do design and implementation scores still correlate with measured performance by task after controlling for task difficulty?

We verified that on the fully correct subset, implementation-score trend correlate with performance trend on 19/26 tasks for Qwen and 21/26 for GPT-6.1-Sol. This compares implementations within the same task, addressing the concern about differences in task difficulty.

GPT-6-Astra produced correct implementations for all 26 tasks in both conditions. The comparisons hold task identity and workloads fixed, with expert materials provided to the judges. For example, MoonEP's Qwen score increased from 45.3 to 80.7, and the guided implementation ran 1.498× as fast. We count a task as an agreement when the implementation score and measured speed change in the same direction.

The exceptions clarify what the score measures. For DeepGEMM, Qwen's score increased from 80.7 to 87.7 while speed fell to 0.97×. The guided implementation realizes additional layout-related design properties, but its taller tiles and finer scale packing can add work. On the profiled workload, instructions rose from 176.4 to 223.7 million despite lower DRAM traffic. The expert's L3 score fell from 85 to 80. Design-property realization therefore does not guarantee faster execution.

JSdesign\mathrm{JS}_{\text{design}} assesses a separate proposal, which the implementation run does not consume. We report no task-adjusted association between that score and final kernel speed. Its role is to assess independent design formulation. The paired code analysis above provides the more direct implementation–performance comparison.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.