acceptodds
Under review as a conference paper at ICLR 2027

DICE: Diffusion Large Language Models Excel at Generating CUDA Kernels

Abstract

Diffusion large language models (dLLMs) have emerged as a compelling alternative to autoregressive (AR) LLMs, owing to their capacity for parallel token generation. Their capacity for holistic structural planning and non-sequential refinement motivates exploring dLLMs for code generation. Despite this potential, tailoring dLLMs for CUDA kernel generation remains challenging, obstructed not only by the high specialization but also by the severe lack of high-quality training data. To address these challenges, we construct CuKe, an augmented supervised fine-tuning (SFT) dataset optimized for high-performance CUDA kernels. On top of it, we propose a bi-phase curated reinforcement learning (BiC-RL) framework consisting of a CUDA kernel infilling stage and an end-to-end CUDA kernel generation stage. With data scheduling during training, the model learns how to write optimized CUDA kernels progressively. Leveraging this training framework, we introduce DICE, a series of diffusion large language models designed for CUDA kernel generation, spanning three parameter scales: 1.7B, 4B, and 8B. Extensive experiments on KernelBench and MultiKernelBench demonstrate that DICE significantly outperforms both autoregressive and diffusion LLMs of comparable scale, establishing a new state-of-the-art for CUDA kernel generation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Diffusion Large Language Model, CUDA Kernel Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QPkiOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper applies RL to post-train dLLMs for the CUDA kernel generation task. It combines curated SFT data with a curriculum progressing from scaffolded kernel infilling to complete generation, and from simple to complex tasks.

DICE first learns from examples of CUDA code. During RL training, it generates implementations that are compiled and tested. Successful attempts receive positive feedback, encouraging the model to generate correct code.

Training has two phases. First, the model writes only the core computation inside a supplied program scaffold. This reduces mistakes in the surrounding code and limits shortcuts such as simply calling the original PyTorch operations. Second, it generates the complete program, including the code that compiles and calls its CUDA implementation. The idea is to learn the central skill first, then handle the whole task. RL rewards correctness; execution speed is encouraged through the examples used in the initial training.

Critical strengths

Improves CUDA generation correctness across several settings. Includes useful ablations and output inspection to identify implementations that bypass custom CUDA kernels.

Critical weaknesses
  • Limited novelty: The method mainly applies familiar post-training techniques to dLLMs, with little new methodology.
  • Limited diffusion-specific contribution: The architecture, decoding method, and RL algorithm come from prior work. The proposed curriculum also works with AR models.
  • Weak supporting analysis: The experiments do not show which diffusion properties drive the gains or separate the curriculum's effect from additional training and data.

Baselines and comparisons: The paper includes many AR and diffusion baselines, including Qwen3 trained with the same curriculum. However, different output budgets and inference engines complicate comparisons. The ablations do not fully control for additional training and data exposure.

Taken together, these limitations make the contribution insufficient for acceptance in its current form. I recommend rejection in the current form.

Questions for authors

None.

2 comments
AuthorsOpenReview ↗

Response Part 1/2

Dear Reviewer QPki,

Thank you for the detailed and constructive review. We address each concern below and have revised the manuscript with additional analyses and clarifications.


Weaknesses1: Limited novelty.

  • Performance-oriented data construction. CuKe is not obtained by merely enlarging or concatenating existing datasets. Starting from ConCuR, we re-measure candidate kernels under a consistent execution environment and add 1,425 PyTorch–CUDA pairs selected using a strict 2.0×2.0\times speedup threshold, together with 36 execution-validated structural examples. Tables 5 and 6 evaluate the resulting quality–quantity trade-off and the effectiveness of the complete curated dataset.

  • CUDA-specific RL curriculum. BiC-RL introduces a task-specific curriculum designed around common CUDA-generation failure modes. During kernel infilling, an executable scaffold fixes the imports, compilation and binding logic, ModelNew wrapper, and invocation path, while the model generates only the missing CUDA/C++ core. This reduces the initial exploration space and restricts the three shortcut patterns described in Section 4.2. The subsequent end-to-end stage then restores responsibility for generating the complete program. Tables 7 and 15 show that, relative to single-stage RL, the complete curriculum improves 20 of 27 scale–level–metric comparisons and 7 of 9 execution-correctness comparisons. Table 16 provides additional evidence consistent with the intended role of the scaffold in reducing invalid CUDA-generation shortcuts.

  • Specialized dLLM evidence. To the best of our knowledge, DICE is the first specialized diffusion-LM family trained and systematically evaluated for CUDA kernel generation across 1.7B, 4B, and 8B scales. After the complete training pipeline, the total number of correct kernels increases from 0 to 35, 8 to 57, and 8 to 89 relative to the corresponding SDAR base models, respectively. We also evaluate DICE against both autoregressive and diffusion baselines rather than presenting specialization results for only a single model or scale.


Weaknesses2: Limited diffusion-specific contribution.

Our diffusion-specific contribution lies in the systematic specialization and evaluation of an SDAR-based dLLM family for CUDA kernel generation.

  • Matched post-training comparison. Table 8 compares DICE with Qwen3 under closely matched post-training data and curriculum settings. DICE achieves higher aggregate execution correctness at all three scales, even though the original SDAR checkpoints are weaker than the corresponding Qwen3 checkpoints.

  • Limited-data adaptation. Table 10 and Figure 6 compare SDAR-Chat-4B and Qwen3-4B using identical CuKe subsets and SFT configurations at each data fraction, providing additional evidence that the evaluated diffusion family adapts more favorably under limited task-specific data.

  • Practical inference evidence. Figure 1 and Table 9 show that DICE retains favorable generation throughput across multiple model scales and batch sizes, demonstrating the practical efficiency associated with its diffusion-style parallel decoding.


Weaknesses3: Weak supporting analysis.

dLLM evidence. In SDAR, block diffusion, iterative denoising, and the associated decoding procedure are jointly encoded in the pretrained checkpoint. Replacing one of these components would therefore constitute a different trained model rather than a controlled removal from an otherwise unchanged checkpoint. Accordingly, we evaluate their effect through closely matched model-family comparisons. Our evidence therefore characterizes the adaptation behavior of the SDAR-based diffusion family under closely matched post-training conditions.

  • Table 8 compares DICE with Qwen3 using closely matched post-training data and curriculum settings, providing model-family-level evidence for their different adaptation behaviors.

  • Table 10 and Figure 6 further compare SDAR-Chat-4B and Qwen3-4B using the same CuKe subsets and SFT configurations at each data fraction, showing more favorable limited-data adaptation for the dLLM family.

Together, these results provide complementary evidence for the practical suitability of the SDAR-based diffusion family under the evaluated CUDA-generation setting.

AuthorsOpenReview ↗

Response Part 2/2

Curriculum components. Our curriculum contains two key components: data scheduling and a kernel infilling stage followed by end-to-end generation. Tables 7 and 15 provide component-level evidence for their respective roles across the three model scales:

  • The RL baseline and data-scheduling variant use the same SFT initialization and end-to-end RL budget. This comparison isolates data scheduling under a matched training budget.

  • The comparison between SFT and SFT + kernel-infilling checkpoint measures the incremental effect of introducing this stage. It improves overall execution correctness, although the speed metrics do not improve uniformly.

  • Continuing from the infilling checkpoint to end-to-end generation produces the strongest overall performance, supporting the role of the final stage in translating the scaffolded initialization into complete kernel-generation capability.


Baselines and comparisons. Output budgets. The different maximum output lengths in the task evaluation are intentional and follow the generation characteristics of each model family. Reasoning models require a larger maximum budget to avoid truncating their reasoning traces and CUDA implementations, whereas fixed-length diffusion models such as LLaDA and Dream become prohibitively expensive and may produce repetitive content at unnecessarily long output lengths. For models supporting EOS termination, the specified token budget is only an upper bound, and generation stops when EOS is produced. Table 13 further shows that reducing cudaLLM’s maximum generation length from 32,768 to 4,096 substantially degrades its results, indicating that a uniformly short budget would disadvantage long-reasoning models. Conversely, the length-sensitivity results in Table 16 show that increasing the fixed generation length of LLaDA-Instruct and LLaDA-1.5 from 1,024 to 4,096 does not yield an overall performance improvement. Together, these results motivate the use of model-compatible output budgets rather than imposing a single generation length.

Inference backends. The evaluated model families use different decoding procedures and attention mechanisms, and a single equally optimized backend is not available for all evaluated decoding algorithms under our experimental setup. We therefore use an optimized and model-compatible inference backend for each model family. We clarify that the inference engines reported in Figure 1 are used only for measuring inference throughput and are not involved in the KernelBench evaluation.

Ablation controls and data exposure. We appreciate this concern. Our paper groups several ablation analyses. These experiments address different questions:

  • Table 5 uses the same SDAR-Chat-4B model and SFT configuration to examine how different speedup thresholds affect the resulting dataset and downstream performance. Thresholding changes both dataset composition and size, this experiment therefore characterizes the trade-off between the performance quality of the retained kernels and the quantity of training data.

  • Table 6 uses the same SDAR-Chat-8B model and SFT configuration to compare training on the cudaLLM, ConCuR, and CuKe datasets. It evaluates the practical quality–quantity trade-off among the complete datasets, rather than constituting a size-matched comparison.

  • Tables 7 and 15 decompose the training pipeline into SFT, single-stage RL, data scheduling, kernel infilling, and the complete BiC-RL pipeline across the 8B, 4B, and 1.7B scales. These variants share the same SFT initialization and evaluation protocol.

  • Table 8 compares SDAR and Qwen3 using the same CuKe SFT data, RL data, and curriculum structure. Only model-compatible modifications required by their different generation mechanisms are retained. This provides a comparison under closely matched post-training exposure, although it may not isolate the generation paradigm as the only causal factor.

To examine SFT data exposure more directly, we additionally train SDAR-Chat-4B and Qwen3-4B using 25%, 50%, and 75% of CuKe. At each fraction, the two models receive the same number of training examples. The results are reported in Table 10 and Figure 6 of Appendix A.2. Despite some non-monotonicity, SDAR exhibits more favorable limited-data adaptation under these matched data budgets.

In conclusion, Tables 5–6 evaluate complete data-curation choices, Tables 7 and 15 analyze the stage-wise training pipeline, Table 8 provides a matched post-training comparison across model families, and the new data-scaling experiment directly examines the effect of SFT data exposure.

Thank you again for the constructive feedback. We hope the additional analyses and clarifications address your concerns.

Reviewer_9uXcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents DICE, a family of diffusion language models for CUDA kernel generation, with three different model sizes. To address the lack of high-quality CUDA kernel training data, the authors build the CuKe dataset and propose BiC-RL, a two-stage RL framework. The training starts with kernel infilling and then moves to end-to-end kernel generation. Experiments show that DICE improves correctness over its base models and achieves competitive results across different model sizes.

Critical strengths
  1. Using diffusion models for code generation is an interesting and relatively novel direction. The iterative generation process also seems quite natural for code refinement.
  2. The authors take reward hacking into account, which is a good consideration in RL-based training.
Critical weaknesses
  1. The reward function in BiC-RL does not directly optimize kernel performance.
  2. The benefit of using diffusion models is still unclear. The paper lacks a convincing comparison with AR-LLMs, and the broad SOTA claims seem somewhat overstated.
Questions for authors

This paper studies CUDA kernel generation using diffusion language models. I think the CuKe dataset, the two-stage RL framework, and the discussion of reward hacking are all useful. However, I have two main concerns.

Q1: Scaling and kernel performance.

From the experiments, DICE seems to work best at the 1.7B scale, while its advantage becomes smaller at 4B and 8B. Does this suggest that DICE may not scale as well as autoregressive LLMs? Another concern is kernel performance. In Table 7, DICE-8B improves correctness, but its Fast2 score drops quite significantly. Fast1 also drops noticeably on Lvl-1 and Lvl-3. Does this mean that the proposed method mainly improves correctness rather than kernel performance? I also noticed that the reward function only uses binary correctness. Could this explain why the generated kernels do not become faster? Since performance is one of the main goals of CUDA kernel optimization, I think this issue deserves more discussion.

Q2: Why diffusion models?

I am not fully convinced that diffusion models are better suited for CUDA kernel generation than autoregressive LLMs. In Table 8, DICE does not show a clear advantage over Qwen3, especially for larger models. The Introduction argues that CUDA kernels have long-range dependencies, such as shared memory and synchronization, and that bidirectional attention and non-sequential generation make diffusion models more suitable for this task. However, DICE uses block diffusion, which generates code within blocks rather than the entire sequence at once. It is unclear how this helps with long-range dependencies across blocks. I think the paper needs stronger evidence to support this motivation.

Also two smaller questions about the experiments:

  • In Figure-5, The paper discusses training efficiency, but the experiments only use the CuKe dataset. Would it be more useful to compare different amounts of training data (e.g., 10%, 20%, etc.) to better show the data efficiency of the method?

  • In Figure-1, I am not sure whether tokens/s is a fair metric for comparing the inference speed of diffusion and autoregressive models (maybe time to solution?). Also, different models use different inference engines, which may affect the comparison.

3 comments
AuthorsOpenReview ↗

Response Part 1/2

Dear Reviewer 9uXc,

Thank you for the detailed and constructive review. We address each question below and have revised the manuscript with additional analyses and clarifications.


Q1: Scaling and kernel performance.

We respectfully disagree that “its advantage becomes smaller at 4B and 8B.” We aggregate the results over all 250 KernelBench tasks across Levels 1–3 below. Each cell reports the overall percentage before and after the complete training pipeline, followed by the absolute change.

ScaleModel pipelineAggregate Exec (%)Aggregate fast₁ (%)Aggregate fast₂ (%)
1.7BQwen3 → Qwen3 + SFT + BiC-RL0.4 → 10.0 (+9.6)0.0 → 6.0 (+6.0)0.0 → 1.6 (+1.6)
SDAR-Chat → DICE0.0 → 14.0 (+14.0)0.0 → 7.2 (+7.2)0.0 → 2.8 (+2.8)
4BQwen3 → Qwen3 + SFT + BiC-RL15.6 → 14.4 (−1.2)6.8 → 8.4 (+1.6)2.4 → 3.2 (+0.8)
SDAR-Chat → DICE3.2 → 22.8 (+19.6)0.0 → 12.4 (+12.4)0.0 → 6.0 (+6.0)
8BQwen3 → Qwen3 + SFT + BiC-RL14.0 → 22.8 (+8.8)5.2 → 11.6 (+6.4)2.8 → 5.2 (+2.4)
SDAR-Chat → DICE3.2 → 35.6 (+32.4)0.8 → 14.8 (+14.0)0.4 → 4.0 (+3.6)

The DICE gains in aggregate execution correctness increase from 14.0 to 19.6 to 32.4 percentage points across the three scales, while the fast₁ gains increase from 7.2 to 12.4 to 14.0 points. The fast₂ gains remain positive at every scale, although they are not monotonic. Moreover, after full training, the aggregate execution-correctness margins of DICE over the corresponding Qwen3 models are 4.0, 8.4, and 12.8 points at 1.7B, 4B, and 8B, respectively. Within the evaluated 1.7B–8B range, we therefore do not observe diminishing gains with model scale.

Interpretation of the speedup metrics. We agree that the speedup improvements are not uniform. However, using SDAR-Chat-8B + SFT as the pre-RL reference, the statement that "Fast1 also drops noticeably on Lvl-1 and Lvl-3" may not match the reported values. On fast₁, Level 1 changes only from 16.0 to 15.0, while Levels 2 and 3 improve from 11.0 to 19.0 and from 2.0 to 6.0, respectively. On fast₂, Levels 1 and 3 improve from 4.0 to 6.0 and from 0.0 to 6.0, although Level 2 decreases from 8.0 to 1.0. These results show a favorable aggregate trend.

Correctness-only reward. We omit a speed reward because reliable latency measurement requires isolated execution, warm-up, and repeated runs, making it noisy and expensive during distributed RL; it also increases incentives to exploit evaluator loopholes, which Robust-KBench[1] shows can produce spurious 50–120×50\text{–}120\times speedups. We prioritize correctness as it is the current bottleneck and required by every fastp\mathrm{fast}_p metric. Aggregated over all scales and levels, Exec, fast1\mathrm{fast}_1, and fast2\mathrm{fast}_2 counts increase from 141/74/28 after SFT to 181/86/32 after BiC-RL. Although correctness is the direct RL objective, the aggregate numbers of correct kernels satisfying fast₁ and fast₂ also increase after BiC-RL.

[1] Towards Robust Agentic CUDA Kernel Benchmarking, Verification, and Optimization


Q2: Why diffusion models?

Matched comparison with Qwen3. We appreciate the reviewer raising this concern. As summarized in Q1 and reported in Table 8, DICE achieves higher aggregate execution correctness at all three scales, and its absolute margin over the corresponding Qwen3 model increases from 10 to 21 to 32 tasks as the model scale grows. DICE also performs better in 8 of the 9 scale–level comparisons, with 8B Level 3 being the only exception. Therefore, these results do not indicate that DICE’s advantage diminishes at larger scales.

Block size and long-range dependencies. Block size of block diffusion is a configurable architectural choice in SDAR. The original SDAR study trains and evaluates variants with B∈{4,8,16,32,64}B \in \{4,8,16,32,64\}, while its official implementation uses B=4B=4 by default. DICE inherits this checkpoint and default setting. Following SDAR’s block-size-scaling strategy, extending DICE to larger blocks could increase the span of within-block bidirectional modeling, and we regard this as a promising future direction.

AuthorsOpenReview ↗

Response Part 2/2


Q3: Data efficiency across different training-set sizes.

Thanks for your valuable suggestion. To directly evaluate data efficiency in our task-specific setting, we additionally fine-tuned SDAR-Chat-4B and Qwen3-4B using 25%, 50%, 75%, and 100% of CuKe under the same three-epoch SFT. Each table entry reports Overall Exec / fast1 / fast2 as an overall percentage across all 250 KernelBench tasks.

ModelCuKe SFT dataOverall Exec / fast₁ / fast₂ (%)
Qwen3-4B (base)0%15.6 / 6.8 / 2.4
SDAR-Chat-4B (base)0%3.2 / 0.0 / 0.0
Qwen3-4B + SFT25%7.2 / 5.6 / 1.6
SDAR-Chat-4B + SFT25%7.2 / 5.2 / 1.2
Qwen3-4B + SFT50%4.4 / 3.2 / 1.2
SDAR-Chat-4B + SFT50%15.2 / 8.0 / 2.8
Qwen3-4B + SFT75%6.4 / 3.2 / 0.8
SDAR-Chat-4B + SFT75%13.2 / 6.4 / 2.0
Qwen3-4B + SFT100%10.8 / 2.4 / 0.0
SDAR-Chat-4B + SFT100%22.0 / 13.6 / 4.4

At 25% data, SDAR matches Qwen3 in aggregate execution correctness while substantially improving over its own base checkpoint. From 50% onward, SDAR outperforms Qwen3 across all three aggregate metrics. Despite some non-monotonicity, these results indicate more favorable adaptation to CuKe for SDAR under the evaluated limited-data SFT setting.

The complete evaluation results and the corresponding data-scaling curves are provided in Table 10 and Figure 6 of Appendix A.2 in the revised manuscript.


Q4: Inference speed comparison.

Inference-speed metric. Tokens/s is a standard throughput metric used in recent dLLM studies, including WeDLM[2], Fast-dLLM[3], D2F[4], WINO[5], and ES-dLLM[6]. Several of these studies also use tokens/s to compare diffusion and autoregressive models directly.

Inference backends. The evaluated model families use different decoding procedures and attention mechanisms, and a single equally optimized backend is not available for all evaluated decoding algorithms under our experimental setup. We therefore use an optimized and model-compatible inference backend for each model family.

[2] WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference

[3] Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding

[4] Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcing

[5] Wide-In, Narrow-Out: Revokable Decoding for Efficient and Effective DLLMs

[6] ES-dLLM: Efficient Inference for Diffusion Large Language Models by Early-Skipping

Thank you again for the constructive feedback. We hope the additional analyses and clarifications address your concerns.

Thanks for the rebuttal, and I believe my questions are resolved.

Reviewer_UcEvOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper presents a dataset and a training recipe to improve diffusion large language models (dLLMs) on CUDA kernel writing. SDAR, a block dLLM, is fine-tuned for PyTorch to CUDA translation at 1.7B, 4B, and 8B scales. The SFT stage uses a dataset called CuKe, comprised of i) ConCuR, ii) the part of the cudaLLM SFT set with >2× speedup, iii) 36 kernels generated by Mercury Coder.

RL (BiC-RL) uses TraceRL with a binary correctness reward. It adds a short “infilling” stage, where the model writes only the kernel body inside a fixed scaffold to limit reward hacking, and an easy-to-hard curriculum labeled by Mercury Coder. DICE-8B reaches 47/35/14% execution correctness on KernelBench Levels 1–3. The paper motivates dLLMs for this task by faster decoding, bidirectional generation, and data efficiency.

Critical strengths

The output inspection in Table 14 is the most useful part of the paper. It shows that several baselines’ KernelBench scores are almost entirely shortcuts: DiffuCoder drops from 46% to 0% on Level 2, and LLaDA-Instruct from 26% to 0% on Level 3. The authors also report their own model’s Level 3 drop. Beyond that, it’s a reasonable data point that a fine-tuned 8B block-diffusion model, with no reasoning and a 4k-token budget, matches cudaLLM-8B on Levels 1–2. The authors also attempted the right control by running the same pipeline on an AR model.

Critical weaknesses

(W1) The motivation presented in the introduction is unsupported by evidence presented throughout the paper. Reviewing the proposed benefits of dLLMs as presented in the introduction:

  • Speed. Nothing in this argument is specific to kernels, and writing a kernel isn’t latency-bound. Generation cost could matter in RL rollouts and large-scale sampling. But there each candidate also has to be compiled and benchmarked, which typically costs more than decoding it, and these workloads run at large batch sizes where AR decoding is efficient. Figure 1 instead measures batch size 1, with a different inference engine per model family. It compares against reasoning models that emit up to 32k thinking tokens, so any latency advantage comes mostly from DICE and not reasoning. The core issue is that the speed of dLLMs doesn't seem relevant to writing cuda kernels.

  • Non-causal attention. DICE decodes in blocks of 4 and is causal across blocks, so its bidirectional context is four tokens. I don't see how that helps with dependencies like shared-memory layout and synchronization, which span hundreds of tokens. Plots e.g. figure 2 show essentially causal left-to-right generation.

The infilling stage puts the prefix and suffix in the prompt (A.6.1). That is ordinary conditional generation, which any AR model can do, and FIM-trained coders support true infilling anyway. Among untrained baselines, the fully bidirectional dLLMs (LLaDA, Dream, DiffuCoder) are the weakest, which is the opposite of the paper's argument.

  • Data efficiency. The paper presents one SFT run per model, at a single dataset size, for 3 epochs. There’s no curve over data quantity or repetition. The cited work (Ni et al.; Prabhudesai et al.) is about pretraining over many repeated epochs, not a few epochs of task SFT.

The only direct test is the matched AR comparison in Table 8, and its AR arm seems broken since training makes Qwen3 worse than the untrained model. On Level 2, Qwen3-8B scores 25% untrained, 13% after SFT on CuKe, and 14% after the full pipeline. Qwen3-4B goes from 29% to 13% to 11%. The two arms also use different RL algorithms with untuned hyperparameters, and Qwen3’s thinking mode and token budget aren’t reported. Meanwhile, cudaLLM-8B, an AR model trained with SFT and RL, already matches DICE on Levels 1–2 (e.g. 46/34 vs. 47/35).

(W2) Incremental contributions

The dataset contributions seem marginal. CuKe is the ConCuR dataset as-is combined with a speed-filtered slice of the public cudaLLM data, plus 36 Mercury Coder samples (amounting to 0.6% of the dataset). Against ConCuR specifically, it gains 2–3 tasks per level, with 30% more data. There doesn't seem to be a size-matched control. The 2.0× speedup threshold was chosen on KernelBench itself.

BiC-RL is TraceRL, a GRPO-style objective, plus 20 steps of a scaffolded subtask and easy-to-hard data ordering. It isn’t compared against the obvious alternatives: putting the authors’ own shortcut detector in the reward, or zero-variance group filtering. The ablations are inconsistent. At 4B, DICE is below plain SFT on Level 1 and identical to infilling-only on Levels 1 and 3. At 8B, the RL baseline beats DICE on Level 3. Since the reward ignores speed, RL also lowers 8B Level 2 fast₂ from 8.0 to 1.0.

The evaluation can’t resolve small differences reported. Results come from a single greedy sample on 100/100/50 tasks with no confidence intervals, and DICE-8B’s margin over cudaLLM-8B on Levels 1–2 is one task each. The speedup measurements are also questionable. Each of the three showcased Level 3 speedups (2-4x) comes from replacing one cheap op (a ReLU, a softmax, a global average pool) while everything else stays in PyTorch. Could this point to a timing artifact (the timing protocol isn’t described)? “Significantly outperforms” and “new state-of-the-art” in the abstract aren’t supported.

Questions for authors

Which decoding mode produced the accuracy numbers, and which produced the throughput numbers? With top-k = 1 at evaluation, how were pass@10 samples drawn?

Can you give intuition on selecting Mercury Coder, and how were difficulty labels assigned?

4 comments
AuthorsOpenReview ↗

Response Part 1/4

Dear Reviewer UcEv,

Thank you for the detailed and constructive review. We address each concern and question below and have revised the manuscript with additional experiments, implementation details, and claims in the abstract.


W1(a): Speed.

We clarify the scope of our speed claim, the throughput setting in Figure 1, and the additional large-batch results below.

Relevance to the CUDA kernel workflow. We can decompose the latency of the kernel-generation-and-verification workflow as Ttotal=Tgeneration+TverificationT_{\mathrm{total}} = T_{\mathrm{generation}} + T_{\mathrm{verification}}, where TgenerationT_{\mathrm{generation}} is the time required by the model to generate CUDA kernel candidates, and TverificationT_{\mathrm{verification}} includes compilation and benchmarking. Holding the verification workload fixed, any reduction in TgenerationT_{\mathrm{generation}} translates into an equal absolute reduction in TtotalT_{\mathrm{total}}. Faster decoding reduces one concrete component of the overall workflow, while its relative end-to-end impact depends on the fraction of time spent on generation.

Clarification of Figure 1. The throughput values in Figure 1 were not measured using the 32K-token evaluation budget of the reasoning baselines. The 32K budget applies only to task-performance evaluation, whereas all throughput measurements use the same maximum generation budget of 1,024 tokens. We have revised Figure 1.

Large-batch throughput. To address the concern that AR decoding becomes more efficient at larger batch sizes, we additionally measured throughput at batch sizes 2, 4, and 8 across all three model scales. The absolute results at the 8B scale are:

Batch sizeQwen3-8B (tokens/s)DICE-8B (tokens/s)Ratio (DICE/Qwen3)
173.5583.071.13×
2155.65155.270.99×
4306.84346.381.13×
8600.15727.811.21×

At the 8B scale, DICE is essentially on par with Qwen3 at batch size 2, while achieving 1.13× and 1.21× higher throughput at batch sizes 4 and 8, respectively. Thus, the throughput advantage is not limited to batch-size-1 serving. Inference speed description and the complete throughput results (Table 9) are provided in Appendix A.1 of the revised manuscript.

Inference backends. The evaluated model families use different decoding procedures and attention mechanisms, and a single equally optimized backend is not available for all evaluated decoding algorithms under our experimental setup. We therefore use an optimized and model-compatible inference backend for each model family.


W1(b): Non-causal attention.

Block size is a configurable architectural choice in SDAR [1]. The original SDAR study trains and evaluates variants with B∈{4,8,16,32,64}B \in \{4, 8, 16, 32, 64\}, and its official implementation uses B=4B=4 as the default setting. DICE inherits this default checkpoint and setting. Following the block-size-scaling training strategy of SDAR, DICE is in principle extensible to larger blocks, which would expand the span of within-block bidirectional modeling and better capture longer-range dependencies.

[1] SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation


W1(c): Kernel infilling and untrained baselines.

Kernel-infilling stage. Unlike generic FIM pretraining, which teaches broad span-completion capabilities, our kernel-infilling stage uses a CUDA-specific executable scaffold. The provided prefix and suffix fix the imports, compilation and binding logic, ModelNew wrapper, and invocation path, while the model generates only the missing CUDA/C++ core. The completed program receives a positive reward only if it compiles and matches the reference output. This construction narrows the initial RL exploration space and restricts the three shortcut patterns described in Section 4.2, before the subsequent end-to-end stage restores responsibility for generating the complete program. Tables 16 in our paper provides supporting evidence for this intended effect.

Interpretation of the untrained baselines. We agree that LLaDA, Dream, and DiffuCoder perform poorly on this task. However, these results do not rule out the potential benefits of bidirectional attention for CUDA kernel generation and therefore do not contradict our motivation. Table 1 is intended to compare the overall CUDA-generation capabilities of different models, rather than to isolate the effect of their attention mechanisms. These models differ substantially in their pretraining data, code capabilities, instruction tuning, and domain specialization. Therefore, their performance cannot be causally attributed to bidirectional attention alone.

AuthorsOpenReview ↗

Response Part 2/4

Indeed, weak performance is not unique to the fully bidirectional models. The autoregressive Seed-Coder-Reasoning baseline achieves execution correctness of only 0.0, 2.0, and 0.0 across the three levels. It would be equally invalid to infer from this result that autoregressive modeling is unsuitable for CUDA kernel generation. Similarly, the results of LLaDA, Dream, and DiffuCoder show that bidirectional attention alone is not sufficient to solve this highly specialized task, but they do not show that the mechanism is unhelpful.


W1(d): Data efficiency.

Scope of the comparison. Our intention here is not to draw a direct comparison with findings from pretraining, but rather to offer an analogy. As we explicitly state in Appendix A.2: "Although our setting focuses on task-specific fine-tuning rather than pre-training, this trend is qualitatively consistent with prior evidence that diffusion language models are more data-efficient under limited-data conditions."

Data-quantity experiment. To directly evaluate data efficiency in our task-specific setting, we additionally fine-tuned SDAR-Chat-4B and Qwen3-4B using 25%, 50%, 75%, and 100% of CuKe under the same three-epoch SFT. Each table entry reports Overall Exec / fast1 / fast2 as an overall percentage across all 250 KernelBench tasks.

ModelCuKe SFT dataOverall Exec / fast₁ / fast₂ (%)
Qwen3-4B (base)-15.6 / 6.8 / 2.4
SDAR-Chat-4B (base)-3.2 / 0.0 / 0.0
Qwen3-4B + SFT25%7.2 / 5.6 / 1.6
SDAR-Chat-4B + SFT25%7.2 / 5.2 / 1.2
Qwen3-4B + SFT50%4.4 / 3.2 / 1.2
SDAR-Chat-4B + SFT50%15.2 / 8.0 / 2.8
Qwen3-4B + SFT75%6.4 / 3.2 / 0.8
SDAR-Chat-4B + SFT75%13.2 / 6.4 / 2.0
Qwen3-4B + SFT100%10.8 / 2.4 / 0.0
SDAR-Chat-4B + SFT100%22.0 / 13.6 / 4.4

At the 25% setting, SDAR improves over its base checkpoint by +4.0, +5.2, and +1.2 percentage points on overall Exec, fast₁, and fast₂, respectively. In contrast, although Qwen3 starts from a stronger base checkpoint, its corresponding metrics change by −8.4, −1.2, and −0.8 percentage points after SFT. Despite some non-monotonicity, SDAR therefore exhibits more favorable limited-data adaptation.

The complete evaluation results and the corresponding data-scaling curves are provided in Table 10 and Figure 6 of Appendix A.2 in the revised manuscript.


W1(e): Matched AR comparison.

SFT configurations and Qwen3 training behavior. Our SFT settings are based on the public training recipes for the corresponding model families. For Qwen3, we follow the official LlamaFactory Qwen3 SFT recipe, modifying only the dataset-related fields (dataset, cutoff_len, and max_samples) to accommodate CuKe. For SDAR, we similarly follow its official training recipe and modify only the dataset-related fields. Both families are trained for 3 epochs with a learning rate of 1×10−51\times10^{-5}. Thus, both arms use reproducible family-specific implementations.

We notice the reported Level 2 results. However, the Level 2 decrease does not indicate a global training failure for Qwen3-8B:

  • On Level 1, Qwen3-8B improves execution correctness from 8.0 to 25.0 to 34.0 across the base, SFT, and full-pipeline stages.
  • On Level 3, it improves from 4.0 to 16.0 to 18.0.
  • Across all 250 KernelBench tasks, its overall execution correctness increases from 14.0 to 18.4 to 22.8.

Relative to the untrained Qwen3-4B base model, the full pipeline improves 5 of the 9 reported metrics. The result therefore reflects non-monotonic metric variations rather than a broken AR training arm.

RL implementation. The mechanisms of AR LLMs and dLLMs are different. Consequently, the same TraceRL implementation cannot be directly reused for AR LLMs. We therefore use GRPO for Qwen3. Apart from this model-specific implementation difference, we align training configurations like the CUDA training data, bi-phase task construction, data schedule, reward function, number of sampled problems, training steps, and so on.

Thinking mode and output budget. We perform RL on Qwen3 in thinking mode, and the token budget is also 32,768.

Comparison with cudaLLM. cudaLLM is a strong AR baseline. But our intended point is not that DICE uniformly dominates cudaLLM, but that it achieves this performance from a weaker base model and with a substantially smaller RL dataset. As reported in Table 14, we only use approximately 6.9% of cudaLLM’s RL data, while achieving comparable or even better performance.

AuthorsOpenReview ↗

Response Part 3/4


W2(a): CuKe contribution and threshold selection.

CuKe contributions. The contribution of the CuKe dataset is primarily the standardized remeasurement and performance-based curation of public CUDA kernel data, together with a set of structurally targeted examples, rather than dataset scale alone. We have several observations why performance-based curation is necessary:

  • After remeasuring the cudaLLM SFT data, we find that 14% of its kernels are slower than the corresponding PyTorch implementations, while only 16% achieve more than 2.0×2.0\times speedup (Table 18). This observation is consistent with EvoEngineer[2], which re-evaluates the released Sakana kernel dataset[3] and reports that its median speedup decreases from 1.13×1.13\times to 0.82×0.82\times, while the number of successful tasks decreases from 63 to 22.

  • More broadly, prior work has identified the scarcity of openly available, high-performance CUDA kernels as a central obstacle to kernel-generation research[4,5].

Threshold selection. We clarify that no KernelBench task, solution, or generated kernel is included in CuKe or used as training data, so the threshold selection does not introduce training-data leakage.

Importantly, the conclusion is not dependent on a single sharply optimal threshold. The 1.5×1.5\times and 2.5×2.5\times variants remain competitive, and both obtain stronger results than 2.0×2.0\times on some Level 3 metrics. The 2.0×2.0\times variant instead provides the strongest aggregate results on Levels 1 and 2. We therefore select it as a practical default under the evaluated setting.

[2] EvoEngineer: Mastering Automated CUDA Kernel Code Evolution with Large Language Models
[3] Sakana AI, AI-CUDA-Engineer-Archive [4] ConCuR: Conciseness Makes State-of-the-Art Kernel Generation
[5] Towards Automated Kernel Generation in the Era of LLMs


W2(b): BiC-RL design, alternatives, and ablations.

BiC-RL contributions. Our contribution is not a new policy objective. Rather, BiC-RL organizes training along two CUDA-specific curriculum dimensions. In the training dimension, the model first completes the CUDA kernel core within a fixed scaffold and then transitions to end-to-end kernel generation. In the data dimension, task complexity progresses from basic operations through fused operations to complete model structures.

Zero-variance filtering and shortcut-detector reward. Zero-variance group filtering is already implemented: groups in which all sampled responses receive the same correctness score, including all-correct and all-incorrect groups, are explicitly excluded before policy updates. We have clarified this in Section 4.2. We agree that a detector-augmented reward is a useful alternative, but due to the limited time window of rebuttal, we will leave the comparison to future work.

BiC-RL ablation results. Compared to single-stage RL, DICE improves 20 of 27 metric comparisons, ties on 2, and decreases on 5; execution correctness improves in 7 of 9 comparisons. At 8B, the total number of correct kernels increases from 58 to 89. We will therefore describe the result as a favorable aggregate trend.

Correctness-only reward. We omit a speed reward because reliable latency measurement requires isolated execution, warm-up, and repeated runs, making it noisy and expensive during distributed RL; it also increases incentives to exploit evaluator loopholes, which Robust-KBench[6] shows can produce spurious 50–120×50–120\times speedups. We prioritize correctness as it is the current bottleneck and required by every fastp\mathrm{fast}_p metric. Aggregated over all scales and levels, Exec, fast1\mathrm{fast}_1, and fast2\mathrm{fast}_2 counts increase from 141/74/28 after SFT to 181/86/32 after BiC-RL.

[6] Towards Robust Agentic CUDA Kernel Benchmarking, Verification, and Optimization

AuthorsOpenReview ↗

Response Part 4/4


W2(c): cudaLLM comparison and timing protocol.

Reporting benchmark results without confidence intervals is common in related CUDA kernel generation works[6, 7].

Across all 250 tasks, DICE-8B solves 89 tasks, compared with 83 tasks for cudaLLM-8B. The more informative conclusion is therefore that DICE-8B achieves competitive overall correctness and better observed Level 3 performance, despite using only 4,992 RL samples, compared with 71,996 for cudaLLM (approximately 6.9%), and starting from the substantially weaker SDAR-Chat-8B base model.

Timing protocol. We follow the official KernelBench v0.1 evaluation protocol. For each candidate, functional correctness is evaluated on five randomized input trials. Runtime is measured for the complete ModelNew.forward, using three warm-up runs followed by 100 timed runs with CUDA events and explicit device synchronization. The PyTorch reference Model.forward is profiled on the same GPU using the same warm-up and timing procedure, and speedup is computed as the ratio of their mean execution times. Compilation is completed before profiling and is not included in either runtime.

Claim correction. Thank you for raising this point. We have removed them from the abstract and revised the claim to state that DICE outperforms existing diffusion LLMs of comparable scale and matches or exceeds strong autoregressive baselines across multiple evaluation settings.

[6] Kevin: Multi-Turn RL for Generating CUDA Kernels

[7] CUDA-L1: Improving CUDA Optimization via Contrastive Reinforcement Learning


Q1: Decoding configurations.

The accuracy numbers are obtained using static decoding. The throughput results are measured using dynamic decoding. The reported top-k=1 setting applies only to the static single-sample evaluation. The pass@10 results were obtained separately using SDAR’s official dynamic decoding configuration, with temperature=1.0, top-p=1.0, and top-k=0.


Q2: Mercury Coder selection and difficulty labels.

We selected Mercury Coder because it is introduced as the first commercial-scale dLLM and is specifically optimized for code generation. It therefore provides a capable and practically accessible diffusion-based generator aligned with our goal of studying dLLMs for CUDA kernel generation.

Difficulty labels. We prompted Mercury Coder to assign each PyTorch–CUDA pair to one of the three levels (single operations, fused operations, model structures). We provide the complete prompt in Appendix A.7.3 of the revised manuscript.

Thank you again for the constructive feedback. We hope the additional experiments and clarifications address your concerns.

Comments on the paper

1 comment

Dear Reviewers,

This is a gentle reminder that only two days remain in the rebuttal/discussion period. Please take some time to review the authors' responses and engage in the discussion, especially if there are any outstanding questions or concerns, and update your assessment if appropriate based on the authors' responses.

Your timely feedback is greatly appreciated and will be helpful in reaching a well-informed final decision.

Thank you for your time and effort!

Best regards, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.