acceptodds
Under review as a conference paper at ICLR 2027

Multi-turn RL with Structural and Performance Aware Rewards for CUDA Kernel Generation

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a powerful technique to enhance the reasoning capacity of LLMs for optimized code generation. However, existing RLVR approaches primarily rely on outcome-based signals such as correctness and speedup, overlooking performance-critical structural properties of programs that are essential for generating optimized code. In this work, we propose \ourtool, a reflective RL framework that incorporates both verifiable execution rewards and structural code-aware rewards derived from parallelization features (e.g., memory coalescing, occupancy, Arithmatic Intensity, and synchronization patterns). \ourtool operates in two stages: (1) an offline pairwise ranking module that learns to distinguish strong and weak program candidates via contrastive comparisons, and (2) an online RL training phase that jointly optimizes for correctness, performance, and structural efficiency through a unified reward signal. To further enhance learning, \ourtool utilizes iterative refinement using execution feedback enabling progressive improvement of generated candidates. We also introduce a dataset comprising 2.9k C CUDA and 1k PyTorch CUDA programs, each paired with diverse input configurations and multiple CUDA implementations encompassing diverse optimization strategies. \ourtool is evaluated across multiple benchmarks comprising both C CUDA and PyTorch CUDA transformations. Empirical findings suggest that \ourtool significantly outperforms strong baselines, including Qwen-3-32B (for C CUDA) and CUDA Agent (for PyTorch CUDA) by achieving up to 5X & 3.32X improvements in speedup, and 17% & 7% improvements in correctness, respectively.

Primary area:
reinforcement learning
Keywords:
LLM, Multi-turn RL, HPC, CUDA, Code Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Tob3OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CUDAPERF, a reinforcement learning framework for CUDA program generation that augments outcome-based verifiable rewards with structure-aware rewards derived from CUDA programs. Specifically, it trains an offline pairwise ranker over performance-relevant structural features, such as memory coalescing, arithmetic intensity, occupancy, control divergence, synchronization, and atomic operations, and combines the resulting structural score with execution-based signals for correctness and speedup during online multi-turn RL. Experiments on C→CUDA and PyTorch→CUDA translation show that CUDAPERF improves the performance of Qwen models, particularly Qwen-3-32B, and outperforms several existing baseline models in correctness and execution speed.

Critical strengths
  1. Strong practical motivation. Automatically generating high-performance CUDA kernels is valuable for accelerating scientific computing and machine-learning workloads. Improving LLM-based CUDA generation therefore addresses a meaningful systems problem with clear practical relevance.

  2. Addresses the sparsity and coarseness of outcome-based rewards. Existing RL-based code-generation methods often rely primarily on signals such as compilation success and test-case correctness, which can be sparse and provide limited guidance for improving performance. CUDAPERF augments these signals with CUDA-specific structural features, including memory coalescing, arithmetic intensity, occupancy, control divergence, synchronization, and atomic operations. These features provide a more informative learning signal, helping guide the policy toward generating CUDA programs that are both functionally correct and computationally efficient.

  3. Provides evidence that structural and verifiable rewards are complementary. The ablation study compares structural-only, verifiable-only, and joint rewards across the CUDAPERF, BabelTower, and KernelBench datasets. The joint reward consistently achieves the strongest correctness and speedup results, indicating that structural and verifiable rewards provide complementary benefits.

Critical weaknesses
  1. Lacks direct comparisons with other RL frameworks and reward designs. The main results in Table 1 compare CUDAPERF primarily with different models, such as OpenAI O4-mini and Qwen2.5-Coder-32B. However, the paper does not compare CUDAPERF with alternative RL frameworks or reward designs under the same base policy model and training data. This makes it difficult to determine whether the reported gains are specifically attributable to the proposed structural RL framework.

  2. Uncontrolled comparison with baselines. CUDAPERF is obtained by fine-tuning Qwen-3-32B, while the proposed CUDAPERF dataset already shows Qwen-3-32B outperforming all other reported baselines in both correctness and speedup. Therefore, CUDAPERF’s superior performance on this dataset is partly expected from inheriting the capabilities of a stronger base model. More importantly, the comparison with QiMeng-MuPa is not controlled at the backbone level. As reported in Appendix A.16, the authors use the translator-Llama-3-8B model for QiMeng-MuPa, which is a fine-tuned version of Meta-Llama-3-8B, whereas CUDAPERF is trained from Qwen-3-32B. The substantial difference in model scale and backbone makes it difficult to attribute the performance gap to the proposed RL framework.

  3. The test-time multi-turn protocol is insufficiently specified. Section 2.2.2 describes a multi-turn refinement process during online RL training, in which compilation errors, failed test cases, speedup measurements, and structural features are fed back into subsequent prompts. However, the paper does not specify what information is provided at test time, or whether the baseline models receive the same feedback and refinement opportunities. This makes it difficult to determine whether the reported gains are attributable to the proposed reward design or to differences in the test-time interaction protocol.

  4. The generalization of the structural ranker is not sufficiently validated. The ranker learns a mapping from a CUDA program’s structural feature vector to a scalar score, where the induced ordering is expected to match the candidates’ measured speedup. However, the paper does not evaluate whether this mapping generalizes to unseen source programs, newly generated CUDA candidates, different workloads, or different hardware settings. The reported training loss is insufficient to establish such generalization, since the ranker may simply learn dataset-specific correlations between the handcrafted features and speedup.

  5. The paper overlooks recent work on partial-success verifiable rewards. CUDAPERF uses the test-case pass rate directly in its correctness penalty, i.e., Rcorr=−αwrong(1−p(y))R_{\text{corr}}=-\alpha_{\text{wrong}}(1-p(y)) for partially correct programs. Recent work (Beyond Binary: https://arxiv.org/pdf/2601.03525) has shown that naive pass-rate rewards, despite providing denser supervision than binary outcome rewards, may suffer from cardinality bias: frequently passed or easy test cases can dominate the learning signal, while progress on difficult frontier tests receives relatively little credit. The authors should discuss this issue and clarify whether it applies to their CUDAPERF.

Questions for authors
  1. Can the authors compare CUDAPERF with alternative RL frameworks?One possible experiment would be to train Qwen-3-32B using CUDAPERF and an existing RL framework/reward configuration under the same training data, and then compare both the training cost (e.g., total GPU hours) and final performance using the metrics reported in Table 1.
  2. Can the authors compare CUDAPERF with QiMeng-MuPa under a controlled setting? Since the paper evaluates QiMeng-MuPa using the translator-Llama-3-8B model, one possible experiment would be to train CUDAPERF starting from the same Meta-Llama-3-8B backbone and compare it with translator-Llama-3-8B under the same data and evaluation protocol. Alternatively, if feasible, the authors could implement QiMeng-MuPa using Qwen-3-32B and compare it directly with CUDAPERF based on Qwen-3-32B. Such comparisons would help disentangle the effects of the underlying model backbone from those of the proposed training framework.
  3. Can the authors clarify the exact test-time multi-turn protocol for both CUDAPERF and all baseline models? In particular, please specify the prompt contents, feedback provided at each refinement step, number of refinement turns, etc.
  4. Can the authors evaluate the structural ranker on source programs and CUDA candidates held out entirely from training, and report a simple metric such as pairwise ranking accuracy or Spearman correlation with measured speedup?
  5. Can the authors discuss recent work on pass-rate-based reward design? In particular, they should discuss methods that address the potential bias and limitations of using a naive test-case pass rate as a reward.
1 comment
AuthorsOpenReview ↗

Response to Reviewer Tob3

We appreciate the reviewer's detailed suggestions and acknowledgment of the importance of structural rewards.

1. Controlled RL Comparison: Table 3 already provides a controlled experiment using the same Qwen-3-32B backbone, comparing structural-only, verifiable-only, and combined rewards. Across all three datasets, the combined reward consistently achieves the highest correctness and speedup. Appendix A.15 also reports training costs.

2. Backbone Fairness: We acknowledge that QiMeng-MuPa uses a different backbone, as explicitly documented in Appendix A.16. Our central evidence for the structural reward is therefore the controlled ablation in Table 3, rather than cross-backbone comparisons.

3. Multi-turn Evaluation: Section 2.2.2 describes the refinement feedback, including compilation errors, failed test cases, speedup measurements, and structural properties. Appendix A.17 already compares single-turn and multi-turn training reward progression, showing lower reward saturation for single-turn training.

4. Ranker Generalization: We agree that training loss alone is insufficient to establish generalization. We will report pairwise ranking accuracy and Spearman correlation on entirely held-out programs and candidates in the revised submission.

5. Partial-success Rewards: We appreciate the reference to Beyond Binary. We acknowledge that pass-rate-based rewards may suffer from cardinality bias and will discuss this limitation and investigate frontier-aware reward weighting in the camera-ready version.

Reviewer_vFvmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents CUDAPerf, a multi-turn RL framework for CUDA kernel generation that combines execution-based rewards with performance rewards derived from code structure. What makes CUDAPerf different from other feedback-driven kernel generation frameworks is that it uses static analysis and an offline-trained pairwise ranker to capture structural properties of CUDA programs. The learned structural reward is then incorporated into the online, feedback-driven, multi-turn RL process. The authors evaluate CUDAPerf on both C-to-CUDA and PyTorch-to-CUDA translation tasks. The results show improvements in both correctness and execution performance over several baselines.

Critical strengths
  1. The paper introduces a new structural reward into RL, providing richer information than traditional approaches that rely only on execution outcomes.

  2. The proposed approach is relatively easy to integrate and could potentially work well with other kernel generation frameworks.

Critical weaknesses
  1. The current ablation studies and experimental evidence are not sufficient to fully establish the effectiveness of the proposed structural reward.

  2. The structural reward seems to depend heavily on the similarity between training and evaluation workloads, raising concerns about its generalization.

Questions for authors

Q1. My biggest concern with this paper is the soundness of the structural reward, which is one of its main contributions. Some of the structural features used in the reward are rather rough approximations of GPU execution behavior, rather than actual measurements. For example, occupancy is mainly estimated from threads per block, while actual occupancy also depends on register usage, shared memory, and hardware resource limits. Similarly, arithmetic intensity and memory coalescing are estimated through simplified static analysis.

I'm not fully convinced that a ranker trained on these features can generalize well, especially since it does not explicitly account for the interactions between different optimization patterns or their dependence on input sizes and workload configurations. In practice, some optimization patterns only work well when combined with others, and the same pattern can have very different performance effects across different workloads.

This makes me question how well such a feature-based ranker can generalize beyond its training data. I also wonder whether we could achieve similar improvements by simply incorporating these CUDA optimization guidelines into an agent as a knowledge base, rather than learning a separate structural reward model.

Q2. I also have some concerns about the reliability and fairness of the experimental evaluation. The paper mentions that part of the offline ranker's training data comes from BabelTower and CUDA Agent, while BabelTower and KernelBench are also used for evaluation. In addition, the performance-aware few-shot examples are sampled from the authors' curated dataset.

It is not clear whether there is any overlap between the data used for offline ranker training, policy training, few-shot prompting, and final evaluation.

Furthermore, CUDAPerf uses several techniques beyond RL training, including performance-aware few-shot prompting, multiple candidate generations, and up to five rounds of execution-feedback-based refinement, but the paper does not clearly explain whether the baselines are evaluated with comparable generation budgets, refinement rounds, and opportunities to use execution feedback. For example, are other agent-based baselines given the same CUDA structural knowledge through their prompts?

It is hard to tell whether the improvements really come from the learned structural reward, or simply from the fact that CUDAPerf has access to CUDA-specific optimization knowledge that the other models do not.

1 comment
AuthorsOpenReview ↗

Response to Reviewer vFvm

We thank the reviewer for the constructive and useful feedback.

1. Structural Reward Soundness: We agree that static structural features approximate rather than fully capture GPU execution behavior. However, our approach does not assign fixed performance values to individual features. Instead, an MLP learns their combined influence through pairwise comparisons of CUDA implementations ranked by measured speedup.

2. Structural Reward vs. Optimization Guidelines: Importantly, Appendix A.19 already incorporates CUDA optimization guidelines directly into the policy prompts. Furthermore, Table 3 isolates the structural reward by disabling individual reward components while retaining the same backbone and overall framework. The consistent improvements demonstrate the additional benefit of the learned structural reward beyond textual optimization guidance.

3. Dataset Overlap and Evaluation Fairness: We acknowledge that the dataset separation and baseline execution budgets require clearer documentation. We in all cases make sure that there are no leakage of training data in the evaluation. For example, BabelTower has a separate evaluation set of 233 c->CUDA pairs. So, we use that evaluation set for evaluation and rest for training.

Reviewer_SV4GOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CUDAPerf, a reinforcement learning framework for translating C and PyTorch programs into efficient CUDA implementations. Its main contribution is a learned structural reward that complements execution-based correctness and performance rewards. An offline MLP ranker learns pairwise preferences between CUDA implementations using features intended to capture memory coalescing, arithmetic intensity, occupancy, divergence, and synchronization. The ranker then provides an additional reward during multi-turn GRPO training, where the policy iteratively refines programs using execution and structural feedback.

The authors curate approximately 2.9k C-to-CUDA and 1k PyTorch-to-CUDA tasks and evaluate on their held-out dataset, BabelTower, and KernelBench. They report improvements in correctness and geometric-mean speedup, with ablations on reward components and policy model size.

Critical strengths
  • Combining structural program information with execution rewards is a useful and novel research direction. Execution rewards identify whether a program works and how fast it runs, but provide limited guidance about which implementation choices are responsible. A lightweight structural ranker could provide complementary guidance during learning. This is a concrete, testable contribution to performance-oriented code generation.
  • The reward ablations provide encouraging evidence for the proposed combination. Table 3 reports improvements over execution-only rewards across all three evaluation datasets. For example, on KernelBench, adding structural rewards increases reported correctness from 84% to 93% and geometric-mean speedup from 4.3× to 6.41×. These comparisons are more informative about the proposed contribution than comparisons against untuned base models alone.
Critical weaknesses
  • The evaluation protocol and comparison fairness are unclear. The relationship between multi-turn refinement and pass@5 is insufficiently explained, including whether baselines receive comparable generation budgets and execution feedback. The paper also does not clearly specify how failed tasks enter the geometric-mean speedup calculation or establish task-level separation between data construction and evaluation. These uncertainties make the reported gains difficult to assess.

  • The structural reward lacks sufficient validation. Several features are coarse approximations of GPU performance, such as estimating occupancy from threads per block and arithmetic intensity from static code counts. The paper does not report held-out ranking accuracy or runtime correlation, leaving unclear whether the ranker generalizes to unseen programs. The calibration of structural scores against execution rewards and their application to incorrect programs also require clarification.

  • The baseline coverage is incomplete, particularly for recent agentic kernel-optimization frameworks. Although CUDA Agent is included, relevant approaches such as StitchCUDA [1], CUDAForge [2], and KernelPro [3] are missing from the evaluation. These frameworks explore iterative feedback, profiling, or explicit search, and some also incorporate signals beyond execution outcomes, such as arithmetic intensity or rubric-based rewards. Comparisons with these methods would help establish both the empirical advantage of CUDAPerf and the distinct contribution of its structural reward.

[1] StitchCUDA: An Automated Multi-Agents End-to-End GPU Programing Framework with Rubric-based Agentic Reinforcement Learning, ICML 2026

[2] Cudaforge: An agent framework with hardware feedback for cuda kernel optimization, Zhang et al.

[3] Optimizing CUDA like a Human: Micro-Profiling Tools as Expert Surrogates for LLM-Based GPU Kernel Optimization, Gai et al.

Questions for authors
  • Clarification—structural reward: What is the ranker’s accuracy on held-out tasks, and how is its output scaled relative to execution rewards? Is structural reward applied to incorrect or uncompilable programs?
  • Under matched training and inference budgets, how does multi-turn training compare with single-turn training on held-out correctness and runtime?
  • How were the two proposed datasets (C-to-CUDA and PyTorch-to-CUDA) curated? Please clarify the source composition, candidate-generation procedure, correctness and performance filtering criteria, and deduplication process. How were training and evaluation splits constructed to prevent overlap with BabelTower and KernelBench? These details are necessary to assess dataset quality and the validity of the evaluation.
1 comment
AuthorsOpenReview ↗

Response to Reviewer SV4G

We thank the reviewer for the constructive feedback and for recognizing the novelty of our structural reward.

1. Structural Reward Effectiveness: Table 3 already provides a controlled ablation using the same Qwen-3-32B backbone. On KernelBench, combining structural and verifiable rewards improves correctness from 84% to 93% and speedup from 4.3× to 6.41×, demonstrating the complementary contribution of structural rewards. Our ranker learns preferences from measured speedups rather than relying on manually assigned feature weights. We will additionally report held-out ranking accuracy, Spearman correlation, reward calibration, and the treatment of invalid candidates in the revised submission.

2. Evaluation Protocol: Table 4 reports pass@1, pass@3, and pass@5 results. Section 2.2.2 describes up to five refinement turns during training. Also, multi-turn vs single-turn results are shown in Figure 5 in Appendix.

3. Dataset Construction: Section 3 describes generating multiple CUDA implementations and retaining programs with at least five correct variants and one achieving ≥1.5× speedup.

4. Missing Baselines: We appreciate the suggested references and will incorporate comparisons with StitchCUDA, CUDAForge, and KernelPro where reproducible implementations are available in the camera-ready as these experiments will take time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.