acceptodds
Under review as a conference paper at ICLR 2027

CUDA-Harness: Harnessing Agentic CUDA Kernel Generation and Optimization from Natural Language

Abstract

Developing high-performance CUDA kernels demands specialized knowledge in algorithm implementation, correctness validation, and hardware-aware parallel optimization, creating a substantial expertise barrier and making generating CUDA kernels directly from natural language (Text2CUDA) essential. Meanwhile, the general-purpose code generation capability of Large Language Models (LLMs) prompts a series of works exploring LLM-based CUDA kernel generation. They mainly focus on transpilation from high-level frameworks such as PyTorch to CUDA (Torch2CUDA) rather than Text2CUDA, where models must understand the high-level input semantics and handle low-level kernel implementation and validation. Additionally, these methods are vulnerable to reward hacking due to reliance on predefined test inputs. In this paper, we propose CUDA-Harness, a framework for harnessing agentic CUDA kernel generation and optimization from natural language. Specifically, we introduce Intermediate-Structured Generation to connect high-level semantic understanding with low-level kernel generation. To dilute reward hacking in Text2CUDA, we construct Synthesis-Based Verification to provide isolated test data and progressive validation. Furthermore, we propose Feedback-Adaptive Evolution, a kernel evolution strategy that prioritizes correctness while optimizing performance. Finally, through extensive experiments, we demonstrate the effectiveness of CUDA-Harness, with further evaluations illustrating generalization across LLMs, hardware platforms, and to C-to-CUDA transpilation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Agentic Harness, Text2CUDA, Agentic Code Generation, CUDA Kernel Generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.60 over 5 reviews, above 10% of the others.
Reviewer_CA4kOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents CUDA-Harness, an agentic framework for generating and optimizing CUDA programs from natural-language descriptions. The framework combines three components: Intermediate-Structured Generation, which constructs a kernel implementation manifest and a self-contained CUDA scaffold; Synthesis-Based Verification, which generates test inputs and reference outputs separately from kernel generation; and Feedback-Adaptive Evolution, which combines correctness-first refinement with performance optimization. The evaluation includes comparisons with existing agent frameworks, ablation studies, verification-error analysis, and generalization experiments across LLMs and hardware platforms.

Critical strengths
  1. Important problem and well-motivated challenges. Generating CUDA programs directly from natural language is practically valuable. The challenges identified by the paper, particularly interpreting underspecified requirements and validating implementations without predefined test data, are real and deserve attention.

  2. A coherent solution tailored to Text2CUDA. The framework explicitly separates semantic interpretation, implementation, verification, and optimization. The manifest and isolated test-generation process provide a reasonable approach to organizing these tasks.

  3. Detailed empirical evaluation. The paper reports improvements over several baselines and includes component ablations and an analysis of discrepancies between synthesized and benchmark validation. These experiments help explain both the benefits and limitations of the framework.

Critical weaknesses
  1. Semantic fidelity of the generated manifest and tests remains unresolved. The agent generates both the implementation manifest and the test inputs and reference outputs. How does the framework establish that these artifacts faithfully represent the original natural-language requirements? Context isolation can reduce leakage between implementation and test generation, but both processes can still inherit the same semantic misunderstanding. A kernel may therefore pass the synthesized tests while implementing the wrong computation. This concern is supported by Section 5.2.3: synthesized verification accepts incorrect programs in 8.7% of all evaluated cases, and the authors attribute 38.1% of these false positives to prompt misinterpretation.

  2. The methodological novelty beyond manifest construction and test synthesis is limited. The most distinctive contribution appears to be the explicit construction of intermediate specifications and synthesized tests for Text2CUDA. In contrast, Feedback-Adaptive Evolution largely follows an established pattern of compiling, testing, profiling, and iteratively refining generated kernels. Prior systems, e.g., CUDAForge(your baseline) and [1,2,3,4] already employ related feedback-driven optimization workflows. The paper should more clearly identify what is technically new in this component and explain how it differs from existing approaches beyond its integration into a Text2CUDA pipeline.

  3. The evaluation provides insufficient evidence for complex end-to-end Text2CUDA workloads. Although CUDABench is relevant to the problem setting, its tasks are organized around individual kernels. Additional evaluation on more demanding end-to-end workloads would strengthen the paper. For example, CUDAHercules[4] includes module-level and full-application tasks with domain-specific semantic validators, making it a relevant complementary benchmark. I encourage the authors to evaluate an appropriate subset and report both semantic correctness and end-to-end performance.

  4. The hardware coverage is insufficient to support broad cross-architecture generalization. Both A40 and Jetson AGX Orin use Ampere, while GTX 1660 SUPER uses the older Turing architecture. Modern GPU architectures evolve rapidly, and many advanced optimization techniques depend on features introduced in newer generations. The current evaluation therefore provides insufficient evidence of cross-architecture generalization. Testing on at least one Hopper or Blackwell GPU would strengthen this claim.

[1] StitchCUDA: An Automated Multi-Agents End-to-End {GPU} Programing Framework with Rubric-based Agentic Reinforcement Learning. ICML 2026.

[2] Hawkeye: Hardware-Aware GPU Kernel Optimization with Minimal Supervision. https://www.alphaxiv.org/abs/2608.hawkeye-hardware-aware-gpu-kernel-optimization

[3] CAKE: Compiler-Agent Co-Design for Frontier Kernel Evolution. https://arxiv.org/abs/2608.12629

[4] CUDAHercules: Benchmarking Hardware-Aware Expert-level CUDA Optimization for LLMs. https://arxiv.org/abs/2605.08467

Questions for authors
  1. How do you verify that the generated manifest and tests preserve the original natural-language semantics, rather than share the same misinterpretation?

  2. What technical novelty does Feedback-Adaptive Evolution offer beyond existing feedback-driven kernel optimization frameworks?

  3. Can you evaluate the framework on more complex end-to-end Text2CUDA tasks, such as those in CUDAHercules?

  4. Do the gains extend to Hopper or Blackwell GPUs, where advanced optimizations depend on newer architectural features?

1 comment
AuthorsOpenReview ↗

Response to Reviewer CA4k

We thank the reviewer for the detailed review. The discussion of prompt misinterpretation is addressed in the General Response.

Novelty of Feedback-Adaptive Evolution

While developing the iterative framework, we found that when the current kernel has correctness issues, the agent is still steered toward optimizing performance by a prompt that asks for kernel optimization, while neglecting the importance of correctness. We therefore introduced correctness-first refinement, whereas existing feedback-driven kernel optimization frameworks tend to overlook this point, assuming that the model itself attends to correctness over performance in such cases.

More Complex Text2CUDA Tasks

We thank the Reviewer for the constructive review. We will add the experimental results and discussion of each method on CUDAHercules in the revision.

Results on Blackwell GPU

We verify our approach on an RTX 5090 GPU (Blackwell architecture), and the results are shown in the table below.

MethodComp.Func.RScore
Baseline90.160.8186.1
CUDA-Harness99.780.2250.1

As can be seen, CUDA-Harness performs well on the RTX 5090, further indicating that CUDA-Harness can generalize across different hardware platforms.

Reviewer_FZBkOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies Text2CUDA: generating and optimizing self-contained CUDA programs directly from natural language descriptions.

Existing LLM-based CUDA kernel generation method mainly focus on Torch2CUDA transpilation from executable PyTorch code where algorithmic details and test inputs are already specified, whereas Text2CUDA starts from underspecified prompts without predefined test data and is prone to reward hacking. To address this, the authors propose CUDA-Harness, an agentic workflow that combines structured manifest/scaffold generation, isolated test data synthesis, and multi-round kernel refinement.

Critical strengths
  1. Well-motivated decomposition for Text2CUDA Separating high-level semantic completion from low-level kernel coding, and prioritizing functional correctness before performance optimization is reasonable.

  2. Evaluation across multiple models, GPUs, and prompt levels The evaluation spans three prompt difficulty levels on CUDABench, three LLMs, three GPU architectures.

Critical weaknesses

W1. Replacing CUDABench's official score with a flawed metric. In Section 5.1.2, instead of using CUDABench's official performance score, the authors define RScore = 1 / execution time (ms) and assign 0 to failed kernels. This metric has two fundamental flaws: (1) Runtime performance across heterogeneous kernels should be evaluated using speedup over a reference implementation. Directly averaging raw 1 / execution time across different tasks is non-standard and unreasonable. (2) Because failed kernels are assigned 0, turning a failed kernel into a passing one changes the sum from 1/t + 0 to 1/t + positive number. As long as the functional pass rate increases, RScore is guaranteed to increase.

W2. Unfair compute comparison and missing token consumption evaluation: Baseline and CUDA-ISG only perform a single generation pass (Pass@1), whereas CUDA-Harness involve N = 3 rounds with G = 3 candidate kernels per round. The paper does not report token consumption, API call counts, or wall-clock time across methods, nor does it compare against a Pass@k / Best-of-k baseline under the same token budget.

W3. Impossible metric value in Table 3 Table 3 (Page 8, Line 387) reports Func. = 101.8% for GLM 5.1 (while Comp. = 99.2%). A pass rate above 100% is impossible and makes the evaluation script and the rest of Table 3 hard to trust.

W4. (minor) Unfinished sentence (Page 4, Section 4.1.2, Line 203).

Questions for authors
  1. Why did the authors abandon CUDABench's official performance score in favor of 1 / execution time (ms)? What are the results when evaluated using CUDABench's official performance score and speedup over reference kernels?

  2. Please report the average token consumption, LLM API calls, and wall-clock time per task for all compared methods.

  3. How could a percentage pass rate reach Func. = 101.8 for GLM 5.1 in Table 3 (while Comp. = 99.2)? This impossible value calls the credibility of the entire Table 3 and the underlying evaluation pipeline into question.

  4. What was the intended completion of the unfinished sentence at line 203 in Section 4.1.2?

1 comment
AuthorsOpenReview ↗

Response to Reviewer FZBk

We thank the reviewer for the detailed review. The token consumption of CUDA-Harness versus the baseline methods, the discussion of the evaluation metric and more experimental results, and the clarification of the wrong data in Table 3 are addressed in the General Response.

Unfinished Sentence

We thank the Reviewer for the careful review. We mistakenly left this unfinished sentence in the paper during revision. The complete sentence should read: "Therefore, guided by the self-contained CUDA scaffold, the agent needs to first generate the launcher and the corresponding launch configuration, and complete the kernel implementation under the given launch configuration."

Reviewer_mdhHOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

In this work, the authors propose a framework for generating and optimizing CUDA kernels from natural-language descriptions.

Critical strengths

This paper focuses on generating efficient CUDA implementations when executable specifications and test data are unavailable. The separation of test synthesis from implementation is a sensible design choice, and the reported improvements in functional correctness over several baselines are encouraging.

Critical weaknesses

Question:

  • Table 3 reports an impossible correctness rate of 101.8%, how could you get a correctness rate over 100%?

Weakness:

1.CUDA-Harness uses three optimization rounds with three candidates per round, but the paper does not report comparable token budgets, candidate counts, tool calls, repair attempts, or wall-clock costs for the competing systems

2.Defined metrics fairness:RScore averages inverse execution times across tasks and assigns zero to incorrect kernels, so I think the score can improve simply because more kernels become correct, even if the execution time of already-correct kernels does not improve. Also, averaging unnormalized inverse latencies gives disproportionate influence to inherently short-running tasks. The reported increase from 80.1 to 110.7 therefore cannot be interpreted as a corresponding runtime speedup.

3.The paper should compare isolated and shared-context test generation and measure whether incorrect kernels exploit the verification process

4.Generated tests can share the kernel’s misunderstanding, and if both interpret an ambiguous request incorrectly, they may agree while solving the wrong problem.

5.The ablations do not isolate the main components for this engineering approach

Questions for authors

See weaknesses listed above

1 comment
AuthorsOpenReview ↗

Response to Reviewer mdhH

We thank the reviewer for the detailed review. The token consumption of CUDA-Harness versus the baseline methods, the discussion of context isolation, the discussion of the evaluation metric and more experimental results, the clarification of the wrong data in Table 3, and the discussion of prompt misinterpretation are addressed in the General Response.

Reviewer_ckXYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CUDA-Harness, an agentic framework for generating and optimizing CUDA kernels from natural-language descriptions (Text2CUDA). The framework consists of three components: Intermediate-Structured Generation for converting natural-language requirements into structured kernel specifications, Synthesis-Based Verification for generating independent test data and validating correctness, and Feedback-Adaptive Evolution for correctness-first refinement and performance optimization. Experiments on CUDABench show improvements in functional correctness and runtime performance, with additional evaluations across different LLMs, GPU platforms, and C-to-CUDA translation.

Critical strengths

The paper addresses Text2CUDA generation from underspecified natural-language descriptions, which requires both semantic understanding and low-level CUDA implementation. The proposed framework provides a structured approach to generation, verification, and optimization, making it relevant to practical GPU programming. CUDA-Harness improves functional correctness from 60.9% to 80.5% and demonstrates improvements across different difficulty levels, LLMs, and hardware platforms. These results suggest that the framework can improve the reliability of agentic CUDA kernel generation.

Critical weaknesses

CUDA-Harness uses multiple optimization rounds and candidate generations, but the paper does not specify whether the baselines receive comparable computational budgets. This makes it difficult to determine whether the improvements come from the proposed methods or additional test-time computation. Without budget-matched comparisons, the effectiveness of the framework remains difficult to assess.

Questions for authors

How are the generation and optimization budgets controlled across CUDA-Harness and the compared baselines? Can the authors provide comparisons under matched LLM calls, candidate generations, or computational budgets to clarify whether the improvements come from the proposed framework?

1 comment
AuthorsOpenReview ↗

Response to Reviewer ckXY

We thank the reviewer for the comments. The token consumption of CUDA-Harness versus the baseline methods is addressed in the General Response.

Reviewer_tjQgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CUDA-Harness generates CUDA kernels from natural-language specifications using a structured implementation manifest and scaffold, independently synthesized validation data, and correctness-first iterative optimization. The framework keeps test synthesis separate from kernel generation and summarizes successful and unsuccessful edits to guide later candidates. On CUDABench, it reports 80.5% functional correctness versus 60.9% for the benchmark baseline, with additional comparisons to agentic systems and evaluations across models, hardware, and C-to-CUDA translation. Its runtime score averages inverse latency with incorrect kernels assigned zero.

Critical strengths
  1. The paper addresses a useful setting where both the implementation and its tests must be derived from natural language. The separation of semantic planning, executable scaffolding, and test synthesis is a practical way to organize this problem, and evaluation uses benchmark tests withheld during optimization rather than relying exclusively on the generated tests (Sections 4 and 5.2.3).

  2. The correctness-first variant reports meaningful gains: adding it to CUDA-Harness* increases overall functional correctness from 70.8% to 80.5%, and Level 3 correctness from 57.6% to 70.4% (Table 1). These results show a practical improvement in the evaluated configuration, particularly on the least specified prompts.

Critical weaknesses
  1. It remains unclear whether the workflow outperforms the alternatives under comparable total search resources. Section 5.1 fixes the underlying model and specifies three optimization rounds with three candidates each, but does not adequately explain the shared allowance for generation and repair across the principal baselines and the correctness-first ablation. Additional repair opportunities could contribute to the reported gains. This limits attribution to the proposed organization, including the specific correctness-first design.

  2. The verification study does not isolate the protection attributed to context separation. Section 4.2 combines isolated test synthesis with broader numeric input coverage, while Section 5.2.3 evaluates the resulting pipeline. Agreement with benchmark validation supports the combined procedure but does not distinguish these mechanisms’ effects on false acceptance or reward hacking. The reported benchmark data-loading defects also complicate interpreting disagreement as generated-kernel error. The paper should support the claimed mechanism or limit its conclusion to the observed reliability of the complete pipeline.

Questions for authors
  1. Were CUDA-Harness, the principal baselines, and the correctness-first ablation given comparable total generation and repair allowances? Which existing settings or results show that the gains persist under those constraints?

  2. What evidence distinguishes the reliability benefit of context isolation from broader test-input coverage? Please clarify which claims the existing verification study supports, or narrow them to the observed reliability of the complete pipeline.

1 comment
AuthorsOpenReview ↗

Response to Reviewer tjQg

We thank the reviewer for the comments. The token consumption of CUDA-Harness versus the baseline methods and the discussion of context isolation are addressed in the General Response.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (1/2)

Token Consumption [tjQg, ckXY, mdhH, FZBk]

We report the token consumption of all evaluated methods in the table below.

Method#Avg Input Tokens#Avg Output Tokens
Baseline11611138
OpenCode277981894
Codex96474349
CudaForge18591301
KernelSkill15831340
CUDA-Harness10544435496

For the NN-round iterative optimization of CUDA-Harness, since the GG agents in each round share the same system prompt and user prompt, the number of cached input tokens attributable to prefix caching (Zheng et al., 2023) is 66072, with a cache hit rate of 62.66%. Moreover, for the iterative optimization process of CUDA-Harness, correctness-first refinement (CFR) and reward-driven optimization (RDO) share the round budget, i.e., NCFR+NRDO=NN_{\text{CFR}} + N_{\text{RDO}} = N.

In the current N=3, G=3 setting, CUDA-Harness takes about 6 minutes per operator on average. Kernel development happens offline, so taking more development time for better deployed performance is acceptable in many practical scenarios.

References

Zheng, Lianmin, et al. "Sglang: Efficient execution of structured language model programs." Advances in neural information processing systems 37 (2024): 62557-62583.

Evaluation Metric (RScore & CUDABench-Score & Speedup) [mdhH, FZBk]

Since CUDA-Harness optimizes the generated kernels for latency on specific hardware, we propose RScore to evaluate the latency of a kernel on a specific hardware. RScore handles incorrect kernels in the same way as the CUDABench-Score proposed by CUDABench (Zhu et al., 2026). Specifically, the CUDABench-Score of an incorrect kernel is set to 0, and hence the CUDABench-Score also increases with the functional correctness rate.

Therefore, we additionally report the CUDABench-Score of each method, together with the metric fast1\text{fast}_1 (i.e., fastp\text{fast}_p with p=1p=1) introduced by KernelBench (Ouyang et al., 2025) (fastp=1N∑i=1N1(correcti∧(speedupi>p))\text{fast}_p = \frac{1}{N} \sum _{i=1}^{N} \mathbb{1}(\text{correct}_i \land ( \text{speedup}_i > p ))) and the speedup over the reference implementation provided by CUDABench. To rule out the interference of functional correctness, we report both all cases (the speedup of an incorrect kernel is set to 0 and still counted in the denominator) and correct only (incorrect kernels are excluded from the average). Moreover, since the reference sorting implementation provided by CUDABench is too inefficient (single-threaded bubble sort), the speedups on sorting tasks exhibit large outlier values, so that the functional correctness on sorting tasks severely biases the speedup statistics. We therefore also report the speedups after excluding the sorting tasks.

MethodComp.Func.RScoreCUDABench-Scorefast1\text{fast}_1Speedup (All Cases)Speedup (Correct Only)Speedup Excluding Sorting (All Cases)Speedup Excluding Sorting (Correct Only)
Baseline90.160.980.129.339.1100.5165.515.825.8
OpenCode92.160.678.728.245.115.826.314.624.3
Codex98.369.586.532.138.360.688.028.841.9
KernelSkill99.463.786.130.340.517.627.917.026.8
CudaForge99.867.787.931.844.318.527.717.626.2
CUDA-ISG94.563.984.430.742.1100.6158.216.125.1
CUDA-Harness*99.370.894.936.049.3200.9284.332.646.2
CUDA-Harness99.780.5110.739.160.5629.5782.338.247.5

As shown in the table, CUDA-Harness outperforms the other methods in both CUDABench-Score and speedup statistics. Meanwhile, the Speedup Excluding Sorting (Correct Only) results also indicate that the speedup gains achieved by CUDA-Harness are not obviously correlated with the improvement in functional correctness.

References

Zhu, Jiace, et al. "Cudabench: Benchmarking llms for text-to-cuda generation." arXiv preprint arXiv:2603.02236 (2026). Ouyang, Anne, et al. "Kernelbench: Can llms write efficient gpu kernels?." arXiv preprint arXiv:2502.10517 (2025).

AuthorsOpenReview ↗

General Response (2/2)

Context Isolation [tjQg, mdhH]

Context isolation is intended to prevent the agent from referring to the test-data generation script when generating kernels, which could lead to reward hacking. Although reward hacking may not necessarily occur in practice, we still choose to further reduce this risk at the system-design level. We will add ablation experiments on context isolation in the revision.

Wrong Data of Table 3 [mdhH, FZBk]

We thank the Reviewer for the careful review. This issue arises because we repeatedly transposed this table while revising the paper. In the last transposition, the data (currently GLM 5.1 Func. 101.8) and the target data (currently DeepSeek V3.2 RScore 80.1) were accidentally left swapped. Therefore, the correct data are: DeepSeek V3.2 achieves an RScore of 101.8, and GLM 5.1 achieves a Func. of 80.1.

Prompt Misinterpretation [mdhH, CA4k]

The kernel-generation process and the test-generation process may indeed share the same misinterpretations of the prompt, but this is closely related to the capability of the underlying model. Moreover, in Sec. 5.2.3, we analyzed this with an LLM-as-a-judge (Zheng et al., 2023) method, using DeepSeek-V4-Pro as the judge, and found that 50 of the 1500 instances exhibited prompt misinterpretation. We will elaborate on this analysis in the revision, including the prompt misinterpretation rate of the baseline method.

References

Zheng, Lianmin, et al. "Judging llm-as-a-judge with mt-bench and chatbot arena." Advances in neural information processing systems 36 (2023): 46595-46623.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.