acceptodds
Under review as a conference paper at ICLR 2027

KernelMosaic: An Open Dataset and Agentic Framework for CUDA Kernel Generation

10 views

Abstract

Training large language models to generate efficient CUDA kernels requires supervision spanning diverse computational requirements and iterative code optimization. Constructing this supervision is challenging because of heterogeneous specifications, and useful implementations emerge through iterative complicated testing and refinement. We introduce KernelMosaic, an open dataset pairing diverse CUDA tasks with verified solutions and trajectories. Human-guided review and diversity-aware selection curate tasks expressed as natural-language descriptions, reference programs, mathematical specifications, and their intersections. To build the solutions and trajectories, our agentic framework guides a strong LLM through implementation, correctness repair, and performance optimization with reusable CUDA guidance and execution feedback. Supervised fine-tuning of a small open-source model on KernelMosaic raises correctness from 4%, 0%, and 0% to 88%, 82%, and 16% on KernelBench L1, L2, and L3, respectively, while increasing the fraction of tasks with kernels faster than PyTorch eager at every level. Further training experiments show that supervision from the curated trajectories substantially outperforms direct responses and promotes sustained testing and revision. These results demonstrate that KernelMosaic enables a general-purpose model to achieve substantial gains in CUDA kernel generation.

Primary area:
datasets and benchmarks
Keywords:
Kernel generation, open dataset, kernel generation harness, data curation, agents

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_x2m7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors present KernelMosaic, an open dataset of 14k+ CUDA tasks with corresponding tests, a verified solution, and a trajectory of an agent developing the solution. Specifically, the trajectories are generated by a strong LLM in an OpenCode-based harness with CUDA skills. While verified agentic CUDA trajectories have been released in prior work (e.g., Nemotron-SFT-CUDA-v1), the primary distinction here is the scale and the diversity of specifications (PyTorch, NL, math, etc.). Among other findings, the authors show that SFT-ing a 9B model on the dataset broadly improves KernelBench scores, and that trajectory supervision is more effective than direct responses.

Agentic kernel generation is quite relevant given the demand for efficient GPU kernels in LLM training and serving, and open trajectory-level data in this domain is still scarce.

That said, there remains a question of the attribution of the improvements. The main claim is that KernelMosaic "enables a general-purpose model to achieve substantial gains," but gains relative to what? The base model largely fails at using the harness itself, so a big jump could be expected from any agentic SFT data. We would need to see a controlled experiment where the same student is trained on existing trajectory data under the same harness, along with the teacher's score as upper bound. Similarly, trajectory over direct SFT is intuitively correct, but the comparison is potentially confounded by data mix and amount of supervision, and the direct SFT model never see the tools in training. Currently, the curation steps are not connected to downstream performance either.

Critical strengths
  • Large dataset with tests, verified solutions, diverse specs, and full trajectories for this domain would be useful for enabling better capabilities of all LLMs, which benefits us.
  • The post-training study and eval on KernelBench shows the utility of the dataset for a 9B model (arguably small), with detailed analysis on agentic behavior after training.
Critical weaknesses
  • The contribution is mostly about the scale of the dataset; most tasks are form existing datasets whose verified agentic CUDA trajectories already exist.
  • The claim about "KernelMosaic enables the gains" is not rigorously tested, as the base model fails mostly at the harness rather than CUDA.
  • The task distribution is biased towards KernelBench-style PyTorch modules while contamination is checked only by exact match, which reduces the significance of the findings.
Questions for authors
  • How would the SFT on Nemotron perform with the same student and harness?
  • What's the near-duplicate rate against KernelBench instances (e.g., cos >= 0.9)?
Reviewer_2gLkOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes KernelMosaic, which collects 14,566 CUDA tasks from 41 sources, each with a selected implementation and an iterative development trajectory. An agent uses execution feedback to construct solutions, and the resulting data fine-tunes Qwen3.5-9B. Under the paper's shared agentic evaluation protocol on 250 KernelBench tasks (100/100/50 at Levels 1/2/3), full-data training improves correctness from 4/0/0% to 88/82/16%, or from four to 178 solved tasks; the corresponding Fast1 rates are 64/62/12%.

Critical strengths
  1. Preserving specifications, tests, revisions, feedback, and final implementations could make the dataset useful for studying iterative kernel development rather than final-code imitation alone.
  2. The same-base-model comparison shows large gains under a common evaluation protocol, and the paper documents trajectory windowing, training, and timing procedures in considerable detail. Its three warmup and nine timed forward calls per implementation are clearly specified.
  3. The authors describe source curation, category composition, benchmark-path exclusion, and normalized-text duplicate checks.
Critical weaknesses
  1. The supplied PDF describes an open dataset and framework but does not give a clear anonymous artifact location, complete provenance/license inventory, or an inspectable sample. I could not verify the central data contribution from the supplied materials. A separate OpenReview attachment may exist, so the request is for the precise reviewer-access route; PDF link absence alone should not determine the score.
  2. The methodological novelty appears limited: successful tool-use trajectories are generated by a stronger agent and used for supervised fine-tuning, with source curation and context-window handling as additional components. The public Nemotron-SFT-CUDA-v1 dataset card already describes full OpenCode agent traces for CUDA tasks, filtered by executable tests and intended for SFT. KernelMosaic is larger (14,566 versus 2,276 records) and spans more source and specification types, which could be a meaningful data contribution, but the paper mentions Nemotron without including it in Table 1 or explaining the concrete schema, test-quality, and trajectory differences.
  3. The 1,000-example trajectory-versus-direct comparison does not isolate trajectory supervision. The cohorts share only 61 task IDs, differ greatly in task composition, and contain about 28.95M versus 11.42M supervised tokens and 315 versus 125 optimizer steps. The direct-response data are also not equivalently execution-filtered. The comparison shows that the training packages differ in effectiveness, not that trajectory form alone causes the difference; this limitation does not erase the full-data same-model result. The Appendix B.3 verified-budget sensitivity still yields 29 versus zero solved tasks, so three uncertain budget records do not explain the gap.
  4. The original Qwen3.5-9B checkpoint solves only 4/250 tasks in this agentic CUDA-C++ harness, an unexpectedly weak starting point. It submits candidates on 249/250 tasks, but at first evaluation 73 omit the required cpp_sources argument, 61 encounter module-state loading errors, and 20 use forbidden PyTorch tensor computation; across its runs, 60.1% of completed file writes repeat exactly, versus 2.6% after SFT (p. 24). These are concrete interface and workflow failures. An unknown part of the 4-to-178 gain may therefore reflect learning this harness rather than general CUDA kernel skill. The paper's Qwen3-8B single-turn Triton result is from a different language and protocol and cannot settle this question.
  5. The paper performs source-aware curation, embedding-based diversity selection within the training pool, benchmark-path exclusion, and normalized-text duplicate checks against all 250 KernelBench tasks. These steps do not establish the absence of semantic or implementation-neighbor overlap between training and evaluation. Three input seeds and 0.01 absolute/relative tolerances also leave uncertainty beyond tested cases. KernelBench-Verified motivates distribution-shift checks, but its effect sizes cannot be transferred to KernelMosaic.
  6. The 3,600-second per-task active clock excludes model inference waits, so actual wall-clock costs remain unclear. Appendix F.4 does provide a useful recorded-trajectory budget curve: the full-data model solves about 129/250 tasks by 65,536 output tokens and 158/250 by 131,072, versus 178/250 at the full 262,144-token allowance with the same time limit. These post hoc cutoffs show substantial budget dependence but not the per-task token and wall-clock cost distribution. The paper states source-group admission thresholds and upstream checks, but does not summarize independent test quality or failure rates by source.
Questions for authors
  1. Where can reviewers access the anonymous dataset, code, source provenance, license inventory, and a small end-to-end reproduction example? If these are separate OpenReview attachments, please identify them.
  2. How does KernelMosaic differ from Nemotron-SFT-CUDA-v1 in task provenance, test quality, trace contents, and training use, beyond dataset size? A concrete side-by-side characterization would clarify the resource contribution.
  3. Can you narrow the causal claim about trajectories, or analyze a small same-task, matched-token subset from existing data? A full model retraining study is not needed for this clarification.
  4. Given the original checkpoint's loader/interface failures and repeated writes, can you report its final-outcome error categories and specify its serving, tool-call parsing, and sampling settings? On a small paired subset, would an explicit valid extension scaffold or a simpler direct-generation prompt materially improve its correctness? This would clarify what the 4/0/0% baseline measures.
  5. What semantic or implementation-neighbor audit against KernelBench exists beyond the reported path and normalized-text checks? The embedding-based diversity selection within the training pool does not answer this train/test overlap question.
  6. What are the distribution of model tokens and actual wall-clock time per KernelBench task, including inference waits?
Reviewer_NFtDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces KernelMosaic, a dataset of 14,566 CUDA programming tasks collected and curated from 41 sources. Task curation combines human inspection, AI-assisted review, and diversity-aware selection. The authors use GLM-5.3 to generate and repair executable tests and implementation interfaces, then use Qwen3.8-Flash in an agentic framework to generate CUDA implementations and development trajectories through iterative testing, debugging, and performance optimization. They use these trajectories to fine-tune Qwen3.5-9B and evaluate the resulting model on KernelBench under an agentic evaluation protocol. Fine-tuning substantially improves correctness and the fraction of tasks with implementations faster than PyTorch eager, particularly on individual operators and operator sequences. Performance on model-level workloads improves but remains limited.

Critical strengths

There are two strengths of the paper:

  • The paper is written in clean manner and introduces a sizable dataset of 14,566 CUDA programming tasks, pairing task specifications with tested implementations and development trajectories. These trajectories capture intermediate failures, execution feedback, and revisions, making the dataset a potentially valuable resource for training models to perform iterative kernel development.

  • The main evaluation compares the base and fine-tuned Qwen3.5-9B under a agentic protocol, showing substantial improvements in correctness and faster-than-PyTorch-eager coverage, on KernelBench L1 and L2. The accompanying behavioral analysis shows more productive testing and revision, providing evidence of improved development behavior beyond aggregate benchmark scores.

Critical weaknesses

There are three weaknesses of the paper, as stated below:

  • Dataset accessibility is unclear and provides no link to the claimed dataset release, there is critical for a dataset paper.

  • The evaluation is limited to KernelBench, leaving generalization across workloads and specification formats untested. Additional evaluation on benchmarks such as TritonBench, BackendBench or Atrex-Bench would further support the claims.

  • Human curation and source provenance are insufficiently documented. The paper does not quantify human-review coverage, reviewer expertise, distinguish human and AI contributions. And the paper does not enumerate all 41 sources.

Questions for authors
  • When and where will the author release the dataset?
  • What fraction of tasks was individually reviewed by humans, who performed these reviews, and how were human judgments distinguished from AI assessments?
  • Could the authors evaluate the fine-tuned model on additional benchmarks, such as BackendBench and TritonBench, to further demonstrate the effectiveness of the dataset?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.