- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors present KernelMosaic, an open dataset of 14k+ CUDA tasks with corresponding tests, a verified solution, and a trajectory of an agent developing the solution. Specifically, the trajectories are generated by a strong LLM in an OpenCode-based harness with CUDA skills. While verified agentic CUDA trajectories have been released in prior work (e.g., Nemotron-SFT-CUDA-v1), the primary distinction here is the scale and the diversity of specifications (PyTorch, NL, math, etc.). Among other findings, the authors show that SFT-ing a 9B model on the dataset broadly improves KernelBench scores, and that trajectory supervision is more effective than direct responses.
Agentic kernel generation is quite relevant given the demand for efficient GPU kernels in LLM training and serving, and open trajectory-level data in this domain is still scarce.
That said, there remains a question of the attribution of the improvements. The main claim is that KernelMosaic "enables a general-purpose model to achieve substantial gains," but gains relative to what? The base model largely fails at using the harness itself, so a big jump could be expected from any agentic SFT data. We would need to see a controlled experiment where the same student is trained on existing trajectory data under the same harness, along with the teacher's score as upper bound. Similarly, trajectory over direct SFT is intuitively correct, but the comparison is potentially confounded by data mix and amount of supervision, and the direct SFT model never see the tools in training. Currently, the curation steps are not connected to downstream performance either.
Critical strengths
- Large dataset with tests, verified solutions, diverse specs, and full trajectories for this domain would be useful for enabling better capabilities of all LLMs, which benefits us.
- The post-training study and eval on KernelBench shows the utility of the dataset for a 9B model (arguably small), with detailed analysis on agentic behavior after training.
Critical weaknesses
- The contribution is mostly about the scale of the dataset; most tasks are form existing datasets whose verified agentic CUDA trajectories already exist.
- The claim about "KernelMosaic enables the gains" is not rigorously tested, as the base model fails mostly at the harness rather than CUDA.
- The task distribution is biased towards KernelBench-style PyTorch modules while contamination is checked only by exact match, which reduces the significance of the findings.
Questions for authors
- How would the SFT on Nemotron perform with the same student and harness?
- What's the near-duplicate rate against KernelBench instances (e.g., cos >= 0.9)?