- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SynKer presents a pipeline for generating synthetic GPU-kernel tasks, searching for verified implementations, and post-training language models on the resulting archive. Task descriptions evolve from a seed, while evolutionary kernel search produces CUDA, Triton, and HIP implementations checked by execution. The resulting resource contains 2,365 task environments and 105,689 verified kernel triples. SFT experiments study task diversity and evaluate held-out BackendBench correctness; exploratory RL experiments examine training behavior under different initializations.
Critical strengths
-
The verified multi-backend task and kernel archive is a useful potential research resource. Building configurable environments together with implementations and verification feedback addresses an important bottleneck in kernel-generation training.
-
The fixed-kernel-budget task-diversity study is more informative than simply scaling the number of examples. Combined with held-out BackendBench results, it provides evidence that broader task coverage can improve the usefulness of synthetic training data. I lean toward acceptance primarily for this resource and empirical contribution.
Critical weaknesses
-
The pipeline combines established task generation, evolutionary search, and post-training methods. Its main novelty seems to be the verified multi-backend resource, rather than a new algorithm.
-
The downstream results mainly support the dataset and SFT stage; the RL evidence is still exploratory. The paper does not yet show that RL improves held-out performance or that the fine-tuned models generate faster kernels.
Questions for authors
-
Beyond combining synthetic task generation, existing evolutionary code search, and standard post-training, what is the distinctive methodological contribution of SynKer? Please identify which design decision is necessary for the reported findings and is not inherited from the underlying components.
-
The task-generation novelty filter is based on task-description similarity. What evidence in the existing analysis shows that the resulting diversity reflects different computational structures or kernel implementation requirements, rather than primarily different application descriptions? How does that distinction affect the interpretation of the fixed-budget diversity study?
-
Should the central contribution be understood as a verified multi-backend resource and an SFT data-diversity finding, or as a validated end-to-end SFT-plus-RL method? Given the exploratory status of RL and the correctness-focused downstream evaluation, which conclusions are specifically attributable to the RL stage rather than the dataset and SFT initialization?
Response to Reviewer oH3R
Thank you for highlighting the two main contributions of our paper: the verified multi-backend archive and the fixed-budget task-diversity study. Totally agree with you to distinguish between data construction, SFT and RL. Thank you for taking the time!
Algorithm 1 in Section 3 and Appendix A detail our task generation procedure, which starts with a single seed task and iteratively mutates task descriptions. We screen the proposals against the verified multi-backend archive and create an executable PyTorch reference with configurable inputs/correctness tests for each proposal. This allows us to verify candidates on all three backends. We leverage the existing ShinkaEvolve search to obtain multiple implementations for each task. This allows us to disentangle tasks & implementations, which is crucial for our fixed-budget study of task diversity.
While we do not claim that any single generation step is crucial, this approach allows us to treat the same task as a verification for all three backends.
The novelty filter and Figure 2 both operate on description-level properties. Table 1 gives an overview of different computational requirements (lattice stencil update, 1-D cosine transform, point-cloud plane fitting) and how search is used to incentivize alternative implementations for the same task. Section 3 covers this in more detail. While there are different computations in the archive, we do not track how often tasks share an operator or implementation pattern.
Fig. 4-5 provides some evidence that there is a positive relationship between the number of executable tasks covered within a fixed example budget & SFT improvements. However, it says nothing about the contribution of any single computational pattern.
Figure 5 reports the SFT-only results. For RL, we perform a new rebuttal evaluation on 114 held-out NVIDIA ComputeEval CUDA tasks. After SFT, Qwen3-32B has a pass@1 of 19.59% ± 1.34%. After RL, this improves to 22.81% ± 1.75% (mean ± SD over three runs). Note that this is with thinking disabled. We claim that our RL transfer results only hold for this setting. Appendix B.4 reports on the median speedups of search-generated kernels versus eager mode for PyTorch. We observe that Triton and CUDA benefit from 4.37× and 10.37× speedup, respectively. Note that we do not evaluate the runtime of the kernels generated by fine-tuned models.