- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The work presents a hardware-in-the-loop RL framework for optimizing OpenCL kernels on mobile GPUs. There are two datasets introduced, OpenCLOC and OpenCLKernelBench, along with an execution-driven training infrastructure that addresses challenges in stablizing the reward computation. Using GRPO to fine-tune Qwen2.5-Coder-32B, the method can improve kernel correctness and performance on mobile GPUs. The paper claims this is the first RLVR applied to OpenCL kernels.
Critical strengths
- First to apply RLVR for OpenCL kernel optimization. Mobile devices may be more of practical value since they need to run inference and there may be a growing need for that.
- There is a clear dataset contribution.
- Empirical results seem promising.
Critical weaknesses
-
Unclear practical motivation for RL training. This paper avoids benchmark frontier models like Astra or Claude. If kernel optimization is a one-time effort before deployment, why don't just use the frontier model? what's the gap between the RL trained model and the best frontier model?
-
This paper need more experiments to compare against RL training approach with other approaches such as supervised fine-tuning (rollout and select the best ones for imitation learning), iterative refinement, best-of-N sampling, which has been done for other related works like SuperCoder (but for assembly domains other than OpenCL) and this paper should discuss the missing citations as well.
-
A detailed study on what the model learns and what program transformations are done is missing. Please consider adding them.
Questions for authors
- How is the reward function designed? There is a big exploration space there; is there any scientific study of it? How much does it affect the training? In what way?
2 comments
part 2/2
Unclear practical motivation for RL training. This paper avoids benchmark frontier models like Astra or Claude. If kernel optimization is a one-time effort before deployment, why don't just use the frontier model? what's the gap between the RL trained model and the best frontier model?
We have added ChatGPT-6.1 Sol and Claude Opus 5.5 as proprietary frontier baselines (Tables 1–2). Both achieve strong results comparable to GLM-5.3 on OpenCLOC, while Claude Opus 5.5 is particularly strong on OpenCLKernelBench.
We acknowledge that the strongest frontier models outperform our RL-trained model, and we do not claim that HIL-RLVR training on a smaller model is necessarily preferable to using a frontier model for optimizing a single kernel with a large inference-time budget. Our motivation is instead to investigate whether hardware optimization capabilities can be learned from verified rewards on the target hardware. Our experiments establish that this approach is effective, with the trained model substantially outperforming its base model and achieving performance competitive with or superior to the considerably larger GPT-OSS-120B on unseen tasks. These results are encouraging and motivate follow-up investigation into training larger open-weight models under the same setup, but we do not claim parity with the strongest frontier models.
More broadly, kernel optimization is not necessarily a one-time effort, particularly as both GPU architectures and algorithms continue to evolve. We view improving model capabilities through training and scaling inference-time search as complementary approaches. Learning reusable optimization capabilities for a target hardware architecture, rather than relying entirely on per-kernel inference-time search, is therefore a worthwhile research direction.
A detailed study on what the model learns and what program transformations are done is missing. Please consider adding them.
We have added an analysis of the optimization strategies used by the base and trained models and frontier baselines (Appendix F). We use AST-based heuristics and an LLM judge to categorize program transformations and compare their frequency across our trained model, the base model, and frontier baselines. On OpenCLOC, the trained policy uses value caching (60 vs. 25 instances), loop hoisting (50 vs. 25), and unrolling (42 vs. 16) substantially more frequently than the base model, and uses native math functions more often than any baseline. On OpenCLKernelBench, the trained policy instead favors host-side optimizations, particularly non-blocking transfers and command-queue synchronization changes, leaving kernel code unchanged on 65% of tasks. We additionally quantify the extent and location of code modifications, highlighting how learned optimization behavior differs across tasks and models.
How is the reward function designed? There is a big exploration space there; is there any scientific study of it? How much does it affect the training? In what way?
We have added detailed reward function ablations (Appendix A), investigating the speedup reward transformation (linear vs. logarithmic), the minimum speedup threshold used in assigning rewards, and the effect of disabling the exact-copy gate.
The results show that these choices substantially affect training outcomes. Lowering the speedup threshold increases correctness but reduces optimization performance, with the model favoring smaller, more conservative modifications. For example, on OpenCLOC, reducing the threshold from 1 to 0 increases correct@1 from 0.53 to 0.85 but decreases mean speedup from 2.07 to 1.26. Using a linear rather than logarithmic speedup reward also reduces performance across speedup metrics, including higher thresholds. Finally, disabling the copy gate increases exact copying and significantly reduces optimization performance on both datasets. We further examine how reward choices affect the magnitude and diversity of code modifications, providing additional insight into the observed tradeoffs. These experiments support the reward design decisions described in Section 4 and demonstrate that reward formulation is important for learning effective optimizations from noisy hardware feedback.
part 1/2
Thank you for your thoughtful review and suggestions. We have added experiments addressing all four concerns: comparisons with SFT self-distillation and iterative refinement, two proprietary frontier model baselines, analysis of the learned optimization strategies, and detailed reward function ablations. We discuss each below. Changes to the revised manuscript are flagged in orange.
We agree that comparisons against other training and inference-time approaches are valuable, and have added the following experiments.
Best-of-N sampling vs. iterative refinement (Appendix D). Our main evaluations use single-turn best-of-32 sampling. We now compare this against eight independent trajectories with four turns each, matching the total budget of 32 generated candidates. Each refinement turn receives execution feedback from the target hardware, including compilation/runtime errors, correctness results, and measured speedups. Our results (Figures 6–7) show that the RL-trained model performs similarly under both test-time strategies, whereas the base model benefits from iterative refinement on OpenCLKernelBench but performs better with independent sampling on OpenCLOC.
Crucially, the trained model substantially outperforms the base model under either approach. Even with multiple rounds of execution-based feedback, the base model does not approach the performance achieved by the trained model without refinement. This suggests that, under the evaluated budgets, providing hardware feedback at inference time does not substitute for learning from that feedback during RL training.
SFT self-distillation (Appendix E). We have also added an experiment following a similar approach to SuperCoder, using iterative self-distillation from the base Qwen2.5-Coder-32B-Instruct model. We sample 16 candidates per training task, select the best under our verified reward function, and use these as supervised targets, approximately matching the generation and training-step budgets of GRPO. SFT improves correct@1 from 0.27 to 0.41 and fast$_{1.1}$@8 from 0.18 to 0.26 on OpenCLOC, but substantially underperforms GRPO (0.51 and 0.46, respectively). Moreover, mean best-found speedup actually decreases from 1.63 to 1.37 under SFT. Our analysis also shows that self-distillation reduces candidate diversity relative to the base model, whereas GRPO increases it. We hypothesize that self-distillation increasingly concentrates the policy on conservative modifications with relatively modest speedups, limiting its ability to discover larger optimizations.
These results are consistent with prior kernel optimization work, including TritonRL, Dr. Kernel, and AutoTriton, which report substantial improvements from RL beyond SFT, including when supervision is distilled from stronger models. We have added SuperCoder to Related Work (L133–134) and discuss these comparisons in Appendix E. We also identify SFT using optimizations distilled from frontier models as a promising direction for future work.