acceptodds
Under review as a conference paper at ICLR 2027

Hardware-in-the-Loop RL for OpenCL Optimization on Mobile GPUs

Abstract

As large language models (LLMs) make rapid strides in code generation, excitement is growing about their potential to optimize system kernels, but the field's datasets and evaluation infrastructure are primarily anchored in the NVIDIA ecosystem. OpenCL offers a common programming interface across vendors whose portable implementations can benefit from hardware-specific optimizations. However, learning these optimizations requires executable, correctness-verifiable workloads and reliable feedback from the target device. We investigate whether LLMs can use live execution feedback during RL training to optimize OpenCL kernels for resource-constrained mobile GPUs. We build and open-source two datasets: OpenCLKernelBench, which adapts KernelBench Level 1 with OpenCL reference implementations, and OpenCLOC, 620 curated kernels mined from GitHub. Both pair kernel implementations with test cases and executable host code. We also present a framework for hardware-in-the-loop reinforcement learning with verified rewards (HIL-RLVR) that includes a custom training environment, protocols for quantifying and reducing device noise, and novel reward hacking interventions. Finally, we train models to optimize kernels for Adreno GPUs on Snapdragon mobile processors. Relative to their shared Qwen2.5-Coder-32B-Instruct base model, increases from 9% to 26% on OpenCLOC and from effectively 0% to 18% on OpenCLKernelBench, achieving performance competitive with GPT-OSS-120B on both datasets. We also train an OpenCL policy on an NVIDIA Jetson AGX Thor and find that the learned optimizations are hardware-specific; policies remain correct when evaluated on the other device type, but their speedups do not transfer. To our knowledge, this is the first application of LLM-based RLVR to OpenCL kernel optimization. Our datasets and framework make resource-constrained deployment hardware a practical source of feedback for learning kernel optimizations.

Primary area:
reinforcement learning
Keywords:
large language model, reinforcement learning, kernel optimization, hardware-in-the-loop, verified rewards

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GfJ4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The work presents a hardware-in-the-loop RL framework for optimizing OpenCL kernels on mobile GPUs. There are two datasets introduced, OpenCLOC and OpenCLKernelBench, along with an execution-driven training infrastructure that addresses challenges in stablizing the reward computation. Using GRPO to fine-tune Qwen2.5-Coder-32B, the method can improve kernel correctness and performance on mobile GPUs. The paper claims this is the first RLVR applied to OpenCL kernels.

Critical strengths
  1. First to apply RLVR for OpenCL kernel optimization. Mobile devices may be more of practical value since they need to run inference and there may be a growing need for that.
  2. There is a clear dataset contribution.
  3. Empirical results seem promising.
Critical weaknesses
  1. Unclear practical motivation for RL training. This paper avoids benchmark frontier models like Astra or Claude. If kernel optimization is a one-time effort before deployment, why don't just use the frontier model? what's the gap between the RL trained model and the best frontier model?

  2. This paper need more experiments to compare against RL training approach with other approaches such as supervised fine-tuning (rollout and select the best ones for imitation learning), iterative refinement, best-of-N sampling, which has been done for other related works like SuperCoder (but for assembly domains other than OpenCL) and this paper should discuss the missing citations as well.

  3. A detailed study on what the model learns and what program transformations are done is missing. Please consider adding them.

Questions for authors
  1. How is the reward function designed? There is a big exploration space there; is there any scientific study of it? How much does it affect the training? In what way?
2 comments

Thank you for your thoughtful review and suggestions. We have added experiments addressing all four concerns: comparisons with SFT self-distillation and iterative refinement, two proprietary frontier model baselines, analysis of the learned optimization strategies, and detailed reward function ablations. We discuss each below. Changes to the revised manuscript are flagged in orange.

This paper need more experiments to compare against RL training approach with other approaches such as supervised fine-tuning (rollout and select the best ones for imitation learning), iterative refinement, best-of-N sampling, which has been done for other related works like SuperCoder (but for assembly domains other than OpenCL) and this paper should discuss the missing citations as well.

We agree that comparisons against other training and inference-time approaches are valuable, and have added the following experiments.

  • Best-of-N sampling vs. iterative refinement (Appendix D). Our main evaluations use single-turn best-of-32 sampling. We now compare this against eight independent trajectories with four turns each, matching the total budget of 32 generated candidates. Each refinement turn receives execution feedback from the target hardware, including compilation/runtime errors, correctness results, and measured speedups. Our results (Figures 6–7) show that the RL-trained model performs similarly under both test-time strategies, whereas the base model benefits from iterative refinement on OpenCLKernelBench but performs better with independent sampling on OpenCLOC.

    Crucially, the trained model substantially outperforms the base model under either approach. Even with multiple rounds of execution-based feedback, the base model does not approach the performance achieved by the trained model without refinement. This suggests that, under the evaluated budgets, providing hardware feedback at inference time does not substitute for learning from that feedback during RL training.

  • SFT self-distillation (Appendix E). We have also added an experiment following a similar approach to SuperCoder, using iterative self-distillation from the base Qwen2.5-Coder-32B-Instruct model. We sample 16 candidates per training task, select the best under our verified reward function, and use these as supervised targets, approximately matching the generation and training-step budgets of GRPO. SFT improves correct@1 from 0.27 to 0.41 and fast$_{1.1}$@8 from 0.18 to 0.26 on OpenCLOC, but substantially underperforms GRPO (0.51 and 0.46, respectively). Moreover, mean best-found speedup actually decreases from 1.63 to 1.37 under SFT. Our analysis also shows that self-distillation reduces candidate diversity relative to the base model, whereas GRPO increases it. We hypothesize that self-distillation increasingly concentrates the policy on conservative modifications with relatively modest speedups, limiting its ability to discover larger optimizations.

    These results are consistent with prior kernel optimization work, including TritonRL, Dr. Kernel, and AutoTriton, which report substantial improvements from RL beyond SFT, including when supervision is distilled from stronger models. We have added SuperCoder to Related Work (L133–134) and discuss these comparisons in Appendix E. We also identify SFT using optimizations distilled from frontier models as a promising direction for future work.

Unclear practical motivation for RL training. This paper avoids benchmark frontier models like Astra or Claude. If kernel optimization is a one-time effort before deployment, why don't just use the frontier model? what's the gap between the RL trained model and the best frontier model?

We have added ChatGPT-6.1 Sol and Claude Opus 5.5 as proprietary frontier baselines (Tables 1–2). Both achieve strong results comparable to GLM-5.3 on OpenCLOC, while Claude Opus 5.5 is particularly strong on OpenCLKernelBench.

We acknowledge that the strongest frontier models outperform our RL-trained model, and we do not claim that HIL-RLVR training on a smaller model is necessarily preferable to using a frontier model for optimizing a single kernel with a large inference-time budget. Our motivation is instead to investigate whether hardware optimization capabilities can be learned from verified rewards on the target hardware. Our experiments establish that this approach is effective, with the trained model substantially outperforming its base model and achieving performance competitive with or superior to the considerably larger GPT-OSS-120B on unseen tasks. These results are encouraging and motivate follow-up investigation into training larger open-weight models under the same setup, but we do not claim parity with the strongest frontier models.

More broadly, kernel optimization is not necessarily a one-time effort, particularly as both GPU architectures and algorithms continue to evolve. We view improving model capabilities through training and scaling inference-time search as complementary approaches. Learning reusable optimization capabilities for a target hardware architecture, rather than relying entirely on per-kernel inference-time search, is therefore a worthwhile research direction.

A detailed study on what the model learns and what program transformations are done is missing. Please consider adding them.

We have added an analysis of the optimization strategies used by the base and trained models and frontier baselines (Appendix F). We use AST-based heuristics and an LLM judge to categorize program transformations and compare their frequency across our trained model, the base model, and frontier baselines. On OpenCLOC, the trained policy uses value caching (60 vs. 25 instances), loop hoisting (50 vs. 25), and unrolling (42 vs. 16) substantially more frequently than the base model, and uses native math functions more often than any baseline. On OpenCLKernelBench, the trained policy instead favors host-side optimizations, particularly non-blocking transfers and command-queue synchronization changes, leaving kernel code unchanged on 65% of tasks. We additionally quantify the extent and location of code modifications, highlighting how learned optimization behavior differs across tasks and models.

How is the reward function designed? There is a big exploration space there; is there any scientific study of it? How much does it affect the training? In what way?

We have added detailed reward function ablations (Appendix A), investigating the speedup reward transformation (linear vs. logarithmic), the minimum speedup threshold used in assigning rewards, and the effect of disabling the exact-copy gate.

The results show that these choices substantially affect training outcomes. Lowering the speedup threshold increases correctness but reduces optimization performance, with the model favoring smaller, more conservative modifications. For example, on OpenCLOC, reducing the threshold from 1 to 0 increases correct@1 from 0.53 to 0.85 but decreases mean speedup from 2.07 to 1.26. Using a linear rather than logarithmic speedup reward also reduces performance across speedup metrics, including higher thresholds. Finally, disabling the copy gate increases exact copying and significantly reduces optimization performance on both datasets. We further examine how reward choices affect the magnitude and diversity of code modifications, providing additional insight into the observed tradeoffs. These experiments support the reward design decisions described in Section 4 and demonstrate that reward formulation is important for learning effective optimizations from noisy hardware feedback.

Reviewer_Rg8bOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies LLM-based OpenCL optimization using execution feedback from mobile GPUs. It introduces OpenCLOC for kernel-only optimization and OpenCLKernelBench for joint host/device optimization, together with a hardware-in-the-loop GRPO framework incorporating correctness checks, timing controls, and reward-hacking defenses. Training Qwen2.5-Coder-32B-Instruct on Snapdragon devices raises reported fast1.5@8 from 9% to 26% on OpenCLOC and from 0% to 17.5% on OpenCLKernelBench (Tables 1–2). Experiments with Jetson Thor show that correctness largely transfers across devices, while most learned speedup gains remain hardware-specific.

Critical strengths
  1. The executable tasks and verification infrastructure provide a useful foundation for learning optimizations beyond the usual CUDA setting. Nondeterminism checks, no-op rejection, reference-output comparisons, and operator-specific input ranges address concrete ways that apparently correct programs can bypass useful computation. This practical groundwork makes mobile GPU feedback more accessible for subsequent research.
Critical weaknesses
  1. The effectiveness of the central reliability measures is insufficiently established. The paper does not quantify how much interleaved timing, AST-based copy detection, or the chosen reward shape improve measurement and training reliability. Section 4.3.2 describes the drawbacks of rewarding slower kernels or using linear rewards without comparative results. Consequently, the importance of these measures to the reported optimization gains remains unclear.

  2. Missing statistical details and inconsistent reporting make the main results difficult to assess. Section 4.2 defines a geometric mean over timings, but does not fully specify aggregation over test inputs or the bootstrap-smoothed fast metric; the cited Appendix E.1 contains only a heading. Section 6 reports 95% intervals over five protocols spanning only two training runs without explaining interval construction or shared-run dependencies. This matters particularly for Table 2's trained fast1.5@8 estimate of 0.175, with an interval of [0.000, 0.493]. Conflicting result descriptions and task counts further obscure the evaluated population.

Questions for authors
  1. Please reconcile the references to Figures 5.1/6.1/6.2 with Figures 2/3/4 and the §6.2 numbers with Table 2, which reports correct@1 increasing from 0.294 to 0.800. How do Table 2's 100 held-out tasks relate to Table 3's 20 tasks, given that §3.2 describes 100 tasks in total? Please specify the training/test split and whether these discrepancies reflect textual errors or different experimental versions.

  2. Please complete Appendix E.1 with the bootstrap procedure, aggregation across test inputs, and construction of 95% intervals. How are dependencies from shared training runs handled across the five protocols, and what are the individual protocol results?

  3. Please reconcile the OpenCLOC count: 575 + 45 − 21 gives 599, whereas §3.1 reports 620. What are the counts at each stage and the dataset version underlying Table 1?

  4. Please quantify the effects of interleaved timing, copy detection, and reward shaping using controlled comparisons. How often are identical programs incorrectly classified as faster under each timing strategy, and how do the reward variants in §4.3.2 affect independently remeasured optimization gains?

2 comments

We thank the reviewer for their detailed and constructive feedback. Reliability of hardware-based rewards and rigorous evaluation are central concerns of our work, and we appreciate the opportunity to further strengthen their empirical characterization. In response to the reviewer's suggestions, we have added controlled experiments quantifying the effects of interleaved timing, AST-based copy detection, and reward shaping; expanded our discussion of evaluation protocols and statistical reporting; and corrected several textual inconsistencies. We address each point individually below, with the corresponding changes highlighted in blue in the revised manuscript.

Quantifying effectiveness of interleaved timing

The paper does not quantify how much interleaved timing […] improve measurement and training reliability.

Please quantify the effects of interleaved timing […] using controlled comparisons. How often are identical programs incorrectly classified as faster under each timing strategy?

We have added controlled experiments demonstrating that interleaved timing substantially reduces measurement errors (Section 4.1.2, Figure 2; Appendix H).

Using our hardware profiling data, we compare interleaved and blocked timing by measuring each kernel against itself, such that any deviation from a speedup of one is necessarily measurement error. Figure 2 shows that self-speedup errors exceeding 10% occur at least 60 times more frequently with blocked timing than with interleaved timing, across both datasets. The blocked strategy even produces apparent speedups of 2× or greater when measuring identical implementations.

We additionally investigate systematic variation between devices in the same pool (Appendix H.1, Figure 12). When baseline and optimized timings are taken from different devices, 12% of OpenCLOC kernels and 15% of OpenCLKernelBench kernels exhibit apparent speedup errors of at least 5%. In contrast, keeping each baseline/optimized timing pair on the same device reduces these rates to approximately 0.1–0.5%, even when different pairs are distributed across devices. Finally, Appendix H.2 (Figures 13–14) provides additional ablations comparing GMOR against ROM speedup estimation and randomized against fixed baseline-first measurement ordering. We find that interleaving is substantially more important than either of these secondary choices.

Together, these experiments quantify the importance of paired, interleaved measurements for obtaining reliable performance feedback on our target hardware.

Quantifying effects of AST-based copy detection and reward shaping

The paper does not quantify how much […] AST-based copy detection, or the chosen reward shape improve measurement and training reliability. Section 4.3.2 describes the drawbacks of rewarding slower kernels or using linear rewards without comparative results.

Please quantify the effects of […] copy detection, and reward shaping using controlled comparisons. How […] do the reward variants in §4.3.2 affect independently remeasured optimization gains?

We have added controlled training ablations of the exact-copy gate and reward function design (Appendix A, Tables 5–8).

AST-based copy detection (Appendix A.1). Disabling the copy gate during training increases copy@8 from approximately 1% to 17% on OpenCLOC and from 18% to 45% on OpenCLKernelBench. This behavior also substantially degrades optimization performance: fast_1.5@8 decreases from 0.29 to 0.18 on OpenCLOC and from 0.18 to 0.05 on OpenCLKernelBench. These results establish that the copy gate is important not only for preventing reward hacking, but also for learning effective optimizations.

Reward shaping (Appendix A.2). We ablate logarithmic versus linear speedup rewards and vary the minimum speedup threshold below which candidates receive the same reward as incorrect implementations. The results reveal a clear tradeoff between correctness and optimization performance. For example, on OpenCLOC, lowering the threshold from 1 to 0 increases correct@1 from 0.53 to 0.85 but reduces mean best-found speedup from 2.07 to 1.26. Our additional edit-distance analysis (Appendix A.2.1) suggests that this occurs because more permissive rewards encourage smaller, more conservative modifications and reduce candidate diversity.

Using a linear rather than logarithmic speedup reward also reduces optimization performance, including at higher speedup thresholds. We hypothesize that this reflects less favorable training dynamics when large speedups dominate relative advantages within GRPO groups.

All reward variants are evaluated using our held-out tasks and hardware-based evaluation protocol, including measurements on a separate device pool. These experiments substantiate the reward design choices motivated in Section 4 and demonstrate their effects on both learned behavior and independently evaluated optimization performance.

Clarifying statistical reporting and performance aggregation

We have substantially expanded our evaluation methodology sections (Sections 4.2 and 6.1; Appendices B and C).

Aggregation across test inputs and tasks. Section 4.2.1 now explicitly describes how we aggregate speedup across test inputs: each candidate must pass correctness checks on five test inputs, and correct candidates are timed on all five. We compute the geometric mean of paired timing ratios (GMOR) independently for each test input, then average the five per-input speedups. Section 4.2.2 and Appendix C clarify aggregation over candidates and tasks. Our main speedup metric selects the best correct candidate among 32 attempts per task, scores tasks with no correct attempts as zero, and reports the arithmetic mean over all held-out tasks. Appendix C also provides results under alternative aggregation conventions and expanded fast$_p$@kk metrics.

Statistical reporting and evaluation dependencies. We have revised the reporting from 95% confidence intervals to means ± one standard error, and added Appendix B to explain how these values are calculated. In particular, we distinguish between variability across independent training seeds and variability from evaluation sampling and hardware measurements, rather than treating evaluations sharing a training run as independent training replicates. Each trained model is evaluated from two independently seeded training runs, with three evaluations per training seed: one reference evaluation, one with newly sampled completions, and one remeasurement of the reference completions on a different device pool. Appendix B now provides the individual evaluation results and the decomposition of the reported standard errors (Tables 9–10).

Correcting inconsistencies and omissions

We thank the reviewer for identifying several reporting errors in the original submission, all of which we have corrected:

  • Figure references and result descriptions: References to Figures 5.1/6.1/6.2 have been corrected, and the numerical descriptions in Section 6 have been reconciled with the results tables.
  • Clarification of dataset sizes and train/test splits: OpenCLOC contains 595 kernels retained from LLMPerf and 46 newly scraped kernels, followed by removal of 21 kernels after manual auditing, yielding 620 kernels in total (520 training, 100 evaluation). The original intermediate counts (545 and 45 instead of 595 and 46) were incorrect. OpenCLKernelBench separately contains 100 tasks (80 training, 20 evaluation). Revised Tables 1–2 explicitly identify the relevant dataset and split, while Tables 3–4 report cross-hardware results on the corresponding held-out tasks.
  • Bootstrap-smoothed fast$_p$@kk: We investigated this alternative metric but found no measurable effect on the results. We therefore removed it from the reported evaluation rather than introducing an unnecessary additional estimator. The single textual reference and empty appendix heading in the original submission have been deleted.

We appreciate the reviewer's attention to these issues. The revised manuscript now provides direct experimental evidence for the reliability measures, a more complete and transparent evaluation methodology, and corrected reporting throughout.

Reviewer_JjQzOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a systematic study of RL for LLM-based OpenCL kernel optimization for mobile GPUs. It introduces two datasets, OpenCLKernelBench and OpenCLOC, sourced and curated from KernelBench level 1 problems and GitHub, respectively. The paper further introduces a hardware-in-the-loop RL framework with measurement for device noise reduction and reward hacking prevention. Experiments with the RL framework on the collected dataset show the effectiveness of the proposed framework for learning correct and fast OpenCL kernels, yet the speedup does not transfer across devices.

Critical strengths
  1. The paper studies a timely and important research topic on kernel optimization, with a novel focus on resource-constrained mobile devices.
  2. The introduced datasets are rigorously constructed and are valuable resources for learning and evaluating LLMs for kernel optimization on mobile devices.
  3. The paper presents rigorous procedures for mitigating device noise, establishing faithful speedup measurements, and mitigating reward hacking.
  4. The proposed RL framework efficiently adapts to the mobile hardware and demonstrates its effectiveness through experiments. It is a good reference for future development of on-device RL.
Critical weaknesses
  1. While the same-device performance gain is promising, as shown in Table 3, the speedup gains obtained through policy learning on one type of device can not be reliably transferred to other devices, limiting the potential application.
Questions for authors

Is there any potential approach that can help improve cross-device transfer?

1 comment

Thank you for your thoughtful review.

While the same-device performance gain is promising, as shown in Table 3, the speedup gains obtained through policy learning on one type of device can not be reliably transferred to other devices, limiting the potential application.

We appreciate you highlighting this important question. We have revised Section 7 to better distinguish hardware-specific optimization from transferable optimization capabilities, and expanded our cross-hardware comparisons to include the untrained base model evaluated on both architectures.

We agree that the ability to generalize across hardware targets is valuable. However, hardware-specific optimization is also an important motivation for HIL-RLVR—while OpenCL provides a portable programming interface, the optimizations that yield the best performance can differ across GPU architectures. Learning those adaptations directly from the target hardware is therefore one of our intended goals, rather than necessarily a limitation of the approach.

Importantly, our revised results (Tables 3–4) suggest that the models learn both transferable optimization strategies and hardware-specific adaptations. For example, on OpenCLOC evaluated on Snapdragon/Adreno, fast$_{1.1}$@88 increases from 18% for the untrained base model to 32% for the Thor-trained model, compared with 46% for the Adreno-trained model. Conversely, on Thor, the Adreno-trained model achieves 32%, compared with 25% for the base model and 31% for the Thor-trained model. Thus, although training on the target hardware generally yields the strongest performance, training on a different architecture can still improve optimization capabilities relative to the untrained model.

Is there any potential approach that can help improve cross-device transfer?

We see several promising directions for future work. One is multi-device HIL-RLVR, in which the policy receives verified rewards from multiple GPU architectures during training, potentially conditioned on the target device's characteristics. Another approach is to adapt a pretrained optimization policy to a new hardware target using a smaller amount of on-device RL feedback, rather than training a new policy from scratch. We have not yet evaluated these approaches, but believe our cross-platform datasets and training infrastructure provide a useful foundation for investigating them.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Additional Experiments and Responses to Reviewer Feedback

We thank the ACs and reviewers for their time and consideration. We have added several additional experiments and expanded our analyses to address the reviewers' questions. We summarize the most significant additions below; detailed responses to each concern are provided in the individual reviewer rebuttals.

Reliability of speedup measurement and reward design. New controlled experiments demonstrate that interleaved timing reduces large measurement errors by at least 60-fold compared with conventional blocked timing, and that pairing measurements appropriately also mitigates systematic differences between devices in the same pool. We additionally provide extensive reward function ablations, quantifying how exact-copy detection, speedup clipping, and linear versus logarithmic rewards affect training outcomes. These experiments demonstrate that our methodological choices materially improve optimization performance and reveal important tradeoffs between correctness, speedup, and candidate diversity.

Comparisons against other training and inference-time approaches. We have added an SFT self-distillation experiment and a comparison of best-of-N sampling against iterative refinement with execution feedback, using matched generation budgets. The RL-trained model substantially outperforms SFT and maintains its advantage over the base model even when the latter receives multiple rounds of on-device execution feedback during inference. These results provide additional evidence for the value of learning optimization capabilities through hardware-in-the-loop RL.

Stronger baselines and analysis of learned capabilities. We have added two closed frontier baselines, ChatGPT-6.1 Sol and Claude Opus 5.5, complementing the open-weight models in our original submission. We have also added a comparative analysis of the program transformations used by the trained model and the baselines, providing insight into the optimization strategies learned.

Hardware specialization and cross-hardware transfer. We have expanded the cross-hardware results to include the untrained base model on both target architectures. These comparisons show that, although optimization performance generally benefits from training on the target hardware, the learned policies also exhibit meaningful transfer to other devices. This supports a more nuanced interpretation of our findings: hardware-in-the-loop training can teach both transferable optimization capabilities and hardware-specific adaptations.

Finally, we have expanded the description of our evaluation protocol, clarified aggregation across timings, test inputs, and tasks, revised our statistical reporting, and corrected textual and numerical inconsistencies identified by the reviewers. We also report individual evaluation results across training seeds and device pools, making the reported variability directly inspectable.

We believe these additions substantially strengthen the empirical foundation of the paper and sharpen its central contribution: demonstrating that resource-constrained deployment hardware can provide reliable, actionable feedback for learning kernel optimization capabilities through RL. We appreciate the reviewers' suggestions, which have helped us more thoroughly characterize both the effectiveness and the limitations of this approach.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.