- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3(+1)
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
ProcessKernel learns an outcome reward model (ORM) from executed complete Triton candidates and calibrates its scores with measured anchors. The calibrated ORM supplies approximate continuation labels for training a process reward model (PRM). At inference, the PRM guides segment-level beam search and the ORM chooses a finished program without executing each search candidate. Across eight generation backbones on KernelBench Levels 1 and 2, the method improves correctness and Fast1 over first-sample generation and includes matched search controls. Its reported 76× saving compares approximately 707 GPU-hours of estimated execution-based continuation labeling with 9.3 GPU-hours of ORM scoring for the same continuations.
Critical strengths
- The calibrated ORM and PRM have clear complementary roles: measured outcomes inform partial-code search, and finished candidates receive a final learned selection score.
- Results span eight generators and include Random Beam, DeepConf, candidate-pool analysis, ranking-fidelity measurements, and bootstrap intervals for some analyses. Table 1 also reports speed coverage against
torch.compile, and Figure 9 gives a concrete non-delegating Triton max-pooling improvement. These controls and examples provide stronger evidence than a comparison with single-sample generation alone. - Training, search, grading, and H100 settings are documented, including five correctness trials and 100 timing trials. The manuscript provides an anonymous replication-package link.
Critical weaknesses
- Many selected correct programs delegate heavy operations to PyTorch: 40.0% on Level 1 and 99.0% on Level 2, versus 24.2% and 98.0% among correct first completions (Table 13, p. 26). In the harder Level-1 cases solved only by search, 64 of 75 use such delegation (p. 27). This is allowed by the task prompt, and the paper shows a non-delegating max-pooling gain; Level-2 delegation was already common before search. The missing result is an aggregate performance breakdown for genuinely low-level Triton work versus the permitted library-delegation strategy.
- The 13,309 training tasks are derived from KernelBook/PyTorch modules, yet the manuscript does not report a semantic-neighbor, graph-template, or near-code overlap audit against KernelBench. The absence of that audit limits the unseen-task interpretation; it does not establish leakage.
- The 76× ratio compares estimated execution time for labeling sampled continuations with ORM scoring time. It omits initial measured-data acquisition, reward-model training, continuation generation, and inference scoring, so it should not be read as an end-to-end cost reduction. The paper otherwise gives useful details of five FP32 correctness trials and 100 timing trials.
- ORM ranking is strong for correct-versus-incorrect finished kernels (92.3% pair accuracy) but near chance across all speed pairs (51.2%), improving to 76.6% when speeds differ by at least 2×. The evidence for fine-grained speed selection among correct candidates is therefore weaker than the evidence for finding correct ones.
- Performance relative to
torch.compileremains more limited than relative to eager PyTorch, despite being reported in Table 1. Across the eight generators, ProcessKernel's Level-2 Fast1 coverage is 23–57% against eager but only 6–14% againsttorch.compile; AutoTriton Level 2 falls from 9% to 7% againsttorch.compileeven while correctness and eager-relative Fast1 improve. This bounds the speed claim while leaving the same-budget correctness gains intact.
Questions for authors
- Can you stratify existing Correct, Fast1, and Random Beam comparisons by heavy PyTorch delegation or by how much computation the Triton kernel performs? A result on the non-delegating subset would clarify the low-level generation claim without changing the current benchmark rules.
- What semantic, graph-template, and near-code overlap checks were performed between the training tasks and KernelBench, and what happens if the closest neighbors are excluded?
- What is the amortized end-to-end cost after measured-data collection, two reward-model training runs, continuation generation, anchor labeling, and inference scoring are included?
- On existing candidate pools containing only correct implementations, how close is ORM-selected latency to a hardware-measured oracle and to random selection?
7 comments
Q2: The 13,309 training tasks are derived from KernelBook/PyTorch modules, yet the manuscript does not report a semantic-neighbor, graph-template, or near-code overlap audit against KernelBench. The absence of that audit limits the unseen-task interpretation; it does not establish leakage. What semantic, graph-template, and near-code overlap checks were performed between the training tasks and KernelBench, and what happens if the closest neighbors are excluded?
While meaningful exact or semantic overlap between kernel-generation problems is not straightforward to define precisely, we adopted the filtering approach proposed in [1] as a practical way to assess potential overlap. We examined both semantic, graph-template and near-code overlap between the KernelBook problems used to train our ORM and PRM and the KernelBench L1/L2 evaluation problems:
- For semantic overlap, we use similarity of jina-embeddings-v3 embeddings (the model of the filter in [1]), reported as the L2 distance τ used there.
- For the near-code overlapping: we use exact match and token Jaccard similarity of the problem files [1].
- For Graph-template overlapping: two problems can differ in text but perform the same computation. In the spirit of structure-based clone detection, which compares programs by their structure rather than their text [2], we trace each module's forward pass on PyTorch's meta device and record its sequence of ATen operators, without running any kernel. One check requires an identical sequence of at least three operators (identical template). The other compares the operators of two problems by their Jaccard similarity [1] (op-set match).
- Exact and syntactic matching. We found no overlap under exact matching or token-level Jaccard similarity, across thresholds ranging from 0.70 to 0.95 (compared with the threshold of >0.8 suggested in [1]).
- Semantic matching. We used jina-embeddings-v3 to identify potentially overlapping problems based on their embedding-space L2 distance, with threshold (i.e., cosine similarity > ). After excluding potentially overlapping evaluation problems under each semantic threshold, we recomputed the aggregated results in Table 1. As shown below, ProcessKernel remains the best method in varying generators and task levels. For example, at = 0.40 its Correctness is 49.4 vs. 35.4 for the strongest baseline on L1 (+14.0%), and 59.5 vs. 35.6 on L2 (+23.9%), and at = 0.45 its Correctness is 56.2 vs. 39.1 for the strongest baseline on L1 (+17.1%), and 58.0 vs. 38.6 on L2 (+19.4%) ([1] suggested threshold < 0.45).
- Graph-template: after removing the graph-template overlapping problems, ProcessKernel remains the best method in Correct and Fast₁ under every filter (Table B). With an identical template, its Correct is 54.1 vs. 35.0 for the strongest baseline on L1; at op-set Jaccard ≥ 0.6, it is 53.7 vs. 29.6 on L2.
Table A1. τ = 0.35.
| Method | L1 Correct | L1 Fast₁ | L1 Mean | L2 Correct | L2 Fast₁ | L2 Mean |
|---|---|---|---|---|---|---|
| Nucleus sampling | 29.7 | 18.6 | 0.35 | 31.0 | 21.2 | 0.40 |
| USC | 34.3 | 20.2 | 0.41 | 37.4 | 27.4 | 0.45 |
| Likelihood | 26.5 | 14.7 | 0.30 | 28.9 | 20.5 | 0.38 |
| Random Beam | 31.0 | 16.8 | 0.35 | 29.9 | 19.2 | 0.40 |
| DeepConf | 29.8 | 15.6 | 0.33 | 31.8 | 20.5 | 0.41 |
| ProcessKernel (Ours) | 52.6 | 32.1 | 0.59 | 60.6 | 44.5 | 0.73 |
Table A2. τ = 0.40.
| Method | L1 Correct | L1 Fast₁ | L1 Mean | L2 Correct | L2 Fast₁ | L2 Mean |
|---|---|---|---|---|---|---|
| Nucleus sampling | 31.7 | 17.4 | 0.38 | 28.4 | 19.2 | 0.30 |
| USC | 35.4 | 19.2 | 0.45 | 35.6 | 25.1 | 0.39 |
| Likelihood | 29.6 | 15.2 | 0.34 | 26.6 | 18.5 | 0.30 |
| Random Beam | 32.6 | 16.1 | 0.39 | 27.7 | 16.6 | 0.36 |
| DeepConf | 34.1 | 15.5 | 0.39 | 29.7 | 18.0 | 0.35 |
| ProcessKernel (Ours) | 49.4 | 27.4 | 0.56 | 59.5 | 42.7 | 0.65 |
Table A3. τ = 0.45.
| Method | L1 Correct | L1 Fast₁ | L1 Mean | L2 Correct | L2 Fast₁ | L2 Mean |
|---|---|---|---|---|---|---|
| Nucleus sampling | 37.5 | 17.2 | 0.29 | 28.0 | 20.5 | 0.28 |
| USC | 40.6 | 20.3 | 0.36 | 38.6 | 29.5 | 0.38 |
| Likelihood | 35.9 | 18.8 | 0.33 | 25.0 | 17.8 | 0.26 |
| Random Beam | 37.9 | 16.0 | 0.26 | 27.7 | 17.1 | 0.28 |
| DeepConf | 39.1 | 15.6 | 0.25 | 30.7 | 20.1 | 0.31 |
| ProcessKernel (Ours) | 56.2 | 35.9 | 0.45 | 58.0 | 43.9 | 0.62 |
Table B1. Identical template (≥ 3 operators).
| Method | L1 Correct | L1 Fast₁ | L1 Mean | L2 Correct | L2 Fast₁ | L2 Mean |
|---|---|---|---|---|---|---|
| Nucleus sampling | 30.9 | 20.1 | 0.38 | 30.8 | 21.3 | 0.40 |
| USC | 35.0 | 21.2 | 0.39 | 37.1 | 27.0 | 0.44 |
| Likelihood | 27.2 | 15.3 | 0.28 | 28.7 | 20.3 | 0.38 |
| Random Beam | 32.2 | 17.8 | 0.35 | 29.7 | 18.9 | 0.40 |
| DeepConf | 31.1 | 16.8 | 0.32 | 31.4 | 20.1 | 0.41 |
| ProcessKernel | 54.1 | 33.8 | 0.58 | 60.5 | 44.2 | 0.73 |
Table B2. Identical op set (Jaccard = 1.0).
| Method | L1 Correct | L1 Fast₁ | L1 Mean | L2 Correct | L2 Fast₁ | L2 Mean |
|---|---|---|---|---|---|---|
| Nucleus sampling | 28.8 | 19.2 | 0.42 | 30.7 | 21.2 | 0.40 |
| USC | 39.4 | 26.0 | 0.65 | 37.2 | 27.2 | 0.45 |
| Likelihood | 26.9 | 14.4 | 0.35 | 29.0 | 20.5 | 0.38 |
| Random Beam | 35.0 | 20.4 | 0.51 | 29.6 | 18.7 | 0.40 |
| DeepConf | 37.5 | 21.2 | 0.53 | 31.5 | 20.0 | 0.41 |
| ProcessKernel | 50.0 | 35.6 | 0.65 | 60.5 | 44.1 | 0.73 |
Table B3. Op-set Jaccard ≥ 0.6.
| Method | L1 Correct | L1 Fast₁ | L1 Mean | L2 Correct | L2 Fast₁ | L2 Mean |
|---|---|---|---|---|---|---|
| Nucleus sampling | 23.6 | 9.7 | 0.18 | 22.5 | 12.3 | 0.30 |
| USC | 34.7 | 19.4 | 0.35 | 29.6 | 19.7 | 0.33 |
| Likelihood | 26.4 | 12.5 | 0.23 | 21.3 | 11.8 | 0.28 |
| Random Beam | 27.6 | 12.7 | 0.20 | 23.2 | 11.6 | 0.33 |
| DeepConf | 29.2 | 9.7 | 0.17 | 25.9 | 13.7 | 0.34 |
| ProcessKernel | 47.2 | 30.6 | 0.40 | 53.7 | 36.8 | 0.65 |
[1] Tehrani, Ali, et al. "Fine-Tuning GPT-5 for GPU Kernel Generation." arXiv preprint arXiv:2602.11000 (2026). [2] Komondoor, Raghavan, and Susan Horwitz. "Using Slicing to Identify Duplication in Source Code." SAS 2001.
Q3: The 76× ratio compares estimated execution time for labeling sampled continuations with ORM scoring time. It omits initial measured-data acquisition, reward-model training, continuation generation, and inference scoring, so it should not be read as an end-to-end cost reduction. The paper otherwise gives useful details of five FP32 correctness trials and 100 timing trials. What is the amortized end-to-end cost after measured-data collection, two reward-model training runs, continuation generation, anchor labeling, and inference scoring are included?
We have revised the text and updated our cost accounting to include the previously omitted costs. Specifically, profiling the ORM training data requires approximately 7 H100 GPU-hours, anchor profiling 2 GPU-hours, ORM training 4 GPU-hours, and ORM-based labeling (including scoring and calibration) 9 GPU-hours, totaling approximately 22 H100 GPU-hours. After including these costs, the estimated GPU-time savings relative to profiling-based labeling (around 707 H100 GPU-hours) are approximately 32X speedup (corresponding to around 96.89% time cost reduction), while providing high-quality supervision for PRM training. We have also clarified that this comparison accounts for the listed training and labeling costs, rather than representing an end-to-end cost reduction.
Q4: ORM ranking is strong for correct-versus-incorrect finished kernels (92.3% pair accuracy) but near chance across all speed pairs (51.2%), improving to 76.6% when speeds differ by at least 2×. The evidence for fine-grained speed selection among correct candidates is therefore weaker than the evidence for finding correct ones.
Although our ORM’s overall speed-pair ranking accuracy is 51.2%, this aggregate result is dominated by near-tied comparisons: more than 91% of code pairs in the test set differ in performance by at most 0.5%. For pairs with larger performance gaps, the ORM identifies the faster implementation much more reliably.
Table A. Accuracy on speed pairs, by speed gap (the 73,075 speed pairs of Table 12, from 46 problems; 95% bootstrap CI over problems). A random order gives 50.0% in every row, as in Table 12.
| Speed gap between the two kernels | Random order (%) | ORM (%) |
|---|---|---|
| All | 50.0 | 51.2 |
| ≥ 1.05× | 49.7 | 60.8 |
| ≥ 1.2× | 50.3 | 62.9 |
| ≥ 1.5× | 49.8 | 71.2 |
| ≥ 2.1× | 50.4 | 76.6 |
| ≥ 3.0× | 50.1 | 79.4 |
We sincerely thank all reviewers for their constructive feedback and overall positive rating of our work! We are pleased that all reviewers appreciate the ideas behind our approach. Reviewer iH7x praised the “Learning to Rank + process reward modeling idea” and described our segment-level beam search as “an interesting idea on where to start using the PRM”. The reviewers further recognize turning sparse execution feedback into reusable process guidance as “a sensible and meaningful direction” (Reviewer vsyQ) and highlight the “clear complementary roles” of the calibrated ORM and PRM (Reviewer TiWp), while also praising our “very comprehensive and well-designed” evaluation (Reviewer vsyQ) and “Strong results” (Reviewer iH7x).
Q1: Many selected correct programs delegate heavy operations to PyTorch: 40.0% on Level 1 and 99.0% on Level 2, versus 24.2% and 98.0% among correct first completions (Table 13, p. 26). In the harder Level-1 cases solved only by search, 64 of 75 use such delegation (p. 27). This is allowed by the task prompt, and the paper shows a non-delegating max-pooling gain; Level-2 delegation was already common before search. The missing result is an aggregate performance breakdown for genuinely low-level Triton work versus the permitted library-delegation strategy. Can you stratify existing Correct, Fast1, and Random Beam comparisons by heavy PyTorch delegation or by how much computation the Triton kernel performs? A result on the non-delegating subset would clarify the low-level generation claim without changing the current benchmark rules.
The table below reports performance gains separately for kernels that delegate operations to PyTorch and those that do not. These results suggest that our method can effectively leverage PyTorch delegation while also generating efficient Triton kernels without delegation.