- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
CUDACoach trains a Qwen3-8B planner to guide a frozen CUDA coder through kernel repair and optimization. It constructs supervised examples by recovering strategies from successful teacher edits, then retaining a strategy only if the coder can implement it successfully without seeing the teacher's improved kernel. Subsequent GRPO training combines execution and format rewards with a diversity reward that uses historical technique frequencies and within-state log-determinant diversity. At inference, the planner can compare multiple correct kernels and their profiles, while strategy clustering limits redundant implementations. Experiments cover all 250 KernelBench tasks with three downstream coders. The largest improvement is with Qwen3-Coder-Plus: correctness rises from 44.8% to 60.4% and fast1 from 13.2% to 19.2% relative to single-stage iteration.
Critical strengths
-
Verified strategy mining is a concrete and useful supervision design. The frozen coder must realize the recovered strategy without access to the teacher's improved kernel, and its implementation must pass execution checks. This is stronger evidence of actionable guidance than accepting a plausible retrospective explanation alone. The detailed prompts and reward specification also make the approach tractable.
-
The learned planner has strong results in a useful matched-harness comparison. With Qwen3-Coder-Plus, it improves correctness from 42.4% to 60.4% over self-planned compositional iteration, with fast1 increasing from 13.6% to 19.2% (Table 1). Against single-stage iteration, the gains correspond to 39 additional correct tasks and 15 additional faster-than-reference tasks. Evaluation across all three KernelBench levels and multiple coders is informative, even though benefits vary by coder.
Critical weaknesses
-
The mechanism-specific evidence does not yet isolate the proposed contributions. Most directly, the correctness gain attributed to multi-kernel context is difficult to understand with the stated algorithm. Section 3.4 and Appendix use that context only when a correct-kernel pool already exists; an initially incorrect task undergoes repair with an empty pool until its first success, after which historical-best retention preserves correctness. Changing only the optimization context therefore has no apparent causal path to task correctness when repair trajectories are held fixed. Yet Table 2 reports 60.4% versus 52.8% correctness, and Section 4.4 reports a seven-task gap already at iteration 1, before multiple correct parents can be available. More broadly, the paper does not compare SFT-only training with RL, verified with unverified reconstructed strategies, or historical weighting with local diversity alone. Because planner-coder separation (Zhu et al., 2026) and determinant-based diversity (Chen et al., 2026b) have precedents, these controls matter for establishing the distinctive contribution.
-
The acceleration claims need stronger measurement and uncertainty controls. Appendix selects the archived maximum measured speedup while explicitly allowing reference timings to vary across evaluations; the selected kernel can therefore differ from the minimum-latency kernel. It does not state an independent final remeasurement against a common reference. The diversity reward adds only four fast1 tasks and two fast2 tasks (Table 2), while the Qwen3.8-Flash fast1 gain is one task and its correctness falls by 16 tasks (Table 1). These facts do not negate the substantial Qwen3-Coder-Plus correctness gain, but they limit confidence in the claimed optimization benefits and their generality.
Questions for authors
-
What differs between the full and no-multi-kernel arms before the first correct implementation is found? Please explain the iteration-1 correctness gap and show a comparison that shares identical repair prompts, generations, and outcomes until first success, then varies only the optimization context. If independent sampling explains the gap, please separate that variation from the claimed context benefit.
-
How much do execution verification and the two diversity levels add beyond the SFT checkpoint and execution-reward RL? Targeted, comparably budgeted controls for verified versus unverified strategies and global versus local diversity would materially affect my assessment. Please also clarify the contribution of the auxiliary analyses.
6 comments
Response to the Concerns of Reviewer [Part 2]
Q2: Contributions of execution verification, reinforcement learning, and historical frequency weighting.
A2: We thank the reviewer for requesting a clearer account of the training components. Strategy verification tests whether a reconstructed strategy can guide a successful implementation, while RL trains the planner using feedback from the implementations produced by the frozen coder. Historical frequency weighting adds information about technique repetition across training states to the local diversity objective.
(1) Execution-verified supervision and reinforcement learning. A successful teacher edit does not by itself establish that a reconstructed strategy provides actionable guidance. We therefore ask the frozen coder to apply each strategy to the original kernel without seeing the teacher's improved implementation. Repair strategies must pass compilation and numerical checks, while optimization strategies must additionally reduce latency relative to the parent kernel.
Role of the auxiliary analyses. The 4,700 CASS kernel analyses provide supervision on algorithms, memory access, thread mapping, synchronization, and performance bottlenecks. They are mixed with verified repair and optimization strategies during SFT, combining understanding of the current implementation with learning actionable changes from execution feedback.
All configurations below use Qwen3-Coder-Plus on the 250 KernelBench tasks.
| Planner configuration | Correct tasks | fast₁ | fast₂ |
|---|---|---|---|
| SFT on unverified strategies | 106 | 35 | 20 |
| SFT on verified strategies | 114 | 37 | 20 |
| RL without the diversity reward | 133 | 44 | 21 |
| Full CUDACoach | 151 | 48 | 23 |
In the table above, verified-strategy SFT covers eight additional correct tasks and two additional fast₁ tasks, with fast₂ unchanged. Because verification excludes strategies whose downstream implementations fail execution checks, the retained strategy supervision consists of guidance that the coder has successfully implemented. We will add these SFT comparison results to the revised manuscript.
RL without the diversity reward improves over SFT on all three metrics. Compared with this RL baseline, full CUDACoach yields 18 additional correct tasks, four additional fast₁ tasks, and two additional fast₂ tasks.
(2) Historical frequency weighting beyond local diversity. The reward ablation above removes both diversity components, so we additionally examine saved local log-det and historical-weighted configurations.
Local log-det evaluates complementarity only among strategies for the same state. Local diversity can remain high even when a planner repeatedly recombines the same small set of techniques across tasks. Historical frequency weighting reduces common techniques' weights in the Gram matrix, adding information across states while retaining complementarity within each state.
We compare the recorded training configurations at updates 10, 20, and 30 on the same 199 optimization states. At each checkpoint, four strategies per state give 796 strategies per configuration. Each strategy receives four coder implementations, and its outcome is determined by the best execution reward. These are strategy-level training-validation rates. Correctness requires a correct implementation. Acceleration success additionally requires speedup greater than 1.1 times the execution-reward baseline.
| Training update | Local correctness | Weighted correctness | Local acceleration success | Weighted acceleration success |
|---|---|---|---|---|
| 10 | 74.87% | 77.14% | 56.78% | 57.66% |
| 20 | 80.53% | 81.41% | 61.31% | 62.69% |
| 30 | 80.78% | 83.42% | 62.69% | 64.32% |
The historical-weighted configuration has higher correctness and acceleration success at all three reported checkpoints. At update 30, its correctness is 2.64 percentage points higher. The acceleration-success gap grows from 0.88 to 1.38 to 1.63 percentage points over updates 10, 20, and 30.
Response to the Concerns of Reviewer [Part 3]
Q3: Consistency of the diversity-reward gains and reliability of speedup measurement.
A3: We examined whether the diversity-reward gains persist when the no-diversity baseline is repeated. The full result reported in the paper exceeds the original baseline and both additional baseline runs in correctness, fast₁, and fast₂. We first clarify the measurement protocol, then report these comparisons.
Consistent reference timing. The appendix inadvertently retained wording from an earlier implementation. CUDACoach uses a fixed PyTorch reference latency for each task. Among candidates that pass verification, maximizing speedup is therefore equivalent to minimizing kernel latency. Recomputing the speedups using a common fixed reference per task leaves the fast₁ and fast₂ counts for CUDACoach and all three single-stage baselines unchanged. We will correct the appendix to state this protocol clearly.
Repeated no-diversity baseline. We compare the original baseline result with two additional inference runs on the same 250-task benchmark. The additional runs use the same trained checkpoint, shared initial kernels, coder, and evaluation settings. Each run has limits of 10 iterations and 40 candidate slots per task. Means and sample standard deviations summarize the three baseline outcomes.
| Metric | Original baseline | Additional run 1 | Additional run 2 | Baseline mean ± SD | Full CUDACoach, paper result |
|---|---|---|---|---|---|
| Correct tasks | 133 | 137 | 137 | 135.67 ± 2.31 | 151 |
| fast₁ | 44 | 41 | 42 | 42.33 ± 1.53 | 48 |
| fast₂ | 21 | 17 | 18 | 18.67 ± 2.08 | 23 |
Comparing the reported full result with these three baseline outcomes gives gains of 14–18 correct tasks, 4–7 fast₁ tasks, and 2–6 fast₂ tasks. The advantage is therefore present with each of the three baseline outcomes. We will add the additional results and summary statistics to the revised manuscript.
Response to the Concerns of Reviewer [Part 1]
A1: We thank the reviewer for identifying the need to separate repair from optimization. Multi-kernel context is used after a correct implementation becomes available. Its intended contribution is to improve the search for faster implementations. When repair generations and outcomes are shared, this mechanism does not change whether the first correct implementation is found. Historical-best retention then preserves that correct implementation during optimization.
(1) Why the original correctness curves differ. The original comparison did not enforce identical sampled repair trajectories. Although both arms started from the same initial kernels and used the same repair procedure, their generated repair strategies and implementations could differ. The aggregate correctness difference of 60.4% versus 52.8% therefore should not be interpreted as the isolated effect of optimization context.
We traced the seven-task gap at iteration 1 to a net difference of six successful repairs and one task counted inconsistently in the plotted results. The first repair step succeeded on 24 initially incorrect tasks in one arm and 18 in the other. We will align the treatment of the remaining task in the revised curve. The early gap reflects differing repair outcomes and counting, rather than a benefit from comparing multiple correct kernels.
(2) Evaluation from a shared repair trajectory. To address the reviewer's requested comparison, we preserve the same repair prompts, generated strategies, candidate implementations, and evaluation outcomes until the first successful repair. Both optimization branches then start from the same correct kernel, feedback, and profiling information. We reuse the recorded single-context continuation and run multi-context continuations from this shared state. Single-context optimization uses the best correct parent, while multi-context optimization can use multiple correct parents as they become available.
This evaluation includes all 40 initially incorrect tasks that were successfully repaired in the recorded single-context trajectory and had optimization iterations remaining.
We obtain the following results. The geometric mean measures speedup over each task's shared first correct implementation. The fast₁ and fast₂ counts use the same fixed PyTorch reference timings for both configurations.
The multi-context configuration yields three additional tasks faster than PyTorch and a higher geometric-mean improvement over the shared starting kernels.
(3) Optimization of initially correct tasks. We also report a subset analysis of the paper's results on all 91 tasks with the same correct initial implementation in both configurations. These tasks enter optimization without requiring an initial repair.
On these initially correct tasks, CUDACoach yields eight additional tasks faster than PyTorch and a higher geometric-mean improvement over the shared initial kernels. Together, the two analyses cover optimization from an initially correct implementation and from a shared successful repair. We will revise the discussion to distinguish these optimization results from the variation in repair outcomes before the first correct implementation is found.