- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies LLM-driven GPU kernel optimization when only a few target-device evaluations are allowed per task. Offline, a strong teacher model distills public code, patches and failure cases into records that link an optimization action to its applicability conditions, coupled code changes and source checks (405 cards with 587 options for KernelBench Level 2); a second model checked selected records against their sources, the authors revised the records, and developer-maintained constructors and checks are versioned with them. Online, a fixed model uses the records in two layers. Search-space pruning removes design routes whose conditions are false for the task and hardware, and the model generates candidates within the remaining routes. Candidate-set pruning inspects, repairs, blocks and ranks the generated programs before measurement, and a persistent pool keeps unmeasured programs for later evaluations. On all 100 KernelBench Level 2 tasks (RTX 4090, non-thinking DeepSeek-V4.1-Flash, an initial pool of up to eight candidates, at most three evaluations per task), the method reaches Fast1 coverage of 46%, 72% and 77% after one, two and three evaluations, and strict Fast1.05 coverage of 45% after three. Five baselines (Best-of-N, Iterative, OpenEvolve, and local ports of STARK and EGG) need five to nine evaluations to reach 77%, which the paper reports as 40.0–66.7% fewer evaluations; at that coverage the method has used 94.8M online tokens versus 1.6–7.2M, and 12.6 versus 28.5–58.0 GPU-worker hours. The paper also reports an H100 run, a 24-task run with GLM-5.3-Flash, knowledge ablations on 24 tasks, and a Level 1 pool-size study.
Critical strengths
-
The problem is well posed and the protocol is careful. Device evaluations are treated as the scarce resource: every attempted evaluation, including failures, consumes budget; all tasks stay in the denominator; and tokens and GPU time are reported separately. Under this protocol the method reaches Fast1 coverage earlier than every baseline on RTX 4090 and H100 (77% versus 63% after three evaluations on RTX 4090), and on RTX 4090 it also leads on the strict 1.05× criterion (45 versus 39 tasks). Where target hardware is scarce or remote, this is a useful result.
-
The experimental record is extensive and self-critical. The appendices audit baseline results above 10× that pass the numerical checks but implement the wrong formula (task 42), re-measure selected sources in fresh processes alongside reference-equivalent controls, classify every winning program by the path it actually executes (Table 7), and include several knowledge studies. This makes the claims checkable; most of the concerns below come from these appendices.
Critical weaknesses
1. The novelty is limited, and the gains are not shown to come from the distilled knowledge. The first layer is essentially a rule filter over applicability conditions, and the second is code review, repair and ranking before measurement; both are familiar parts of agentic kernel optimization. The contribution therefore rests on the distilled knowledge, but the experiments do not isolate its effect:
- The removal ablations also remove constructors, specialized checks and automatic patches (Table 10), and are compared with an earlier full-system run on a different model service, with one run per arm.
- When tools and construction support are held fixed in the same batch (Table 34), adding the knowledge changes the Fast1 and strict counts after three evaluations by at most one of 24 tasks.
- All 77 RTX 4090 winners keep a library convolution or matrix-multiplication body plus a custom tail kernel (Table 7), the split
torch.compilemakes by default, and a one-sentence hint to do this matches the full system's Fast1, though not its strict count. The task 82 case shows that the relevant records were in the prompt, not that they were needed.
2. The evaluation does not show an advantage over strong alternatives or a favorable total cost.
- Metric. The headline numbers use Fast1, which reference-equivalent code can pass (the paper's controls measure up to 1.014×). Under the strict 1.05× criterion, the method leads on RTX 4090 but not on H100 or with GLM-5.3-Flash.
- Baselines. The baselines are local ports, read at the first three evaluations of ten-evaluation runs, and at matched coverage they use 13–61× fewer tokens than the method. There is no token-matched baseline and no general-purpose coding agent (e.g., Claude Code or Codex) with the same evaluator and budget.
- Cost. The method saves GPU evaluations by using about 40× more tokens than Iterative, and neither the end-to-end time per task nor the offline cost of building the knowledge is reported, so whether the trade pays off is unclear.
Questions for authors
- Attribution (experiment). In one batch and on one model service, with more than one run per arm, how does FULL compare with GENERIC PREP and with a removal arm that keeps the constructors, checks and patches and drops only the records?
- Baselines on equal terms (experiment). How do a token-matched baseline without teacher records (e.g., the same model generating eight candidates and reviewing and ranking them before each evaluation) and a general-purpose coding agent perform, both with the same evaluator and three evaluations?
- Cost (clarification). What did building the knowledge cost (teacher and reviewer tokens, author and developer hours), and what is the end-to-end time per task, including model latency, for the method and the baselines?
4 comments
Q3. How much does a library body with a custom tail explain?
We appreciate your question about how much the library body explains. All 77 faster RTX 4090 implementations used a library body and a custom tail. To assess the contribution of that design, we evaluated torch.compile with default Inductor on the full reference for each of the 24 tasks. The compiler used one fixed program per task and zero model calls. It achieved stable correctness on 24 tasks, stable Fast₁ on 15, and strict acceleration on 12. FULL’s first B3 search per task yielded 23, 22, and 17, respectively. A one-sentence hint matched the historical FULL run’s 19/24 Fast₁ tasks, with 6 vs. 13 strict successes. All comparisons timed the full forward pass.
On task 82, the selected implementation moved tanh after the window maximum. Because tanh is monotone, the rewrite cut 16 tanh evaluations per full window to one. The full forward pass achieved a 2.126× speedup in the original measurement and 2.129–2.130× in fresh processes. The source shows which operation changed. The matched comparisons above quantify the contribution of record delivery across tasks.
Table 29. FULL and the fixed compiler on the 24 tasks
| Configuration | GPU slots | Stable correct | Stable Fast₁ | Strict | Tokens | HTTP |
|---|---|---|---|---|---|---|
| FULL | 72 | 23 | 22 | 17 | 21,286,089 | 518 |
| torch.compile | 24 | 24 | 15 | 12 | 0 | 0 |
Each search uses B3 and one run per task. The compiler evaluates the original full task once per task with default Inductor and zero model calls.
Table 30. Single route hint compared with historical FULL
| Study | Configuration | Tasks | Correct | Fast₁ | Strict | Tokens (M) |
|---|---|---|---|---|---|---|
| Capabilities | Prior FULL-C | 24 | 23 | 19 | 13 | 21.909 |
| Capabilities | Coarse route hint | 24 | 24 | 19 | 6 | 5.978 |
Table 31. Source transformation in Level 2 task 82
| Configuration | Measured speedup | Executed work |
|---|---|---|
| Raw-source search | 1.003762× | Separate tanh, scaling, bias and pooling |
| FULL | 2.125992× | Window max before monotone tanh, 16 calls → 1 |
| FULL fresh-process repeats | 2.1298 / 2.1291 / 2.1299× | Three remeasurements |
Q4. How do the threshold, hardware, and online model affect coverage?
Thank you for asking us to separate threshold effects from hardware and model changes. On the 100 RTX 4090 tasks at B3, FULL led the best baseline 77 vs. 63 on Fast₁ and 45 vs. 39 under the strict criterion. Unfiltered counts were 77 vs. 65 and 45 vs. 40. The 12 reference-equivalent controls measured between 0.9931× and 1.0136×, below the strict threshold. On H100, FULL reached 70/100 Fast₁ at B2 vs. 52/100 for the best baseline and 75/100 vs. 69/100 at B3. Baselines reaching 75 needed B4–B7, so FULL used 25.0%–57.1% fewer slots per task. With GLM, FULL led Fast₁ 23/24 vs. 20/24 at B3. EGG led strict coverage on H100 (36 vs. 34), and Iterative led with GLM (15 vs. 14).
Table 32. B3 coverage with and without the 10× screen
| Method | Fast₁ <10× / all | Strict <10× / all |
|---|---|---|
| FULL | 77 / 77 | 45 / 45 |
| Best-of-N | 52 / 53 | 13 / 14 |
| Iterative | 63 / 65 | 29 / 31 |
| OpenEvolve | 60 / 60 | 28 / 28 |
| STARK-port | 57 / 60 | 34 / 38 |
| EGG-port | 56 / 57 | 39 / 40 |
Table 33. All 100 Level 2 tasks on H100
| Method | Fast₁ B1 | Fast₁ B2 | Fast₁ B3 | Correct B3 | Stable B3 | Strict B3 | Match 75 Fast₁ |
|---|---|---|---|---|---|---|---|
| FULL | 47 | 70 | 75 | 89 | 88 | 34 | B3 |
| Best-of-N | 30 | 51 | 61 | 75 | 72 | 23 | B7 |
| Iterative | 28 | 52 | 69 | 95 | 93 | 23 | B4 |
| STARK-port | 20 | 49 | 63 | 85 | 84 | 33 | B6 |
| EGG-port | 38 | 52 | 58 | 70 | 68 | 36 | — |
Each method uses DeepSeek-V4.1-Flash. Fast₁ target crossings use observed prefixes. The B3 reduction is 25.0%–57.1% for baselines reaching 75 tasks.
Table 34. GLM-5.3-Flash on the same 24 Level 2 tasks
| Method | B3 C / F / S | B3 closed | Max B | Max B closed | Max B F / S | Tokens (M) | Match 23 Fast₁ |
|---|---|---|---|---|---|---|---|
| FULL | 23 / 23 / 14 | 23 | B3 | 23 | 23 / 14 | 18.569 | B3 |
| Best-of-N | 16 / 14 / 11 | 23 | B10 | 23 | 24 / 19 | 3.205 | B8 |
| Iterative | 19 / 19 / 15 | 23 | B10 | 21 | 23 / 18 | 3.799 | B8 |
| OpenEvolve | 20 / 18 / 12 | 24 | B10 | 24 | 23 / 18 | 4.268 | B9 |
| STARK-port | 21 / 20 / 14 | 23 | B10 | 21 | 23 / 18 | 4.958 | B7 |
| EGG-port | 15 / 13 / 9 | 24 | B10 | 23 | 20 / 17 | 4.313 | — |
C / F / S denotes Correct / Fast₁ / Strict. Closed counts are completed prefixes out of 24. Max-B tokens count reported online usage.
Table 35. GLM resources at 23 Fast₁ tasks
| Method | B | GPU receipts | Worker h | Tokens (M) | HTTP |
|---|---|---|---|---|---|
| FULL | 3 | 69 | 2.74 | 18.569 | 437 |
| Best-of-N | 8 | 177 | 5.35 | 2.572 | 223 |
| Iterative | 8 | 175 | 7.91 | 3.049 | 230 |
| OpenEvolve | 9 | 205 | 9.26 | 3.736 | 256 |
| STARK-port | 7 | 151 | 6.16 | 3.063 | 313 |
Q5. How do stronger baselines perform under the same GPU budget?
We appreciate your concern that the stopping budget can affect search behavior. The submitted B3 baselines use the first three evaluations from B10 runs of our shared-evaluator ports. Following your suggestion, we ran OpenEvolve directly at B3. The standard version achieved 5/24 strict successes. A variant explicitly told the budget achieved 2/24, compared with 17/24 for FULL’s first repeat. In the preparation-budget control, OpenEvolve used 71.84M reported tokens vs. 63.97M for FULL across 72 paired B3 searches. OpenEvolve achieved 45 vs. 60 stable Fast₁ successes and 22 vs. 46 strict successes. Both used DeepSeek-V4.1-Flash.
At B3, GPT-6 Astra in OpenEvolve achieved strict speedups on 17/24 tasks and stable correctness on 24/24. DeepSeek in FULL reached 17/24 and 23/24, respectively. Applying public prices to reported token use gives estimated API costs of USD 7.69–USD 8.35 for Astra vs. USD 3.13 for FULL. FULL’s estimate corresponds to 1.31–1.35 Astra evaluations per task.
Running Astra inside dual pruning would let us compare strict coverage and API cost with Astra–OpenEvolve while holding the online model and GPU budget fixed. To test a stronger teacher independently, we could ask it to distill the same public sources and evaluate its records with DeepSeek online. The general-purpose coding agent you suggest could use the same evaluator with three GPU evaluations per task.
Table 36. Direct search and compiler controls on 24 tasks
| Configuration | GPU slots | Stable correct | Stable Fast₁ | Strict | Tokens | HTTP |
|---|---|---|---|---|---|---|
| FULL | 72 | 23 | 22 | 17 | 21,286,089 | 518 |
| DeepSeek light | 72 | 23 | 17 | 8 | 10,136,690 | 502 |
| DeepSeek deep | 72 | 22 | 16 | 8 | 9,985,002 | 493 |
| Standard OpenEvolve | 72 | 18 | 13 | 5 | 320,963 | 72 |
| OpenEvolve informed of B3 | 72 | 14 | 10 | 2 | 331,861 | 72 |
| torch.compile | 24 | 24 | 15 | 12 | 0 | 0 |
Each search uses B3 and one run per task. The compiler evaluates the original full task once per task with default Inductor and zero model calls.
Table 37. Matched preparation budgets across paired searches
| Repeat | Pairs | FULL | OpenEvolve | FULL tokens | OE tokens |
|---|---|---|---|---|---|
| Pooled | 72 | 66 / 60 / 46 | 59 / 45 / 22 | 63,966,656 | 71,841,885 |
| Complete pairs | 69 | 63 / 57 / 44 | 57 / 43 / 21 | 61,719,106 | 69,304,809 |
Same DeepSeek-V4.1-Flash model, tasks and B3 budget. Triplets are stable correct / stable Fast₁ / strict. HTTP attempts total 1,530 and 3,468, with 87 and 4 attempts lacking usage, respectively.
Table 38. Astra and DeepSeek at the same B3 evaluation budget
| Configuration | Correct / stable | All F / S | <10× F / S | Tokens | API cost |
|---|---|---|---|---|---|
| Astra + OpenEvolve B3 | 24 / 24 | 23 / 17 | 22 / 16 | 597,185 | USD 7.69–USD 8.35 |
| DeepSeek + FULL B3 | 23 / 23 | 22 / 17 | 22 / 17 | 21,286,089 | USD 3.13 |
Same 24 tasks, one search per task. F / S denotes stable Fast₁ / strict. The equal-spend comparison uses FULL’s B3 API cost. One Astra and 34 FULL requests lack usage.
Table 39. Initial candidate pool on 24 Level 1 tasks at B3
| Pool | Correct | Stable | Fast₁ | Strict | AM / GM | Tokens (M) | HTTP |
|---|---|---|---|---|---|---|---|
| 4 | 20 | 20 | 9 | 7 | 1.270 / 1.038 | 19.943 | 435 |
| 8 | 24 | 24 | 15 | 7 | 1.273 / 1.101 | 26.581 | 608 |
| 16 | 24 | 23 | 19 | 7 | 1.328 / 1.189 | 37.251 | 788 |
Nine operator families. AM and GM are arithmetic and geometric means over correct tasks. Submitted Table 4.
Q6. What did online search and offline knowledge construction cost?
Thank you for asking for a fuller accounting of online search and knowledge construction. At matched Fast₁ coverage on the 100-task benchmark, FULL used 12.64 GPU-worker hours and 94.763M reported online tokens. The baselines used 28.51–58.02 hours and 1.565M–7.225M tokens. In the paired B3 study, the median accumulated API and worker times per search were 186 s and 206 s for FULL and 268 s and 148 s for expanded-compute OpenEvolve. End-to-end latency also depends on queueing and concurrency.
Our historical Level 2 records document the sources and revisions. We recognize the value of a fuller cost record. Our records for that build regrettably lack aggregate teacher and reviewer token counts and hands-on hours. To give a measured breakdown, we tracked extraction and audit use in the new 33-source study. Initial extraction used 0.699M reported tokens and auditing used 0.659M more. Extending the Level 2 workflow to Level 3 took about eight calendar days of Codex-assisted development, including integration and testing.
Table 40. Recorded resources to reach 77 Fast₁ tasks
| Method | B | GPU receipts | Worker h | Tokens (M) | HTTP attempts |
|---|---|---|---|---|---|
| FULL | 3 | 300 | 12.64 | 94.763 | 2,031 |
| Best-of-N | 9 | 887 | 32.51 | 1.565 | 900 |
| Iterative | 5 | 498 | 29.83 | 2.387 | 500 |
| OpenEvolve | 9 | 893 | 58.02 | 6.630 | 900 |
| STARK-port | 8 | 793 | 33.32 | 7.225 | 1,398 |
| EGG-port | 7 | 694 | 28.51 | 5.096 | 1,087 |
Worker time is accumulated over evaluation receipts. Tokens are the reported online usage in the submission.
Table 41. Median accumulated time per paired B3 search
| Phase | FULL | OpenEvolve |
|---|---|---|
| Median API time / search | 186 s | 268 s |
| Median worker time / search | 206 s | 148 s |
Table 42. Offline build effort on the same public sources
| Stage | Sources | Input tokens | Output tokens | Total tokens | HTTP |
|---|---|---|---|---|---|
| Initial extraction | 33 | 530,319 | 169,071 | 699,390 | 68 |
| Additional audit | 33 | 539,753 | 119,294 | 659,047 | 52 |
| Audited library total | 33 | 1,070,072 | 288,365 | 1,358,437 | 120 |
The selected Level 2 catalog contains 405 cards and 587 options. Records document its sources and revisions. The table measures the new matched extraction study.
Thank you for helping us clarify the contribution and improve the comparisons. We will include the supporting analyses and tables in the revision and would be grateful to know whether they answer your concerns.
We are grateful for the time you devoted to the review. Your questions showed where the shared record needed a more concrete explanation and guided our additional controls.
For ease of reading, we recap our notation. FULL denotes our complete method. B caps GPU evaluations per task and search. Fast₁ counts tasks with a correct implementation faster than the reference. Stable Fast₁ adds timing stability. Strict successes pass stability checks and exceed 1.05× in every seed and reference bracket.
Q1. What does the shared knowledge record contribute?
Thank you for asking us to make the shared record more concrete. It links a design choice to checks on its implementation. For a row of length N reduced in partitions of width b, the number of partial results, ceil(N/b), must agree across the launch grid, scratch allocation and merge bounds. A mean still divides by N. During design, the record helps choose a valid route. During inspection, it identifies the coupled source edits to check. A tool-confirmed violation can block evaluation. A concern raised by the model can prompt inspection or repair. Appendices L and N document these dependencies and checks.
Table 25. Source checks for a partitioned row reduction
Q2. What changes when only the delivery of teacher records varies?
We appreciate your request to isolate record delivery. The four configurations shared the same official DeepSeek service, tools, constructors, checks, patches, and evaluator. We used the 24 representative KernelBench Level 2 tasks selected for the submitted paper. Across three independent B3 searches per task, FULL achieved 46/72 strict successes, compared with 34/72 for generation-only delivery, 30/72 for assessment-only delivery, and 31/72 for generic preparation. FULL led generic preparation by 6, 6, and 3 strict successes across the repeats. Assessment-only delivery led on stable correctness, with 69 successes to FULL’s 66.
When the system retrieved plain summaries of 33 public source files, it achieved 26 strict successes, compared with 45 for FULL, across 69 completed matched pairs. In the separate same-source excerpt study (Table 40), the historical full-delivery run reached 19 Fast₁ tasks using 22.76M reported tokens. With excerpts, the system reached 13 tasks using 27.37M tokens.
Table 26. Four configurations with shared tools and support
Triplets are stable correct / stable Fast₁ / strict. Each repeat has 24 tasks. The last column applies the submission’s speed screen to strict results. The task-cluster bootstrap interval for FULL minus generic strict coverage is 8.3–34.7 percentage points (95%).
Table 27. Submitted knowledge and source comparisons at B3
Rows are comparable within the named study. FULL-C and FULL-E are the corresponding historical references from the submission.
Table 28. Plain summaries in place of the distilled records
Triplets are stable correct / stable Fast₁ / strict. The summaries cover 33 public sources. The one-time summarization cost is 320,308 reported tokens.