- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces MetaKernelBench, a benchmark that evaluates whether knowledge gained while optimizing a GPU computation in one domain-specific language can help a fresh attempt optimize the same computation in another. The authors use CuTe DSL and TIRx and evaluate six models, including DeepSeek and Qwen, on 74 problems. More specifically, they first have a model attempt the task in one language and write down what it learned as a natural-language skill. They then give this skill to a fresh attempt by the same model to solve the same task in the other language, without providing the original code or execution trajectory. They find that transfer improves some attempts but harms others.
Critical strengths
There are two strenghts of the paper:
-
The first strength is that this paper asks an interesting research question about whether knowledge gained in one DSL can transfer to another DSL for the same task. To answer this question, the authors conduct a substantial set of experiments involving six models, using a straightforward experimental setup.
-
The second strength is that the authors present several interesting case studies that help explain why knowledge transfer helps or harms an attempt. e.g. the Figure 6 aobut the GLM5.3 transfer case.
Critical weaknesses
There are two main weaknesses of the paper:
-
The first weakness is that the paper draws conclusions that provide limited practical guidance. Although the experiments show that knowledge transfer can help or hurt, they do not establish reliable conditions for successful transfer. The case studies suggest possible explanations, but the authors do not directly test them, for example, by removing misleading claims from a skill and checking whether performance improves. This limits the actionable insights that future work can draw from the paper.
-
The second weakness is that the repeatability of the results is insufficiently established. The authors use a single agent attempt per condition, making it difficult to determine whether the observed gains and regressions are consistent across attempts. They also use 10 warm-up calls and 50 timed calls without reporting timing variability or showing whether increasing these counts produces similar results.
Questions for authors
My questions are mainly based on the weakness points listed above:
-
Can the authors identify conditions that reliably indicate whether transfer will help or hurt, such as task type or model type?
-
Can the authors repeat the agent attempts and report the variability across runs, as well as the variability across the 50 timed calls when evaluating ops? Reporting low variability in the timed calls and showing that more warm-up and timed calls yield similar results would strengthen confidence in the runtime measurements.
4 comments
Response to Reviewer JwmG (Part 2/2): Repeatability & Timing (W2, Q2)
2. Repeatability of Agent Runs and Timing Variability W2 + Q2
W2: ... The authors use a single agent attempt per condition, making it difficult to determine whether the observed gains and regressions are consistent across attempts. They also use 10 warm-up calls and 50 timed calls without reporting timing variability or showing whether increasing these counts produces similar results.
Q2: Can the authors repeat the agent attempts and report the variability across runs, as well as the variability across the 50 timed calls when evaluating ops? ...
Each evaluation run covers all six models, all 74 problems, and both transfer directions, regenerating source skills and running fresh skill-conditioned attempts alongside same-run no-skill controls under the same 40-minute target budget. Both new runs reproduce our core findings that help and harm coexist and that help rates track the source attempt's relative standing. In all three runs, 12 of 12 model-direction combinations exhibit both help and harm, with overall help rates of 31.2%, 30.1%, and 25.9%, harm rates of 25.1%, 20.3%, and 21.2%, and same-run above-versus-below source-standing help rates of 71.2% / 10.6%, 69.1% / 16.2%, and 65.2% / 14.2% across Runs 1, 2, and 3. Section 3.1, Sections 4.1–4.2, and Tables 1–2 report three-run arithmetic means, Appendix B.1 Tables 3–4 report per-run lifts alongside across-run sample standard deviations for every model and direction, and Appendix B.2 Table 5 reports 95% percentile intervals from 10,000 problem-cluster bootstrap draws together with across-run problem-level outcome counts.
Across all 2,705 non-skipped final evaluations satisfying the implementation and scoring rules in Runs 2 and 3, the 50 timed calls show tight dispersion around each evaluation's median, with a median kernel interquartile range of 1.34% relative to the median and median sample coefficients of variation of 2.43% for candidate kernels and 2.02% for the eager reference. Every evaluation performs 10 warmup calls followed by 50 CUDA-event timed calls after device synchronization, and the 50 per-call samples reproduce the reported median runtimes across all 2,705 evaluations. Under the verification protocol in Section 3.3 and Appendix A.3, the 50 timed calls evaluate 50 distinct input seeds drawn from the same generator and matched one-to-one between the candidate implementation and the eager reference, so wider spread on input-dependent workloads reflects shared workload variation across the 50 paired inputs rather than per-call timing noise. To keep all three evaluation runs directly comparable under a unified grading contract, we held the 10-warmup and 50-timed-call schedule fixed rather than running separate sweeps over higher call counts, while the 50 per-call samples above directly quantify fixed-implementation timing stability under this protocol. This timing-dispersion analysis appears in Appendix A.3.
Thank you again for your constructive feedback, which helped us strengthen both the empirical robustness and the practical takeaways of our paper.
Thanks to the author for the extra effort in addressing the questions. The extra evaluation is solid. The practical value of the findings remains somewhat unclear, but the paper meets the bar for a benchmark paper. So I am increasing my score and leaning toward a weak accept
Thank you very much for checking our additional results and raising your score. We really appreciate your constructive comments, and we will continue making the practical takeaways clearer in the paper.
Response to Reviewer JwmG (Part 1/2): Transfer Conditions (W1, Q1)
Thank you for your thoughtful review and for recognizing our research question on cross-DSL experience reuse, our six-model evaluation, and our case studies illustrating when transfer helps or harms.
We have completed two additional full evaluation runs across all six models, and both new runs reproduce the finding that benefit and regression coexist in experience reuse, with specific effect sizes now reported as three-run means alongside their uncertainty. We have also distilled empirical conditions and practical guidelines for skill reuse and added a 50-call timing-dispersion analysis across 2,705 final evaluations in the paper and this response. Revisions in the paper are marked in blue, and key conclusions below are highlighted in bold. We present our response in two parts, covering W1 and Q1 in Part 1 and W2 and Q2 in Part 2.
1. Conditions for Successful Transfer and Practical Guidance W1 + Q1
Across all 2,664 paired comparisons in three runs, the source attempt's performance tier relative to the target baseline, model scale, and the transfer direction and operator family consistently indicate when skill transfer is beneficial. For relative standing, when the source attempt lies in a higher outcome tier than the same-run target solo baseline, 68.6% of comparisons improve and 9.2% regress, whereas when the source lies below the target baseline, 13.5% improve and 44.9% regress. Even when target standing is estimated from the mean of the other two independent solo runs so that grouping and scoring share no baseline noise, help rates still rise monotonically across source-below, source-same, and source-above groups from 21.3% to 27.0% and 39.9%, giving an above-versus-below gain of 18.5 percentage points with a 95% problem-cluster bootstrap interval of [12.6, 24.5].
Across model scale, transfer directions, and operator families, transferring from source attempts at 2× speedup or better yields mean paired lifts of 16.3% to 30.1% for the three larger models, GLM-5.3, DeepSeek-V4-Pro, and GPT-5.6 Sol, compared with 4.3% to 13.1% for the three Flash-tier models. Across directions, TIRx to CuTe DSL achieves positive three-run mean lift for all six models from +2.5% to +26.6%, and five of six models remain positive after removing each model's best operator family, whereas three of the five positive lifts reverse from CuTe DSL to TIRx. Within a given model, outcomes also vary sharply by family and direction, as GLM-5.3 gains +87.0% from TIRx to CuTe DSL on KV Cache versus −20.8% in the reverse direction while achieving positive mean lift in both directions on Linear Attention.
For practical system design, the skill-content and trajectory analyses in Sections 4.3–4.4 translate directly into actionable guidelines for filtering and using distilled skills. In five of six models, source skills containing unsupported closure statements that rule out alternatives without sufficient evidence appear 7 to 17 percentage points more often in harmed than in helped comparisons, and 65% to 86% of assigned target-side failures for the three larger models fall into search truncation or unverified adoption. When building a kernel-agent experience library in practice, one should therefore prioritize skills distilled from high-speedup source attempts, filter out unverified negative exclusions before storing a skill, and prompt target agents to treat exclusionary claims in attached skills as hypotheses to verify rather than hard constraints. Our main benchmark retains raw source skills to evaluate end-to-end distillation and reuse, while Appendix E.2 outlines removing or revising individual guidance statements as a follow-up direction. These results and discussions appear in Sections 4.1–4.4, Tables 1–2, Figures 4–7, Appendix B.5 Figure 9, Appendix B.6 Table 7, and Appendix E.2.