acceptodds
Under review as a conference paper at ICLR 2027

MetaKernelBench: Measuring GPU Kernel Knowledge Transfer Beyond Code

Abstract

Recent GPU kernel optimization agents retain what they learn in knowledge bases or as distilled skills. Kernel benchmarks score each attempt's implementation for correctness and speed but leave the reuse value of retained experience unmeasured. We introduce MetaKernelBench, which measures whether experience distilled from an attempt in one kernel domain-specific language (DSL) improves a fresh attempt at the same problem in another. Its 74 problems are fused subgraphs in six families, each posed as a pair of CuTe DSL and TIRx variants that differ only in the DSL. The agent first attempts each variant solo and is instructed to distill what it learns into a natural-language skill, which is transferred whether or not the source attempt passes verification. The skill is the only extra input to a skill-conditioned attempt by the same model in the other DSL. We compare each skill-conditioned attempt with the solo attempt on the same variant under matched per-attempt budgets, scoring correctness and end-to-end runtime. Across six models and both directions, paired lift over solo attempts ranges from % to %. Four models gain in both directions, yet regressions occur on 16% to 45% of problems in every model and direction. Outcomes follow the source attempt's result relative to the target's solo attempt rather than source success alone, improving in 71% of comparisons when the source stands above and regressing in 54% when it stands below. MetaKernelBench complements implementation-quality metrics by measuring same-problem cross-DSL kernel knowledge transfer.

Primary area:
datasets and benchmarks
Keywords:
GPU Kernel Optimization, Knowledge Transfer

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JwmGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MetaKernelBench, a benchmark that evaluates whether knowledge gained while optimizing a GPU computation in one domain-specific language can help a fresh attempt optimize the same computation in another. The authors use CuTe DSL and TIRx and evaluate six models, including DeepSeek and Qwen, on 74 problems. More specifically, they first have a model attempt the task in one language and write down what it learned as a natural-language skill. They then give this skill to a fresh attempt by the same model to solve the same task in the other language, without providing the original code or execution trajectory. They find that transfer improves some attempts but harms others.

Critical strengths

There are two strenghts of the paper:

  • The first strength is that this paper asks an interesting research question about whether knowledge gained in one DSL can transfer to another DSL for the same task. To answer this question, the authors conduct a substantial set of experiments involving six models, using a straightforward experimental setup.

  • The second strength is that the authors present several interesting case studies that help explain why knowledge transfer helps or harms an attempt. e.g. the Figure 6 aobut the GLM5.3 transfer case.

Critical weaknesses

There are two main weaknesses of the paper:

  • The first weakness is that the paper draws conclusions that provide limited practical guidance. Although the experiments show that knowledge transfer can help or hurt, they do not establish reliable conditions for successful transfer. The case studies suggest possible explanations, but the authors do not directly test them, for example, by removing misleading claims from a skill and checking whether performance improves. This limits the actionable insights that future work can draw from the paper.

  • The second weakness is that the repeatability of the results is insufficiently established. The authors use a single agent attempt per condition, making it difficult to determine whether the observed gains and regressions are consistent across attempts. They also use 10 warm-up calls and 50 timed calls without reporting timing variability or showing whether increasing these counts produces similar results.

Questions for authors

My questions are mainly based on the weakness points listed above:

  1. Can the authors identify conditions that reliably indicate whether transfer will help or hurt, such as task type or model type?

  2. Can the authors repeat the agent attempts and report the variability across runs, as well as the variability across the 50 timed calls when evaluating ops? Reporting low variability in the timed calls and showing that more warm-up and timed calls yield similar results would strengthen confidence in the runtime measurements.

4 comments
AuthorsOpenReview ↗

Response to Reviewer JwmG (Part 1/2): Transfer Conditions (W1, Q1)

Thank you for your thoughtful review and for recognizing our research question on cross-DSL experience reuse, our six-model evaluation, and our case studies illustrating when transfer helps or harms.

We have completed two additional full evaluation runs across all six models, and both new runs reproduce the finding that benefit and regression coexist in experience reuse, with specific effect sizes now reported as three-run means alongside their uncertainty. We have also distilled empirical conditions and practical guidelines for skill reuse and added a 50-call timing-dispersion analysis across 2,705 final evaluations in the paper and this response. Revisions in the paper are marked in blue, and key conclusions below are highlighted in bold. We present our response in two parts, covering W1 and Q1 in Part 1 and W2 and Q2 in Part 2.

1. Conditions for Successful Transfer and Practical Guidance W1 + Q1

W1: ... Although the experiments show that knowledge transfer can help or hurt, they do not establish reliable conditions for successful transfer. The case studies suggest possible explanations, but the authors do not directly test them, for example, by removing misleading claims from a skill and checking whether performance improves. ...

Q1: Can the authors identify conditions that reliably indicate whether transfer will help or hurt, such as task type or model type?

Across all 2,664 paired comparisons in three runs, the source attempt's performance tier relative to the target baseline, model scale, and the transfer direction and operator family consistently indicate when skill transfer is beneficial. For relative standing, when the source attempt lies in a higher outcome tier than the same-run target solo baseline, 68.6% of comparisons improve and 9.2% regress, whereas when the source lies below the target baseline, 13.5% improve and 44.9% regress. Even when target standing is estimated from the mean of the other two independent solo runs so that grouping and scoring share no baseline noise, help rates still rise monotonically across source-below, source-same, and source-above groups from 21.3% to 27.0% and 39.9%, giving an above-versus-below gain of 18.5 percentage points with a 95% problem-cluster bootstrap interval of [12.6, 24.5].

Across model scale, transfer directions, and operator families, transferring from source attempts at 2×2\times speedup or better yields mean paired lifts of 16.3% to 30.1% for the three larger models, GLM-5.3, DeepSeek-V4-Pro, and GPT-5.6 Sol, compared with 4.3% to 13.1% for the three Flash-tier models. Across directions, TIRx to CuTe DSL achieves positive three-run mean lift for all six models from +2.5% to +26.6%, and five of six models remain positive after removing each model's best operator family, whereas three of the five positive lifts reverse from CuTe DSL to TIRx. Within a given model, outcomes also vary sharply by family and direction, as GLM-5.3 gains +87.0% from TIRx to CuTe DSL on KV Cache versus −20.8% in the reverse direction while achieving positive mean lift in both directions on Linear Attention.

For practical system design, the skill-content and trajectory analyses in Sections 4.3–4.4 translate directly into actionable guidelines for filtering and using distilled skills. In five of six models, source skills containing unsupported closure statements that rule out alternatives without sufficient evidence appear 7 to 17 percentage points more often in harmed than in helped comparisons, and 65% to 86% of assigned target-side failures for the three larger models fall into search truncation or unverified adoption. When building a kernel-agent experience library in practice, one should therefore prioritize skills distilled from high-speedup source attempts, filter out unverified negative exclusions before storing a skill, and prompt target agents to treat exclusionary claims in attached skills as hypotheses to verify rather than hard constraints. Our main benchmark retains raw source skills to evaluate end-to-end distillation and reuse, while Appendix E.2 outlines removing or revising individual guidance statements as a follow-up direction. These results and discussions appear in Sections 4.1–4.4, Tables 1–2, Figures 4–7, Appendix B.5 Figure 9, Appendix B.6 Table 7, and Appendix E.2.

AuthorsOpenReview ↗

Response to Reviewer JwmG (Part 2/2): Repeatability & Timing (W2, Q2)

2. Repeatability of Agent Runs and Timing Variability W2 + Q2

W2: ... The authors use a single agent attempt per condition, making it difficult to determine whether the observed gains and regressions are consistent across attempts. They also use 10 warm-up calls and 50 timed calls without reporting timing variability or showing whether increasing these counts produces similar results.

Q2: Can the authors repeat the agent attempts and report the variability across runs, as well as the variability across the 50 timed calls when evaluating ops? ...

Each evaluation run covers all six models, all 74 problems, and both transfer directions, regenerating source skills and running fresh skill-conditioned attempts alongside same-run no-skill controls under the same 40-minute target budget. Both new runs reproduce our core findings that help and harm coexist and that help rates track the source attempt's relative standing. In all three runs, 12 of 12 model-direction combinations exhibit both help and harm, with overall help rates of 31.2%, 30.1%, and 25.9%, harm rates of 25.1%, 20.3%, and 21.2%, and same-run above-versus-below source-standing help rates of 71.2% / 10.6%, 69.1% / 16.2%, and 65.2% / 14.2% across Runs 1, 2, and 3. Section 3.1, Sections 4.1–4.2, and Tables 1–2 report three-run arithmetic means, Appendix B.1 Tables 3–4 report per-run lifts alongside across-run sample standard deviations for every model and direction, and Appendix B.2 Table 5 reports 95% percentile intervals from 10,000 problem-cluster bootstrap draws together with across-run problem-level outcome counts.

Across all 2,705 non-skipped final evaluations satisfying the implementation and scoring rules in Runs 2 and 3, the 50 timed calls show tight dispersion around each evaluation's median, with a median kernel interquartile range of 1.34% relative to the median and median sample coefficients of variation of 2.43% for candidate kernels and 2.02% for the eager reference. Every evaluation performs 10 warmup calls followed by 50 CUDA-event timed calls after device synchronization, and the 50 per-call samples reproduce the reported median runtimes across all 2,705 evaluations. Under the verification protocol in Section 3.3 and Appendix A.3, the 50 timed calls evaluate 50 distinct input seeds drawn from the same generator and matched one-to-one between the candidate implementation and the eager reference, so wider spread on input-dependent workloads reflects shared workload variation across the 50 paired inputs rather than per-call timing noise. To keep all three evaluation runs directly comparable under a unified grading contract, we held the 10-warmup and 50-timed-call schedule fixed rather than running separate sweeps over higher call counts, while the 50 per-call samples above directly quantify fixed-implementation timing stability under this protocol. This timing-dispersion analysis appears in Appendix A.3.


Thank you again for your constructive feedback, which helped us strengthen both the empirical robustness and the practical takeaways of our paper.

Thanks to the author for the extra effort in addressing the questions. The extra evaluation is solid. The practical value of the findings remains somewhat unclear, but the paper meets the bar for a benchmark paper. So I am increasing my score and leaning toward a weak accept

Thank you very much for checking our additional results and raising your score. We really appreciate your constructive comments, and we will continue making the practical takeaways clearer in the paper.

Reviewer_h8gdOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

MetaKernelBench investigates the ability of LLMs to transfer knowledge acquired while optimizing GPU kernels from one Domain-Specific Language (DSL) to another. More specifically, it evaluates knowledge transfer between CuTe DSL and TIRx, in both directions. The authors introduce a benchmark consisting of 74 GPU kernel optimization problems and evaluate six LLMs. For each problem, the model first attempts to optimize a kernel in one DSL and summarizes its findings into a natural-language skill, which is then provided when solving the same problem in the other DSL. The resulting performance is compared against a baseline without transferred knowledge. The results are quite mixed, with performance changes ranging from -19% to 29% depending on the model and transfer direction. Although four of the six models benefit from knowledge transfer in both directions, all models experience regressions on some problems. The authors also show that the effectiveness of transfer strongly depends on the relative quality of the source solution, since poorly validated knowledge can mislead the model into following suboptimal optimization strategies. The paper demonstrates that cross-DSL knowledge transfer can be beneficial but remains unreliable.

Critical strengths

The paper tries to understand how optimization knowledge can be transferred and reused across GPU kernel implementations. Optimization strategies developed for one attention variant could potentially benefit related variants. Similarly, transferring hardware-aware optimization insights across different kernel DSLs could reduce redundant optimization efforts.

The benchmark covers a diverse and representative set of GPU kernels derived from real-world workloads. The inclusion of different kernel families, such as attention, linear attention, MoE, and quantization, makes the evaluation relevant to practical GPU optimization challenges. The experimental evaluation is extensive.

The authors evaluate a broad range of language models, including both larger and lightweight models, and investigate not only successful transfers but also cases where transferred knowledge negatively impacts performance. The detailed analysis of failure cases is particularly interesting, as it helps identify when and why knowledge transfer can be beneficial or harmful.

Critical weaknesses

My main concern is knowledge between DSLs for the same kernel on the same GPU seems less useful than transferring optimizations across GPU architectures or between related kernels (e.g., attention variants). The benchmark therefore seems to miss more practical transfer scenarios.

The evaluation relies on single runs of agents. Without repeated experiments or confidence intervals, it is difficult to assess the significance and robustness of the reported improvements.

It is unclear whether the transferred skills contain genuinely useful knowledge from previous attempts or simply general GPU optimization tricks. A baseline using generic optimization guidelines would help clarify this.

Only two relatively low-level DSLs are evaluated. Since both expose similar hardware-level concepts, it is unclear how much the results demonstrate meaningful cross-DSL transfer.

Given CUDA's importance in GPU programming, choosing CUDA or Cutlass as a source language is a relevant candidate that's missing.

Questions for authors

What motivates the proposed transfer-learning approach over developing a reusable, general-purpose skill or transferring across variations of a problem within the same DSL? What advantages does your approach offer over these alternatives? You would need to run strong pair of test-time experiments applied to both the baseline (within a DSL), and across the DSLs.

What results do you obtain with GPT-6 (available at the time of submission)? Does the proposed approach still provide improvements when using a stronger model?

3 comments
AuthorsOpenReview ↗

Response to Reviewer h8gd (Part 1/3): Scope & Cross-DSL Setting (W1, Q1)

Thank you for your thoughtful review and for recognizing the benchmark's coverage of real-world workloads, the breadth of our model evaluation, and the value of our negative-transfer analysis.

We have completed two additional full evaluation runs across all six models, and both new runs reproduce the finding that benefit and regression coexist in experience reuse, with specific effect sizes now reported as three-run means alongside their uncertainty. We have also clarified the motivation for our same-problem cross-DSL design, the task specificity of source skills, the relationship between CuTe DSL and CUTLASS, and directions for extending the benchmark in the paper and this response. Revisions in the paper are marked in blue, and key conclusions below are highlighted in bold. We present our response in three parts, covering W1 and Q1 in Part 1, W2, W3, and Q1 in Part 2, and W4, W5, and Q2 in Part 3.

1. Benchmark Scope and Value of the Cross-DSL Setting W1 + Q1

W1: My main concern is knowledge between DSLs for the same kernel on the same GPU seems less useful than transferring optimizations across GPU architectures or between related kernels (e.g., attention variants). ...

Q1: What motivates the proposed transfer-learning approach over developing a reusable, general-purpose skill or transferring across variations of a problem within the same DSL? ... You would need to run strong pair of test-time experiments applied to both the baseline (within a DSL), and across the DSLs.

MetaKernelBench holds the computation and hardware fixed in order to isolate and measure an agent's ability to reuse problem-specific optimization knowledge across implementation interfaces. Repeating the same problem within a single DSL lets DSL-specific implementation recipes and toolchain mechanics carry over unchanged, making it difficult to separate reusing surface interface recipes from transferring underlying algorithmic and hardware insights. Conversely, changing the operator variant or GPU architecture alters the workload semantics or hardware bottlenecks and changes which prior findings remain valid, so when an attempt regresses, one cannot separate whether prior findings no longer apply to the new workload or hardware from whether the agent failed to translate applicable insights. Fixing the computation graph, workload configuration, and B200 GPU while switching between CuTe DSL and TIRx and withholding source code and trajectories isolates a single core question, namely whether optimization knowledge acquired in a prior attempt can be translated into an effective implementation under a different set of programming primitives.

Within each target DSL, the skill-conditioned and solo attempts share the same model, offline documentation, verifier, and 40-minute target budget, with the source skill as the sole additional input, so the paired runtime ratio directly measures the net effect of the transferred experience. Appendices E.1–E.2 clarify this boundary relative to cross-problem and cross-hardware transfer and outline matched within-DSL transfers, transfers across related problem variants, and comparisons with generic optimization guides as complementary follow-up directions. Relevant discussion and revisions appear in Section 1, Sections 2.1–2.2, and Appendices E.1–E.2.

AuthorsOpenReview ↗

Response to Reviewer h8gd (Part 2/3): Repeated Runs & Skills (W2, W3, Q1)

2. Repeated Experiments and Statistical Reliability W2

W2: The evaluation relies on single runs of agents. Without repeated experiments or confidence intervals ...

We have completed two additional full evaluation runs for all six models across all 74 problems and both directions, regenerating source skills and running fresh skill-conditioned attempts alongside same-run solo controls under the same 40-minute target budget. Both new runs reproduce our core findings that help and harm coexist and that help rates track the source attempt's relative standing. Across the 2,664 paired comparisons in the three runs, all 12 model-direction combinations exhibit both help and harm in every run, with overall help rates of 25.9% to 31.2%, harm rates of 20.3% to 25.1%, and same-run above-versus-below source-standing help rates of 71.2% / 10.6%, 69.1% / 16.2%, and 65.2% / 14.2% in Runs 1, 2, and 3.

The main tables now report three-run arithmetic means to summarize aggregate effects, paired with across-run sample standard deviations and 95% percentile intervals from 10,000 problem-cluster bootstrap draws resampling the 74 problems with replacement to quantify run-to-run variation and task-resampling uncertainty, respectively.

ModelCuTe DSL→TIRx 3-run mean [95% CI]TIRx→CuTe DSL 3-run mean [95% CI]
GLM-5.3-Flash+1.1% [−8.4%, +11.6%]+9.2% [−2.2%, +22.0%]
Gemini 3.7 Flash+3.4% [−6.4%, +15.9%]+10.0% [+0.5%, +20.9%]
Qwen3.8-Flash−1.8% [−10.8%, +8.9%]+10.1% [−1.5%, +23.9%]
DeepSeek-V4-Pro+3.5% [−4.7%, +13.6%]+17.3% [+7.3%, +29.3%]
GLM-5.3+8.7% [−2.2%, +21.4%]+26.6% [+13.9%, +41.9%]
GPT-5.6 Sol+5.0% [−4.7%, +17.2%]+2.5% [−8.7%, +15.9%]

Five of the six models achieve positive three-run mean lift in both transfer directions, and Gemini 3.7 Flash, DeepSeek-V4-Pro, and GLM-5.3 have 95% interval lower bounds strictly above zero from TIRx to CuTe DSL. Intervals that span zero correspond to conditions where substantial help and harm coexist across problems within the same model and direction and offset in the aggregate mean. The updated three-run means appear in Section 3.1, Sections 4.1–4.2, Tables 1–2, and Figures 3–4, per-run lifts and across-run standard deviations appear in Appendix B.1 Tables 3–4, and bootstrap intervals alongside across-run problem-level outcome counts appear in Appendix B.2 Table 5.


3. Task-Specific Experience and Generic Optimization Guidelines W3 + Q1

W3: It is unclear whether the transferred skills contain genuinely useful knowledge ... or simply general GPU optimization tricks. A baseline using generic optimization guidelines would help ...

Transferred source skills contain concrete, problem-specific findings grounded in the source attempt's execution feedback rather than merely generic GPU optimization tips. Across the 841 source skills analyzed in the first run, 86% record bottleneck analyses and timing comparisons tied to specific implementation changes, 84% document workload shapes, sizes, or operator properties beyond the task prompt, 78% record bounded failure diagnoses from tested candidates, and 54% contain algorithmic or algebraic reformulations verified on the GPU. Moreover, if skills only supplied static generic advice independent of the specific source attempt, transfer outcomes would not vary sharply with the source attempt's performance tier relative to the target baseline, whereas our three-run evaluation shows a 68.6% help rate when the source stands above the same-run target baseline versus 13.5% when it stands below, and an 18.5 percentage-point advantage with a 95% problem-cluster bootstrap interval of [12.6, 24.5] even when target standing is estimated from the other two independent runs. The p68 case in Figure 6 of Section 4.3 illustrates this specificity directly, as the source skill records that promoting inputs to float64 leaves the numerical error unchanged and thereby isolates the fault to indexing, a diagnostic control that the target attempt reruns to fix the gate-index computation.

In our evaluation protocol, both skill-conditioned and solo attempts already receive the same offline DSL documentation and workflow guidance in the task instruction, so paired lift measures the marginal effect of adding experience distilled from a prior attempt on top of shared static guidance. Addressing W3 and the comparison with reusable general-purpose skills in Q1, comparing attempt-distilled source skills directly against static generic optimization guides can further quantify the gain of task-specific exploration over static priors, and we have included this comparison in Appendix E.2 as a follow-up direction. Relevant analyses and revisions appear in Sections 4.2–4.3, Figures 5–6, Appendix A.1 Figure 8, Appendix C.1 Table 8, and Appendix E.2.

AuthorsOpenReview ↗

Response to Reviewer h8gd (Part 3/3): DSLs & Model Scaling (W4, W5, Q2)

4. Choice of DSLs and Coverage of CUDA and CUTLASS W4 + W5

W4: Only two relatively low-level DSLs are evaluated. Since both expose similar hardware-level concepts, it is unclear how much the results demonstrate meaningful cross-DSL transfer.

W5: Given CUDA's importance in GPU programming, choosing CUDA or Cutlass as a source language is a relevant candidate that's missing.

Although CuTe DSL and TIRx both expose low-level hardware resources, they rely on distinct programming abstractions to express tiling, memory hierarchy placement, and thread scheduling, so source kernel code cannot be copied across DSLs and optimization strategies must be re-implemented in the target DSL's primitives. As detailed in Section 2.1, CuTe DSL uses explicit layout algebra to partition data and computation across threads, whereas TIRx declares hardware distribution through storage layouts that guide compiler dispatch of tile primitives. In the p74 case shown in Appendix D Figure 11, for instance, the TIRx source skill records a strategy that assigns one thread block per batch-head pair and one thread per state-matrix column to fuse the recurrence in registers, and the CuTe DSL target attempt, without access to the source code, must independently construct a native recurrence kernel using CuTe tensor layouts and loop primitives, cutting end-to-end runtime from 48.00 ms in solo to 4.18 ms for an 11×11\times speedup.

Furthermore, the CuTe DSL evaluated in our benchmark is NVIDIA CUTLASS 4's official Python kernel programming interface, evaluated with CUTLASS DSL 4.7.0, and shares the same layout algebra, tensor abstractions, and underlying CUDA toolchain as CUTLASS and CuTe C++. Selecting Python-embedded CuTe DSL and TIRx allows agents to compose eager PyTorch operations with custom kernels under a unified interface for end-to-end verification and timing. Because our task specifications and verification protocol are DSL-neutral, the benchmark extends naturally to native CUDA C++ and CUTLASS C++. Relevant clarifications and revisions appear in Section 2.1, Appendix A.2, Appendix D Figure 11, and Appendix E.2.


5. Model Coverage and Extension to GPT-6 Q2

Q2: What results do you obtain with GPT-6 (available at the time of submission)? Does the proposed approach still provide improvements when using a stronger model?

Across the three larger models in our evaluation panel, GLM-5.3, DeepSeek-V4-Pro, and GPT-5.6 Sol, cross-DSL skill transfer achieves positive three-run mean lift in both directions and delivers its largest gains when the source attempt produces a high-speedup implementation. Within the same-family pair, GLM-5.3 achieves +8.7% and +26.6% mean lift across the two directions compared with +1.1% and +9.2% for the lighter GLM-5.3-Flash, and GLM-5.3 and DeepSeek-V4-Pro reach three-run mean lifts of +26.6% and +17.3%, respectively, from TIRx to CuTe DSL, both with 95% interval lower bounds above zero. As shown in the right panel of Appendix B.5 Figure 9, when transferring from source attempts that reach 2×2\times speedup or better, the three larger models achieve mean paired lifts of 16.3% to 30.1%, outperforming the three Flash-tier models at 4.3% to 13.1%. At the same time, every model still exhibits both help and harm in both directions across all three runs, and 65% to 86% of assigned target-side failures in the three larger models fall into search truncation or unverified adoption, showing that stronger model capability amplifies gains from strong source skills without automatically eliminating negative transfer from suboptimal guidance.

Our revision completes systematic three-run evaluations across a fixed six-model panel spanning three Flash-tier models and three larger models, and outlines extending the panel to newer frontier models such as GPT-6 under the same task set and 40-minute paired budgets in Appendix E.2 as a follow-up direction. Relevant results and revisions appear in Section 3.1, Sections 4.1 and 4.4, Appendix B.5 Figure 9, and Appendix E.2.


Thank you again for your valuable comments, which helped us clarify the motivation and empirical findings of our work.

Reviewer_SAHROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

MetaKernelBench evaluates same-problem knowledge transfer between CuTe DSL and TIRx on 74 fused-subgraph tasks. Six models distill natural-language skills from source attempts and receive those skills in fresh target attempts with matched per-attempt budgets. Across 888 paired comparisons, reported lift ranges from −19.1% to +29.3%. The paper relates transfer outcomes to source quality, skill content, and target trajectories.

Critical strengths
  1. A well-defined information-transfer setting. The target receives a natural-language skill while source code and trajectories are withheld. Matched task variants and target budgets make the transferred information explicit, and the paper appropriately distinguishes per-attempt budgets from total source-plus-target computation.

  2. The analysis includes negative transfer. Failed sources are retained, and results are broken down by model, direction, and task family. Appendix B.2 adds randomization analyses of direction asymmetry and source tracking, supplementing the aggregate lift results.

Critical weaknesses
  1. Benefit and harm are not separated from agent-run variability sufficiently. Each target arm is a single fresh attempt, without repeated no-skill controls or confidence intervals for model/direction lifts. Appendix B.2 supports aggregate dependence, but does not validate individual harm labels or small gains such as 2.3%. Thus, the benchmark does not yet provide a reliable quantitative account of when transfer helps or hurts.

  2. The mechanistic conclusions rely on unvalidated post-hoc labels. LLM annotations have no reported expert agreement, and Appendix C.2 assigns causes by precedence, with search truncation covering remaining slowdowns. Section 4.3 acknowledges that traces do not isolate causality, yet Section 4.4 treats these labels as causes of harm. Controlled skill edits are needed to support that interpretation; descriptive label frequencies alone do not establish it.

Questions for authors
  1. Across repeated source and target attempts, how do skill-conditioned outcomes compare with fresh no-skill retries under the same target budget? Please report model/direction confidence intervals.

  2. Does removing or correcting statements implicated in unsupported closure or search truncation reverse the predicted harm on matched tasks? How well do blinded expert annotations agree with the LLM labels?

  3. Does source-relative tier still predict transfer when target baseline quality is estimated from independent repeated attempts? The current grouping and paired outcome both reuse the same solo result.

  4. Do the main conclusions persist under other ratio clip bounds and rescue/break weights? Please report runtime changes on jointly passing pairs separately from pass/fail transitions, since the assigned 4 and 1/4 values are scores rather than measured speedups.

3 comments
AuthorsOpenReview ↗

Response to Reviewer SAHR (Part 1/3): Repeated Runs & CIs (W1, Q1)

Thank you for your thoughtful review and for recognizing MetaKernelBench's well-defined transfer setting and systematic analysis of negative transfer.

We have completed two additional full evaluation runs across all six models, and both new runs reproduce the finding that benefit and regression coexist in skill reuse, with specific effect sizes now reported as three-run means alongside their uncertainty. We have also verified and clarified the baseline grouping, scoring decomposition, and annotation grounding in the paper and this response. Revisions in the paper are marked in blue, and key conclusions below are highlighted in bold. We present our response in three parts, covering W1 and Q1 in Part 1, W2 and Q2 in Part 2, and Q3 and Q4 in Part 3.

1. Repeated Runs and Uncertainty W1 + Q1

W1: Benefit and harm are not separated from agent-run variability sufficiently. Each target arm is a single fresh attempt, without repeated no-skill controls or confidence intervals for model/direction lifts. Appendix B.2 supports aggregate dependence, but does not validate individual harm labels or small gains such as 2.3%. ...

Q1: Across repeated source and target attempts, how do skill-conditioned outcomes compare with fresh no-skill retries under the same target budget? Please report model/direction confidence intervals.

Each evaluation run covers all six models, all 74 problems, and both transfer directions, regenerating source skills and running fresh skill-conditioned attempts alongside same-run no-skill controls under the same 40-minute target budget. Both new runs reproduce our core findings that help and harm coexist and that help rates track the source attempt's relative standing.

ObservationRun 1Run 2Run 3
Model and direction cells showing both help and harm12/1212/1212/12
Help rate across all paired comparisons31.2%30.1%25.9%
Harm rate across all paired comparisons25.1%20.3%21.2%
Help rate for same-run source above / below71.2% / 10.6%69.1% / 16.2%65.2% / 14.2%

Across all 2,664 paired comparisons over three runs, 774 are helped, or 29.1%, and 591 are harmed, or 22.2%. We update the main tables to report aggregate effects as three-run arithmetic means, and combine across-run sample standard deviations with 95% percentile intervals from 10,000 problem-cluster bootstrap draws resampling the 74 problems with replacement to quantify run-to-run variation and task-resampling uncertainty, respectively.

ModelCuTe DSL→TIRx 3-run mean [95% CI]TIRx→CuTe DSL 3-run mean [95% CI]
GLM-5.3-Flash+1.1% [−8.4%, +11.6%]+9.2% [−2.2%, +22.0%]
Gemini 3.7 Flash+3.4% [−6.4%, +15.9%]+10.0% [+0.5%, +20.9%]
Qwen3.8-Flash−1.8% [−10.8%, +8.9%]+10.1% [−1.5%, +23.9%]
DeepSeek-V4-Pro+3.5% [−4.7%, +13.6%]+17.3% [+7.3%, +29.3%]
GLM-5.3+8.7% [−2.2%, +21.4%]+26.6% [+13.9%, +41.9%]
GPT-5.6 Sol+5.0% [−4.7%, +17.2%]+2.5% [−8.7%, +15.9%]

Five of six models achieve positive three-run mean lift in both directions, and three models have 95% interval lower bounds above zero from TIRx to CuTe DSL. Small mean lifts and intervals spanning zero primarily reflect substantial help and harm coexisting within the same condition and offsetting in the aggregate mean. The first-run 2.3% gain you noted corresponds to Gemini 3.7 Flash from CuTe DSL to TIRx, which now has a +3.4% three-run mean lift, a 27.5% mean help rate, a 25.7% mean harm rate, and 26 of 74 problems showing both help and harm across the three runs. An individual comparison's harm label records the actual gap between that run's skill-conditioned attempt and its matched same-run solo control, while the across-run statistics separate per-problem trajectory variation from aggregate transfer trends. These updates appear in Section 3.1, Sections 4.1–4.2, Tables 1–2, Figures 3–4, Appendix B.1 Tables 3–4, and Appendix B.2 Table 5.

AuthorsOpenReview ↗

Response to Reviewer SAHR (Part 2/3): Labels & Mechanism (W2, Q2)

2. First-Run Annotations and Mechanistic Interpretation W2 + Q2

W2: The mechanistic conclusions rely on unvalidated post-hoc labels. LLM annotations have no reported expert agreement, and Appendix C.2 assigns causes by precedence, with search truncation covering remaining slowdowns. ... Controlled skill edits are needed to support that interpretation; descriptive label frequencies alone do not establish it.

Q2: Does removing or correcting statements implicated in unsupported closure or search truncation reverse the predicted harm on matched tasks? How well do blinded expert annotations agree with the LLM labels?

The skill-content and trajectory analyses link specific statements in a source skill to target-side uptake actions and evaluator feedback to document verifiable behavioral patterns rather than claiming single-factor causality from label frequencies alone. As shown in Figure 6 of Section 4.3, the skill-conditioned attempt on p68 reuses the source skill's thread mapping and reruns its float64 diagnostic control to fix an indexing bug and fuse the recurrence, whereas the attempt on p71 cites the source skill's unsupported closure ruling out a fused recurrence, retains eager execution, and never completes the native recurrence implemented by solo. When a harmed trajectory exhibits multiple issues, Appendix C.2 applies a fixed precedence rule to assign one primary failure class per target-side failure so that the single-label shares in the left panel of Figure 7 do not double-count comparisons, with search truncation placed last to cover attempts that retain the source design without exploring a better target-DSL option as well as remaining slowdowns without a more specific execution or adoption fault.

To evaluate the reliability of the first-run Claude annotations, we drew a stratified sample of 100 artifacts across all six models and both directions, comprising 60 source skills and 40 harmed trajectories, and conducted both cross-model GPT review and blinded human expert annotation. The GPT review audited the skill text, target-side skill-conditioned and solo trajectories, and evaluator reports against the original rubric, while the human expert annotated the same items independently under fully blinded conditions without access to either the original labels or the GPT outputs.

Audit itemGPT review matched / totalGPT review agreementBlinded expert matched / totalBlinded expert agreement
All 11 skill-content binary judgments633 / 66095.9%612 / 66092.7%
Unsupported closure judgments54 / 6090.0%51 / 6085.0%
Primary failure class24 / 4060.0%32 / 4080.0%

Across all 11 skill-content categories and on unsupported closure specifically, both the cross-model GPT review and the blinded human expert achieve 85.0% to 95.9% agreement with the original labels, and blinded human expert agreement on the primary failure class reaches 80.0%.

Our main benchmark retains the raw skills distilled by source attempts without manual editing in order to evaluate end-to-end distillation and reuse, while Appendix E.2 outlines controlled skill editing as a follow-up direction. Removing or revising individual unsupported-closure or search-limiting statements on matched tasks under the same target budget provides a direct way to isolate the effect of individual guidance statements. The corresponding revisions appear in Section 1, Sections 4.3–4.4, Figure 7, and Appendix C.2 Table 9.

AuthorsOpenReview ↗

Response to Reviewer SAHR (Part 3/3): Baseline Split & Scoring (Q3, Q4)

3. Independent Baseline Grouping Q3

Q3: Does source-relative tier still predict transfer when target baseline quality is estimated from independent repeated attempts? The current grouping and paired outcome both reuse the same solo result.

We decouple target-side relative-standing grouping from paired scoring by retaining the current source attempt that produced the transferred skill, estimating target baseline standing from the mean tier of the other two target solo runs, and using the current target solo attempt only to score the paired outcome. Encoding failed attempts as 0, passing attempts below 2×2\times as 1, and passing attempts at 2×2\times or better as 2 yields the following comparison between independent target grouping and same-run grouping across all 2,664 comparisons.

GroupingSource above nn / Help / HarmSource same nn / Help / HarmSource below nn / Help / HarmAbove−Below help gap [95% CI]
Independent target from other two runs710 / 39.9% / 25.1%1,316 / 27.0% / 20.1%638 / 21.3% / 23.2%+18.5 pp [12.6, 24.5]
Same-run target baseline437 / 68.6% / 9.2%1,790 / 23.2% / 19.8%437 / 13.5% / 44.9%+55.1 pp [49.3, 60.8]

When target baseline standing is estimated from independent repeated runs, help rates still rise monotonically with the source attempt's relative standing from 21.3% to 27.0% and 39.9%, giving an above-versus-below gain of 18.5 percentage points with a 95% problem-cluster bootstrap interval of [12.6, 24.5]. At the same time, harm rates narrow from 9.2%–44.9% under same-run grouping to 20.1%–25.1% across tiers, with an above-versus-below difference of +1.9 percentage points and a 95% interval of [−3.4, +7.0]. This comparison shows that shared target-baseline noise contributes to the wider contrast under same-run grouping, yet the source attempt's relative standing continues to predict beneficial transfer once that shared noise is removed, whereas regressions also depend on specific skill claims and target-side execution. These results are reported in Section 4.2 and Appendix B.6 Table 7.


4. Scoring Sensitivity and Measured Speedups Q4

Q4: Do the main conclusions persist under other ratio clip bounds and rescue/break weights? Please report runtime changes on jointly passing pairs separately from pass/fail transitions ...

Separating correctness transitions from measured runtimes shows that 72.5% of helped and 67.0% of harmed comparisons come from runtime changes on jointly passing pairs, and 10 of the 12 model-direction cells retain a geometric-mean speedup above 1.0×1.0\times on unclipped runtimes alone. Across the 774 helped and 591 harmed comparisons over three runs, verdict transitions account for 213 fail-to-pass and 195 pass-to-fail cases, while the remaining 561 helped and 396 harmed cases come from measured runtime changes across the 1,845 executed, jointly passing pairs without using the 44 or 1/41/4 transition weights. The table below reports unclipped baseline-to-treatment runtime-ratio geometric means on these jointly passing pairs, reaching 1.095×1.095\times to 1.443×1.443\times across all six models from TIRx to CuTe DSL.

ModelCuTe DSL→TIRx 9-setting lift rangeCuTe DSL→TIRx jointly passing ratio GMTIRx→CuTe DSL 9-setting lift rangeTIRx→CuTe DSL jointly passing ratio GM
GLM-5.3-Flash[−0.59%, +3.29%]0.937×[+4.41%, +13.85%]1.289×
Gemini 3.7 Flash[+2.09%, +3.98%]1.053×[+8.48%, +11.40%]1.103×
Qwen3.8-Flash[−3.00%, +0.06%]0.903×[+5.55%, +15.03%]1.274×
DeepSeek-V4-Pro[−1.14%, +7.08%]1.114×[+8.56%, +22.49%]1.317×
GLM-5.3[+5.08%, +12.21%]1.113×[+15.14%, +35.12%]1.443×
GPT-5.6 Sol[+1.69%, +7.29%]1.088×[−1.07%, +6.72%]1.095×

Across the 3×33 \times 3 grid of clipping bounds c∈{2,4,8}c \in \lbrace 2, 4, 8 \rbrace and correctness-transition weights w∈{2,4,8}w \in \lbrace 2, 4, 8 \rbrace, all comparison classifications remain identical and 8 of the 12 model-direction cells stay strictly positive across all nine settings. Because the 1.251.25 and 0.80.8 help and harm thresholds lie inside all tested bounds and weights, every comparison's help, harm, or neutral label and all relative-standing help rates are unchanged across the nine settings, while the only 4 cells whose mean lifts cross zero are those whose baseline three-run mean lifts in Table 1 already lie near zero between −1.8% and +3.5%. These updates and the full per-cell decomposition appear in Section 4.1 and Appendix B.3 Table 6.


Thank you again for your constructive feedback, which helped us strengthen the statistical and empirical analyses of the paper.

Comments on the paper

1 comment

Thanks to the authors for the effort put into the two additional full runs and the extra analyses. The 3-run intervals, the independent baseline split, the separate runtime breakdown on jointly passing pairs, and the blinded human check make the empirical results much more solid than in the initial submission, so I am raising my Soundness score from 2 to 3.

At the same time, the decoupled split also shows that around 25% harm still happens even when transferring from stronger source runs, clear aggregate gains mainly appear in the TIRx to CuTe DSL direction, and direct skill-editing tests are still left to future work. I will keep my overall rating as is for now and discuss the final recommendation further with the AC and the other reviewers during the discussion phase.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.