acceptodds
Under review as a conference paper at ICLR 2027

MegaKernelBench: A System-Level Benchmark for Megakernel Generation and Optimization

Abstract

Small-model, small-batch LLM decoding launches many fine-grained GPU kernels, making launch overhead a bottleneck for interactive inference. Megakernels mitigate this overhead by executing part or all of a computation graph within a single persistent work kernel. However, their performance depends jointly on operator implementation and scheduling, which existing GPU kernel benchmarks do not adequately evaluate. We present MegaKernelBench, a system-level benchmark for megakernel generation and optimization with four progressive tasks: operator generation, scheduling generation, joint operator-scheduling generation, and generation without a prebuilt runtime. It evaluates correctness, structural validity, and end-to-end performance. Existing methods can generate correct megakernels with a prebuilt runtime but generally underperform mainstream inference engines; generation without one remains more challenging. Strong operator-scheduling coupling further makes joint optimization essential. MegaKernelBench enables unified, reproducible evaluation of megakernel generation by compilers and agents.

Primary area:
datasets and benchmarks
Keywords:
Megakernel, LLM Inference, GPU Code Generation, Operator–Scheduler Co-optimization, Benchmarking

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Zkv8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MegaKernelBench is a system-level benchmark for evaluating the generation and optimization of megakernels for LLM decode. It defines four tasks: operator optimization, scheduling optimization, joint kernel optimization and kernel scheduling using a provided runtime, and complete megakernel generation without a prebuilt runtime. Experiments evaluate 3 agent frameworks driven by 3 language models across 6 target models on NVIDIA A800 and H100 GPUs, checking functional correctness, structural validity, and decoding latency. The results show that agents can often generate correct implementations when a runtime is provided, but fewer than 10% of submissions succeed without one. The generated implementations remain substantially slower than Mirage Persistent Kernel (MPK) on the small models where it is evaluated.

Critical strengths

Megakernels are powerful ways to gain performance, so this is a timely benchmark. The paper identifies that generating efficient kernels for a neural network and organizing their execution within a megakernel are two separate problems. Task partitioning, dependencies, synchronization, and GPU resource allocation require evaluation beyond just an individual operator performance.

The four tasks separate operator generation, scheduling generation, joint optimization, and generation without a prebuilt runtime. This structure makes it easier to identify which capabilities an agent has and where it fails.

Although KernelBench Levels 3 and 4 cover full models, whole-model correctness and speedup do not by themselves isolate the challenges of megakernel execution. MegaKernelBench improves that by explicitly evaluating GPU-side scheduling and structural requirements, making it better suited to this use case.

The experiments show that substantial improvements in individual operators can yield only small end-to-end gains when scheduling remains fixed. This supports the motivation for evaluating operator and scheduling optimization together.

Critical weaknesses

The performance evaluation lacks competitive baselines. PyTorch eager is useful as a reference, but it is insufficient as the main performance baseline for an inference optimization benchmark. Speedups over eager execution do not establish practical competitiveness. What are the end-to-end speedups against TensorRT-LLM, torch.compile , SGLang, and vLLM under matched workloads and configurations, including appropriate use of CUDA Graphs? MPK is included for only a subset of models, where it substantially outperforms the agent-generated implementations.

The agent evaluation should include stronger contemporary models. The paper should include leading OpenAI and Claude models to assess whether the reported limitations persist with stronger coding agents (leading at the time of the submission like GPT-6 Astra). Comparisons should use comparable optimization budgets and tool access. Many of these conclusions would change.

The reference baseline needs more description. For Tasks A and B, the baseline is an agent-generated AutoMegaKernel implementation. Which agent and model produced it, with what optimization budget and selection procedure? These details matter because the reported speedups depend on the quality of that reference implementation.

Multi-GPU applicability remains unclear. The evaluation is limited to a single GPU, leaving communication, cross-device synchronization, and computation–communication overlap unexplored. Would this task decomposition and validation protocol apply to multi-GPU megakernels as well? Which additional scheduling and correctness constraints would be required, and would the benefits of joint operator–scheduling optimization still be there when communication of the multi-GPU setup becomes the bottleneck?

Questions for authors

The benchmark decomposition is promising, but competitive inference baselines and stronger agent comparisons are needed to establish its empirical contribution. In particular, I have the following questions for the authors:

Do the conclusions still hold when comparing against modern inference baselines such as TensorRT-LLM, SGLang, vLLM, and torch.compile, with appropriately configured CUDA Graphs? How do the generated megakernels compare with MPK across a broader range of models?

How was the reference implementation constructed and optimized? Which agent, model, optimization budget, and selection procedure were used, and how sensitive are the reported speedups to the quality of this reference?

Beyond single-step decoding latency, what throughput do the implementations achieve, measured in tokens per second across a few different batch sizes and context lengths?

5 comments
AuthorsOpenReview ↗

Response for Reviewer Zkv8(Part #1)

We thank the reviewer for the detailed feedback and constructive suggestions. Below we address the concerns point-by-point.

W1 & Q1: No competitive baselines; MPK on only a subset of models.

The performance evaluation lacks competitive baselines. PyTorch eager is useful as a reference, but it is insufficient as the main performance baseline for an inference optimization benchmark. Speedups over eager execution do not establish practical competitiveness. What are the end-to-end speedups against TensorRT-LLM, torch.compile , SGLang, and vLLM under matched workloads and configurations, including appropriate use of CUDA Graphs? MPK is included for only a subset of models, where it substantially outperforms the agent-generated implementations.

Do the conclusions still hold when comparing against modern inference baselines such as TensorRT-LLM, SGLang, vLLM, and torch.compile, with appropriately configured CUDA Graphs? How do the generated megakernels compare with MPK across a broader range of models?

W1 & Q1 (1/2):

We agree that mature inference engines and compilers are important references for gauging real-world deployment competitiveness, but they serve a different role from the baseline this paper uses for unified evaluation of generation capability; the primary goal of MegaKernelBench is to diagnose whether existing methods can generate correct and efficient megakernels, not to use speedups over PyTorch eager to claim superiority over production-grade inference systems. This paper therefore needs two kinds of complementary references at the same time: the unified baseline (PyTorch) for comparable capability evaluation across different models, tasks, and generation methods, and strong inference-engine/compiler baselines for locating how large the remaining performance gap between the generated results and mature deployment solutions is.

First, PyTorch eager is not the only performance baseline for the four task types. Tasks A/B use a fixed reference megakernel, directly measuring the improvement of the generated operators or schedules relative to the same system starting point; only the end-to-end Tasks C/D use native PyTorch as the unified reference. PyTorch was chosen because it can provide essentially consistent API/model semantics for models across hardware platforms, whereas TensorRT-LLM, vLLM, SGLang, and torch.compile do not support and optimize specific models, operators, hardware, and configurations in a fully consistent manner. These system baselines can therefore answer "do the generated implementations have deployment competitiveness," but they are not better suited than PyTorch to serve as the unified baseline shared by all models and generation methods. Nor does the choice of baseline change the correctness of the submissions, their absolute latencies, or the relative ordering of generation methods by latency within the same task.

Second, MPK appears on only a subset of models because we report results only on the models its current implementation can accommodate, not because we deliberately excluded unfavorable comparisons. More importantly, we do not regard MPK's clear lead as a result to be avoided: it directly shows that, within the existing coverage, dedicated compilation methods for megakernels still significantly outperform current agent-generated solutions, revealing a capability gap that agents have not yet closed in structural optimization and joint design. This is precisely the current state of affairs that the benchmark intends to measure, not a claim by this paper that the generated results have already reached deployment optimality.

Your suggestion can further supplement the dimension of deployment competitiveness. We have added TensorRT-LLM, torch.compile, SGLang, vLLM, and PyTorch + CUDA Graph baselines, reporting their absolute latencies under aligned models, input states, and timing scopes, while retaining PyTorch as the unified reference. This can more directly demonstrate the actual performance gap between the generated megakernels and mature compilers/inference engines.

AuthorsOpenReview ↗

Response for Reviewer Zkv8(Part #2)

W1 & Q1 (2/2):

The supplementary experimental data are as follows (H100, batch=1, continuous 128-token greedy decoding, bf16; the table shows the speedup of each inference system relative to HF PyTorch eager):

Modeleager (tps)PyTorch + CUDA Graphtorch.compile (best variant)vLLMSGLangTensorRT-LLMbest agent megakernel (task C)best agent megakernel (task D)
Llama-3.2-1B101.03.79×5.85× (+graph)6.86×5.32×5.11×1.80×12.26×
Qwen3-0.6B48.44.86×11.18× (+graph)13.50×11.03×9.97×3.55×28.64×
gpt-oss-20b36.3✗0.90× (fullgraph=False)8.44×7.72×4.67×0.87×✗
DeepSeek-V2-Lite19.4✗✗14.82×11.76×✗ run failed12.75×✗
GLM-4.7-Flash4.58✗0.93× (fullgraph=False)37.56×✗ unsupported✗ run failed2.50×✗
Qwen3-30B-A3B7.96✗0.81× (fullgraph=False)27.17×26.37×20.43×3.85×✗

Four observations can be drawn from the table above:

(1) vLLM is the fastest system on all 4 MoE tracks, while on the 2 dense tracks the megakernels generated by agents in task D are faster;

(2) a "stronger PyTorch baseline" cannot be constructed on any of the MoE models. PyTorch's native acceleration paths (CUDA Graph capture, compile fullgraph) all fail structurally, and the fullgraph=False fallback is instead slower than eager, so for these tracks eager is already the ceiling natively attainable with PyTorch;

(3) the best agent submissions in Task C of this benchmark surpass only PyTorch eager on most tracks and still lag behind mainstream inference engines, matching mainstream inference engines only on DeepSeek-V2-Lite in performance, but the generation success rate of Task C on the MoE tracks is markedly higher than that of Task D.

(4) In Task D, agents are already able to independently implement megakernels that outperform mainstream inference engines on some structurally simple, small-scale dense models, but on MoE models, agents can hardly implement a functionally correct megakernel within the given time budget (90 min).

AuthorsOpenReview ↗

Response for Reviewer Zkv8(Part #3)

W2: Stronger models such as GPT-6 Astra should be included.

The agent evaluation should include stronger contemporary models. The paper should include leading OpenAI and Claude models to assess whether the reported limitations persist with stronger coding agents (leading at the time of the submission like GPT-6 Astra). Comparisons should use comparable optimization budgets and tool access. Many of these conclusions would change.

The existing experiments do not rely on weaker models to reach their conclusions. Under a limited experimental budget, we chose GPT-5.6-Sol, GLM-5.3, and the DeepSeek-V4 series, which cover currently representative strong coding models; although they are not SOTA models, they already possess strong code-generation and reasoning capabilities. The goal of this paper is to examine, through multiple representative models, the capabilities and common difficulties of current mainstream LLMs/agents on different megakernel generation tasks. To address your concern, we will additionally run experiments with Claude and GPT-6-Astra and include the results in the appendix of the final version. That said, the models adopted in this paper are already representative: GPT-5.6-Sol already outperforms the vast majority of mainstream LLMs, which also indicates that, for the vast majority of LLMs, megakernel generation tasks are extremely difficult — a conclusion that will not change by adding generation experiments with SOTA models.

Reviewer_LGHxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents MegaKernelBench, a benchmark for megakernel generation and optimization in small-model and small-batch LLM decoding. It builds on AutoMegaKernel and defines four progressive tasks: operator generation with fixed schedule, scheduling generation with fixed operators, joint operator-scheduling generation, and generation without a prebuilt runtime. Evaluation show that current agents still struggle with correctness and performance of megakernel generation.

Critical strengths
  • Fills an important evaluation gap for megakernel generation, a promising direction for low-latency inference that remains an open problem for LLMs.
  • Covers diverse model architectures, with extensive evaluation across agent harnesses and underlying LLMs.
Critical weaknesses
  • Strong reliance on AutoMegaKernel (AMK) limits the benchmark's scope, making it more like an AMKBench than a general megakernel benchmark. AMK’s approach is not an established best way to write megakernels.
  • The benchmark only includes full-model megakernels, which are difficult for agents to generate directly. Layer- and subgraph-level megakernels are also relavant, and full-model megakernels may not always be optimal for low-latency LLM decoding.
Questions for authors

The following comments and questions are related to the aforementioned weaknesses.

  • Tasks A-C are built heavily on AMK, reusing AMK’s data structure, scheduling and runtime to benchmark the operator and scheudling components separately. However, AMK’s approach is not established to represent megakernel authoring in general. Mirage Persistent Kernels already follows a different paradigm with good performance. How best to design a megakernel is still an open problem.
  • Full-model megakernels put scheduling logic of an entire model into one kernel, which current LLMs could struggle to build in a few rounds. A full-model megakernel is also not always the best practice for low-latency decoding. Multiple kernels with CUDA graph + Programmatic Dependent Launch is a strong alternative. While a single kernel per decode step is required by the benchmark, why it is the right criterion? Would the authors consider adding layer- and subgraph-level tasks, such as attention variants and MoE layers? They would add an easier tier in the benchmark and may also outperform full-model megakernels.
  • Task A fixes the schedule and varies only the operators, while in a megakernel the two are higly coupled. Is each operator submitted as a device function or as a standalone CUDA kernel? Can a submission change tile shapes or shared memory use (both depend on schedule)? Is operator performance measured standalone or inside the megakernel?
  • Even in single request low-latency settings, speculative decoding is widely used. MegaKernelBench only evaluates batch size 1, where most of the computations are GEMV and cannot utilize GPU tensor cores. Consider extending the benchmark to some small batch sizes, e.g., 8 and 16.
  • PyTorch eager mode is not a competitive baseline. For LLM inference, better ML compilers (TorchInductor, TensorRT) or LLM serving engines (vLLM, Sglang) should be used.
  • A large proportion of the megakernel performance gains come from software pipelining, which benefits from async GPU features, while on tensor cores on Amphere GPUs are synchronous. Blackwell B200/B300 GPUs provide async tcgen05 instructions and tensor memory, which can reduce accumulator register pressure and enable further megakernel optimizations. Can you extend the evaluation to these GPUs?
6 comments
AuthorsOpenReview ↗

Response for Reviewer LGHx(Part #1)

We thank the reviewer for the detailed feedback and constructive suggestions. Below we address the concerns point-by-point.

W1:Too reliant on AMK; more an AMKBench than a general benchmark.

Strong reliance on AutoMegaKernel (AMK) limits the benchmark's scope, making it more like an AMKBench than a general megakernel benchmark. AMK’s approach is not an established best way to write megakernels.

We agree that AMK is not an established "best" paradigm for megakernels, but MegaKernelBench is not equivalent to a benchmark designed specifically for AMK. The design intent of this work is to evaluate and distinguish the operator-implementation, schedule-design, joint-optimization, and runtime-construction capabilities of LLMs/agents when building megakernels, and to evaluate the ability of other generation paradigms to construct megakernels. AMK provides an executable, verifiable infrastructure for the decomposed evaluation (Tasks A–C); it is not this paper's sole formal definition of a megakernel. Indeed, precisely because the design methods and implementation forms of current megakernels remain unsettled, establishing a benchmark with unified task boundaries, correctness requirements, and performance metrics is all the more valuable: it enables different generation methods to be compared systematically, and provides a common research foundation for the subsequent definition, design methodology, and evaluation criteria of megakernels.

We also need to clarify the actual role of AMK in the benchmark. Tasks A–C do reuse AMK's data structures, schedule representation, and the runtime that we improved on top of it, treating it as a representative controlled experimental environment. This design does not presuppose that AMK is the best paradigm; rather, it serves to control variables: Task A fixes the schedule and the runtime to evaluate operator generation, Task B fixes the operators and the runtime to evaluate schedule generation, and Task C opens both the operators and the schedule to evaluate their joint optimization. Without fixing this common infrastructure, the effects of operators, scheduling, and the runtime would be confounded, and the benchmark could only compare how fast the final systems are, without being able to determine which part of the construction capability an LLM/agent actually possesses. This hierarchical diagnostic capability is precisely the core value of this benchmark relative to a single end-to-end performance test.

At the same time, AMK does not define the coverage boundary of the entire benchmark. MegaKernelBench evaluates different levels, from component generation to system construction, through four complementary task types; among them, Task D provides no pre-built runtime and does not require adopting AMK's task graph, schedule format, or search strategy. For methods in which operators and scheduling are tightly coupled, that adopt different execution representations such as MPK, or that design their own data structures and runtimes, as long as they satisfy the unified functional and single-work-kernel constraints, they can all receive end-to-end evaluation through Task D. Therefore, Tasks A–C are tasks for capability attribution of LLMs/agents: they evaluate the ability to independently generate either the operator side or the scheduling side, to jointly design and optimize both, and they probe the degree of coupling between the two sides in a megakernel; although the performance of their outputs is to some extent limited by the AMK runtime, this is irrelevant to the capabilities we evaluate and to the questions we investigate. Task D, in turn, is the open track that accommodates different megakernel generation paradigms; the two together constitute this benchmark, rather than taking AMK as the sole legitimate definition of a megakernel.

In the revision, we will more explicitly distinguish "the execution infrastructure inherited from AMK" from "the hierarchical tasks and unified evaluation design newly introduced in this paper"; we will also clarify that the conclusions of Tasks A–C apply to this representative static-scheduling environment. Although this cannot cover all possible megakernel paradigms, we believe this limitation does not diminish the value of the benchmark: it provides component-level generation tasks for attributing the generation capabilities of LLMs/agents, and through Task D we also retain an open evaluation track for new generation paradigms.

AuthorsOpenReview ↗

Response for Reviewer LGHx(Part #2)

W2: Only full-model megakernels are included; they may not always be optimal.

The benchmark only includes full-model megakernels, which are difficult for agents to generate directly. Layer- and subgraph-level megakernels are also relavant, and full-model megakernels may not always be optimal for low-latency LLM decoding.

This comment in fact involves two distinct questions: which fusion granularity is optimal in a given deployment, and how to evaluate the ability of LLMs/agents to build megakernels; that the former has no unified answer does not mean that the latter must cover all fusion granularities simultaneously in order to be valid. The goal of MegaKernelBench is not to select a universally optimal scheme among full-model, layer-level, and subgraph-level implementations, but to establish a unified, reproducible capability evaluation for megakernels as an important system-construction target. The value of a benchmark does not depend on whether its subject of evaluation can replace all other technical approaches, but on whether it can continuously and comparably measure the capabilities of methods around a well-defined target. Therefore, the fact that full-model megakernels do not necessarily achieve the lowest latency under all models, batch sizes, and hardware conditions does not diminish the necessity of evaluating the ability to generate them automatically.

We deliberately chose full-model tasks, because they expose the system-level problems in megakernel generation more completely. A candidate method must not only implement the local computations, but also handle task partitioning and scheduling across multiple operators over multiple layers, dependency and synchronization, buffer management, resource allocation, and the correctness of continuous decoding. By contrast, layer- or subgraph-level tasks mainly cover partial combinations of these capabilities, and can hardly answer by themselves whether a method can organize an entire model into a correct and efficient megakernel decoding implementation. You point out that full-model megakernels are harder for agents to generate directly; we believe this difficulty indicates precisely that the task is suitable as a test of the capability ceiling and of system-construction ability, rather than being a reason to avoid it.

At the same time, MegaKernelBench does not merely offer a single full-model end-to-end task. Tasks A–C open up operators, scheduling, and their joint generation respectively, to determine which part a failure or performance bottleneck comes from; Task D then further evaluates the ability to build a complete system without relying on a pre-built runtime. This hierarchical design is a deliberate design choice made in response to the difficulty of full-model tasks: it preserves the system complexity required by a complete megakernel, while making it possible to analyze the capability boundaries of LLMs/agents at different construction stages. Therefore, the contribution of this work is to establish a decomposable, attributable evaluation methodology around this target.

While there is not yet a consensus on the definition, implementation form, and design methodology of megakernels, such a benchmark can further provide common task boundaries, correctness requirements, and performance metrics for different methods, and drive subsequent design evolution through comparable results. The layer- and subgraph-level tasks you mention are also valuable for other forms of megakernel research, and we will consider extending the benchmark with them in a subsequent version; however, they answer additional research questions and are not prerequisites for the validity of the current full-model tasks and their core conclusions.

AuthorsOpenReview ↗

Response for Reviewer LGHx(Part #3)

Q1: Tasks A–C are built on AMK, which is not an established general approach.

Tasks A-C are built heavily on AMK, reusing AMK’s data structure, scheduling and runtime to benchmark the operator and scheudling components separately. However, AMK’s approach is not established to represent megakernel authoring in general. Mirage Persistent Kernels already follows a different paradigm with good performance. How best to design a megakernel is still an open problem.

MegaKernelBench does not presuppose AMK as a general or optimal scheme; instead, it decomposes and evaluates the operator- and schedule-generation capabilities of LLMs/agents in a representative controlled environment through Tasks A–C, and openly evaluates other, different complete megakernel construction paradigms, including MPK, through Task D. The goal of this work is not to prove that AMK is superior to MPK or other schemes, but to evaluate what operator-implementation, schedule-design, joint-optimization, and runtime-construction capabilities LLMs/agents respectively possess when building megakernels. A megakernel integrates model computation and scheduling into a single kernel, posing system-level challenges to automatic generation methods that are distinct from single-operator generation; whichever design paradigm ultimately proves better, unified task boundaries, correctness requirements, and performance metrics are needed to compare generation methods, and on this basis to progressively advance the definition, design methodology, and evaluation criteria of megakernels.

We need to state the relationship between AMK and this work more precisely: Tasks A–C do reuse AMK's data structures, schedule-representation format, and the runtime that we improved on top of it. This choice is deliberate experimental control, not a definition of AMK as the sole form of a general megakernel. A megakernel consists of two parts, operators and scheduling, and is executed through a runtime adapted to its operators and scheduling. To evaluate the operator- and schedule-generation capabilities of LLMs/agents separately, a benchmark must provide a common, executable, verifiable infrastructure in which one of the two sides can be held fixed; otherwise, different submissions would change operators, scheduling, and the runtime at the same time, and the final performance differences could not be attributed. AMK provides such a representative controlled environment, enabling Task A, Task B, and Task C to separately measure the single-side generation and joint-design capabilities of LLMs/agents. The new contribution of this paper is the hierarchical tasks, unified validation, and capability-attribution methodology established around this environment for LLMs/agents; it does not claim that AMK's runtime or scheduling mechanism itself is a general-purpose scheme.

Therefore, the role of Tasks A–C is to provide a benchmark for analyzing the ability of LLMs/agents to generate each part of a megakernel, while the role of Task D is to provide a benchmark for analyzing the ability of LLMs/agents and other, different paradigms to generate a complete megakernel. Task D does not provide AMK's reference schedule, data structures, or runtime; candidate methods may design their own internal representations, operator–schedule coupling, and runtime. The different paradigm demonstrated by MPK further justifies the necessity of opening Task D: as long as MPK or another generation paradigm adapts to the unified functional interface and satisfies the structural constraints of the task, it can retain its own design and receive end-to-end evaluation in Task D, without first converting to AMK's format.

Experiments show that on the track where MPK currently runs (Llama-3.2-1B), its measured throughput on an H100 is approximately 725 tokens/s, comparable to the best agent-generated megakernel submission in Task D (768 tokens/s). This confirms the value of opening Task D: complete megakernel implementations of different paradigms can obtain directly comparable measured numbers under unified functional constraints and a unified timing protocol, without first being unified into one particular internal representation.

MegaKernelBench thus adopts a design of "generation-capability decomposition + open-paradigm construction": the former answers what LLMs/agents can do at each construction stage of a megakernel, and the latter answers whether LLMs/agents and other, different complete paradigms can ultimately generate correct and efficient systems. In the revision, we will explicitly distinguish the execution infrastructure inherited from AMK from the benchmark design newly introduced in this paper.

Reviewer_XVvnOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MegaKernelBench, a benchmark for evaluating megakernel generation and optimization. It defines four tasks: operator generation, scheduling generation, joint operator and scheduling generation, and generation without a prebuilt runtime. The problem is practically relevant, and the paper is generally well organized, with clearly defined tasks and useful experimental insights. However, the manuscript does not yet clearly establish its contribution beyond prior work. Concerns also remain about the reliability of the correctness criteria and the consistency of the reported experimental statistics.

Critical strengths
  1. The paper examines operator implementations, task scheduling, and their interactions, helping assess the practical capabilities of automatic generation methods in complete GPU inference systems.

  2. Tasks A–D clearly distinguish the scope of generation across operators, scheduling, and the runtime. This design provides a structured basis for assessing each method’s strengths and limitations across these components.

  3. The paper analyzes why improvements in individual operator performance do not necessarily translate into comparable end-to-end gains. These observations provide useful insights into bottlenecks in automated megakernel optimization.

Critical weaknesses
  1. Substantial overlap with existing evaluation protocols. KernelBench and FlashInfer-Bench already combine correctness and performance evaluation, while AutoMegaKernel provides operator and scheduling optimization with structural validation. The additional diagnostic value of Tasks A–D remains unclear. The authors should distinguish inherited mechanisms from new designs and demonstrate what existing evaluations miss.

  2. Insufficient validation of the token-level tolerance rule. Accepting non-top-1 tokens within a logit threshold may conceal computational errors. The authors should provide threshold calibration, sensitivity analysis, and tests using known incorrect implementations. They should also clarify how reference trajectories and KV cache states are handled after token divergence.

  3. Inconsistent correctness statistics. Section 4.3.1 reports an 80% static-validation failure rate for Task D, whereas Table 10 reports 70%, alongside 20% compilation failures. These figures should be reconciled, with any differences in reporting conventions explicitly explained.

Questions for authors
  1. Can the authors provide experiments or examples showing that Tasks A–D reveal capability differences or failure modes missed by existing evaluations in KernelBench, FlashInfer-Bench, or AutoMegaKernel?

  2. What justifies the top-5 criterion and logit-gap threshold for accepting non-top-1 tokens? Have the authors conducted sensitivity analyses and tested known incorrect implementations to assess whether this rule admits substantive computational errors?

  3. Section 4.3.1 reports an 80% static-validation failure rate for Task D, whereas Table 10 reports 70%. Do these figures reflect different experimental runs or reporting conventions?

5 comments
AuthorsOpenReview ↗

Response for Reviewer XVvn(Part #1)

We thank the reviewer for the detailed feedback and constructive suggestions. Below we address the concerns point-by-point.

W1: Substantial overlap with existing evaluation protocols.

Substantial overlap with existing evaluation protocols. KernelBench and FlashInfer-Bench already combine correctness and performance evaluation, while AutoMegaKernel provides operator and scheduling optimization with structural validation. The additional diagnostic value of Tasks A–D remains unclear. The authors should distinguish inherited mechanisms from new designs and demonstrate what existing evaluations miss.

The added value of MegaKernelBench does not lie in once again combining correctness and performance testing, but in treating the megakernel as a system-generation object composed jointly of operator implementation, device-side cross-operator scheduling, and runtime logic, and, through the task hierarchy of Tasks A–D, diagnosing which parts of the construction capability a method actually possesses and whether local optimizations translate into full decoding gains.

A megakernel differs from an ordinary kernel by integrating multiple operators of a complete model and their scheduling logic into a single work kernel, rather than implementing one computational primitive. Its correctness and performance depend jointly on operator code quality, cross-operator dependencies, task partitioning and SM mapping, synchronization, buffer management, and runtime state maintenance; in essence, it is an inference system. A megakernel reduces kernel-launch and intermediate scheduling overhead and opens up cross-operator data reuse and joint computation–scheduling optimization; yet precisely because these components are highly interrelated, generating high-performance operators does not mean being able to construct a correct and efficient complete megakernel. What is currently missing is an evaluation of these part-wise construction capabilities; establishing decomposable, comparable standards for them is in itself a necessary foundation for advancing megakernel generation methods.

MegaKernelBench evaluates a different task from existing benchmarks. Existing benchmarks mainly evaluate single-kernel generation tasks rather than megakernel generation tasks, and naturally cannot evaluate mechanisms specific to megakernels. KernelBench, for example, evaluates implementing and optimizing a given computational task; even a large computational structure is still a computational implementation, without the device-side scheduling and runtime construction of a complete megakernel. FlashInfer-Bench evaluates generating inference operators and verifying their effect inside an existing inference system, but "replacing a component in an existing system" differs from "generating the entire inference system itself." AutoMegaKernel is a specific construction method rather than a general evaluation benchmark, and cannot evaluate other paradigms such as compiler-generated megakernels. We reuse its data structures, schedule representation, improved runtime, and structural-check mechanisms as the experimental basis of Tasks A–C, and do not claim these inherited mechanisms as contributions of this paper; the new contributions are instead the systematic control over which components of the full-model megakernel generation pipeline are fixed and which are generated by the candidate method, and the design of a task that opens up megakernel generation paradigms.

Specifically, Task A fixes the schedule and the runtime to evaluate operator generation, Task B fixes the operators and the runtime to evaluate schedule generation, Task C opens both to evaluate joint design, and Task D removes the pre-built runtime to further evaluate the overall construction capability spanning operators, scheduling, and the runtime. This design resembles a controlled ablation in systems research: it not only gives an overall verdict of "correct or not, faster or not," but can also locate whether a performance gain or a failure comes from the operators, the schedule, the coupling of the two, or runtime construction. The difference between local optimization and end-to-end gains, the joint-optimization cases, and the failure phenomena of Task D reported in the paper are precisely the diagnostic information that existing evaluations can hardly provide but Tasks A–D can reveal. The existing Appendix A already lists the evaluation goals, the information provided, the submission contents, and the speedup baselines of Tasks A–D, and the concrete inheritance relation of Tasks A–C to the runtime and data structures is already explained in Section 3 of the main text. In the revision, we will strengthen the cross-references among these existing parts, making the inherited mechanisms and the newly added evaluation design of MegaKernelBench clearer.

AuthorsOpenReview ↗

Response for Reviewer XVvn(Part #2)

Q1: Missed capability differences or failure modes need evidence.

Can the authors provide experiments or examples showing that Tasks A–D reveal capability differences or failure modes missed by existing evaluations in KernelBench, FlashInfer-Bench, or AutoMegaKernel?

Three sets of results in the paper demonstrate this concretely:

  1. Generating high-performance operators does not imply being able to optimize a complete megakernel. In Task A, the generated GEMV kernels achieve an average speedup of 1.75× over the reference implementation and the attention kernels reach up to 4.45×, yet the end-to-end speedup of the complete megakernel is only 1.05×. With the schedule fixed, Task A further shows that more than 70% of the latency comes from waiting caused by dependency and resource constraints; faster local computation therefore did not change the critical path. KernelBench can identify whether these operators themselves are correct and fast, and FlashInfer-Bench can observe the final effect after generated operators are integrated into an existing system, but neither makes the device-side scheduling of the megakernel an independent generation axis, so it is difficult for them to determine, through controlled comparison, whether limited gains come from operator quality, the fixed schedule, or a mismatch between the two. What the comparison between Task A and Tasks B/C reveals is precisely the difference between "local optimization capability" and "the capability to convert it into system-level gains."

  2. Possessing operator-optimization and schedule-optimization capabilities separately does not imply possessing the capability to jointly design the two. K-split GEMV requires the operator to produce partial results across SMs while requiring the schedule to modify task partitioning, SM assignment, dependencies, and the reduction flow, so neither side can be changed alone. Another fusion case changes operator boundaries and the dependency graph at the same time, reducing the dependency levels per layer from 13 to 7 and those of the full model from 211 to 115; although the number of tasks decreases by only 3%, end-to-end performance still improves by about 10%. In addition, directly combining the independent optimizations from Task A and Task B yields only 1.26×, about 4% lower than the expected product of gains, further showing that two local optima do not automatically constitute a global system optimum. The task boundaries of A/B/C make this interaction explicitly measurable: A/B answer "can the current component be optimized with the other side fixed," and C answers "can the two sides be co-designed according to the new execution behavior." Evaluating only generated operators or final system results cannot distinguish these two capabilities.

  3. Being able to generate a megakernel inside an existing runtime does not imply being able to construct a runtime. In the same GLM-5.3 + Codex case study, all 10 Task C submissions passed, whereas only 1/10 passed in Task D; among the remaining submissions, 2/10 failed to compile and 7/10 failed at the static structural-validation stage. This contrast separates failures in numerical computation or schedule generation from errors at the runtime level in state management, synchronization, dependencies, and execution structure. Neither KernelBench nor FlashInfer-Bench requires generating a complete decode runtime; AutoMegaKernel, for its part, already provides the runtime, data structures, and validation mechanisms, so in its original evaluation, "knowing how to use the existing infrastructure" and "being able to construct an execution system by oneself" were not two independent capabilities. The difference between Task C and Task D directly exposes this infrastructure dependence, and the open interface of Task D also allows the evaluation of other megakernel construction paradigms that do not depend on the AutoMegaKernel runtime.

Tasks A–D therefore do not simply cut the same evaluation into four pieces; they form a set of system-level controlled ablations: A/B isolate component capabilities, C measures cross-component coordination, and D measures system-construction capability without a pre-built runtime. KernelBench, FlashInfer-Bench, and AutoMegaKernel respectively provide valuable operator evaluation, system-integration evaluation, or a concrete generation method, but none of them organizes the three kinds of capabilities above into dimensions comparable across methods under a unified protocol.

AuthorsOpenReview ↗

Response for Reviewer XVvn(Part #3)

W2 & Q2: The token-level tolerance rule may conceal computational errors.

Insufficient validation of the token-level tolerance rule. Accepting non-top-1 tokens within a logit threshold may conceal computational errors. The authors should provide threshold calibration, sensitivity analysis, and tests using known incorrect implementations. They should also clarify how reference trajectories and KV cache states are handled after token divergence.

What justifies the top-5 criterion and logit-gap threshold for accepting non-top-1 tokens? Have the authors conducted sensitivity analyses and tested known incorrect implementations to assess whether this rule admits substantive computational errors?

W2 & Q2(1/2):

The token tolerance rule of this paper does not replace numerical correctness with loose token matching; rather, after the complete logits have passed an element-wise error check, it specifically handles the autoregressive trajectory divergence caused by near-tie tokens in BF16 inference. The rule accepts an alternative equivalent trajectory only when the first top-1 disagreement is a numerical near-tie; it does not accept arbitrary non-top-1 outputs.

This phenomenon arises from the combined effect of finite-precision computation and the argmax operation. Different but numerically correct kernels may adopt different fusion schemes, reduction orders, and accumulation paths; under BF16 precision, these differences introduce very small perturbations into the logits. At most positions the top-1 candidate is clearly separated from the other candidates, so the perturbation does not change the output; but when the logits of the top-1 and top-2 candidates are very close, a tiny perturbation can swap their order even though the logits as a whole remain within the correct numerical error range. The longer the decoded sequence, the higher the probability of encountering such near-tie positions; requiring the top-1 token of every step of a long sequence to exactly match the reference implementation is therefore not a robust correctness criterion for BF16.

More importantly, after the first token divergence in autoregressive decoding, the candidate token and the reference token each become the input of the next step and are written into their respective KV caches, so the subsequent logits no longer correspond to the same conditional trajectory, and exact token-by-token equality can no longer serve as a necessary condition for numerical correctness. We therefore compare step by step from the shared prompt and the initial KV cache up to the first divergence; if the candidate token is not in the reference top-5, or its logit difference from the reference top-1 exceeds the threshold, the submission is immediately judged incorrect. Only when the first divergence satisfies these two near-tie conditions is it treated as a floating-point-equivalent branch; thereafter the candidate and the reference maintain their own token and KV-cache states, and token-by-token equality between the two different trajectories is no longer used to judge whether the computation is correct.

The two criteria we use for judging correctness each rule out a different risk. top-5 alone is not sufficient, because a high rank does not mean that the logit is close to the top-1, and the second-ranked candidate may still have a significant probability gap; we therefore additionally require that the logit difference between the candidate token and the reference top-1 not exceed the threshold. Conversely, the logit gap alone is not sufficient to explicitly bound the candidate set, so the top-5 constrains the acceptable candidates to a small number. The actual verdict starts from the shared prompt and the initial KV cache and checks these two conditions at the first disagreement: a candidate token outside the top-5, or a logit gap exceeding the threshold, causes an immediate failure. Consequently, an incorrect implementation with a systematically biased output distribution cannot pass merely because some token happens to fall into the top-5.

This way of handling the first divergence via a near-tie candidate set is also consistent with correctness testing in mainstream inference systems. For example, vLLM's official check_logprobs_close, when the first token disagrees, checks whether the two tokens are each within the top-N candidates of the other's logits; if so, it treats them as numerically near-tied and stops comparing the already-diverged logprobs that follow. MegaKernelBench goes one step further by simultaneously requiring "reference top-5" and an explicit logit gap, bounding the acceptable range along two dimensions: rank and numerical distance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.