- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
MegaKernelBench is a system-level benchmark for evaluating the generation and optimization of megakernels for LLM decode. It defines four tasks: operator optimization, scheduling optimization, joint kernel optimization and kernel scheduling using a provided runtime, and complete megakernel generation without a prebuilt runtime. Experiments evaluate 3 agent frameworks driven by 3 language models across 6 target models on NVIDIA A800 and H100 GPUs, checking functional correctness, structural validity, and decoding latency. The results show that agents can often generate correct implementations when a runtime is provided, but fewer than 10% of submissions succeed without one. The generated implementations remain substantially slower than Mirage Persistent Kernel (MPK) on the small models where it is evaluated.
Critical strengths
Megakernels are powerful ways to gain performance, so this is a timely benchmark. The paper identifies that generating efficient kernels for a neural network and organizing their execution within a megakernel are two separate problems. Task partitioning, dependencies, synchronization, and GPU resource allocation require evaluation beyond just an individual operator performance.
The four tasks separate operator generation, scheduling generation, joint optimization, and generation without a prebuilt runtime. This structure makes it easier to identify which capabilities an agent has and where it fails.
Although KernelBench Levels 3 and 4 cover full models, whole-model correctness and speedup do not by themselves isolate the challenges of megakernel execution. MegaKernelBench improves that by explicitly evaluating GPU-side scheduling and structural requirements, making it better suited to this use case.
The experiments show that substantial improvements in individual operators can yield only small end-to-end gains when scheduling remains fixed. This supports the motivation for evaluating operator and scheduling optimization together.
Critical weaknesses
The performance evaluation lacks competitive baselines. PyTorch eager is useful as a reference, but it is insufficient as the main performance baseline for an inference optimization benchmark. Speedups over eager execution do not establish practical competitiveness. What are the end-to-end speedups against TensorRT-LLM, torch.compile , SGLang, and vLLM under matched workloads and configurations, including appropriate use of CUDA Graphs? MPK is included for only a subset of models, where it substantially outperforms the agent-generated implementations.
The agent evaluation should include stronger contemporary models. The paper should include leading OpenAI and Claude models to assess whether the reported limitations persist with stronger coding agents (leading at the time of the submission like GPT-6 Astra). Comparisons should use comparable optimization budgets and tool access. Many of these conclusions would change.
The reference baseline needs more description. For Tasks A and B, the baseline is an agent-generated AutoMegaKernel implementation. Which agent and model produced it, with what optimization budget and selection procedure? These details matter because the reported speedups depend on the quality of that reference implementation.
Multi-GPU applicability remains unclear. The evaluation is limited to a single GPU, leaving communication, cross-device synchronization, and computation–communication overlap unexplored. Would this task decomposition and validation protocol apply to multi-GPU megakernels as well? Which additional scheduling and correctness constraints would be required, and would the benefits of joint operator–scheduling optimization still be there when communication of the multi-GPU setup becomes the bottleneck?
Questions for authors
The benchmark decomposition is promising, but competitive inference baselines and stronger agent comparisons are needed to establish its empirical contribution. In particular, I have the following questions for the authors:
Do the conclusions still hold when comparing against modern inference baselines such as TensorRT-LLM, SGLang, vLLM, and torch.compile, with appropriately configured CUDA Graphs? How do the generated megakernels compare with MPK across a broader range of models?
How was the reference implementation constructed and optimized? Which agent, model, optimization budget, and selection procedure were used, and how sensitive are the reported speedups to the quality of this reference?
Beyond single-step decoding latency, what throughput do the implementations achieve, measured in tokens per second across a few different batch sizes and context lengths?
5 comments
Response for Reviewer Zkv8(Part #2)
W1 & Q1 (2/2):
The supplementary experimental data are as follows (H100, batch=1, continuous 128-token greedy decoding, bf16; the table shows the speedup of each inference system relative to HF PyTorch eager):
| Model | eager (tps) | PyTorch + CUDA Graph | torch.compile (best variant) | vLLM | SGLang | TensorRT-LLM | best agent megakernel (task C) | best agent megakernel (task D) |
|---|---|---|---|---|---|---|---|---|
| Llama-3.2-1B | 101.0 | 3.79× | 5.85× (+graph) | 6.86× | 5.32× | 5.11× | 1.80× | 12.26× |
| Qwen3-0.6B | 48.4 | 4.86× | 11.18× (+graph) | 13.50× | 11.03× | 9.97× | 3.55× | 28.64× |
| gpt-oss-20b | 36.3 | ✗ | 0.90× (fullgraph=False) | 8.44× | 7.72× | 4.67× | 0.87× | ✗ |
| DeepSeek-V2-Lite | 19.4 | ✗ | ✗ | 14.82× | 11.76× | ✗ run failed | 12.75× | ✗ |
| GLM-4.7-Flash | 4.58 | ✗ | 0.93× (fullgraph=False) | 37.56× | ✗ unsupported | ✗ run failed | 2.50× | ✗ |
| Qwen3-30B-A3B | 7.96 | ✗ | 0.81× (fullgraph=False) | 27.17× | 26.37× | 20.43× | 3.85× | ✗ |
Four observations can be drawn from the table above:
(1) vLLM is the fastest system on all 4 MoE tracks, while on the 2 dense tracks the megakernels generated by agents in task D are faster;
(2) a "stronger PyTorch baseline" cannot be constructed on any of the MoE models. PyTorch's native acceleration paths (CUDA Graph capture, compile fullgraph) all fail structurally, and the fullgraph=False fallback is instead slower than eager, so for these tracks eager is already the ceiling natively attainable with PyTorch;
(3) the best agent submissions in Task C of this benchmark surpass only PyTorch eager on most tracks and still lag behind mainstream inference engines, matching mainstream inference engines only on DeepSeek-V2-Lite in performance, but the generation success rate of Task C on the MoE tracks is markedly higher than that of Task D.
(4) In Task D, agents are already able to independently implement megakernels that outperform mainstream inference engines on some structurally simple, small-scale dense models, but on MoE models, agents can hardly implement a functionally correct megakernel within the given time budget (90 min).
Response for Reviewer Zkv8(Part #3)
W2: Stronger models such as GPT-6 Astra should be included.
The agent evaluation should include stronger contemporary models. The paper should include leading OpenAI and Claude models to assess whether the reported limitations persist with stronger coding agents (leading at the time of the submission like GPT-6 Astra). Comparisons should use comparable optimization budgets and tool access. Many of these conclusions would change.
The existing experiments do not rely on weaker models to reach their conclusions. Under a limited experimental budget, we chose GPT-5.6-Sol, GLM-5.3, and the DeepSeek-V4 series, which cover currently representative strong coding models; although they are not SOTA models, they already possess strong code-generation and reasoning capabilities. The goal of this paper is to examine, through multiple representative models, the capabilities and common difficulties of current mainstream LLMs/agents on different megakernel generation tasks. To address your concern, we will additionally run experiments with Claude and GPT-6-Astra and include the results in the appendix of the final version. That said, the models adopted in this paper are already representative: GPT-5.6-Sol already outperforms the vast majority of mainstream LLMs, which also indicates that, for the vast majority of LLMs, megakernel generation tasks are extremely difficult — a conclusion that will not change by adding generation experiments with SOTA models.
Response for Reviewer Zkv8(Part #1)
We thank the reviewer for the detailed feedback and constructive suggestions. Below we address the concerns point-by-point.
W1 & Q1: No competitive baselines; MPK on only a subset of models.
W1 & Q1 (1/2):
We agree that mature inference engines and compilers are important references for gauging real-world deployment competitiveness, but they serve a different role from the baseline this paper uses for unified evaluation of generation capability; the primary goal of MegaKernelBench is to diagnose whether existing methods can generate correct and efficient megakernels, not to use speedups over PyTorch eager to claim superiority over production-grade inference systems. This paper therefore needs two kinds of complementary references at the same time: the unified baseline (PyTorch) for comparable capability evaluation across different models, tasks, and generation methods, and strong inference-engine/compiler baselines for locating how large the remaining performance gap between the generated results and mature deployment solutions is.
First, PyTorch eager is not the only performance baseline for the four task types. Tasks A/B use a fixed reference megakernel, directly measuring the improvement of the generated operators or schedules relative to the same system starting point; only the end-to-end Tasks C/D use native PyTorch as the unified reference. PyTorch was chosen because it can provide essentially consistent API/model semantics for models across hardware platforms, whereas TensorRT-LLM, vLLM, SGLang, and torch.compile do not support and optimize specific models, operators, hardware, and configurations in a fully consistent manner. These system baselines can therefore answer "do the generated implementations have deployment competitiveness," but they are not better suited than PyTorch to serve as the unified baseline shared by all models and generation methods. Nor does the choice of baseline change the correctness of the submissions, their absolute latencies, or the relative ordering of generation methods by latency within the same task.
Second, MPK appears on only a subset of models because we report results only on the models its current implementation can accommodate, not because we deliberately excluded unfavorable comparisons. More importantly, we do not regard MPK's clear lead as a result to be avoided: it directly shows that, within the existing coverage, dedicated compilation methods for megakernels still significantly outperform current agent-generated solutions, revealing a capability gap that agents have not yet closed in structural optimization and joint design. This is precisely the current state of affairs that the benchmark intends to measure, not a claim by this paper that the generated results have already reached deployment optimality.
Your suggestion can further supplement the dimension of deployment competitiveness. We have added TensorRT-LLM, torch.compile, SGLang, vLLM, and PyTorch + CUDA Graph baselines, reporting their absolute latencies under aligned models, input states, and timing scopes, while retaining PyTorch as the unified reference. This can more directly demonstrate the actual performance gap between the generated megakernels and mature compilers/inference engines.