- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents EdgeKernelBench, a benchmark and framework for LLM-based kernel generation on edge devices. It includes tasks from basic operators to small model graphs, with a shared intermediate representation and TVM support for different backends. The paper also introduces LEKA, a layered agent system that organizes optimization along progressive IR lowering. Each stage preserves earlier optimizations and exposes more low-level details to the agent at that level, while device feedback guides further changes. Experiments across three edge backends show performance gains over one-shot generation and the Claude Code baseline.
Critical strengths
- The benchmark provides useful coverage of edge kernel generation. It covers three different backends and includes tasks from basic operators to small model graphs. This makes it a useful testbed for studying both local kernel optimization and larger graph-level decisions on edge devices, which is a substantive contribution of the paper.
- LEKA shows a clear practical gain under matched evaluation settings. With the same base model and task time budgets, LEKA increases the percentage of tasks that outperform TVM from 26.1% for Claude Code to 43.5% on 69 task instances. This supports the practical value of the full system.
Critical weaknesses
- Some key parts of the method are not clearly explained. These include the control of cross-level optimization and the use of memory budgets. This makes it difficult to understand how the method works in practice.
- The paper does not establish the value of each component separately. The results show gains for the complete system. However, they do not show which components produce these gains or how much each component contributes.
Questions for authors
-
How does LEKA revise higher-level decisions?
The appendices describe cross-level requests and rebuilding later stages. Could the authors show, using an existing run, how LEKA decides which level and earlier candidate to revisit?
-
How does LEKA control memory use?
Do the experiments use explicit memory budgets? If yes, what memory does each budget include? How does LEKA check memory use and handle programs that exceed the budget? Do the existing results show a memory problem in a baseline method that LEKA solves?
-
What is the contribution of each main component?
Could you provide ablation results for layer-specific agents, the candidate tree, and hardware feedback? How does removing each component affect performance? Please use the same base model and time budget for each comparison.
4 comments
W1.2 & Q2.2: Handling Budget Violations.
We thank the reviewer for requesting concrete examples. LEKA addresses excess temporary memory through graph-level restructuring and kernel-level workspace reduction. The following cases illustrate how these changes bring implementations within the assigned budgets.
For the convolution, GELU, and global average pooling task on GPU/CUDA, the tuned TVM baseline already uses fusion. LEKA changes the fusion boundary by combining bias addition, GELU, and pooling in the kernel following convolution. This kernel directly produces the pooled output without materializing the full activation. The peak temporary memory is 35.04 MiB for LEKA, compared with 70.08 MiB for TVM. LEKA satisfies the 48 MiB budget that TVM exceeds.
For the bottleneck residual block on CPU/LLVM, LEKA folds batch normalization into convolution parameters and fuses bias, activation, and residual computations. Its kernels also eliminate redundant input copies in the convolutions. The largest dynamic workspace request drops from 2.29 MiB in the same-campaign fixed TVM reference to 0.60 MiB in LEKA, a 73.81% reduction. Including live intermediate tensors, LEKA's peak temporary memory is 1.74 MiB, within the 2 MiB budget.
These cases show how cross-layer optimization can make over-budget implementations fit within the assigned memory budget. Detailed optimization steps are provided in Appendix E.3.3 (Memory Optimization Case Studies) of the revised manuscript.
We appreciate your suggestion and hope the above clarification addresses your concern.
W2 & Q3: Contributions of the individual components.
Thank you for this valuable suggestion. We have added a component ablation study to examine the contributions of hierarchical specialization, the candidate tree, and hardware feedback. All configurations use the same base model and task-level time budget. The table reports GM speedup relative to Claude Code. Parentheses give each method's GM minus LEKA's GM.
| Backend | LEKA | No HW Feedback | No Hierarchy | No Tree | Claude Code |
|---|---|---|---|---|---|
| CPU/LLVM | 10.13 | 6.93 (−3.20) | 2.58 (−7.55) | 3.78 (−6.35) | 1.00 (−9.13) |
| GPU/OpenCL | 2.58 | 1.86 (−0.72) | 1.65 (−0.93) | 1.73 (−0.85) | 1.00 (−1.58) |
| GPU/CUDA | 1.81 | 1.60 (−0.21) | 1.20 (−0.61) | 1.48 (−0.33) | 1.00 (−0.81) |
Removing the hierarchy produces the largest GM drop on all three backends. This suggests that layered decomposition reduces task complexity and enables more thorough optimization at each IR level. The candidate tree and hardware feedback provide further gains through search history and device guidance, with the smallest drop observed when hardware feedback is removed. These ablations support the individual contributions of all three components to LEKA's performance.
Your feedback has been important in making our manuscript more convincing. The detailed ablation results can be found in the revised PDF (Section 5.3.3, COMPONENT ABLATION, Table 3).
Overall, EdgeKernelBench provides a systematic benchmark for LLM-based kernel generation across heterogeneous edge platforms with fragmented software stacks and tight memory budgets. We also introduce LEKA, a layered multi-agent framework for effective edge-kernel optimization. We believe this work will inspire further research on LLM-based kernel generation for edge devices.
We welcome any further questions and remain happy to discuss. Thank you very much for the detailed comments!
Thank you for the detailed responses and additional experiments. The component ablation helps clarify the contributions of the layered design, candidate tree, and hardware feedback. I also appreciate the added memory evaluation and the concrete examples illustrating cross-layer optimization.
These additions address most of my concerns and make the design and practical value of the system clearer.
Thank you very much for your positive feedback . Please let us know if you have any remaining concerns; we would be happy to address them. If you have no further concerns, could you please gently consider raising the score as a recognition to our rebuttal efforts?
We sincerely thank the reviewer for the constructive feedback and for recognizing EdgeKernelBench as a useful testbed for edge kernel generation, as well as the practical value of LEKA. Below, we respond to each question in turn. In the revised manuscript, we have marked all changes in blue; in this response, we have also highlighted key points for ease of reading.
We thank the reviewer for the opportunity to clarify how cross-layer requests guide optimization in LEKA. LEKA determines which IR level to revisit from the specific constraints identified during lower-level optimization.
On the CPU/LLVM task combining depthwise and pointwise convolution, L1 first improves the convolution kernels through parallelization and vectorization. It then attempts to fuse the pointwise convolution with bias addition to eliminate a separate elementwise pass. However, a bias reshape separates these operations in the current graph, preventing them from satisfying L1's requirement for adjacent fusion bindings. Since this constraint comes from the graph's operation order, L1 asks L0 to prepare the bias before the pointwise convolution and leave the convolution and addition adjacent. L0 incorporates this request into its planned NHWC layout rewrite and adjusts the operation order accordingly. The resulting graph candidate passes correctness checks and achieves a 1.28× speedup over the baseline for the preceding graph.
We have added the case study “Cross-layer requests guide graph refinement” to Appendix E.3.2 to show how a constraint identified at a lower level triggers reconsideration of the higher-level graph and how L0 responds to the request. Please see the text marked in blue in the revised PDF for details. Thank you for your contribution to improving our revised manuscript.
We thank the reviewer for this suggestion. Our evaluation uses explicit peak temporary memory budgets. We have clarified the metric in Section 5.1 and added a memory-budget evaluation in Section 5.3.4 (Table 4) of the revised manuscript.
The memory budget satisfaction rate measures the fraction of tasks whose peak temporary memory usage stays within the assigned budget. This peak covers simultaneously live intermediate tensors and dynamic operator workspaces during one inference call. Memory accounting follows their allocation lifetimes and storage reuse in host or device global memory.
We configure budgets according to the edge application scenario, task, input shape, and target backend. All methods share the same budget for each configuration, with detailed settings provided in Appendix B.5. Section 5.3.4 (Table 4) reports the memory budget satisfaction rates across the three backends.
Memory budget satisfaction rate (%)
The experiments therefore assess implementations against explicit memory budgets tailored to each task configuration.
We appreciate your suggestion and hope the above clarification addresses your concern.