acceptodds
Under review as a conference paper at ICLR 2027

EdgeKernelBench: Unified LLM-Driven Kernel Generation for Edge Backends

Abstract

Recent works show that LLMs can generate kernels with strong performance on server GPUs. Edge deployment presents distinct challenges because the hardware is heterogeneous, the software stacks are fragmented, and the resource budgets are tight. LLM kernel generation across diverse edge backends has yet to be studied systematically. We introduce EdgeKernelBench, which combines a unified IR deployment framework with a hierarchical benchmark for edge workloads. The unified IR captures reusable computation, scheduling, and memory decisions and connects to each target backend through a compiler interface. The benchmark spans progressively larger program scopes, from individual operators to compact model graphs, across representative edge application domains. We instantiate the design with TVM as a reference compiler and validate it across heterogeneous edge backends. With a single generation attempt, the best model achieves 64.5% Correct@1 but only 25.5% Fast\@1.0. This gap shows that functional correctness does not consistently translate into performance gains. We therefore introduce the Layered Edge Kernel Agent (LEKA), which organizes optimization through a layered IR. On the agent evaluation subset, LEKA achieves 43.5% Fast\@1.0, compared with 13.0% for direct generation and 26.1% for Claude Code. LEKA and Claude Code use the same base model and optimization budget. Together, EdgeKernelBench and LEKA support unified evaluation and layered optimization of kernels generated by LLMs across edge backends.

Primary area:
datasets and benchmarks
Keywords:
Kernel Generation, Edge Computing

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_RX2MOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents EdgeKernelBench, a benchmark and framework for LLM-based kernel generation on edge devices. It includes tasks from basic operators to small model graphs, with a shared intermediate representation and TVM support for different backends. The paper also introduces LEKA, a layered agent system that organizes optimization along progressive IR lowering. Each stage preserves earlier optimizations and exposes more low-level details to the agent at that level, while device feedback guides further changes. Experiments across three edge backends show performance gains over one-shot generation and the Claude Code baseline.

Critical strengths
  1. The benchmark provides useful coverage of edge kernel generation. It covers three different backends and includes tasks from basic operators to small model graphs. This makes it a useful testbed for studying both local kernel optimization and larger graph-level decisions on edge devices, which is a substantive contribution of the paper.
  2. LEKA shows a clear practical gain under matched evaluation settings. With the same base model and task time budgets, LEKA increases the percentage of tasks that outperform TVM from 26.1% for Claude Code to 43.5% on 69 task instances. This supports the practical value of the full system.
Critical weaknesses
  1. Some key parts of the method are not clearly explained. These include the control of cross-level optimization and the use of memory budgets. This makes it difficult to understand how the method works in practice.
  2. The paper does not establish the value of each component separately. The results show gains for the complete system. However, they do not show which components produce these gains or how much each component contributes.
Questions for authors
  1. How does LEKA revise higher-level decisions?

    The appendices describe cross-level requests and rebuilding later stages. Could the authors show, using an existing run, how LEKA decides which level and earlier candidate to revisit?

  2. How does LEKA control memory use?

    Do the experiments use explicit memory budgets? If yes, what memory does each budget include? How does LEKA check memory use and handle programs that exceed the budget? Do the existing results show a memory problem in a baseline method that LEKA solves?

  3. What is the contribution of each main component?

    Could you provide ablation results for layer-specific agents, the candidate tree, and hardware feedback? How does removing each component affect performance? Please use the same base model and time budget for each comparison.

4 comments

We sincerely thank the reviewer for the constructive feedback and for recognizing EdgeKernelBench as a useful testbed for edge kernel generation, as well as the practical value of LEKA. Below, we respond to each question in turn. In the revised manuscript, we have marked all changes in blue; in this response, we have also highlighted key points for ease of reading.


W1.1 & Q1: Revisiting higher-level decisions and earlier candidates.

We thank the reviewer for the opportunity to clarify how cross-layer requests guide optimization in LEKA. LEKA determines which IR level to revisit from the specific constraints identified during lower-level optimization.

On the CPU/LLVM task combining depthwise and pointwise convolution, L1 first improves the convolution kernels through parallelization and vectorization. It then attempts to fuse the pointwise convolution with bias addition to eliminate a separate elementwise pass. However, a bias reshape separates these operations in the current graph, preventing them from satisfying L1's requirement for adjacent fusion bindings. Since this constraint comes from the graph's operation order, L1 asks L0 to prepare the bias before the pointwise convolution and leave the convolution and addition adjacent. L0 incorporates this request into its planned NHWC layout rewrite and adjusts the operation order accordingly. The resulting graph candidate passes correctness checks and achieves a 1.28× speedup over the baseline for the preceding graph.

We have added the case study “Cross-layer requests guide graph refinement” to Appendix E.3.2 to show how a constraint identified at a lower level triggers reconsideration of the higher-level graph and how L0 responds to the request. Please see the text marked in blue in the revised PDF for details. Thank you for your contribution to improving our revised manuscript.


W1.2 & Q2.1: Memory Budgets.

We thank the reviewer for this suggestion. Our evaluation uses explicit peak temporary memory budgets. We have clarified the metric in Section 5.1 and added a memory-budget evaluation in Section 5.3.4 (Table 4) of the revised manuscript.

The memory budget satisfaction rate measures the fraction of tasks whose peak temporary memory usage stays within the assigned budget. This peak covers simultaneously live intermediate tensors and dynamic operator workspaces during one inference call. Memory accounting follows their allocation lifetimes and storage reuse in host or device global memory.

We configure budgets according to the edge application scenario, task, input shape, and target backend. All methods share the same budget for each configuration, with detailed settings provided in Appendix B.5. Section 5.3.4 (Table 4) reports the memory budget satisfaction rates across the three backends.

Memory budget satisfaction rate (%)

BackendLEKAClaude CodeTVM
CPU/LLVM82.6182.6191.30
GPU/OpenCL91.3082.6195.65
GPU/CUDA91.3082.6186.96

The experiments therefore assess implementations against explicit memory budgets tailored to each task configuration.

We appreciate your suggestion and hope the above clarification addresses your concern.

W1.2 & Q2.2: Handling Budget Violations.

We thank the reviewer for requesting concrete examples. LEKA addresses excess temporary memory through graph-level restructuring and kernel-level workspace reduction. The following cases illustrate how these changes bring implementations within the assigned budgets.

For the convolution, GELU, and global average pooling task on GPU/CUDA, the tuned TVM baseline already uses fusion. LEKA changes the fusion boundary by combining bias addition, GELU, and pooling in the kernel following convolution. This kernel directly produces the pooled output without materializing the full activation. The peak temporary memory is 35.04 MiB for LEKA, compared with 70.08 MiB for TVM. LEKA satisfies the 48 MiB budget that TVM exceeds.

For the bottleneck residual block on CPU/LLVM, LEKA folds batch normalization into convolution parameters and fuses bias, activation, and residual computations. Its kernels also eliminate redundant input copies in the 1×11\times1 convolutions. The largest dynamic workspace request drops from 2.29 MiB in the same-campaign fixed TVM reference to 0.60 MiB in LEKA, a 73.81% reduction. Including live intermediate tensors, LEKA's peak temporary memory is 1.74 MiB, within the 2 MiB budget.

These cases show how cross-layer optimization can make over-budget implementations fit within the assigned memory budget. Detailed optimization steps are provided in Appendix E.3.3 (Memory Optimization Case Studies) of the revised manuscript.

We appreciate your suggestion and hope the above clarification addresses your concern.


W2 & Q3: Contributions of the individual components.

Thank you for this valuable suggestion. We have added a component ablation study to examine the contributions of hierarchical specialization, the candidate tree, and hardware feedback. All configurations use the same base model and task-level time budget. The table reports GM speedup relative to Claude Code. Parentheses give each method's GM minus LEKA's GM.

BackendLEKANo HW FeedbackNo HierarchyNo TreeClaude Code
CPU/LLVM10.136.93 (−3.20)2.58 (−7.55)3.78 (−6.35)1.00 (−9.13)
GPU/OpenCL2.581.86 (−0.72)1.65 (−0.93)1.73 (−0.85)1.00 (−1.58)
GPU/CUDA1.811.60 (−0.21)1.20 (−0.61)1.48 (−0.33)1.00 (−0.81)

Removing the hierarchy produces the largest GM drop on all three backends. This suggests that layered decomposition reduces task complexity and enables more thorough optimization at each IR level. The candidate tree and hardware feedback provide further gains through search history and device guidance, with the smallest drop observed when hardware feedback is removed. These ablations support the individual contributions of all three components to LEKA's performance.

Your feedback has been important in making our manuscript more convincing. The detailed ablation results can be found in the revised PDF (Section 5.3.3, COMPONENT ABLATION, Table 3).


Overall, EdgeKernelBench provides a systematic benchmark for LLM-based kernel generation across heterogeneous edge platforms with fragmented software stacks and tight memory budgets. We also introduce LEKA, a layered multi-agent framework for effective edge-kernel optimization. We believe this work will inspire further research on LLM-based kernel generation for edge devices.

We welcome any further questions and remain happy to discuss. Thank you very much for the detailed comments!

Thank you for the detailed responses and additional experiments. The component ablation helps clarify the contributions of the layered design, candidate tree, and hardware feedback. I also appreciate the added memory evaluation and the concrete examples illustrating cross-layer optimization.

These additions address most of my concerns and make the design and practical value of the system clearer.

Thank you very much for your positive feedback . Please let us know if you have any remaining concerns; we would be happy to address them. If you have no further concerns, could you please gently consider raising the score as a recognition to our rebuttal efforts?

Reviewer_cqq4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to use LLMs to generate high-performance kernels for heterogeneous edge devices and introduces two main components.

The first is EdgeKernelBench: 331 tasks given as PyTorch reference programs, organized by program scope into three levels (128 L1 basic computation patterns, 162 L2 operator-composition graphs, and 41 L3 compact model graphs) and instantiated with static shapes for three application scenarios (Mobile and Personal Intelligence, Embodied Robotics, and Smart AIoT), for 650 instances in total. Instead of writing native code for each backend, the model expresses computation graphs, schedules, and memory decisions in a unified IR, which backend adapters then lower to the target platform. The paper implements this design with TVM (Relax for computation graphs, TIR for computation and scheduling, with LLVM IR, CUDA C, or OpenCL C as the final output) and evaluates it on a Snapdragon 8 Gen 3 phone CPU (LLVM, Mobile tasks), a Jetson AGX Orin (CUDA, Robotics tasks), and the Mali-G610 GPU of an RK3588 (OpenCL, AIoT tasks). The performance baseline is TVM MetaSchedule with search budgets calibrated per workload class; MNN and TensorRT serve as additional comparisons. In one-shot generation with five LLMs, the best model reaches 64.5% Correct@1 and 25.5% [email protected] (the fraction of tasks that are correct and faster than the baseline); at L3, the best [email protected] on each of the three backends is only 0–4%.

The second is LEKA: three agents are each responsible for one level, L0 (graph and algorithm), L1 (loops and scheduling), and L2 (target source code); a read-only hardware-sensing agent attributes on-device measurements to the level that needs optimization; and a candidate tree records the candidates at each level and their lineage. On 69 tasks drawn by stratified random sampling (23 per backend), with GLM-5.3-Flash as the base model, LEKA reaches 43.5% [email protected], compared with 13.0% for one-shot generation with the same model. With the same base model, evaluator, task contracts, and time budget (90 minutes per task at L1 and 155 minutes at L2/L3), Claude Code reaches 26.1%, and LEKA's geometric-mean speedup over Claude Code is 3.8×.

Critical strengths

S1: The benchmark covers three backends on real edge devices with one task suite and evaluation protocol, and its program scopes extend up to compact model graphs. Existing kernel-generation benchmarks mostly target server-class hardware; as the paper describes it, MobileKernelBench mainly consists of single-operator tasks on mobile devices. This paper evaluates five models in one-shot generation on 650 instances across a phone CPU, a Jetson, and a Mali GPU. It reports compilation success, functional correctness, and whether programs beat the baseline separately, gives the distribution of failure types and its breakdown by backend and model, and compares against MNN and TensorRT. These results clearly show where current models struggle on edge backends: the best [email protected] is only 16.2% at L2 and 4.0% at L3, and memory errors and syntax/API errors together account for about 59% of failures. I consider this the paper's main value.

S2: Measurement and baselines are handled more carefully than in comparable work. Candidates and baselines are measured in pairs, interleaved within the same device session, under a unified stability criterion of rCV ≤ 5% for latencies and paired speedups; CPU cores are pinned, the Jetson is fixed in MAXN mode, and the RK3588 GPU frequency is locked. The TVM baseline is tuned with MetaSchedule using search budgets calibrated for convergence over six workload classes, and two production inference engines are added as references. The agent comparison controls the base model, evaluator, task contracts, and time budget, which makes it more convincing than a comparison with one-shot generation alone. I therefore consider the individual measurements trustworthy; my concerns are about how the conclusions are interpreted and attributed.

Critical weaknesses

W1: The agent experiments do not establish that LEKA's layered design is necessary.**

LEKA is one of the paper's main contributions, and both the abstract and the conclusion use its advantage over Claude Code as the main evidence.

  • Base model. For budget reasons, both agents use only GLM-5.3-Flash, which Table 1 does not evaluate although [email protected] there varies widely with the model (21.4–51.0% at CUDA L1), and the paper itself suggests that the smaller L3 gain may reflect the base model's limits. It is therefore unclear whether a general-purpose coding agent with a stronger frontier model would already come close to LEKA, and whether layering would still help.
  • Claude Code baseline. On CPU, LEKA is 8.6× faster than Claude Code in geometric mean yet beats tuned TVM on only 8 of 23 tasks, which suggests, unless LEKA's wins over TVM are very large, that Claude Code's CPU programs are several times slower than tuned TVM. Appendix F.2.1 does not specify which IR levels Claude Code can edit or what evaluator feedback it receives, and the comparison matches the wall-clock budget but reports no tokens, LLM calls or on-device measurements.
  • Attribution. Besides layering, LEKA adds hardware-sensing diagnostics, a candidate tree and per-level prompts, with no ablations or repeated agent runs, and OpenCL alone accounts for 8 of the net 12-task gain in [email protected]. The experiments therefore cannot separate the layered division of work from better tools and feedback.

W2: The claims about the benchmark's edge-specific and unified-IR aspects go beyond what the experiments verify.

The task format, the L1/L2/L3 levels and the Fast@p metric all follow the KernelBench protocol, so the benchmark's increment over prior work lies mainly in the edge dimension, where the evidence is thin:

  • Memory. The paper argues that edge kernels must optimize performance and memory jointly within a fixed resource budget, but the evaluation neither systematically measures peak memory or temporary buffers nor imposes a memory budget; evidence of memory savings comes only from individual cases in the appendix.
  • Unified IR. No optimization obtained on one backend is applied to another, and there is no comparison with direct native-code generation, so there is no evidence on whether the unified IR makes generation easier or harder.
  • Backends. Each scenario is evaluated on only one backend, so the differences between backends in §5.2.1 are not separated from differences in tasks and shapes.
Questions for authors
  1. Claude Code baseline (clarification). Which IR levels can Claude Code edit and what evaluator feedback does it receive, compared with LEKA's hardware-sensing agent, and what are the geometric-mean speedups over tuned TVM of one-shot generation, Claude Code and LEKA?

  2. Controls (experiment, on a small stratified subset). Please add (a) a single agent with LEKA's base model, IR access, hardware diagnostics, candidate archive and budget but without the fixed L0/L1/L2 division of work, and (b) Claude Code and LEKA each run with a stronger frontier model (e.g., Claude Opus 5.5 or GPT-6 Astra), reporting the tokens, LLM calls and on-device measurements per task. Control (a) separates layering from tools and feedback, and control (b) shows whether layering still helps with stronger models.

  3. Unified IR and memory (experiment). For several tasks that have instances on more than one backend, what happens when the L0 (graph and algorithm level) optimizations obtained on one backend are applied directly to the others, and what peak memory or temporary-buffer usage do the generated programs have?

5 comments

We sincerely thank the reviewer for the detailed feedback and for recognizing EdgeKernelBench's coverage of real edge devices and our careful measurement methodology. Below, we respond to each question in turn. In the revised manuscript, we have marked all changes in blue; in this response, we have also highlighted key points for ease of reading.


W1.1 & Q2(b): Value of layering with a stronger base model.

We thank the reviewer for raising this question. We have added a comparison using the stronger frontier model Opus-5.5 to drive operator optimization in both LEKA and Pure Claude Code. This experiment examines whether LEKA's layered design remains beneficial as the base model becomes more capable. Tables 9 and 10 summarize the performance and optimization resources.

Table 9: Runtime performance. GM speedup relative to Pure Claude Code (Pure).

BackendPureLEKATVM
CPU/LLVM1.0001.7120.392
GPU/CUDA1.0001.1630.376
GPU/OpenCL1.0001.6600.501
Overall1.0001.4900.419

Table 10: Optimization resources. LEKA/Pure ratios of model inference requests and token usage.

BackendLLM callsTokens
CPU/LLVM3.6252.189
GPU/CUDA4.0332.342
GPU/OpenCL1.8521.013
Overall2.9791.636

LEKA achieves an overall GM of 1.490× over Pure Claude Code, with improvements across all three backends. Therefore, the layered design remains effective with a stronger model. However, the relative gains are smaller than those obtained with GLM-5.3-Flash. This difference should be considered alongside the strength of the baseline. Claude Code already achieves backend-level GMs of approximately 2.00× to 2.66× over TVM, placing further optimization on a higher-performance baseline. Obtaining large additional relative speedups from this higher starting point is also more challenging.

At the same time, the performance gap grows with task complexity. On GPU/CUDA, LEKA achieves GMs of 1.027×, 1.134×, and 1.570× at Levels 1, 2, and 3, respectively. The methods perform similarly on simpler tasks with the stronger model. As task complexity increases, LEKA achieves larger speedups, making the difference between the methods more pronounced.

Table 10 also quantifies the additional optimization resources. Overall, LEKA uses 2.979× as many model inference requests and 1.636× the token usage of Pure Claude Code. These figures make the resource cost accompanying the performance gains explicit.

These results support the conclusion that LEKA's layered optimization remains effective with a stronger base model and can deliver larger benefits on more complex tasks. Per-task token counts, inference-request counts, and on-device performance measurements are provided in Table 11 of the revised PDF appendix. We thank the reviewer for this constructive suggestion. Your feedback has been important in making our manuscript more convincing.


W1.2 & Q1.1: Claude Code's IR access and evaluator feedback.

We thank the reviewer for the opportunity to clarify the IR access and evaluator feedback of the Claude Code baseline.

The baseline can edit the same three representation levels as LEKA. These comprise Relax for computation graphs (L0), TIR for computation and scheduling (L1), and target-specific CUDA C, OpenCL C, or LLVM IR (L2). It optimizes complete candidates through a single persistent session. We do not prescribe layer roles or a fixed optimization order for it.

The evaluator provides correctness outcomes, whole-program execution latency for valid candidates, and concise diagnostics for compilation or execution failures. Compared with the Claude Code baseline, LEKA additionally receives per-operation or per-kernel execution times, call counts, and runtime shares through its hardware-sensing agent. Available profiling metrics include SM and L2 cache throughput on Jetson GPUs, and process-level cache misses and instructions per cycle (IPC) on CPUs.

We have added these details to Appendix C.4 of the revised manuscript. We appreciate your suggestion and hope the above clarification addresses your concern.

Q1.2: Absolute performance relative to tuned TVM.

We thank the reviewer for the opportunity to clarify the geometric mean (GM) speedups relative to tuned TVM. The results are reported below, with tuned TVM normalized to 1.000. For each backend, GM is computed over the common set of tasks that pass correctness checks for all compared methods.

BackendTuned TVMOne-shot generationPure Claude CodeLEKA
Jetson1.0000.2930.4920.516
RK35881.0000.2740.4320.840
vivo1.0000.1740.2060.320

Speedup is defined as tuned TVM latency divided by the method's latency. Values below 1 therefore indicate slower execution than tuned TVM on these common task sets. The limited speedups of LEKA and Claude Code over the TVM baseline may partly reflect the weaker capabilities of the GLM-5.3-Flash base model. In the supplementary experiments with the stronger Opus-5.5 model, Pure Claude Code achieves backend-level GMs of approximately 2.00× to 2.66× over tuned TVM.

We appreciate your suggestion and hope this clarification addresses your concern.


W1.3 & Q2(a): Separating layering from tools and feedback.

Thank you for this valuable suggestion. To distinguish the contribution of hierarchical design from that of tools and feedback, we added a component ablation study. No Hierarchy uses the same base model as LEKA and retains IR access, hardware diagnostics, and the same optimization budget. It uses a single agent without fixed L0/L1/L2 responsibilities. No HW Feedback retains the hierarchy but removes the hardware feedback agent. The table reports GM speedup relative to Claude Code. Parentheses give each method's GM minus LEKA's GM.

BackendLEKANo HW FeedbackNo HierarchyClaude Code
CPU/LLVM10.136.93 (−3.20)2.58 (−7.55)1.00 (−9.13)
GPU/OpenCL2.581.86 (−0.72)1.65 (−0.93)1.00 (−1.58)
GPU/CUDA1.811.60 (−0.21)1.20 (−0.61)1.00 (−0.81)

Across all three backends, removing the hierarchy produces a larger GM drop than removing hardware feedback. Since No Hierarchy retains the tools, diagnostics, and budget, these results support the added value of hierarchical design beyond hardware feedback.

Your feedback has been important in making our manuscript more convincing. The detailed ablation results can be found in the revised PDF (Section 5.3.3, COMPONENT ABLATION, Table 3).


W2.1: Memory Budgets.

We thank the reviewer for the opportunity to clarify how memory constraints are considered in our framework. We explicitly apply memory budgets in the agent evaluation. We have clarified the memory metric in Section 5.1 and provided a supplementary memory-budget evaluation in Section 5.3.4 (Table 4) of the revised manuscript.

The memory budget satisfaction rate measures the fraction of tasks whose peak temporary memory usage stays within the assigned budget. This peak covers simultaneously live intermediate tensors and dynamic operator workspaces during one inference call. Memory-related decisions are exposed across IR levels. Graph-level fusion and decomposition determine intermediate tensor materialization and lifetimes, while lower-level agents optimize data reuse and temporary storage.

For the revised evaluation, we systematically assign peak temporary memory budgets according to the edge application scenario, task, input shape, and target backend. All methods share the same budget for each configuration, with detailed settings provided in Appendix B.5. Section 5.3.4 (Table 4) reports the resulting memory budget satisfaction rates across the three backends.

Memory budget satisfaction rate (%)

BackendLEKAClaude CodeTVM
CPU/LLVM82.6182.6191.30
GPU/OpenCL91.3082.6195.65
GPU/CUDA91.3082.6186.96

Memory is therefore considered in the framework's optimization space, and memory budgets are explicitly included in the evaluation.

We appreciate your suggestion and hope the above clarification addresses your concern.

W2.2: Unified IR versus direct native-code generation.

Thank you for this valuable suggestion. To assess the benefits of unified IR generation, we have added a comparison between unified IR generation and direct native IR generation (LLVM, CUDA, and OpenCL) to the manuscript. The experiment is reported in Section 5.2.4 of the revised manuscript (pp. 8–9).

Table 2. Unified versus native IR. Δ denotes unified minus native generation in percentage points. GM is relative to native generation.

BackendModelΔCSRΔFCRΔF@1GM
GPU/CUDAGPT-5.6-sol−18.2−9.10.01.420×
DeepSeek-V4-Pro−27.3−9.10.02.030×
GPU/OpenCLGPT-5.6-sol0.0−9.1+9.11.284×
DeepSeek-V4-Pro0.0−9.1+18.21.729×
CPU/LLVMGPT-5.6-sol+36.4+45.5+45.53.208×
DeepSeek-V4-Pro+36.4+45.5+36.44.966×
Overall+4.5+9.1+18.21.943×

Overall, Table 2 shows higher CSR, FCR, and F@1 for unified IR. GM speedups over native generation are substantial across all three backends. These gains show that unified IR generation is beneficial compared with direct native IR generation for fragmented edge backends.

Nevertheless, native CUDA generation achieves higher CSR and FCR. CUDA has an established LLM kernel optimization ecosystem [R1, R2, R3]. Meanwhile, models may be more familiar with CUDA, and these factors may partly explain its advantage.

However, edge deployments span fragmented hardware and software stacks. These native backend IRs often lack a comparably mature ecosystem for LLM kernel generation. This disparity may contribute to the weaker execution performance of native generation, particularly on LLVM. Therefore, targeting unified IR can generally make code generation easier for fragmented edge backends.

Thank you again for your constructive feedback, which has helped us substantially strengthen the manuscript.

References

[R1] Ouyang A, Guo S, Arora S, et al. KernelBench: Can LLMs write efficient GPU kernels? ICML 2025.

[R2] Baronio C, Marsella P, Pan B, et al. Kevin: Multi-turn RL for generating CUDA kernels. ICLR 2026.

[R3] Zhang Z, Wang R, Li S, et al. CudaForge: An agent framework with hardware feedback for CUDA kernel optimization. arXiv:2511.01884, 2025.


W2.3: Separating backend effects from task and shape differences.

Thank you for this thoughtful feedback. Our previous analysis of backend differences in §5.2.1 was not sufficiently accurate. We have revised this discussion and added a supplementary comparison on matched workloads in Appendix E.1. These updates align the detailed analysis with the main experimental results, and the conclusions summarized below remain supported.

Specifically, we compare CPU/LLVM with GPU/CUDA on Jetson, and CPU/LLVM with GPU/OpenCL on vivo, using GPT-5.6-sol and DeepSeek-V4-Pro. Within each device, the application scenario, task definitions, tensor shapes, and parameters are held fixed. The results are summarized below; all values are percentages, and F@1 is measured against the corresponding backend's TVM baseline.

Table 8: One-shot generation on matched CPU and GPU workloads.

DeviceModelCPU CSRCPU FCRCPU F@1GPU CSRGPU FCRGPU F@1
JetsonGPT-5.6-sol66.761.528.266.756.435.9
JetsonDeepSeek-V4-Pro74.459.020.559.043.623.1
vivoGPT-5.6-sol69.259.010.361.551.320.5
vivoDeepSeek-V4-Pro82.174.412.879.556.412.8

According to these results, CPU achieves higher FCR than GPU, possibly because thread partitioning and parallel mapping make valid GPU implementations harder to generate. In addition, CUDA exceeds CPU in F@1 by 2.6–7.7 percentage points. We hypothesize that this advantage partly reflects the active CUDA kernel-generation ecosystem [R1, R2].

Therefore, these supplementary backend comparisons using identical tasks and tensor shapes within the same application scenario remove tensor-shape differences as a confounding factor, and the backend-related observations above remain supported.

Thank you for your contribution to improving our revised manuscript. The detailed revisions can be found in §5.2.1 and Appendix E.1 (Table 8) of the revised PDF.

References

[R1] Ouyang A, Guo S, Arora S, et al. KernelBench: Can LLMs write efficient GPU kernels? ICML 2025.

[R2] Baronio C, Marsella P, Pan B, et al. Kevin: Multi-turn RL for generating CUDA kernels. ICLR 2026.

Reviewer_AYrxOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper concerns LLM code generation for edge devices. Compared to servers, edge devices are far more heterogeneous in hardware and compilation frameworks. The paper proposes that instead of generating code directly for native hardware, LLM's should generate code for a uniform higher level IR (instantiated as TVM in the paper), which then is lowered by the backend specific compiler to native code. To address performance, the paper proposes LEKA, an agent framework that uses hardware profiling to invoke optimization at appropriate levels of the IR. Additionally the paper proposes a hierarchical suite of benchmark codes ranging from operators to more complex composition of operators to more complex graphs. The combination of the benchmarks and the IR deployment framework is referred to as EdgeKernelBench. The evaluation shows that for the simpler operator kernels, one shot generated code is often correct, but often not performant. As the benchmark code complexity increases, both the fraction of correct and performant code reduces substantially. Using LEKA for optimization improves the situation considerably - the paper claims this is because focusing within a specific IR level at a time is more effective.

Critical strengths
  • The problem of generating code for heterogeneous edge hardware is important.
  • The idea of using a uniform layered IR as the target for LLM code generation is interesting and arguably non-intuitive. Such an idea is now widely used by traditional compilers, and it is not at all intuitive that LLM code generation frameworks should be similarly structured.
  • The idea of using IR layer specific optimization agents iteratively informed by hardware profiling is also interesting and again not necessarily intuitive.
  • The evaluation results (if substantiated) are interesting.
Critical weaknesses
  1. I am ambivalent about what the results mean. For the one shot generation vs TVM (Table 1), except for level 1 benchmarks, for most models, less than half the programs are functionally correct and very few are performant. (For those that are slower than TVM, the table does not say how slow.) Aren't these results rather weak for the hypothesis? Iterative optimization with Leka does indeed improve these numbers impressively, but I am not sure of the methodology - see below.
  2. It is not clear that the evaluation is showing that compiling to a uniform IR is beneficial. Why not compare against code generated directly for the lowest level? I understand that the argument is that this is hard because such hardware is evolving. But here the hardware/low level IR used is quite stable (Jetson/CUDA, Snapdragon/LLVM, Mali/OpenCL). Wouldn't this be an important baseline to show that compilation to a higher level uniform IR is effective?
  3. The main body of the paper is not clear on how LEKA works. How are the hardware profiling results used to determine which IR level to optimize? In this case, the paper uses a well established and now somewhat conventional TVM IR. How would this generalize to an IR that was designed specifically for LLM code generation (I would imagine this would be part of the future work)? There is not much insight into how the optimization framework works and for what type of codes (only 23 tasks, mostly level 1 and 2, are considered).
  4. The introduction makes a big deal out of memory constraints, but there is no information about how memory is considered in the framework nor are there any evaluation results. Is memory considered?
  5. Will this be open sourced?
Questions for authors

Please answer the five questions raised in the weakness section.

6 comments

We sincerely thank the reviewer for the thoughtful feedback and for recognizing the importance of kernel generation for edge devices and the promise of unified IR and layered optimization. Below, we respond to each question in turn. In the revised manuscript, we have marked all changes in blue; in this response, we have also highlighted key points for ease of reading.


W1: Interpretation of the one-shot results in Table 1.

We thank the reviewer for raising this question. We understand the concern about whether the correctness and performance results support our research hypothesis. We believe Table 1 should be interpreted in the context of its one-shot evaluation setting. We address this concern from two perspectives: optimization quality and time cost.

First, Table 1 measures current models under one-shot generation. Both base-model capability and the absence of iterative feedback affect these results. To further examine feasibility, we added experiments using the stronger Opus-5.5 model for iterative optimization with both Claude Code and LEKA. Both methods' final implementations pass correctness checks on the stratified evaluation subset, achieving 100% functional correctness. Table A reports geometric mean speedups over the tuned TVM baseline, normalized to 1.

Table A: Runtime performance with Opus-5.5 and iterative optimization. Higher GM is better.

BackendTVMClaude CodeLEKA
CPU/LLVM1.0002.5524.368
GPU/CUDA1.0002.6623.096
GPU/OpenCL1.0001.9973.315
Overall1.0002.3853.552

Claude Code and LEKA achieve overall GMs of 2.385× and 3.552× over TVM, respectively. These results show that stronger models combined with iterative optimization can produce correct and competitive edge-kernel implementations. Please see Appendix E.5 of the revised PDF for the experimental details.

Second, direct LLM generation reduces the time spent searching for optimized implementations. In our one-shot workflow, generating an optimized operator program typically takes approximately 5 to 10 minutes,whereas TVM MetaSchedule tuning often requires several hours or more per operator. MetaSchedule repeatedly compiles candidate schedules and measures them on the device, while direct LLM generation avoids this repeated search. Reducing the time needed to obtain optimized implementations is therefore another benefit of using LLMs for edge-kernel optimization.

These results support the feasibility of LLM-based edge-kernel generation and optimization in terms of implementation quality and optimization time. Table 1 primarily evaluates current models' one-shot capabilities and identifies challenges across task complexity levels and backends, providing a common basis for future research. We therefore view Table 1 as evidence of current one-shot generation challenges, while the supplementary iterative results further support our research hypothesis.

We appreciate your suggestion and hope these additional results and explanations address your concern.

W2: Unified IR versus direct native-code generation.

Thank you for this valuable suggestion. To assess the benefits of unified IR generation, we have added a comparison with direct native IR generation (LLVM, CUDA, and OpenCL) to the manuscript. The experiment is reported in Section 5.2.4 of the revised manuscript (pp. 8–9).

Table 2. Unified versus native IR. Δ denotes unified minus native generation in percentage points. GM is relative to native generation.

BackendModelΔCSRΔFCRΔF@1GM
GPU/CUDAGPT-5.6-sol−18.2−9.10.01.420×
DeepSeek-V4-Pro−27.3−9.10.02.030×
GPU/OpenCLGPT-5.6-sol0.0−9.1+9.11.284×
DeepSeek-V4-Pro0.0−9.1+18.21.729×
CPU/LLVMGPT-5.6-sol+36.4+45.5+45.53.208×
DeepSeek-V4-Pro+36.4+45.5+36.44.966×
Overall+4.5+9.1+18.21.943×

Overall, Table 2 shows higher CSR, FCR, and F@1 for unified IR. GM speedups over native generation are substantial across all three backends. Unified IR generation is thus effective for fragmented edge backends.

Nevertheless, native CUDA generation achieves higher CSR and FCR. CUDA has an established LLM kernel optimization ecosystem [R1, R2, R3]. Meanwhile, models may be more familiar with CUDA, and these factors may partly explain its advantage.

However, edge deployments span fragmented hardware and software stacks.Their native targets often lack a comparably mature ecosystem for LLM kernel generation. This disparity may contribute to the weaker execution performance of native generation, particularly on LLVM. Therefore, unified rather than native IR generation is beneficial for fragmented edge backends.

Thank you again for your constructive feedback, which has helped us substantially strengthen the manuscript.

References

[R1] Ouyang A, Guo S, Arora S, et al. KernelBench: Can LLMs write efficient GPU kernels? ICML 2025.

[R2] Baronio C, Marsella P, Pan B, et al. Kevin: Multi-turn RL for generating CUDA kernels. ICLR 2026.

[R3] Zhang Z, Wang R, Li S, et al. CudaForge: An agent framework with hardware feedback for CUDA kernel optimization. arXiv:2511.01884, 2025.


W3.1: Using hardware profiling to choose the optimization level.

We thank the reviewer for the opportunity to clarify how hardware profiling guides optimization in LEKA. The hardware-sensing agent combines profiling results with the current IR implementation to identify the main bottleneck and recommends the IR level that exposes the transformations needed to address it.

On the vivo CPU/LLVM global-average-pooling task, for example, profiling attributes 98.27% of the total reported time to the mean reduction. The hardware-sensing agent then inspects the implementation and finds a serial scalar reduction under an eight-thread runtime configuration. It recommends L1, which handles loops and scheduling, because channel partitioning and independent partial accumulation require reorganizing the reduction loops. L1 subsequently introduces channel parallelism and private vector accumulators while preserving the graph and call interfaces. The implementation passes correctness checks and, in search-stage measurements, achieves a 10.7× speedup over the initial schedule on the same graph.

We have added the case study “Profiling-guided selection of the IR level” to Appendix E.3.2 to clarify how profiling evidence informs the choice of IR level and the resulting optimization recommendation. Please see the text marked in blue in the revised PDF for details.

W3.2: Generalization to IRs designed for LLM code generation.

We thank the reviewer for raising this point. Extending our approach to IRs designed for LLM code generation is a promising direction for future work. Agents could use kernel optimization feedback to design IR abstractions for themselves and improve the corresponding compiler infrastructure. These abstractions could be adapted to the target hardware and specific optimization tasks. This would extend optimization within a fixed IR to the joint evolution of kernel implementations and compiler support [R4]. For edge deployment, this direction would need to account for resource constraints and deployment requirements across heterogeneous backends. We would investigate how the IR and agent responsibilities could evolve together, so that experience gained during optimization can benefit subsequent tasks.

References

[R4] Ye Z, Huang Y, Jin H, et al. CAKE: Compiler-agent co-design for frontier kernel evolution. arXiv:2608.12629, 2026.


W3.3: Insights into LEKA's optimization process.

We thank the reviewer for this constructive suggestion. We have expanded Appendix G.2 (“More Insights and Future Work”) in the manuscript to provide further insights into LEKA's optimization process. LEKA's optimization process illustrates the dependencies between decisions across layers and the need to combine focused optimization within each layer with cross-layer feedback.

Building on this observation, we discuss how multilevel compiler infrastructure such as IREE could expose finer-grained program representations and support different divisions of work among agents. Future work could compare combinations of layer boundaries, context sharing, and cross-layer feedback to narrow each agent's search space while retaining opportunities for joint optimization. We also discuss co-designing LLM-oriented IRs, layered agents, and compiler support for specific edge hardware, guided by optimization feedback, so that accumulated experience can benefit subsequent tasks.

We hope these clarifications address your concerns.


W4: Consideration of Memory Constraints.

We thank the reviewer for the opportunity to clarify how memory constraints are considered in our framework. We explicitly take memory budgets into account. We have clarified the memory metric in Section 5.1 and added a memory-budget evaluation in Section 5.3.4 (Table 4) of the revised manuscript.

Specifically, the memory budget satisfaction rate measures the fraction of tasks whose peak temporary memory usage stays within the assigned budget. This peak covers simultaneously live intermediate tensors and dynamic operator workspaces during one inference call. Memory-related decisions are exposed across IR levels. Graph-level fusion and decomposition determine intermediate tensor materialization and lifetimes, while lower-level agents optimize data reuse and temporary storage.

In Section 5.3.4 (Table 4), we evaluate memory budget satisfaction using budgets tailored to the edge application scenarios, tasks, input shapes, and target backends. All methods share the same budget for each configuration, with detailed settings provided in Appendix B.5. The results are summarized below.

Memory budget satisfaction rate (%)

BackendLEKAClaude CodeTVM
CPU/LLVM82.6182.6191.30
GPU/OpenCL91.3082.6195.65
GPU/CUDA91.3082.6186.96

Memory is therefore considered in both the framework's optimization space and its evaluation.


W5: Open-source availability.

We will open-source EdgeKernelBench and the LEKA implementation to support reproducibility and further research.


Overall, EdgeKernelBench provides a systematic benchmark for LLM-based kernel generation across heterogeneous edge platforms with fragmented software stacks and tight memory budgets. We also introduce LEKA, a layered multi-agent framework for effective edge-kernel optimization. We believe this work will inspire further research on LLM-based kernel generation for edge devices.

We welcome any further questions and remain happy to discuss. Thank you very much for the detailed comments!

Comments on the paper

2 comments
Reviewer_AYrxOpenReview ↗

Response to authors

Dear authors,

Thank you for your detailed response. I have changed my rating to clear acceptance.

AuthorsOpenReview ↗

Summary of Reviews and Author-Reviewer Discussion

Dear AC, SACs, and PCs,

We sincerely thank all reviewers for their time and constructive feedback. As the discussion period closes, we summarize the main points of each review, how we addressed them, and the rating changes during the discussion.

Overview

ReviewerInitialCurrentStatus
AYrx3 (Weak acceptance)4 (Clear acceptance)Raised 19 Nov
RX2M3 (Weak acceptance)3 (Weak acceptance)Most concerns addressed
cqq42 (Weak rejection)2 (Weak rejection)No reply

Reviewer AYrx: 4 (Clear acceptance)

Strengths: The importance of edge-device kernel generation and the promise of unified IR and layered optimization.

Weaknesses & our responses:

  • W1: Interpretation of the one-shot results in Table 1. ✅ Addressed. Table 1 shows one-shot challenges; iterative Opus-5.5 achieves substantial speedups over tuned TVM on the stratified subset, further supporting our hypothesis (App. E.5).
  • W2: Unified IR versus direct native-code generation. ✅ Addressed. Added comparisons and analysis showing unified IR's benefits over native IR for fragmented edge backends (Table 2).
  • W3.1–W3.3: Using hardware profiling to choose the optimization level; Generalization to IRs designed for LLM code generation; Insights into LEKA's optimization process. ✅ Addressed. Clarified profiling-guided level selection (App. E.3.2) and expanded LLM–IR co-design discussion (App. G.2).
  • W4: Consideration of Memory Constraints; W5: Open-source availability. ✅ Addressed. Added matched-budget peak temporary-memory evaluations across backends (Table 4), explicitly including memory in optimization and evaluation; confirmed open-source plans.

Rating change: On 19 Nov: "I have changed my rating to clear acceptance." 3 → 4 (Clear acceptance).

Reviewer RX2M: 3 (Weak acceptance)

Strengths: The value of EdgeKernelBench as a testbed for edge kernel generation and the practical value of LEKA.

Weaknesses & our responses:

  • W1.1 & Q1: Revisiting higher-level decisions and earlier candidates. ✅ Addressed. Expanded “Cross-layer requests guide graph refinement” to explain LEKA's use of cross-layer requests (App. E.3.2).
  • W1.2 & Q2: Memory Budgets and Handling Budget Violations. ✅ Addressed. Added explicit, matched-budget peak temporary-memory evaluations across backends (Table 4) and memory optimization cases (App. E.3.3).
  • W2 & Q3: Contributions of the individual components. ✅ Addressed. Added component ablations: hierarchy removal causes the largest GM drop on every backend; the candidate tree and hardware feedback each add gains (Table 3).

Status: On 16 Nov, RX2M wrote: "These additions address most of my concerns and make the design and practical value of the system clearer."

Reviewer cqq4: 2 (Weak rejection)

Strengths: EdgeKernelBench's coverage of real edge devices and its careful measurement methodology.

Weaknesses & our responses:

  • W1.1 & Q2(b): Value of layering with a stronger base model. ✅ Addressed. Added Opus-5.5 comparisons and resource-usage results showing LEKA's layered optimization remains effective with a stronger model (App. E.5).
  • W1.2 & Q1.1: Claude Code's IR access and evaluator feedback. ✅ Addressed. Clarified Claude Code's L0/L1/L2 access and how its evaluator feedback differs from LEKA's profiling (App. C.4).
  • Q1.2: Absolute performance relative to tuned TVM. ✅ Addressed. Reported common-subset GMs relative to tuned TVM.
  • W1.3 & Q2(a): Separating layering from tools and feedback. ✅ Addressed. Added component ablations isolating the contribution of layering from tools and feedback (Table 3).
  • W2.1: Memory Budgets. ✅ Addressed. Added matched-budget peak temporary-memory evaluations across backends (Table 4), explicitly including memory in optimization and evaluation.
  • W2.2: Unified IR versus direct native-code generation. ✅ Addressed. Added comparisons and analysis showing unified IR's benefits over native IR for fragmented edge backends (Table 2).
  • W2.3: Separating backend effects from task and shape differences. ✅ Addressed. Added matched-workload comparisons (App. E.1) and revised the discussion; the conclusions remain supported.
  • Q3: Cross-backend reuse of L0 optimizations and memory usage. ✅ Addressed. Added two representative transfer cases and temporary-buffer usage results: unified IR preserves upper-level optimizations across hardware and supports adaptation around reused parts (App. E.4).

Status: We posted a four-part response on 16 Nov and followed up on 17 and 19 Nov, but received no reply.


Overall, AYrx and RX2M rated our work 4 (Clear acceptance) and 3 (Weak acceptance), respectively. We regret that we could not engage with cqq4 during the discussion. We believe our final responses have addressed cqq4's concerns and hope they support a favorable final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.