acceptodds
Under review as a conference paper at ICLR 2027

EvokeAgent: A Hardware-Aware, Knowledge-Base-Co-Driven Self-Evolving Agent System for GPU Kernel Optimization

Abstract

Automated GPU operator optimization is critical for accelerating deep learning workloads, prompting the growing use of Large Language Models (LLMs) to generate and tune high-performance kernels. While existing systems increasingly incorporate runtime profiling feedback, the selection of hardware evidence is often not explicitly coupled to optimization actions; relying on the model alone to analyze all such hardware data makes it difficult to pinpoint the right optimization opportunities amid noisy metrics. We present EvokeAgent, an NCU-native GPU operator optimization agent built around a self-evolving, bottleneck-indexed knowledge base (KB). EvokeAgent establishes a bottleneck-guided routing layer that contracts voluminous NCU metrics into hypothesis-relevant diagnostic evidence downward while mapping the diagnosis to targeted KB policy retrieval upward. Evaluated on all three KernelBench levels using an untuned LLM, EvokeAgent achieves 100% correctness with geometric-mean speedups of 1.93, 2.65, and 2.94 on CUDA, and 1.68, 2.56, and 1.64 on Triton across L1–L3, respectively. Furthermore, ablation shows that this routing-and-retrieval design outperforms the non-KB baseline by 1.08, 1.48, and 1.55 on L1–L3 under capped optimization rounds.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
GPU kernel optimization, LLM agents, CUDA, Triton, Nsight Compute, knowledge-base retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_XXnuOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

EvokeAgent is an LLM-based system for GPU kernel optimization, combining feedback loop with an evolving database of optimization policies indexed by the profiler-diagnosed bottleneck. Its central mechanism uses diagnosed bottlenecks to select relevant profiler measurements and retrieve optimization strategies. Successful strategies can be stored for reuse across tasks, with separate CUDA and Triton knowledge bases. Evaluated on 250 KernelBench tasks using an NVIDIA H20-3e and GLM-5.2, it reports CUDA geometric-mean speedups of 1.93×, 2.65×, and 2.94× across three levels. and Triton geometric-mean speedups of 1.68×, 2.56×, and 1.64×. Ablations support the combined design but do not isolate individual contributions; external comparisons lack matched experimental conditions. Moreover, the idea seems incremental over KernelBlaster

Critical strengths

EvokeAgent uses the diagnosed bottleneck to select relevant NCU measurements and retrieve targeted optimization strategies, giving the LLM both diagnostic evidence and actionable guidance. Its implementation across CUDA and Triton demonstrates applicability beyond a single kernel language.

Critical weaknesses
  1. Details of the evolution process are insufficiently documented. The KB starts with manually curated policies and grows during evaluation, but the overlap between human written seeds and the tasks are unclear. The order of KernelBench tasks could affect the KB construction but I did not find the analysis of this.

  2. External comparisons use different hardware, LLMs, budgets, and evaluation protocols. Although they can serve as useful contextual information, they cannot establish EvokeAgent's superiority over close methods such as KernelBlaster. The main results also require clarification. For example, the maximum 1093.11× speedup lacks a independent validation. This outlier is not proof of reward hacking, but warrants inspection of the generated code, correctness coverage, and timing.

Questions for authors
  1. Could you provide ablation evidence to attribute gains specifically to retrieved policies, selection algorithm, or their coupling? What about using other algorithms like MAP-Elite as the selection algorithm?
  2. Could you provide break down of retrieved and/or adopted policies by manual seeds versus newly generated ones? The inventory counts do not suggest retrieval or usage counts
  3. Could you identify the tasks involving suspicious speedup like the one with 1093.11× speedup? This will affect the assessment of what the LLM actually learns.
  4. Can a close competitor such as KernelBlaster be evaluated using the same LLM, GPU, tasks, and budget to establish the comparative benefit?
  5. Since the KB are shared across the tasks, how does the order of tasks affect the final speedup?
  6. How were the thresholds in the two-level bottleneck classifier chosen (e.g., 60% memory/SM throughput and 40% occupancy in Table 11, and the sub-category cut-offs in Tables 12–13)? Does “calibrated for the H20-3e” mean they were tuned on KernelBench problems, and how sensitive are the results to changing them?
Reviewer_dqirOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies GPU kernel optimization agents, motivated by the difficulty of identifying bottlenecks when an agent is directly given large amounts of potentially noisy NCU metrics and the difficulty of mapping identified bottlenecks to existing optimization experience. It proposes EvokeAgent, which uses a two-stage diagnosis process to identify bottleneck types: the first stage uses coarse-grained NCU metrics to estimate the bottleneck category, and the second selects fine-grained evidence. The identified bottleneck is then used as an index to retrieve relevant optimization strategies from a constructed knowledge base, which provides guidance to the LLM and can absorb new experience from the optimization process. The authors evaluate the system on KernelBench. For CUDA, the reported geometric mean speedups on L1-L3 are 1.93×, 2.65×, and 2.94×; for Triton, the corresponding speedups are 1.68×, 2.56×, and 1.64×.

Critical strengths

The paper proposes a practical workflow for a kernel optimization agent that connects NCU-based bottleneck diagnosis with a knowledge base. The overall workflow is clear.

Critical weaknesses
  1. The method lacks sufficient originality and technical depth. The paper mainly compares the proposed system with an entirely untuned setting and does not sufficiently demonstrate that its core design goes beyond combinations of existing optimization techniques.

  2. The ablation study does not isolate the contributions of the individual components, and the management of context during iterative optimization is not sufficiently explained.

Questions for authors
  1. The paper argues that directly using NCU information can overwhelm the agent with information. Could a simpler approach achieve similar results by prompting the same LLM to first identify the bottleneck, focus on the relevant fine-grained metrics, and then retrieve optimization experience?

  2. Have the authors evaluated such a baseline under the same model, initial candidates, hardware, and optimization budget?

Reviewer_FumZOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents EvokeAgent, an LLM agent system for automatic GPU kernel optimization. It uses the bottleneck type diagnosed from NCU profiling as a unified routing variable that selects relevant fine-grained hardware metrics and retrieves optimization policies from a bottleneck-indexed, self-evolving knowledge base, followed by multi-round validated refinement. On all 250 KernelBench problems, with an untuned LLM, it achieves 100% correctness and geometric mean speedups of 1.93×, 2.65×, and 2.94×, and a controlled ablation shows 1.08×, 1.48×, and 1.55× improvements over a no-KB baseline.

Critical strengths
  • Solid empirical results with full benchmark coverage. Evaluated on all 250 KernelBench problems across three levels, the system achieves 100% correctness with geometric mean speedups of 1.93×, 2.65×, and 2.94× using an untuned LLM, and demonstrates portability across both CUDA and Triton backends, placing it among the stronger results in this area.

  • The ablation protocol is reasonably well controlled. The With-KB versus No-KB comparison is conducted under matched hardware, model, initial candidate, and budget, with a round-by-round convergence analysis provided as supporting evidence, which is more complete than the ablation practice in many comparable works.

Critical weaknesses
  • Limited novelty at the method level, as the core components are combinations of known techniques. The claimed contribution is using the diagnosed bottleneck as a routing variable for both evidence selection and policy retrieval. However, on closer inspection, the two-level metric filtering is essentially conditional subset selection over NCU metrics, and bottleneck-indexed policy retrieval is a variant of retrieval-augmented experience memory in the kernel optimization domain, differing from existing memory-based methods mainly in the choice of retrieval key. The paper neither argues why the bottleneck label is a fundamentally better indexing key nor demonstrates that the proposed coupling is necessary for the observed gains rather than a convenient engineering arrangement. The overall contribution reads as a solid but incremental system combination.

  • No controlled comparison against retrieval- or memory-augmented baselines. Since the central claim concerns how optimization experience should be organized and retrieved, the most convincing experiment would be a head-to-head comparison with existing memory-based methods under matched hardware, model, and budget. The cross-paper numbers reported in the paper are not comparable across hardware, baselines, timing protocols, task coverage, or correctness criteria, a caveat the authors themselves acknowledge, so the implied claim that bottleneck-indexed retrieval is superior to existing memory mechanisms remains unverified.

Questions for authors

Since the paper argues that bottleneck-indexed experience organization is superior, could you conduct a direct comparison against at least one existing memory-based method under matched hardware, model, and budget to substantiate this claim?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.