- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
EvokeAgent is an LLM-based system for GPU kernel optimization, combining feedback loop with an evolving database of optimization policies indexed by the profiler-diagnosed bottleneck. Its central mechanism uses diagnosed bottlenecks to select relevant profiler measurements and retrieve optimization strategies. Successful strategies can be stored for reuse across tasks, with separate CUDA and Triton knowledge bases. Evaluated on 250 KernelBench tasks using an NVIDIA H20-3e and GLM-5.2, it reports CUDA geometric-mean speedups of 1.93×, 2.65×, and 2.94× across three levels. and Triton geometric-mean speedups of 1.68×, 2.56×, and 1.64×. Ablations support the combined design but do not isolate individual contributions; external comparisons lack matched experimental conditions. Moreover, the idea seems incremental over KernelBlaster
Critical strengths
EvokeAgent uses the diagnosed bottleneck to select relevant NCU measurements and retrieve targeted optimization strategies, giving the LLM both diagnostic evidence and actionable guidance. Its implementation across CUDA and Triton demonstrates applicability beyond a single kernel language.
Critical weaknesses
-
Details of the evolution process are insufficiently documented. The KB starts with manually curated policies and grows during evaluation, but the overlap between human written seeds and the tasks are unclear. The order of KernelBench tasks could affect the KB construction but I did not find the analysis of this.
-
External comparisons use different hardware, LLMs, budgets, and evaluation protocols. Although they can serve as useful contextual information, they cannot establish EvokeAgent's superiority over close methods such as KernelBlaster. The main results also require clarification. For example, the maximum 1093.11× speedup lacks a independent validation. This outlier is not proof of reward hacking, but warrants inspection of the generated code, correctness coverage, and timing.
Questions for authors
- Could you provide ablation evidence to attribute gains specifically to retrieved policies, selection algorithm, or their coupling? What about using other algorithms like MAP-Elite as the selection algorithm?
- Could you provide break down of retrieved and/or adopted policies by manual seeds versus newly generated ones? The inventory counts do not suggest retrieval or usage counts
- Could you identify the tasks involving suspicious speedup like the one with 1093.11× speedup? This will affect the assessment of what the LLM actually learns.
- Can a close competitor such as KernelBlaster be evaluated using the same LLM, GPU, tasks, and budget to establish the comparative benefit?
- Since the KB are shared across the tasks, how does the order of tasks affect the final speedup?
- How were the thresholds in the two-level bottleneck classifier chosen (e.g., 60% memory/SM throughput and 40% occupancy in Table 11, and the sub-category cut-offs in Tables 12–13)? Does “calibrated for the H20-3e” mean they were tuned on KernelBench problems, and how sensitive are the results to changing them?