- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work introduces AutoProfiler, a framework for dynamically selecting hardware profiling metrics to guide LLM-based GPU kernel optimization. This is motivated by the observation that collecting more profiling information does not necessarily improve optimization quality and that useful metrics can vary across kernels. AutoProfiler learns a deterministic profiling tree through two LLM-guided operations: REWRITE, which introduces additional profiling metrics, and SPLIT, which partitions kernels into different profiling regimes. Candidate modifications are evaluated using downstream kernel optimization feedback, while inference-time profiling decisions require no additional LLM calls. Experiments on KernelBench report an average speedup of 1.7$\times$, outperforming several agentic baselines, with reduced LLM invocation overhead. The paper argues that learning kernel-specific profiling strategies can improve the efficiency and effectiveness of LLM-driven GPU kernel optimization.
Critical strengths
AutoProfiler introduces a feedback-driven method for learning a deterministic profiling policy, eliminating the need for additional LLM calls during profiling decisions at inference time. The separation between policy construction and deployment is appealing from a systems perspective. Furthermore, the evaluation trains the profiling policy on a separate dataset and reports improvements on KernelBench, providing encouraging evidence that the learned policy can generalize to unseen workloads.
Critical weaknesses
The evaluation does not convincingly establish how much of the reported improvement can be attributed to the proposed profiling policy. In Table 1, AutoProfiler is compared against agentic frameworks with different optimization strategies and numbers of LLM calls. Although the authors standardize the backbone model and optimization rounds, these comparisons do not isolate the contribution of adaptive profiling. The more relevant controlled comparison in Table 2 shows only a modest improvement over static profiling (1.379$\times$ vs. 1.314$\times$) on 48 kernels, without confidence intervals. Furthermore, Table 1 reports speedup only over correctly solved problems, making comparisons potentially misleading when correctness rates differ. Finally, while Figure 3 reports overall optimization time, the paper does not separately quantify profiling overhead or account for the offline cost of constructing the profiling policy. These limitations make it difficult to assess the actual performance and efficiency benefits of the proposed approach.
Questions for authors
As LLMs become increasingly capable of understanding GPU kernel performance, stronger models may already be able to identify relevant profiling metrics or dynamically decide what to profile without requiring a separately trained profiling policy. Have the authors considered comparing AutoProfiler against an LLM that directly selects profiling metrics based on the kernel code and previously collected hardware information? Such a comparison would help establish whether the learned profiling tree provides meaningful benefits beyond the capabilities of modern LLMs, particularly when accounting for the cost of constructing the policy.
Response to the Weaknesses and Questions
Thank you for your detailed comments on the experimental evaluation. We have conducted additional controlled experiments and analyses to clarify the contribution of adaptive profiling, the comparability of evaluation results, and the associated profiling costs. We address these concerns below.
(1) Isolating the contribution of adaptive profiling.
To directly assess the contribution of our profiling policy, we conducted an additional controlled experiment using KernelAgent, which employs a fixed static profiling configuration. Specifically, we replaced only its static profiling component with the profiling tree learned by AutoProfiler, while keeping the rest of the optimization framework unchanged. Under static profiling, KernelAgent achieves a speedup of 1.410×. With AutoProfiler's adaptive profiling, the speedup increases to 1.587×, corresponding to a 12.6% relative improvement.
This controlled comparison provides direct evidence that the learned profiling policy improves downstream kernel optimization, beyond the effects of different agentic optimization frameworks.
(2) Response to the Comparability of Speedup Results under Different Correctness Rates
We follow the common practice in LLM-based kernel optimization of reporting speedup only for kernels that pass correctness verification, as performance improvements are meaningful only for functionally correct implementations. To provide a more comprehensive evaluation, we therefore report both correctness and speedup to provide a comprehensive assessment of each method. Notably, AutoProfiler achieves the highest correctness and speedup among all compared methods, demonstrating that its performance gains do not come at the expense of correctness.
(3) Response to Profiling Overhead and Offline Policy Construction Cost
Offline Profiling Policy Construction Cost. We quantify the offline cost of constructing the profiling tree below. Tree construction takes 28.2 hours of wall-clock time and consumes 2.98M tokens. This is a one-time offline cost: the resulting tree is frozen and reused across kernel optimization tasks, without requiring reconstruction for individual kernels.
Runtime Profiling Overhead and Optimization Efficiency. AutoProfiler incurs 21.5 s of profiling time per task, comparable to 23.8 s with static profiling. Thus, its efficiency advantage does not come from reducing profiling overhead, but from providing more informative profiling signals for downstream LLM-based optimization. These signals reduce unnecessary LLM calls and shorten optimization iterations. As shown in Figure 3, AutoProfiler achieves higher speedups than the baseline under a comparable cumulative wall-clock budget, demonstrating the optimization benefits of its adaptive profiling policy.
(4) Response to Whether Direct LLM Reasoning Can Replace a Learned Profiling Policy
To investigate whether a capable LLM can directly determine effective profiling metrics without a separately learned policy, we conducted an additional experiment using GPT-5.6-Sol as a profiling decision-maker. At each round, the LLM selects at most six profiling metrics based on the kernel code and profiling results collected in previous rounds, iteratively proposing additional metrics over three rounds. The metrics selected across all three rounds are then collected and used for kernel optimization under the same optimization procedure as AutoProfiler. We evaluate both approaches on 50 kernels. AutoProfiler achieves a speedup of 1.642×, whereas direct LLM-based profiling reaches only 1.441×. This result indicates that directly reasoning about profiling metric selection with a strong LLM does not match the effectiveness of the learned profiling tree, highlighting the value of learning a dedicated profiling policy for downstream kernel optimization.