- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether LLM-based optimization agents can effectively exploit iterative feedback when task-specific prior knowledge is limited. It studies black-box numerical optimization and hardware-aware kernel optimization, varying programming representation (CUDA versus TVM IR), input shapes, and agent architecture. The experiments compare LLM proposers with CMA-ES, hybrid optimization, TVM MetaSchedule, and controlled parameter search using Optuna-TPE. The authors observe localized LLM search trajectories, declining proposal validity during TVM IR refinement, and repeated use of common kernel parameters across input sizes. They interpret these findings as evidence that LLM optimization depends more on pretrained knowledge than on accumulated feedback.
Critical strengths
-
The paper addresses an interesting and important question: to what extent does LLM-based optimization rely on prior knowledge versus search guided by feedback? Attempting to disentangle these contributions is valuable for understanding the capabilities and limitations of LLM optimization agents.
-
The black-box optimization perspective provides a useful analytical lens. Framing LLM agents and classical optimizers within a shared propose–evaluate–revise loop enables direct comparisons and motivates examining exploration behavior alongside final performance.
Critical weaknesses
-
The title and central conclusions overgeneralize beyond the experimental scope. Hardware-aware code optimization encompasses a broad range of tasks, hardware platforms, and optimization objectives, whereas this paper primarily studies selected GPU kernel workloads on A100 hardware. The evaluation also covers only a few model families, with different model subsets used across experiments. These results establish failure cases for the evaluated configurations, but do not justify broad conclusions that LLM agents depend more on prior knowledge than on search or accumulated feedback. This gap between the scope of the evidence and the framing of the contribution substantially weakens the paper.
-
The experiments do not separate limitations of the models from limitations of the agent designs. Whether an LLM can use execution feedback depends on how that feedback is represented, diagnosed, and incorporated into subsequent decisions. Although the paper evaluates an advisor/diagnoser pipeline and a sampling-based alternative, these cover only a narrow set of designs. More informative feedback, explicit search state, structured scheduling actions, or targeted context management could potentially improve adaptation. The observed failures therefore cannot be confidently attributed to insufficient prior knowledge or frozen model weights. Without stronger controls on feedback and agent design, the claim that these are fundamental limitations of LLM-based optimization remains insufficiently supported.
Questions for authors
- Clarification of the central claim: What evidence distinguishes insufficient prior knowledge from representation-specific difficulty, compiler constraints, or differences in the agent prompts? Given the acknowledged confounds, should the main conclusion be restricted to the observed behavior of the evaluated systems?
- Clarification of comparison fairness and reliability: How many hardware evaluations and how much tuning time does MetaSchedule receive per task? How are invalid LLM outputs handled in Figure 5? Are the reported win rates and iterative trends stable across independent optimization runs?
- The choice of decoding hyperparameters is insufficiently justified (temperature, top-p, etc). Why these settings were chosen and whether conclusions hold under alternatives