acceptodds
Withdrawn from ICLR 2027

Prior Knowledge or Search? A Study of LLM Agents in Hardware-Aware Code Optimization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

LLM discovery and optimization systems are increasingly applied across domains, implementing a common propose-evaluate-revise loop. Such optimization or discovery progresses via context conditioning on feedback received from an environment. In classical optimization methods, the optimizer's search state is updated, while in LLM-based systems the context is updated but the weights stay fixed. In this work we show that the rich prior encoded in an LLM's weights can be useful in some scenarios, but misleading and detrimental when prior knowledge does not transfer to the task. To demonstrate this, we develop three controlled scenarios, one is optimization over unknown functions, and two low-level code optimization tasks. We find that (1) in function optimization, LLMs act as greedy optimizers and underperform classic methods. (2) In low-level code optimization, performance sharply degrades when generating kernels for uncommon sizes and providing explicit input-size information has no measurable effect. (3) Iterative generation degrades when a less common language is used. Compiled-and-correct rates generally increase for CUDA with iterations but decrease for TVM IR. (4) In several scenarios, classical optimization methods outperform LLM-based systems, even on code optimization tasks. Our results empirically highlight a core limitation of current LLM-based optimization and discovery systems as their performance depends more on pretrained knowledge than on the accumulated context and feedback.

Primary area:
generative models
Keywords:
LLM agents, agent behavior analysis, failure modes, exploration and exploitation, black-box optimization, prior knowledge, distribution shift, GPU kernel optimization, code generation, iterative feedback

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_CDpWOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether LLM-based optimization agents can effectively exploit iterative feedback when task-specific prior knowledge is limited. It studies black-box numerical optimization and hardware-aware kernel optimization, varying programming representation (CUDA versus TVM IR), input shapes, and agent architecture. The experiments compare LLM proposers with CMA-ES, hybrid optimization, TVM MetaSchedule, and controlled parameter search using Optuna-TPE. The authors observe localized LLM search trajectories, declining proposal validity during TVM IR refinement, and repeated use of common kernel parameters across input sizes. They interpret these findings as evidence that LLM optimization depends more on pretrained knowledge than on accumulated feedback.

Critical strengths
  • The paper addresses an interesting and important question: to what extent does LLM-based optimization rely on prior knowledge versus search guided by feedback? Attempting to disentangle these contributions is valuable for understanding the capabilities and limitations of LLM optimization agents.

  • The black-box optimization perspective provides a useful analytical lens. Framing LLM agents and classical optimizers within a shared propose–evaluate–revise loop enables direct comparisons and motivates examining exploration behavior alongside final performance.

Critical weaknesses
  • The title and central conclusions overgeneralize beyond the experimental scope. Hardware-aware code optimization encompasses a broad range of tasks, hardware platforms, and optimization objectives, whereas this paper primarily studies selected GPU kernel workloads on A100 hardware. The evaluation also covers only a few model families, with different model subsets used across experiments. These results establish failure cases for the evaluated configurations, but do not justify broad conclusions that LLM agents depend more on prior knowledge than on search or accumulated feedback. This gap between the scope of the evidence and the framing of the contribution substantially weakens the paper.

  • The experiments do not separate limitations of the models from limitations of the agent designs. Whether an LLM can use execution feedback depends on how that feedback is represented, diagnosed, and incorporated into subsequent decisions. Although the paper evaluates an advisor/diagnoser pipeline and a sampling-based alternative, these cover only a narrow set of designs. More informative feedback, explicit search state, structured scheduling actions, or targeted context management could potentially improve adaptation. The observed failures therefore cannot be confidently attributed to insufficient prior knowledge or frozen model weights. Without stronger controls on feedback and agent design, the claim that these are fundamental limitations of LLM-based optimization remains insufficiently supported.

Questions for authors
  • Clarification of the central claim: What evidence distinguishes insufficient prior knowledge from representation-specific difficulty, compiler constraints, or differences in the agent prompts? Given the acknowledged confounds, should the main conclusion be restricted to the observed behavior of the evaluated systems?
  • Clarification of comparison fairness and reliability: How many hardware evaluations and how much tuning time does MetaSchedule receive per task? How are invalid LLM outputs handled in Figure 5? Are the reported win rates and iterative trends stable across independent optimization runs?
  • The choice of decoding hyperparameters is insufficiently justified (temperature, top-p, etc). Why these settings were chosen and whether conclusions hold under alternatives
Reviewer_J4ozOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper performs an empirical analysis of whether LLMs are good at discovery through their ability to search or because they have prior knowledge of the search space. The paper evaluates this by performing multiple studies: 1) Comparing LLMs against Black box optimizers and 2) Comparing kernel code optimization capabilities across 2 different languages (CUDA and TVM-IR) and a shape bias study on CUDA.

Critical strengths

Using entropy as a metric for prior knowledge was an interesting strategy since usually the datasets are proprietary. The shape bias results are also interesting, and they also motivate further study in the future.

Critical weaknesses
  1. Experiment 1 does not vary the prior knowledge for the different problems so the main takeaway is that LLMs tend to prefer greedy search and how greedy search compares to traditional optimizers (the paper's own related works section mentions Ferreira et al. (2026) and Zhang et al. (2026) which deal with nearly the same problem but the paper doesn't explain how it differs from these). For experiment 2, in the limitations section, the paper acknowledges that "...preventing a causal attribution to prior familiarity alone" but still concludes that the effects observed were from difference in prior knowledge. CUDA is translated from pytorch while TVM starts from an unscheduled program for same pytorch (different tasks) and TVM is forced to make at least one change while it’s unclear whether CUDA was given the same constraints which diminishes the degradation results. Also, they use different pipelines: TVM is analysis, code, summary while CUDA is advisor/diagnoser, coder (different prompts even excluding language specific instructions). This implies prior knowledge was not truly isolated which diminishes the main claim of the paper.
  2. Figures 6 and 13 score each new candidate, although the loop retains the best program. Declining proposal validity does not establish worse final solutions. Moreover, all 12 TVM trajectories contain intermediate increases, contradicting the claim that correctness “decreases at each step”.
  3. The paper’s evaluation budget and tuning time are unspecified, while the LLM receives at most five iterations. MetaSchedule also selects scheduling transformations, whereas the LLM rewrites TVMScript and must preserve syntax and semantics. Figure 5 leaves candidate selection, failures, and denominators unclear, preventing attribution of its win rates to superior search under comparable conditions.
  4. Figure 7 reports cumulative feedback improvement but scores only sampling’s sixth, post-aggregation candidate. It does not clarify whether both methods return their best valid candidate or how the first five samples affect the final score. This comparison therefore does not establish a general advantage of sequential feedback over parallel sampling.
  5. Section 2 compares classical optimizers’ evolving search state with LLMs’ fixed weights. Yet classical update rules are also fixed, while LLM context provides evolving task-specific state. Frozen weights alone therefore do not explain exploration failures or establish an inability to adapt from feedback.
Questions for authors

Please address the weaknesses with additional experiments or explanations.

Reviewer_53kBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether LLM propose–evaluate–revise loops learn from feedback or mostly replay pretrained priors, framing them as black-box optimization with frozen weights (§2). With 50 evaluations, three open LLMs beat CMA-ES on synthetic "Functions" (82–18 average wins) but lose on "Physical" tasks (40–60) and BBOB-5D (13–35); low coverage and straight trajectories are read as greedy search (Table 1). On 200 KernelBench tasks, a ≤5-iteration feedback agent mostly raises CUDA correctness (43 → 73%, DeepSeek-V4-Pro, Base L2) but lowers TVM-TIR rates (65 → 35%, gpt-oss-120b, Small L2; Fig. 6), and MetaSchedule beats the LLM's TVM loop on 65–71% of reduced-size kernels (Fig. 5). Recurring launch and tile parameters cost up to 1.76× (Table 8), and Optuna-TPE wins 14 of 18 fixed-kernel searches (Table 9).

Critical strengths

S1. The question is timely, and the paper tests against classical optimizers (CMA-ES, Optuna-TPE, MetaSchedule), not only LLM variants. Some probes are well controlled: App. K fixes the kernel code to remove generation validity as a confound (Table 9), and App. J tests whether the recurring parameters are universal defaults (Table 8).

S2. Contrasting CUDA with TVM-TIR on the same computation, via an automatically generated unscheduled TIR reference (App. L), is a distinctive way to vary representation familiarity. It is supported by Stack token counts (Fig. 4) and by higher TVM-IR generation entropy on all seven operations under both models (Table 2). Prompts and settings are fully disclosed (Apps. D, G, H, M, N).

Critical weaknesses

W1. Baseline budgets and statistics are missing: MetaSchedule runs with "default settings" (p. 6) and no stated trials or tuning time against at most 10 LLM calls (Table 12), so the "roughly 70%" result may reflect budget rather than prior; the Small panels of Fig. 5 appear to show far fewer than 200 kernels with no inclusion rule; and Tables 1, 3, 4, 9 and Fig. 6 report no repetitions or tests. Suggestion: report trials, wall-clock and cost, add a matched-budget run, count failures explicitly, and use ≥ 3 seeds with CIs.

W2. The CUDA-versus-TVM comparison is confounded by starting point (correct TIR versus translating PyTorch), pipeline and prompts (App. M vs. App. N), and the rule that the TVM coder must "never return code unchanged" (p. 28); Fig. 6's TVM decline appears to track the latest candidate rather than the best-so-far program (Fig. 12). The prior proxies also disagree (SYCL and Triton are rarer than TVM in The Stack, Fig. 4, yet have lower entropy, Fig. 9), and contribution 2 is phrased causally despite the caveat on p. 9. Suggestion: use one pipeline, allow no-op iterations, report best-so-far rates, and intervene on the prior (e.g., retrieved TVM examples).

W3. The "sharply degrades" claim (abstract) conflicts with the paper's own data: pass rates are "essentially flat" across sizes (p. 9), TVM correctness is higher at Small than at Base (e.g., 62% vs. 28%, Fig. 6), and canonical parameters cost 1.66–1.76× only at large shapes versus 1.01–1.03× at the smallest (Table 8). "Three values cover up to 78%" sums overlapping TILE_K/M/N shares (25.5–27.5% each, Table 7), and the "rarely encountered" sizes (p. 1) are not measured. Suggestion: report joint configuration frequencies, per-shape optimality gaps and Stack shape counts.

W4. The optimization design limits the conclusions: minimizing |F| yields continua of zero-loss minima rather than "many local minima" (App. C, p. 4); the BBOB prompt names the benchmark despite "never told which function" (pp. 3, 16); and the greedy probe uses a unimodal sphere (p. 5). Table 1 also contradicts the text: the LLM beats CMA-ES on BBOB-2D (26–22), DeepSeek-V4-Pro's Functions coverage (70.9%) exceeds CMA-ES's (44.8%), and LLM-MCTS loses to the plain LLM in all four families. Suggestion: report final regret, remove the benchmark name, use a multimodal probe, and compare Centaur with CMA-ES directly.

W5. The optimization findings largely restate cited work (LLMs as local refiners, Zhang et al., 2026; CMA-ES beating LLM agents, Ferreira et al., 2026; p. 14), and §2 adds notation rather than predictions; TVM, Ansor, MetaSchedule and the TPE paper are uncited although MetaSchedule and Optuna-TPE serve as baselines. Suggestion: state the delta over these works and cite the autotuning literature.

Questions for authors

Q1. How many measured trials and how much wall-clock did MetaSchedule use per kernel? How do the Fig. 5 percentages change at a matched evaluation budget, and how are incorrect LLM kernels counted?

Q2. Are the iteration-5 rates in Fig. 6 latest-candidate or best-so-far? Does the TVM decline persist without the "status must be 'improved'" constraint (pp. 28–30) and with identical agent roles?

Q3. What fraction of Matmul kernels set TILE_M = TILE_N = TILE_K = 32 jointly? What are the explicit- and implicit-prompt frequencies separately, given that App. I calls them "identical"?

Q4. How many seeds per optimization problem, and how are near-ties scored? Do the conclusions hold with mean ± std final regret, a prompt without the benchmark name, and a multimodal probe?

Q5. Does an intervention on the prior, such as retrieving TVM examples or fine-tuning on TVM, reverse the feedback-loop degradation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.