Prior Knowledge or Search? A Study of LLM Agents in Hardware-Aware Code Optimization

LLM discovery and optimization systems are increasingly applied across domains, implementing a common propose-evaluate-revise loop. Such optimization or discovery progresses via context conditioning on feedback received from an environment. In classical optimization methods, the optimizer's search state is updated, while in LLM-based systems the context is updated but the weights stay fixed. In this work we show that the rich prior encoded in an LLM's weights can be useful in some scenarios, but misleading and detrimental when prior knowledge does not transfer to the task. To demonstrate this, we develop three controlled scenarios, one is optimization over unknown functions, and two low-level code optimization tasks. We find that (1) in function optimization, LLMs act as greedy optimizers and underperform classic methods. (2) In low-level code optimization, performance sharply degrades when generating kernels for uncommon sizes and providing explicit input-size information has no measurable effect. (3) Iterative generation degrades when a less common language is used. Compiled-and-correct rates generally increase for CUDA with iterations but decrease for TVM IR. (4) In several scenarios, classical optimization methods outperform LLM-based systems, even on code optimization tasks. Our results empirically highlight a core limitation of current LLM-based optimization and discovery systems as their performance depends more on pretrained knowledge than on the accumulated context and feedback.