acceptodds
Under review as a conference paper at ICLR 2027

Better Feedback, Simpler Agents: Rethinking Agentic Code Optimization

Abstract

Modern coding agents are often made more capable by adding specialized agents, tools, memory, and additional reasoning stages. But does adding more machinery actually produce better optimized code? We study this question systematically in code optimization, where an agent must discover faster implementations while keeping the optimization process affordable. We develop a controlled \system with 24 workflows spanning four design choices: direct versus delegated coding, raw versus analyzed execution feedback, optional tools and skills, and how earlier attempts are stored. We evaluate these workflows with GPT-OSS-20B, GPT-OSS-120B, and Qwen3.8-27B on 54 NPBench CPU kernels. The trade-offs are hard to anticipate: more elaborate workflows do not consistently perform better, and most components help in one setting while costing tokens or speed in another. The clearest gain comes from what the main agent receives between attempts: an analysis agent turns execution results into concise guidance, and each turn starts from a compact context. Across both coding modes, analyzed feedback makes programs 1.15-1.46 faster for all three models; with direct coding, the GPT-OSS models gain 1.47-1.51 and use 25-55% fewer tokens, although searches take longer. With analyzed direct coding, tools and skills raise token use by 36-134% without a significant speed-up, and history helps GPT-OSS direct coding more than delegated coding. The configuration with every component enabled uses 54-146% more tokens than direct coding with analysis and history, again without a significant speed-up. On three further models, analyzed feedback helps both Qwen3.5 models and Kimi-K2.7-Code's direct coding, while the value of the other components changes: Qwen3.5-397B, unlike GPT-OSS, gains substantially from delegation. Our results show that code-optimization performance does not grow monotonically with workflow complexity: simpler combinations can produce faster code with fewer tokens.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM agents, code optimization, agentic workflows, multi-agent systems, execution feedback, context management, tool use, performance engineering, token efficiency

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_LTeMOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how common workflow design choices affect program speed and search cost in agentic code optimization. The authors develop a controlled, configurable framework that combines four design choices—direct versus delegated coding, raw versus analyzed execution feedback, tools and skills, and history—into 24 workflows under a common task contract and evaluation protocol. The main experiments evaluate all 24 configurations with three models on 54 NPBench CPU kernels, with additional transfer tests on three further models using smaller fixed panels. The evaluation measures achieved program speed-up, token cost, and search time to examine the benefits of different design choices and how they depend on the model and the surrounding configuration. The results show that increasing workflow complexity does not consistently improve the trade-off between program speed and search cost. Analyzed feedback, combined with rebuilding the lead agent’s context at each turn, provides relatively consistent performance gains, while the benefits of delegated coding, tools and skills, and history depend more strongly on the specific setting.

Critical strengths

In my view, one of the paper’s main strengths is the systematic design of its experimental framework. I particularly appreciate how the authors bring coding mode, feedback, tools and skills, and history into a common optimization loop, with the same task contract and evaluator across the 24 workflows. The framework uses a shared controller to generate, validate, and benchmark candidates, while allowing changes to who writes the code, how execution feedback reaches the lead agent, and how earlier attempts are revisited. What I find especially useful is that the configuration design supports matched comparisons across different combinations of these choices. For example, it allows the authors to examine whether history that helps direct coding remains useful under delegated coding. Compared with ablations conducted only around a single full configuration, this provides broader coverage of how component benefits depend on the surrounding workflow and the underlying model. I also appreciate the joint evaluation of achieved program speed-up, token cost, and search time, including token cost by role. This helps assess whether the gains from additional components justify their search cost and gives the study practical value for workflow design.

Critical weaknesses

My concerns fall under two critical weaknesses: unquantified variability across independent optimization searches, and insufficient verifiable evidence connecting workflow configurations, task requirements, and the code changes that produce the reported gains. To make the priorities of this review explicit, I distinguish six points:

  1. Proprietary deployment models: an optional extension, with implications for the scope of the claims. I would find exploratory tests valuable, but I do not require them. I do expect a clear distinction between what has been directly validated and what remains untested in deployed coding-agent systems.
  2. Repeatability across independent optimization searches: an expected response. The authors need to address variability in generated code, optimization behavior, correctness, and final gains.
  3. Optimization trajectories and workflow configurations: an expected response. I need an analysis connecting configuration changes to the actual sequence of candidate-level code changes.
  4. Task requirements, code optimization, and workflow configurations: an expected response. I need a joint analysis establishing which component capabilities the workloads exercise and how this relates to the observed effects.
  5. Optimization in a real software repository: an optional extension. This would be an interesting additional case study. Its absence does not mean a change to the current claims or count against the paper in this review.
  6. Complete evidence from actual historical workflow executions: an expected response. I expect auditable records linked to representative runs underlying the reported results.

The paper provides a substantial comparison of final performance and resource consumption across workflows. However, a critical layer of evidence is missing from the step between these results and practical workflow-design recommendations: how different models use feedback, history, tools and skills, and delegation on different tasks during the optimization loop; how this use translates into code changes; how differences in workflow components correspond to differences in optimization behavior; and whether the experiments establish that these behaviors and gains are repeatable, meaningfully test the relevant component capabilities, and support consequential design recommendations. Insufficient analysis of generation variability, weak connections between task requirements and code changes, and incomplete disclosure and analysis of trajectories in relation to workflow components jointly limit the transferability and actionability of the recommendations. The absence of validation on proprietary deployment models raises a separate question about their scope. If these gaps in evidence and analysis were addressed, I would be much more confident in treating this paper as a reliable empirical basis for future code-optimization agent design and LLM coding-agent development. Unfortunately, the current evidence and analysis do not convince me that it has reached that point. I strongly encourage the authors to address these gaps so that I can assess the practical significance of this work more positively.

Regarding scope, I encourage the authors to distinguish more explicitly between validated configuration comparisons and deployment recommendations that remain untested. I recognize that the paper already demonstrates model dependence, particularly the substantially different gains from delegation observed for Qwen3.5-397B compared with GPT-OSS. However, these experiments do not establish whether the practical recommendations hold for proprietary deployment models, and this matters for the scope of the paper's advice. I support the use of open-weight models for workflow research: they facilitate version control and deeper analysis. The key question for me is whether the finding that code-optimization performance does not grow monotonically with workflow complexity, and the recommendations about the order in which components should be adopted, continue to hold across a broader range of frontier models. I understand that extending these findings to proprietary models is not the central focus of the study. Nevertheless, it is relevant to the applicability of its recommendations. I would welcome a qualitative assessment of whether programmers using models such as GPT-6 Astra, Claude Opus 5.5, GPT-6.1 Sol, and Claude Fable 5.1 should adjust their code-optimization workflows in light of these results, and whether developers of coding-agent products, including Claude Code, should adjust their built-in workflows. Broader coverage could also include additional open-weight models such as DeepSeek-V4-Pro. That would address a complementary question rather than provide proprietary-model validation. As the authors note, differences in how models use feedback, history, tools and skills, and delegation may materially change the value of these components. The full-suite experiments currently cover only three open-weight models, while the additional models are evaluated on smaller panels selected using GPT-OSS results. Readers therefore still cannot determine how much of the observed component benefit reflects transferable workflow principles and how much depends on the particular behavior of the evaluated models. The recommendation to start with analyzed feedback, choose the coding mode according to the model, and add tools and skills last has no direct or behavioral validation on proprietary coding models. I am therefore not yet convinced of its value for these everyday deployment settings or for developing agents intended to operate in them. If resources permit, I encourage exploratory tests on a small set of representative tasks and configurations, with explicit model versions, prompts, budgets, repeated searches, and clearly stated limits on applicability. Even then, I would treat the results as informative but limited transfer evidence: changes or uncertainty in model versions, serving conditions, sampling, and potentially request routing would make broad generalization difficult. That evidence could still provide valuable practical intuition. If the authors choose not to conduct these experiments, I fully understand, and that choice alone will not count against the paper. I do, however, expect the recommendations to state explicitly that direct validation is currently limited to the evaluated open-weight models and experimental conditions, while applicability to untested deployed coding-agent systems remains open.

Proprietary-model testing is an extension. The following concerns are central to my assessment, and I strongly expect detailed responses and, where needed, additional experiments. My first major reservation is that variability across independent optimization searches under the same configuration has not been quantified. My concern is primarily about the process of generating and modifying code, rather than the timing protocol. Section 3 explicitly states that each model-kernel-configuration combination receives one search, and all models use a temperature of 0.7. Each result therefore represents one optimization search with stochastic sampling and its associated candidate selection; the distribution of outcomes across independent searches under the same configuration has not been characterized. The authors also explicitly acknowledge that the kernel bootstrap does not measure variability between repeated optimization searches. Repeated timing and checks using mean versus median runtimes likewise cannot replace independent searches that regenerate candidates. These checks clarify sensitivity to workload composition, timing variation for fixed candidates, and the choice of aggregation statistic, respectively. They still do not quantify the probability and distribution of finding useful optimizations, passing correctness checks, and obtaining performance gains when the same configuration is rerun independently. Because later candidates depend on earlier generated content and feedback, different sampled outputs may alter subsequent attempts; the current single-search design cannot determine how much this affects the configuration comparisons. Beyond repeated searches with fixed prompts, semantically equivalent prompt paraphrases on a small set of representative tasks would provide an additional sensitivity check. Prompt paraphrasing is optional. Independent repeated searches are the central requirement. Appendix J describes joint accumulation across output channels and 4 x 4 spatial tiling for convolution, as well as parallel trailing-matrix updates in the LU factorization stage of ludcmp, and reports performance differences between the corresponding final programs. These examples do not establish how frequently the strategies are discovered and correctly implemented across repeated searches, or quantify the effects of small changes in generation. As the authors acknowledge at the end of Appendix J, each configuration contributes one search, so the examples describe selected programs rather than the probability that a component discovers a particular strategy. This acknowledged limitation still requires targeted repeated experiments. I ask the authors to conduct independent searches on a preselected subset of representative tasks and key configurations, jointly reporting the distribution of final performance, the frequency of useful strategies, rates of correct implementation and successful repair, and whether subsequent candidates retain previously effective transformations. What needs to be established is whether a workflow reliably improves the distribution of final outcomes across independent searches and increases the success rate of useful optimization behavior. I am not asking for identical code transformations or search paths in every run. Timing variation for fixed programs has already received substantial attention; variation in final outcomes and behavior across independent searches still needs to be quantified.

I recognize the paper's joint comparison of speed-up, correctness, token use, and search time, as well as its per-kernel results, context and role-level cost statistics, and Appendix J's analysis of final implementation differences. My second major reservation is that these analyses do not adequately establish how the benefits of different configurations arise through concrete code changes, or whether these explanations hold across models and tasks. The framework requires the model to identify bottlenecks, choose optimization directions, implement transformations, preserve correctness, and use feedback to decide whether to continue, revise, or abandon an attempt. Final correctness, the speed of the selected program, and total token use cannot distinguish these stages. Existing process statistics provide useful clues, but they have not been systematically connected to candidate-level code changes. For example, a component might help discover a new effective strategy, primarily reduce repair or repeated exploration costs, or help later candidates retain and combine previously effective transformations. These effects also need to be distinguished from the controller's own mechanism for retaining the best measured valid candidate. They have different implications for workflow design, yet the current analysis does not adequately distinguish them using candidate code and the feedback actually received. The available clues have not been developed into verifiable patterns of code modification across models and tasks. Figures 9 and 10 show that component effects vary with the kernel, model, and coding mode, but these differences have not been systematically linked to specific transformations, failure types, and feedback use. The few final-program examples in Appendix J do not establish that connection: they are comparisons of selected implementations, not an analysis of the optimization processes that produced them. For representative models, tasks, and configurations, I ask for a verifiable classification of optimization transformations, correctness repairs, repeated attempts, and reversions, using each candidate and its actual base implementation as the unit of analysis, and linking these actions to the model, task, workflow, and evaluation outcome. For the same model and workload under different workflows, this should include code diffs for every candidate in each turn relative to its actual base, so that readers can inspect where and how the code changed. Readers could then assess which behaviors produce the observed configuration advantages, whether they repeatedly yield gains in independent searches, and under what conditions they work. I accept execution-based evaluation of candidate performance and the evidence that matched configuration comparisons provide about overall component benefits. What remains insufficient is a verifiable explanation of how those benefits arise. Candidate source code, the context actually supplied, and targeted controls where needed to distinguish mechanisms would help fill that gap.

I am equally concerned about whether the requirements of the experimental tasks make particular components especially valuable, or leave little opportunity for them to contribute. Data layout, dependency structure, numerical constraints, and the available optimization space can all affect the usefulness of feedback, history, tools and skills, and delegation. The paper still lacks a systematic analysis connecting these task properties to the observed configuration differences and establishing whether the workloads adequately test the capabilities in question. Table 4, the reference implementations, the full results matrix, and the evaluation protocol help readers inspect the workloads and measurements. However, task names, reference-code line counts, and final speed-ups are not enough to characterize the optimization reasoning and component capabilities that each task requires. Appendix J provides useful explanations of performance differences between final programs, but does not trace those differences through the actual optimization process. For example, which feedback or tool evidence led to convolution tiling or parallel updates in LU factorization? At which turn were they implemented? Did they require repair? Were they attempted unsuccessfully under other configurations? This is currently unclear. More fundamentally, can these transformations be connected to a component's particular contribution, to the changes it makes to the model's available context, or to an interaction between the two? This distinction is especially relevant because analyzed feedback combines an analysis report with a compact-context rebuild by design. A comparison of this combined setting does not, by itself, identify the contribution of each mechanism. One possibility is that some workloads place little demand on particular component capabilities. If so, using those workloads to draw general conclusions about those capabilities without establishing that they were meaningfully exercised would be methodologically unsound. The paper does not provide the joint analysis of optimization behavior, search trajectories, task properties, and component configurations needed to examine this possibility and substantiate the proposed explanations of component effects. Task simplicity is not itself a problem. However, if the goal is to explain a component's role, an account of the final optimized code is no longer sufficient. The connection between optimization behavior and configuration must be examined jointly with task requirements, so that readers can distinguish an unhelpful component from a workload that offers little opportunity to use it, and understand their interaction. As a separate, optional extension, I would be very interested in optimization within a real repository or a larger codebase. In my view, such settings would be particularly valuable testbeds for broader advice about coding agents and would be closer to everyday use. They often create substantial opportunities to use feedback, history, tools and skills, and delegation, potentially allowing more informative comparisons of their roles. I am not making such a case study a requirement for the current claims or my assessment. My concern is whether the present tasks, trajectories, and code changes support the component-level interpretations being offered. The current cases do not yet allow me to determine whether changes in components produce stable patterns of optimization behavior, or how those patterns would carry over to practical use.

I strongly value the effort to organize code-optimization workflows within a unified experimental framework, the discussion of their key components, and the quantitative analysis of program speed, token consumption, and search cost. Comparing program performance jointly with the cost of finding the program has clear practical value. My main reservation is that the current evidence does not adequately support the paper's positioning as a contribution that moves from configuration comparisons to workflow-design principles. The paper shows that configurations produce different outcomes and that those outcomes depend on the model and task. It does not yet sufficiently explain how the differences arise through actual code changes, whether they are repeatable, how workflow changes correspond to modification patterns, whether the workloads meaningfully exercise the component capabilities being assessed, or how these observations transfer to other deployment models and task requirements. Adding model names, listing more tasks, or further decomposing resource costs cannot by themselves resolve these questions. I would certainly welcome broader open-weight model coverage, more repeated searches, more detailed analysis, and more comprehensive workloads. What is needed is a connected body of evidence: observe how components are used across the evaluated models and configurations; establish whether useful optimization behavior and final gains are repeatable across independent searches; connect task requirements to actual code transformations; and use the actual prompts and complete trajectories to document the progression from information to action to outcome. Additional open-weight experiments and limited exploratory proprietary-model tests could broaden that evidence, but proprietary-model tests remain optional. The appendices provide a starting point, and the code artifact supplies prompt templates and their construction logic. However, the supplied material does not provide a complete account of the actual optimization histories for a given task and model under different configurations: from the initial prompt and the context supplied to each model call, through feedback and component use, to candidate changes, their actual bases, correctness and repair outcomes, candidate selection, and the final code transformation. I specifically expect a report of actual historical executions that contributed to the paper's reported results, including a clearly identified model-task-configuration run underlying a main-table result. A generic description of how the harness operates, or a new illustrative run without a demonstrated connection to the reported results, would not meet this request.

I therefore regard these gaps as substantive weaknesses affecting the paper's main contribution. Even if all the reported performance numbers are accurate, I still find it difficult to determine which workflow to choose for a new task, why to choose it, and which lessons remain trustworthy after changing the model. I need the authors to establish what bounded, testable, and transferable design knowledge the paper provides through evidence on independent-search repeatability and the connections among task requirements, trajectories, code changes, and workflow configurations. I expect a direct and substantive response to each of the four expected points: independent repeated searches; the connection between configurations and candidate-level optimization trajectories; the joint analysis of task requirements, optimization behavior, and component effects; and complete, auditable historical execution evidence linked to the reported results. General summaries or another description of final implementations would not be sufficient. A convincing response, supported by specific evidence and analysis, would improve my assessment of the paper, strengthen my confidence in the authors' interpretation of its contribution, and lead me to raise my rating. If these issues remain unanswered, or the response does not provide the evidence needed to examine them, my assessment will become less favorable and I will lower my rating accordingly. The optional proprietary-models and real-repositories extensions are separate from these requirements. Declining to undertake them will not itself count against the paper. I have high expectations for the ideas and potential of this work, and I would sincerely appreciate a response that resolves these concerns and gives me a sound basis for a more positive recommendation.

Questions for authors

What do the authors see as the core value of this work, and how should readers understand the scope of its conclusions?

I would appreciate a clearer statement of the paper’s intended contribution. Is the main aim to develop an empirical understanding of code-optimization workflows for the evaluated open-weight models under controlled conditions, or to provide guidance for everyday code optimization and the development of LLM coding-agent systems? If both are intended, please distinguish what the current evidence establishes from the broader practical value the authors hope to offer.

This distinction matters to me because the paper goes beyond comparing configurations and offers recommendations about adopting analyzed feedback, choosing the coding mode, and adding tools and skills. I would like to understand what the authors believe readers should take away from these recommendations. Which decisions can users and developers reasonably make based on the present results, and which lessons should still be treated as hypotheses to test when the model, workload, or environment changes? I recognize that the paper already discusses model dependence, but I would welcome a more explicit connection between those limitations and the scope of its advice.

My expectations are high because I see considerable potential in this work. If the authors can connect task requirements, feedback, component use, candidate-level code changes, and final outcomes into a complete and verifiable account of the optimization process, and establish that the useful behaviors and gains recur across independent searches, the paper would provide a much stronger basis for workflow design. If its key conclusions also hold in exploratory tests with proprietary models and real software repositories, it could become an important empirical foundation for future research and engineering practice.

At the same time, I would not regard a more focused contribution as inherently low-impact. A study of the evaluated open-weight models, a fixed experimental environment, and a bounded set of clearly specified workloads, with some comparisons conducted on selected panels, can still be valuable. If this is the intended scope, I would simply ask the authors to make that positioning explicit and keep the recommendations within the boundaries of the evidence, so that conditional findings are not read as generally applicable guidance.

With this positioning clarified, I would appreciate a direct, point-by-point response to the six questions already raised in my weaknesses. Points 2, 3, 4, and 6 are central to my assessment. Additional experiments under Points 1 and 5 are optional, and their absence alone would not count against the paper. I would nevertheless welcome the authors’ views on the corresponding limits of applicability.

1. Applicability to proprietary deployment models. How do the authors expect the findings to inform programmers using proprietary models and developers of coding-agent systems? Please distinguish directly supported recommendations from untested expectations. If resources permit, could a small exploratory evaluation help clarify this boundary?

2. Repeatability across independent optimization searches. Could the authors conduct independent repeated searches on a preselected subset of representative tasks and key configurations, reporting final performance distributions, correctness and repair success, and how consistently useful strategies are discovered, implemented, and retained? Repeated timing of fixed programs does not address this question.

3. The connection between workflow configurations and optimization trajectories. Could the authors use candidate-level code changes, measured relative to each candidate’s actual base implementation, to explain how configurations affect strategy discovery, repair, repeated exploration, and reversions—and how these behaviors produce the reported gains?

4. The relationship among task requirements, code optimization, and component effects. Could the authors jointly analyze task properties and actual optimization trajectories to distinguish an unhelpful component from a workload that offers little opportunity to use it? Please also clarify the attribution limits of combined mechanisms, particularly analyzed feedback and context rebuilding.

5. Applicability to real software repositories. How do the authors view the transfer of these findings to optimization in real repositories or larger codebases? An exploratory case study would be welcome, but is optional. Without one, which aspects of that transfer can reasonably be discussed, and which remain open?

6. Complete historical execution evidence linked to the reported results. Could the authors provide auditable records of actual runs underlying the reported results, including a clearly identified run contributing to a main-table result? These should connect the actual prompts and contexts, feedback and component use, candidates and their bases, evaluation and repair outcomes, candidate selection, and final program.

4 comments

1. Proprietary models and harness transfer.

Claude Code tests harness transfer with GPT-OSS-120B (G3). Our six-model sweep guides targeted API tests of feedback/context, delegation, and history, retaining controls without assuming rankings transfer to proprietary models.

2. Repeatability and repairs.

All five repetitions favor reports (G2). Within-kernel highest/lowest ratios are:

ConditionMedianMaximum
Raw, append1.47×36.62×
Reports, append2.67×38.27×
Raw, rebuild1.76×146.03×
Reports, rebuild1.70×39.12×

The audit covers all 560 searches across five repetitions and 5,599 evaluated submissions. One ten-turn search submits nine candidates.

ConditionPassed first evaluationFailed then repaired to passPassed after repairs
Raw, append876/1400498/5241374/1400
Reports, append600/1400760/8001360/1400
Raw, rebuild935/1399442/4641377/1399
Reports, rebuild749/1400594/6511343/1400

Of 2,439 initially unsuccessful submissions, 2,294 pass after repairs; 5,454 pass in total. Failures include compilation problems, output mismatches, and timeouts. Reports require more repairs but produce faster selected programs. All 560 final selections pass and retain the fastest valid candidate measured, including 433 searches with later slower or unsuccessful submissions.

3. Recommendations and code changes.

Historical GPT-OSS-120B conv2d/ludcmp traces in Appendix J connect prompts, code, correctness, and selection. These trajectory ratios use native median timings; final paper figures use the shared mean-baseline statistic.

Both conv2d workflows start near 16× Numba. A report recommends output-channel SIMD before the next recovered candidate implements it and reaches 106×. Later candidates add 4×4 output tiling and weight reuse. The valid turn-7 winner reaches 584× and stays selected. Raw feedback retains its first candidate near 16×.

Recorded starting code allows verified comparisons for six delegated ludcmp candidates. Reports first produce SIMD at 1.13×. A report on remaining serial work precedes an accepted request for parallel row-wise trailing-matrix updates, reaching 23.20×. A report on per-pivot parallel-region overhead precedes one persistent region, reaching 28.51× after repair and correctness checks. Raw feedback ends after SIMD at 1.12×.

4. Task requirements and component use.

Conv2d reuses inputs across channels and weights across neighboring outputs. Ludcmp parallelism must respect dependencies.

We classified 400 post-repair candidates from 20 searches per kernel across five repetitions. Conv2d output-channel SIMD appears in 10/10 report searches versus 4/10 raw searches. Trailing-matrix LU updates appear in 5/10 versus 1/10. Every discovered feature remains in the selected code. These counts describe selected cases, not benchmark-wide behavior.

Across all 560 searches, rebuilt context has successful best-source reads in 138/140 raw and 136/140 report searches. At least one next proposal copies that source exactly in 121/140 versus 54/140 searches; appended-context counts are 13/140 versus 0/140. These counts exclude modified reuse and do not establish wasted work. Core retrieval is separate from optional tools and graphs, disabled here.

The four historical paired cases contain five successful best-source reads, one graph query returning no rows, and nonempty automatic history in 33/35 lead requests. The 17 denied file requests are excluded from successful use. Exposure does not establish benefits; full-suite comparisons measure performance effects.

5. Repository functions.

Extracting a function with its dependencies, representative inputs, and correctness constraints could provide a comparable optimization contract. The extraction must preserve caller assumptions, data layouts, and integration constraints. Reports could help identify relevant performance evidence in that context, but performance transfer remains untested. Reintegration requires application tests and timings; application speed-up also depends on the function's runtime share and integration overhead.

6. Auditable historical evidence.

Five selected programs have model, kernel, workflow, timings, and source hashes. Conv2d C06 contributes to Table 5 and reproduces Appendix J's 569.85× under the mean-based statistic. All 94 calls in the four paired searches match responses and message counts. Both conv2d winners and 17/20 candidate sources are recovered; three intermediates are missing. Direct-coding traces establish chronology, without explicit parent hashes. The evidence contains all five winners, 29/36 candidate sources, six parent diffs, prompts, responses, validation, and timings. The manifest lists hashes and missing sources.

Thank you again for your encouraging review. Could you let us know whether the revision resolves your concerns? If anything is still unclear, we would be glad to address it in the final revision.

Reviewer_LTeMOpenReview ↗

Sorry for the late reply

Thank you for the detailed reply. Sorry for the little reply and limited revision time. Forgive that I am a little busy....

Your reply convinced me about the repeatability and how the task component used. (I have seen the revision in the appendix)

I don't find any useful info in the reply of repo functions. Maybe there's some misunderstanding. Proprietary models and harness transfer here I want to see the data about its performance. But these are just for my own curiosity and this is not related to my confidence about your paper.

The key problem is that I still miss the info of a full life-turn optimization behavior. Could you expand it in Appendix J by walking through the report-guided and raw-feedback runs side by side, with the available code diffs for each candidate against the version it started from? I’d particularly like to see the qualitative analysis between the code diff and component's role.

Please revise the paper to add the optimization behavior includes the code diff and qualitative analysis between the code diff and component's role. Especially the code diff provided by agents! I have raised my score accordingly, but still want to see these details to be mentioned in the appendix.

Sorry again for the late reply and limited revision time!

Dear Reviewer,

Thank you for raising your score and for your suggestions. To examine how report guidance affects optimization trajectories, we traced the original raw-feedback and report-guided searches side by side, covering code changes, failed evaluations, correctness repairs, and final selections.

Two examples illustrate the observed behavior:

  • conv2d: Following an analysis recommendation, the report-guided agent adds SIMD across output channels, then spatial tiling and weight reuse. Later repairs remove these optimizations, but the agent retrieves and returns to an earlier optimized implementation. The raw-feedback agent repeatedly submits a slower implementation, while the controller retains its first valid candidate.
  • ludcmp: The report-guided workflow progresses from SIMD reductions to parallel row updates and a persistent OpenMP region, following the corresponding recommendations. The recorded starting implementations let us distinguish delegated code changes from correctness repairs. We also observed a direct-coding search that retrieves and reuses an earlier blocked implementation after repairs remove that structure.

These trajectories connect feedback recommendations to subsequent code changes and show how agents recover earlier optimizations after unsuccessful attempts. The raw-feedback searches do not exhibit the same successful sequences in these examples.

We also added source-provenance recording and an on-demand visualization linking code changes, feedback, repairs, and evaluation outcomes. Across 100 additional searches with GPT-OSS-20B and GPT-OSS-120B, we verified source records for all 1,731 evaluations. In one softmax search, a repair replaces an attempted vector exponential with scalar code, while a later candidate builds on that repaired source to produce a valid vectorized implementation.

Appendix J includes a complete optimization walkthrough, with actual code diffs and the feedback received at each step.

Best regards, The Authors

Reviewer_tn2HOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper systematically investigates whether increasingly complex agentic workflows improve code optimization performance. The authors introduce a unified framework covering 24 workflow configurations across four dimensions: direct versus delegated coding, raw versus analyzed execution feedback, optional tools and skills, and different history-management strategies. Experiments on 54 NPBench CPU kernels with three primary models, supplemented by three additional models, evaluate generated-code speed-up, token consumption, and search time. The main finding is that analyzed feedback consistently improves optimization performance, whereas delegation, tools, and history provide model-dependent benefits. Simpler workflows can therefore achieve competitive performance at substantially lower token costs.

Critical strengths
  1. Systematic and controlled evaluation. The 24-configuration design enables matched comparisons of interacting workflow components under a common evaluation protocol, providing useful empirical evidence beyond isolated component ablations.

  2. Meaningful efficiency findings. The study jointly considers generated-code performance and optimization cost. In particular, analyzed feedback improves full-suite performance by 1.15–1.46×, while the complete workflow consumes 54–146% more tokens than a simpler alternative without a statistically significant performance advantage.

  3. Cross-model insights. Experiments across six models reveal substantial differences in the effectiveness of delegation, history, and feedback analysis, challenging the assumption that one agent architecture generalizes across model families.

Critical weaknesses
  1. Limited methodological novelty. The individual mechanisms, including feedback analysis, delegation, context reconstruction, and history retrieval, are established techniques. The primary contribution is their systematic empirical comparison rather than a fundamentally new optimization methodology or theoretical insight.

  2. Confounded feedback-analysis mechanism. Section 2 explicitly combines execution-feedback analysis with context reconstruction into one configuration choice. Consequently, the improvements in Section 4.2 cannot be attributed specifically to better feedback rather than shorter contexts, reduced truncation, or their interaction. This weakens the paper's central explanatory claim.

  3. Limited statistical reliability. Section 3 specifies only one optimization search per model–kernel–configuration combination. As acknowledged in Appendix E.4, the bootstrap intervals capture variation across kernels but not stochastic variation across independent agent searches. The absence of repeated searches limits confidence in the reported effects, particularly smaller improvements and comparisons characterized as statistically insignificant.

  4. Restricted generalizability. The main experiments focus on NPBench CPU kernels and C++ implementations, while the additional models are evaluated on smaller, effect-selected subsets. It remains unclear whether the conclusions transfer to GPU kernels, repository-level optimization, or other agentic coding tasks.

Questions for authors
  1. Isolating feedback effects: Can the authors separately evaluate context reconstruction without an analysis agent, and analyzed feedback without context reconstruction? This would clarify which mechanism drives the improvements reported in Section 4.2.

  2. Statistical robustness: Can the authors provide independent repeated searches for representative configurations to quantify run-to-run variation and assess whether the main performance conclusions remain stable?

  3. Budget comparability: Given that analyzed feedback changes token consumption, search duration, and sometimes the number of evaluated candidates, how would the conclusions change under matched token, wall-clock, or candidate-evaluation budgets?

  4. Generalization: To what extent do the authors expect the observed advantages of analyzed feedback and simpler workflows to hold for GPU optimization or repository-level performance engineering?

4 comments

1. What the interaction study teaches us.

The study shows why a component that helps one workflow may add little to another. For GPT-OSS-120B with analyzed feedback and optional tools off, history/no-history ratios on 40 matched full-suite kernels are 1.10× with direct coding and 0.81× with delegation. The ratio between these effects is 1.36× (95% kernel interval 1.02–1.82). A single average history effect would hide this distinction. The practical lesson is to choose history together with the coding mode, rather than assuming that a benefit transfers between workflows.

Reports help under both context policies (General Response G1). Across the three matched full-suite models, the complete stack costs more tokens without a statistically clear speed advantage over a simpler workflow. These findings support starting with analyzed feedback and testing additional components in the workflow where they will be used. Holding the model, tasks, and evaluator fixed makes these lessons possible. We revised the introduction accordingly.

2. Isolating feedback effects.

G1 reports the requested four conditions. Reports improve performance with appended context (1.50×; 95% kernel interval 1.31–1.74) and rebuilt context (1.62×; 1.38–1.93). Rebuilding alone does not explain the gain. Reports still combine interpretation and compression, a distinction now stated in the paper.

3. Independent repeated searches.

All five repetitions favor reports under both context policies, and all 560 searches retain valid programs (G2). Individual implementations still vary. In an illustrative ludcmp comparison, the slower program uses row-wise dot products with strided column reads. The faster program uses parallel, contiguous trailing-row updates with SIMD. Candidate medians are 1,290 ms and 75 ms, while reference medians differ by 1.44%. Saved sources and timings support this comparison. Search variability and kernel-bootstrap uncertainty are reported separately.

4. Matched budgets.

We compare the best valid candidate available at each cutoff, counting work across all agents. Retrospective analyses cover all 560 searches across five repetitions; separate fresh searches enforce a five-minute deadline. Elapsed time includes generation, analysis, evaluation, repairs, and waiting on the fixed serving setup.

At 30 minutes, reports improve performance: 1.44× with appended context (95% kernel interval 1.26–1.68; 139/140 valid pairs) and 1.45× with rebuilding (1.21–1.75; 140/140 pairs). These retrospective comparisons retain only candidates completed before the cutoff.

At five minutes, the advantage is uncertain. We ran 112 fresh watchdog-limited searches, counting only evaluations completed within 300 seconds. Report/raw ratios are 1.22× (95% kernel interval 0.85–1.90; 20 matched valid pairs) and 1.08× (0.93–1.27; 21 pairs). Both intervals include 1; 89/112 searches produce valid code before the deadline.

At 250K input-plus-output tokens across all agents, conservative bounds for missing usage retain all 140 pairs per policy. Report/raw ratios are 1.407–1.424× with appended context and 1.602–1.606× with rebuilding. The respective 95% kernel-bootstrap envelopes are 1.22–1.66 and 1.35–1.92. These are retrospective limits, not additional token-capped runs. The supplement includes smaller-budget curves, evaluator-call comparisons, validity counts, token bounds, and watchdog details.

5. Generalization.

G3 reports the completed GPU and Claude Code experiments. On GPU, reports produce more valid programs and use 36% fewer tokens; the 1.31× speed estimate remains uncertain. In Claude Code, direct coding gains 1.74× on jointly valid tasks, with lower valid-program coverage (24/28 versus 27/28). Both experiments test reports with rebuilt context against raw results with appended context; G1 isolates these choices on CPU.

Extracting a function with its dependencies, representative inputs, and correctness constraints could provide a comparable optimization contract. The extraction must preserve caller assumptions, data layouts, and integration constraints. Reports could help identify relevant performance evidence in that context, but performance transfer remains untested. Reintegration requires application tests and timings; application speed-up also depends on the function's runtime share and integration overhead.

Thank you again for your encouraging review. Could you let us know whether the revision resolves your concerns? If anything is still unclear, we would be glad to address it in the final revision.

Thank you for your careful reply. I think the quality of the paper still needs to be further improved and I will maintain my score.

Dear Reviewer,

Thank you for your reply and for taking the time to reconsider our paper. We appreciate your feedback and understand that some concerns remain.

We aimed to address the main points raised in your review through the new feedback/context ablation, independent repeated searches, matched-budget comparisons, and GPU and Claude Code experiments. We also revised the manuscript to clarify the findings and their limitations.

Could you clarify which of your original concerns you feel remain insufficiently addressed? We would appreciate any specific feedback on where the new evidence or explanations still fall short, as this would help us further strengthen the paper and address any remaining gaps in the revision.

We would be happy to provide further details or clarifications.

Thank you again for your time and consideration.

Best regards, The Authors

Reviewer_t3sGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether adding more machinery to a coding agent actually produces faster code. One controller combines four design choices into 24 workflows. The lead agent either writes the code itself or delegates it, receives either raw execution results or a report from an analysis agent that also resets its context, works with or without tools and skills, and can revisit earlier attempts in one of three ways. Every workflow runs once on each of the 54 NPBench CPU kernels with GPT-OSS-20B, GPT-OSS-120B and Qwen3.8-27B, and smaller panels test three more models. Analyzed feedback gives the most consistent gain, making programs 1.15 to 1.46 times faster across the three main models, and for GPT-OSS direct coding it also cuts token use. Tools, history and delegation help in some settings but not others, and enabling every component costs far more tokens than a simpler workflow without being significantly faster.

Critical strengths
  1. The paper tests the components together instead of one at a time. Systems such as PerfAgent, STARK and KernelSkill bundle several of these components into one design. Studies that separate them, such as ContextEvolve, remove one part at a time from the full system. This paper crosses all four design choices under one controller, as Section 2 describes. It can therefore ask whether a component still helps once the others are present. Section 4.3 shows why this matters. History tends to help GPT-OSS when the lead writes the code but not when it delegates.

  2. The paper judges each component by the speed it buys and the tokens it costs. Optimization benchmarks such as GSO, SWE-Perf and KernelBench report how much faster the code becomes. The harness study of Fan et al. reports task success and cost, but not program speed. Here every workflow is placed on both measures, as Figure 1 and Section 4.4 show. This joint view is what exposes the paper's most practical result. The complete stack spends far more tokens than a simpler workflow, yet its programs are not significantly faster.

Critical weaknesses
  1. My main concern is that the paper credits the gain to better feedback, but its design cannot separate feedback from context management. Analyzed feedback changes two things at once. An analysis agent writes a report, and the harness rebuilds the lead's prompt at every turn. With raw feedback, results are appended to the conversation, which is only shortened once it no longer fits the window, as Appendix K.8 describes. No workflow rebuilds the prompt while passing raw results. Section 2 argues that the two parts only work together, but this does not show which part produces the gain. The paper's own evidence points both ways. For GPT-OSS direct coding, the gain comes with a much smaller lead prompt, and most raw-feedback searches had to be truncated. For Qwen3.8 delegated coding, analysis still helps while lead occupancy barely changes, as Table 13 shows. These may be two different mechanisms, yet the title and conclusion present them as one lesson about better feedback.

  2. The findings on tools, history and the complete stack rest on a panel chosen from GPT-OSS outcomes. Section 4.1 keeps the 28 kernels on which GPT-OSS configurations differ most, and notes that analysis is the largest factor there. A component that helps mainly on other kernels can then look weak. The full-suite numbers in the paper point that way. Section 4.3 reports that tools make Qwen3.8 direct coding faster on the full suite, while the panel shows no significant gain. In Table 5, the complete stack reaches a higher mean speed-up than the simpler workflow for 20B and Qwen3.8. These averages cover slightly different kernels and carry no intervals, and the paper compares workflows kernel by kernel on the full suite only for analysis. The abstract still says, without mentioning the panel, that tools and the complete stack add tokens without a significant speed-up. It is unclear whether this also holds on the full suite.

Overall, I find this work positive. The study is carefully run, and its question matters for how coding agents are built. My concerns are mainly about how far the current results support the main claims. If the authors can address these two points, I am willing to raise my score.

Questions for authors

I have no additional questions. I would appreciate the authors' response to the two concerns raised in the weaknesses above.

2 comments

1. Separating feedback from context management.

Thank you for suggesting these controls. General Response G1 reports the new 2×2 experiment, which compares raw results and reports under each context policy and then compares context policies under each feedback type. Reports improve performance with either policy: 1.50× with appended context and 1.62× with rebuilding, with both 95% kernel intervals above 1. All five independent repetitions show the same direction (G2). These results separate the report benefit from rebuilding; reports still combine interpretation and compression. We distinguish these controls from the original full-suite experiment, which combines reports and rebuilding.

2. Panel selection and full-suite comparisons.

The 28-kernel panel targets kernels where workflows differ substantially. Many other kernels perform similarly across configurations. We selected kernels by the size of the differences, regardless of which workflow was faster, and by timing coverage. Selection used GPT-OSS results; the other models use the same fixed panel.

Matched full-suite comparisons assess whether these findings hold across the broader benchmark. For direct coding, analysis gains are 1.51× on the full suite versus 1.79× on the panel for 20B, and 1.47× versus 1.71× for 120B. Full-suite tool gains are 1.21× for 20B delegated coding (95% kernel interval 1.06–1.37) and 1.13× for Qwen3.8 direct coding (1.01–1.29). We updated the abstract and discussion to name these settings and distinguish panel findings from broader comparisons.

Complete-stack speed and token comparisons now use identical timed kernels, addressing the differing populations in Table 5.

ModelMatched kernelsComplete / simpler speed ratio95% kernel intervalAdditional reported tokens
GPT-OSS-20B411.00×0.76–1.29+141%
GPT-OSS-120B420.80×0.61–1.03+48%
Qwen3.8-27B481.15×0.97–1.41+99%

The complete stack uses more reported tokens for all three models without a clear speed advantage. These estimates condition on available paired timings and do not show equivalence. Qwen's interval, for instance, still allows a meaningful gain.

Dear Authors,

Thank you for the detailed response. I have checked the updated paper and the added experiments. They address my concerns well, espeically on separating feedback from context management, and showing that feedback is the cruicial part that contributes most to the gain. I have raised my score accordingly.

Best regards.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response

We thank the reviewers for their constructive feedback and for recognizing the value of comparing interacting workflow choices under a common evaluator. We revised the paper to separate feedback from context rebuilding, quantify search variability, test constrained budgets, and examine transfer to GPU kernels and another harness. We summarize the shared findings here and address reviewer-specific questions in separate replies.

G1. Reports help under either context policy.

To distinguish feedback from context management, we ran all four combinations of raw results/reports and appended/rebuilt context with GPT-OSS-120B on the fixed 28-kernel panel. Five fresh searches per kernel and condition give 560 searches, all with valid final programs. Direct coding, model settings, and the evaluator were fixed; optional tools and history graphs were disabled.

Held fixedBaselineComparisonSpeed ratio95% kernel interval
Appended contextRawReports1.50×1.31–1.74
Rebuilt contextRawReports1.62×1.38–1.93
Raw resultsAppendedRebuilt1.02×0.89–1.15
ReportsAppendedRebuilt1.10×0.99–1.23

Ratios are geometric means of paired speed-up ratios; values above 1 favor the comparison. Each contrast has 140 pairs. Intervals resample the 28 kernels, keeping all five repetitions together. Reports help on 26/28 kernels with appended context and 27/28 with rebuilding after averaging repetitions. Average lead prompts fall from 58.8K to 13.0K tokens and from 62.1K to 6.2K, respectively. Rebuilding alone does not explain the gain. Reports still combine interpretation and compression.

G2. The report benefit persists across five independent searches.

Every repetition favors reports under both context policies. Report/raw ratios are 1.44×, 1.48×, 1.47×, 1.32×, and 1.84× with appended context, and 1.65×, 1.39×, 1.77×, 1.98×, and 1.39× with rebuilding. Across conditions, the median within-kernel highest/lowest speed-up ratio ranges from 1.47× to 2.67×. Search variability is reported separately from kernel-bootstrap intervals. The behavioral audit and retrospective budget analyses cover all 560 searches across five repetitions.

G3. Transfer experiments support specific benefits beyond the original CPU harness.

For GPU transfer, we used 24 KernelBench tasks from the KernelBench submodule of HPCAgent-Bench: 10 level-1, eight level-2, and six level-3 tasks. GPT-OSS-120B generated HIP implementations evaluated on AMD GPUs against the original PyTorch references. Reports produce valid programs on 19/24 tasks versus 15/24 for raw feedback and use 36% fewer reported tokens across all agents and tasks. On 13 jointly valid kernels, reports win nine and give a 1.31× speed ratio (95% kernel interval 0.85–1.89). The speed interval includes 1. All 34 valid programs pass five correctness trials with the HIP compute guard enabled.

In Claude Code, keeping GPT-OSS-120B, our tools, and the NPBench evaluator fixed, direct coding with reports gives 1.74× (1.27–2.50), winning 19/24 jointly valid kernels. Valid-program coverage is lower: 24/28 versus 27/28. Excluding searches retried after agent-role failures gives 1.83× (1.29–2.74; 21 pairs). Delegated coding gives 1.25× (0.99–1.62; 22 pairs). Adding history to reports gives 1.12× for direct coding (0.99–1.27; 19 pairs) and 0.89× for delegation (0.73–1.07; 21 pairs), with neither interval establishing an additional speed benefit.

All 168 Claude Code searches have terminal outcomes. Valid counts for raw/reports/reports with history are 27/24/23 out of 28 for direct coding and 24/22/22 for delegation. Interrupted searches and agent-role failures were resumed or retried with the original turn budgets. Failures remain in coverage counts; speed ratios use jointly valid tasks. Both transfer experiments compare reports with rebuilt context against raw feedback with appended context. G1 separates those choices on CPU. Claude Code tests harness transfer using an open-weight model, not proprietary-model transfer.

Area_Chair_97VrOpenReview ↗

Discussion

Dear Reviewers,

We are reaching the end of the discussion period! If the authors have responded to your review, please read their response, leave a comment, and update your rating if your assessment has changed.

Thank you,

AC.

We would like to thank you all for your time, detailed feedback, and constructive discussion throughout the rebuttal period. Your comments have helped us substantially improve the paper.

During the rebuttal, we revised the manuscript and conducted several additional experiments and analyses:

  • Feedback and context management: We added a controlled ablation separating the effects of analysis reports and context rebuilding.
  • Repeatability and budgets: We conducted independent repeated searches, analyzed correctness and repair behavior, and evaluated performance under matched time and token budgets.
  • Generalizability: We extended the evaluation to AMD GPU kernels and tested workflow transfer to Claude Code.
  • Optimization behavior: We expanded the analysis of historical execution trajectories, connecting feedback, code changes, correctness repairs, and final outcomes through source-code diffs.
  • Clarity and scope: We refined the presentation of the results, added matched full-suite comparisons, and clarified the conclusions and their limitations.

We believe these additions have strengthened the experimental evidence and made the paper's findings and practical implications clearer.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.