acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Program Search for AutoML: A Reachability Analysis of LLM-Edited Training Programs

Abstract

Do the code edits an LLM agent makes to a training program reach anything a hyperparameter optimizer could not? End scores cannot tell a new mechanism from a renamed knob. We make the question executable: relative to a frozen program, its configuration space, and an extension declared in advance, we sort every edit into four reachability classes by comparing parent and child forward outputs at fixed configurations. Only edits outside the extension can credit program search, and only against a tuner given the same extension. We audit 467 edits a Claude agent proposed to LoRA adapter code and run controlled searches on Pythia-160M and 410M over ARC-Easy, RACE, and OpenBookQA. Of 332 primary edits, 76% are hyperparameter-level: 56% reconstruct exactly inside the extension and 20% rescale one coordinate. At an equal budget of 492 evaluations, neither our controller nor OpenEvolve or ShinkaEvolve separated on validation from TPE on the unedited program. A best-child acceptance rule overstates progress: in our controller's archives, 38 of 51 admitted programs entered because the best score of a short local tuning run beat the parent's by more than one point, below single-run noise, not because the edit helped at matched configurations. Given a wider surface, the agent produced uncovered programs in every seed, yet none showed an established gain over a tuner given the extension; on OpenBookQA that tuner matched the agent's sealed-test gain. Program search should be credited only for uncovered edits, against a coverage-matched control.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Agent, AutoML, LLM Evolve

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PdwGOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This article investigates whether modifying the training code of large language models can bring benefits beyond hyperparameter optimization. The author proposes a reachability analysis framework, which divides modifications into four categories: unchanged behavior, re-expression of existing parameters, reproducible by a pre-declared extended search space, and beyond this space. The author also builds a SelfTune platform to evaluate the effect of modifications through modification validation checks, independent program parameter adjustment, and comparison of parent and child programs under the same configuration. The author conducted editing corpus auditing, same-budget comparison, and coverage matching experiments on the Pythia model and ARC-Easy, RACE, and OpenBookQA tasks. About 76% of the 332 major edits were classified as parameter-level changes, and the significant advantage of program search over parameter adjustment control was not established under the tested settings. The main contribution of this paper is to provide an evaluation framework based on actual implementation evidence for distinguishing search space expansion from additional benefits of program search.

Critical strengths
  • The authors propose a meaningful evaluation framework to assess the reproducibility, validity, and independent data gains of the modifications.
  • The experiments analyze editing types, archival inclusion, execution variances, and configuration interactions, while also addressing data coverage and statistical limitations.
Critical weaknesses
  • The core ratio lacks item-by-item verification: 76% comes from type mapping across cohorts, the impact of implementation differences on this ratio is unclear.

  • Early pairwise comparisons are biased: The historical best results of the parent program are affected by selection bias and noise, which limits the judgment of the effect of modifications.

Questions for authors
  1. Section 3 and Appendix B.4 state that a fixed four-anchor-point approach is used, whereas Appendix B.3 and Table 6 refer to the use of the four best configurations from the parent program. Why is this the case?

  2. Is there any code or source documentation supporting implementation consistency across cohorts? What does the 76% figure represent when considering only edits with verifiable correspondences?

  3. Are there independent parent-child re-evaluation results? If so, how do the paired labels from historical score comparisons differ from those obtained through independent re-evaluation?

  4. The significance criteria in Section 5 are inconsistent with the results in Table 9. Please clarify whether this is a typographical error or due to different testing methods, and indicate whether it affects the relevant conclusions.

Reviewer_WAr7OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper is an empirical study on LLM conducting program search by directly editing code, and try to compare its performance against more conventional hyperparameter optimizers. To evaluate the performance, the paper defines a novel set of reachability classes from equivalence to uncovered by an existing set of extension or reparameterization. The paper then proposes SelfTune, a controlled platform for nested search where LLM agent is responsible for modifying training code at the outer loop, while TPE optimizes hyperparameters from the inner loop. The result on several models and datasets reveals many findings, such as that LLM-based program search does not outperform conventional hyperparam optimizer, most modification by LLM is hyperparam optimization, and agent's open edit does not necessarily help.

Critical strengths
  1. This paper studies a very interesting problem: the effect of LLM autoML agent, which is timely in this age of Recursive Self-Improvement (RSI). Based on this problem, the paper proposes a reasonable and principled framework to evaluate LLM agent's ability by defining the reachability classes.

  2. The evaluation of this paper is relatively solid, with many findings reported. The reproducibility of this paper is also backed by a very detailed description of experiment details in Appendix B.

Critical weaknesses

I think a key limitation of this paper is that the construction of the "frozen extension set" Λ+\Lambda_+. This is the most important reason that supports rejection. The definition of this set largely infleunces whether an edit can be classified as R2 or R3, which heavily rely on the authors' prior knowledge.

a) If R2 is too large, then it is obvious that almost all edits made by LLM will not fall into R3, and vice versa. Thus, it is hard to well-define the LLM agent's ability to perform edits beyond Λ+\Lambda_+ as it is heavily influenced by prior knowledge on R2;

b) In page 4, the authors mentioned that the Λ+\Lambda_+ of the corpus is assembled "from the subtypes the agent had produced". Since the generation of Λ+\Lambda_+ involves the LLM agent, it is likely that the design of Λ+\Lambda_+ already contains what the agent could propose to modify. Thus, a hindsight bias exists, and it is not surprising that the agents cannot often propose R3 edits. If Λ+\Lambda_+ is sufficiently large, then whether an edit is R2 or R3 does not necessarily matter; what matters is the efficiency of LLM agent under limited budget.

c) The edit rules may further limits the LLM agent's ability to perform R3-level edits, which underestimates the creativity of the LLM agent, since "the frozen base model, data, objective, optimizer, schedule, precision, batching, and target task" are preserved (line 764-765). Adding regularizer or changing optimizer / learning rate scheduler / batch size are all common hyperparam search human would do; limiting edits to them may force the LLM agent to conduct R1/R2 edits.

Minor Weakness

The abbreviation "TPE" is not shown in full in the paper, and also not explained in the abstract / introduction but discussed.

Questions for authors
  1. What is the upper bound for autoML optimization on the evaluated tasks? While I understand that the exact numbers may be hard to answer, there is a possibility that LLM agent might not achieving better performance because there is no room for further improvement in the first place.

  2. It would be better if a qualitative example can be shown for the experiment. Except for helping the readers to understand how the agent works, it could also help us to see how negative experiences in the memory actually affects the proposal of memory-based selftune, such that it performs worse than no memory (Fig. 2).

4 comments

Thanks for the response. However, even if the ability of the agent with Λ+\Lambda_+ was tested on new edits, it is still reliant on the model's output on other data. Thus, the definition of Λ+\Lambda_+ differs when summarized from a corpus of size 100 and 10000. With the exact same behavior of the agent on the new 143 edits, R2/R3 edit rate can vary wildly; if the R3 edit rate is low, this does not necessarily mean that the agent does not know how to perform a novel edit - it may be just that the corpus is too big. Overall, I think my question is: can the R2/R3 rate be interpreted beyond a specific choice of corpus and test set? If not, then a low R3 edit rate may not indicate "agents cannot often propose R3 edits" even on another pair of corpus and test set of the same task domain.

No. The R2/R3 rate cannot be read beyond the specific (Λ+,(\Lambda_+, test set$)$ pair, and we do not read it that way. You are right that a catalog built from 10,000 edits would lower the R3 rate even if the agent behaved identically. So the rate is not a measure of what the agent can do, and a low R3 rate would not show that agents cannot propose novel edits. We make no such claim. Section 3 defines R3 as "a statement about what Λ+\Lambda_+ did not enumerate, not a claim of non-enumerability."

The rate supports one narrower statement, and only together with the catalog size: our five-family catalog, built from 332 edits, covers 76% of the corpus and 58% of held-out edits. That describes how concentrated these edits are relative to this catalog, nothing more.

Our conclusions do not depend on the rate. The agent clearly can propose R3 edits: it did in every seed of Finding 5, and 10 of 78 held-out edits were new designs. What we test is whether such edits help. In Finding 1, which uses no catalog, the agent ended −0.08 points from TPE (p=0.71p=0.71). In Finding 5, the R3 programs showed no established gain over a tuner given the frozen Λ+\Lambda_+. Neither result changes with the size of the corpus.

Thank you for your response. I think the fact that R2/R3 cannot be read beyond a specific pair is a limitation of this work as future researchers who would apply this metric to their task need to carefully design and interpret their Λ+\Lambda_+ in a case-by-case manner, which limits the generalizability of the work. This being said, I have no further question and will increase my score from 2 to 3.

Thank you for the follow-up and for raising your score. We agree that R2/R3 are tied to the specific pair they are computed on, and that applying the metric to a new task requires choosing and interpreting Λ+\Lambda_+ for that task. We will state this explicitly in the limitations section of the revised version, along with guidance on how to choose Λ+\Lambda_+. Thank you again for your time and the constructive discussion.

Reviewer_uzhKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates the question of when an LLM improves training code, how often does it come from novel methodological improvements, and when it does make an improvement, can a conventional hyperparameter optimizer match that performance. The authors introduce a taxonomy of reachability that compares parent and child outputs at fixed configs. Investigation shows that a majority of LLM generated improvements are on the hyperparameter level, and none of the systems show advantage over TPE optimization.

Critical strengths

The paper separates algorithmic/methodological improvements versus hyperparameter tuning cleanly by evaluating parent and child programs at fixed configurations. The findings showed that most earlier studies overstated their progress, and proposed a practical way of solving the problem, using LLMs to propose new mechanism families for improvement and use classical optimizers like TPE to squeeze the performance out of them.

Critical weaknesses

The main experiments mostly consider lora adapter modifications, and the broader study uses a MLP-site program. The scope is rather limited and it is less clear on if the edits will be concentrated on hyperparameter level edits or not when we generalize the LLM optimization methods to the entire model or complete ML pipelines. Also, the extenstion Λ+\Lambda_+ used for the analysis is constructed retrospectively from the modifications that the agent proposed. Defining it from the observed edits makes it easier for the proposed edits to fall into R2, this is informative about what fraction of the edits can be represented with a compact conditional search space, but a independently specified HPO baseline might not cover the same search space. A analysis of the edit distribution on a broader set of structures/tasks will make the conclusions more general.

Also, there seems to be a May 2026 paper 'What Do Evolutionary Coding Agents Evolve?' that asks the very similar question of is the agent discovering new algorithmic structure/re-tuning parameters/overfitting evaluator. It also investigates the OpenEvolve/ShinkaEvolve/etc harnesses and try to classify the type of edits into HP tuning, architectural change etc. They also came to the same observation that a Bayesian optimization run can match most of the improvements from evolutionary runs. Although that study evaluates mathematical and algorithm-design tasks instead of ML training workloads, it still overlaps with a large part of this paper's motivation. The authors should discuss this work and clarify how the reachability taxonomy and controlled training-program experiments go beyond prior work, this would help distinguishing the methodological contribution from extending the prior work on a new experimental setting.

  1. What Do Evolutionary Coding Agents Evolve? https://arxiv.org/abs/2605.20086
Questions for authors

When using Bayesian optimization for hyperparameter optimization, we would iterative make observations and update proposed parameters. However, the LLM only did single shot adapter edits, would it make a difference if we allow multi-turn code edits that takes the training run as feedback?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.