acceptodds
Under review as a conference paper at ICLR 2027

AutoLLMResearch: Training Agents for Automating LLM Experiment Configuration — Learn from Cheap, Optimize Expensive

Abstract

Effectively configuring scalable large language model (LLM) experiments, spanning architecture design, hyperparameter tuning, and beyond, is crucial for advancing LLM research, as poor configuration choices can waste substantial computational resources and prevent models from realizing their full potential. Prior automated methods are designed for low-cost settings where repeated trial and error is feasible, but scalable LLM experiments are too expensive for such extensive iteration. To our knowledge, no work has addressed the automation of high-cost LLM experiment configurations, leaving this problem labor-intensive and dependent on expert intuition. Motivated by this gap, we propose AutoLLMResearch, an agentic framework that mimics how human researchers learn generalizable principles from low-fidelity experiments and extrapolate to efficiently identify promising configurations in expensive LLM settings. The core challenge is how to enable an agent to learn, through interaction with a multi-fidelity experimental environment that captures the structure of the LLM configuration landscape. To achieve this, we propose a systematic framework with two key components: 1) LLMConfig-Gym, a multi-fidelity environment encompassing four critical LLM experiment tasks, supported by over one million GPU hours of verifiable experiment outcomes; 2) A structured training pipeline that formulates configuration research as a long-horizon Markov Decision Process and accordingly incentivizes cross-fidelity extrapolation reasoning. Extensive evaluation against diverse strong baselines on held-out experiments demonstrates the effectiveness, generalization, and interpretability of our framework, supporting its potential as a practical and general solution for scalable real-world LLM experiment automation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
AI for AI research, Agentic reinforcement learning, Automated machine learning research, multi-fidelity optimization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_rKjqOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors introduce RL-environments which aim to train LLM agents to perform AI research. They focus on generalizing from low-fidelity to high-fidelity settings. Low-fidelity here means simple environments and high-fidelity means complex environments, closer to frontier research. Although the high-fidelity environments are also very simple / small-scale.

Critical strengths

None

Critical weaknesses
  1. very simplistic environments, with only 1 tool and maximum 5 steps
  2. the high-fidelity settings are not really high-fidelity, but still quite small. E.g. they use models of up to 537M parameters in the pre-training task and GSM8K in the GRPO task.
Questions for authors
  1. How could you create stronger high-fidelity settings, as opposed to having simple high-fidelity baselines as in the current setting?
  2. How could the setting be made open-ended? Right now the action space of the agents is very limited.
Reviewer_97K2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LLM experiments are always heavy to run, so the feedback and reward is hard to get for training. The paper studies choosing configurations for expensive LLM experiments by learning from cheap ones. It introduces LLMConfig-Gym, an offline lookup-table environment with four tasks and several fidelity levels. An agent receives a task description, the Top-K configurations from lower-fidelity experiments with their fidelity metadata, and a budget, and proposes configurations through a tool call. Training combines trajectory simulation, multi-turn SFT, and multi-turn GRPO. On held-out high-fidelity experiments, the trained 4B agent achieves average regret 0.035 vs 0.164 for its prompt-only backbone, 0.043–0.071 for frontier prompt-only agents (Claude Opus 5 / Fable 5 / GPT-5.6) given the same context, and 0.22–0.32 for meta-BO baselines (NAP / MetaBO / FSBO).

Critical strengths
  1. I think the target problem is important. Rewards and feedbacks from real experiments are becoming harder to get, with tasks and environments getting more and more complex and heavy. This paper targets precisely on this important problem.

  2. Environment construction and training recipe are relatively complete.

Critical weaknesses
  1. The decision problems are small finite selections, which can limit the impact. Task 1 is "pick 1 of 20 listed candidates", Task 4 is "pick 1 of 16 listed mixtures", Task 3 is an 18-cell grid, and the budget goes to 5, though I do understand resources as potential limitations.

  2. NAP / MetaBO / FSBO receive numeric inputs with no fidelity information and must score coordinates they never saw. The natural competitor for the "learn from cheap, optimize expensive" claim is a multi-fidelity surrogate with fidelity as an input (e.g., a GP or a parametric fit of the optimal LR/BS as a function of (N, D) on the training sweeps), which is what practitioners actually use. Excluding Step Law because it was fitted on the same sweep is not convincing, since the agent's training data also comes from that sweep. Without such a baseline, it is unclear whether an RL-trained 4B agent adds anything over a few lines of extrapolation on tasks of this size.

Questions for authors
  1. For Task 2, how does a parametric extrapolation fit on the training experiments compare at budgets 1–5?

  2. Can the Gym expose larger candidate sets (all 76 mixtures for Task 4; hundreds of GPT-L architectures), and does the agent's advantage still hold there?

Reviewer_pjVdOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The authors propose an LLM-based framework for automated parameter search, using GRPO to train a search agent and transferring the results of low-fidelity search to high-fidelity settings.

Critical strengths

N/A

Critical weaknesses

The paper never defines low fidelity and high fidelity or specifies their scope. For example, is searching for parameters on small models and transferring them to large models a problem this paper aims to address? Or must the model size remain unchanged? It is unclear what problem the authors are trying to solve.

The paper emphasizes low-cost parameter search, but none of the experiments report cost-related data, even though the framework also requires training one (possibly more than one) search agent.

Is there a theoretical basis for extrapolating results obtained with fewer than 8B tokens to 20B–100B tokens? In practice, there are many cases where training runs that perform well early on are overtaken later.

Was parameter search performed for the search agent itself? How well does it generalize? How can the authors demonstrate that it can handle various hyperparameter search tasks?

Questions for authors

In the current automated search framework, if a similar or even identical task comes in (assuming different users who are unaware of each other’s tasks), will the system run the search again? I did not see anything like a memory module.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.