acceptodds
Under review as a conference paper at ICLR 2027

EvoForge: Agentic Online Hyperparameter Tuning for Pretraining

Abstract

Pretraining hyperparameters are typically fixed before training begins, while the optimal configuration drifts as training proceeds. Existing automated methods, such as random search, Bayesian optimization, multi-fidelity scheduling, and population-based training, presuppose multiple trials or concurrent replicas; a single large-scale pretraining run yields one trajectory, precluding a second trial. We present EvoForge, to the best of our knowledge, the first system that places LLM agents in the control loop of a running distributed pretraining job. We cast online tuning as a single-trajectory, partially observable sequential decision problem, where observations are induced by past actions, and organize the system around three cooperating agents, namely, one for tuning, one for safety, and one for cross-task reflection. EvoForge distills rules into a two-layer experience store only after sufficient evidence has accumulated. In GPT-2 124M trained on 10B FineWeb-Edu tokens, EvoForge reduces the final validation loss by 4.15% relative to a tuned cosine schedule.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
agentic systems, online hyperparameter tuning, pretraining, rollback recovery, cross-task experience memory, closed-loop control

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_EdziOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EvoForge, a multi-agent framework for online hyperparameter optimization during a single distributed pretraining run. EvoForge treats tuning as a sequential decision problem where observations depend on earlier interventions. The framework consists of three agents: a tuning agent that adjusts hyperparameters based on training dynamics, a safety agent that validates interventions and supports checkpoint rollback, and a reflection agent that maintains reusable experience across training tasks. Experiments on GPT-2 124M trained on 10B FineWeb-Edu tokens show a 4.15% reduction in validation loss compared with a tuned cosine schedule and 2.49% compared with WSD.

Critical strengths
  1. A practical and relevant problem formulation. The paper addresses online hyperparameter adjustment under a single pretraining trajectory, where conventional multi-trial search is impractical. It recognizes that training statistics are influenced by previous tuning decisions and that harmful interventions may require recovery.

  2. Thoughtful system design with attention to safety and distributed execution. EvoForge combines adaptive tuning, deterministic guardrails, checkpoint rollback, and experience management in a coherent framework. The implementation addresses practical issues such as synchronized hyperparameter updates across data-parallel replicas, maintaining data-loader consistency after batch changes, and restoring controller state after rollback. These considerations make the work more substantial than a straightforward LLM-based hyperparameter proposer.

  3. Promising empirical improvements across multiple configurations. EvoForge improves validation loss and downstream accuracy over the evaluated static schedules. The ablations indicate that removing the safety or experience components degrades performance, while experiments on different model architectures and controller backends provide some evidence of robustness.

Critical weaknesses
  1. The experimental comparisons do not sufficiently isolate the benefit of agentic online optimization. The main baselines use a fixed global batch size of 512, whereas EvoForge starts at 64 and dynamically increases it. EvoForge also modifies the learning-rate trajectory, weight decay, and Adam's second-moment coefficient during training. Thus, the gains may arise from better hyperparameter schedules or more optimizer updates rather than the LLM-driven decision process itself. Comparisons against simple adaptive schedules or rule-based online controllers under matched initialization and action spaces are needed to establish the value of agentic control.

  2. The contribution of cross-task reflection is not convincingly demonstrated. The reflection agent operates after a training task finishes, yet the main ablation evaluates its effect within a single training run. The authors explain that the experience store already contains priors that affect the controller's decisions. Consequently, the large degradation without reflection may reflect the removal of useful prior guidance rather than the benefit of learning from previous tasks. No clear sequence of training tasks demonstrates that automatically accumulated experience improves performance on subsequent tasks.

  3. The empirical evidence remains limited in scale and statistical reliability. The largest evaluated model has only 0.7B parameters, and all experiments run on one eight-GPU node. The main comparisons appear to rely largely on single training runs. Furthermore, the larger-model experiments compare primarily against a cosine baseline rather than demonstrating that the gains persist against independently tuned alternatives such as WSD. These limitations make it difficult to assess whether the reported improvements generalize to more demanding pretraining settings rather than a small scale prototype.

Questions for authors
  1. How much of the improvement comes from LLM-driven decisions rather than the resulting hyperparameter schedule? EvoForge changes several hyperparameters and starts with a substantially smaller batch size than the static baselines. Could the authors evaluate simpler alternatives under comparable starting configurations and allowed actions, such as a hand-designed progressive batch schedule, an adaptive WSD schedule, or a rule-based controller that adjusts learning rate based on validation-loss trends? Replaying EvoForge's discovered hyperparameter trajectory as a fixed schedule would also help determine whether the LLM's online adaptation provides additional value beyond discovering an effective schedule. Otherwise it's hard to argue the LLM-driven decisions are actually helpful.

  2. Can the authors demonstrate the benefit of cross-task experience accumulation independently of predefined priors? Since the reflection agent updates the experience store only after a task completes, what experience is available when the primary GPT-2 training run begins? Could the authors evaluate a sequence of distinct pretraining tasks, comparing an initially empty experience store, a fixed store containing the same prior guidance, and an evolving store that incorporates reflection from earlier tasks? This would directly test the proposed cross-task learning mechanism.

  3. What evidence supports the scalability of EvoForge to substantially larger distributed pretraining model and workloads? The implementation handles eight-way data parallelism, but the current experiments do not exercise larger scale model and system.

Reviewer_3W1cOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces EvoForge, that utilizes LLM agents in the pertaining process to enable single-trajectory optimization. EvoForge runs independently of the training process and has three agents including Tuning agent, Safety agent, and Reflection agent.

Critical strengths
  1. Combines LLM and pretraining, and different agents for different tasks.
  2. The problem itself is important and worth exploring.
  3. Tested on different model size and architecture, EvoForge suggests a good generalization.
Critical weaknesses
  1. There are already some existing works similar to EvoForge, e.g., SOLAR1SOLAR^{1}, AgentHPOAgentHPO, this work needs to show more novelty other than simply combining several ideas that have individually been explored.
  2. The claim of this paper should be narrowed. For example the wording in the abstract is too broad.
  3. The baselines are relatively weak, to establish a stronger claim please use more existing online learning-rate optimization methods.
  4. The ablation study is broad, needs more systematic evaluation including one agent for all tasks, full system with randomized or irrelevant experience.,,

[1] Shang, Q., Wang, B., Qiao, Y., Rao, S., Wu, Z., & Yuan, K. (2026). SOLAR: A State-Driven Online Learning Rate Scheduler for LLM Pretraining.

Questions for authors
  1. EvoForge produces a sequence of hyperparameter adjustments during training. After completing a run, the researchers could record the resulting schedule and replay it in a new training run without using any agents. Compare:
  • EvoForge with active agents.
  • Fixed replay of the schedule discovered by EvoForge.
  • Original cosine schedule.
  1. If EvoForge repeatedly tries hyperparameter adjustments and rolls back when they fail, how is this fundamentally different from exploring multiple training trajectories?
Reviewer_g4yBOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces an AI agent controller for training an LLM. The agent learns to tune the hyper-parameters from experience through trial and error. It has the ability to look up historical decisions, outcomes, propose new decisions and store their outcomes in the history. The agent acts as a controller, observing LLM training trajectories and makes online training hyper-parameter decisions including rollbacks to previous checkpoints if the training loss is unrecoverable.

Traditional hyper-parameter tuning models have to run several experiments to gather data about the loss landscape before being able to tune hyper-parameters. EvoForge on the other hand is able to learn from single trajectories. Experiment results on common benchmarks show improved scores compared to baselines.

Critical strengths
  • This is a paper towards improving LLM training using LLMs (self improvement). Unlike traditional HPO methods which do not have any 'intelligence', EvoForge is able to make complex decisions.
  • While standard hyper-parameters are mostly constant or follow a specified schedule through training, they often require manual monitoring and rollbacks. EvoForge replaces such human effort with AI agents.
  • Experimental results show improved scores on benchmark datasets.
Critical weaknesses

The seems to be AI written with a lot of missing details on the method. The overall idea is understood, however the specific details of each of the components are missing or very obscure and requires a huge amount of effort to read and understand.

  • Missing details of the method: Figure 2 gives and overall overview of the method, but none of the components are described well in the text with sufficient explanation and details (see questions).
  • Novelty: while this specific method may be new, LLMs have been used as controllers before: FT-Dojo: Towards Autonomous LLM Fine-Tuning with Language Agents. A simple google / AI search would reveal many such papers. Discussion on existing work needs to be expanded.
  • A large fraction of the AI generated text is not understandable (see questions)
Questions for authors
  • Section 4.3 talks about cross task experience, and the introduction says that the method can run on one trajectory only. Where is cross task experience coming from then?
  • What is experience distillation, what is being distilled to what? It is nowhere defined. What is the meaning of line 298? Section 4.1 doesn't seem to talk about any type of distillation.
  • Missing details of the method:
    • What are three layer guardrails? how did you come up with them and why?
    • How does the phase classifier work? how was it designed?
    • What are 'validation checks'?
  • What do these sentences mean?
    • Figure 2: "... the two-layer experience store, and emits a fixed-schema proposal; the safety agent checks it at proposal, validation, and admission, choosing the restore point and post-rollback hyperparameters on rejection or drift"
    • "A fixed control interval is inherently mistuned: after a large change it is too short, tempting the agent to judge before the change has taken effect, which is exactly where oscillation comes from; in a calm period it is too long, delaying an adjustment that could have been made." how exactly is the control interval decided then?
  • "Validation explains rather than corrects: a rejected proposal returns to the model with the reason and the acceptable range, because silently applying a modified version would equally contaminate the history the model reasons from. Memory safety is predictive rather than reactive: the largest safe micro-batch is reported during prompt construction, and proposals with insufficient predicted headroom are rejected, since waiting for a real out-of-memory error would interrupt the job." what exactly is this sentence trying to say? feels out of context
Reviewer_6H3xOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes using pretrained LLM agents to tune AdamW hyperparameters during training, rather than relying on multiple trials or parallel runs. A tuning agent receives a recency-biased summary of training statistics and decides on hyperparameter updates and the next observation time. A safety agent validates updates and manages checkpoint recovery, while a reflection agent distills lessons from completed runs for future training tasks. None of the controller LLMs is trained.

Critical strengths

1- Combining human-established learning-rate schedules and optimization heuristics with training history and accumulated experience can lead to useful adaptive schedules.

2- The paper demonstrates a potentially useful engineering system for automated online hyperparameter tuning.

Critical weaknesses

1- Framing and related work: The paper's main motivation that existing HPO methods require multiple trials or parallel workers overlooks an important class of online meta-gradient methods, including IDBD (1992), SMD (1999), and MetaOptimize (2025). These methods adapt hyperparameters within a single training trajectory by propagating meta-gradients with respect to the hyperparameters through learning dynamics, with small computational overhead.

2- Methodology and generalization: Fundamentally, EvoForge appears to primarily apply existing human knowledge through an LLM-based decision interface. It is unsurprising that combining established scheduling heuristics (like decay schedules and decrease during plateaus) with experience from previous runs can be useful on familiar tasks. However, generalization to substantially different datasets, model scales, training dynamics, and nonstationary or continual-learning settings remains untested. It is unclear whether the approach provides a generalizable optimization mechanism beyond applying familiar heuristics. In particular, it is unclear how the controller can reliably distinguish loss changes caused by suboptimal hyperparameters from those arising from changes in the underlying data distribution or training dynamics, potentially leading to inappropriate hyperparameter adjustments or rollbacks.

3- Experimental limitations: The controllers include LLMs with hundreds of billions of parameters, while the trained models have only 124M–700M parameters. There is no comparison with a simpler non-LLM adaptive controller. The usage details of the initial reflection memory is unclear, particularly whether it includes previous runs on the same task. Finally, validation loss is used for both online adaptation and primary evaluation, raising concerns about evaluation bias.

Questions for authors

1- Equation (1) describes SGD with gradient clipping rather than AdamW, which is used in the experiments. Why is the actual optimizer not represented in the formulation?

2- Why are separate tuning and safety agents necessary? Much of the safety mechanism consists of deterministic constraints and checkpoint management. Could these functions be integrated into one agent with deterministic safeguards? The ablation does not establish the necessity of a separate safety agent.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.