- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes EvoForge, a multi-agent framework for online hyperparameter optimization during a single distributed pretraining run. EvoForge treats tuning as a sequential decision problem where observations depend on earlier interventions. The framework consists of three agents: a tuning agent that adjusts hyperparameters based on training dynamics, a safety agent that validates interventions and supports checkpoint rollback, and a reflection agent that maintains reusable experience across training tasks. Experiments on GPT-2 124M trained on 10B FineWeb-Edu tokens show a 4.15% reduction in validation loss compared with a tuned cosine schedule and 2.49% compared with WSD.
Critical strengths
-
A practical and relevant problem formulation. The paper addresses online hyperparameter adjustment under a single pretraining trajectory, where conventional multi-trial search is impractical. It recognizes that training statistics are influenced by previous tuning decisions and that harmful interventions may require recovery.
-
Thoughtful system design with attention to safety and distributed execution. EvoForge combines adaptive tuning, deterministic guardrails, checkpoint rollback, and experience management in a coherent framework. The implementation addresses practical issues such as synchronized hyperparameter updates across data-parallel replicas, maintaining data-loader consistency after batch changes, and restoring controller state after rollback. These considerations make the work more substantial than a straightforward LLM-based hyperparameter proposer.
-
Promising empirical improvements across multiple configurations. EvoForge improves validation loss and downstream accuracy over the evaluated static schedules. The ablations indicate that removing the safety or experience components degrades performance, while experiments on different model architectures and controller backends provide some evidence of robustness.
Critical weaknesses
-
The experimental comparisons do not sufficiently isolate the benefit of agentic online optimization. The main baselines use a fixed global batch size of 512, whereas EvoForge starts at 64 and dynamically increases it. EvoForge also modifies the learning-rate trajectory, weight decay, and Adam's second-moment coefficient during training. Thus, the gains may arise from better hyperparameter schedules or more optimizer updates rather than the LLM-driven decision process itself. Comparisons against simple adaptive schedules or rule-based online controllers under matched initialization and action spaces are needed to establish the value of agentic control.
-
The contribution of cross-task reflection is not convincingly demonstrated. The reflection agent operates after a training task finishes, yet the main ablation evaluates its effect within a single training run. The authors explain that the experience store already contains priors that affect the controller's decisions. Consequently, the large degradation without reflection may reflect the removal of useful prior guidance rather than the benefit of learning from previous tasks. No clear sequence of training tasks demonstrates that automatically accumulated experience improves performance on subsequent tasks.
-
The empirical evidence remains limited in scale and statistical reliability. The largest evaluated model has only 0.7B parameters, and all experiments run on one eight-GPU node. The main comparisons appear to rely largely on single training runs. Furthermore, the larger-model experiments compare primarily against a cosine baseline rather than demonstrating that the gains persist against independently tuned alternatives such as WSD. These limitations make it difficult to assess whether the reported improvements generalize to more demanding pretraining settings rather than a small scale prototype.
Questions for authors
-
How much of the improvement comes from LLM-driven decisions rather than the resulting hyperparameter schedule? EvoForge changes several hyperparameters and starts with a substantially smaller batch size than the static baselines. Could the authors evaluate simpler alternatives under comparable starting configurations and allowed actions, such as a hand-designed progressive batch schedule, an adaptive WSD schedule, or a rule-based controller that adjusts learning rate based on validation-loss trends? Replaying EvoForge's discovered hyperparameter trajectory as a fixed schedule would also help determine whether the LLM's online adaptation provides additional value beyond discovering an effective schedule. Otherwise it's hard to argue the LLM-driven decisions are actually helpful.
-
Can the authors demonstrate the benefit of cross-task experience accumulation independently of predefined priors? Since the reflection agent updates the experience store only after a task completes, what experience is available when the primary GPT-2 training run begins? Could the authors evaluate a sequence of distinct pretraining tasks, comparing an initially empty experience store, a fixed store containing the same prior guidance, and an evolving store that incorporates reflection from earlier tasks? This would directly test the proposed cross-task learning mechanism.
-
What evidence supports the scalability of EvoForge to substantially larger distributed pretraining model and workloads? The implementation handles eight-way data parallelism, but the current experiments do not exercise larger scale model and system.