acceptodds
Under review as a conference paper at ICLR 2027

EvE: An Alternate Optimizer to Adam

Abstract

Adam and its variants dominate neural network training, but a single training run only reveals whether one configuration works well after most of its budget is already spent. The concept is a poor fit for hyperparameter or architecture search, where many configurations must be ranked cheaply and pruned early. As an alternative to Adam, we introduce EvE (Evolutionary Explorer), a steady-state, population-of-four differential evolution (DE) optimizer with a targeted Adam fallback: each iteration proposes one candidate via the DE operator. EvE only runs a short burst of gradient descent on it if the DE step fails to improve on the incumbent. Selection is greedy, so on a deterministic objective the best-so-far value is provably monotone non-increasing, and because gradients are used only as a targeted rescue, per-iteration cost stays within a constant factor of a single Adam step regardless of problem dimension. Under a fixed, evaluation-cost-matched budget, EvE wins or ties Adam on 76% of 70 (problem, dimension) cells across seven classical scalable benchmarks scaling up to one million variables. On three real neural-network training tasks (an MLP on MNIST, and LoRA fine-tuning of a 1.5B-parameter language model on two datasets) EvE consistently finishes the same charged budget 1.7 to 3.9x faster in wall-clock time, at a modest cost in final quality (about one accuracy point on MNIST, roughly 9-11% higher relative test loss on the two language fine-tuning tasks; on GSM8K's full test set, Adam is also about 5 accuracy points more accurate at matched budgets, and fine-tuning lowers accuracy below the base model for both optimizers). Inside successive halving on UCI Adult Income, EvE completes hyperparameter and architecture searches 3.1-3.5x faster, and its ranking of candidate configurations agrees with Adam's about as well as Adam's agrees with itself across training seeds (Kendall's tau of 0.66-0.69). These results position EvE not as a total replacement for Adam as a final-stage trainer, but as a fast, gradient-aware proxy for the search-heavy, budget-constrained regime one level up: ranking candidate configurations, or pruning clearly bad ones early, where a fast, approximate signal is worth more than a slow, precise one. Code will be made publicly available, with a link, in the final version.

Primary area:
optimization
Keywords:
evolutionary optimization, differential evolution, hybrid optimizer, gradient-free optimization, population-based methods, hyperparameter optimization, LoRA / parameter-efficient fine-tuning, Adam / adaptive optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_epVeOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This article argues that in some low-cost scenarios such as hyperparameter sorting and architecture search, the optimization cost of ADAM can be computationally expensive. Therefore, the author proposed EvE, a method that combines differential evolution with gradient-based optimization. The author conducted experiments on three real network training tasks and claimed that the proposed method can reduce wall-clock time while maintaining reasonably competitive performance, although at some cost to final model quality.

Critical strengths
  1. The paper is well-written and well-structured.
  2. The motivation behind the proposed method is reasonable and the results are competitive.
Critical weaknesses
  1. The novelty of the author's core approach is rather limited. There have been some studies[1-3] on hybrid evolutionary algorithms and gradient optimizers. However, the author failed to clearly delineate the core contributions of the proposed method from the existing works.

[1] Xue Y, Tong Y, Neri F. An ensemble of differential evolution and Adam for training feed-forward neural networks[J]. Information Sciences, 2022, 608: 453-471.

[2] Yu Q, Liang X, Liu J, et al. Adaptive Niching-Based Gradient-Accelerated Differential Evolution for High-Dimensional Nonconvex Optimization[J]. IEEE Transactions on Neural Networks and Learning Systems, 2026.

[3] Neural DE: An Evolutionary Method Based on Differential Evolution Suitable for Neural Network Training.

  1. Among the comparison methods chosen by the author, there is a lack of methods that combine with evolutionary algorithms and gradient optimizers.

  2. The author did not conduct detailed ablation experiments to verify the role of each component, which makes it difficult to convince people that each component contributes to the proposed method.

Questions for authors

See weaknesses above.

3 comments

Dear Reviewer,

Thank you for your careful review, and for pointing us to these three related works. Your comments have helped us strengthen the paper.

We have added Appendix I ("Relation to Hybrid Evolutionary–Gradient Methods"), which starts at the section of that name after Appendix H. In brief, it contains:

  • A design comparison with EDEAdam, AdaptiveGDE and Neural DE, based on what each paper reports. EDEAdam evolves two sub-populations in parallel, one with a population version of Adam and one with a modified DE, and they interact only through periodic exchange of good individuals. AdaptiveGDE forms niches by similarity and applies a gradient step after a DE mutation for every individual. Neural DE is a DE variant with an occasional bounded local search. EvE is a four-member population in which an Adam burst is triggered only when a greedy DE proposal fails to improve its slot, with persistent per-slot Adam state, and all forward and backward passes are charged against one common cost budget (Appendix A.1). We state clearly that combining evolution with gradient steps is not new, and that our contribution is this particular design point.
  • A map from our ablations to these design families. Appendix G ("Tuned Baselines and Component Ablations") compares full EvE with variants that resemble each family, under the same tuned hyperparameters and budget: DE only with no gradient fallback (resembling Neural DE without the local search), an Adam burst after every proposal (resembling the gradient step after every DE mutation in AdaptiveGDE, although our variant keeps greedy selection), four Adam trajectories with no DE (resembling the Adam sub-population of EDEAdam), and a single Adam trajectory with greedy accept/revert. Removing the gradient fallback or replacing DE with Adam is never better than full EvE. These variants are controls inside our framework, not re-implementations of the cited methods. Appendix G also reports diagnostics of the fallback, such as how often the DE child is accepted and how often the Adam burst improves a slot.

On running these three methods as baselines. We looked for reference implementations and could not locate any for EDEAdam or AdaptiveGDE in their manuscripts. For Neural DE, the authors' code is part of their general-purpose library OPTIMUS, which we did not integrate. Re-implementing these methods from their descriptions would risk comparing against a method that differs from the original, which is why we did not do so.

If you happen to have implementations of any of these three methods, we would be glad to run them under our equal-cost budget and add the comparison as another subsection of Appendix I. We would also be grateful for any pointer to code we may have missed.

The reference implementation of EvE is available at https://anonymous.4open.science/r/eve-2D59/. It is a standard torch.optim.Optimizer and should be very easy to run; the README gives a short usage example.

We hope the new appendix resolves your concerns. If it does, we would be grateful if you would consider revising your score. We are happy to answer any further questions.

Thank you again for your time and effort.

Best regards, The Authors

The authors have adequately addressed most of my concerns. Therefore, I have raised my score.

Dear Reviewer,

Thank you for your reply and for raising your score. Could you tell us which concerns remain? We would be glad to address them before the discussion ends on November 18.

Best regards, The Authors

Reviewer_UH9TOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

In this paper, authors introduce EvE, a four-member differential-evolution optimizer that applies short Adam updates when an evolutionary proposal fails to improve the incumbent. The intended application is inexpensive configuration ranking during hyper-parameter or architecture search, rather than final model training. Experiments cover synthetic optimization problems, neural-network training, and successive halving on Adult. The authors report 1.7–3.9× shorter training times under their budget accounting, generally at the expense of predictive performance, and early-stage configuration rankings that agree reasonably well with Adam.

Critical strengths
  1. The configuration-ranking application is interesting, EVE need not match Adam's final accuracy to be useful for screening configurations. Comparing EVE-Adam rank agreement with Adam's agreement across training seeds is a sensible reference and makes the correlation results easier to interpret.

  2. Authors also report unfavorable results, including the predictive-performance gap and the GSM8K accuracy degradation after fine-tuning. The validation-based learning rate check is also useful, uspecially since it exposes a dependence on the selection rule in the GSM8K results.

Critical weaknesses
  1. I found that it's difficult to tell whether the proposed failure-triggered Adam fallback is responsible for the gains. The paper lacks comparisons with DE alone, Adam bursts under the same greedy acceptance rule, and a hybrid that applies gradient updates unconditionally.

  2. The synthetic comparison seems weakened by using Adam with one untuned learning rate of 0.1 across all problems and dimensions, while EvE's initialization was chosen after preliminary comparisons. Without these controls and a stronger Adam baseline, the aggregate win count provides limited evidence for the proposed design.

  3. Table 10 compares EvE and Adam at the first search rung, not EvE's early rankings against Adam's full-training results, the problem here is, agreement at an early stage doesn't eastablish that the right configurations survive pruning.

Questions for authors
  1. Wow well do EvE's early rankings predict Adam's rankings after full training?

  2. When EvE selected configurations are retrained with Adam, how does their final test performance compare with an Adam-based search at the same total wall-clock budget?

  3. Do the synthetic conclusions survive tuning Adam's learning rate?

1 comment

Dear Reviewer,

Thank you for your detailed and careful review. Your comments have helped us strengthen the paper.

We have added two appendices that address your three concerns and your questions directly.

Appendix G ("Tuned Baselines and Component Ablations")

  • Tuned Adam on the synthetic suite (your question on the untuned learning rate 0.1). Adam and EvE each get their own tuning study with the same search procedure and the same number of trials. Adam's search covers the learning rate, both betas, weight decay, AMSGrad, warmup and decay. EvE's search includes its own hyperparameters, and the paper's configuration is trial 0, so tuning is never worse than the paper's setting on the tuning seeds. Selection uses tuning seeds, confirmation uses fresh seeds, and the final comparison uses separate evaluation seeds. Over the 70 (problem, dimension) cells, EvE is better in 25, ties in 21, and is worse in 24. EvE wins every Rastrigin cell and most Schwefel cells; Adam is ahead on Rosenbrock. So the earlier advantage does not extend to every problem once Adam is tuned, and the appendix reports it that way. A full per-problem table shows every cell.
  • Component ablations (DE alone, Adam bursts, unconditional gradients). Over the same 70 cells we compare full EvE with four variants: DE only with no gradient fallback; an Adam burst after every proposal (the unconditional-gradient hybrid); four Adam trajectories with no DE; and one Adam trajectory with the same greedy accept/revert rule. Full EvE is better than DE only in 54 cells (never worse), better than four Adam trajectories in 59 (never worse), better than the unconditional-burst variant in 18 with 51 ties and 1 loss, and better than the single greedy Adam trajectory in 51 with 13 ties and 6 losses. All variants use the tuned hyperparameters and budget of full EvE.
  • Fallback diagnostics. The DE child is accepted in 50–69% of steps; when the fallback fires, its refined point beats the DE child in 37–59% of bursts.

Appendix H ("Early Rankings, Selection Regret, and Retraining Cost on UCI Adult")

  • Early EvE ranking versus Adam's full-training ranking (your first question). On the same 48 configurations, rung schedule and 5 seeds as the Adult search, we compute Kendall's tau between each method's ranking at a rung and Adam's final-rung ranking on other seeds. The ground truth is always averaged over seeds the method did not train on. We also report Adam's own early ranking from a different seed, which shows how much agreement seed noise alone allows. EvE's early ranking predicts Adam's final ranking nearly as well as Adam's own early ranking does (gap at most about 0.1 in tau), and at the last rung before the final one both are almost close to the seed-to-seed ceiling. At the earliest rung both are well below that ceiling.
  • Configurations selected by EvE, retrained with Adam (your second question). The top-ranked configuration from each search is retrained with Adam to the full budget. Selection regret and test accuracy of EvE-selected and Adam-selected configurations are within seed spread: for example 0.0010 vs 0.0012 regret and 0.8540 vs 0.8536 test accuracy on the hyperparameter space. With equal total wall-clock, where the Adam-based search receives only the random subset of 44 of the 48 configurations that fit in EvE's search-plus-retraining time, the result is unchanged (regret 0.0013 vs 0.0010; test accuracy 0.8536 vs 0.8540 on the hyperparameter space).

The Appendix H measurements are on the two Adult search spaces, as stated there.

Both appendices come after Appendices A–F.

The reference implementation is available at https://anonymous.4open.science/r/eve-2D59/. It is a standard torch.optim.Optimizer and should be very easy to run; the README gives a short usage example.

We hope the new appendices resolve your concerns. If they do, we would be grateful if you would consider revising your score. We are happy to answer any further questions.

Thank you again for your time and effort.

Best regards, The Authors

Reviewer_MiDVOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a four-member differential evolution optimizer with an Adam fallback when a candidate fails to improve. It evaluates EvE on synthetic functions, MNIST, Adult, and LoRA fine-tuning. EvE exhausts the proposed evaluation-cost budget faster, generally with worse final accuracy or loss. The intended application is cheap configuration ranking before more expensive training.

Critical strengths

The idea of using a cheaper optimizer as a search proxy is worth investigating. The Adult ranking experiment addresses this directly, and comparing EvE–Adam agreement with Adam's agreement across seeds is useful. I also appreciate the reporting of unfavorable results, including higher memory use, greater variance, and lower downstream accuracy.

Critical weaknesses

The evaluation is insufficient for the paper's scope. MNIST and Adult provide limited evidence for an alternative neural-network optimizer, and LoRA fine-tuning does not establish performance when training all parameters from scratch. I would expect at least a small language-model pretraining experiment. This is feasible without a large cluster: established setups such as nanochat and the nanoGPT speedrun provide reproducible workflows in less than ten GPU hours. A carefully controlled small-model experiment would be more informative than adding further synthetic dimensions.

Hyperparameter tuning is central to a credible optimizer comparison. The literature contains many apparent improvements that depend on the tuning protocol rather than the update rule. Choi et al. demonstrate this sensitivity, Schmidt et al. provide systematic tuned comparisons, and AlgoPerf explicitly addresses tuning and time-to-result evaluation. The paper needs to place its results in this context. Here, Adam uses an untuned learning rate of 0.1 throughout the synthetic suite. Neural-network tuning is largely restricted to five learning rates, with several LoRA baselines selecting the lowest value tested. There is no systematic examination of schedules, warmup, moment parameters, or optimizer-specific regularization. The main results also select learning rates using test performance, although a validation-based check is provided.

For the claimed advantage to be convincing, I would expect established, well-tuned baselines and training settings. A pinned nanochat or nanoGPT speedrun recipe would provide a concrete starting point, with architecture, data, precision, and implementation held fixed while the optimizer is changed and tuned appropriately. Muon is absent, as are other competitive matrix-based methods. Recent comparisons of Muon, SOAP, and AdamW further underline the importance of tuning and training horizon. Using the same numerical learning-rate grid for every optimizer is not sufficient.

The budget accounting is another concern. Appendix A.1 converts backward time into evaluation units using each run's own average forward time. Equal charged budgets therefore need not represent equal computation. Finishing this budget faster does not by itself demonstrate better efficiency. The relevant measurements are held-out quality at equal wall-clock time and time to a fixed quality target, including Adam stopped earlier.

The search-proxy claim also remains incomplete. Rank agreement is measured against Adam at the first, small-budget rung, rather than against performance after substantial training. Both early rankings could agree while discarding configurations that eventually perform well. The paper should test whether configurations selected by EvE remain competitive after training with Adam, reporting selection regret and the total search-plus-training cost.

Finally, the contribution of the hybrid components is unclear without comparisons to DE alone, an unconditional fallback, and multiple Adam trajectories. No implementation link is provided; release is promised for the final version. This makes the unusual accounting and reported speedups difficult to verify.

No code was provided in the supplementary file or link at submission time, as far as I can see, which is again not up to the standards for an optimizer paper.

Questions for authors
  1. Does EvE remain competitive in a small, established language-model pretraining setup against carefully tuned AdamW and Muon? (Example: NanoChat)

  2. Do the gains survive broader validation-based tuning and equal-time or time-to-target comparisons?

  3. Do configurations selected by EvE perform well after full Adam training, including the total cost of selection and training?

  4. Can the authors provide the implementation, timing breakdown, fallback statistics, and component ablations?

1 comment

Dear Reviewer,

Thank you for your detailed and careful review. Your comments have helped us strengthen the paper.

Why we compare at equal charged cost. Differential evolution and Adam count progress in different units. DE proposes a candidate and scores it with a forward pass. Adam takes a step that needs a forward and a backward pass. Comparing "number of steps" or "number of batches" would therefore favour one method by construction. Counting evaluations is the standard convention in evolutionary computation, but it says nothing about what a backward pass should cost, so we fill that gap explicitly (Appendix A.1). Scoring one candidate with a forward pass costs 1 unit. A backward pass costs its measured time divided by the mean forward time of that run. Every forward and backward pass either method performs, including the Adam bursts inside EvE, is charged at this rate, and both methods stop when the same budget is spent. Neither method gets free evaluations, and neither is counted in a unit that suits it. The backward cost is measured on the actual hardware and model rather than assumed from a fixed ratio. We report wall-clock only as a secondary measure, because it depends on the data pipeline and the machine, as the CPU-fetch versus GPU rows of the nanoGPT table in Appendix G show.

We have added Appendix G ("Tuned Baselines and Component Ablations"), which addresses the concerns in your review. In brief, it contains:

  • Tuned baselines on the synthetic suite. Adam is tuned with the same search procedure and the same trial budget as EvE (selection on tuning seeds, confirmation on fresh seeds, final comparison on separate evaluation seeds). Against tuned Adam, EvE is better on 25 cells, ties on 21, and is worse on 24 of the 70 cells. A full per-problem table is included so every result can be checked.
  • Component ablations. We remove the gradient fallback, replace DE with independent Adam trajectories, apply the Adam burst after every proposal, and use a single Adam trajectory with greedy accept/revert. Removing the fallback or replacing DE with Adam is never better than full EvE.
  • Fallback diagnostics. How often the DE child is accepted, how often the Adam burst improves a slot, how many bursts occur per 100 steps, and what share of the budget they use.
  • nanoGPT (Shakespeare-char) against tuned AdamW and Muon. We report the time each optimizer takes to exhaust the same charged budget, and the best validation loss it reaches.

Appendix G comes after Appendices A–F.

The reference implementation is available at https://anonymous.4open.science/r/eve-2D59/. It is a standard torch.optim.Optimizer and should be very easy to run; the README gives a short usage example.

We hope the new appendix resolves your concerns. If it does, we would be grateful if you would consider revising your score. We are happy to answer any further questions.

Thank you again for your time and effort.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.