- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates a counterintuitive phenomenon in evolution strategies: even when all perturbed models perform worse than the original model, aggregating the perturbations using weights based on their relative rewards can still improve the original model. Through LLM fine-tuning and MNIST experiments, together with theoretical analysis, the authors show that relative performance among worse candidates still contains useful directional information, while aggregation reduces noise.
Critical strengths
The paper investigates an interesting question: how can aggregating perturbations weighted by their relative rewards improve the original model, even when all perturbed models perform worse?
Critical weaknesses
- The paper reports final results only on the training data, leaving generalization to held-out test data unverified. The authors also do not report the wall-clock time per iteration. Since each iteration requires multiple forward passes, training appears less efficient than SFT or RL.
- The discussion of related work is insufficient, making it difficult to assess the paper’s contributions and novelty. For example, The Blessing of Dimensionality in LLM Fine-tuning [1] has already analyzed why ES can effectively optimize billion-parameter models using only around 30 perturbations. Evolution Strategies – A Comprehensive Introduction [2] has already explained the mechanism by which aggregation preserves beneficial directional components while reducing orthogonal noise.
Citations:
- Qiyao Liang, Jinyeop Song, Yizhou Liu, Jeff Gore, Ila Fiete, Risto Miikkulainen, and Xin Qiu. “The Blessing of Dimensionality in LLM Fine-tuning: A Variance-Curvature Perspective.” arXiv preprint arXiv:2602.00170, 2026.
- Beyer, HG., Schwefel, HP. Evolution strategies – A comprehensive introduction. Natural Computing 1, 3–52 (2002).
Questions for authors
- What implications do the authors believe this work has for applying perturbation-based optimization to modern LLMs? Line 116 states, “These findings thus demonstrate a previously unknown mechanism that allows scaling perturbation-based optimization to modern LLMs.” However, beyond explaining the mechanism, I do not see any improvements to the application of ES to LLMs. Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning [3] has already demonstrated that ES can directly fine-tune all parameters of billion-parameter LLMs.
- Could the authors compare their work with the related studies mentioned in the weaknesses section to clarify this paper’s novelty and contributions?
- Could the authors clarify the experimental settings used to obtain the reported results, including the training/test splits, and provide an analysis of training efficiency, including wall-clock time per iteration and comparisons with SFT or RL?
Citations: 3. Xin Qiu, Yulu Gan, Conor F. Hayes, Qiyao Liang, Yinggan Xu, Roberto Dailey, Elliot Meyerson, Babak Hodjat, and Risto Miikkulainen. “Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning.” arXiv preprint arXiv:2509.24372, 2026
4 comments
Q1-Q3 responses
Q1. What implications do the authors believe this work has for applying perturbation-based methods to modern LLMs? Line 116 states, "These findings thus demonstrate a previously unknown mechanism that allows scaling perturbation-based optimization to modern LLMs." However, beyond explaining the mechanism, I do not see any improvements to the application of ES to LLMs. Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning [3] has already demonstrated that ES can directly fine-tune all parameters of billion-parameter LLMs.
We believe that identifying the phenomenon itself changes which modifications and methods may be applied to ES for fine-tuning. The most immediate example is elitism or Best-of-N selection methods, which our results indicate will often fail to contribute to optimization performance. Additionally, our analysis provides a framework for understanding the relationship between optimization and population size, and how curvature governs optimal perturbation and population sizes.
Q2. Could the authors compare their work with the related studies mentioned in the weaknesses section to clarify this paper's novelty and contributions?
We intend to add more comparison with the aforementioned works, both in our response to W2 above and in the paper's introduction, to be uploaded before the rebuttal deadline.
Q3. Could the authors clarify the experimental settings used to obtain the reported results, including the training/test splits, and provide an analysis of training efficiency, including wall-clock time per iteration and comparisons with SFT or RL?
For our LLM fine-tuning setup in both Countdown and Sudoku, the training set was a static set of 800 questions, and the batch size was set to the entire training set. The test set for Countdown is 1,400 held-out problems, while for Sudoku it is 200 held-out problems. Training for both was done for 500 iterations. We do not have equivalent RL times on the machine these tests were run on; however, we do have an RL comparison with a smaller batch size of 200 on 8×H200 GPUs, as reported in our response to W1:
| Model | ES | GRPO |
|---|---|---|
| Qwen 2.5 1.5B | 2h 48m | 16h 58m |
| Qwen 2.5 3B | 5h 35m | 25h 12m |
For the 1.5B model, naive extrapolation to batch size 800 would place the ES time at 11h 12m, still competitive with GRPO on a per-iteration level. As mentioned in our response to W1, we believe the performance and competitiveness of ES vs. RL for fine-tuning is already well established by Qiu et al. [22] and follow-ups (Hoy et al. [18], Hayes et al. [23]).
References
References
[1] H.-G. Beyer and H.-P. Schwefel. Evolution strategies – A comprehensive introduction. Natural Computing, 1(1):3–52, 2002.
[2] H.-G. Beyer. Toward a theory of evolution strategies: On the benefits of sex — the (μ/μ, λ) theory. Evolutionary Computation, 3(1):81–111, 1995.
[3] Y. Akimoto, A. Auger, T. Glasmachers, and D. Morinaga. Global linear convergence of evolution strategies on more than smooth strongly convex functions. SIAM Journal on Optimization, 32(2):1402–1429, 2022. doi:10.1137/20M1373815
[4] O. Ait Elhara, A. Auger, and N. Hansen. A median success rule for non-elitist evolution strategies: Study of feasibility. In Proc. GECCO 2013, pp. 415–422. ACM, 2013. doi:10.1145/2463372.2463429
[5] D. V. Arnold and H.-G. Beyer. Local performance of the (1+1)-ES in a noisy environment. IEEE Transactions on Evolutionary Computation, 6(1):30–41, 2002.
[6] I. Loshchilov. A computationally efficient limited memory CMA-ES for large scale optimization. In Proc. GECCO 2014, pp. 397–404. ACM, 2014. arXiv:1404.5520
[7] D. Wierstra, T. Schaul, T. Glasmachers, Y. Sun, J. Peters, and J. Schmidhuber. Natural evolution strategies. Journal of Machine Learning Research, 15:949–980, 2014.
[8] O. Li, J. Harrison, J. Sohl-Dickstein, V. Smith, and L. Metz. Variance-reduced gradient estimation via noise-reuse in online evolution strategies. In Advances in Neural Information Processing Systems 36 (NeurIPS), 2023.
[9] P. Vicol. Low-variance gradient estimation in unrolled computation graphs with ES-Single. In Proc. ICML 2023, PMLR 202, 2023.
[10] P. Vicol, L. Metz, and J. Sohl-Dickstein. Unbiased gradient estimation in unrolled computation graphs with persistent evolution strategies. In Proc. ICML 2021, PMLR 139:10553–10563, 2021.
[11] Y. Tang, K. Choromanski, and A. Kucukelbir. Variance reduction for evolution strategies via structured control variates. In Proc. AISTATS 2020, PMLR 108:845–855, 2020.
[12] T. Glasmachers, T. Schaul, Y. Sun, D. Wierstra, and J. Schmidhuber. Exponential natural evolution strategies. In Proc. GECCO 2010, pp. 393–400. ACM, 2010.
[13] Y. Sun, D. Wierstra, T. Schaul, and J. Schmidhuber. Efficient natural evolution strategies. In Proc. GECCO 2009, pp. 539–546. ACM, 2009.
[14] X. Zhang, J. Clune, and K. O. Stanley. On the relationship between the OpenAI evolution strategy and stochastic gradient descent. arXiv:1712.06564, 2017.
[15] T. Salimans, J. Ho, X. Chen, S. Sidor, and I. Sutskever. Evolution strategies as a scalable alternative to reinforcement learning. arXiv:1703.03864, 2017.
[16] J. Lehman, J. Chen, J. Clune, and K. O. Stanley. ES is more than just a traditional finite-difference approximator. In Proc. GECCO 2018, pp. 450–457. ACM, 2018. arXiv:1712.06568
[17] S. Malladi, T. Gao, E. Nichani, A. Damian, J. D. Lee, D. Chen, and S. Arora. Fine-tuning language models with just forward passes. In Advances in Neural Information Processing Systems 36 (NeurIPS), 2023.
[18] W. Hoy, B. Wang, and X. Pan. Matching accuracy, different geometry: Evolution strategies vs GRPO in LLM post-training. arXiv:2604.01499, 2026.
[19] Q. Liang, J. Song, Y. Liu, J. Gore, I. Fiete, R. Miikkulainen, and X. Qiu. The blessing of dimensionality in LLM fine-tuning: A variance-curvature perspective. arXiv:2602.00170, 2026.
[20] Y. Gan and P. Isola. Neural thickets: Diverse task experts are dense around pretrained weights. In Proc. ICML 2026. arXiv:2603.12228
[21] B. Sarkar, M. Fellows, J. A. Duque, et al. Evolution strategies at the hyperscale. PMLR 306, 2026. arXiv:2511.16652
[22] X. Qiu, Y. Gan, C. F. Hayes, et al. Evolution strategies at scale: LLM fine-tuning beyond reinforcement learning. PMLR 306, 2026. arXiv:2509.24372
[23] C. F. Hayes, E. Meyerson, K. Schweighofer, et al. Beyond the best guess: Improving LLM solution coverage with evolution strategies. arXiv:2608.12679, 2026.
Thank you for clarifying the held-out results and reporting the ES and GRPO runtimes. I remain unconvinced that the population-size analysis establishes a new mechanism for scaling ES to LLMs, and I would have liked to see whether using it to choose the population size improves fine-tuning in practice. Therefore I plan to keep my original rating.
W1 and W2 responses
We would like to thank the reviewer for the constructive feedback and careful analysis of our work. We respond to the identified weaknesses and questions below.
W1. "The paper reports final results only on training data…" and concerns on training efficiency.
Thank you for your comment. We want to clarify that we have also observed the learning curves on held-out test datasets, and the performance improvement does generalize to the test sets. Therefore, the paradox phenomenon occurs for both the training and test datasets. We agree with the reviewer that generalization is important for understanding the paradox, and we have added the test-set results to the revision. Figure 1 is now amended to show both training and held-out test curves, both of which show consistent improvement given worse perturbations.
Within ES, each iteration requires multiple forward passes per population member; however, only one rollout is required per population member, so the number of rollouts via population size can be considered analogous to the number of rollouts via group size in GRPO. Additionally, RL requires a backpropagation step, which can take 2–3 times longer than a rollout step. Given this, on a per-iteration level, ES can often be faster than GRPO. When performing 500 training iterations on the Countdown fine-tuning task on 8×H200 GPUs, we observe the following wall-clock times:
However, the focus of this paper is on the phenomenon of a population-based method performing optimization while all population members remain worse than the existing solution. We believe the performance and competitiveness of ES vs. RL for fine-tuning is already well established by Qiu et al. [22], as well as numerous follow-ups (Hoy et al. [18], Hayes et al. [23]), and we will add discussion based on these prior works.
W2. The discussion of related work is insufficient.
We agree with the reviewer that the related work section was underdeveloped. We intend to amended it to more carefully compare with the aforementioned works and will be greatly extending the list of citations to cover classical ES, NES, and modern ES for LLMs in more detail.
Specifically, with regard to the two papers mentioned by the reviewer:
The Blessing of Dimensionality [19] identifies only that perturbation-based methods can be used, not how to determine optimal population sizes. It additionally focuses on "Best of N" perturbations as a measure of learnability via ES, implying that better perturbations are needed for learning. In contrast, our analysis shows that outperforming offspring are not necessary for optimization.
Evolution Strategies – A Comprehensive Introduction [1] analyzes a separate form of ES, which averages among elites in (μ/μ,λ)-ES, rather than the NES-style ES used in LLM fine-tuning. While it does identify that averaging reduces noise, it does not identify that the signal portion remains constant with respect to population size, since it analyzes a different form of ES in low-dimensional space.