acceptodds
Withdrawn from ICLR 2027

The Evolution Strategies Paradox: Better Models from Worse Perturbations

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

How is it possible for perturbation-based methods like Evolution Strategies (ES) to optimize billion-parameter large language models with population sizes of thirty? Classical evolutionary methods are highly dependent on finding good candidates within generated populations. However, when performing ES in various fine tuning tasks for large language models, a phenomenon can occur where all population members consistently perform worse than the current model, yet with each iteration the model consistently improves. This paradox occurs across models, tasks, and even domains. This paper first reports this ES paradox, then examines how the local reward landscape as well as the ES estimator properties contribute to this paradox: in high-dimensional spaces, exploration leads to worse performance, but still highlights good directions; aggregating those directions then allows making local improvements. These findings provide guidance for taking advantage of perturbation-based methods in LLM fine-tuning and in high-dimensional optimization in general.

Primary area:
reinforcement learning
Keywords:
Evolution Strategies, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Dp8jOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates a counterintuitive phenomenon in evolution strategies: even when all perturbed models perform worse than the original model, aggregating the perturbations using weights based on their relative rewards can still improve the original model. Through LLM fine-tuning and MNIST experiments, together with theoretical analysis, the authors show that relative performance among worse candidates still contains useful directional information, while aggregation reduces noise.

Critical strengths

The paper investigates an interesting question: how can aggregating perturbations weighted by their relative rewards improve the original model, even when all perturbed models perform worse?

Critical weaknesses
  1. The paper reports final results only on the training data, leaving generalization to held-out test data unverified. The authors also do not report the wall-clock time per iteration. Since each iteration requires multiple forward passes, training appears less efficient than SFT or RL.
  2. The discussion of related work is insufficient, making it difficult to assess the paper’s contributions and novelty. For example, The Blessing of Dimensionality in LLM Fine-tuning [1] has already analyzed why ES can effectively optimize billion-parameter models using only around 30 perturbations. Evolution Strategies – A Comprehensive Introduction [2] has already explained the mechanism by which aggregation preserves beneficial directional components while reducing orthogonal noise.

Citations:

  1. Qiyao Liang, Jinyeop Song, Yizhou Liu, Jeff Gore, Ila Fiete, Risto Miikkulainen, and Xin Qiu. “The Blessing of Dimensionality in LLM Fine-tuning: A Variance-Curvature Perspective.” arXiv preprint arXiv:2602.00170, 2026.
  2. Beyer, HG., Schwefel, HP. Evolution strategies – A comprehensive introduction. Natural Computing 1, 3–52 (2002).
Questions for authors
  1. What implications do the authors believe this work has for applying perturbation-based optimization to modern LLMs? Line 116 states, “These findings thus demonstrate a previously unknown mechanism that allows scaling perturbation-based optimization to modern LLMs.” However, beyond explaining the mechanism, I do not see any improvements to the application of ES to LLMs. Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning [3] has already demonstrated that ES can directly fine-tune all parameters of billion-parameter LLMs.
  2. Could the authors compare their work with the related studies mentioned in the weaknesses section to clarify this paper’s novelty and contributions?
  3. Could the authors clarify the experimental settings used to obtain the reported results, including the training/test splits, and provide an analysis of training efficiency, including wall-clock time per iteration and comparisons with SFT or RL?

Citations: 3. Xin Qiu, Yulu Gan, Conor F. Hayes, Qiyao Liang, Yinggan Xu, Roberto Dailey, Elliot Meyerson, Babak Hodjat, and Risto Miikkulainen. “Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning.” arXiv preprint arXiv:2509.24372, 2026

4 comments
AuthorsOpenReview ↗

W1 and W2 responses

We would like to thank the reviewer for the constructive feedback and careful analysis of our work. We respond to the identified weaknesses and questions below.

W1. "The paper reports final results only on training data…" and concerns on training efficiency.

Thank you for your comment. We want to clarify that we have also observed the learning curves on held-out test datasets, and the performance improvement does generalize to the test sets. Therefore, the paradox phenomenon occurs for both the training and test datasets. We agree with the reviewer that generalization is important for understanding the paradox, and we have added the test-set results to the revision. Figure 1 is now amended to show both training and held-out test curves, both of which show consistent improvement given worse perturbations.

Within ES, each iteration requires multiple forward passes per population member; however, only one rollout is required per population member, so the number of rollouts via population size can be considered analogous to the number of rollouts via group size in GRPO. Additionally, RL requires a backpropagation step, which can take 2–3 times longer than a rollout step. Given this, on a per-iteration level, ES can often be faster than GRPO. When performing 500 training iterations on the Countdown fine-tuning task on 8×H200 GPUs, we observe the following wall-clock times:

ModelESGRPO
Qwen 2.5 1.5B2h 48m16h 58m
Qwen 2.5 3B5h 35m25h 12m

However, the focus of this paper is on the phenomenon of a population-based method performing optimization while all population members remain worse than the existing solution. We believe the performance and competitiveness of ES vs. RL for fine-tuning is already well established by Qiu et al. [22], as well as numerous follow-ups (Hoy et al. [18], Hayes et al. [23]), and we will add discussion based on these prior works.

W2. The discussion of related work is insufficient.

We agree with the reviewer that the related work section was underdeveloped. We intend to amended it to more carefully compare with the aforementioned works and will be greatly extending the list of citations to cover classical ES, NES, and modern ES for LLMs in more detail.

Specifically, with regard to the two papers mentioned by the reviewer:

  • The Blessing of Dimensionality [19] identifies only that perturbation-based methods can be used, not how to determine optimal population sizes. It additionally focuses on "Best of N" perturbations as a measure of learnability via ES, implying that better perturbations are needed for learning. In contrast, our analysis shows that outperforming offspring are not necessary for optimization.

  • Evolution Strategies – A Comprehensive Introduction [1] analyzes a separate form of ES, which averages among elites in (μ/μ,λ)(\mu/\mu, \lambda)-ES, rather than the NES-style ES used in LLM fine-tuning. While it does identify that averaging reduces noise, it does not identify that the signal portion remains constant with respect to population size, since it analyzes a different form of ES in low-dimensional space.

AuthorsOpenReview ↗

Q1-Q3 responses

Q1. What implications do the authors believe this work has for applying perturbation-based methods to modern LLMs? Line 116 states, "These findings thus demonstrate a previously unknown mechanism that allows scaling perturbation-based optimization to modern LLMs." However, beyond explaining the mechanism, I do not see any improvements to the application of ES to LLMs. Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning [3] has already demonstrated that ES can directly fine-tune all parameters of billion-parameter LLMs.

We believe that identifying the phenomenon itself changes which modifications and methods may be applied to ES for fine-tuning. The most immediate example is elitism or Best-of-N selection methods, which our results indicate will often fail to contribute to optimization performance. Additionally, our analysis provides a framework for understanding the relationship between optimization and population size, and how curvature governs optimal perturbation and population sizes.

Q2. Could the authors compare their work with the related studies mentioned in the weaknesses section to clarify this paper's novelty and contributions?

We intend to add more comparison with the aforementioned works, both in our response to W2 above and in the paper's introduction, to be uploaded before the rebuttal deadline.

Q3. Could the authors clarify the experimental settings used to obtain the reported results, including the training/test splits, and provide an analysis of training efficiency, including wall-clock time per iteration and comparisons with SFT or RL?

For our LLM fine-tuning setup in both Countdown and Sudoku, the training set was a static set of 800 questions, and the batch size was set to the entire training set. The test set for Countdown is 1,400 held-out problems, while for Sudoku it is 200 held-out problems. Training for both was done for 500 iterations. We do not have equivalent RL times on the machine these tests were run on; however, we do have an RL comparison with a smaller batch size of 200 on 8×H200 GPUs, as reported in our response to W1:

ModelESGRPO
Qwen 2.5 1.5B2h 48m16h 58m
Qwen 2.5 3B5h 35m25h 12m

For the 1.5B model, naive extrapolation to batch size 800 would place the ES time at 11h 12m, still competitive with GRPO on a per-iteration level. As mentioned in our response to W1, we believe the performance and competitiveness of ES vs. RL for fine-tuning is already well established by Qiu et al. [22] and follow-ups (Hoy et al. [18], Hayes et al. [23]).

AuthorsOpenReview ↗

References

References

[1] H.-G. Beyer and H.-P. Schwefel. Evolution strategies – A comprehensive introduction. Natural Computing, 1(1):3–52, 2002.

[2] H.-G. Beyer. Toward a theory of evolution strategies: On the benefits of sex — the (μ/μ, λ) theory. Evolutionary Computation, 3(1):81–111, 1995.

[3] Y. Akimoto, A. Auger, T. Glasmachers, and D. Morinaga. Global linear convergence of evolution strategies on more than smooth strongly convex functions. SIAM Journal on Optimization, 32(2):1402–1429, 2022. doi:10.1137/20M1373815

[4] O. Ait Elhara, A. Auger, and N. Hansen. A median success rule for non-elitist evolution strategies: Study of feasibility. In Proc. GECCO 2013, pp. 415–422. ACM, 2013. doi:10.1145/2463372.2463429

[5] D. V. Arnold and H.-G. Beyer. Local performance of the (1+1)-ES in a noisy environment. IEEE Transactions on Evolutionary Computation, 6(1):30–41, 2002.

[6] I. Loshchilov. A computationally efficient limited memory CMA-ES for large scale optimization. In Proc. GECCO 2014, pp. 397–404. ACM, 2014. arXiv:1404.5520

[7] D. Wierstra, T. Schaul, T. Glasmachers, Y. Sun, J. Peters, and J. Schmidhuber. Natural evolution strategies. Journal of Machine Learning Research, 15:949–980, 2014.

[8] O. Li, J. Harrison, J. Sohl-Dickstein, V. Smith, and L. Metz. Variance-reduced gradient estimation via noise-reuse in online evolution strategies. In Advances in Neural Information Processing Systems 36 (NeurIPS), 2023.

[9] P. Vicol. Low-variance gradient estimation in unrolled computation graphs with ES-Single. In Proc. ICML 2023, PMLR 202, 2023.

[10] P. Vicol, L. Metz, and J. Sohl-Dickstein. Unbiased gradient estimation in unrolled computation graphs with persistent evolution strategies. In Proc. ICML 2021, PMLR 139:10553–10563, 2021.

[11] Y. Tang, K. Choromanski, and A. Kucukelbir. Variance reduction for evolution strategies via structured control variates. In Proc. AISTATS 2020, PMLR 108:845–855, 2020.

[12] T. Glasmachers, T. Schaul, Y. Sun, D. Wierstra, and J. Schmidhuber. Exponential natural evolution strategies. In Proc. GECCO 2010, pp. 393–400. ACM, 2010.

[13] Y. Sun, D. Wierstra, T. Schaul, and J. Schmidhuber. Efficient natural evolution strategies. In Proc. GECCO 2009, pp. 539–546. ACM, 2009.

[14] X. Zhang, J. Clune, and K. O. Stanley. On the relationship between the OpenAI evolution strategy and stochastic gradient descent. arXiv:1712.06564, 2017.

[15] T. Salimans, J. Ho, X. Chen, S. Sidor, and I. Sutskever. Evolution strategies as a scalable alternative to reinforcement learning. arXiv:1703.03864, 2017.

[16] J. Lehman, J. Chen, J. Clune, and K. O. Stanley. ES is more than just a traditional finite-difference approximator. In Proc. GECCO 2018, pp. 450–457. ACM, 2018. arXiv:1712.06568

[17] S. Malladi, T. Gao, E. Nichani, A. Damian, J. D. Lee, D. Chen, and S. Arora. Fine-tuning language models with just forward passes. In Advances in Neural Information Processing Systems 36 (NeurIPS), 2023.

[18] W. Hoy, B. Wang, and X. Pan. Matching accuracy, different geometry: Evolution strategies vs GRPO in LLM post-training. arXiv:2604.01499, 2026.

[19] Q. Liang, J. Song, Y. Liu, J. Gore, I. Fiete, R. Miikkulainen, and X. Qiu. The blessing of dimensionality in LLM fine-tuning: A variance-curvature perspective. arXiv:2602.00170, 2026.

[20] Y. Gan and P. Isola. Neural thickets: Diverse task experts are dense around pretrained weights. In Proc. ICML 2026. arXiv:2603.12228

[21] B. Sarkar, M. Fellows, J. A. Duque, et al. Evolution strategies at the hyperscale. PMLR 306, 2026. arXiv:2511.16652

[22] X. Qiu, Y. Gan, C. F. Hayes, et al. Evolution strategies at scale: LLM fine-tuning beyond reinforcement learning. PMLR 306, 2026. arXiv:2509.24372

[23] C. F. Hayes, E. Meyerson, K. Schweighofer, et al. Beyond the best guess: Improving LLM solution coverage with evolution strategies. arXiv:2608.12679, 2026.

Thank you for clarifying the held-out results and reporting the ES and GRPO runtimes. I remain unconvinced that the population-size analysis establishes a new mechanism for scaling ES to LLMs, and I would have liked to see whether using it to choose the population size improves fine-tuning in practice. Therefore I plan to keep my original rating.

Reviewer_TFpzOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This review was entirely human-written, GPT-6 Astra was used after the fact to adversarially contradict my review. I manually corrected some of my statements that I judged to have been misrepresented based on this.

This paper presents a theoretical and empirical analysis of an apparent paradox: evolution strategies (ES) on LLMs for post-training are able to find good directions from poor perturbations. This implies that ES is not simply doing "search" in the weight landscape. The authors present experiments to (1) highlight this phenomenon, (2) analyze why this phenomenon would occur with theoretical tools. They show that as the population size increases, the update size decreases both using their mathematical model and in experiments. They finally use their mathematical model to show that ES population sizes of 30 might be a "sweet spot" for LLM finetuning.

Overall, this paper has potentially interesting insights, but these are framed incorrectly, I believe the core thesis of the paper to be framed against a strawman (at least from the papers they referenced).

Critical strengths

S1 The paper is very well written, reads nicely and is not filled with jargon.

S2 The experiments are well executed and documented, the math looks correct. I did not thoroughly check the appendix, but the progression of the mathematical model is sensible in the main text.

Critical weaknesses

W1 The most pressing issue is that the framing of the paper is against a strawman, symptomatic of a missing related work section, this substantially hurts the paper.

  • They present an apparent "paradox": they claim L49-52 that results from Liang et al., (2026) and Gan & Isola, (2026) suggest that "ES is not based on gradient estimation, but relies on finding good candidates among sampled perturbations and moving towards them, like other evolutionary methods do." This is a substantial inference from these works as they do not directly make such a claim, the authors need to heavily revise this. The rest of the paper presents evidence trying to disprove this claim and frame it as a "paradox".

  • One of these citations: Liang et al. (2026) presents a hypothesis that LLMs are low-dimensional in curvature, which can explain how an informative gradient can be obtained from a small population.

  • The work does not situate itself in analysing ES methods: works such as Sarkar et al. 2026, Hoy et al. 2026 specifically looks at this in LLMs.

  • I counted a total of 15 unique citations in the paper, while obviously counting citations is a terrible proxy for paper quality, I am making a point that for a work analysing ES approximate gradients (for which there is a very expansive literature), this is simply too little, it is unclear if the authors are simply rediscovering previously known results.

W2 Inconsistent claims and poor presentation

  • Looking at Figure 1, there are clearly perturbations better than the parent model; especially in LLM countdown, the authors claim L79 "In each case, offspring models perform worse than the parent model for nearly all of training": this does not agree with their own figure. The central claim of the paper is at odds with its first figure.
  • Placeholder citations L386, L443.
  • Equation references are duplicated, leading them to point to the appendix rather than main text: this is confusing for the reader.
  • Figure 7 caption and legend do not match: one claims 1e-10 perturbations but the legend says 1e-3: the authors need to clarify this.
  • L451 "10" should be "Figure 10".
Questions for authors
  • Why do you believe that ES learning from worse perturbations is a "paradox"? I imagine as a toy model, a 1D quadratic function with noise, it is clear in this case that perturbations about the optimum will have worse fitnesses than the parent model, but still point to the right optimum in aggregate. See e.g. Figure 2 in Liang et al. (2026).
  • Have you extensively reviewed the literature to check that any of your results are novel?
  • Could you put the code in an anonymous repository so that I could have a look and understand the experiments in better detail? This would help me in my assessment of the paper.
4 comments
AuthorsOpenReview ↗

W1 and W2 response

W1. Framing the paper as a strawman

a) We thank the reviewer for the feedback. We agree that claiming Liang et al. [19] and Gan & Isola [20] imply a selection interpretation without providing sufficient explanation is confusing. To clarify, we believe Liang et al. [19] at least partially suggest a selection mechanism via their analysis of improving directions. They perform a "Best of N" analysis and estimate the ability of ES to optimize post-training based on how much uplift perturbations provide over the parent. We believe Best of N aligns with a selection interpretation of ES. Gan & Isola's paper presents a method that samples a large number of perturbations around the base model and either selects the best, or creates an ensemble subset of the best (note this is not a weight average but response voting). This is a selection method that, while not exactly ES, we believe heavily suggests that ES works through selection.

b) We agree that the work of Liang et al. [19] provides evidence that intrinsic dimension can explain how perturbation-based methods can optimize LLMs; however, it does not extend to what population size should be used. Given that LLM work like [21] uses populations in the thousands, while previous neural network ES used tens of thousands of samples, low intrinsic dimension only provides evidence that such problems can be optimized by perturbations, but not what population sizes can optimize them. Our analysis reveals this is a function of the total curvature, and once enough noise is removed from the estimate (at N≈30N \approx 30), the curvature penalty is removed and the optimal population size is achieved.

c) We thank the reviewer for identifying important literature to contextualize our work. In addition to the two identified references, we plan to include a number of additional citations in the paper, covering the following:

  • Classical evolution strategies: Evolution strategies – A comprehensive introduction [1], Toward a theory of evolution strategies: On the benefits of sex [2], Global linear convergence of evolution strategies on more than smooth strongly convex functions [3], A median success rule for non-elitist evolution strategies: Study of feasibility [4], and Local performance of the (1+1)-ES in a noisy environment [5]. These works study the progress rate, or per-step reward improvement, of classical ES methods ((1+1)(1+1), (μ/μ+λ)(\mu/\mu + \lambda), etc.) and identify a "success rate" criterion: in these forms of ES, optimal improvement comes when 1/5 of offspring outperform the parent.

  • NES-related works that treat ES as a gradient estimator: A computationally efficient limited memory CMA-ES for large scale optimization [6], Variance-reduced gradient estimation via noise-reuse in online evolution strategies [8], Low-variance gradient estimation in unrolled computation graphs with ES-Single [9], Unbiased gradient estimation in unrolled computation graphs with persistent evolution strategies [10], Variance reduction for evolution strategies via structured control variates [11], Exponential natural evolution strategies [12], and Efficient natural evolution strategies [13]. These works study progress rate with natural evolution strategies and frame ES as a gradient estimator; however, they work in regimes where the population is not significantly smaller (10,000×) than the dimension of the landscape.

  • Modern ES at LLM scale: Fine-tuning language models with just forward passes [17]. This was the first LLM-scale work to invoke low intrinsic dimension, later followed by Liang et al. [19].

W2. Inconsistent claims and poor presentation

a) With the presentation of the paradox, our intention is to show training runs where progress is clearly not attributable to discovered offspring that outperform the parent. In the original Countdown training run, although better children could occur, after 200 iterations the majority of iterations (78.7%) did not contain any children that improved over the parent. We will add clarifications to avoid this confusion.

b–e) We thank the reviewer for identifying the missing citations, incorrect equation references, and incorrect figure references. These have been fixed.

AuthorsOpenReview ↗

Q1-Q3 responses

Q1. Why do you believe that ES learning from worse perturbations is a "paradox"? I imagine as a toy model, a 1D quadratic function with noise, it is clear in this case that perturbations about the optimum will have worse fitnesses than the parent model, but still point to the right optimum in aggregate.

The reason we find the paradox to be a paradox is simple: it is counterintuitive that averaging population members could lead to improvement when all population members are worse than the existing solution.

The reviewer brings up a simple, intuitive counter-example; however, we feel that on further inspection, such simple counter-examples represent degenerate or uninteresting subsets of optimization, and realistic examples only start to occur at higher dimensions that are not possible to visualize.

In the 1D parabola, for the scenario to occur, offspring must either be biased to one side of the parent or be further from the parent than the optimal solution. While this may represent the very end of a training run, such a scenario does little to explain why the majority of training follows this paradox.

If the parabola is extended to nn dimensions, however, the paradox becomes possible for an extended period of optimization. We ran ES on nn-dimensional parabolas (sphere functions). The starting point was set 20σ20\sigma away from the optimum, where σ\sigma is the perturbation scale, and the run stopped when the distance to the optimum was ≤σ\le \sigma. We tracked the fraction of iterations in which all offspring perform worse than the parent at various dimensions. For low dimensions (d=1d = 1–33), the paradox essentially does not occur. As the dimension increases, it eventually becomes guaranteed, as shown in the table below.

Dimension dd123451050100
All perturbations worse (%)0.000.000.030.241.0910.8470.7598.80

Given that the paradox essentially cannot occur at human-understandable dimensions, yet becomes likely in high dimensions, we feel "paradox" is still an apt description of the phenomenon.

Q2. Have you extensively reviewed the literature to check that any of your results are novel?

We have extensively reviewed the literature on classical ES, NES, and modern ES variants. We found one instance of a hypothetical scenario in which an update could improve with worse population members (now cited as [2]); however, this hypothetical is limited in a number of ways. The offspring had to be hand-chosen rather than randomly generated, and the update mechanism used elite selection and averaging, not a z-score-weighted reward over all candidates. Furthermore, we find no instances of the paradox being identified in any real-world optimization problem with ES or other zeroth-order optimization methods. Additionally, we find no analysis of the distribution of random perturbation performance at various scales in LLM post-training, nor any analysis of the correlation in relative performance of such perturbations.

Q3. Could you put the code in an anonymous repository so that I could have a look and understand the experiments in better detail? This would help me in my assessment of the paper.

Yes, the code is available here: https://anonymous.4open.science/r/es_paradox_review-8AD5/

AuthorsOpenReview ↗

References

References

[1] H.-G. Beyer and H.-P. Schwefel. Evolution strategies – A comprehensive introduction. Natural Computing, 1(1):3–52, 2002.

[2] H.-G. Beyer. Toward a theory of evolution strategies: On the benefits of sex — the (μ/μ, λ) theory. Evolutionary Computation, 3(1):81–111, 1995.

[3] Y. Akimoto, A. Auger, T. Glasmachers, and D. Morinaga. Global linear convergence of evolution strategies on more than smooth strongly convex functions. SIAM Journal on Optimization, 32(2):1402–1429, 2022. doi:10.1137/20M1373815

[4] O. Ait Elhara, A. Auger, and N. Hansen. A median success rule for non-elitist evolution strategies: Study of feasibility. In Proc. GECCO 2013, pp. 415–422. ACM, 2013. doi:10.1145/2463372.2463429

[5] D. V. Arnold and H.-G. Beyer. Local performance of the (1+1)-ES in a noisy environment. IEEE Transactions on Evolutionary Computation, 6(1):30–41, 2002.

[6] I. Loshchilov. A computationally efficient limited memory CMA-ES for large scale optimization. In Proc. GECCO 2014, pp. 397–404. ACM, 2014. arXiv:1404.5520

[7] D. Wierstra, T. Schaul, T. Glasmachers, Y. Sun, J. Peters, and J. Schmidhuber. Natural evolution strategies. Journal of Machine Learning Research, 15:949–980, 2014.

[8] O. Li, J. Harrison, J. Sohl-Dickstein, V. Smith, and L. Metz. Variance-reduced gradient estimation via noise-reuse in online evolution strategies. In Advances in Neural Information Processing Systems 36 (NeurIPS), 2023.

[9] P. Vicol. Low-variance gradient estimation in unrolled computation graphs with ES-Single. In Proc. ICML 2023, PMLR 202, 2023.

[10] P. Vicol, L. Metz, and J. Sohl-Dickstein. Unbiased gradient estimation in unrolled computation graphs with persistent evolution strategies. In Proc. ICML 2021, PMLR 139:10553–10563, 2021.

[11] Y. Tang, K. Choromanski, and A. Kucukelbir. Variance reduction for evolution strategies via structured control variates. In Proc. AISTATS 2020, PMLR 108:845–855, 2020.

[12] T. Glasmachers, T. Schaul, Y. Sun, D. Wierstra, and J. Schmidhuber. Exponential natural evolution strategies. In Proc. GECCO 2010, pp. 393–400. ACM, 2010.

[13] Y. Sun, D. Wierstra, T. Schaul, and J. Schmidhuber. Efficient natural evolution strategies. In Proc. GECCO 2009, pp. 539–546. ACM, 2009.

[14] X. Zhang, J. Clune, and K. O. Stanley. On the relationship between the OpenAI evolution strategy and stochastic gradient descent. arXiv:1712.06564, 2017.

[15] T. Salimans, J. Ho, X. Chen, S. Sidor, and I. Sutskever. Evolution strategies as a scalable alternative to reinforcement learning. arXiv:1703.03864, 2017.

[16] J. Lehman, J. Chen, J. Clune, and K. O. Stanley. ES is more than just a traditional finite-difference approximator. In Proc. GECCO 2018, pp. 450–457. ACM, 2018. arXiv:1712.06568

[17] S. Malladi, T. Gao, E. Nichani, A. Damian, J. D. Lee, D. Chen, and S. Arora. Fine-tuning language models with just forward passes. In Advances in Neural Information Processing Systems 36 (NeurIPS), 2023.

[18] W. Hoy, B. Wang, and X. Pan. Matching accuracy, different geometry: Evolution strategies vs GRPO in LLM post-training. arXiv:2604.01499, 2026.

[19] Q. Liang, J. Song, Y. Liu, J. Gore, I. Fiete, R. Miikkulainen, and X. Qiu. The blessing of dimensionality in LLM fine-tuning: A variance-curvature perspective. arXiv:2602.00170, 2026.

[20] Y. Gan and P. Isola. Neural thickets: Diverse task experts are dense around pretrained weights. In Proc. ICML 2026. arXiv:2603.12228

[21] B. Sarkar, M. Fellows, J. A. Duque, et al. Evolution strategies at the hyperscale. PMLR 306, 2026. arXiv:2511.16652

[22] X. Qiu, Y. Gan, C. F. Hayes, et al. Evolution strategies at scale: LLM fine-tuning beyond reinforcement learning. PMLR 306, 2026. arXiv:2509.24372

We thank the authors for their detailed response. I will consider raising my score to 2 if I see 1) a detailed related works section and 2) the other issues addressed in the draft.

This is despite my main concern remaining: the framing of the paper is against a strawman: the authors do not convincingly establish that prior works show that best-of-N sampling is the mechanism by which ES optimizes LLMs.

ES methods used in this work form an approximate gradient with no requirement on the individual population members having a higher fitness than the parent. This is well-known and a standard finding of ES, the authors incorrectly claim that prior work does not show this: this is highlighted in the cited Beyer et al. 2002 ES survey that the rebuttal mentioned. Figure 7 explicitly shows all perturbations being worse, and yet the recombination is better than the parent. I maintain that ES finding good models from bad perturbations is not a paradox.

Where I can see a "paradox" framing is the surprising fact demonstrated by prior work that ES finds good directions in LLMs with small population sizes (~30), but this is not how the paper is framed, thus my concern. The theoretical model the authors develop in this paper is novel from what I can tell, but is not enough of a contribution on top of Liang et al. 2026.

Reviewer_bmDSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates an “ES paradox”: parent models improve during evolution-strategy fine-tuning even when all sampled offspring perform worse than the parent. It proposes an explanation combining curvature-induced degradation, useful relative reward information across perturbation directions, and aggregation that preserves signal while reducing orthogonal noise. A local quadratic model with observation noise characterizes this tradeoff and derives a candidate optimal perturbation scale. Empirical validation includes LLM fine-tuning on Sudoku and Countdown, CNN training on MNIST, cross-scale reward correlations, a scaled best-of-N comparison, and perturbation-scale and population-size experiments.

Critical strengths
  1. A compelling empirical phenomenon with relevance to ES-based fine-tuning. The paper documents that parent models can continue improving even when all sampled offspring perform worse, with examples spanning LLM fine-tuning and CNN training (Fig. 1). This distinguishes candidate quality from the usefulness of candidate-derived information and provides a concrete phenomenon that an adequate explanation of ES behavior should account for.

  2. Complementary experiments that probe the proposed explanation. Cross-scale reward correlations, the scaled best-of-N comparison, and population-size experiments examine different links between exploration and effective updates (Figs. 4, 8, and 10). Although they do not fully establish the mechanism, these probes make the contribution more substantive than an isolated observation and provide useful evidence for understanding ES in high-dimensional parameter spaces.

Critical weaknesses
  1. Limited novelty. The proposed explanation builds on established views of ES as a gradient estimator and closely resembles existing variance–curvature analyses, particularly Liang et al. (2026, The Blessing of Dimensionality in LLM Fine-tuning: A Variance–Curvature Perspective)[https://arxiv.org/pdf/2602.00170]. Beyond documenting the all-worse-offspring phenomenon, the paper should clarify what new mechanistic insight its analysis provides and which conclusions cannot be obtained from prior frameworks.

  2. Limited quantitative validation of the small-population explanation. The paper does not sufficiently establish why only a few dozen perturbations can effectively optimize billion-parameter models. The experiments show trends consistent with the proposed explanation, but do not directly estimate the key signal, noise, and curvature quantities or use independently estimated quantities to predict the optimal perturbation scale or required population size.

  3. Inconsistent treatment of sample and population normalization. The implemented algorithm standardizes rewards using the sample mean and standard deviation, whereas the derivations replace these with population statistics. Some subsequent steps nevertheless invoke exact sample-normalization constraints.

Questions for authors
  1. Does the analysis of the all-worse-offspring phenomenon yield predictions that cannot be derived from existing ES gradient-estimation and variance–curvature frameworks?

  2. Do you have quantitative measurements of the signal, noise, or curvature terms that test the proposed explanation?

4 comments
AuthorsOpenReview ↗

W1 & Q1 response

We would like to thank the reviewer for the kind words on identification of a novel empirical phenomenon as well as experimental methodology. We hope to address each of the identified weaknesses and reviewer questions below.

W1 & Q1. Limited Novelty: The authors believe that this paper’s novelty lies in the discovery of a phenomenon not documented in all of Evolution Strategies literature, from classical (1+1)ES [1], to NES [7], to modern ES and zeroth order work in LLMs [16]-[22]. In our search through literature, we have not seen any research that identifies optimization happening without improving offspring. We additionally provide theoretical understanding of this phenomenon. However, we take note of the similarity of some conclusions with the mentioned ES work Liang et al. [19] Blessing of Dimensionality [19]. We would like to note Liang et al’s work focuses on intrinsic dimension and improving directions via “Best of N” sampling to imply low population sizes can be used. It provides no analysis on what that low population size should be or how it may be related to curvature. Additionally its “best of N” analysis implicitly assumes improving perturbations are found at standard sigmas, and that these are a measure of capability for ES to improve the model. Our analysis shows this is not the case, and improvement can occur well after improving perturbations stop being found. While variance curvature analyses exist in the high parameter dimension/ low intrinsic dimensional setting of llm fine tuning via [17-19],[21], these analyses do not explore why given perturbation sizes work or what might be optimal, nor do they examine the performance of a method at various population sizes. Lastly we intend to add more specification to conclusions taken from our own work, including:

  • futility of elite selection in ES for LLM fine tuning
  • understanding of the selection of population size, how population size is related to curvature in high-dimension/low-intrensic dimension regime, and how optimal population size may be selected.
  • estimation of optimal perturbation size, and how this may occur at scales where all population members underperform the parent.

We will add these discussions to the revision, which we will upload before the rebuttal deadline.

AuthorsOpenReview ↗

W2 and Q2 response

W2 & Q2. Limited Quantitative validation of the small-population explanation: We agree with the reviewer that these are important validations to supplant our original analysis. However, estimation of these quantities, specifically observation noise and the trace of the squared hessian, tr(Q2), are highly uncertain in practice due to higher order reward landscape irregularity and the presence of higher order terms. We believe that the purpose of the provided analysis is to inform general trends and tradeoffs, rather than make specific estimates. That said, with some accuracy we can estimate optimal perturbation size as well as the 1-step performance of a population size as a percent of an infinite population size. With perturbation size, optimal perturbation coincides with the minimum of the variance of the reward at sigma over sigma squared, or Var(σ)/σ2. Using the same data from our correlation tests to sweep sigma, we got the following sigmas to meet the optimal value at iterations 0, 100, 200, 300, 400, and 500.

Iteration0100200300400500
Best σ\sigma (×10−3\times 10^{-3})2.000.750.751.001.000.75

After 100 iterations the optimal perturbation size lines up similarly to results tested in [22]. The early recommendation of large perturbation size lines up with the early regime where curvature is low and improving offspring are easy to find.

To measure the effectiveness of population size, we can use the equation 14 along with measured reward improvement, and empirical estimates of curvature to estimate the expected reward achievement for a given population size.

Iteration0100200300400500
N=30 (%)979669525155
N=128 (%)999993898889

Here we see saturation in expected 1-step improvement plateaus but at a higher population size than we see from our experiments. We believe this may be due to the estimation of curvature. Higher order terms in the loss landscape may artificially inflate the curvature, and thus push down the estimated 1-step improvement from smaller population sizes.

We intend to add these to the results section of the paper before the rebuttal deadline.

W3. Inconsistent treatment of sample and population normalization. We thank the reviewer for noting this treatment and we believe it should be better justified in the text. This treatment was done to make analysis in the paper easier to follow. Derivation with correct sample statistics is now provided in the appendix. It adds some complexity to the derivations but results in a simple correction factor ~= (1-3/4N). We have rewritten the new expected Signal, Noise, and Reward Change below for convenience, the appendix has also been updated to include them.

Table: Per-update signal, noise, and expected reward change of z-scored ES, using the sample statistics the algorithm computes

LandscapeSignal E[g]Noise E∥Δθ⊥∥2\sqrt{\mathbb{E}\lVert \Delta\theta_\perp \rVert^2}Expected reward change E[R]
Linear, exact rewardsN(d-1)/NακN∥g∥\alpha\kappa_N\lVert g\rVert
Linear, observation noiseN(d-1)/NαρκN∥g∥\alpha\rho\kappa_N\lVert g\rVert
Quadratic, observation noiseN(d-1)/NαρκN∥g∥−α22Ntr[Q]\alpha\rho\kappa_N\lVert g\rVert - \frac{\alpha^2}{2N}\mathrm{tr}[Q]

Definitions.

  • d: number of parameters; N: population size; : perturbation scale; : learning rate.
  • =Ni=1NZii: the ES update, with perturbations iN(0,Id) and Zi the rewards R(+i) z-scored with their sample mean and standard deviation.
  • g=R(): reward gradient; g^=g/∥g∥\hat g = g/\lVert g\rVert: its direction.
  • =(I-gg): the part of the update orthogonal to the gradient. The signal is the expected step along g, and the noise is the root-mean-square size of the orthogonal part.
  • Q: negative Hessian of the reward (curvature); : standard deviation of the observation noise on each reward evaluation.
  • σR2=σ2∥g∥2+σ42tr[Q2]+σξ2\sigma_R^2 = \sigma^2\lVert g\rVert^2 + \tfrac{\sigma^4}{2}\mathrm{tr}[Q^2] + \sigma_\xi^2: population variance of the perturbed rewards. Its three terms come from the gradient, the curvature, and the observation noise.
  • ρ=σ∥g∥/σR\rho = \sigma\lVert g\rVert/\sigma_R: correlation between a member's reward and its projection onto the gradient. It equals σ∥g∥/σ2∥g∥2+σξ2\sigma\lVert g\rVert/\sqrt{\sigma^2\lVert g\rVert^2+\sigma_\xi^2} in the linear noisy case, and 1 with exact linear rewards.
  • κN=2/N Γ(N2)/Γ(N−12)≈1−34N\kappa_N = \sqrt{2/N}\,\Gamma(\tfrac N2)/\Gamma(\tfrac{N-1}2) \approx 1 - \tfrac{3}{4N}: finite-population factor from using sample rather than population statistics (300.975).
AuthorsOpenReview ↗

References

References

[1] H.-G. Beyer and H.-P. Schwefel. Evolution strategies – A comprehensive introduction. Natural Computing, 1(1):3–52, 2002.

[2] H.-G. Beyer. Toward a theory of evolution strategies: On the benefits of sex — the (μ/μ, λ) theory. Evolutionary Computation, 3(1):81–111, 1995.

[3] Y. Akimoto, A. Auger, T. Glasmachers, and D. Morinaga. Global linear convergence of evolution strategies on more than smooth strongly convex functions. SIAM Journal on Optimization, 32(2):1402–1429, 2022. doi:10.1137/20M1373815

[4] O. Ait Elhara, A. Auger, and N. Hansen. A median success rule for non-elitist evolution strategies: Study of feasibility. In Proc. GECCO 2013, pp. 415–422. ACM, 2013. doi:10.1145/2463372.2463429

[5] D. V. Arnold and H.-G. Beyer. Local performance of the (1+1)-ES in a noisy environment. IEEE Transactions on Evolutionary Computation, 6(1):30–41, 2002.

[6] I. Loshchilov. A computationally efficient limited memory CMA-ES for large scale optimization. In Proc. GECCO 2014, pp. 397–404. ACM, 2014. arXiv:1404.5520

[7] D. Wierstra, T. Schaul, T. Glasmachers, Y. Sun, J. Peters, and J. Schmidhuber. Natural evolution strategies. Journal of Machine Learning Research, 15:949–980, 2014.

[8] O. Li, J. Harrison, J. Sohl-Dickstein, V. Smith, and L. Metz. Variance-reduced gradient estimation via noise-reuse in online evolution strategies. In Advances in Neural Information Processing Systems 36 (NeurIPS), 2023.

[9] P. Vicol. Low-variance gradient estimation in unrolled computation graphs with ES-Single. In Proc. ICML 2023, PMLR 202, 2023.

[10] P. Vicol, L. Metz, and J. Sohl-Dickstein. Unbiased gradient estimation in unrolled computation graphs with persistent evolution strategies. In Proc. ICML 2021, PMLR 139:10553–10563, 2021.

[11] Y. Tang, K. Choromanski, and A. Kucukelbir. Variance reduction for evolution strategies via structured control variates. In Proc. AISTATS 2020, PMLR 108:845–855, 2020.

[12] T. Glasmachers, T. Schaul, Y. Sun, D. Wierstra, and J. Schmidhuber. Exponential natural evolution strategies. In Proc. GECCO 2010, pp. 393–400. ACM, 2010.

[13] Y. Sun, D. Wierstra, T. Schaul, and J. Schmidhuber. Efficient natural evolution strategies. In Proc. GECCO 2009, pp. 539–546. ACM, 2009.

[14] X. Zhang, J. Clune, and K. O. Stanley. On the relationship between the OpenAI evolution strategy and stochastic gradient descent. arXiv:1712.06564, 2017.

[15] T. Salimans, J. Ho, X. Chen, S. Sidor, and I. Sutskever. Evolution strategies as a scalable alternative to reinforcement learning. arXiv:1703.03864, 2017.

[16] J. Lehman, J. Chen, J. Clune, and K. O. Stanley. ES is more than just a traditional finite-difference approximator. In Proc. GECCO 2018, pp. 450–457. ACM, 2018. arXiv:1712.06568

[17] S. Malladi, T. Gao, E. Nichani, A. Damian, J. D. Lee, D. Chen, and S. Arora. Fine-tuning language models with just forward passes. In Advances in Neural Information Processing Systems 36 (NeurIPS), 2023.

[18] W. Hoy, B. Wang, and X. Pan. Matching accuracy, different geometry: Evolution strategies vs GRPO in LLM post-training. arXiv:2604.01499, 2026.

[19] Q. Liang, J. Song, Y. Liu, J. Gore, I. Fiete, R. Miikkulainen, and X. Qiu. The blessing of dimensionality in LLM fine-tuning: A variance-curvature perspective. arXiv:2602.00170, 2026.

[20] Y. Gan and P. Isola. Neural thickets: Diverse task experts are dense around pretrained weights. In Proc. ICML 2026. arXiv:2603.12228

[21] B. Sarkar, M. Fellows, J. A. Duque, et al. Evolution strategies at the hyperscale. PMLR 306, 2026. arXiv:2511.16652

[22] X. Qiu, Y. Gan, C. F. Hayes, et al. Evolution strategies at scale: LLM fine-tuning beyond reinforcement learning. PMLR 306, 2026. arXiv:2509.24372

Thank you for the clarification. My remaining reservation is novelty. I still do not think the paper meets the ICLR standard. I will keep my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.