acceptodds
Under review as a conference paper at ICLR 2027

Understanding Evolution Strategies for LLM Reasoning: Broader Reasoning Coverage than GRPO

Abstract

Evolution Strategies (ES) have recently emerged as a memory-efficient post-training paradigm for LLM reasoning. However, the optimization behavior of ES remains understudied, making it hard to define its advantage scope compared to mainstream post-training paradigms (e.g., Group Relative Policy Optimization (GRPO)). This paper systematically investigates the behavior and design of ES post-training. we : broader reasoning coverage that better exploits the reasoning capabilities of pretrained LLMs. Empirically, ES generally improves Pass@1 over the base model and achieves higher Pass@ than GRPO, which can exhibit entropy collapse. Theoretically, we derive sufficient conditions under which an ES update improves the center policy's Pass@. We further propose a sequential ES–GRPO training method in both orders, which can combine GRPO's Pass@1 strength with ES's Pass@ gains. we find that despite substantial parameter drift, removing most small coordinate updates of ES largely preserves target-task Pass@1 performance. Held-out evaluations further show that this large drift need not imply broad forgetting. we study how ES design choices affect its effectiveness, finding that comparable training rewards can be maintained with smaller ES populations in larger LLMs. These findings position ES as a distinct post-training paradigm for LLM reasoning rather than a less effective, memory-efficient alternative to GRPO. Our code is available at https://anonymous.4open.science/r/understanding-es-3D73.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Post-Training, Evolution Strategies, Reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_sfn6OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

This work examines a broad look at evolution strategies for fine tuning models and describes broad benefits of this method over GRPO. The author(s) provide analytical reasoning for the expected improvement on pass@k by ES over GRPO. The work then demonstrates improved pass@k performance by ES in a variety of fine tuning tasks. Next the work demonstrates the large parameter drift that occurs with ES, and how this parameter drift is not an indicator of performance or off task degradation, showing that held out task performance does not significantly decrease during training. Next design choices are evaluated, including reward normalization, perturbation scale, population size, and mirror sampling.

Overall the work runs a number of very useful ablations for evolution strategies in LLM fine tuning, however it ends up spreading too thin across all the tests. Either more thorough testing should be done or some of the testing should be dropped to allow for greater focus on a subset of tests.

Critical strengths

The paper runs some very useful experiments within the broader field of evolution strategies for post training in LLM's. The demonstration of improved pass@k performance across tasks and models using evolution strategies is novel along concurrent work (Hayes et al), and is valuable to have for the community. The tests of sequential training (ES, then GRPO and vice versa) are a good tests to determine if the benefits of both methods can be aggregated. Held out performance tests are a good approach to check for catastrophic forgetting. The test of reward function choice (none vs z-score) is a good ablation to have as well.

Critical weaknesses

The paper may be trying to test too many things at once. There may be a version of this that works well, but in its current state this leaves some of the most interesting tests left under-examined.

  1. some of the most interesting ablations seem under-examined. The z-score reward comparison just references one training run on the first figure, with no mention of what test it was, what model was used, what the settings where, etc. This is a very valuable test to run but it should really be run across multiple benchmarks/models, and the test conditions are strictly needed.

2)section 5.1 mentions describes perturbation scale as if it is something tested in the paper, but makes no reference to a table, figure, or experiment. I found mention of an experiment in the appendix, but if this is referenced in the body of the paper it really needs to be tied to an experiment in the paper.

  1. one vs two point estimation - again a very interesting ablation, it appears there is only one training run of this?
Questions for authors
  1. The conclusion in section 3.1, with regards to ES update improving Pass@k, seems to rely on lemma 1, which itself relies on an assumption from appendix B.2 - "For the local relation, let p_i = p_bar + \eps_i." - this assumes that the perturbation pass probability is centered at the centerpoint model probability. How is this justified? From my own work with ES this may be true at small scales, but trivially the perturbation size could be made large enough that every population members probability falls to zero and this becomes trivially untrue.

  2. the Z-score normalization comparison - what task/model was this run on? Section 5.1 references a figure on the first page, but that figure seems to have no extra information, and seems to be only one training run.

  3. On the perturbation scaling - it is mentioned in section 5.1 but no tables or figures or experiments are referenced. What testing of perturbation scale was done here?

6 comments
AuthorsOpenReview ↗

Response to Reviewer sfn6 (Part 1/4)

Thank you for your time and effort in reviewing our work. We are glad to know that you find our Pass@K findings valuable to the community and our sequential-training, held-out evaluation, and reward-normalization experiments useful.

We address your concerns as follows, and all revisions are marked in blue in the revised PDF.

Weakness 1 & Question 2. Z-Score Normalization across Models and Tasks: "the Z-score normalization comparison - what task/model was this run on?" ... "it should really be run across multiple benchmarks/models, and the test conditions are strictly needed."

Thank you for your suggestion. The original comparison uses Qwen2.5-1.5B-Instruct on GSM8K. We have added comparisons across four models in the Easy and Hard settings (Appendix L, Figures 14–15).

The Easy-setting models are trained and evaluated on GSM8K. DeepSeek-R1-Distill-Qwen-1.5B is trained on DeepScaleR and evaluated on MATH500, AIME24, AIME25, and AMC23. The population size is N=32N=32 for Qwen2.5-0.5B-Instruct and N=16N=16 for the others. The tables report training rewards and task Pass@1 (%) across training steps, with paired values ordered as without normalization / with z-score normalization.

Qwen2.5-0.5B-InstructSmoothed rewardPass@1
Step 390.4502 / 0.485445.9 / 46.8
Step 780.4530 / 0.500845.5 / 47.5
Step 1170.4560 / 0.507346.0 / 46.9
Step 1560.4595 / 0.511745.7 / 47.4
Step 1950.4609 / 0.511145.5 / 47.1
Step 2340.4655 / 0.514645.7 / 47.3
Qwen2.5-1.5B-InstructSmoothed rewardPass@1
Step 390.7611 / 0.805069.9 / 72.3
Step 780.7635 / 0.819570.1 / 72.4
Step 1170.7709 / 0.829569.6 / 72.5
Step 1560.7754 / 0.833769.5 / 72.6
Step 1950.7785 / 0.832670.0 / 72.7
Step 2340.7872 / 0.838170.2 / 72.4
Llama-3.2-3B-InstructSmoothed rewardPass@1
Step 390.5978 / 0.724977.9 / 78.4
Step 780.6369 / 0.800478.3 / 78.5
Step 1170.6712 / 0.839978.2 / 80.0
Step 1560.7027 / 0.869578.1 / 81.3
Step 1950.7186 / 0.887077.7 / 81.2
Step 2340.7423 / 0.900377.5 / 81.2
DeepSeek-R1-Distill-Qwen-1.5BSmoothed rewardMATH500AIME24AIME25AMC23
Step 300.4513 / 0.456380.0 / 81.224.0 / 24.122.4 / 22.963.1 / 64.0
Step 600.4435 / 0.463279.8 / 81.725.0 / 24.822.5 / 23.063.7 / 65.2
Step 900.4355 / 0.462879.8 / 81.823.1 / 22.822.0 / 23.963.5 / 67.7
Step 1200.4383 / 0.471680.2 / 82.223.9 / 25.322.4 / 23.064.6 / 68.1
Step 1500.4402 / 0.479580.3 / 82.323.8 / 24.622.0 / 22.964.8 / 69.1
Step 1800.4392 / 0.483680.7 / 82.624.2 / 27.223.2 / 23.063.4 / 69.3
Step 2100.4390 / 0.487780.4 / 82.423.9 / 25.422.5 / 23.964.3 / 67.9
Step 2400.4368 / 0.491480.6 / 82.523.3 / 24.722.6 / 22.663.8 / 69.8
Step 2700.4420 / 0.500180.5 / 82.423.8 / 24.321.7 / 22.662.5 / 70.6
  1. Easy setting: Z-score normalization consistently improves training rewards and GSM8K Pass@1 throughout training across all three models.

  2. Hard setting: Z-score normalization generally improves training rewards and Pass@1 during training, with higher final Pass@1 on all four benchmarks.

These results support the effectiveness of z-score normalization across models and tasks.

AuthorsOpenReview ↗

Response to Reviewer sfn6 (Part 2/4)

Weakness 2 & Question 3. Experimental Evidence for Perturbation Scale: "On the perturbation scaling - it is mentioned in section 5.1 but no tables or figures or experiments are referenced. What testing of perturbation scale was done here?"

Thank you for pointing this out. We have added a perturbation-scale sweep and an explicit reference to it in Section 5.1 (Appendix M, Figure 16).

The sweep varies σ\sigma on GSM8K for the three models below. The tables report smoothed mean training rewards.

Qwen2.5-0.5B-Instruct (N=32)Step 1Step 10Step 50Step 100Step 234
σ=0.0010.60090.59290.59040.59790.6015
σ=0.0020.26940.28220.33170.36130.3911
σ=0.0030.01750.01680.03150.06010.1170
σ=0.0040.00260.00210.00280.00490.0223
Qwen2.5-1.5B-Instruct (N=16)Step 1Step 10Step 50Step 100Step 234
σ=0.0010.83050.81280.83440.84300.8517
σ=0.0020.76950.74350.78320.80010.8158
σ=0.0030.61510.60810.66190.70030.7364
σ=0.0060.00460.00770.01200.01690.0370
Qwen2.5-3B-Instruct (N=8)Step 1Step 10Step 50Step 100Step 234
σ=0.0020.89390.90140.89360.89120.8927
σ=0.0030.72360.76100.77820.79610.8121
σ=0.0040.42320.43750.46130.49650.5521
σ=0.0060.00350.00340.00290.00430.0075
  1. Excessively large perturbations: At the largest tested scales, all three models start with very low rewards and show limited recovery during training.

  2. Excessively small perturbations: The smallest tested scales preserve higher rewards but show little subsequent improvement. From a parameter-space search perspective, excessively small perturbations restrict exploration to the vicinity of the current parameters, which may limit access to other beneficial regions.

  3. Intermediate perturbations: Intermediate scales produce larger reward gains during training than the smallest tested scales, while avoiding the severe reward degradation observed at the largest scales.

These results support choosing a perturbation scale that preserves task performance while allowing further training progress.

AuthorsOpenReview ↗

Response to Reviewer sfn6 (Part 3/4)

Weakness 3. One-Point versus Two-Point Comparison across Models and Tasks: one vs two point estimation - again a very interesting ablation, it appears there is only one training run of this?

Thank you for your comment. We have expanded the one-point versus two-point comparison to four models across Easy and Hard settings (Appendix O, Figures 18–19).

The models and tasks follow the normalization comparison above. Each step evaluates the same number of perturbed models: 32 for Qwen2.5-0.5B-Instruct and 16 for the others. Paired values denote one-point / two-point results, with task performance reported as Pass@1 (%).

Qwen2.5-0.5B-InstructSmoothed rewardPass@1
Step 390.4854 / 0.487446.8 / 46.6
Step 780.5008 / 0.501447.5 / 46.9
Step 1170.5073 / 0.507246.9 / 46.5
Step 1560.5117 / 0.509547.4 / 47.1
Step 1950.5111 / 0.508647.1 / 46.5
Step 2340.5146 / 0.511147.3 / 46.9
Qwen2.5-1.5B-InstructSmoothed rewardPass@1
Step 390.8050 / 0.808672.5 / 72.3
Step 780.8195 / 0.820172.6 / 72.6
Step 1170.8295 / 0.829472.6 / 72.9
Step 1560.8337 / 0.832872.5 / 72.3
Step 1950.8326 / 0.831272.5 / 72.5
Step 2340.8381 / 0.835772.5 / 72.3
Llama-3.2-3B-InstructSmoothed rewardPass@1
Step 390.7249 / 0.724778.4 / 78.1
Step 780.8004 / 0.802778.5 / 77.9
Step 1170.8399 / 0.840080.0 / 79.3
Step 1560.8695 / 0.867281.3 / 81.1
Step 1950.8870 / 0.884481.2 / 81.3
Step 2340.9003 / 0.899981.2 / 80.9
DeepSeek-R1-Distill-Qwen-1.5BSmoothed rewardMATH500AIME24AIME25AMC23
Step 300.4563 / 0.457481.2 / 80.824.1 / 24.722.9 / 23.164.0 / 65.1
Step 600.4632 / 0.461481.7 / 81.524.8 / 23.923.0 / 21.665.2 / 65.8
Step 900.4628 / 0.459781.8 / 81.422.8 / 25.723.9 / 23.067.7 / 66.6
Step 1200.4716 / 0.466882.2 / 81.625.3 / 25.023.0 / 22.668.1 / 65.9
Step 1500.4795 / 0.472682.3 / 81.924.6 / 23.222.9 / 21.769.1 / 68.1
Step 1800.4836 / 0.476582.6 / 82.027.2 / 23.523.0 / 22.369.3 / 68.0
Step 2100.4877 / 0.480482.4 / 82.025.4 / 25.323.9 / 23.067.9 / 68.6
Step 2400.4914 / 0.483282.5 / 82.324.7 / 25.822.6 / 23.869.8 / 68.3
Step 2700.5001 / 0.491882.8 / 82.726.1 / 25.222.6 / 21.970.6 / 68.4
  1. Training rewards: The two estimators have similar training rewards in the Easy setting, while one-point ES generally achieves higher rewards in the Hard setting.

  2. Test performance: One-point ES generally achieves higher Pass@1 during training and outperforms two-point ES at the final step across all evaluated models and benchmarks.

These results show that one-point ES remains competitive across the tested models and tasks under the same evaluation budget.

Reviewer_SRcjOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work investigates Evolution Strategies (ES) as a memory‑efficient post‑training approach for LLM reasoning and contrasts it with GRPO. It derives sufficient conditions for ES updates to improve a center policy’s Pass@K and empirically demonstrates ES yields broader reasoning coverage and higher Pass@K at the cost of slightly lower Pass@1 compared to GRPO. The paper also analyzes ES’s large‑magnitude parameter drift, shows that discarding most small‑magnitude parameter changes preserves target‑task performance, explores practical ES design choices, and proposes sequential ES‑GRPO training to combine their complementary strengths.

Critical strengths

The paper introduces a practical hybrid training method, which merges ES’s Pass@K reasoning‑coverage capability with GRPO’s Pass@1 strength.

Critical weaknesses
  1. It is unclear whether the work introduces any substantial, effective modifications to the vanilla ES algorithm.

  2. The observation that pruning most small-magnitude ES updates retains target-task Pass@1 is heuristic without theoretical support, and its transferability across different base models and validation datasets is unconfirmed.

Questions for authors
  1. Could you compare your method against entropy-regularized GRPO variants, which are designed specifically to mitigate entropy collapse and lack of response diversity and bring notable Pass@K gains?

  2. Most experiments are conducted on relatively small base models with limited reasoning capacity. Could you provide results on stronger 7B/14B reasoning-focused backbones such as Qwen2.5-7B-Math and DeepSeek-R1-Distill-Qwen-7B?

  3. GSM8K is common for ablations, but would ablations on harder benchmarks like AIME24/25 better demonstrate the effectiveness of your approach?

  4. Several results in Table 2 (e.g., on MBPP) show your method performs similarly to or worse than the baseline. What explains this behavior, and does it suggest limited generalization beyond mathematical reasoning tasks?

  5. The training curves in Figure 2 appear to have sparse data points and terminate at only 234 steps. Does this imply that the model has not yet fully converged?

7 comments
AuthorsOpenReview ↗

Response to Reviewer SRcj (Part 1/4)

Thank you for your time and effort in reviewing our work. We are glad to know that you find our sequential ES–GRPO training method practical for combining ES’s reasoning coverage with GRPO’s single-sample accuracy.

We address your concerns as follows, and all revisions are marked in blue in the revised PDF.

Weakness 1. Contribution beyond Vanilla ES: It is unclear whether the work introduces any substantial, effective modifications to the vanilla ES algorithm.

Thank you for your comment. We do not introduce substantial modifications to the vanilla ES estimator. Our main contribution is to systematically understand ES for LLM reasoning post-training. We develop this understanding in three aspects:

  1. Reasoning coverage: We identify ES’s broader coverage relative to GRPO and derive sufficient conditions for improving the center policy’s Pass@K.
  2. Parameter drift and capability retention: We examine how parameter changes relate to retained capabilities and provide practical guidance through design ablations.
  3. Sequential ES–GRPO training: Our methodological contribution combines the two methods in both orders to bring together GRPO’s Pass@1 gains and ES’s large-KK coverage.

Weakness 2. Theoretical Support and Transferability of Update Pruning: The observation that pruning most small-magnitude ES updates retains target-task Pass@11 is heuristic without theoretical support, and its transferability across different base models and validation datasets is unconfirmed.

Thank you for your comment. We would like to make a clarification: pruning most small-magnitude ES updates is not a heuristic without theoretical support. Our magnitude-thresholding criterion is supported by Gaussian significance testing, and our experiments provide transferability evidence across base models and benchmarks.

  1. Theoretical support. Our magnitude-thresholding criterion is theoretically motivated by Gaussian significance testing. When a coordinate's perturbations are independent of the reward weights, its single-step ES update is zero-mean Gaussian conditional on those weights. When the random update components are uncorrelated across training steps and their accumulated variances are similar across parameter coordinates, larger absolute updates are less readily explained by random fluctuations, supporting the retention of large-magnitude updates.
  2. Transferability evidence across models and datasets. Our manuscript already reports magnitude-thresholded ES results across four models evaluated on GSM8K or MATH-500. Removing approximately 78–79% of nonzero updates keeps Pass@1 within 0.5 percentage points of the ES-trained model in all four cases (Section 4.2 and Appendix J.2, Tables 31–32). With the same number of retained coordinates, magnitude thresholding also achieves higher Pass@1 than random truncation across all three tested models (Appendix J.3, Table 33), supporting update magnitude as a criterion for selecting which updates to retain.
AuthorsOpenReview ↗

Response to Reviewer SRcj (Part 2/4)

Question 1. Comparison with Entropy-Regularized GRPO: Could you compare your method against entropy-regularized GRPO variants, which are designed specifically to mitigate entropy collapse and lack of response diversity ...?

Thank you for your suggestion. We have added comparisons with entropy-regularized GRPO across all three Easy-setting models (Table 2 and Appendix G, Table 21). The entropy bonus weight is 0.001, consistent with prior GRPO-based studies [1, 2]. Values below are average Pass@1 / Pass@16 / Pass@32 (%) across GSM8K, HotpotQA, Countdown, and MBPP.

MethodQwen2.5-1.5B-InstructLlama-3.2-3B-InstructQwen2.5-7B-Instruct
Base41.0 / 68.0 / 72.743.5 / 67.2 / 71.561.8 / 76.7 / 78.6
GRPO42.7 / 67.7 / 72.246.0 / 66.4 / 70.262.5 / 75.8 / 77.8
Entropy-Regularized GRPO42.8 / 67.4 / 72.346.1 / 67.2 / 71.261.4 / 75.3 / 77.3
ES41.4 / 68.3 / 73.145.2 / 68.6 / 72.861.1 / 76.8 / 78.9
ES→GRPO42.3 / 67.4 / 71.946.0 / 67.5 / 71.462.3 / 76.4 / 78.1
GRPO→ES41.8 / 67.3 / 71.945.4 / 66.7 / 71.062.1 / 76.3 / 78.3
  1. Entropy regularization partially mitigates GRPO’s coverage degradation. It improves average Pass@16/32 in some settings, but coverage degradation relative to the base model persists across all three backbones.

  2. Sequential training shows advantages over entropy-regularized GRPO. Both orders improve all three average metrics on Qwen2.5-7B-Instruct, while ES→GRPO also improves multi-sample coverage on Llama-3.2-3B-Instruct. On Qwen2.5-1.5B-Instruct, ES→GRPO and entropy-regularized GRPO each lead in Pass@16/32 on two of four tasks (Table 2 and Appendix G, Table 21).

  3. ES’s coverage advantage supports the motivation for sequential training. ES achieves higher average Pass@16/32 than entropy-regularized GRPO across all three models, motivating its combination with GRPO’s strength in Pass@1.


Question 2. Evaluation on Stronger Reasoning Models: ... Could you provide results on stronger 7B/14B reasoning-focused backbones such as Qwen2.5-7B-Math and DeepSeek-R1-Distill-Qwen-7B?

Thank you for your suggestion. We have added experiments comparing ES and GRPO on DeepSeek-R1-Distill-Qwen-7B in the Hard setting (Appendix Q, Table 45). With N=8N=8 for ES and G=8G=8 for GRPO, ES uses approximately one quarter of GRPO’s estimated training FLOPs (Appendix A.3). Values below are Pass@1 / Pass@16 / Pass@32, all in percent.

DeepSeek-R1-Distill-Qwen-7BAIME24AIME25AMC23MATH-500Average
Base42.2 / 74.5 / 76.733.1 / 59.6 / 66.781.2 / 95.9 / 97.590.2 / 98.3 / 98.661.7 / 82.1 / 84.9
GRPO49.9 / 77.2 / 80.036.0 / 61.6 / 63.388.2 / 95.0 / 95.093.5 / 99.2 / 99.666.9 / 83.2 / 84.5
ES42.8 / 78.5 / 80.031.6 / 64.3 / 66.783.2 / 97.2 / 100.091.4 / 98.5 / 99.262.2 / 84.6 / 86.5

These results show that ES’s average multi-sample coverage advantage extends to a stronger 7B reasoning model.

AuthorsOpenReview ↗

Response to Reviewer SRcj (Part 3/4)

Question 3. Ablations on Harder Benchmarks: GSM8K is common for ablations, but would ablations on harder benchmarks like AIME24/25 better demonstrate the effectiveness of your approach?

Thank you for your suggestion. We have extended the reward-normalization and one-point versus two-point ablations to the Hard setting (Appendix L, Figure 15, and Appendix O, Figure 19). The tables report training rewards and task Pass@1 (%) across training steps.

Z-score normalization: Paired values denote without normalization / with z-score normalization.

DeepSeek-R1-Distill-Qwen-1.5BSmoothed rewardMATH500AIME24AIME25AMC23
Step 300.4513 / 0.456380.0 / 81.224.0 / 24.122.4 / 22.963.1 / 64.0
Step 600.4435 / 0.463279.8 / 81.725.0 / 24.822.5 / 23.063.7 / 65.2
Step 900.4355 / 0.462879.8 / 81.823.1 / 22.822.0 / 23.963.5 / 67.7
Step 1200.4383 / 0.471680.2 / 82.223.9 / 25.322.4 / 23.064.6 / 68.1
Step 1500.4402 / 0.479580.3 / 82.323.8 / 24.622.0 / 22.964.8 / 69.1
Step 1800.4392 / 0.483680.7 / 82.624.2 / 27.223.2 / 23.063.4 / 69.3
Step 2100.4390 / 0.487780.4 / 82.423.9 / 25.422.5 / 23.964.3 / 67.9
Step 2400.4368 / 0.491480.6 / 82.523.3 / 24.722.6 / 22.663.8 / 69.8
Step 2700.4420 / 0.500180.5 / 82.423.8 / 24.321.7 / 22.662.5 / 70.6

One-point versus two-point estimation: Paired values denote one-point / two-point.

DeepSeek-R1-Distill-Qwen-1.5BSmoothed rewardMATH500AIME24AIME25AMC23
Step 300.4563 / 0.457481.2 / 80.824.1 / 24.722.9 / 23.164.0 / 65.1
Step 600.4632 / 0.461481.7 / 81.524.8 / 23.923.0 / 21.665.2 / 65.8
Step 900.4628 / 0.459781.8 / 81.422.8 / 25.723.9 / 23.067.7 / 66.6
Step 1200.4716 / 0.466882.2 / 81.625.3 / 25.023.0 / 22.668.1 / 65.9
Step 1500.4795 / 0.472682.3 / 81.924.6 / 23.222.9 / 21.769.1 / 68.1
Step 1800.4836 / 0.476582.6 / 82.027.2 / 23.523.0 / 22.369.3 / 68.0
Step 2100.4877 / 0.480482.4 / 82.025.4 / 25.323.9 / 23.067.9 / 68.6
Step 2400.4914 / 0.483282.5 / 82.324.7 / 25.822.6 / 23.869.8 / 68.3
Step 2700.5001 / 0.491882.8 / 82.726.1 / 25.222.6 / 21.970.6 / 68.4
  1. Z-score normalization: Normalization generally improves training rewards and Pass@1 during training, with higher final Pass@1 on all four benchmarks.
  2. One-point versus two-point estimation: One-point ES generally achieves higher training rewards and Pass@1 during training and higher final Pass@1 on all four benchmarks.

These results support the effectiveness of z-score normalization and the competitiveness of one-point ES on harder reasoning tasks.

Reviewer_5UN7OpenReview ↗
Rating
3(+2)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates how evolution strategies (ES) differ from GRPO in LLM reasoning post-training, focusing on solution coverage, capability retention, and optimization choices. Across the evaluated configurations, ES achieves higher average pass@K than GRPO at larger sampling budgets, while GRPO generally produces stronger pass@1 gains. The authors provide a conditional theoretical account connecting population diversity to coverage and explore sequential combinations of ES and GRPO. They also show that substantial parameter drift under ES does not necessarily imply broad capability degradation, and that retaining a relatively small subset of larger parameter updates preserves much of the measured task performance. Additional experiments examine reward normalization, population size, and one-point versus two-point estimators.

Critical strengths

The paper provides a useful empirical comparison that goes beyond single-sample accuracy. Evaluating pass@K, majority-vote accuracy, and held-out performance across multiple models and training settings reveals meaningful differences between ES and GRPO that pass@1 alone would miss. The explicit compute-matching analysis further strengthens the comparison.

The parameter-update analysis provides an informative investigation of ES’s behavior. By removing small updates and reevaluating the resulting models, the authors provide direct evidence that much of the task performance can be retained using only a subset of the parameter changes. Together with the held-out evaluations, this supports a more nuanced interpretation of the relationship between parameter drift, functional change, and forgetting.

Critical weaknesses

See questions below

Questions for authors
  1. Could the authors provide comparisons against GRPO with an entropy bonus and MaxRL, or other methods explicitly designed to improve exploration or solution coverage?

  2. Relatedly, the paper reports higher pass@K and includes majority-vote evaluations. Could the authors evaluate whether this coverage advantage translates into better downstream performance when ES and GRPO trained models are used within the same feedback-guided test-time search procedure, under matched inference budgets? For example, iterative code refinement using execution feedback would test whether ES’s broader coverage benefits adaptive search beyond independent sampling and voting. Such evidence would strengthen the practical significance of the findings by demonstrating their usefulness within a downstream search pipeline.

  3. Were hyperparameter sweeps conducted for GRPO? Given the paper’s claim that ES achieves broader reasoning coverage than GRPO, it is important to establish that this advantage persists against a well-tuned baseline rather than reflecting a particular GRPO configuration.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 5UN7 (Part 1/3)

Thank you for your time and effort in reviewing our work. We are glad to know that you find our comparison beyond single-sample accuracy useful and our parameter-update analysis informative.

We address your concerns as follows, and all revisions are marked in blue in the revised PDF.

Question 1. Comparisons with Entropy-Regularized GRPO and MaxRL: Could the authors provide comparisons against GRPO with an entropy bonus and MaxRL, or other methods explicitly designed to improve exploration or solution coverage?

Thank you for your suggestion. We have added comparisons with entropy-regularized GRPO and MaxRL [1] across all three Easy-setting models. For entropy-regularized GRPO, we use an entropy bonus weight of 0.001, consistent with prior GRPO-based studies [2, 3]. Values below are average Pass@1 / Pass@16 / Pass@32 (%) across GSM8K, HotpotQA, Countdown, and MBPP (per-task results in Appendix G, Table 21).

MethodQwen2.5-1.5B-InstructLlama-3.2-3B-InstructQwen2.5-7B-Instruct
Base41.0 / 68.0 / 72.743.5 / 67.2 / 71.561.8 / 76.7 / 78.6
GRPO42.7 / 67.7 / 72.246.0 / 66.4 / 70.262.5 / 75.8 / 77.8
MaxRL42.5 / 67.4 / 72.144.1 / 65.8 / 70.062.5 / 76.1 / 78.0
Entropy-Regularized GRPO42.8 / 67.4 / 72.346.1 / 67.2 / 71.261.4 / 75.3 / 77.3
ES41.4 / 68.3 / 73.145.2 / 68.6 / 72.861.1 / 76.8 / 78.9

Entropy regularization improves average Pass@16/32 over GRPO on Llama-3.2-3B-Instruct, as does MaxRL on Qwen2.5-7B-Instruct. Nevertheless, ES outperforms both baselines on these metrics across all three models.

AuthorsOpenReview ↗

Response to Reviewer 5UN7 (Part 2/3)

Question 2. Feedback-Guided Search under Matched Budgets: ... Could the authors evaluate whether this coverage advantage translates into better downstream performance when ES and GRPO trained models are used within the same feedback-guided test-time search procedure, under matched inference budgets? For example, iterative code refinement using execution feedback would test whether ES's broader coverage benefits adaptive search beyond independent sampling and voting. ...

Thank you for your suggestion. We have added feedback-guided code refinement on MBPP (Appendix H). Both methods use 32 initial solutions per problem and three repair rounds, generating 32 revised solutions per round for unresolved problems using execution feedback. Below are cumulative success rates (%) from initial sampling (Round 1) through three repairs (Round 4).

Qwen2.5-1.5B-InstructRound 1Round 2Round 3Round 4
ES86.486.486.486.4
GRPO86.086.086.086.0
Llama-3.2-3B-InstructRound 1Round 2Round 3Round 4
ES84.184.484.886.0
GRPO79.882.583.383.3
Qwen2.5-7B-InstructRound 1Round 2Round 3Round 4
ES87.688.790.390.7
GRPO87.689.189.589.9
DeepSeek-R1-Distill-Qwen-1.5BRound 1Round 2Round 3Round 4
ES87.289.190.391.1
GRPO88.389.589.990.3

After three repair rounds, ES outperforms GRPO on all four models. ES maintains its initial advantage on Qwen2.5-1.5B-Instruct and continues improving after GRPO plateaus on Llama-3.2-3B-Instruct. On Qwen2.5-7B-Instruct, ES pulls ahead from the same starting performance. On DeepSeek-R1-Distill-Qwen-1.5B, it overtakes GRPO despite starting lower. These results show that ES’s coverage advantage remains useful within feedback-guided search.

AuthorsOpenReview ↗

Response to Reviewer 5UN7 (Part 3/3)

Question 3. GRPO Hyperparameter Tuning: Were hyperparameter sweeps conducted for GRPO? ... it is important to establish that this advantage persists against a well-tuned baseline rather than reflecting a particular GRPO configuration.

Thank you for your question. We have conducted a GRPO learning-rate sweep on GSM8K across all three Easy-setting models, covering {5×10−7,1×10−6,2×10−6}\{5\times10^{-7}, 1\times10^{-6}, 2\times10^{-6}\}, following the learning-rate grid used in prior GRPO work [4]. Values below are Pass@1 / Pass@16 / Pass@32, all in percent.

MethodQwen2.5-1.5B-InstructLlama-3.2-3B-InstructQwen2.5-7B-Instruct
GRPO (lr=5e-7)75.6 / 94.4 / 96.084.0 / 95.9 / 96.892.2 / 97.1 / 97.6
GRPO (lr=1e-6, original)77.3 / 94.3 / 96.084.7 / 95.6 / 96.492.3 / 97.0 / 97.3
GRPO (lr=2e-6)79.5 / 93.1 / 94.484.3 / 95.4 / 96.591.0 / 96.6 / 97.0
ES73.2 / 94.9 / 96.681.6 / 96.4 / 97.991.0 / 97.6 / 98.1
  1. Compared with the lower learning rate: Our original setting achieves higher Pass@1 on all three models, with similar Pass@16 and Pass@32.

  2. Compared with the higher learning rate: Our original setting achieves higher Pass@1, Pass@16, and Pass@32 in most model–metric comparisons.

These results support our original GRPO setting as a reasonable baseline choice. Moreover, ES retains higher Pass@16 and Pass@32 across all tested learning rates, showing that its broader reasoning coverage is not specific to the GRPO learning rate used in our main experiments.


We sincerely hope our responses have addressed all your concerns. If you have any further questions, please feel free to follow up, and we are glad to address them.

References

[1] Fahim Tajwar, Guanning Zeng, Yueer Zhou, Yuda Song, Daman Arora, Yiding Jiang, Jeff Schneider, Ruslan Salakhutdinov, Haiwen Feng, and Andrea Zanette. Maximum Likelihood Reinforcement Learning. Proceedings of the 43rd International Conference on Machine Learning (ICML), 2026.

[2] Renren Jin, Pengzhi Gao, Yuqi Ren, Zhuowen Han, Tongxuan Zhang, Wuwei Huang, Wei Liu, Jian Luan, and Deyi Xiong. Revisiting Entropy in Reinforcement Learning for Large Reasoning Models. Findings of the Association for Computational Linguistics: ACL 2026.

[3] Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Liyuan Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, Weizhu Chen, Shuohang Wang, Simon Shaolei Du, and Yelong Shen. Reinforcement Learning for Reasoning in Large Language Models with One Training Example. Advances in Neural Information Processing Systems 38 (NeurIPS), 2025.

[4] Gang Li, Ming Lin, Tomer Galanti, Zhengzhong Tu, and Tianbao Yang. DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimization. Advances in Neural Information Processing Systems 38 (NeurIPS), 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.