- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3(+1)
- Soundness
- 3(+1)
Read review
Paper summary
This work examines a broad look at evolution strategies for fine tuning models and describes broad benefits of this method over GRPO. The author(s) provide analytical reasoning for the expected improvement on pass@k by ES over GRPO. The work then demonstrates improved pass@k performance by ES in a variety of fine tuning tasks. Next the work demonstrates the large parameter drift that occurs with ES, and how this parameter drift is not an indicator of performance or off task degradation, showing that held out task performance does not significantly decrease during training. Next design choices are evaluated, including reward normalization, perturbation scale, population size, and mirror sampling.
Overall the work runs a number of very useful ablations for evolution strategies in LLM fine tuning, however it ends up spreading too thin across all the tests. Either more thorough testing should be done or some of the testing should be dropped to allow for greater focus on a subset of tests.
Critical strengths
The paper runs some very useful experiments within the broader field of evolution strategies for post training in LLM's. The demonstration of improved pass@k performance across tasks and models using evolution strategies is novel along concurrent work (Hayes et al), and is valuable to have for the community. The tests of sequential training (ES, then GRPO and vice versa) are a good tests to determine if the benefits of both methods can be aggregated. Held out performance tests are a good approach to check for catastrophic forgetting. The test of reward function choice (none vs z-score) is a good ablation to have as well.
Critical weaknesses
The paper may be trying to test too many things at once. There may be a version of this that works well, but in its current state this leaves some of the most interesting tests left under-examined.
- some of the most interesting ablations seem under-examined. The z-score reward comparison just references one training run on the first figure, with no mention of what test it was, what model was used, what the settings where, etc. This is a very valuable test to run but it should really be run across multiple benchmarks/models, and the test conditions are strictly needed.
2)section 5.1 mentions describes perturbation scale as if it is something tested in the paper, but makes no reference to a table, figure, or experiment. I found mention of an experiment in the appendix, but if this is referenced in the body of the paper it really needs to be tied to an experiment in the paper.
- one vs two point estimation - again a very interesting ablation, it appears there is only one training run of this?
Questions for authors
-
The conclusion in section 3.1, with regards to ES update improving Pass@k, seems to rely on lemma 1, which itself relies on an assumption from appendix B.2 - "For the local relation, let p_i = p_bar + \eps_i." - this assumes that the perturbation pass probability is centered at the centerpoint model probability. How is this justified? From my own work with ES this may be true at small scales, but trivially the perturbation size could be made large enough that every population members probability falls to zero and this becomes trivially untrue.
-
the Z-score normalization comparison - what task/model was this run on? Section 5.1 references a figure on the first page, but that figure seems to have no extra information, and seems to be only one training run.
-
On the perturbation scaling - it is mentioned in section 5.1 but no tables or figures or experiments are referenced. What testing of perturbation scale was done here?
6 comments
Response to Reviewer sfn6 (Part 2/4)
Weakness 2 & Question 3. Experimental Evidence for Perturbation Scale: "On the perturbation scaling - it is mentioned in section 5.1 but no tables or figures or experiments are referenced. What testing of perturbation scale was done here?"
Thank you for pointing this out. We have added a perturbation-scale sweep and an explicit reference to it in Section 5.1 (Appendix M, Figure 16).
The sweep varies on GSM8K for the three models below. The tables report smoothed mean training rewards.
| Qwen2.5-0.5B-Instruct (N=32) | Step 1 | Step 10 | Step 50 | Step 100 | Step 234 |
|---|---|---|---|---|---|
| σ=0.001 | 0.6009 | 0.5929 | 0.5904 | 0.5979 | 0.6015 |
| σ=0.002 | 0.2694 | 0.2822 | 0.3317 | 0.3613 | 0.3911 |
| σ=0.003 | 0.0175 | 0.0168 | 0.0315 | 0.0601 | 0.1170 |
| σ=0.004 | 0.0026 | 0.0021 | 0.0028 | 0.0049 | 0.0223 |
| Qwen2.5-1.5B-Instruct (N=16) | Step 1 | Step 10 | Step 50 | Step 100 | Step 234 |
|---|---|---|---|---|---|
| σ=0.001 | 0.8305 | 0.8128 | 0.8344 | 0.8430 | 0.8517 |
| σ=0.002 | 0.7695 | 0.7435 | 0.7832 | 0.8001 | 0.8158 |
| σ=0.003 | 0.6151 | 0.6081 | 0.6619 | 0.7003 | 0.7364 |
| σ=0.006 | 0.0046 | 0.0077 | 0.0120 | 0.0169 | 0.0370 |
| Qwen2.5-3B-Instruct (N=8) | Step 1 | Step 10 | Step 50 | Step 100 | Step 234 |
|---|---|---|---|---|---|
| σ=0.002 | 0.8939 | 0.9014 | 0.8936 | 0.8912 | 0.8927 |
| σ=0.003 | 0.7236 | 0.7610 | 0.7782 | 0.7961 | 0.8121 |
| σ=0.004 | 0.4232 | 0.4375 | 0.4613 | 0.4965 | 0.5521 |
| σ=0.006 | 0.0035 | 0.0034 | 0.0029 | 0.0043 | 0.0075 |
-
Excessively large perturbations: At the largest tested scales, all three models start with very low rewards and show limited recovery during training.
-
Excessively small perturbations: The smallest tested scales preserve higher rewards but show little subsequent improvement. From a parameter-space search perspective, excessively small perturbations restrict exploration to the vicinity of the current parameters, which may limit access to other beneficial regions.
-
Intermediate perturbations: Intermediate scales produce larger reward gains during training than the smallest tested scales, while avoiding the severe reward degradation observed at the largest scales.
These results support choosing a perturbation scale that preserves task performance while allowing further training progress.
Response to Reviewer sfn6 (Part 3/4)
Weakness 3. One-Point versus Two-Point Comparison across Models and Tasks: one vs two point estimation - again a very interesting ablation, it appears there is only one training run of this?
Thank you for your comment. We have expanded the one-point versus two-point comparison to four models across Easy and Hard settings (Appendix O, Figures 18–19).
The models and tasks follow the normalization comparison above. Each step evaluates the same number of perturbed models: 32 for Qwen2.5-0.5B-Instruct and 16 for the others. Paired values denote one-point / two-point results, with task performance reported as Pass@1 (%).
| Qwen2.5-0.5B-Instruct | Smoothed reward | Pass@1 |
|---|---|---|
| Step 39 | 0.4854 / 0.4874 | 46.8 / 46.6 |
| Step 78 | 0.5008 / 0.5014 | 47.5 / 46.9 |
| Step 117 | 0.5073 / 0.5072 | 46.9 / 46.5 |
| Step 156 | 0.5117 / 0.5095 | 47.4 / 47.1 |
| Step 195 | 0.5111 / 0.5086 | 47.1 / 46.5 |
| Step 234 | 0.5146 / 0.5111 | 47.3 / 46.9 |
| Qwen2.5-1.5B-Instruct | Smoothed reward | Pass@1 |
|---|---|---|
| Step 39 | 0.8050 / 0.8086 | 72.5 / 72.3 |
| Step 78 | 0.8195 / 0.8201 | 72.6 / 72.6 |
| Step 117 | 0.8295 / 0.8294 | 72.6 / 72.9 |
| Step 156 | 0.8337 / 0.8328 | 72.5 / 72.3 |
| Step 195 | 0.8326 / 0.8312 | 72.5 / 72.5 |
| Step 234 | 0.8381 / 0.8357 | 72.5 / 72.3 |
| Llama-3.2-3B-Instruct | Smoothed reward | Pass@1 |
|---|---|---|
| Step 39 | 0.7249 / 0.7247 | 78.4 / 78.1 |
| Step 78 | 0.8004 / 0.8027 | 78.5 / 77.9 |
| Step 117 | 0.8399 / 0.8400 | 80.0 / 79.3 |
| Step 156 | 0.8695 / 0.8672 | 81.3 / 81.1 |
| Step 195 | 0.8870 / 0.8844 | 81.2 / 81.3 |
| Step 234 | 0.9003 / 0.8999 | 81.2 / 80.9 |
| DeepSeek-R1-Distill-Qwen-1.5B | Smoothed reward | MATH500 | AIME24 | AIME25 | AMC23 |
|---|---|---|---|---|---|
| Step 30 | 0.4563 / 0.4574 | 81.2 / 80.8 | 24.1 / 24.7 | 22.9 / 23.1 | 64.0 / 65.1 |
| Step 60 | 0.4632 / 0.4614 | 81.7 / 81.5 | 24.8 / 23.9 | 23.0 / 21.6 | 65.2 / 65.8 |
| Step 90 | 0.4628 / 0.4597 | 81.8 / 81.4 | 22.8 / 25.7 | 23.9 / 23.0 | 67.7 / 66.6 |
| Step 120 | 0.4716 / 0.4668 | 82.2 / 81.6 | 25.3 / 25.0 | 23.0 / 22.6 | 68.1 / 65.9 |
| Step 150 | 0.4795 / 0.4726 | 82.3 / 81.9 | 24.6 / 23.2 | 22.9 / 21.7 | 69.1 / 68.1 |
| Step 180 | 0.4836 / 0.4765 | 82.6 / 82.0 | 27.2 / 23.5 | 23.0 / 22.3 | 69.3 / 68.0 |
| Step 210 | 0.4877 / 0.4804 | 82.4 / 82.0 | 25.4 / 25.3 | 23.9 / 23.0 | 67.9 / 68.6 |
| Step 240 | 0.4914 / 0.4832 | 82.5 / 82.3 | 24.7 / 25.8 | 22.6 / 23.8 | 69.8 / 68.3 |
| Step 270 | 0.5001 / 0.4918 | 82.8 / 82.7 | 26.1 / 25.2 | 22.6 / 21.9 | 70.6 / 68.4 |
-
Training rewards: The two estimators have similar training rewards in the Easy setting, while one-point ES generally achieves higher rewards in the Hard setting.
-
Test performance: One-point ES generally achieves higher Pass@1 during training and outperforms two-point ES at the final step across all evaluated models and benchmarks.
These results show that one-point ES remains competitive across the tested models and tasks under the same evaluation budget.
Response to Reviewer sfn6 (Part 1/4)
Thank you for your time and effort in reviewing our work. We are glad to know that you find our Pass@K findings valuable to the community and our sequential-training, held-out evaluation, and reward-normalization experiments useful.
We address your concerns as follows, and all revisions are marked in blue in the revised PDF.
Thank you for your suggestion. The original comparison uses Qwen2.5-1.5B-Instruct on GSM8K. We have added comparisons across four models in the Easy and Hard settings (Appendix L, Figures 14–15).
The Easy-setting models are trained and evaluated on GSM8K. DeepSeek-R1-Distill-Qwen-1.5B is trained on DeepScaleR and evaluated on MATH500, AIME24, AIME25, and AMC23. The population size is N=32 for Qwen2.5-0.5B-Instruct and N=16 for the others. The tables report training rewards and task Pass@1 (%) across training steps, with paired values ordered as without normalization / with z-score normalization.
Easy setting: Z-score normalization consistently improves training rewards and GSM8K Pass@1 throughout training across all three models.
Hard setting: Z-score normalization generally improves training rewards and Pass@1 during training, with higher final Pass@1 on all four benchmarks.
These results support the effectiveness of z-score normalization across models and tasks.