- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper investigates forgetting during Evolution Strategies (ES) fine-tuning of LLMs and proposes Anchored Weight Decay (AWD) to reduce it. Across reasoning tasks, model families, and model sizes, the authors observe that prior-task performance can decline and subsequently recover during ES training, suggesting that some apparent forgetting reflects performance drift rather than permanent degradation. They attribute this behavior primarily to random-walk movement in parameter directions weakly constrained by the target task. Increasing the ES population reduces this drift but requires more model evaluations; AWD instead pulls parameters toward the initial model after each update. Experiments show that AWD substantially improves retention while largely maintaining target-task gains, and comparisons with GRPO demonstrate that forgetting is not exclusive to ES.
Critical strengths
The paper provides a thorough empirical analysis of ES forgetting across multiple tasks, model families, and model sizes. Tracking performance throughout training reveals recovery patterns that final-checkpoint evaluations would miss. Population-size experiments and analyses of parameter displacement and KL divergence provide complementary evidence for the proposed drift explanation.
AWD is a simple and practical mitigation that substantially reduces prior-task degradation while largely preserving target-task improvements. It requires no additional model evaluations and adds only 1–2% reported runtime overhead. Penalty-strength ablations and the transfer of selected settings across tasks and models further support its practical value.
Critical weaknesses
The methodological novelty of AWD appears limited, given its close relationship to existing approaches that regularize parameters toward pretrained weights. Nevertheless, its simplicity and demonstrated effectiveness are valuable. I view the main contribution as the empirical investigation of ES forgetting and the practical adaptation of an established regularization principle to ES, rather than a fundamentally new regularization method.
Questions for authors
To help me better understand the differences between ES and GRPO, could the authors clarify what drives GRPO’s substantially higher target-task KL divergence despite its smaller parameter displacement and comparable accuracy. Does this primarily reflect changes in answer confidence, reasoning trajectories, and/or output formatting?
2 comments
Thank you for the explanation. I will leave my current score of 3 (weak accept).
Thank you for the detailed and constructive review and for pointing out the empirical analysis as thorough as well as the proposed AWD being practical and well supported. The raised weakness and question are addressed below:
Anchoring weights to a reference model is indeed an established technique in gradient descent based optimization (e.g. EWC, SI, L2-SP; discussed in Sec. 2.2 and App. C), and AWD can be seen as a principled adaptation of these techniques to fine-tuning LLMs with ES as the optimization algorithm, rather than a fundamentally new paradigm for weight regularization. This paper makes three technical contributions: (i) it grounds the method on an ES-specific failure mode, with the aim to limit random walk in directions unconstrained by the target task. Prior work on forgetting under gradient descent focused on task interference and retaining weights important for prior tasks, not random walk during optimization on a new target task; (ii) it provides a regularizer for backpropagation-free ES, where the penalty cannot enter via a loss and is instead applied on the update rule; and (iii) it provides an efficient implementation of AWD at 1-2% runtime overhead.
That said, it is correct that the main contribution of this work is the analysis of ES forgetting, the findings of which then motivate AWD as a simple mitigation strategy. We will state these contributions explicitly in the revised version.
In terms of behavior of GRPO and ES fine-tuned models, their reasoning trajectories and output formatting are similar. The differences are primarily seen in the confidence of their answers.
This observation is consistent with recent work on solution coverage: ES outperforms RL in pass@k [1], and the base model catches up and even outperforms RL at sufficiently high k [2]. In contrast, Hayes et. al [1] shows that ES remains better than the base model even for very large k, sometimes at the cost of a slightly lower pass@1 than RL. These results indicate that RL concentrates probability mass on particular high-reward answers, while ES preserves more of the base model's distribution. Since KL(pθ0∣∣pθT) is reported, which is large precisely when the fine-tuned model assigns low probability to outputs of the base model, this sharpening of the output distribution directly translates to higher target-task KL divergence. We added this discussion to Sec. 6 in the paper.
Related to that, preliminary layer-wise analysis (which we intend to publish as part of a future study) points to the same mechanism: GRPO updates are sparse and concentrated in the final layers. Furthermore, hidden representations remain nearly identical to the base model until the last 2-3 layers and then diverge sharply (much more than with ES). The output logits are therefore strongly reshaped, while the overall update norm remains small. In contrast, ES changes weights uniformly across all layers, which makes the overall update norm large; hidden representations drift more homogeneously across layers and the logits change less, leading to the observed smaller change in KL albeit the higher update norm.
[1] Beyond the Best Guess: Improving LLM Solution Coverage with Evolution Strategies (2026), ArXiv
[2] Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (2025), NeurIPS