acceptodds
Under review as a conference paper at ICLR 2027

Overcoming Forgetting in LLM Fine-Tuning with Evolution Strategies

Abstract

Evolution Strategies (ES) has recently emerged as a competitive alternative to reinforcement learning (RL) for large language model (LLM) fine-tuning, offering advantages through simplicity, scalability, and inference-only training. However, recent work suggests that ES fine-tuning on new tasks may induce forgetting of prior tasks. First, this paper shows that prior task forgetting (1) is better characterized as performance drift rather than irreversible forgetting, with prior-task performance often recovering during ES training; and (2) is not a specific failure mode of ES, but can also arise for fine-tuning with RL methods. Second, it analyzes when and why such drift arises, highlighting its dependence on ES training dynamics, particularly random walk behavior in weakly constrained directions of the weight space. Third, based on these insights, it introduces Anchored Weight Decay (AWD) as a parameter-space regularization technique that constrains optimization toward the initial model parameters. AWD effectively stabilizes prior-task performance while preserving target-task performance, achieving benefits comparable to large ES population sizes at much lower computational cost. Thus, contrary to previous beliefs, the paper shows that prior-task forgetting under ES is largely avoidable, positioning ES as a promising approach for continual learning in LLMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Evolution Strategies, LLM fine-tuning, post-training, catastrophic forgetting, continual learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_muDoOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates forgetting during Evolution Strategies (ES) fine-tuning of LLMs and proposes Anchored Weight Decay (AWD) to reduce it. Across reasoning tasks, model families, and model sizes, the authors observe that prior-task performance can decline and subsequently recover during ES training, suggesting that some apparent forgetting reflects performance drift rather than permanent degradation. They attribute this behavior primarily to random-walk movement in parameter directions weakly constrained by the target task. Increasing the ES population reduces this drift but requires more model evaluations; AWD instead pulls parameters toward the initial model after each update. Experiments show that AWD substantially improves retention while largely maintaining target-task gains, and comparisons with GRPO demonstrate that forgetting is not exclusive to ES.

Critical strengths

The paper provides a thorough empirical analysis of ES forgetting across multiple tasks, model families, and model sizes. Tracking performance throughout training reveals recovery patterns that final-checkpoint evaluations would miss. Population-size experiments and analyses of parameter displacement and KL divergence provide complementary evidence for the proposed drift explanation.

AWD is a simple and practical mitigation that substantially reduces prior-task degradation while largely preserving target-task improvements. It requires no additional model evaluations and adds only 1–2% reported runtime overhead. Penalty-strength ablations and the transfer of selected settings across tasks and models further support its practical value.

Critical weaknesses

The methodological novelty of AWD appears limited, given its close relationship to existing approaches that regularize parameters toward pretrained weights. Nevertheless, its simplicity and demonstrated effectiveness are valuable. I view the main contribution as the empirical investigation of ES forgetting and the practical adaptation of an established regularization principle to ES, rather than a fundamentally new regularization method.

Questions for authors

To help me better understand the differences between ES and GRPO, could the authors clarify what drives GRPO’s substantially higher target-task KL divergence despite its smaller parameter displacement and comparable accuracy. Does this primarily reflect changes in answer confidence, reasoning trajectories, and/or output formatting?

2 comments

Thank you for the detailed and constructive review and for pointing out the empirical analysis as thorough as well as the proposed AWD being practical and well supported. The raised weakness and question are addressed below:

Novelty of AWD

Anchoring weights to a reference model is indeed an established technique in gradient descent based optimization (e.g. EWC, SI, L2-SP; discussed in Sec. 2.2 and App. C), and AWD can be seen as a principled adaptation of these techniques to fine-tuning LLMs with ES as the optimization algorithm, rather than a fundamentally new paradigm for weight regularization. This paper makes three technical contributions: (i) it grounds the method on an ES-specific failure mode, with the aim to limit random walk in directions unconstrained by the target task. Prior work on forgetting under gradient descent focused on task interference and retaining weights important for prior tasks, not random walk during optimization on a new target task; (ii) it provides a regularizer for backpropagation-free ES, where the penalty cannot enter via a loss and is instead applied on the update rule; and (iii) it provides an efficient implementation of AWD at 1-2% runtime overhead.

That said, it is correct that the main contribution of this work is the analysis of ES forgetting, the findings of which then motivate AWD as a simple mitigation strategy. We will state these contributions explicitly in the revised version.

Why GRPO has substantially higher target-task KL divergence despite smaller parameter displacement and comparable accuracy

In terms of behavior of GRPO and ES fine-tuned models, their reasoning trajectories and output formatting are similar. The differences are primarily seen in the confidence of their answers.

This observation is consistent with recent work on solution coverage: ES outperforms RL in pass@k [1], and the base model catches up and even outperforms RL at sufficiently high k [2]. In contrast, Hayes et. al [1] shows that ES remains better than the base model even for very large k, sometimes at the cost of a slightly lower pass@1 than RL. These results indicate that RL concentrates probability mass on particular high-reward answers, while ES preserves more of the base model's distribution. Since KL(pθ0∣∣pθT)KL(p_{\theta_0}||p_{\theta_T}) is reported, which is large precisely when the fine-tuned model assigns low probability to outputs of the base model, this sharpening of the output distribution directly translates to higher target-task KL divergence. We added this discussion to Sec. 6 in the paper.

Related to that, preliminary layer-wise analysis (which we intend to publish as part of a future study) points to the same mechanism: GRPO updates are sparse and concentrated in the final layers. Furthermore, hidden representations remain nearly identical to the base model until the last 2-3 layers and then diverge sharply (much more than with ES). The output logits are therefore strongly reshaped, while the overall update norm remains small. In contrast, ES changes weights uniformly across all layers, which makes the overall update norm large; hidden representations drift more homogeneously across layers and the logits change less, leading to the observed smaller change in KL albeit the higher update norm.


[1] Beyond the Best Guess: Improving LLM Solution Coverage with Evolution Strategies (2026), ArXiv

[2] Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (2025), NeurIPS

Thank you for the explanation. I will leave my current score of 3 (weak accept).

Reviewer_9urQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies forgetting during ES finetuning and proposes Anchored Weight Decay (AWD), which pulls parameters toward the initial model after each update. Larger populations and AWD both reduce prior-task degradation while retaining most target-task gains. The authors attribute this to reducing random walk in directions unrelated to the target task.

Critical strengths

The paper measures prior-task performance throughout training. The observation that some capabilities recover without being trained on again is interesting and qualifies conclusions drawn from individual checkpoints.

AWD substantially improves retention with little overhead and only a small reduction in target performance. Using the same coefficients across several settings makes this more convincing.

Critical weaknesses

The regularization idea seems quite standard. Keeping weights close to the pretrained checkpoint has been studied extensively. Applying it to ES is useful, but the paper needs to be clearer about what is new beyond this application.

The random-walk explanation is plausible but not isolated by the experiments. Larger populations improve the estimator, and AWD limits all displacement, including useful changes. These results alone do not show that suppressing task-irrelevant random walk explains the retention gains.

Questions for authors

How does AWD compare with smaller ES step sizes, early stopping, or interpolation toward the base checkpoint, at matched target accuracy?

What does AWD add over existing regularization toward pretrained weights? A direct comparison would help assess the contribution.

Does it work across a sequence of tasks? The current experiments show retention during one adaptation, which seems insufficient to support the lifelong-learning claims.

7 comments

Thank you for the careful review and pointing out the insightfulness of the analysis of prior-task performance over the course of training, as well as AWD being an effective and practical way to retain prior capabilities. Below the raised weaknesses and questions are addressed:

W1: Novelty of AWD

Anchoring weights to a reference model is indeed an established technique in gradient descent based optimization (e.g. EWC, SI, L2-SP; discussed in Sec. 2.2 and App. C), and AWD can be seen as a principled adaptation of these techniques to fine-tuning LLMs with ES as the optimization algorithm, rather than a fundamentally new paradigm for weight regularization. This paper makes three technical contributions: (i) it grounds the method on an ES-specific failure mode, with the aim to limit random walk in directions unconstrained by the target task. Prior work on forgetting under gradient descent focused on task interference and retaining weights important for prior tasks, not random walk during optimization on a new target task; (ii) it provides a regularizer for backpropagation-free ES, where the penalty cannot enter via a loss and is instead applied on the update rule; and (iii) it provides an efficient implementation of AWD at 1-2% runtime overhead.

Furthermore, as also pointed out by reviewer muDo, the main contribution of this paper is the analysis of ES forgetting and its cause, the findings of which motivate AWD as a simple mitigation strategy that adapts the weight regularization principle to ES.

We will state these contributions explicitly in the revised version.

W2: Isolating the random-walk explanation

While it may not be possible to rule out every conceivable explanation, the experiments aim to rule out the most likely confounders, i.e.- target task, model size and model family. The experiments in section 4.2 show that there is no correlation between any of those factors and prior task forgetting. Indeed, larger populations improve the estimator at each step - reducing its variance - which is equivalent to reducing random walk along the optimization trajectory.

While it is not possible to run ES without any random walk, the opposite experiment was run to provide additional evidence for the hypothesis (that random walk is causing prior task forgetting). In particular, ES was run for 500 iterations with random fitness values, with the setup otherwise identical to that during training (Qwen2.5-3B-Instruct, population size 30). This experiment gives the following results:

CountdownGSM8KProofWriterHellaSwagPIQAARC-CMMLU-ProAverage
-5.8 ± 5.2-15.9 ± 26.8+3.6 ± 6.9-5.9 ± 5.5-1.7 ± 0.5-2.4 ± 0.1-5.6 ± 2.4-4.8 ± 2.8

Thus, pure random walk shows similar forgetting as standard ES training, supporting the hypothesis that random walk is the explanation for the phenomenon. We will add this new experiment and further clarifications to the revision.

Note also two further insights about this experiment: First, the accuracy for ProofWriter improved in two runs. The task requires deciding whether some logical statement is true, false or undecidable given the information. Base models are often overconfident on this task, underusing undecidable. Moving a few samples to undecidable during random walk - empirically the most likely change to the model - often improves performance on this task compared to the base model. Second, there is a high variance on GSM8K. This task has strict format requirements for answer extraction, which the model starts to completely ignore in one seed. Under ES training this was not observed to happen, most likely because the target task always requires to follow a certain format rule, penalizing the model ignoring the formatting instructions in the prompt.

In addition to the above new experiment, another new experiment described below (in response to Q1/Q2) also supports the random walk hypothesis. The step size α\alpha was reduced, thus reducing random walk (according to Eq. 2). As a result, forgetting was also reduced but often at the cost of target task performance. This result can be seen as additional evidence that random walk is the main cause of forgetting.

Q1/Q2: Comparing AWD to alternative methods

Three new experiments comparing to alternative regularization methods were run for this rebuttal:

Early stopping: Prior task forgetting starts relatively early on, e.g. -2.0% on ProofWriter after 25 iterations and -5.1% on GSM8K after 100 iterations, while target task accuracy is still far from the final performance. Furthermore, forgetting is not monotonic per task, making checkpoint selection difficult, and requires costly prior task evals. Early stopping is thus not a scalable remedy to forgetting under ES.

IterationCountdown (target)GSM8KProofWriterHellaSwagPIQAARC-CMMLU-ProAverage (excl. target)
25+6.8 ± 1.7-0.1 ± 0.4-2.0 ± 3.2+1.3 ± 0.8+0.1 ± 0.3+0.0 ± 1.0-0.3 ± 1.0-0.2 ± 0.4
50+17.8 ± 1.9+0.5 ± 1.3+1.0 ± 2.3+0.5 ± 1.2-0.2 ± 0.2+0.7 ± 0.7-1.8 ± 1.7+0.1 ± 0.2
75+27.6 ± 1.5-2.5 ± 4.7+2.6 ± 4.5-0.6 ± 0.6-0.3 ± 0.4-0.4 ± 0.8-0.9 ± 0.9-0.4 ± 1.4
100+32.6 ± 1.4-5.1 ± 10.6+2.5 ± 5.6-1.9 ± 1.2-0.7 ± 0.1-0.9 ± 0.8-0.5 ± 1.6-1.1 ± 2.8
200+39.9 ± 0.9+0.3 ± 1.9+2.8 ± 6.0-3.5 ± 3.6-2.0 ± 1.3-1.3 ± 0.0-2.1 ± 2.3-1.0 ± 1.3
300+43.0 ± 1.4-4.1 ± 4.5+6.8 ± 3.9-5.6 ± 3.5-2.0 ± 1.2-1.9 ± 1.2-3.0 ± 1.8-1.6 ± 1.6
400+44.1 ± 1.4-4.5 ± 5.1+3.5 ± 5.8-8.3 ± 5.7-2.9 ± 0.8-2.5 ± 0.4-6.0 ± 2.7-3.4 ± 2.6
500+45.3 ± 0.3-9.5 ± 11.5-2.0 ± 5.7-8.2 ± 9.0-2.8 ± 1.0-2.0 ± 0.5-6.5 ± 2.6-5.2 ± 4.7

Interpolation towards base model: The results show that as soon as interpolation attains reasonable performance on the target task (w=0.5, halfway between base and final), forgetting sets in. At w=0.75 where nearly the full performance is reached, there is only moderately less forgetting than for the final model. It is thus not a viable alternative to AWD.

wCountdown (target)GSM8KProofWriterHellaSwagPIQAARC-CMMLU-ProAverage (excl. target)
0.25+15.0 ± 5.7+0.1 ± 0.8+1.2 ± 3.3-0.8 ± 1.9-0.2 ± 0.2-0.3 ± 0.4-1.7 ± 0.9-0.3 ± 0.8
0.5+38.2 ± 1.2-4.5 ± 7.6+0.4 ± 2.5-2.2 ± 4.5-0.8 ± 0.5-0.6 ± 0.4-3.4 ± 1.2-1.8 ± 2.3
0.75+43.9 ± 0.6-9.2 ± 13.9-0.0 ± 4.4-5.4 ± 7.4-1.8 ± 1.0-1.3 ± 0.4-4.9 ± 2.0-3.8 ± 4.3
1+45.3 ± 0.3-9.5 ± 11.5-2.0 ± 5.7-8.2 ± 9.0-2.8 ± 1.0-2.0 ± 0.5-6.5 ± 2.6-5.2 ± 4.7

Smaller step size α\alpha: As predicted by Eq. 2, reducing the step size shrinks the random-walk component and thereby forgetting. In this experiment α\alpha was reduced from 5e-4 (the value recommended in [1] and used throughout) to 3e-4 and 1e-4. For Qwen 2.5 3B Instruct, 3e-4 attains a similar tradeoff between target task performance and prior task forgetting as AWD, while 1e-4 severely reduces target task performance:

α\alphaCountdown (target)GSM8KProofWriterHellaSwagPIQAARC-CMMLU-ProAverage (excl. target)
5e-4+45.3 ± 0.3-9.5 ± 11.5-2.0 ± 5.7-8.2 ± 9.0-2.8 ± 1.0-2.0 ± 0.5-6.5 ± 2.6-5.2 ± 4.7
3e-4+44.6 ± 0.6-1.5 ± 3.0+2.9 ± 5.4-1.6 ± 2.7-0.9 ± 0.6-0.7 ± 0.6-3.0 ± 2.4-0.8 ± 0.6
1e-4+35.0 ± 0.7+0.4 ± 0.7+2.5 ± 0.3+0.9 ± 0.0+0.0 ± 0.4+0.0 ± 0.7+0.7 ± 0.3+0.7 ± 0.2
AWD (ℓ2\ell_2)+43.8 ± 0.7-2.5 ± 5.0+1.0 ± 6.8-0.7 ± 1.6-0.9 ± 0.2-0.5 ± 1.1-2.6 ± 0.9-1.0 ± 2.4

However, for Llama 3.2 3B Instruct, AWD improves on 3e-4 in prior task forgetting at matched target task performance, and gains 5.2 higher target task performance than 1e-4 at similar levels of forgetting:

α\alphaCountdown (target)GSM8KProofWriterHellaSwagPIQAARC-CMMLU-ProAverage (excl. target)
5e-4+46.9 ± 1.0-5.1 ± 3.4+3.9 ± 6.0-12.6 ± 2.0-3.3 ± 3.3-4.2 ± 1.1-5.4 ± 3.8-4.4 ± 1.7
3e-4+44.1 ± 0.6-2.9 ± 1.5-0.3 ± 2.8-3.0 ± 2.9-1.0 ± 2.8-1.8 ± 0.2-6.7 ± 1.4-2.6 ± 1.4
1e-4+39.4 ± 0.4+3.2 ± 1.8-0.0 ± 1.8-1.4 ± 0.5+0.7 ± 0.4-0.7 ± 0.3-2.4 ± 2.2-0.1 ± 0.7
AWD (ℓ2\ell_2)+44.6 ± 0.1+2.9 ± 2.1-1.7 ± 3.6-2.8 ± 0.7+1.6 ± 2.2-0.9 ± 0.6-3.5 ± 3.5-0.7 ± 1.0

However, tuning the step size is not a substitute for AWD for three reasons: (i) The step taken in the signal direction gg over all iterations is ≈αTg\approx \alpha T g, so decreasing α\alpha caps convergence speed. Matching target progress requires proportionally more iterations, trading forgetting for compute similar to increasing the population size. That α\alpha = 3e-4 performs well here is likely due to Countdown saturating after about 300 iterations, well below the 500 iteration budget (see Fig 1a). (ii) A smaller α\alpha only slows the random walk, it doesn’t stop it. Once the target task saturates (about 300 iterations on Countdown, Fig. 1a), the drift keeps growing with no target-task benefit. AWD instead pulls back every direction that the fitness signal does not actively sustain, so drift stays bounded however long training runs. α\alpha = 3e-4 looks competitive within the 500-iteration budget, yet the difference is expected to grow in longer runs (e.g. 1500–2500 iterations for ARC-AGI and Sudodu runs in [1]) and generally in continual learning. (iii) The step size is primarily an optimization hyperparameter. Using it to also trade off target performance against forgetting couples two concerns and could become prohibitive, e.g. under step-size schedules as [2] explored for more complex agentic tasks. AWD decouples them.

Therefore, while smaller step size is not a solution to forgetting, the experiment provided further insight into the underlying random-walk mechanism.

We will add these comparisons and insights into the revision, where they will strengthen the results. Thanks for suggesting them!

Reviewer_keo1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies prior-task forgetting when LLMs are fine-tuned with Evolution Strategies. It argues that this forgetting is better described as performance drift that often recovers during training, that it also appears under GRPO, and that it is driven by a random walk of ES in weight directions weakly constrained by the target task. To counter this drift, the paper proposes Anchored Weight Decay, which adds an ℓ1\ell_1 or ℓ2\ell_2 decay toward the initial weights to every ES update. Experiments on Qwen2.5 and Llama-3.2 models with Countdown, GSM8K and ProofWriter as target tasks report that AWD with a population of 30 reduces prior-task degradation to a level similar to much larger populations.

Critical strengths
  1. Timely question studied systematically. Whether ES fine-tuning preserves prior abilities matters as ES emerges as an alternative to RL, and the paper studies it across multiple target tasks, prior tasks, model families and sizes.

  2. A simple mitigation derived from a clear hypothesis. AWD follows directly from the drift hypothesis by anchoring the decay at the pretrained weights, and it also works on Llama-3.2 3B, reducing the average prior-task change from −4.4 to −0.7.

Critical weaknesses
  1. Attributing forgetting to the random walk does not fully hold. Sec. 4.3 names the random walk as the primary driver of forgetting, and Eq. 2 and Fig. 5 show that ∥θt−θ0∥\|\theta_t - \theta_0\| under ES(30) keeps growing throughout training, yet Fig. 2 shows prior-task accuracy recovering in the same runs. Fig. 5 further shows that AWD (ℓ1\ell_1) ends with a norm only about a fifth below ES(30) while cutting the average forgetting from −5.2-5.2 to −0.9-0.9, so prior-task degradation is not explained by how far the weights drift from θ0\theta_0.

  2. AWD essentially reduces variance, and the equally simple alternatives are not compared. Like a larger population, AWD bounds the accumulated noise in the update, which explains why the two behave similarly. By the same logic, a smaller step size α\alpha, iterate averaging, or the early stopping proposed by Hoy et al. 2026, which the paper itself cites, could have a similar effect, but none of them is evaluated against AWD.

Questions for authors
  1. What is the actual cost advantage of AWD over larger populations? The paper reports the cost of standard ES(30) runs and a 11–2%2\% overhead for AWD, but not the cost of ES(128) or ES(256), so the claim of much lower cost is not quantified. The comparison also covers only prior tasks: on Countdown, ES(256) reaches +48.4+48.4 while AWD reaches +44.3+44.3 (ℓ1\ell_1) and +43.8+43.8 (ℓ2\ell_2).

  2. What supports positioning ES as promising for continual learning? The abstract and conclusion make this claim, but every experiment fine-tunes on a single target task. A sequential multi-task setting, such as the one in Hoy et al. 2026, would show whether AWD still preserves earlier tasks when several tasks are learned in turn.

4 comments

Thank you for the careful review, and especially identifying value in both the analysis of forgetting under ES and the mitigation strategy that results from it. Each of the raised weaknesses and questions are addressed in turn below.

W1: Attribution of forgetting to random walk

Indeed the norm of weight updates alone is not the reason for forgetting (as pointed out e.g. in lines 443,444 and 455 of the original version), contrary to prior work [1] that attributed prior task forgetting under ES to dense, high-magnitude weight updates. The hypothesis is that the root cause is random walk, and its effects are performance drift on prior tasks as well as increased update norm due to dense, high-magnitude updates.

Importantly, an increased absolute norm does not necessarily mean that the random walk leads the model into a region where it degrades on a specific prior task. Yet the higher the update norm, the higher the potential to change the model on any prior task. Therefore, spontaneous performance recovery (Fig.2) and monotonically increasing update norm (Eq.2 and Fig.5) are not incompatible. Furthermore, the performance of AWD may be attributed less to constraining the absolute update norm and rather to decaying overly strong deviations from the base model in directions irrelevant to the target task, which is why ℓ1\ell_1 and ℓ2\ell_2 decay lead to similar forgetting, but different update norm.

To provide further evidence for this hypothesis (that random walk causes forgetting), ES was run for 500 iterations with random fitness values, with the setup otherwise identical to that during training (Qwen2.5-3B-Instruct, population size 30). This experiment gives the following results:

CountdownGSM8KProofWriterHellaSwagPIQAARC-CMMLU-ProAverage
-5.8 ± 5.2-15.9 ± 26.8+3.6 ± 6.9-5.9 ± 5.5-1.7 ± 0.5-2.4 ± 0.1-5.6 ± 2.4-4.8 ± 2.8

Thus, pure random walk shows similar forgetting as standard ES training, supporting the hypothesis that random walk is the explanation for the phenomenon. We will add this new experiment and further clarifications to the revision.

Note also two further insights about this experiment: First, the accuracy for ProofWriter improved in two runs. The task requires deciding whether some logical statement is true, false or undecidable given the information. Base models are often overconfident on this task, underusing undecidable. Moving a few samples to undecidable during random walk - empirically the most likely change to the model - often improves performance on this task compared to the base model. Second, there is a high variance on GSM8K. This task has strict format requirements for answer extraction, which the model starts to completely ignore in one seed. Under ES training this was not observed to happen, most likely because the target task always requires to follow a certain format rule, penalizing the model ignoring the formatting instructions in the prompt.

In addition to the above new experiment, another new experiment described below (in response to W2) also supports the random walk hypothesis. The step size α\alpha was reduced, thus reducing random walk (according to Eq. 2). As a result, forgetting was also reduced but often at the cost of target task performance. This result can be seen as additional evidence that random walk is the main cause of forgetting.

W2: Comparing AWD to alternatives

Three new experiments comparing to alternative regularization methods were run for this rebuttal:

Early stopping: Prior task forgetting starts relatively early on, e.g. -2.0% on ProofWriter after 25 iterations and -5.1% on GSM8K after 100 iterations, while target task accuracy is still far from the final performance. Furthermore, forgetting is not monotonic per task, making checkpoint selection difficult, and requires costly prior task evals. Early stopping is thus not a scalable remedy to forgetting under ES.

IterationCountdown (target)GSM8KProofWriterHellaSwagPIQAARC-CMMLU-ProAverage (excl. target)
25+6.8 ± 1.7-0.1 ± 0.4-2.0 ± 3.2+1.3 ± 0.8+0.1 ± 0.3+0.0 ± 1.0-0.3 ± 1.0-0.2 ± 0.4
50+17.8 ± 1.9+0.5 ± 1.3+1.0 ± 2.3+0.5 ± 1.2-0.2 ± 0.2+0.7 ± 0.7-1.8 ± 1.7+0.1 ± 0.2
75+27.6 ± 1.5-2.5 ± 4.7+2.6 ± 4.5-0.6 ± 0.6-0.3 ± 0.4-0.4 ± 0.8-0.9 ± 0.9-0.4 ± 1.4
100+32.6 ± 1.4-5.1 ± 10.6+2.5 ± 5.6-1.9 ± 1.2-0.7 ± 0.1-0.9 ± 0.8-0.5 ± 1.6-1.1 ± 2.8
200+39.9 ± 0.9+0.3 ± 1.9+2.8 ± 6.0-3.5 ± 3.6-2.0 ± 1.3-1.3 ± 0.0-2.1 ± 2.3-1.0 ± 1.3
300+43.0 ± 1.4-4.1 ± 4.5+6.8 ± 3.9-5.6 ± 3.5-2.0 ± 1.2-1.9 ± 1.2-3.0 ± 1.8-1.6 ± 1.6
400+44.1 ± 1.4-4.5 ± 5.1+3.5 ± 5.8-8.3 ± 5.7-2.9 ± 0.8-2.5 ± 0.4-6.0 ± 2.7-3.4 ± 2.6
500+45.3 ± 0.3-9.5 ± 11.5-2.0 ± 5.7-8.2 ± 9.0-2.8 ± 1.0-2.0 ± 0.5-6.5 ± 2.6-5.2 ± 4.7

Iterate averaging: Iterate averaging (with exponential moving average) can boost target task performance (β\beta = 0.9), but leads to similar levels of forgetting as not using iterate averaging (β\beta = 0). Higher β\beta gradually decreases forgetting, yet also reduces target task performance, striking a similar trade-off as early stopping.

β\betaCountdown (target)GSM8KProofWriterHellaSwagPIQAARC-CMMLU-ProAverage (excl. target)
0+45.3 ± 0.3-9.5 ± 11.5-2.0 ± 5.7-8.2 ± 9.0-2.8 ± 1.0-2.0 ± 0.5-6.5 ± 2.6-5.2 ± 4.7
0.9+46.5 ± 1.7−10.4 ± 13.6+2.9 ± 1.3−8.0 ± 8.5−4.8 ± 2.5−4.7 ± 1.1−5.6 ± 3.3−5.1 ± 0.1
0.99+45.5 ± 2.1−3.3 ± 4.9−0.7 ± 1.2−7.3 ± 9.0−3.3 ± 1.6−2.5 ± 0.1−4.1 ± 2.7−3.5 ± 1.2
0.995+44.4 ± 2.2−0.5 ± 2.7−0.7 ± 5.2−6.1 ± 6.6−2.1 ± 1.0−1.1 ± 0.5−2.6 ± 1.2−2.2 ± 0.1
0.997+41.9 ± 2.6+0.3 ± 3.0−1.1 ± 0.6−4.2 ± 4.8−1.7 ± 0.6−1.2 ± 1.2−1.7 ± 2.6−1.6 ± 0.5
0.999+19.2 ± 6.9+1.2 ± 2.0−2.0 ± 1.3−1.5 ± 1.3−0.6 ± 0.0−0.3 ± 0.8+0.2 ± 1.2−0.5 ± 0.3

Smaller step size α\alpha: As predicted by Eq. 2, reducing the step size shrinks the random-walk component and thereby forgetting. In this experiment α\alpha was reduced from 5e-4 (the value recommended in [2] and used throughout) to 3e-4 and 1e-4. For Qwen 2.5 3B Instruct, 3e-4 attains a similar tradeoff between target task performance and prior task forgetting as AWD, while 1e-4 severely reduces target task performance:

α\alphaCountdown (target)GSM8KProofWriterHellaSwagPIQAARC-CMMLU-ProAverage (excl. target)
5e-4+45.3 ± 0.3-9.5 ± 11.5-2.0 ± 5.7-8.2 ± 9.0-2.8 ± 1.0-2.0 ± 0.5-6.5 ± 2.6-5.2 ± 4.7
3e-4+44.6 ± 0.6-1.5 ± 3.0+2.9 ± 5.4-1.6 ± 2.7-0.9 ± 0.6-0.7 ± 0.6-3.0 ± 2.4-0.8 ± 0.6
1e-4+35.0 ± 0.7+0.4 ± 0.7+2.5 ± 0.3+0.9 ± 0.0+0.0 ± 0.4+0.0 ± 0.7+0.7 ± 0.3+0.7 ± 0.2
AWD (ℓ2\ell_2)+43.8 ± 0.7-2.5 ± 5.0+1.0 ± 6.8-0.7 ± 1.6-0.9 ± 0.2-0.5 ± 1.1-2.6 ± 0.9-1.0 ± 2.4

However, for Llama 3.2 3B Instruct, AWD improves on 3e-4 in prior task forgetting at matched target task performance, and gains 5.2 higher target task performance than 1e-4 at similar levels of forgetting:

α\alphaCountdown (target)GSM8KProofWriterHellaSwagPIQAARC-CMMLU-ProAverage (excl. target)
5e-4+46.9 ± 1.0-5.1 ± 3.4+3.9 ± 6.0-12.6 ± 2.0-3.3 ± 3.3-4.2 ± 1.1-5.4 ± 3.8-4.4 ± 1.7
3e-4+44.1 ± 0.6-2.9 ± 1.5-0.3 ± 2.8-3.0 ± 2.9-1.0 ± 2.8-1.8 ± 0.2-6.7 ± 1.4-2.6 ± 1.4
1e-4+39.4 ± 0.4+3.2 ± 1.8-0.0 ± 1.8-1.4 ± 0.5+0.7 ± 0.4-0.7 ± 0.3-2.4 ± 2.2-0.1 ± 0.7
AWD (ℓ2\ell_2)+44.6 ± 0.1+2.9 ± 2.1-1.7 ± 3.6-2.8 ± 0.7+1.6 ± 2.2-0.9 ± 0.6-3.5 ± 3.5-0.7 ± 1.0

However, tuning the step size is not a substitute for AWD for three reasons: (i) The step taken in the signal direction gg over all iterations is ≈αTg\approx \alpha T g, so decreasing α\alpha caps convergence speed. Matching target progress requires proportionally more iterations, trading forgetting for compute similar to increasing the population size. That α\alpha = 3e-4 performs well here is likely due to Countdown saturating after about 300 iterations, well below the 500 iteration budget (see Fig 1a). (ii) A smaller α\alpha only slows the random walk, it doesn’t stop it. Once the target task saturates (about 300 iterations on Countdown, Fig. 1a), the drift keeps growing with no target-task benefit. AWD instead pulls back every direction that the fitness signal does not actively sustain, so drift stays bounded however long training runs. α\alpha = 3e-4 looks competitive within the 500-iteration budget, yet the difference is expected to grow in longer runs (e.g. 1500–2500 iterations for ARC-AGI and Sudodu runs in [2]) and generally in continual learning. (iii) The step size is primarily an optimization hyperparameter. Using it to also trade off target performance against forgetting couples two concerns and could become prohibitive, e.g. under step-size schedules as [3] explored for more complex agentic tasks. AWD decouples them.

Therefore, while smaller step size is not a solution to forgetting, the experiment provided further insight into the underlying random-walk mechanism.

We will add these comparisons and insights into the revision, where they will strengthen the results. Thanks for suggesting them!

Q1: Cost advantage of AWD over larger populations

The cost advantage of AWD is substantial compared to larger populations as the computational cost of ES is linear in the population size. Thus ES(128) costs more than four times as much as ES(30) and ES(256) is more than eight times as expensive. So compared to ES(30), ES(30)+AWD adds 1-2%, ES(128) more than 300% and ES(256) more than 700% overhead. We added this clarification in the revision.

Indeed, the larger populations do give higher target accuracy on Countdown at a fixed number of iterations, yet at 4-8 times the cost.

Q2: Support for ES being promising for continual learning

The main finding of the paper is that prior-task forgetting under ES is largely avoidable: with AWD, prior-task performance remains stable at low computational cost, and ES itself is simple and scalable. This finding does not mean that ES solves continual learning, only that these properties position it as a well-suited basis for continual learning systems. We adapted the abstract, discussion and conclusion accordingly.

Nevertheless, a sequential learning task provides an insightful test of the method. In a new experiment, the model was fine tuned first on Countdown then ProofWriter and the reverse, applying AWD with the anchor set to the model from the previous stage. All values are changes relative to the start of each stage; bold marks the respective target task.

SettingCountdownGSM8KProofWriterHellaSwagPIQAARC-CMMLU-ProAverage (excl. target)
Base -> C+43.8 ± 0.7-2.5 ± 5.0+1.0 ± 6.8-0.7 ± 1.6-0.9 ± 0.2-0.5 ± 1.1-2.6 ± 0.9-1.0 ± 2.4
C -> P-3.2 ± 0.4+3.3 ± 4.5+32.0 ± 6.7-1.3 ± 1.3-0.5 ± 1.0-0.5 ± 2.6-0.7 ± 1.8-0.5 ± 1.4
Base -> P-2.3 ± 3.4+1.3 ± 0.6+34.4 ± 0.4-1.1 ± 3.1-0.9 ± 1.1-0.5 ± 0.2+0.5 ± 0.8-0.5 ± 0.6
P -> C+45.3 ± 3.7-0.2 ± 1.7-2.1 ± 2.9-2.1 ± 2.5-0.9 ± 0.5-1.0 ± 0.2-0.8 ± 1.1-1.2 ± 0.5

The second task is learned about as well as when training directly from the base model, and over 92% of the first task’s gain is retained (−3.2 of +43.8 on Countdown, −2.1 of +34.4 on ProofWriter). Changes on the tasks that are never trained stay small and do not scale across stages. Anchoring to the previous model is the simplest choice; for longer sequences and very different tasks, anchoring at the center of previous models may work better.

As control, the following results show the same experiment without AWD:

SettingCountdownGSM8KProofWriterHellaSwagPIQAARC-CMMLU-ProAverage (excl. target)
Base -> C+45.3 ± 0.3-9.5 ± 11.5-2.0 ± 5.7-8.2 ± 9.0-2.8 ± 1.0-2.0 ± 0.5-6.5 ± 2.6-5.2 ± 4.7
C -> P-14.2 ± 4.8+3.8 ± 11.3+36.3 ± 5.8-1.1 ± 4.2-3.2 ± 1.8-3.2 ± 2.0-1.8 ± 1.7-3.3 ± 3.0
Base -> P-0.6 ± 4.8-0.3 ± 0.3+35.6 ± 2.7-1.4 ± 0.2-1.9 ± 0.8-2.5 ± 1.6-0.7 ± 0.5-1.3 ± 0.8
P -> C+46.0 ± 5.9-5.9 ± 1.2-4.4 ± 2.8-3.2 ± 2.6-2.3 ± 0.4-2.4 ± 1.1-6.3 ± 2.1-4.1 ± 0.6

Compared to training with AWD, there is both increased overall forgetting for each stage, as well as stronger forgetting on the first task when training on the second task. This is especially pronounced for training on ProofWriter after Countdown. Without AWD, Countdown performance drops severely when training on ProofWriter (−14.2±4.8-14.2 ± 4.8 compared to $ -3.2 ± 0.4$). When training on Countdown after ProofWriter, this is less extreme in absolute numbers, but without AWD there is still twice as much ProofWriter forgetting (−4.4±2.8-4.4 ± 2.8 compared to −2.1±2.9-2.1 ± 2.9).

Overall, the experiment provides an intriguing possibility that AWD could extend to various continual learning settings in the future. We will add these results and the suggestion for future work in the revised version of the paper.


[1] Evolutionary Strategies lead to Catastrophic Forgetting in LLMs (2026) ArXiv

[2] Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning (2026) ICML

[3] Agentic ESOpt: Fine-Tuning Long-Horizon LLM Agents with Minimal GPU Requirements (2026) ArXiv

Comments on the paper

2 comments

We thank all reviewers for their careful and constructive reviews. The pointed out strengths of our work were encouraging, such as the empirical study of forgetting under ES being systematic and thorough across tasks, model families and sizes (keo1, muDo). Reviewers also found that tracking prior-task performance throughout training, and the recovery effects this reveals, adds insight beyond final-checkpoint evaluations (9urQ, muDo). Finally, all reviewers considered AWD a simple and practical mitigation that substantially improves prior task retention at little overhead (keo1, 9urQ, muDo).

Following the reviewers’ suggestions, several new experiments were conducted. The results are presented in the individual responses to each reviewer and will also be incorporated into an updated version of the paper to be uploaded early next week. Properly adding and discussing them in the paper will take a bit more time. We will post another message as soon as this version is available. Regarding clarifications and revisions, we already updated the respective sections in the paper (changes in green) and uploaded a new version to directly show the proposed changes.

Below is a summary of the major points. Please see each individual response to the corresponding reviewer for more details.

New experiments

  1. ES with random fitness values (keo1, 9urQ): To further investigate whether random walk can be the cause of forgetting, a random fitness function was applied, thus all weight updates are a random walk. This experiment shows forgetting comparable to regular ES training, supporting random walk as the cause of forgetting.
  2. Alternatives to AWD - early stopping, interpolation, iterate averaging and smaller step size (keo1, 9urQ): Early stopping, interpolation, iterate averaging cannot avoid forgetting to a similar degree as AWD without losing target-task performance. A smaller step size reduces forgetting, as Eq. 2 predicts, further supporting the random walk explanation, but AWD gives an equal or better tradeoff and has additional benefits (as discussed in the rebuttals to keo1 and 9urQ).
  3. Sequential multi-task training (keo1, 9urQ): The second task is learned as well as when training from the base model, and over 92% of the first task’s gain is retained.

Clarifications and revisions

  1. Main contributions (muDo, keo1): The main contribution of this work is the analysis of ES forgetting, the findings of which then motivate AWD as a simple mitigation strategy. An explicit list of contributions was added to Sec. 1.
  2. Positioning of AWD (9urQ, muDo): AWD is an adaptation of established anchoring methods to ES (necessary as there is no loss + backpropagation) and is motivated by the paper's empirical analysis, which suggests that random walk causes forgetting in ES training. Sec. 1 (contributions) and Sec. 5 were updated to more explicitly state this.
  3. Continual-learning claims (keo1, 9urQ): The abstract, discussion and conclusion were updated to more clearly present ES(+AWD) as a promising direction for continual learning, not as a full solution to it.
  4. Update norm vs. forgetting (keo1): The results point at random walk as the root cause and both performance drift and the growing norm as its effects. The norm alone can be misleading and may grow without necessarily leading to performance drift. It depends on which weights are most affected by the random walk. Sec. 6 was updated to make this more clear.
  5. Computational cost (keo1): AWD adds 1–2% overhead to ES(30). ES(128) adds more than 300% and ES(256) more than 700%. Sec. 7 was updated to explicitly state that comparison.
  6. ES vs. GRPO KL divergence (muDo): GRPO’s higher KL on the target task reflects sharper answer confidence. This result is consistent with recent work showing that ES preserves a more diverse answer distribution, which performs well under pass@k, whereas GRPO concentrates probability mass on a few high-reward answers, favoring pass@1. This discussion was added to Sec. 6.

We thank the reviewers again for their feedback, which allowed us to make the paper stronger. We look forward to answering any further questions during the discussion period.

The updated PDF incorporating all suggested changes and new results of this rebuttal is now available.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.