acceptodds
Under review as a conference paper at ICLR 2027

Continual Reinforcement Learning with Neuroevolution

Abstract

Despite many studies about causes and remedies of plasticity loss in Reinforcement Learning (RL) under continual task changes, no RL method has yet consistently achieved a good balance between adaptation and forgetting. Here we turn to an alternative optimization paradigm, neuroevolution (NE): algorithms, such as evolution strategies (ES) and genetic algorithms (GAs), that search directly in weight space through mutation and selection over a population of neural networks. We compare ES and a GA against state-of-the-art continual RL variants and population-based RL, with no method informed of a task switch, across different kinds of environments and environmental changes and policies ranging from a few hundred parameters to million-parameter networks. We find that ES most consistently achieves a good stability-plasticity trade-off, while the GA is the most plastic method. To explain this, we study the return landscape around each method's solutions. We observe that ES finds the widest neighborhoods, regions of weight space in which perturbed policies still solve the task, and that the size of the overlap between the neighborhoods of consecutive tasks correlates with the stability-plasticity trade-off of a method. The GA has narrower neighborhoods and moves further per task, so it is more plastic but forgets more. Rewarding behavioral diversity in a GA through novelty search makes the population even more plastic at the cost of forgetting. Finally, symptoms of plasticity loss commonly reported in RL do not transfer to NE suggesting that they are specific to gradient-based optimization rather than non-stationarity. Overall, these results establish NE as a competitive alternative to RL under continual task changes, and suggest that training under perturbations in weight space may be a useful mechanism for continual learning more broadly.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
neuroevolution, evolution strategies, genetic algorithms, continual reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4mwgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper compares Neuroevolution (NE) methods against Reinforcement Learning (RL) algorithms and Population Based Training (PBT) variants on continual RL problems. In this setting, long-lived agents face changing environments (in this work: observation space, action space, or dynamics), good neural policy candidates should show plasticity (to be able to adapt to the updated task) and stability (to avoid forgetting their previous behavior). The paper compares NE vs RL methods on this plasticity/stability tradeoff and conclude that NE methods typically perform better under those criteria even though the stationary RL policy on a given separate task can be better in average. The paper suggests that this is explained by the NE methods discovering weights with "wide neighborhoods", i.e. a region of the parameter space where perturbating the parameters still solves the task.The main contribution of the paper is the empirical evaluations and the analysis of weight space properties.

Critical strengths
  • The paper is clearly-motivated and easy to follow. It was a pleasant read.

  • I think the paper makes a point in highlighting some positive properties (wide neighborhoods, more plasticity, less forgetting) of NE methods. SGD is the dominant paradigm, but it comes with some drawbacks and zero-order optimization methods have their strengths.

  • The empirical evaluation is thorough, I think the analysis in parameter space/return landscape makes sense. And I like the fact there is a discussion in the appendix about the fact the weight perturbation analysis does not take into account the geometry of SGD.

Critical weaknesses
  • I think the actual results, shown in the figures (Figure 2 and 3 for instance), appear more nuanced than the interpretation of it developed in the main text . I found myself going back and forth a lot between the figures and the text to try to find out if I had the same interpretation. A thing that doesn't help: the text says a lot "on X out of Y tasks, this method is the best". I think this loses a lot of information (such as "how much better is it") and I think having instead cross-environment, cross-task metric aggregates would really help.

  • I think the main text misses important details. Axes in Figure 1, metrics scaling in Figure 2, number of seeds for all experiments, compute budget in terms of number of NN calls and number of env steps.

  • This is not a critical weakness, but a lot of the details are in the appendix, that is not included in the submission pdf, which makes it a pain to go back and forth finding the info, since references to appendix can't be followed.

Questions for authors
  • Can you provide the missing details mentioned in the "Critical Weakness" section?

  • How was selected the time duration of tasks? Would it help letting more time to the methods to recover?

  • Would average performance across the episode be a valid metric? If I understand correctly you only looked at end of task/end of episode return.

Reviewer_1XTnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies continual RL from the perspective of neuroevolution (NE). It compares ES and a GA with PPO, three continual RL variants (TRAC, ReDo, and C-CHAIN), and PBT-PPO in a boundary-free setting, on classic control, MiniGrid, Brax HalfCheetah, Kinetix, and DeepSea with observation offsets, dynamics changes, and action reversal. The results show that ES gives the best stability-plasticity trade-off, while the GA is the most plastic. The authors further analyze the return landscape and argue that ES finds the widest neighborhoods, and that the overlap between the neighborhoods of consecutive tasks correlates with the trade-off. They also show that novelty search makes the GA more plastic, and that dormant neurons and weight drift do not appear in NE.

Critical strengths
  1. The paper studies an interesting question. Plasticity loss is a known problem in continual RL, and comparing gradient-based RL with NE under the same boundary-free setting gives a different view on it.
  2. The experiments are reasonably broad. The paper covers several environments and types of task change, uses both task sequences and alternating tasks, and reports 10 trials per setting with confidence intervals and statistical tests. The code is also released.
  3. The analysis is easy to follow. The neighborhood, diversity, and dormant neuron analyses give intuitive explanations for the differences between methods.
Critical weaknesses
  • The baseline set is not fully aligned with the paper's main claim. The paper attributes the advantage of NE to perturbation in weight space and leaves weight noise for PPO as future work, but this is the most direct control. PPO with parameter noise and methods such as Shrink & Perturb, Continual Backprop, and Plasticity Injection should be compared, especially since Juliani & Ash (2024), cited in the paper, find that a class of regenerative methods consistently reduces plasticity loss in on-policy RL. Also, all RL baselines target plasticity only, so the conclusion that RL forgets more is not well supported.
  • The novelty is somewhat limited, and some closely related work is missing. Several of its findings are already known. Lehman et al. (2018) showed that ES finds wider and more robust solutions, and the effect of population diversity under changing environments has long been studied in evolutionary dynamic optimization. The shared neighborhood is also closely related to linear mode connectivity in continual learning (Mirzadeh et al., 2021). Other evolutionary methods for continual RL, such as Nevo-CRL, and hybrids of evolution and RL, such as ERL and CEM-RL, are not discussed either. Nevo-CRL assumes known task identity, but it targets the same stability-plasticity problem.
  • The experimental setting is not convincing enough. Only 5 of the 18 settings use 20-task sequences, 4 of which only add observation offsets, and apart from Kinetix, dynamics changes and action reversal are only tested with two alternating tasks. Most environments are classic control tasks with small networks, and standard benchmarks such as Continual World are not used. The main metric LA - F is simply the return on each task at the end of the run, so it can favor methods that learn less and forget less, as on the HalfCheetah sequence. Also, ES has a significantly better trade-off than all RL methods in only 7 of the 18 settings.
  • The analysis is mostly correlational. The link between the shared neighborhood and the trade-off is based on correlations across methods and settings (Figure 4), without any intervention. For example, the ES search width is already varied in Appendix D, and it would help to show whether the shared neighborhood and the trade-off change together. Since the authors also state that a large shared neighborhood is not necessary, the claim that parameter-space exploration is the mechanism behind the trade-off seems too strong. In addition, novelty search only helps the best individual rather than the centroid, and the symptom analysis only covers gradient-specific effects such as dormant neurons.
Questions for authors
  1. Can the authors compare with PPO with parameter noise, Shrink & Perturb, Continual Backprop, or a SAM-based method such as PLASTIC? Does the advantage of NE still hold in this case?
  2. Can the authors add a baseline for forgetting that does not need task boundaries, such as CLEAR, and a reference that trains a fresh network on each task? Do the conclusions also hold for off-policy methods such as SAC or DQN?
  3. What is new compared with prior work on population diversity in evolutionary dynamic optimization, ES in changing environments (e.g., IW-IES), and evolutionary methods for continual RL such as Nevo-CRL?
  4. Under action reversal, does the policy have memory or any task-related input? If not, how can a single policy solve both tasks without forgetting?
  5. How is compute matched between NE and PPO? Were the hyperparameters of the continual RL variants tuned in the non-stationary setting?
  6. Can the authors test the role of the shared neighborhood more directly, for example by varying the ES search width? Why does novelty search help the best individual but not the centroid?
Reviewer_RQwmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper compares Evolution Strategies (ES) and Genetic Algorithms (GA) with state-of-the-art continual reinforcement learning methods and population-based reinforcement learning methods across various environments and types of environmental changes. Based on these comparisons, the authors analyze the strengths and weaknesses of ES and GA. The results suggest that neuroevolution (NE) can be a competitive alternative to reinforcement learning (RL) in continual learning settings.

Critical strengths
  • The paper addresses an important research question by investigating plasticity loss in continual reinforcement learning and exploring neuroevolution as an alternative approach.

  • The authors evaluate different methods across various reinforcement learning tasks, including classic control, sparse-reward navigation, locomotion, and pixel-based physics puzzles.

Critical weaknesses
  • The neuroevolution experiments mainly focus on ES and GA. Comparisons with other recent neuroevolution methods are lacking.

  • Neuroevolution methods typically require evaluating a large number of candidate policies, leading to high sample complexity and computational costs. The paper does not sufficiently discuss how these limitations affect the practical applicability of ES and GA in continual reinforcement learning.

Questions for authors
  • How sensitive are the main findings to the hyperparameter settings of ES and GA? Could different settings change their relative performance?

  • How do ES and GA compare with state-of-the-art continual RL methods in terms of sample efficiency and computational cost?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.