- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper compares Neuroevolution (NE) methods against Reinforcement Learning (RL) algorithms and Population Based Training (PBT) variants on continual RL problems. In this setting, long-lived agents face changing environments (in this work: observation space, action space, or dynamics), good neural policy candidates should show plasticity (to be able to adapt to the updated task) and stability (to avoid forgetting their previous behavior). The paper compares NE vs RL methods on this plasticity/stability tradeoff and conclude that NE methods typically perform better under those criteria even though the stationary RL policy on a given separate task can be better in average. The paper suggests that this is explained by the NE methods discovering weights with "wide neighborhoods", i.e. a region of the parameter space where perturbating the parameters still solves the task.The main contribution of the paper is the empirical evaluations and the analysis of weight space properties.
Critical strengths
-
The paper is clearly-motivated and easy to follow. It was a pleasant read.
-
I think the paper makes a point in highlighting some positive properties (wide neighborhoods, more plasticity, less forgetting) of NE methods. SGD is the dominant paradigm, but it comes with some drawbacks and zero-order optimization methods have their strengths.
-
The empirical evaluation is thorough, I think the analysis in parameter space/return landscape makes sense. And I like the fact there is a discussion in the appendix about the fact the weight perturbation analysis does not take into account the geometry of SGD.
Critical weaknesses
-
I think the actual results, shown in the figures (Figure 2 and 3 for instance), appear more nuanced than the interpretation of it developed in the main text . I found myself going back and forth a lot between the figures and the text to try to find out if I had the same interpretation. A thing that doesn't help: the text says a lot "on X out of Y tasks, this method is the best". I think this loses a lot of information (such as "how much better is it") and I think having instead cross-environment, cross-task metric aggregates would really help.
-
I think the main text misses important details. Axes in Figure 1, metrics scaling in Figure 2, number of seeds for all experiments, compute budget in terms of number of NN calls and number of env steps.
-
This is not a critical weakness, but a lot of the details are in the appendix, that is not included in the submission pdf, which makes it a pain to go back and forth finding the info, since references to appendix can't be followed.
Questions for authors
-
Can you provide the missing details mentioned in the "Critical Weakness" section?
-
How was selected the time duration of tasks? Would it help letting more time to the methods to recover?
-
Would average performance across the episode be a valid metric? If I understand correctly you only looked at end of task/end of episode return.