- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper demonstrates that scaling the architecture depth of (off and on-policy) actor-critic algorithms enables better generalization in out-of-distribution goal-conditioned RL.
Prior work on scaling contrastive reinforcement learning (CRL) algorithm showed that scaling matters for CRL when considering in-distribution training. This paper extends that line of work to other algorithms, showing that scaling doesn't help when goals are sampled in-distribution, but as goals are becoming more OOD, scaling improves generalization and therefore downstream performance.
Critical strengths
- The paper presents a useful and relatively surprising result, which can in the future enable future research on scaling and generalization in reinforcement learning.
- The sparse auto-encoder analysis is novel and insightful.
- Comparing generalization across and network depth and width is a very useful result.
Critical weaknesses
- The result is purely empirical, however the paper uses only four reinforcement learning environments to support its claims.
- More generally some claims can be made more precise: generalization here is only in the context of goal-conditioned RL, not for instance in the context of offline RL or sim-to-real. The "horizon generalization" framing in the title is not as deep as the title may suggest. The above two points are discussed in the limitations.
- The invariant representation is done only using SAC and Humanoid.
- The description of Section 4.6 can be made clearer. For instance, terms like: "cumulative greedy ablation" tend to be somewhat vague.
Questions for authors
Minor comments:
- You can cite BRO and Simba when presenting them in line 99.
- When generating new OOD goals, how did you validate that these goals are sound and that there is a policy that can actually solve them?
I'm happy to increase my score if claims about generalization and "horizon generalization" are made more precise.
Or, even better, if these results extend beyond these four environments and not only within the context of goal-conditioned RL.