Maintain Your Ranks: Enabling Continual Learning in RL Agents with Growing and Pruning

The learning capacity of neural networks in deep reinforcement learning (RL) is essential for extracting task-specific representations, thereby facilitating learning. However, networks' capacity can be lost during training with standard gradient optimizers; timely intervention is needed to provide new capacity or release the underutilized capacity to cope with the non-stationary data distribution in RL. Despite the evolving data distribution, common RL practices overly commit to static network architectures, which can suffer from plasticity loss in the sense of lacking adaptability in learning, an inherent trait that can be captured by their collapsing effective rank (e-rank). We argue that maintaining a sufficient e-rank of the neural networks' layers is a necessary precondition for plasticity. To mitigate plasticity loss, we suggest a dynamic adaptation of the architecture, aiming to maintain the e-rank, by growing and pruning the neurons of a layer. We demonstrate that RL agents with adaptively growing-and-pruning critics can continually learn in both standard continuous control RL and continual learning for sequential RL tasks with superior performance.