Why Subspace Protection Needs a Fixed Rank and Neuron Recycling in Task-Free Continual Learning

Neural networks trained on long task-free streams suffer from two failure modes known as loss of plasticity and catastrophic forgetting. In practice both failures can co-occur and a method commonly used against one can worsen the other, yet few studies address them together. Here we design task-free streams for supervised and reinforcement learning in which tasks recur after long and variable gaps. On these streams we study how subspace protection against forgetting interacts with neuron recycling against loss of plasticity. We find that protection whose subspace continues to grow over the stream eventually saturates and reduces plasticity even when recycling keeps almost every neuron active. A fixed rank avoids saturation, but our analysis identifies a second cost. Protection aligned with the mean input direction of a layer can suppress one of the main pathways through which dormant neurons reactivate. Removing this direction from the protected subspace restores reactivation in our experiments as the analysis predicts. Recycling therefore remains necessary even at a fixed rank. We further show that protecting layers whose input directions are less shared across tasks lowers plasticity on both streams. These results suggest that continually learning neural networks may need a bound on the protected rank along with neuron recycling and a careful choice of the protected layers. They also indicate that studying forgetting and loss of plasticity together may be important for practical continual learning algorithms.