Preserving Plasticity in Continual Robot Learning from Pixels with Stiefel-Constrained Optimization

A robotic agent that continually learns new tasks from pixel-based inputs must maintain the plasticity of deep neural networks that process high-dimensional observations. We show that existing methods based on regularizing neural network parameters to mitigate loss of plasticity remain insufficient in the continual visual-control settings. We show that changing the optimizer from AdamW to Skewon (Solonko et al. 2026) mitigates the loss of plasticity. The Skewon optimizer constrains the parameter matrices of the deep neural network to the Stiefel manifold, meaning their columns are orthonormal. We show that this optimizer does so by preserving the input–output Jacobian norm alongside sustained learning. We demonstrate continual learning from images for up to 100 million frames for locomotion tasks using the proximal policy optimization algorithm and Skewon. We further demonstrate its efficacy in simulated manipulation tasks. We introduce a new optimizer: Gram-Skewon, which preserves pretrained weight Gram matrices, which is applied to continual behavior cloning for manipulation tasks using the SO-101 robotic arm in the physical world and outperforms AdamW. Gram-Skewon achieves 62% success versus 37% for AdamW in the real-world task chain.