Brain-Inspired Efficient Coding Improves Continual Learning in Small Networks

Humans and animals continually learn new tasks and stimuli, but artificial neural networks often struggle to do so without disrupting prior knowledge. A successful continual learning (CL) approach must preserve what has already been learned while using its representational capacity efficiently enough to support future learning. Inspired by efficient coding in information theory and sensory neuroscience, which proposes that neural systems balance response diversity against metabolic cost, we introduce the Efficient Coding Regularizer (ECR), which penalizes response energy while encouraging diverse, high-effective-rank representations. We then study ECR across CL problems of increasing complexity. We first consider domain-incremental learning, where labels remain fixed as the input distribution changes. On permuted MNIST, ECR improves both acquisition of new tasks and retention of previous ones, reduces the rate of task-learning collapse by up to 19.3 percentage points, requires approximately 26% fewer epochs per task, and maintains up to $4.5\times$ the representation diversity (effective rank) of standard training. On CIFAR-100, ECR continues to preserve representational diversity in the corrupted setting but yields smaller and less consistent accuracy gains, and in class-incremental learning, it alone does not prevent forgetting. We therefore develop a nullspace-based approach that protects previous classifiers from destructive updates, providing a foundation on which ECR can act. In this context, applying ECR to the task heads constrains their response scale and reduces global forgetting while keeping accuracy comparable. Together, these results show that efficient-coding principles can preserve representational capacity and regulate network responses, with the clearest benefits in smaller networks and simpler tasks. In larger networks and more complex tasks, efficient coding works best when paired with explicit constraints. Beyond machine learning, these findings inspire neuroscience by suggesting that efficient coding operates most effectively within smaller neural circuits, such as cortical columns, a possibility that remains to be tested.