Input Anchoring Makes Your Model a Deep Analytic Continual Learner

In class-incremental learning, models attempt to learn from a stream of tasks while having limited storage capacity. The strongest guarantee a continual learner can make is `absolute memorization' which, beyond simply avoiding catastrophic forgetting, promises to produce a model identical to its full replay counterpart without needing the original dataset. Recent work in Analytic Class Incremental Learning (ACIL) has demonstrated that linear layers with input dimension $d$ can be made into absolute memorizers with storage complexity $O(d^2)$. A variety of approaches leverage this by applying a wide linear layer atop large frozen pre-trained models. While ACIL methods achieve absolute memorization in linear regression models, no work has yet achieved absolute memorization in deep neural networks. In this paper, we define Input Anchoring and prove that degree-$L$ deep neural networks with this property achieve absolute memorization with storage complexity $O((d+1{)}^{2L})$. We demonstrate that $\Pi$-nets and GaLU networks have the input-anchoring property with degree scaling linearly in their depth, and introduce our own simple and performant input-anchored architecture: the Anchored Multi-Layer Perceptron (AMLP). We show that the benefits of absolute memorization can be achieved with polynomial storage and time complexity via our Loss-Anchoring Points (LAP) approximation method, and leverage this finding to achieve state-of-the-art performance over the latest ACIL methods at matched storage and parameter counts on pre-trained model class-incremental benchmarks.