Affine Compensation and Recoverable Feature Learning for Exemplar-Free Class-Incremental Learning

Exemplar-free class-incremental learning must preserve past knowledge while adapting representations to new classes. Class means and covariances offer a compact memory, but keeping them aligned with an evolving feature space is difficult without old images. Our key observation is that matching these low-order moments leaves more freedom than aligning individual features. This motivates a single affine map shared across classes, fitted on current-task feature pairs to update stored statistics in closed form. Our analysis characterizes affine moment matching and bounds a shared map's old-class errors in terms of feature drift and cross-class response differences. Building on this foundation, we propose Recoverable Feature Learning with Affine Compensation (ReFAC), which combines this memory update with flexible representation learning. ReFAC trains a decoder to recover previous features and their within-class covariances, preserving recoverable information while the representation adapts to new classes. Replacing the statistics updates of four representative learners with the shared affine map largely preserves or improves final accuracy while reducing update cost. Extensive experiments on standard benchmarks show that ReFAC consistently achieves competitive or superior performance compared with state-of-the-art methods.