Not More Parameters, Better Organization: Structured Compression Memory Network for Class-Incremental Learning

Class-incremental learning (CIL) aims to learn new classes sequentially while preserving discriminability over previously seen categories. While fine-tuning pre-trained models (PTMs) yields stronger generalization than training from scratch, existing approaches suffer from an inherent trade-off between parameter overhead and feature regulation. Expandable architectures incur steady parameter inflation as tasks grow, whereas fixed-parameter alternatives struggle to maintain reliable recognition of historical classes. To address this dilemma, we propose the Structured Compression Memory Network (SCMNet), a rehearsal-free, fixed-parameter framework built upon PTMs. Instead of growing parameters with tasks, SCMNet spends a fixed budget on organizing the feature and prototype spaces. Specifically, dynamic neighborhood compression constructs label-constrained batch graphs, suppressing intra-class variance while preserving inter-class separability. The adaptive prototype constraint uses stored prototypes as anchors to stabilize historical decision boundaries. These two modules form a mutually reinforcing loop to mitigate performance degradation over long task sequences. Evaluations across six benchmarks and three pre-trained backbones demonstrate that SCMNet surpasses state-of-the-art methods with only a fixed budget of $\sim$1.7M trainable parameters, validating the benefits of our structured feature organization strategy. Our code is included in the supplementary materials and will be released publicly upon acceptance.