Fidelity for Rehearsal, Coverage for Calibration: Role-Decoupled Memory for Class-Incremental Learning

In Class-Incremental Learning (CIL), limited replay can preserve separable features for historical classes, yet provide insufficient distribution coverage to align the classifier with the historical class distributions in feature space. We observe that representation rehearsal and classifier calibration have different input fidelity requirements: with backbone frozen, calibration can trade input detail for broader historical coverage, provided that compression retains information relevant to classification in feature space. To this end, we propose a role-decoupled memory framework: using high-fidelity exemplars for representation learning, and compressed low-fidelity samples exclusively for subsequent classifier calibration. Drawing on the concept of Average Gradient Outer Product (AGOP), we propose two novel compression schemes, AGOP-PCA and AGOP-guided transform coding, to achieve broad feature coverage and preserve information useful for classification. Experiments across diverse CIL benchmarks show consistent improvements over strong replay, compressed replay and classifier-alignment baselines.