Are We Forgetting or Just Confused?

Class incremental learning enables models to continuously learn new categories sequentially. Nevertheless, mainstream incremental learning approaches follow a disjoint stage-wise training pipeline and inevitably suffer from severe inter-stage class confusion across different learning phases. To address this issue, this paper presents a Cross-stage Confusion Suppression (CCS) framework built upon pre-trained vision models. The framework first develops a Confusion-Aware Connected Component Generation module, which leverages the Union-Find algorithm to dynamically cluster categories with high inter-stage confusion. On this basis, a Connected Component-Guided Differential Adapter mechanism is proposed to alleviate feature confusion induced by isolated task-specific feature spaces. In particular, we learn a shared adapter to capture common feature patterns within each confused class group, and further subtract the shared feature response from task-specific adapters to obtain more discriminative incremental representations. Moreover, to cope with the inherent unavailability of historical data in exemplar-free incremental learning, we develop a Noise-Driven Pseudo-Sample Generation strategy to compensate for the missing prior category information. Finally, we adopt a two-stage inference procedure to adaptively activate differential adapters during testing, which effectively bridges the gap between training and inference. Extensive experiments on four standard benchmarks verify that the proposed CCS method significantly reduces inter-stage category confusion and achieves superior performance in both classification accuracy and forgetting mitigation. Code will be made publicly available.