Past as Guidance, Not Just Constraint: Cross-Level Knowledge Reorganization for Class-Incremental Semantic Segmentation

Class-incremental semantic segmentation (CISS) requires a model to learn new classes while retaining acquired knowledge without full access to historical training data. Existing regularization strategies alleviate catastrophic forgetting by preserving knowledge from previous steps. However, under fixed model capacity, newly introduced classes alter the optimization objective and require the representation to be reorganized. Our optimization analysis shows that adjacent incremental objectives generally correspond to different stationary states, while inherited constraints can restrict current-task-aligned adaptation. Insufficient representation reorganization can consequently increase class distribution overlap. We further empirically reveal a cross-level representation propagation pathway through controlled perturbation analysis: response-unit variations are systematically reflected in downstream feature variations, which are further associated with changes in class-conditional distributions. Together, the optimization analysis provides a rationale for representation reorganization as incremental objectives evolve, while the empirical evidence supports coordinating this reorganization across representation levels. Motivated by these findings, we propose Adaptive Knowledge Inheritance (AKI), a framework that coordinates knowledge reorganization across the response, feature, and distribution levels. Response Capacity Adaptation (RCA) constructs a compact response support and enforces prediction consistency between the complete-response and compact-support branches, reducing excessive dependence on the complete response space. Reusable Projection Alignment (RPA) preserves historical semantics along reusable directions identified from historical discriminability and observed current adaptation. Distribution Entanglement Penalization (DEP) models current old- and new-class distributions and suppresses their pairwise overlap. Extensive experiments on Pascal VOC and ADE20K across nine incremental settings validate the effectiveness of AKI and the complementary roles of its components in balancing historical preservation and current adaptation.