Decoupling "What" and "Where": Cognitive Cascades for Class-Incremental Segmentation

Class-incremental semantic segmentation (CISS) demands preserving previously learned classes while incorporating new ones, yet conventional Softmax-based heads suffer from background shift and incomparable per-task logits. Through a Hessian-based analysis we show that while Strict Parameter Isolation (SPI) provides a parameter-level guarantee against catastrophic forgetting, its naive instantiation still fails end-to-end: the underlying probabilistic factorization entangles "what classes exist in the image" with "where each class lies." In this study, we propose Cognitive Cascade Segmentation (CogCaS), which explicitly decouples these two questions. Phase I (the "what") is a text-anchored multi-label classifier that detects which learned classes are present; Phase II (the "where") activates only the corresponding class-specific binary segmenters. A frozen vision-language text anchor stabilizes the classifier as the vocabulary grows, allowing new classes to be added without retraining any historical module. Under SPI, this decomposition cleanly separates two phenomena commonly conflated in the CISS literature: parameter-level forgetting (which CogCaS provably eliminates) and routing error from the existence classifier (an open-vocabulary recognition problem, orthogonal to forgetting and modularly improvable). On PASCAL VOC 2012 and ADE20K, CogCaS matches or surpasses state-of-the-art CISS methods under both short- and long-sequence protocols, with particularly strong gains in extreme regimes such as VOC 1-1 (20 sequential tasks), while keeping storage at 1.6–2.0X the backbone.