acceptodds
Withdrawn from ICLR 2027

Decoupling "What" and "Where": Cognitive Cascades for Class-Incremental Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Class-incremental semantic segmentation (CISS) demands preserving previously learned classes while incorporating new ones, yet conventional Softmax-based heads suffer from background shift and incomparable per-task logits. Through a Hessian-based analysis we show that while Strict Parameter Isolation (SPI) provides a parameter-level guarantee against catastrophic forgetting, its naive instantiation still fails end-to-end: the underlying probabilistic factorization entangles "what classes exist in the image" with "where each class lies." In this study, we propose Cognitive Cascade Segmentation (CogCaS), which explicitly decouples these two questions. Phase I (the "what") is a text-anchored multi-label classifier that detects which learned classes are present; Phase II (the "where") activates only the corresponding class-specific binary segmenters. A frozen vision-language text anchor stabilizes the classifier as the vocabulary grows, allowing new classes to be added without retraining any historical module. Under SPI, this decomposition cleanly separates two phenomena commonly conflated in the CISS literature: parameter-level forgetting (which CogCaS provably eliminates) and routing error from the existence classifier (an open-vocabulary recognition problem, orthogonal to forgetting and modularly improvable). On PASCAL VOC 2012 and ADE20K, CogCaS matches or surpasses state-of-the-art CISS methods under both short- and long-sequence protocols, with particularly strong gains in extreme regimes such as VOC 1-1 (20 sequential tasks), while keeping storage at 1.6–2.0X the backbone.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
continual semantic segmentation, class-incremental semantic segmentation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_58c4OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a new method for class-incremental semantic segmentation (CISS). The algorithm has two phases: Phase I (the “what”) learns a multi-label classifier that detects which classes are present; Phase II (the “where”) activates the corresponding class-specific binary segmenters. Part of the paper is dedicated to proving a parameter-level guarantee against catastrophic forgetting. The method obtains good results on standard CISS datasets.

Critical strengths
  • Breaking up the task in a 'what' part (which can suffer forgetting) and an isolated 'where' part that will not suffer forgetting seems a reasonable idea.
  • The paper reports good results on standard datasets PASCAL and ADE20K.
Critical weaknesses
  • The presentation of the paper is not good and needs to be improved significantly. Figure 1, does not show a new identified problem, the mathematical 3.1 and 3.2 (on zero-forgetting) are only of little importance to the method (and the reader is unsure about their purpose because they are not well introduced). It would be better to directly start with the method, and later use a (small part) of the math to support the working of the method (although I am yet to be convinced about any relation of the math with the method).

  • The results method should include newer works [a][b] also I think it would be important to show for all method on which backbone the method is based.

  • The method uses much larger network SigLIP2 than other methods (that is probably also why the paper also reports higher joint performance). It would be good to include results in the tables based on only the ResNet101.

[a] Continual Gaussian Mixture Distribution Modeling for Class Incremental Semantic Segmentation, NeurIPS 2025 [b] Learning Yourself: Class-Incremental Semantic Segmentation with Language-Inspired Bootstrapped Disentanglement, ICCV 2025

Questions for authors

I think the improvement to the presentation of the paper needs a new submission. The method obtains good results but comparison should be with more recent state-of-the-art and clearly state required pretrained networks (preferably in the Table). The main architectural idea is good and could lead to a publication in the future; the math requires a clearer link with the method.

See weaknesses.

  • Normally zero forgetting in parameter isolation methods is only obtained in task incremental methods. Can the authors explain why they can claim it ? The multi-label classifier would still suffer forgetting (new heads can obtain higher scores than previous ones on old data).

Minor

  • some of the citations are in the wrong format (cite vs citep)
  • also explain SPI the first time you use it in introduction. I am not sure if HAT is a strict parameter isolation method since it can have soft attention masks.
Reviewer_vce6OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

This paper focuses on class-incremental semantic segmentation, which requires learning new classes while preserving knowledge of previously learned classes. The authors propose Cognitive Cascades (CogCaS), a two-phase framework that decouples class identification from pixel-level localization. This paper introduced a text-anchored classifier to identify classes present in an image and class-specific binary segmentation heads to localize them.

Critical strengths
  1. The motivation of this paper is well-founded: freezing the parameters of previous tasks can prevent forgetting caused by parameter updates, but it does not guarantee that the output scores of segmentation heads trained on different tasks are directly comparable.

  2. The proposed method achieves significant performance gains on mainstream CISS benchmarks, such as VOC.

Critical weaknesses
  1. Scalability and model complexity: Each new category seems to require the introduction of additional parameters (class-specific binary segmentation heads). Does this mean that the model is difficult to scale to larger-scale scenarios or open-domain segmentation tasks?

  2. Generalization across pretrained backbone: Could the authors evaluate the framework with a strong pretrained representation, e.g., GenLI​​P [1*], or provide a more controlled comparison across pretrained encoders? Such experiments would help clarify whether the reported gains mainly arise from the proposed framework or depend on the specific pretrained representations used.

[1*] Let ViT Speak: Generative Language-Image Pre-training, ECCV 2026

Questions for authors

Oracle's experiment is quite valuable, but Oracle's results on VOC are similar to those on the full model, while Oracle's results on ADE are relatively low (43.0). Does this mean that the upper bound of this scheme is not high enough?

Reviewer_ykWMOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

AI Statement: I don't use AI for generating reviews - that is a abdication of duty, and I do not think LLM generation has a place in academic discourse - LLM generation in papers is plagiarism and poor scholarly practice.

Critical strengths

The paper tackles the problem of class incremental semantic segmentation. It incorporates a classifier that selects appropriate segmentation heads by what is in the image. It relies on a pretrained backbone and a pretrained vision language semantic embedding method. The latter allows a class semantic embedding to be defined - the classifier is comparing a current semantic embedding against the trained code book.

Critical weaknesses

As abstract continual learning learning problem-sets go, class-incremental semantic segmentation is more of a realistic challenge than some. But still the paper should make some clear statement as to why in a world a large vision language models this is still a relevant task. Many of these tasks can be done zero shot with a trained VLM and defining class embeddings on the fly. So there is not really a learning problem. Indeed in this paper most of the dynamic learning is finessed via a pretrained VLM and a backbone, so in what sense it actually targetting the inherent problem rather than just hacking the benchmark - class incremental semantic segmentation is no longer an interesting (in a continual learning sense) or a hard problem if I have learnt how to segment and learnt all possible semantic spaces ahead of time.

Recent VLM methods for in-context disentanglement take this conclusion (with methods reporting significantly higher performance than that achieved here) - I am not interested in benchmark chasing, but at least the relevance of the problem and approaches as cast should be established.

Mixture methods for selection of semantic heads is also an old business - a recent example is CoGaMiD in Neurips 2025, but this has been done in different models for this context over the last 5 years. I'm not sure what this adds beyond leveraging the pretrained semantic encoder to define the classifier via prototypes.

The title is not quite accurate - there is no decoupling of what and where - in fact they are fully entangled - the method relies on determining the what to ascertain the where.

The evaluations stop at 2025 and do not include recent state of the art methods.

Altogether, this seems like work that is doing something that might be considered "obvious", and it is not clear to me what academic insight I gain from the work as it stands. This is an academic conference and descriptions of some way of putting bits together to target a particular benchmark task does not provide substantial academic insight of itself, and I feel this is where this paper sits.

Questions for authors

No questions

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.