acceptodds
Under review as a conference paper at ICLR 2027

Past as Guidance, Not Just Constraint: Cross-Level Knowledge Reorganization for Class-Incremental Semantic Segmentation

Abstract

Class-incremental semantic segmentation (CISS) requires a model to learn new classes while retaining acquired knowledge without full access to historical training data. Existing regularization strategies alleviate catastrophic forgetting by preserving knowledge from previous steps. However, under fixed model capacity, newly introduced classes alter the optimization objective and require the representation to be reorganized. Our optimization analysis shows that adjacent incremental objectives generally correspond to different stationary states, while inherited constraints can restrict current-task-aligned adaptation. Insufficient representation reorganization can consequently increase class distribution overlap. We further empirically reveal a cross-level representation propagation pathway through controlled perturbation analysis: response-unit variations are systematically reflected in downstream feature variations, which are further associated with changes in class-conditional distributions. Together, the optimization analysis provides a rationale for representation reorganization as incremental objectives evolve, while the empirical evidence supports coordinating this reorganization across representation levels. Motivated by these findings, we propose Adaptive Knowledge Inheritance (AKI), a framework that coordinates knowledge reorganization across the response, feature, and distribution levels. Response Capacity Adaptation (RCA) constructs a compact response support and enforces prediction consistency between the complete-response and compact-support branches, reducing excessive dependence on the complete response space. Reusable Projection Alignment (RPA) preserves historical semantics along reusable directions identified from historical discriminability and observed current adaptation. Distribution Entanglement Penalization (DEP) models current old- and new-class distributions and suppresses their pairwise overlap. Extensive experiments on Pascal VOC and ADE20K across nine incremental settings validate the effectiveness of AKI and the complementary roles of its components in balancing historical preservation and current adaptation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Class-incremental semantic segmentation, Continual learning, Incremental Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_1HJCOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies class-incremental semantic segmentation without replaying samples from previous tasks. It proposes AKI, a cross-layer knowledge reorganization framework, to mitigate forgetting when learning new classes. AKI comprises three components: RCA uses weight norms to construct response masks and encourages consistent predictions between the full and compact branches; RPA derives class-specific rank-one feature-alignment directions by considering both previously learned knowledge and the needs of the current task; and DEP models class distributions to encourage separation between old and new classes.

Critical strengths
  1. The problem is important, and the overall design is easy to follow. RCA, RPA, and DEP target forgetting at the response, representation, and class-distribution levels, respectively. The three components appear to play complementary roles.
  2. The evaluation is fairly extensive. Besides the main results and ablations, the appendix covers the disjoint setting, different backbones, and compatibility with other frameworks. Table 10 also includes random rank-one and simplified-direction controls, which help clarify the role of RPA.
  3. The derivation of RPA is presented in reasonable detail. The use of a truncated pseudoinverse in the effective feature subspace is explained, and the assumptions and limitations of the analysis are discussed. This gives more support to the projection design than simply introducing it as a heuristic.
Critical weaknesses
  1. The comparability of the baseline results needs clarification. The paper uses MBS† for comparison, but its differences from the original MBS are not sufficiently clear. If the architecture, training budget, or pseudo-labeling strategy has changed, the impact of those changes should be reported. Otherwise, it is difficult to attribute the improvement to AKI itself.
  2. The random-mask control for RCA does not isolate the effect of mask selection. According to Appendix C.9, the random mask stays fixed throughout an incremental step, whereas the norm-based mask in RCA is updated during training. Moreover, the number of suppressed responses is matched only at the beginning of the step. The differences in Table 13 may therefore reflect both mask selection and the update strategy. A dynamically updated random mask and a fixed norm-based mask, with layerwise mask counts matched, would make the comparison more informative.
  3. The actual optimization signal from DEP is not yet clear. DEP computes the Bhattacharyya coefficient in a 768-dimensional feature space. Although working in the log domain helps with intermediate numerical stability, the exponentiated coefficient and its gradients may still be very small. Reporting the range of the DEP loss and its gradient magnitudes during training would help establish whether it provides a meaningful learning signal.
  4. The independent contribution of RPA needs stronger evidence. SPPA, which is already cited in the paper, also uses low-dimensional projection. The distinction between the two methods and the claimed advantage of RPA over a learned projection should be made clearer. The random rank-one control in Table 10 is useful, but it is not a direct comparison with learned projection methods. In addition, Table 9 reports 79.3 for full AKI and 79.2 for RCA+DEP on VOC 15-1, a gain of only 0.1 mIoU. Results over multiple runs, including the mean and standard deviation, would help determine whether this difference exceeds normal experimental variation.
Questions for authors
  1. What exactly differs between MBS† and the original MBS? How much does each change affect performance? Does AKI yield a comparable gain when the original MBS training configuration is kept unchanged?
  2. If the random mask is updated during training and the number of suppressed responses is matched to RCA at each layer throughout training, does the conclusion in Table 13 still hold? Conversely, what happens if the norm-based RCA mask is kept fixed?
  3. How does RPA differ from learned projection methods such as SPPA in its objective, projection rank, and the feature layers to which it is applied? Do the current ablations adequately support its independent contribution?
1 comment

Thanks for the author's detailed rebuttal and additional experiments. I retain the initial rating.

Reviewer_y1tDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Adaptive Knowledge Inheritance (AKI) for replay-free class-incremental semantic segmentation with an unchanged shared architecture. RCA encourages prediction consistency between complete and compact response supports. RPA preserves historical semantics along class-specific directions selected using historical discriminability and observed current adaptation. DEP penalizes overlap between old- and new-class distributions in the current feature space. Conditional optimization observations and a controlled perturbation analysis motivate the design. Experiments cover five Pascal VOC and four ADE20K protocols, with component ablations, matched-rank and gradient controls, disjoint evaluation, backbone compatibility, repeated runs, and efficiency measurements.

Critical strengths
  1. The framework provides a useful way to balance historical preservation and current adaptation. RPA is particularly compelling: its explicit selection criterion preserves discriminative old-class directions while accounting for current feature changes. Together with RCA and DEP, this gives concrete mechanisms for reorganizing knowledge without increasing deployment complexity.

  2. The empirical validation is substantial and investigates the proposed mechanisms beyond final benchmark scores. Tables 3 and 9 test the component combinations; Table 10 compares matched rank-one alternatives and ablates the direction criterion; Table 11 examines parameter-gradient compatibility at a common checkpoint with matched inheritance-update magnitude. These controls support benefits beyond simply weakening feature matching. The additional protocols, repeated runs, and runtime measurements further strengthen the practical evidence.

Critical weaknesses

The main remaining reservation is the interpretation of improvement over MBS. On VOC 15-1, Table 1 reports 78.0 All mIoU for the MBS reproduction and 79.3 for AKI, whereas the original MBS paper reports 80.6 in the overlapping setting. Appendix B provides a reasonable comparison under a common configuration, but the reproduction gap should be explained, including training schedule, loss weights, and evaluation differences. This limits confidence in a broad superiority claim; it does not invalidate the controlled improvements or the value of the proposed framework. I regard this as a fixable concern rather than a sufficient reason to reject.

The optimization properties and perturbation study are best viewed as design motivation. The paper appropriately qualifies their scope in Appendices A and B.1. The strongest contribution is therefore the concrete method and its experimental validation, rather than a general theoretical guarantee; this supports a weak-accept recommendation rather than a clear accept.

Questions for authors
  1. Clarification of the main comparison: What accounts for the difference between the published MBS result and its reproduction, particularly VOC 15-1? Please identify the relevant training, loss-weight, and evaluation differences and distinguish the controlled comparison from claims about the published state of the art. [MBS, ECCV 2024, Table 1, p. 10.]

  2. Clarification of the masking control: Appendix C.9 fixes the random support within each incremental step while recomputing the RCA support during optimization. How much of the Table 13 gap should be attributed to coordinate selection versus this difference in update policy? A clarification of the interpretation would help; I do not regard a new large experimental study as necessary for acceptance.

Reviewer_g44dOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Adaptive Knowledge Inheritance (AKI) for replay-free class-incremental semantic segmentation with fixed model capacity. It combines response masking and prediction consistency (RCA), selective alignment along reusable historical feature directions (RPA), and an old–new class distribution-overlap penalty (DEP). Optimization observations and controlled perturbation analyses motivate the three levels of adaptation. Experiments on Pascal VOC and ADE20K, together with extensive ablations and diagnostic analyses, support the framework’s effectiveness.

Critical strengths
  1. The framework appears technically sound. Its three components address relevant aspects of the trade-off between retaining old knowledge and learning new classes. The paper provides detailed formulations and analyses of how the mechanisms operate.

  2. The experimental support is extensive. The paper evaluates nine incremental settings and includes component combinations, repeated runs, gradient-conflict analysis, and controlled comparisons of response supports and feature inheritance. These analyses provide useful evidence that the framework works and that its components contribute to the results.

Critical weaknesses
  1. RCA is close to established masking and consistency regularization. Section 3.3 builds a response mask from weight norms and enforces prediction consistency between the complete and masked branches. The general recipe resembles earlier consistency regularization, such as R-Drop (NeurIPS 2021). RCA uses a different masking rule, and Table 13 supports that rule over random support. Still, I find the conceptual advance limited. The paper should explain more clearly what RCA adds beyond a particular choice of mask within a familiar consistency objective.

  2. RPA remains closely related to selective feature distillation and projected alignment. Its direction-selection criterion combines historical discriminability with observed current adaptation. Tables 10–11 provide useful evidence beyond full-space, confidence-selected, and random rank-one constraints. However, projected feature alignment for balancing old and new knowledge is already explored in SPPA (Lin et al., ECCV 2022), which the paper also cites. The methodological distinction from this line of work needs a more direct explanation. The results support the selected direction, but the broader claim of a new knowledge-reorganization mechanism is less convincing.

  3. DEP builds on an established Gaussian class-separability criterion. Section 3.5 fits diagonal Gaussian class statistics and penalizes old–new Bhattacharyya overlap. A closely related statistical construction appears in Pándy et al., Transferability Estimation Using Bhattacharyya Class Separability (CVPR 2022), which models classes with Gaussians and measures pairwise overlap using the Bhattacharyya coefficient. That work uses it as a transferability metric rather than a training loss. This prior work shows that the statistical modelling and separability criterion are established, although DEP’s specific CISS formulation differs. Appendix A.5 clearly explains how DEP considers both class centers and dispersion. Still, the paper should clarify what methodological advance comes from applying this familiar criterion to current old- and new-class distributions. Overall, the framework feels like a well-tested combination of established ideas. The extensive analyses establish effectiveness more clearly than a substantial conceptual advance.

Questions for authors
  1. What is RCA’s main technical contribution beyond response masking followed by prediction consistency? Please explain the distinction at the objective or mechanism level, in addition to the random-support comparison.

  2. How does RPA fundamentally differ from prior projected feature alignment, particularly SPPA? Which part of the historical-discriminability/current-adaptation criterion provides the key new insight?

  3. What is the main methodological advance in DEP beyond applying a standard distribution-overlap penalty to current old- and new-class statistics? More broadly, what does coordinating these three components establish beyond their empirical complementarity?

1 comment

Thanks to the authors for the follow-up. I agree that Table 8 and Table 3 use different settings. But my main concerns remain, and the paper's own tables support them.

  1. RCA. The authors say R-Drop works because it is combined with RPA and DEP. But the random-mask variants in Tables 13 and 20 also use RPA and DEP (Appendix C.9, C.15), so this doesn't explain the gap. The "long sequence" explanation doesn't fit either: Table 20 is 19-1, just one step, and Random-Dynamic still collapses to 3.4 All, far below plain MiB (68.9). R-Drop resamples a random mask every forward pass and works fine. So the random-mask baseline looks broken, not like a fair control. Showing R-Drop and random masks side by side on 10-1 and 19-1 would settle this.

  2. RPA. Eq. 11 is the Fisher discriminant, and the "w/o Σ_ada" row in Table 10 is just Fisher LDA. Of the +10.0 New gain over Random-R1, Σ_ada (the only new part) gives +1.9 New / +0.6 All. In Table 9 (15-1), adding all of RPA on top of RCA+DEP moves All from 79.2 to 79.3.

  3. DEP. "DEP improves 43.4 → 53.0" compares against no regularizer, not against GBC. Against the GBC-style loss, the gain is +0.5 All.

  4. Complementarity. Without RCA, RPA alone gives +16.2 and DEP alone +9.6, but together only +17.5. That points to overlap, not coordination. With RCA, the third component adds about 1 point.

Overall, I still think this is a better engineering combination work, but each part stays close to R-Drop, Fisher LDA, and GBC, and the ablations don't support the cross-level framing. I keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.