acceptodds
Under review as a conference paper at ICLR 2027

From Neural Collapse Geometry to Structural Memory: Preserving Decision Structure in Continual Learning

15 views

Abstract

Neural collapse (NC) reveals a structured geometry that links sample representations, class prototypes, and classification decisions, providing a unified perspective on how class knowledge can be organized across learning stages. Existing NC-based continual learning methods adopt fixed or progressively expanded simplex equiangular tight frames (ETFs) as learning targets. However, specifying such a geometric target does not by itself ensure that the induced discriminative structure persists across stages; in particular, globally expanding an ETF shifts prototypes that have already been established for old classes. We propose Structure-Preserving Continual Learning (SPCL), which treats the NC-induced decision structure as structural memory—the class reference, the coordinate response, and the decision margin of the established classes—and preserves it across stages. SPCL freezes established prototypes and embeds each new task's simplex block in the orthogonal complement of the old span; distills each sample's response distribution over the frozen prototypes; and penalizes old-class margin decreases on replayed samples in the post-plasticity window, with mixed-batch feature distillation stabilizing the underlying representation. These mechanisms preserve the class references and sample-level relations anchored to them, letting structural memory persist as the label space expands. Experiments on Seq-CIFAR-100 with a total replay buffer capacity of 500 samples show that SPCL improves final average accuracy while reducing average forgetting by approximately 70% relative to ProNC under matched protocols.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Class-Incremental Learning, Neural Collapse, Equiangular Tight Frame, Catastrophic Forgetting, Knowledge Distillation, Structural Memory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZFDjOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SPCL, a replay-based class-incremental learning method built on neural-collapse prototypes. ProNC rebuilds a global simplex ETF (a set of equally spaced class prototypes) at each task. SPCL instead freezes the old class prototypes, places new ones in their orthogonal complement, and adds distillation and margin losses to preserve old decisions. On Seq-CIFAR-100 and Seq-TinyImageNet, it reduces forgetting and slightly improves accuracy over ProNC.

Critical strengths
  1. Simple construction that removes prototype drift. Freezing old prototypes and placing new ones orthogonally removes drift in the class reference by design, with no extra trainable parameters. This directly addresses a clear limitation of ProNC.
Critical weaknesses
  1. Limited novelty. Fixed, pre-allocated class prototypes have been explored in class-incremental learning [1], and orthogonal prototype placement in [2]. Response distillation, a margin loss on old classes and cosine feature distillation closely resemble LwF-style distillation and LUCIR [3].

  2. Limited experimental scope. The experiments use only a ResNet-18 trained from scratch, on two small benchmarks (CIFAR100 and TinyImageNet).


References

[1] Class-incremental Learning with Pre-allocated Fixed Classifiers. ICPR 2020.

[2] OrCo: Towards Better Generalization via Orthogonality and Contrast for Few-Shot Class-Incremental Learning. CVPR 2024.

[3] Learning a Unified Classifier Incrementally via Rebalancing. CVPR 2019.

Questions for authors
  1. How does SPCL compare with LUCIR, or with a fixed orthogonal classifier trained with the same distillation losses, under the same protocol?
Reviewer_CCVDOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SPCL, which organizes old-class prototypes, the response distributions of samples to prototypes, and classification margins into “structural memory”: it freezes old prototypes, appends a simplex ETF block for new tasks in their orthogonal complement space, and combines response distillation, margin constraints activated in the final stage of training, and feature distillation to preserve old knowledge. Corresponding evaluations are conducted on CIFAR-100 and TinyImageNet.

Critical strengths
  1. The prototype expansion mechanism is clear.

  2. The reported empirical gains are very stable relative to the main baselines.

Critical weaknesses

The novelty of the core geometric motivation and its decision-level explanation are insufficient. For example, ProNC explicitly discusses prototype displacement caused by expansion. Moreover, its expansion formula, which retains the old orthogonal basis, implies that for a fixed feature, the score difference between old classes satisfies si′−sj′=a(si−sj)s'_i-s'_j=a(s_i-s_j). Therefore, the prediction ranking among old classes remains unchanged. Prototype displacement alone does not prove that the decision boundaries between old classes are disrupted.

Questions for authors

As far as I know, the state of the art on CIFAR-100 with 10 tasks should already be above 50 without replay, whereas this paper achieves less than 46 even though it requires storing old samples. This substantially undermines the effectiveness of the proposed method. Also, if your method has other advantages, please highlight them.

Reviewer_67tKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Structure-Preserving Continual Learning (SPCL), which freezes established class prototypes and places new simplex blocks in orthogonal subspaces. It combines this construction with response distillation, a temporally gated margin loss, and feature distillation.

Critical strengths

I really commend Authors for an excellent explanation of the incompatibility between preserving existing prototypes and expanding an exact global simplex ETF. The proposed approach is easy to follow and the paper explicitly acknowledges its dimensional-capacity limit and the absence of guaranteed old–new decision stability. Component ablations and prototype/feature diagnostics provide useful evidence beyond aggregate accuracy.

Critical weaknesses

I found the following four main weaknesses in the paper that currently prohibit me from giving this paper a higher rating:

W1: There is no sufficient separation between reduction in forgetting and reduction in acquisition. Forgetting measures the drop from peak accuracy, so lower peaks can produce lower forgetting without better final retention. This is particularly relevant because removing response distillation and the margin loss improves final accuracy at both buffer sizes: 50.98% versus 49.98% at (B=500), despite higher forgetting. Full accuracy matrices, immediate post-task accuracy, and learning curves are needed to support the claim that these losses preserve knowledge without materially impairing acquisition.

W2: Not sufficient ablation / isolation of the frozen geometry. SPCL changes prototype placement, adds losses, and increases feature-distillation strength. ProNC already combines progressive ETF geometry with alignment and distillation. In my opinion this paper critically needs a factorial comparison of frozen versus expanding references under identical losses and separately tuned distillation strengths. Table 2 retains the frozen construction throughout, so it cannot establish how much of the improvement comes specifically from eliminating prototype movement.

W3: The structural interpretation exceeds the demonstrated guarantees. The propositions establish geometric properties, but prototype movement alone does not establish harmful decision changes. At the same time preserving old-class responses and margins does not prevent new classes from overtaking old predictions. The zero-flip audit is useful but restricted to incompletely specified audited inputs. I see that the response loss is standard distillation under a fixed classifier, while the margin itself is determined by distribution of responses. This allows me to conclude that the methodological novelty therefore lies mainly in the construction and scheduling, rather than three independent memory mechanisms.

W4: Experimental section and reproducibility needs strengthening. Most baseline results are quoted and run by the Authors; the documented ProNC reproduction covers only CIFAR-100 at (B=200), which feels quite weak. The TinyImageNet accuracy gain at (B=500) is only 0.27 points, without providing any baseline uncertainty or standard deviations. Hyperparameters are described as fixed a priori but also selected through performance and activation profiles, without a clear validation protocol. The no-hinge/ungated forgetting values in Section 4.3 differ from Table 2, making me question the preparation and proofreading of the paper.

Questions for authors

I would like for the Authors to address these questions and everything else I mentioned As W1-W4:

Q1: Can you provide complete task-accuracy matrices and immediate post-task accuracies to determine whether the forgetting reduction reflects better retention or lower initial learning?

Q2: How do frozen and expanding prototype constructions compare under identical objectives, with equally thorough tuning of feature-distillation strength?

Q3: Which examples constitute the 45,000 audited inputs, how many are unique, and do the zero-flip findings hold on held-out old-class data, including old–new competition after student updates?

Q4: Can you reproduce ProNC across all four settings with paired seeds, clarify the validation protocol and discrepant ablation values, and specify and evaluate the placement rule after orthogonal capacity is exhausted?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.