acceptodds
Withdrawn from ICLR 2027

NCAR: Neural Collapse Alignment Regularization for Machine Unlearning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Machine unlearning aims to remove the influence of requested training data while efficiently approximating the solution induced by retraining on the retained data. Existing gradient-based methods primarily optimize predictive or output-level objectives for forgetting and retention, yet a discrepancy in the retained representation-classifier structure can remain even when predictive behavior is close to retraining outcomes. To address this structural discrepancy, we propose Neural Collapse Alignment Regularization (NCAR), which uses the idealized Neural Collapse terminal geometry of supervised classification as a structural prior. NCAR constructs a canonical simplex over the remaining classes, aligns it with the current classifier, and adaptively fits its translation and scale to retained features to obtain a classifier-aligned structural target. The resulting retain-side regularizer couples within-class concentration and simplex organization of class means with feature-classifier alignment. NCAR can be incorporated into existing gradient-based unlearning objectives while preserving the base method's forgetting-loss construction and parameter-update restrictions. We evaluate NCAR across random-subset and full-class unlearning, multiple datasets, architectures, and base methods, analyzing predictive behavior and retained representation structure together.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
machine unlearning, neural collapse, representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_xLMMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Neural Collapse Alignment Regularization (NCAR), a retain-side regularizer that can be added to existing gradient-based unlearning methods. It constructs simplex ETF targets for the remaining classes, aligns them with the current classifier, and fits their translation and scale to retained features. The resulting penalty encourages within-class concentration and structured class separation while preserving the base forgetting objective. The paper provides a geometric characterization of the regularizer and evaluates it across three datasets, ResNet and ViT architectures, and random-subset and full-class deletion. The reported results show improved accuracy-based agreement and retained-feature similarity to retrained models across several base methods, supported by component ablations and neural-collapse diagnostics.

Critical strengths

The method has a clear intuition and a coherent design. Using neural-collapse geometry to regularize retained representations is a reasonable way to limit disruption during unlearning. Adapting the targets to the current classifier and feature scale makes the regularizer flexible, and it can be added to existing methods without replacing their forgetting objectives.

The experiments show benefits across several base methods, two architectures, and both random-subset and full-class deletion. Improvements in retained-feature similarity and accuracy-based agreement with retraining suggest that the regularizer has practical value beyond a single configuration, although the gains vary substantially by method.

Critical weaknesses

I like the intuition, but the paper presents a relatively simple regularizer as a more substantial contribution than it is. NCAR combines established ideas in within-class concentration, simplex ETF geometry, and feature–classifier alignment. Its main addition is fitting classifier-aligned targets for retained features during unlearning. The geometric analysis explains this construction, but the novelty over existing NC-guided training and CMF-based unlearning remains limited. Direct comparisons with these approaches and simpler alternatives, such as center loss and feature distillation, are needed to establish the value of this particular design.

The model-selection procedure weakens the empirical evidence. The authors select hyperparameters and checkpoints using agreement with an exactly retrained model, and the selection criterion includes accuracy on the same test set used for reporting. This leaves performance without access to retraining outcomes unresolved and compromises the independence of the test evaluation. Results under a validation-based selection procedure, without access to the retrained reference during selection, are needed to assess whether the reported improvements hold in practice.

Questions for authors

Clarification requests:

  1. What is the main technical contribution beyond existing NC-guided training and CMF-based unlearning? In particular, why should orienting the simplex using the current classifier be preferable to fitting it to retained class means?
  2. The stated selection procedure uses retraining-relative Avg. Gap, which includes test accuracy. Were the same test examples used for checkpoint selection and final reporting? How would the authors select β\beta and the stopping point without an exactly retrained reference?

Additional evidence needed for the current claims:

  1. Do NCAR’s gains persist when hyperparameters and checkpoints are selected using validation data without access to retraining outcomes?
  2. Under comparable tuning and computational budgets, how does NCAR compare with center loss, retained-feature distillation, and existing NC- or CMF-based approaches? These comparisons would determine whether its additional construction provides a meaningful benefit.
1 comment

We thank the reviewer for the thoughtful questions.

Technical contribution and classifier alignment

For the main contribution and distinction from prior NC-based methods, we kindly refer to our response to Reviewer AU12.

Regarding classifier alignment, the intention is to anchor the structural target to the current model's classification directions rather than determine its orientation solely from retained features. The −CA ablation directly compares these alternatives. Under full-class unlearning, classifier alignment reduces Avg. Gap from 3.85 to 2.17 and improves CKA from 0.780 to 0.925.

Hyperparameter and checkpoint selection

We used retraining-relative Avg. Gap, a commonly adopted evaluation criterion in machine unlearning, to select hyperparameters and checkpoints. The base methods were first tuned using this criterion. When applying NCAR, we kept the selected base hyperparameters fixed and selected only the regularization coefficient (\beta) and checkpoint using the same criterion. This allowed us to compare the base methods and their NCAR variants under consistent selection conditions.

The statement that NCAR does not require exact retraining refers specifically to the construction of its regularization targets. NCAR constructs these targets from the current classifier and retained features, with translation and scale fitted analytically. The retrained model is used as a reference for experimental selection and evaluation, but is not involved in constructing the regularization targets.

We recognize the reviewer's concern regarding the inclusion of test accuracy in Avg. Gap. The reported results are therefore comparisons under a common reference-guided selection protocol, rather than an independent validation-only evaluation. We hope this distinction clarifies both the rationale for our experimental procedure and the intended scope of NCAR's retraining-free target construction.

Reviewer_AU12OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

Machine unlearning (MU) aims to remove specific knowledge associated with a subset of the data, while preserving knowledge on the retained data. This paper proposes Neural Collapse Alignment Regularization (NCAR), utilizing the Neural Collapse geometry as a prior, which can be integrated with the existing gradient-based MU methods. The method aims to utilize within-class concentration, class means alignment, and classifier-feature alignment. Experimental results show that NCAR can improve performance across different tasks and baselines.

Critical strengths
  • The authors conduct extensive experiment to evaluate NCAR. The tasks include random-subset unlearning and full-class unlearning, datasets include CIFAR10 and TinyImagenet, and different architectures are evaluated.
Critical weaknesses
  • The contribution and novelty of the paper seems incremental. Firstly, Neural Collapse and NC1, NC2, and NC3 are well-established concepts. As discussed in Related Work, recent methods such as POUR (Le et al., 2026), and CMF (Gao et al., 2026) study Neural Collapse within the unlearning framework. Additionally, there is prior work on using Neural Collapse for regularization in other tasks (e.g., Liu et al., 2023), which I think is worth discussing in the related work section. The main contribution of NCAR seems to be applying Neural Collapse as a regularizer for retained features, utilizing NC1-NC3, which seems incremental relative to prior work. The performance of NCAR is only assessed when applying the regularization to previous baselines, but comparison to more recent work such as POUR is not included. Therefore, the advantage of NCAR over similar work is not yet established.

  • The performance gains are noticeable mainly when NCAR is applied to baselines that are not as strong. When applied to a recent baseline, UGradSL+, which already has strong performance, the gains are not statistically significant. Therefore, NCAR does not provide clear advantages if the baseline already has good performance. Additionally, in other baselines, performance gains are not consistent across all metrics, and NCAR can hurt performance in some cases. It would be helpful if the authors provide discussions on why this could be the case.

Liu, Xuantong, et al. "Inducing neural collapse in deep long-tailed learning." International conference on artificial intelligence and statistics. PMLR, 2023.

Questions for authors

Can the authors further clarify the main novelty and advantage of their method?

1 comment

We thank the reviewer for the opportunity to clarify the main contribution and advantages of NCAR.

The central contribution of NCAR is a structural regularization approach for machine unlearning that can be applied across different deletion targets and underlying unlearning mechanisms. The motivation is that, while existing methods employ different objectives and update rules to achieve forgetting, these updates can also disrupt representations of the retained data. NCAR addresses this shared problem by introducing a structural constraint on retained representations without altering how each base method performs forgetting.

To achieve this, NCAR uses Neural Collapse geometry as a structural prior. It constructs a simplex target over the remaining classes, aligns its orientation with the corresponding weights of the current classifier, and analytically fits its translation and scale to retained features. This allows the regularization target to adapt to both the remaining-class configuration and the evolving representation space, without requiring features from an exactly retrained model.

The distinction from related NC-based approaches lies in how this structural guidance is used within unlearning. Whereas POUR incorporates NC-based projection into the forgetting mechanism and CMF-based approaches use class-mean features for classifier construction, NCAR provides retain-side structural regularization that complements the existing forgetting procedure. The same formulation can therefore be applied to different gradient-based methods while preserving their original forgetting objectives and parameter-update restrictions.

Reviewer_z3fBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

NCAR adds a retained-feature regularizer to existing machine-unlearning methods. It constructs a simplex target, aligns it with the current classifier, and fits its translation and scale to retained minibatch features. Experiments across three image datasets, two architectures, and seven base methods generally improve retained-feature similarity and an aggregate accuracy gap relative to retraining.

Critical strengths
  1. NCAR combines within-class concentration, simplex geometry, and classifier alignment without changing the base forgetting objective. Target construction does not require a retrained feature bank or an additional network.

  2. Offers interpretable mathematical analysis. The residual decomposition separates within-class concentration from classifier-aligned simplex fitting, clarifying what the regularizer encourages.

  3. Adapts target position and scale analytically. The closed-form translation and scale fit accommodates the current feature space without introducing separately learned prototypes.

  4. The paper explicitly states that structural alignment is not a certificate of forgetting and reports computational overhead.

Critical weaknesses
  1. Section 4.1 and Appendix A.4 select hyperparameters and checkpoints by minimizing the reported Avg. Gap, which includes test accuracy and requires an exact-retraining reference. Appendix A.2 explicitly excludes the validation partition from selection. This compromises independent evaluation and leaves the practical selection procedure unresolved, even though target construction itself avoids retraining. Use validation-based selection without a retraining oracle, and reserve test metrics for final evaluation.

  2. The presentation assumes familiarity with Neural Collapse, making it difficult to appreciate the proposed regularizer and its connection to unlearning. Briefly explain NC1–NC4 intuitively, illustrate the geometry, and clarify why encouraging this structure should help approximate retained-data retraining. Distinguish established NC properties from the hypothesis motivating their use in unlearning.

  3. Limited evaluation across deletion fractions. Random-subset experiments consider only 10% deletion, leaving robustness to deletion size unclear. Evaluating smaller and larger fractions, for example, 1%, 5%, 20%, and 50%, would help establish whether NCAR consistently improves the forgetting–retention tradeoff as less retained data remains available.

  4. The connection between Neural Collapse and the two deletion settings needs clearer justification. Section 3.6 explains the construction for random-subset and full-class unlearning, but provides limited intuition for why the same geometric prior should improve retraining agreement when class structure is preserved versus changed. Explain this distinction and evaluate additional deletion rates and class configurations to establish the scope of generalization.

Questions for authors
  1. Is it possible to test class-imbalanced retained sets and deletion rates beyond 10%. A common-radius simplex may be less appropriate when retained class frequencies differ substantially.
  2. Translation and scale are fitted independently per minibatch, whereas the structural interpretation uses a full-data fit. Is it possible to report sensitivity to batch size and class coverage?
  3. Report practical scaling. Table 10 already shows roughly 21–46% additional time per update on CIFAR-10. Extend profiling to larger class counts, where the alignment cost grows quadratically.
1 comment

We thank the reviewer for the constructive feedback and valuable experimental suggestions.

We believe that evaluating additional deletion fractions and class-imbalanced retained sets, examining sensitivity to minibatch size and class coverage, and profiling computational overhead at larger class counts would provide a more comprehensive assessment of NCAR. The comments on model selection and the motivation connecting Neural Collapse to different deletion settings are also helpful for strengthening the presentation and evaluation of our work.

We appreciate these thoughtful suggestions and the opportunity to improve our research. We will take them into account and work to incorporate the suggested evaluations and clarifications when preparing an improved version for submission to another venue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.