acceptodds
Withdrawn from ICLR 2027

When Regularization Takes a Shortcut: Preventing Neural Collapse for OOD Generalization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

10 views

Abstract

Neural Collapse (NC) is a phenomenon observed during supervised deep training of neural networks, in which within-class representations become increasingly compact while different classes become highly separable. Although NC has been associated with improved out-of-distribution (OOD) generalization, state-of-the-art anti-collapse methods often fail to distinguish between within-class collapse and between-class separation. Consequently, current anti-collapse regularization techniques may satisfy their objectives simply by increasing between-class separation while leaving within-class collapse largely unaffected. NC prevention methods have been shown to improve OOD generalization. However, we observe that state-of-the-art anti-collapse regularizers act on the representation through label-agnostic projections without access to the within/between-class decomposition. Therefore, they may inadvertently trade one for the other. In this work, we investigate the interaction between within-class variability and between-class separation in the loss landscape of widely used anti-collapse regularization methods. We show theoretically that existing regularization objectives can exploit an unintended shortcut by increasing between-class separation to satisfy their optimization objectives. This limits their ability to effectively prevent NC. To address this limitation, we introduce a regularization strategy that explicitly preserves within-class variability, while maintaining strong between-class separation. Our approach mitigates NC across a range of collapse regimes while preserving high in-distribution accuracy. Our extensive experimentsOur code is included in the supplementary material and will be made public. on diverse visual datasets further demonstrate improved OOD generalization across different input distributions and label spaces.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Neural Collapse, Transfer Learning, OOD Generalisation, Supervised Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_jx35OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how global distribution-matching regularization can reduce its objective by increasing between-class separation while failing to preserve sufficient within-class variability. After analyzing the optimization behavior of Strong-SIGReg and Weak-SIGReg, the authors propose PIER, which combines per-coordinate scale constraints, Weak-SIGReg covariance regularization, and nearest-neighbor entropy regularization to mitigate neural collapse and improve OOD generalization. Experiments on ImageNet-100 and ImageNet-1K evaluate NC metrics, ID classification accuracy, and linear-probing performance on eight downstream datasets.

Critical strengths
  1. The paper offers a useful theoretical account of the optimization behavior underlying the proposed method.

  2. The experimental section is reasonably broad, covering multiple architectures, datasets, training settings, and ablations.

Critical weaknesses
  1. The novelty of PIER is not yet fully clear. Its scale, covariance, and entropy terms are closely related to existing variance-covariance regularization, Weak-SIGReg, VISReg, and nearest-neighbor entropy regularization. VCReg is particularly relevant because it applies high-variance and low-covariance constraints during supervised pretraining to improve transfer and mitigate representation collapse, yet it is neither discussed nor evaluated. The comparison is also incomplete in other respects: the paper does not evaluate the full method of Harun et al. (2025), which combines encoder entropy regularization with a fixed Simplex ETF projector, and VISReg and VICReg are not included in the ImageNet-1K experiments. HypO and Neural Collapse by Design are also omitted. Finally, the ImageNet-1K results do not include the component ablation comparing Weak-SIGReg, Weak-SIGReg+Scale, Weak-SIGReg+Entropy, and PIER. Without these comparisons, the additional contribution of PIER over Weak-SIGReg remains unclear.

  2. I found an apparent inconsistency in the NC metrics. The main text defines NC1 as Tr(ΣW)/Tr(ΣT)\mathrm{Tr}(\Sigma_W)/\mathrm{Tr}(\Sigma_T), which should lie between 0 and 1, yet the tables report values such as 5.62 and 9.32. The appendix figures label NC1 as “W/B,” while the paper describes NC2 in terms of the growth of class-mean geometry or between-class separation. This also appears inconsistent with the standard NC1 and NC2 definitions attributed to Zhu et al. (2021) and Zhou et al. (2022). The authors should clarify the exact implementation, formulas, units, and directions of the reported metrics before the NC results can be fully assessed.

  3. Some implementation and reporting details also need clarification. The AdamW and SGD experiments change both the optimizer and the weight-decay setting, making it difficult to separate optimizer effects from regularization effects. The paper should specify whether the batch covariance is centered, how the representation mean is handled, how the PIER terms are ordered and weighted, and what computational overhead PIER introduces in practice. Finally, the appendix states that Weak-SIGReg and PIER achieve the best EGENE_{\mathrm{GEN}} in every setting, but Table 14 shows that PIER slightly underperforms Weak-SIGReg on ResNet-18. Claims such as “consistently outperforms” should therefore be qualified.

Questions for authors

See the "Critical Weaknesses".

Reviewer_N3MuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Some existing regularization methods for preventing Neural Collapse may reduce the loss by increasing the separation between different classes, without effectively addressing the collapse of within-class features. The authors analyze this phenomenon by distinguishing between within-class variance and between-class separation. Based on this analysis, they propose PIER, a regularization method that introduces an additional loss term to encourage the model to preserve more within-class feature information.

Critical strengths
  1. The paper is well-written, provides sufficient preliminary background, and is easy to follow.
  2. The paper gives meaningful insight into existing regularization objectives.
Critical weaknesses
  1. The paper claims that its analysis disentangles within-class variability and between-class angular separation in the loss landscape. However, the main methodological change introduced by PIER is the addition of a regularization loss. I think such a loss only imposes a "soft constraint", it is unclear whether PIER can theoretically guarantee the disentanglement of these two geometric dynamics.
Questions for authors
  1. Proposition 4 appears to establish the desired behavior under a hard constraint on the trace of the total covariance matrix and in a simplified two-class setting. However, PIER employs a soft scale penalty and remains label-agnostic. What theoretical guarantees, if any, ensure that PIER actually preserves within-class variability in the general multi-class setting?
  2. I didn't see experiment results that can directly verify that the shortcut is eliminated during training. Moreover, from the ablation results, it seems that scale, which the authors emphasize as a key component, contributes substantially less to OOD performance than Entropy in this experiment. Is the effectiveness of the proposed method primarily due to Scale mitigating shortcuts, or does the existing Entropy regularization account for most of the observed improvements?
Reviewer_cjYrOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes the regularization of the within covariance matrix of the class distributions in order to prevent or control the neural colapse in deep neural networks. There are a lot of works pulished that try to do the same think and thus the novelty, if any, is on the novel approach for addressing this issue. The paper uses extensive mathematical formalism denoting use of LLMs. The authors do declare that they have used generative AI for such purposes and the paper indeed seems to have been heavily AI manipulated/extended. Regarding the methodological ideas I cannot find any significant contribution to support an ICLR paper. The authors seem to overisight the entire literature of Linear Discirminant Analysis (Fisher's criterion) and the numerous works and regularizers that try to control the within-scatter matrix even before the deep learning era in the last 30 years. Instead, they start their proposed approach from NC1 and NC2 which are indeed inspired from Fisher's criterion. The valiadation is extensive but fails to highlight the most significant problem to my view which is the estimation of the within-class covariance in huge space dimensions, with limited batch size, large number of classes (e.g., imagenet-1K) and the strong assumntion of homoscedacity which does not hold in complex classification tasks. Their experiments indicate that indeed in 1K calasses with 512 batch size they cannot improve and this is completely resonable since they do not have even one sample per class in the batch.

Critical strengths

The only strength I can find in the paper is the experiment that indeed show that it is difficult to estimate covariance matrices in deep learning hidden layers.

Critical weaknesses

The methodology oversimplifies the problem of neural colapse with assumptions that the within-class scatter matrix should be pushed to become diagonal, ignoring cross-correlations of the features. The assumption of homoscedacity is not valide for complex classification tasks. The estimation of the matrices with 1K classes is prohibitive since the batch size should become huge. The methimatical formulation is often unnecessary and clearly a product of LLMs that can justify even the most obious thinks id they are asked to do so.

Questions for authors

How you can guarantee proper estimation of the scatter matrices with so few samples inside the batch for each class? Why you consider that each class is homoscedastic and even more, Gaussian, when we know that most of the classes are multimodal? What is the novel idea in the paper that is not incremental to the current literature of regularization based on Fisher's criterion?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.