When Regularization Takes a Shortcut: Preventing Neural Collapse for OOD Generalization

Neural Collapse (NC) is a phenomenon observed during supervised deep training of neural networks, in which within-class representations become increasingly compact while different classes become highly separable. Although NC has been associated with improved out-of-distribution (OOD) generalization, state-of-the-art anti-collapse methods often fail to distinguish between within-class collapse and between-class separation. Consequently, current anti-collapse regularization techniques may satisfy their objectives simply by increasing between-class separation while leaving within-class collapse largely unaffected. NC prevention methods have been shown to improve OOD generalization. However, we observe that state-of-the-art anti-collapse regularizers act on the representation through label-agnostic projections without access to the within/between-class decomposition. Therefore, they may inadvertently trade one for the other. In this work, we investigate the interaction between within-class variability and between-class separation in the loss landscape of widely used anti-collapse regularization methods. We show theoretically that existing regularization objectives can exploit an unintended shortcut by increasing between-class separation to satisfy their optimization objectives. This limits their ability to effectively prevent NC. To address this limitation, we introduce a regularization strategy that explicitly preserves within-class variability, while maintaining strong between-class separation. Our approach mitigates NC across a range of collapse regimes while preserving high in-distribution accuracy. Our extensive experimentsOur code is included in the supplementary material and will be made public. on diverse visual datasets further demonstrate improved OOD generalization across different input distributions and label spaces.