- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies how global distribution-matching regularization can reduce its objective by increasing between-class separation while failing to preserve sufficient within-class variability. After analyzing the optimization behavior of Strong-SIGReg and Weak-SIGReg, the authors propose PIER, which combines per-coordinate scale constraints, Weak-SIGReg covariance regularization, and nearest-neighbor entropy regularization to mitigate neural collapse and improve OOD generalization. Experiments on ImageNet-100 and ImageNet-1K evaluate NC metrics, ID classification accuracy, and linear-probing performance on eight downstream datasets.
Critical strengths
-
The paper offers a useful theoretical account of the optimization behavior underlying the proposed method.
-
The experimental section is reasonably broad, covering multiple architectures, datasets, training settings, and ablations.
Critical weaknesses
-
The novelty of PIER is not yet fully clear. Its scale, covariance, and entropy terms are closely related to existing variance-covariance regularization, Weak-SIGReg, VISReg, and nearest-neighbor entropy regularization. VCReg is particularly relevant because it applies high-variance and low-covariance constraints during supervised pretraining to improve transfer and mitigate representation collapse, yet it is neither discussed nor evaluated. The comparison is also incomplete in other respects: the paper does not evaluate the full method of Harun et al. (2025), which combines encoder entropy regularization with a fixed Simplex ETF projector, and VISReg and VICReg are not included in the ImageNet-1K experiments. HypO and Neural Collapse by Design are also omitted. Finally, the ImageNet-1K results do not include the component ablation comparing Weak-SIGReg, Weak-SIGReg+Scale, Weak-SIGReg+Entropy, and PIER. Without these comparisons, the additional contribution of PIER over Weak-SIGReg remains unclear.
-
I found an apparent inconsistency in the NC metrics. The main text defines NC1 as , which should lie between 0 and 1, yet the tables report values such as 5.62 and 9.32. The appendix figures label NC1 as “W/B,” while the paper describes NC2 in terms of the growth of class-mean geometry or between-class separation. This also appears inconsistent with the standard NC1 and NC2 definitions attributed to Zhu et al. (2021) and Zhou et al. (2022). The authors should clarify the exact implementation, formulas, units, and directions of the reported metrics before the NC results can be fully assessed.
-
Some implementation and reporting details also need clarification. The AdamW and SGD experiments change both the optimizer and the weight-decay setting, making it difficult to separate optimizer effects from regularization effects. The paper should specify whether the batch covariance is centered, how the representation mean is handled, how the PIER terms are ordered and weighted, and what computational overhead PIER introduces in practice. Finally, the appendix states that Weak-SIGReg and PIER achieve the best in every setting, but Table 14 shows that PIER slightly underperforms Weak-SIGReg on ResNet-18. Claims such as “consistently outperforms” should therefore be qualified.
Questions for authors
See the "Critical Weaknesses".