acceptodds
Under review as a conference paper at ICLR 2027

Clustering Stability and Collapse in Deeper Networks

Abstract

We show that as neural networks grow deeper, their internal representations progressively lose the clustering structure of the input and that this degradation, measured by the Frobenius distance between projection operators, is tightly and consistently associated with a sharp drop in clustering accuracy across architectures and datasets. This geometric characterization leads directly to two principled strategies: a structure-aware regularizer that penalizes projection distance at each layer, and an orthogonal initialization strategy that provides a structurally favorable starting point for training. Across more than 60,000 trials spanning five prevalent architectures and eight real-world datasets, our regularizer delays collapse in performance by an average of 7 layers and sustains high clustering accuracy well past the depth at which all unregularized counterparts have collapsed entirely.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
representational collapse, subspace cluster structure, deep representation learning, structure-preserving regularization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_a2ebOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a simple regularizer, the sum over layers of squared Frobenius distances between the projection matrices of the input and of each layer's representation, and an orthogonal initialization that delay the collapse in clustering performance, enabling the training of deeper networks for clustering. The initialization is evaluated on synthetic data, while the regularizer is evaluated on synthetic data and on established image-clustering datasets.

Critical strengths
  • S1: Overall, the writing is clear and the paper is easy to follow.
  • S2: The proposed regularizer is simple and easy to implement.
Critical weaknesses
  • W1: Methodology and experiment setup: The authors train different architectures (e.g., residual networks) from scratch, but hyperparameters such as the number of epochs, batch size, and optimizer are not provided. Each architecture uses a single learning rate across all datasets and depths, which makes it difficult to distinguish an actual performance collapse phenomena from undertrained networks. This is evident for 18-layer networks, which perform poorly in the paper, whereas resnet-18 trained from scratch is commonly used in clustering and achieves strong performance (e.g., 81.8% accuracy on CIFAR-10 [1]).

  • W2: Inconsistencies and mistakes in the paper: Table 1 summarizes the datasets, but the reported numbers of clusters do not match the ground truth, e.g.: MNIST KK =20, CIFAR-10 KK =40. Figure 3 says: "Remarkably, our regularizer never hurts, regardless of the choice of λ\lambda", but Sec. 5.1. contradicts it: "Beyond a certain threshold, further increasing λ\lambda begins to negatively affect performance".

  • W3: Novelty and related work: The orthogonal initialization proposed in Sec. 5 is presented without citation, although it follows the standard orthogonal initialization [2], which is implemented in PyTorch. Its benefits for deep networks are well established [3, 4], e.g., [4] trains CNNs with 10,000 layers using an orthogonal-based initialization. Sec. 2 also claims that prior work does not establish when representational collapse begins with depth and how it can be systematically delayed, but this has been studied as rank collapse [5, 6], where [6] shows that batch normalization prevents it. None of these works are cited.

References:


[1] Van Gansbeke, Wouter, et al. "Scan: Learning to classify images without labels." European conference on computer vision. Cham: Springer International Publishing, 2020.

[2] Saxe, Andrew M., James L. McClelland, and Surya Ganguli. "Exact solutions to the nonlinear dynamics of learning in deep linear neural networks." arXiv preprint arXiv:1312.6120 (2013)

[3] Pennington, Jeffrey, Samuel Schoenholz, and Surya Ganguli. "Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice." Advances in neural information processing systems 30 (2017).

[4] Xiao, Lechao, et al. "Dynamical isometry and a mean field theory of cnns: How to train 10,000-layer vanilla convolutional neural networks." International conference on machine learning. PMLR, 2018.

[5] Dong, Yihe, Jean-Baptiste Cordonnier, and Andreas Loukas. "Attention is not all you need: Pure attention loses rank doubly exponentially with depth." International conference on machine learning. PMLR, 2021.

[6] Daneshmand, Hadi, et al. "Batch normalization provably avoids ranks collapse for randomly initialised deep networks." Advances in Neural Information Processing Systems 33 (2020): 18387-18398.

Questions for authors

See weaknesses.

Reviewer_xwPgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the relationship between network depth, geometric characterization of the deep representation, and clustering performance. The authors report that increasing depth is associated with greater Frobenius distance between input and deep representation projection operators and a sharp drop in clustering accuracy. To mitigate this degradation, they propose a structure-preserving regularizer and a complementary orthogonal initialization strategy.

Critical strengths

The connection between clustering-performance degradation and changes in subspace structure is interesting. The proposed regularizer is conceptually simple, straightforward for implementation, and potentially applicable to different architectures. The evaluation across multiple architectures and datasets provides useful evidence of the observed association and effects of the proposed regularization in some settings.

Critical weaknesses

The paper leaves several important methodological choices insufficiently explained, and its practical contribution is not convincingly established.

  1. The specification of the number of clusters/subspaces K is unclear. I didn't locate anywhere in this paper explaining whether K is assumed known, estimated from data, or obtained from ground-truth labels, nor how the number of output clusters is determined. Specifying K is a nontrivial and challenging in clustering, and its role is particularly important here because it also influences the projection rank. The authors should clarify this choice and acknowledge any use of ground-truth information.

  2. The projection rank selection lacks sufficient justification and sensitivity analysis. The implementation specifies K*d, but does not clearly explain how d is chosen. The authors should justify the selection procedure and evaluate sensitivity to different choices of the rank.

  3. The values of K in Table 1 require clarification. For example, the table reports K=20 for MNIST and K=40 for CIFAR-10, although both datasets have 10 ground-truth classes, to my knowledge. If these values represent something other than the number of clusters, their meaning and relationship to the evaluation labels should be explained.

  4. Table 1 lists CIFAR-10 with 400 samples and projection rank 400. This is problematic. The full column rank for both X and H can make the proposed distance zero between such representations. This requires clarification.

  5. The necessity of preserving the input row space is not sufficiently justified. Useful representation learning may require changing the principal direction of the data in their input space. The authors should explain more clearly their motivation for this proposed regularization, like when and why this preservation is desirable.

  6. The practical benefit and motivation remains unclear. The paper does not establish why greater depth is useful for the considered tasks in the paper. This research may be less valuable if a shallow network already achieves strong clustering performance. Comparisons with shallow models are needed to demonstrate a practical advantage beyond delaying degradation.

Minor comments:

  1. The Figure 3 caption states that the regularizer "never hurts" whereas Section 5.1 acknowledges that over-penalization degrades performance. These statements should be reconciled.
Questions for authors

Please refer to the questions raised in the Weaknesses section.

Reviewer_pqEUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates the phenomenon that as neural network depth increases, the model’s internal representations gradually lose the clustering structure of the input data, leading to sharp performance collapse. To address this issue, the paper introduces two main strategies: a structure-aware regularizer that penalizes the projection distance at each layer, and an orthogonal initialization strategy that provides a favorable geometric starting point. Extensive experiments demonstrate the feasibility of the proposed methods.

Critical strengths
  1. The authors conducted experiments at different network depths across multiple network architectures and datasets.

  2. The proposed method is decoupled from specific architectures and thus has general applicability.

Critical weaknesses
  1. Computing this regularization term requires obtaining the projection operator at each layer via singular value decomposition (SVD) or similar means. This may introduce significant additional computational and memory overhead into the training process, and the paper does not quantitatively evaluate this.

  2. The paper is incomplete. The appendix is still a template and has not been properly written. The entire manuscript needs to be polished.

  3. The authors overlooked performance comparisons with related work.

Questions for authors
  1. The paper fixes all layer widths to the input dimension to isolate depth effects. However, standard ResNet/CNN channels usually expand with depth, and this greater capacity helps preserve feature structure. Without the constant-width constraint, would the observed collapse still be significant? How much of it is a width-bottleneck artifact rather than an inherent flaw of depth?

  2. In practical training, especially when dealing with large batch sizes or ultra-high-dimensional feature inputs, what are the specific computational and time overheads of calculating the projection distance at each layer?

  3. In Section 2, “Related Work,” the authors explicitly mention that recent work in deep clustering has proposed methods to prevent representation collapse, such as hard regularization strategies for online clustering systems and other methods that enforce structural consistency. However, the paper does not compare the “projection distance regularizer” proposed in this work with these existing methods in terms of performance or computational efficiency.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.