- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes a simple regularizer, the sum over layers of squared Frobenius distances between the projection matrices of the input and of each layer's representation, and an orthogonal initialization that delay the collapse in clustering performance, enabling the training of deeper networks for clustering. The initialization is evaluated on synthetic data, while the regularizer is evaluated on synthetic data and on established image-clustering datasets.
Critical strengths
- S1: Overall, the writing is clear and the paper is easy to follow.
- S2: The proposed regularizer is simple and easy to implement.
Critical weaknesses
-
W1: Methodology and experiment setup: The authors train different architectures (e.g., residual networks) from scratch, but hyperparameters such as the number of epochs, batch size, and optimizer are not provided. Each architecture uses a single learning rate across all datasets and depths, which makes it difficult to distinguish an actual performance collapse phenomena from undertrained networks. This is evident for 18-layer networks, which perform poorly in the paper, whereas resnet-18 trained from scratch is commonly used in clustering and achieves strong performance (e.g., 81.8% accuracy on CIFAR-10 [1]).
-
W2: Inconsistencies and mistakes in the paper: Table 1 summarizes the datasets, but the reported numbers of clusters do not match the ground truth, e.g.: MNIST =20, CIFAR-10 =40. Figure 3 says: "Remarkably, our regularizer never hurts, regardless of the choice of ", but Sec. 5.1. contradicts it: "Beyond a certain threshold, further increasing begins to negatively affect performance".
-
W3: Novelty and related work: The orthogonal initialization proposed in Sec. 5 is presented without citation, although it follows the standard orthogonal initialization [2], which is implemented in PyTorch. Its benefits for deep networks are well established [3, 4], e.g., [4] trains CNNs with 10,000 layers using an orthogonal-based initialization. Sec. 2 also claims that prior work does not establish when representational collapse begins with depth and how it can be systematically delayed, but this has been studied as rank collapse [5, 6], where [6] shows that batch normalization prevents it. None of these works are cited.
References:
[1] Van Gansbeke, Wouter, et al. "Scan: Learning to classify images without labels." European conference on computer vision. Cham: Springer International Publishing, 2020.
[2] Saxe, Andrew M., James L. McClelland, and Surya Ganguli. "Exact solutions to the nonlinear dynamics of learning in deep linear neural networks." arXiv preprint arXiv:1312.6120 (2013)
[3] Pennington, Jeffrey, Samuel Schoenholz, and Surya Ganguli. "Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice." Advances in neural information processing systems 30 (2017).
[4] Xiao, Lechao, et al. "Dynamical isometry and a mean field theory of cnns: How to train 10,000-layer vanilla convolutional neural networks." International conference on machine learning. PMLR, 2018.
[5] Dong, Yihe, Jean-Baptiste Cordonnier, and Andreas Loukas. "Attention is not all you need: Pure attention loses rank doubly exponentially with depth." International conference on machine learning. PMLR, 2021.
[6] Daneshmand, Hadi, et al. "Batch normalization provably avoids ranks collapse for randomly initialised deep networks." Advances in Neural Information Processing Systems 33 (2020): 18387-18398.
Questions for authors
See weaknesses.