- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper constructs wide neural networks with the same limiting eigenvalue distribution, yet the same learning rate is stable for one and unstable for the other and validates this phenomenon experimentally. This is because the stable learning rate depends on the largest eigenvalue rather than the limiting eigenvalue distribution. The authors also give conditions under which the output layer preserves or the difference between the largest eigenvalue and the boundary of the limiting spectrum.
Critical strengths
This paper constructs wide neural networks with the same limiting eigenvalue distribution, yet the same learning rate is stable for one and unstable for the other, which raises an important research question that which part of the weights spectra is useful to understand neural network training.
Critical weaknesses
-
Unclear novelty and significance. The main message of the paper appears to be that two neural networks can have the same limiting eigenvalue distribution but different stable learning rates. However, this is a direct consequence of the fact that the stability threshold depends on the largest eigenvalue rather than the limiting spectral distribution. It is already well known that a few outlier eigenvalues can have a significant impact on optimization. Therefore, I find the main contribution of this work unclear. The paper does not sufficiently explain what new insights it provides beyond existing results on spiked random matrices and spectral outliers.
-
Poor presentation and unnatural problem formulation. Section 3 is particularly difficult to follow. The authors introduce a large number of variables, matrices, and technical definitions without clearly explaining their roles. This makes it difficult to understand the main results. The asymptotic limits in these theorems are not clearly defined. Moreover, The theoretical models are not introduced in a way that makes their connection to neural networks immediately clear. For example, Section 3.1 starts with a gated Gaussian random matrix. I believe it would be much more natural to first specify the neural network architecture, activation functions, trainable parameters, and loss function, and then derive the corresponding matrices.
-
Limited connection to actual neural network training. Much of the theoretical analysis concerns the spectra and outliers of certain random matrices, while the connection to training is mainly established through the stability condition determined by the largest NTK eigenvalue. It is unclear how these results inform the training dynamics of neural networks in more realistic learning problems.
Questions for authors
-
Since the importance of spectral outliers for stability is already well understood, what is the main novel contribution of this paper?
-
Can the authors substantially simplify and reorganize Section 3? In particular, can the models be formulated directly as neural networks, with the corresponding Jacobians and NTKs derived afterward?
-
How relevant are the theoretical results to actual neural network training? Can the authors provide evidence that the proposed mechanism affects training in a nontrivial supervised learning setting?