- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper develops a general DMFT in an abstract spiked matrix setting. It then applies this theory in two important cases: Infinite Width network (with constant input dimension, batch size and training time) and deep linear networks (with width, input dimension, batch size and training size large and comparable). These results are then used to study hyperparameter transfer. The authors support their theoretical findings with numerical experiments.
Critical strengths
The problem of identifying outliers in the dynamical setting of training neural networks is hard in general. In that sense, the authors not only develop a theory that works, but they also map two seemingly different cases in the same framework of spiked random matrices with coupled bulk and spikes. The theoretical contribution itself is already enough to justify acceptance. Beyond the theoretical contribution, the paper is clearly written and understandable (although introducing DMFT is never an easy task in a paper).
Critical weaknesses
None.
Questions for authors
- Through the paper the authors always assume that initialization of the parameters is done with a particular mean and variance. I imagine there is a large family of distributions that match the necessary assumptions, but I don't manage to find it in the main text nor the appendix.
- Appendix G states that the ImageNet and GPT experiments, and in particular Figures 6 and 7, use NNs trained by Adam, while the developed theory is written for (S)GD . However, the authors claim that the difference comes from the large output channel. Why is the output dimension the "main suspect" and not the training algorithm, which could also be modifying the bulk of the matrix spectrum?
Critical Strengths
Thank you for such an encouraging review. Bringing both infinite-width nonlinear networks and deep linear networks in the proportional regime into the same framework under SGD dynamics was one of our main goals, so we especially appreciate that you highlighted this connection.
Questions