- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how network width, sample size, and regularization jointly affect generalization in a shallow quadratic neural network with Gaussian inputs and a power-law target spectrum. An exact matrix reformulation makes network width a rank constraint and weight decay a trace penalty. It derives a phase diagram describing different generalization regimes. It also compares optimal width selection, explicit regularization, and post-training spectral pruning. The theoretical predictions rely on a conjecture that is supported by numerical optimization of the original network.
Critical strengths
1 This paper gives details of how finite width interacts with sample size and regularization. Results 1 and 2 describe several distinct behaviors, including increasing error before interpolation, decreasing error after interpolation, and regimes where additional width provides no leading-order benefit.
2 The comparison of width selection, weight decay, and spectral pruning provides a useful account of regularization. This paper (Section 3) identifies conditions under which each approach achieves the Bayes-optimal error exponent. This gives a quantitative basis for understanding finite width as an implicit regularizer.
Critical weaknesses
1 The main results depend on Conjecture 1. My concern is that the paper should clarify which regions of the phase diagram satisfy the replica-symmetry stability condition and how the extension of the state-evolution equations to other scaling regimes is justified. These assumptions underlie both the phase diagram and the conclusions about optimal width. The numerical agreement provides useful support, but it does not establish the validity of these assumptions across all regimes.
2 The description of rank collapse appears inconsistent with the stated behavior of the training loss. On page 6, the paper states that the ERM solution sets all network weights to zero when and . It then states that the training loss vanishes in all under-regularized phases satisfying . These conditions overlap, i.e., the two statements cannot both hold.
Questions for authors
Explain Weakness 2.
Have you observed the main qualitative findings, such as the non-monotonic dependence of test error on width and the regularizing effect of finite width, in other network architectures?
3 comments
I thank the authors for the clarification.
Weakness 2 has been addressed satisfactorily.
For Weakness 1, I understand that Conjecture 1 remains unproven but is supported by numerical evidence. Additional experiments on other architectures, such as CNNs, ResNets, and Transformers (as mentioned in the reply), would help evaluate whether the qualitative findings extend beyond quadratic networks.
I will keep my score.
We thank the reviewer for the follow-up and for acknowledging that Weakness 2 has been resolved.
We agree that experiments on broader architectures would be valuable for assessing the generality of our findings. To provide additional evidence beyond quadratic networks, we have added new experiments on a nonlinear softmax attention model in Figure 11. Specifically, we consider a single-head attention model with tied query and key matrices, trained on a noisy sequence-to-label task.
Figure 11 shows close agreement between the state-evolution predictions and numerical training. In particular, we observe the same low-width scaling and sample-dependent large-width saturation as in the quadratic model. These results provide direct evidence that some of the scaling mechanisms extend to attention architectures.
We acknowledge that this is still a simplified attention model. Nevertheless, we believe these new experiments provide a useful first step toward understanding the generality of our results.
Finally, regarding Weakness 1, we would also like to highlight the new Theorem 1 in Appendix D, which rigorously establishes the scaling exponent of Phase VII independently of Conjecture 1. While the remaining phases are still non-rigorous, this provides an additional theoretical foundation for one of the newly identified scaling regimes.
Reply to W1: We acknowledge that Conjecture 1 remains unproven, which is why we explicitly formulate it as a conjecture. As discussed in Appendix C, the replicon condition is satisfied when the rank constrain is inactive or truncates the spikes, but not when the rank constraint truncates the spectral bulk.
Nevertheless, we emphasize that our numerical experiments provide strong and direct evidence for the accuracy of Conjecture 1 across different regimes. The state-evolution equations quantitatively reproduce the full training and test error curves of the original nonconvex neural network, including interpolation peaks, non-monotonic behavior, and width-independent plateaus. Although the extension to general scaling regimes is not rigorously established, this extensive quantitative agreement supports its predictive validity.
Reply to W2: We thank the reviewer for pointing out this inconsistency. We have identified an error in the phase labeling of Figure 2. Now we have Phases Ia, Ib and Ic. In Phase Ic, all network weights are zero and the training loss does not vanish because of the high regularization. In Phases Ia and Ib, the training loss valishes and not all the weights are zero. We have also corrected the discussions of the rank collapse regime correspondingly.
These corrections do not change the reported leading-order test-error exponents, but distinguish regimes with identical test-error scaling and different spectral and training properties.
Reply to Q1: Please see our response to Weakness 2.
Reply to Q2: We thank the reviewer for this question. Qualitatively similar non-monotonic width dependence has indeed been observed in more general architectures, including CNNs, ResNets, and Transformers (Nakkiran et al., 2020). These observations suggest that the competition between model capacity, noise overfitting, and generalization is not specific to quadratic networks.
Our work provides a quantitative spectral explanation of these effects in a tractable feature-learning model. We expect some of the underlying qualitative mechanisms, particularly the role of finite width as an implicit regularizer and the competition between feature recovery and noise fitting, to extend to broader architectures, e.g., the attention models in Appendix C.4. However, we have not directly verified the full phase diagram in more general architectures, and do not claim that the precise scaling exponents are universal. Establishing which aspects of these mechanisms persist in more general models is an interesting direction for future work.