acceptodds
Under review as a conference paper at ICLR 2027

How Width and Data Shape Generalization Scaling Laws in Quadratic Neural Networks

Abstract

Understanding how performance scales jointly with model size and data is a central problem in modern machine learning. Existing theoretical works on scaling laws typically describe generalization as a function of data or compute, often in fixed-feature or infinite-width regimes and for online SGD. Here, we instead study how generalization scales with the number of trainable parameters and the number of samples in a feature-learning model. We analyze -regularized empirical loss minimization for a quadratic two-layer network in a finite-sample setting with structured data. This setting allows for an explicit characterization of the generalization error as a function of the number of samples, model width, and regularization. Our results reveal a phase diagram with distinct scaling regimes as the number of parameters varies. In particular, the generalization error follows data-dependent power laws controlled by the spectral structure of the target. We further characterize the transitions between regimes, including the onset of interpolation, and their impact on generalization.

Primary area:
learning theory
Keywords:
scaling laws, quadratic networks, empirical risk minimization, approximate message passing

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_PNCdOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how network width, sample size, and regularization jointly affect generalization in a shallow quadratic neural network with Gaussian inputs and a power-law target spectrum. An exact matrix reformulation makes network width a rank constraint and weight decay a trace penalty. It derives a phase diagram describing different generalization regimes. It also compares optimal width selection, explicit regularization, and post-training spectral pruning. The theoretical predictions rely on a conjecture that is supported by numerical optimization of the original network.

Critical strengths

1 This paper gives details of how finite width interacts with sample size and regularization. Results 1 and 2 describe several distinct behaviors, including increasing error before interpolation, decreasing error after interpolation, and regimes where additional width provides no leading-order benefit.

2 The comparison of width selection, weight decay, and spectral pruning provides a useful account of regularization. This paper (Section 3) identifies conditions under which each approach achieves the Bayes-optimal error exponent. This gives a quantitative basis for understanding finite width as an implicit regularizer.

Critical weaknesses

1 The main results depend on Conjecture 1. My concern is that the paper should clarify which regions of the phase diagram satisfy the replica-symmetry stability condition and how the extension of the state-evolution equations to other scaling regimes is justified. These assumptions underlie both the phase diagram and the conclusions about optimal width. The numerical agreement provides useful support, but it does not establish the validity of these assumptions across all regimes.

2 The description of rank collapse appears inconsistent with the stated behavior of the training loss. On page 6, the paper states that the ERM solution sets all network weights to zero when α<1\alpha<1 and ρ<(3α−1)/5\rho<(3\alpha-1)/5. It then states that the training loss vanishes in all under-regularized phases satisfying α<min⁡(2,1+ρ)\alpha<\min(2,1+\rho). These conditions overlap, i.e., the two statements cannot both hold.

Questions for authors

Explain Weakness 2.

Have you observed the main qualitative findings, such as the non-monotonic dependence of test error on width and the regularizing effect of finite width, in other network architectures?

3 comments

W1: The main results depend on Conjecture 1. My concern is that the paper should clarify which regions of the phase diagram satisfy the replica-symmetry stability condition and how the extension of the state-evolution equations to other scaling regimes is justified. These assumptions underlie both the phase diagram and the conclusions about optimal width. The numerical agreement provides useful support, but it does not establish the validity of these assumptions across all regimes.

Reply to W1: We acknowledge that Conjecture 1 remains unproven, which is why we explicitly formulate it as a conjecture. As discussed in Appendix C, the replicon condition is satisfied when the rank constrain is inactive or truncates the spikes, but not when the rank constraint truncates the spectral bulk.

Nevertheless, we emphasize that our numerical experiments provide strong and direct evidence for the accuracy of Conjecture 1 across different regimes. The state-evolution equations quantitatively reproduce the full training and test error curves of the original nonconvex neural network, including interpolation peaks, non-monotonic behavior, and width-independent plateaus. Although the extension to general scaling regimes is not rigorously established, this extensive quantitative agreement supports its predictive validity.

W2: The description of rank collapse appears inconsistent with the stated behavior of the training loss. On page 6, the paper states that the ERM solution sets all network weights to zero when α<1\alpha < 1 and ρ<(3α−1)/5\rho < (3\alpha - 1)/5. It then states that the training loss vanishes in all under-regularized phases satisfying α<min⁡(2,1+ρ)\alpha < \min{(2, 1+\rho)}. These conditions overlap, i.e., the two statements cannot both hold.

Reply to W2: We thank the reviewer for pointing out this inconsistency. We have identified an error in the phase labeling of Figure 2. Now we have Phases Ia, Ib and Ic. In Phase Ic, all network weights are zero and the training loss does not vanish because of the high regularization. In Phases Ia and Ib, the training loss valishes and not all the weights are zero. We have also corrected the discussions of the rank collapse regime correspondingly.

These corrections do not change the reported leading-order test-error exponents, but distinguish regimes with identical test-error scaling and different spectral and training properties.

Q1: Explain Weakness 2.

Reply to Q1: Please see our response to Weakness 2.

Q2: Have you observed the main qualitative findings, such as the non-monotonic dependence of test error on width and the regularizing effect of finite width, in other network architectures?

Reply to Q2: We thank the reviewer for this question. Qualitatively similar non-monotonic width dependence has indeed been observed in more general architectures, including CNNs, ResNets, and Transformers (Nakkiran et al., 2020). These observations suggest that the competition between model capacity, noise overfitting, and generalization is not specific to quadratic networks.

Our work provides a quantitative spectral explanation of these effects in a tractable feature-learning model. We expect some of the underlying qualitative mechanisms, particularly the role of finite width as an implicit regularizer and the competition between feature recovery and noise fitting, to extend to broader architectures, e.g., the attention models in Appendix C.4. However, we have not directly verified the full phase diagram in more general architectures, and do not claim that the precise scaling exponents are universal. Establishing which aspects of these mechanisms persist in more general models is an interesting direction for future work.

I thank the authors for the clarification.

Weakness 2 has been addressed satisfactorily.

For Weakness 1, I understand that Conjecture 1 remains unproven but is supported by numerical evidence. Additional experiments on other architectures, such as CNNs, ResNets, and Transformers (as mentioned in the reply), would help evaluate whether the qualitative findings extend beyond quadratic networks.

I will keep my score.

We thank the reviewer for the follow-up and for acknowledging that Weakness 2 has been resolved.

We agree that experiments on broader architectures would be valuable for assessing the generality of our findings. To provide additional evidence beyond quadratic networks, we have added new experiments on a nonlinear softmax attention model in Figure 11. Specifically, we consider a single-head attention model with tied query and key matrices, trained on a noisy sequence-to-label task.

Figure 11 shows close agreement between the state-evolution predictions and numerical training. In particular, we observe the same low-width scaling E∼p1−2γ\mathcal E\sim p^{1-2\gamma} and sample-dependent large-width saturation as in the quadratic model. These results provide direct evidence that some of the scaling mechanisms extend to attention architectures.

We acknowledge that this is still a simplified attention model. Nevertheless, we believe these new experiments provide a useful first step toward understanding the generality of our results.

Finally, regarding Weakness 1, we would also like to highlight the new Theorem 1 in Appendix D, which rigorously establishes the scaling exponent of Phase VII independently of Conjecture 1. While the remaining phases are still non-rigorous, this provides an additional theoretical foundation for one of the newly identified scaling regimes.

Reviewer_tafeOpenReview ↗
Rating
4(+1)
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates scaling laws for excess test error as a function of training sample size nn, network width pp, and regularization strength λ\lambda in an ℓ2\ell_2-regularized finite-width quadratic neural network with Gaussian inputs and a teacher matrix S⋆S^\star with a power-law spectrum.

Using AMP and state evolution, the authors characterize the spectrum of the learned matrix S^\widehat S and the excess test error of the global ERM minimizer, deriving width–data scaling laws and a phase diagram. These theoretical predictions are empirically validated against numerical optimization results.

The paper shows that network width acts as an implicit regularizer, revealing non-monotonic generalization behavior in the under-regularized regime and a decreasing-then-saturating trend in the over-regularized regime. It further compares optimal width selection, weight decay, and post-training pruning, showing that these approaches can achieve Bayes-optimal excess-error scaling rates under appropriate conditions.

Critical strengths

(1) Quantitative Analysis of Generalization

The paper analyzes scaling regimes through the signal spikes and noise bulk of the spectrum of S^\widehat S, providing quantitative predictions for the scaling exponents and phase boundaries. This offers useful theoretical insights into the non-monotonic relationship between network width and generalization.

(2) Relationship Between Width Constraints and Explicit Regularization The paper shows that, under ℓ<α/2−1\ell < \alpha/2 - 1, selecting an optimal network width can achieve the same Bayes-optimal excess-error scaling exponent as optimally tuned explicit ℓ2\ell_2 regularization. This is a meaningful contribution, as it predicts that constraining network width can serve as an alternative regularization mechanism to explicit regularization.

Critical weaknesses

(1) Limited Novelty Compared to Prior Work

This paper extends the quadratic network setting and AMP/state-evolution framework of Defilippis et al. (2026b) to the p<dp<d regime, but its methodological novelty appears somewhat limited. In particular, the main approach is to extend the spectral analysis developed in prior work to a rank-constrained setting. Moreover, non-monotonic generalization behavior with respect to width and the effects of overparameterization have been investigated in studies such as Martin et al. (2026). While I acknowledge that this paper derives new width-dependent scaling exponents and phase boundaries, I find it unclear to what extent these results provide fundamentally new theoretical insights beyond prior work. This methodological overlap raises concerns about the paper’s novelty.

(2) Lack of Rigorous Justification for the Main Theoretical Results

Conjecture 1 provides an AMP/state-evolution characterization of the global ERM minimizer under a non-convex rank constraint in the p<dp<d regime. However, as the authors acknowledge, a rigorous proof of this characterization remains an open problem, and the analysis relies on a replica-symmetric prediction and a heuristic extension to non-standard scaling regimes. Since the main scaling laws and phase diagram in Results 1,2 are from Conjecture 1, this is not merely a technical detail but a concern that directly affects the theoretical justification of the main results. Taken together with Weakness (1), the lack of rigorous justification for the central characterization raises concerns about the strength of the paper's theoretical contribution, particularly given its limited methodological novelty relative to prior work.

Questions for authors

Q1. I would appreciate the authors' responses to the concerns regarding novelty and theoretical justification raised in the Weaknesses section.

Q2. Could the authors directly compare the spectral characterization S^∼(S⋆+δZ−λ~ϵId)+(p)\widehat S\sim(S^\star+\delta Z-\widetilde\lambda\epsilon I_d)_+^{(p)} in Conjecture 1 with the spectrum of the learned matrix from numerical experiments? For example, a comparison of eigenvalue histograms between theoretical predictions and numerical simulations would be helpful.

Q3. Could the authors provide further theoretical justification for Conjecture 1, for example by establishing it under additional mathematical assumptions, even conditionally?

6 comments

W1: This paper extends the quadratic network setting and AMP/state-evolution framework of Defilippis et al. (2026b) to the p<dp < d regime, but its methodological novelty appears somewhat limited. In particular, the main approach is to extend the spectral analysis developed in prior work to a rank-constrained setting. Moreover, non-monotonic generalization behavior with respect to width and the effects of overparameterization have been investigated in studies such as Martin et al. (2026). While I acknowledge that this paper derives new width-dependent scaling exponents and phase boundaries, I find it unclear to what extent these results provide fundamentally new theoretical insights beyond prior work. This methodological overlap raises concerns about the paper’s novelty.

Reply to W1: We agree that our analysis builds on the AMP/state-evolution framework developed in previous works. However, we believe that the novelty of a theoretical contribution should be assessed not only by the methods employed, but also by the new results and insights obtained.

Defilippis et al. (2026b) study the full-width regime (p\geq d), while our analysis incorporates an active rank constraint and reveals new width-dependent scaling regimes (notably Phases III, V, and VII), together with their phase boundaries and spectral interpretations. Martin et al. (2026) study gradient-flow dynamics in the regime (p=\Theta(d)), (n=\Theta(d^2)). In contrast, our analysis considers the joint scaling (p=\Theta(d^\rho)), (n=\Theta(d^\alpha)).

Importantly, our results go beyond reproducing known qualitative phenomena. They quantitatively characterize how width-dependent approximation and overfitting effects interact, identify optimal width scalings, and establish conditions under which width selection or spectral pruning achieves Bayes-optimal scaling rates. We believe these constitute meaningful new theoretical insights despite the methodological connections with prior work.

W2: Conjecture 1 provides an AMP/state-evolution characterization of the global ERM minimizer under a non-convex rank constraint in the p<dp<d regime. However, as the authors acknowledge, a rigorous proof of this characterization remains an open problem, and the analysis relies on a replica-symmetric prediction and a heuristic extension to non-standard scaling regimes. Since the main scaling laws and phase diagram in Results 1,2 are from Conjecture 1, this is not merely a technical detail but a concern that directly affects the theoretical justification of the main results. Taken together with Weakness (1), the lack of rigorous justification for the central characterization raises concerns about the strength of the paper's theoretical contribution, particularly given its limited methodological novelty relative to prior work.

Reply to W2: We acknowledge that Conjecture 1 remains unproven, which is why we explicitly state it as a conjecture. Nevertheless, several components of the analysis are exact, including the reformulation as a rank-constrained PSD matrix estimation problem and the solution of the corresponding spectral denoising problem (Appendix B).

More importantly, as discussed in our response to the Reviewer D7Z3, Conjecture 1 is extensively validated through direct numerical optimization of the original nonconvex network. The state-evolution equations accurately predict the full training and test error curves across different widths, sample sizes, regularization strengths, and target spectra. We believe this detailed quantitative agreement provides strong evidence for the conjecture, even though its rigorous justification remains an important open problem.

Q1: I would appreciate the authors' responses to the concerns regarding novelty and theoretical justification raised in the Weaknesses section.

Reply to Q1: Please see our responses to Weaknesses 1 and 2.

Q2: Could the authors directly compare the spectral characterization in Conjecture 1 with the spectrum of the learned matrix from numerical experiments? For example, a comparison of eigenvalue histograms between theoretical predictions and numerical simulations would be helpful.

Reply to Q2: We thank the reviewer for this useful suggestion. Conjecture 1 predicts not only the generalization error but also the spectrum of the learned matrix. To directly test this prediction, we add Figure 5 to compare the eigenvalues of S^=W⊤W/pd\widehat S=W^\top W/\sqrt{pd}, obtained by training the original neural network, with those of the theoretical prediction. Figure 5 provides a direct test of the spectral characterization, complementary to our existing comparisons of training and test errors.

Q3: Could the authors provide further theoretical justification for Conjecture 1, for example by establishing it under additional mathematical assumptions, even conditionally?

Reply to Q3: We agree that a rigorous justification of Conjecture 1 would be valuable. As explained in Appendix B, the rank-constrained spectral denoiser is derived exactly using von Neumann's trace inequality. The remaining challenge is to establish that the associated state evolution characterizes the global minimizer of the nonconvex ERM non-asymptotically. This is a challenging task and requires extending existing rigorous AMP results beyond the convex setting and the asymptotical limit.

Nevertheless, some limiting behaviors can be justified independently of Conjecture 1. For example, in the limit n→∞n\to\infty at fixed d,pd,p, the optimal rank-pp estimator is exactly the truncated eigendecomposition of S⋆S^\star. Consequently, $\mathcal E(p)=\frac{1}{\zeta(2\gamma)}\sum_{i=p+1}^{d}i^{-2\gamma} \sim\frac{p^{1-2\gamma}}{(2\gamma-1)\zeta(2\gamma)}$ for 1≪p≪d1\ll p\ll d. This rigorously recovers the exponent predicted in Phase VII, independently of the replica analysis.

While this does not establish the full finite-sample state evolution, it provides an additional theoretical consistency check. Together with the exact spectral denoiser, the existing rigorous full-width results, and the close numerical agreement, it supports the proposed characterization.

Dear Authors,

Thank you for your helpful responses.

I agree that the rank constraint (p<dp < d) introduces a new theoretical challenge, and I acknowledge the novelty and importance of finite-width, non-asymptotic regime problem. My concerns regarding the differences between this work and Martin et al. (2026) have been resolved.

However, I have several remaining questions regarding soundness:

Q1. Joint Scaling

In the response to [W1], the authors explain: "Martin et al. (2026) study gradient-flow dynamics in the regime p=Θ(d)p=\Theta(d), n=Θ(d2)n=\Theta(d^2). In contrast, our analysis considers the joint scaling p=Θ(dρ)p=\Theta(d^\rho), n=Θ(dα)n=\Theta(d^\alpha)."

But, in the response to Q3, the authors discussed the validity of the predictions in the limit n→∞n \to \infty, (not joint scaling pp and nn). Given that the authors emphasize joint scaling as a key distinction from prior work, I find it unclear how a justification based on the limit n→∞n \to \infty alone supports the joint scaling analysis.

Q2. Choice of Analytical Approach

I now recognize that this work involves a distinct mathematical structure and leads to different conclusions from prior work. However, I remain curious about why the authors chose to adopt essentially the same AMP/replica approach despite the different mathematical structure introduced by the rank constraint. I have reviewed the related work, but I find that the manuscript provides insufficient explanation of why AMP/replica methods are an appropriate approach for studying non-convex problems in the non-asymptotic regime. Could the authors provide further justification for this methodological choice, particularly regarding why the approach is expected to remain valid under the rank constraint?

Q3. Theoretical Justification

I acknowledge that rigorously analyzing the low-rank-constrained, non-convex problem in the non-asymptotic regime is a highly challenging and important mathematical problem. I therefore withdraw my earlier concern that the lack of a rigorous proof merely reflects a technical limitation of the analysis.

However, the following statement in the related work section (lines 439–441) does not fully address my concern about the justification of Conjecture 1:

"This is in line with a common route in high-dimensional statistics, where heuristic predictions (Bordelon et al., 2020; Cui et al., 2021; Simon et al., 2023) often precede and guide later rigorous analyses (Cheng & Montanari, 2024; Misiakiewicz & Saeed, 2024; Defilippis et al., 2024)."

While I understand that heuristic predictions have historically preceded rigorous results, this precedent does not itself provide a theoretical justification for the validity of Conjecture 1 in the present setting. I believe this point deserves a more detailed explanation.

Furthermore, my earlier question in Q3 regarding "under additional mathematical assumptions" was not intended to refer to additional assumptions underlying the asymptotic approach, as discussed in the authors' response.

Rather, I was asking whether additional structural assumptions could make the non-convex problem mathematically tractable. For instance, there are approaches to non-convex problems that exploit weak convexity, as discussed in the context of Vilucchio et al., which is cited in the manuscript.

I was wondering whether similar assumptions or conditions, such as restricted strong convexity, could be introduced to establish rigorous results, even if only in a more restricted setting.

I would appreciate clarification on whether such approaches could be applicable to the present problem and, if not, what mathematical obstacles prevent their application.

Q5. Additional Experiments

If a rigorous theoretical justification is currently out of reach, I would appreciate more detailed empirical evidence supporting the soundness of Conjecture 1.

Thank you for adding the eigenvalue diagram to Figure 5.

First, I would like to request additional experiments with smaller sample sizes nn. The current experiments consider n=8844,33064,219316n=8844, 33064, 219316. As discussed in the response to Q3, the authors have provided evidence supporting the prediction as n→∞n \to \infty. However, I believe it is also important to examine whether Conjecture 1 remains valid for smaller nn,(e.g., n=500,1000n=500, 1000) particularly given the paper's emphasis on finite-sample scaling behavior.

Second, Defilippis et al. present experiments with different input dimensions, such as d=100,200,400,800d=100, 200, 400, 800, in their Figure 3. Since the present work specifically investigates the finite-width regime, I wonder whether additional eigenvalue diagrams for smaller values of pp and dd could provide more direct empirical evidence for Conjecture 1.

I believe these additional experiments would help clarify the range of validity of the conjecture and strengthen the empirical support for the proposed theoretical framework.

Reviewer_D7Z3OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how neural network width, training sample size, regularization, and target data structure jointly affect generalization in a two-layer quadratic neural network. By reformulating the network as a rank-constrained positive semidefinite matrix estimation problem, the authors analyze the excess test error of the regularized empirical risk minimizer under Gaussian inputs and a power-law target spectrum.

Using a conjectured state-evolution characterization based on replica and approximate message passing methods, the paper derives width- and sample-dependent scaling laws and identifies multiple generalization regimes. These include low-width approximation error, noise overfitting, interpolation-related behavior, and large-width plateaus. The analysis further suggests that appropriately choosing network width, explicit regularization, or post-training spectral pruning can achieve Bayes-optimal scaling rates under suitable conditions.

The theoretical predictions are evaluated through numerical experiments that train the original quadratic network using L-BFGS on synthetic data. The reported results show close agreement between the state-evolution predictions and empirical training/test errors across different widths, sample sizes, spectral decay exponents, and regularization strengths.

Critical strengths
  1. The paper develops a comprehensive quantitative description of how width, sample size, regularization, and target spectral structure interact in a finite-width quadratic feature-learning model. Unlike analyses restricted to fixed representations or full-width regimes, it explicitly incorporates the rank constraint induced by network width and identifies multiple scaling regimes with corresponding power-law exponents. This extends the scope of existing generalization analyses and provides a more detailed characterization of width-dependent behavior.

  2. By reformulating the quadratic network as a rank-constrained PSD matrix estimation problem, the paper provides an intuitive spectral interpretation of network width as a form of implicit regularization. The analysis explains the competition between recovering informative target directions and fitting noise-dominated spectral components, and connects this mechanism to optimal width selection, explicit regularization, and post-training pruning. The numerical experiments show close agreement with the state-evolution predictions across several parameter settings, lending empirical support to the proposed framework.

Critical weaknesses
  1. The principal scaling-law results and the associated phase diagram rely on Conjecture 1, whose validity has not been established for the nonconvex, rank-constrained ERM problem or for the general scaling regimes considered in the paper. Appendix B explicitly acknowledges these two assumptions, while Appendix C introduces an additional semicircular-bulk-plus-spikes approximation. Although the authors provide a replicon analysis, the resulting condition is only necessary and does not establish the validity of the replica-symmetric characterization across all phases. Furthermore, the numerical validation is concentrated on finite-dimensional comparisons, rather than directly verifying the predicted asymptotic exponents across increasing input dimensions. Since the main contributions depend critically on these assumptions, their insufficient validation raises concerns about the reliability and scope of the proposed scaling laws.

  2. The paper addresses the broad question of how model size and data structure determine neural network generalization, but its analysis is restricted to a highly stylized quadratic teacher–student model with Gaussian inputs, a power-law target spectrum, squared loss, and a fixed second layer. Many of the qualitative behaviors emphasized in the paper, including double descent, width-dependent overfitting, and implicit regularization, have already been extensively studied. The principal novelty therefore appears to lie in obtaining a unified, model-specific quantitative description rather than identifying fundamentally new generalization mechanisms. While such a characterization can be valuable, the paper provides limited evidence that the derived scaling laws or their underlying mechanisms extend beyond this particular setting. This restricts the demonstrated significance of the contribution for broader feature-learning architectures.

Questions for authors
  1. The main results depend on Conjecture 1, which assumes that the replica-symmetric state-evolution characterization remains valid for nonconvex rank-constrained ERM and across general scaling regimes. Appendix C also uses a semicircular-bulk-plus-spikes approximation. Could the authors clarify which of the newly identified phases are supported by the replicon stability analysis and the assumptions underlying the spectral approximation, and which remain purely heuristic? In particular, how should the validity of the predictions be assessed in regimes where the rank constraint truncates the spectral bulk? Could the authors provide more direct numerical evidence for the predicted asymptotic exponents by varying the input dimension dd, rather than primarily comparing fixed-dimensional curves?

  2. The paper provides a unified quantitative characterization of several known qualitative phenomena, including width-dependent overfitting, double descent, and implicit regularization. Could the authors clarify which scaling regimes and conclusions are fundamentally new relative to existing analyses, particularly Defilippis et al. (2026b), Ren et al. (2025), Ben Arous et al. (2026), and Martin et al. (2026)? Moreover, which aspects of the proposed scaling behavior are expected to remain valid beyond the specific quadratic teacher–student model, and which depend critically on the PSD rank-constrained reformulation and power-law target assumption? A clearer separation of model-specific results from potentially generalizable mechanisms would help assess the broader significance of the contribution.

7 comments

W1: The principal scaling-law results and the associated phase diagram rely on Conjecture 1, whose validity has not been established for the nonconvex, rank-constrained ERM problem or for the general scaling regimes considered in the paper. Appendix B explicitly acknowledges these two assumptions, while Appendix C introduces an additional semicircular-bulk-plus-spikes approximation. Although the authors provide a replicon analysis, the resulting condition is only necessary and does not establish the validity of the replica-symmetric characterization across all phases. Furthermore, the numerical validation is concentrated on finite-dimensional comparisons, rather than directly verifying the predicted asymptotic exponents across increasing input dimensions. Since the main contributions depend critically on these assumptions, their insufficient validation raises concerns about the reliability and scope of the proposed scaling laws.

Reply to W1: We thank the reviewer for raising these concerns. While we agree that Conjecture 1 remains unproven, we would like to clarify that our numerical experiments already provide strong and direct evidence for its accuracy, and that fixing the input dimension d is a deliberate choice motivated by the main questions of the paper.

  1. Numerical validation of Conjecture 1.

Conjecture 1 provides a quantitative prediction for the ERM, not merely its asymptotic scaling exponents. Given (n,p,d,λ~,γ,Δ)(n,p,d,\widetilde\lambda,\gamma,\Delta), the state-evolution equations predict the training and test errors. Our numerical experiments directly test these predictions by optimizing the original nonconvex quadratic neural network using L-BFGS. Figures 1, 3, and 6–8 show close quantitative agreement across a broad range of sample sizes, widths, regularization strengths, target spectral exponents, and noise levels. This agreement includes non-monotonic generalization curves, interpolation-related behavior, and large-width plateaus.

We emphasize that matching the entire generalization curves is a substantially stronger test than matching asymptotic slopes alone. The agreement is particularly informative in the rank-constrained regimes, where the validity of the replica-symmetric characterization is not guaranteed by existing theoretical results.

  1. Why we focus on scaling with (n,p,λ~)(n,p,\widetilde\lambda) rather than d.

The primary objective of our work is to understand how generalization depends on training sample size, network width, and regularization. In typical applications, the input dimension is determined by the dataset, whereas the sample size, model width, and regularization are the relevant variables for model selection. For this reason, our experiments deliberately fix d and systematically vary n,p,λ~n,p,\widetilde\lambda. This directly tests the scaling relationships of interest, including the width-dependent laws E∼p1−2γ\mathcal E\sim p^{1-2\gamma}, E∼pd/n\mathcal E\sim pd/n, and E∼n/(pd)\mathcal E\sim n/(pd) in the corresponding phases.

That being said, we add Figure 10 to compare the theoretical exponents with the numerics for different dd. Figure 10 demonstrates that when dd gets larger, the numerics get closer to the predicted exponents. This provides an additional verification for our theory.

  1. Role of the spectral approximation.

We would also like to clarify that the semicircular-bulk-plus-spikes approximation in Appendix C is not used to generate the numerical state-evolution curves shown in our figures. These curves are obtained by solving Conjecture 1 using the finite-dimensional spectrum of S⋆+δZS^\star+\delta Z, as detailed in Appendix D.1. Consequently, the agreement between state evolution and neural-network training provides evidence for Conjecture 1 independently of the additional spectral approximation used to derive the analytical scaling exponents.

In conclusion, we would like to emphasize that the absence of a rigorous proof should be distinguished from the absence of evidence. The close agreement with independently optimized neural networks across the different regimes provides substantial empirical support for the validity and predictive usefulness of the conjecture.

W2: The paper addresses the broad question of how model size and data structure determine neural network generalization, but its analysis is restricted to a highly stylized quadratic teacher–student model with Gaussian inputs, a power-law target spectrum, squared loss, and a fixed second layer. Many of the qualitative behaviors emphasized in the paper, including double descent, width-dependent overfitting, and implicit regularization, have already been extensively studied. The principal novelty therefore appears to lie in obtaining a unified, model-specific quantitative description rather than identifying fundamentally new generalization mechanisms. While such a characterization can be valuable, the paper provides limited evidence that the derived scaling laws or their underlying mechanisms extend beyond this particular setting. This restricts the demonstrated significance of the contribution for broader feature-learning architectures.

Reply to W2: We agree that double descent, overfitting, and implicit regularization are known phenomena. However, we believe that obtaining a quantitative characterization of how these phenomena emerge and interact in a feature-learning model is itself a significant theoretical contribution. Our goal is not to discover entirely new behaviors, but to understand precisely how generalization depends jointly on model width, sample size, regularization, and target structure.

In particular, previous analyses of ERM scaling laws mainly consider the full-width regime (p\geq d), while studies of finite-width feature learning often focus on training dynamics. By incorporating the rank constraint, we identify several new scaling regimes. These regimes characterize how increasing width can successively improve generalization, worsen it through noise overfitting, and improve it again beyond interpolation. Beyond identifying these exponents, we derive the phase boundaries and characterize the optimal width, showing that appropriately choosing the network width can achieve the same Bayes-optimal scaling rate as optimal explicit regularization or spectral pruning. These quantitative conclusions are not available from the previous full-width analyses.

Regarding generality, we deliberately study a minimal yet nontrivial feature-learning model to isolate the underlying mechanisms. While the precise power-law exponents depend on the target spectrum and model, the competition between learning informative directions and fitting noise is not specific to power-law targets. We do not claim that the precise exponents universally extend to deep architectures. Rather, our contribution is to provide a detailed, quantitatively predictive framework explaining how width, data availability, and regularization interact in a genuine feature-learning setting. We believe that establishing such a characterization in a tractable model is a meaningful step toward understanding the corresponding mechanisms in more general architectures.

Q1: The main results depend on Conjecture 1, which assumes that the replica-symmetric state-evolution characterization remains valid for nonconvex rank-constrained ERM and across general scaling regimes. Appendix C also uses a semicircular-bulk-plus-spikes approximation. Could the authors clarify which of the newly identified phases are supported by the replicon stability analysis and the assumptions underlying the spectral approximation, and which remain purely heuristic? In particular, how should the validity of the predictions be assessed in regimes where the rank constraint truncates the spectral bulk? Could the authors provide more direct numerical evidence for the predicted asymptotic exponents by varying the input dimension (d), rather than primarily comparing fixed-dimensional curves?

Reply to Q1: As discussed in our response to Weakness 1, we believe that the excellent quantitative agreement between Conjecture 1 and numerical experiments provides strong evidence for the accuracy of our predictions, including in the nonconvex rank-constrained regimes.

Regarding the replicon stability, we agree that we do not establish replica-symmetric validity in all phases. This is why we explicitly formulate our characterization as a conjecture. Nevertheless, the predictions in these regimes are strongly supported by their close agreement with the original neural-network experiments.

We would like to clarify that the semicircular bulk and BBP outlier mapping are not ad hoc assumptions, but are grounded in established random matrix theory. In particular, when the number of spikes satisfies (K=o(d)), the limiting bulk spectral distribution is semicircular, while the locations of separated outliers are described by BBP-type results (Huang, 2018). These results motivate the asymptotic spectral description used in Appendix C.

Moreover, the theoretical curves in our figures do not rely on this asymptotic spectral description. They are obtained by directly solving Conjecture 1 using finite-dimensional GOE matrices. Their excellent agreement with the original neural-network experiments therefore provides independent evidence for the conjecture, including when the rank constraint truncates the spectral bulk.

As explained in our response to Weakness 1, fixing dd and varying n,p,λ~n,p,\widetilde\lambda directly tests the scaling relationships of primary interest. Moreover, we add Figure 10 to validate the scaling exponents across dd. We believe this provides more comprehensive evidence.

Comments on the paper

1 comment

We thank all reviewers for their constructive feedback. In response to their comments, we have made five main revisions to the manuscript:

  1. Direct spectral validation (new Figure 5). We added numerical experiments comparing the spectrum of the learned weight matrix with the theoretical predictions from Conjecture 1, providing additional evidence for the proposed spectral characterization.

  2. Finite-size validation (new Figure 10). We added experiments across different input dimensions dd, demonstrating the agreement between numerical results and the predicted width-dependent scaling laws, with improving agreement as dd increases.

  3. Clarification of the rank-collapse regime (new Phase Ic). We corrected the phase diagram and corresponding discussion by distinguishing Phase Ic from Phases Ia and Ib, where in Phase Ic and learned weights are zero, and in Phases Ia and Ib the learned weights are nonzero despite having the same leading-order test error.

  4. A rigorous proof for the scaling exponent in Phase VII (new Appendix D).

  5. An experiment for single-head softmax attention (new Figure 11).

All changes are highlighted in blue, These revisions provide additional theoretical and numerical support and clarify the interpretation of the phase diagram. We hope they address the reviewers' main concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.