acceptodds
Under review as a conference paper at ICLR 2027

Depth Improves Training-Time Scaling for Difficult Tasks in Deep Linear Networks

Abstract

Neural scaling laws describe the predictable power-law dependence of model performance on training time, dataset size, and model scale, with the exponents quantifying how performance changes with resources. Recent theory has shown how scaling exponents are shaped by the data distribution and the structure of the learning task, but whether network depth can improve training-time scaling has remained unsettled. Through a refined analysis of deep linear networks, we clarify that the task spectrum determines when increasing network depth improves training-time scaling. For spectrally difficult tasks, increasing depth improves the asymptotic training-time scaling exponent, whereas for easier tasks, the exponent remains independent of depth, with depth affecting only a logarithmic correction at the boundary between these regimes. Remarkably, we find that a reparameterization of time governed by the predictor norm provides a unified description of the dynamics across task difficulties. Under this reparameterization, the loss follows a common scaling law whose exponent is independent of depth. Numerical experiments on a task derived from benchmark image data and with nonlinear networks exhibit qualitatively consistent task-dependent effects of depth on training-time scaling. Our findings advance the theoretical understanding of how task structure shapes the effects of network architecture on learning and offer a basis for studying scaling behavior across broader classes of neural networks.

Primary area:
learning theory
Keywords:
deep linear networks, neural scaling laws, network depth, task spectra, learning dynamics

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_9KW2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper analyses deep linear networks trained by population gradient flow on power-law tasks. It shows that depth improves the training-time exponent when the task is spectrally difficult (p < 1), while the exponent is depth-independent in a reparameterised time defined by the predictor norm.

Critical strengths
  • The analysis is rigorous and clearly organised into three regimes.
  • The time reparameterisation gives a neat unified picture, and the paper identifies a gap in prior arguments.
Critical weaknesses
  • Depth effects and scaling in deep linear networks have been studied extensively (e.g. Saxe et al. 2014, Arora et al. 2018, Bordelon & Pehlevan 2025).
  • Because the end-to-end model is linear, the result is a different take on classical linear-regression learning curves. A random-matrix or deterministic-equivalent treatment would give it directly, and would also handle finite data.
  • In deep linear networks, nonlinearity arises only in how individual parameters vary due to the very large matrices. It does not produce a nonlinear fitting problem. So the relationship to nonlinear networks is limited. The ReLU evidence is only qualitative.
Questions for authors
  • What does this add beyond a standard random-matrix analysis of the end-to-end predictor?
  • Which properties of the mechanism, if any, do you expect to carry over to nonlinear networks, where depth changes the function class?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 9KW2

We thank the reviewer for their thoughtful feedback on our manuscript. We respond to individual points from your review below.

W1. Depth effects and scaling in deep linear networks have been studied extensively (e.g. Saxe et al. 2014, Arora et al. 2018, Bordelon & Pehlevan 2025).

We agree that depth effects and scaling in deep linear networks have been extensively studied. However, to the best of our knowledge, our work is the first to resolve a fundamental, long-standing open question of whether and when increasing depth improves the asymptotic training-time scaling exponent under squared loss, by establishing that such an improvement occurs in the spectrally difficult regime (p<1p<1).

Q1. What does this add beyond a standard random-matrix analysis of the end-to-end predictor?

The central contribution of our work is to explicitly separate the anisotropic structure and overall scale of the NTK using the conservation of adjacent-layer Gram differences in deep linear networks, thereby reducing its complicated dependence on the individual layer weights to a tractable expression in terms of the end-to-end predictor. This allows us to characterize the long-time feature-learning dynamics and show how the depth dependence of the NTK scale affects the training-time scaling exponent.

In the multilayer setting considered here, the gradient of each layer depends on products of the weights of the other layers, making their coupled evolution generally nontrivial to characterize and control. A standard random-matrix or deterministic-equivalent analysis does not, by itself, characterize the depth-dependent evolution of the NTK and the associated long-time feature-learning dynamics induced by this multilayer parameterization. While such methods may be useful for extending our analysis to finite-sample settings, they would still need to account for these interactions among layers.

Q2. Which properties of the mechanism, if any, do you expect to carry over to nonlinear networks, where depth changes the function class?

As discussed above, our analysis shows that the dependence of the NTK scale on network depth plays a central role in determining the training-time loss-decay exponent. We expect that architecture-dependent growth of the NTK scale could similarly affect the loss-decay exponent in nonlinear networks.

In deep linear networks, the conservation of adjacent-layer Gram differences allows us to characterize this dependence explicitly. However, it remains unclear whether the dependence of the NTK scale on network architecture can be characterized similarly in nonlinear networks, where depth also changes the function class. Understanding how this scale evolves with network architecture would therefore be an important step toward extending our framework.

Reviewer_9KW2OpenReview ↗

Response to authors

I thank the authors for their detailed responses to the posed questions. The additional information has clarified questions about the results and their importance. However, the core concern I have about this work's relevance to ICLR is the applicability to standard machine learning practices, of which deep linear networks is not often a part. The point touched upon by the authors in their final response:

"""In deep linear networks, the conservation of adjacent-layer Gram differences allows us to characterize this dependence explicitly. However, it remains unclear whether the dependence of the NTK scale on network architecture can be characterized similarly in nonlinear networks, where depth also changes the function class. Understanding how this scale evolves with network architecture would therefore be an important step toward extending our framework."""

appears to me to be the core problem of interest in a treatment like this one. Answering this question would take this approach and make it relevant to almost every machine learning practitioner. I appreciate the author responses, but I don't see a reason to raise my score.

AuthorsOpenReview ↗

Response to Reviewer 9KW2's Follow-up Comments

We thank the reviewer for the follow-up comments and for acknowledging that our responses have clarified their questions about our results and their importance.

However, we respectfully disagree with the view that limited direct applicability to standard machine learning practices diminishes the relevance of our work to ICLR. In fact, deep linear networks have been extensively studied at major machine learning conferences, including ICLR (Arora et al., 2019; Gissin et al., 2020; Dominé et al., 2025), ICML (Arora et al., 2018; Du & Hu, 2019; Bordelon & Pehlevan, 2025), and NeurIPS (Gidel et al., 2019; Braun et al., 2022; Kunin et al., 2024). These studies use deep linear networks as theoretical models to understand how depth and parameterization affect learning dynamics. We believe that our work makes a meaningful contribution to the ICLR community.

Reviewer_3YWvOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks whether adding depth actually improves the training-time scaling exponent under squared loss, which was left open by the self-consistency argument in Bordelon & Pehlevan (2025).

Critical strengths

The claim is stated precisely and the proof structure is honest about where depth enters , and the three-way classification of predictor-norm growth (polynomial / log / bounded) is a nice object in itself.

Critical weaknesses

The scope is quite narrow relative to how the result is framed: linear teacher and student, population gradient flow, fixed L with all widths scaling with d, and the small-initialization assumption is regime-dependent in a way that feels engineered with the standard-initialization result only covering p<1. The empirical support for the nonlinear claim is thin, Fig. 7 is two synthetic tasks, single seed per curve, described only as "qualitatively consistent" with the effect being noticeably weaker than for linear nets, so it is hard to tell whether the mechanism really survives nonlinearity or whether depth is just doing ordinary optimization speed-up. Finally, no direct numerical comparison against the exponent predicted by Bordelon & Pehlevan (2025) on the same tasks would have made the "previously unresolved" framing much sharper than the current prose-only account.

Questions for authors

The scope is quite narrow relative to how the result is framed: linear teacher and student, population gradient flow, fixed L with all widths scaling with d, and the small-initialization assumption is regime-dependent in a way that feels engineered with the standard-initialization result only covering p<1. The empirical support for the nonlinear claim is thin, Fig. 7 is two synthetic tasks, single seed per curve, described only as "qualitatively consistent" with the effect being noticeably weaker than for linear nets, so it is hard to tell whether the mechanism really survives nonlinearity or whether depth is just doing ordinary optimization speed-up. Finally, no direct numerical comparison against the exponent predicted by Bordelon & Pehlevan (2025) on the same tasks would have made the "previously unresolved" framing much sharper than the current prose-only account.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 3YWv

We thank the reviewer for their valuable feedback on our manuscript. We respond to individual points from your review below.

The scope is quite narrow relative to how the result is framed: linear teacher and student, population gradient flow, fixed L with all widths scaling with d, and the small-initialization assumption is regime-dependent in a way that feels engineered with the standard-initialization result only covering p<1.

We agree that our theoretical analysis is restricted to linear teacher and student networks. However, as discussed in Section 2 (Related Work), deep linear networks allow us to isolate the effect of depth on optimization dynamics from changes in representational power and have been widely studied in this context.

Small initialization is also a common assumption in theoretical studies of deep linear networks (e.g., Gidel et al., 2019; Jacot et al., 2021; Atanasov et al., 2022), as it makes the learning dynamics more tractable. Extending the analysis to standard initialization is generally more challenging. In the difficult-task regime (p<1p<1), however, the predictor norm grows polynomially during training, making the effect of the O(1)O(1) initialization scale asymptotically negligible. As discussed in Section 5, this allows us to establish the same scaling exponent under standard initialization.

The empirical support for the nonlinear claim is thin, Fig. 7 is two synthetic tasks, single seed per curve, described only as "qualitatively consistent" with the effect being noticeably weaker than for linear nets, so it is hard to tell whether the mechanism really survives nonlinearity or whether depth is just doing ordinary optimization speed-up.

We thank the reviewer for this comment. The original experiments in Figure 7 used standard initialization, under which the depth-dependent trend was less clear. We have therefore replaced Figure 7 with experiments using small initialization, which exhibit clearer qualitative consistency with the theoretical predictions for linear networks.

The revised figure reports the mean loss over five random seeds for each depth, together with shaded regions indicating one standard deviation. We also fitted the late-time decay exponents, confirming that they increase with depth for p=0.6p=0.6 but remain approximately constant across depths for p=1.4p=1.4.

We have moved the original standard-initialization experiments to Appendix I.3 (Figure 18).

Finally, no direct numerical comparison against the exponent predicted by Bordelon & Pehlevan (2025) on the same tasks would have made the "previously unresolved" framing much sharper than the current prose-only account.

We have added a black dotted curve to Figure 4 representing the decay exponent predicted by Bordelon & Pehlevan (2025), allowing a direct comparison with our theoretical predictions and numerical results. The predictions coincide for L=2L=2 but differ for L>2L>2 in the difficult-task regime. The exponents (α,β)(\alpha,\beta) used in Bordelon & Pehlevan (2025) correspond to (a,p)(a,p) in our notation. We have explicitly clarified this correspondence in Appendix H.1, in the discussion of Figure 4.

AuthorsOpenReview ↗

Gentle Reminder on Our Rebuttal

Thank you again for your feedback and for taking the time to review our paper.

As the author-reviewer discussion period is coming to an end, we would greatly appreciate it if you could let us know whether our responses have addressed your concerns. We would be happy to provide further clarification if needed.

Reviewer_8sxLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

In this paper, the authors analyze how network depth affects training-time scaling in deep linear networks on tasks with power-law spectral structures. Specifically, the authors develop a theoretical framework based on deep linear networks, where increasing depth does not change the model's representational capacity. The task difficulty is characterized by a spectral exponent pp. When p<1p < 1, the task is considered difficult because the model needs to learn significant target components associated with low-eigenvalue directions of the data covariance matrix.The authors provide a detailed theoretical analysis of how network depth affects training dynamics under the defined settings. They show that depth introduces a preconditioner KLK_L, which changes the optimization dynamics. The authors then introduce a rescaled predictor to eliminate the leading directional anisotropy of the preconditioner. Based on this formulation, they define a time reparameterization that separates the effect of depth from the underlying spectral learning dynamics. Under the reparameterized time, the loss follows a common scaling law whose exponent is determined only by the task exponent pp, independent of network depth.The authors further demonstrate that network depth affects the relationship between the reparameterized time and the actual training time through the growth of the predictor norm. As a result, increasing depth improves the asymptotic training-time scaling exponent for difficult tasks (p<1p < 1), while providing diminishing improvements as depth increases. For easier tasks (p>1p > 1), increasing depth does not change the asymptotic power-law exponent. The authors also provide numerical experiments on synthetic tasks, a regression task derived from real image data, and nonlinear networks to support their theoretical findings.

Critical strengths

1: The motivation of this paper is clear. Understanding how network depth affects training dynamics and model performance is an important research topic.

2: The theoretical analysis separates the effect of network depth from the underlying spectral learning dynamics, providing valuable insights into how depth affects neural network training.

3: The authors provide numerical experiments that are generally consistent with their theoretical predictions, supporting the main conclusions of the paper.

Critical weaknesses

1: The theoretical analysis is based on deep linear networks under specific training settings, such as population gradient flow and squared loss. The authors identify predictor-norm growth as a key mechanism behind the depth-dependent improvement. However, it remains unclear whether this mechanism still holds under more general settings.

2: In practical training, optimization is also affected by factors such as learning rate, training steps, and computational budget. However, the theoretical analysis mainly considers continuous-time gradient flow. The authors should provide additional experiments comparing loss against computational budget or investigating different optimization settings.

3: Insufficient analysis of nonlinear networks: The analysis of nonlinear networks is still limited. Although the authors provide numerical results with ReLU networks, the experiments mainly demonstrate qualitative trends. A more detailed analysis, similar to that conducted for deep linear networks, would help clarify whether the proposed mechanism also applies to nonlinear models.

Questions for authors

1: Can the authors provide further insights into how the proposed theoretical framework can be extended to more general neural network settings?

2: How does the effect of network depth change under different training budgets, especially when considering practical computational costs?

3: Can the authors provide a clearer comparison with previous works, particularly regarding the differences in theoretical assumptions and conclusions?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 8sxL (1/2)

We thank the reviewer for the helpful feedback on our work. We respond to individual points from your review below.

Q1. Can the authors provide further insights into how the proposed theoretical framework can be extended to more general neural network settings?

We thank the reviewer for this thoughtful question. The key idea of our work is to decompose the preconditioner KLK_L governing the end-to-end predictor dynamics into its anisotropic component and overall scale, and to show that the depth dependence of this scale affects the loss-decay exponent. In deep linear networks, the conservation of adjacent-layer Gram differences allows us to characterize this dependence explicitly. However, it remains unclear whether the dependence of the preconditioner scale on network architecture can be characterized similarly in more general nonlinear networks. We believe that understanding how this scale evolves with network architecture would provide a promising direction for extending our framework.

Q2. How does the effect of network depth change under different training budgets, especially when considering practical computational costs?

Our main focus is on how increasing network depth improves the power-law exponent of training-time loss decay, rather than comparing network depths under fixed training or computational budgets.

To provide some insight into comparisons at a fixed training time, we have added a heuristic analysis in Appendix G to predict which network depth achieves a lower loss for a given training time TT and initialization scale ss. For example, Figure 9 predicts that, at T=2000T=2000, the network with L=4L=4 achieves a lower loss than that with L=2L=2 when ss exceeds a crossover scale estimated to be approximately 0.020.02.

AuthorsOpenReview ↗

Response to Reviewer 8sxL (2/2)

Q3. Can the authors provide a clearer comparison with previous works, particularly regarding the differences in theoretical assumptions and conclusions?

We clarify the differences in theoretical assumptions and conclusions between our work and the relevant prior studies below.

Arora et al. (2018) investigate depth-induced acceleration using a simplified, ill-conditioned two-dimensional regression problem with an ℓp\ell_p loss (p>2p>2), based on the depth-dependent preconditioned update derived from deep linear network dynamics. They show that, for certain data configurations and suitable choices of learning rate, the update associated with deeper networks can converge faster than that of shallower networks (Section 7 and Appendix B). In contrast, we consider squared loss under population gradient flow with high-dimensional power-law task spectra and characterize how the loss-decay exponent depends on depth.

Bordelon & Pehlevan (2025) develop a dynamical mean-field theory for randomly initialized deep linear networks in the high-dimensional limit. Their scaling-law analysis considers power-law structured data in the infinite-width, infinite-batch, and gradient-flow limits. Their spectral assumptions, λk∼k−α\lambda_k\sim k^{-\alpha} and λk(wk∗)2∼k−αβ−1\lambda_k(w_k^\ast)^2\sim k^{-\alpha\beta-1}, correspond to ours with α=a\alpha=a and β=p\beta=p. However, they assume α>1\alpha>1 (trace-class covariance), whereas our analysis allows a>0a>0. Regarding the parameterization, they introduce a parameter γ0\gamma_0 by scaling the network output by 1/γ01/\gamma_0 and the learning rate of the raw weights by γ02\gamma_0^2 in addition to width-dependent normalization, whereas we directly control the Gaussian initialization scale ss of each layer. Although these parameterizations can be related through a rescaling of the weights and training time with s=γ0−1/Ls=\gamma_0^{-1/L} for LL trainable layers, their scaling-law analysis additionally centers the network output so that it is zero at initialization. In contrast, we analyze population gradient flow without such centering.

In Appendix E.1, Bordelon & Pehlevan (2025) introduce the ansatz K(t)∼1+γ02tχ−1K(t)\sim 1+\gamma_0^2t^{\chi-1} for a scalar kernel factor K(t)K(t) related to the NTK, and derive the residual dynamics in Eq. (114). To determine χ\chi, they implicitly rely on the kernel-growth relation derived for the two-layer model by Bordelon et al. (2025, Appendix E.2, Eqs. (64)–(67)), where the scalar kernel-growth factor approximately satisfies K˙(t)∝∑kλkwk∗vk(t)\dot K(t)\propto\sum_k\lambda_k w_k^\ast v_k(t). Here, wk∗w_k^\ast and vk(t)v_k(t) denote the target and centered residual coefficients in the covariance eigenbasis, respectively. Using the residual dynamics in Bordelon & Pehlevan (2025, Eq. (114)) and the power-law spectral assumptions, we obtain ∑kλkwk∗vk(t)∼t−βχ\sum_k\lambda_k w_k^*v_k(t)\sim t^{-\beta\chi} when γ02tχ−1≫1\gamma_0^2t^{\chi-1}\gg1, up to γ0\gamma_0-dependent prefactors. Integrating this relation yields K(t)∼t1−βχK(t)\sim t^{1-\beta\chi}, and matching it with the initial ansatz gives the depth-independent loss-decay exponent 2β/(1+β)2\beta/(1+\beta) (corresponding to 2p/(1+p)2p/(1+p) in our notation). However, this kernel evolution relation is derived for a two-layer network, and its validity for general depth is not established. In contrast, we derive the depth-dependent kernel scale from the conservation laws of deep linear networks, leading to the exponent pL/[1+p(L−1)]pL/[1+p(L-1)]. We have also added the exponent predicted by Bordelon & Pehlevan (2025) as a black dotted curve in Figure 4, showing that it coincides with our prediction for L=2L=2 but differs from those for L>2L>2 in the difficult-task regime (p<1p<1).

Thanks for detailed answer from author. I tend to keep my positive score.

Reviewer_e2RJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper analyzes population gradient flow on deep linear networks trained with squared loss on a linear regression task whose input covariance eigenvalues decay as a power law. The task exponent determines whether the target norm diverges with dimension (p < 1; difficult task regime), diverges logarithmically (p = 1) or stays bounded (p > 1). They find that as a function of τ, loss decays as τ^{−p} for any network depth. The squared predictor norm grows as τ^{1−p}, grows as log τ, or stays bounded; converting back to training time gives a loss exponent pL/[1 + p(L − 1)] for p < 1, a depth-dependent logarithmic correction at p = 1, and no depth dependence for p > 1. The results hold under small initialization with bounds uniform in dimension and, for p < 1, under standard initialization. Numerical experiments for dimensions d ~ 10^3-10^4 confirm the exponents and show that finite-size effects delay the asymptotic regime for deeper networks. In a linear regression task from CIFAR-5M, where the fitted value of p ≈ 0.48, they find good agreement with the predicted exponents, and they show that ReLU networks exhibit a qualitatively similar depth dependence as the theory.

Critical strengths
  1. The paper gives a closed-form answer to a well-posed open question: whether depth changes the training-time exponent under squared loss. The predictor-norm time reparameterization shows that the effective learning rate grows as σ^{2(L−1)}, and σ grows only when the target norm diverges.

  2. The disagreement with Bordelon and Pehlevan (2025) is made clear, with the numerical results in Figures 1, 4 and 5 distinguishing the two predictions.

Critical weaknesses
  1. The acceleration result is a continuous-time phenomenon whose meaning in discrete time unclear. In the current paper, the effective step size on the leading modes grows as σ^{2(L−1)}. Gradient descent at a fixed learning rate η becomes unstable once η L λ_1 σ^{2(L−1)} approaches 2. Doesn’t this imply that acceleration cannot be exploited indefinitely at fixed η, and instead the largest stable learning rate in a deeper network must decrease as the predictor norm grows? The experiments use small fixed learning rates and don’t reach this limit. The training-time scaling in steps or in compute therefore would benefit from either a stability analysis or experiments at the largest stable learning rate for each depth. Is it possible that the depth-dependent exponent could be an artifact of comparing depths at the same small learning rate?

  2. In the difficult-task regime p < 1, the target’s norm diverges with dimension. In this regime, population gradient flow represents learning from finite data least well, because the predictor norm grows along low-eigenvalue directions, which finite samples do not constrain. The CIFAR-5M task inherits this problem from its construction. The teacher β_i = e_i/λ_i divides the input–target cross-moment by the eigenvalue, so coefficients at small eigenvalues are amplified and may be dominated by estimation noise. The fitted p_eff ≈ 0.48 may therefore reflect the construction of the teacher rather than the data. The ReLU results (Figure 7) report single seeds, no exponent fits and crossing loss curves, so the data shown don’t support the claim of qualitative agreement.

Questions for authors
  1. Clarification: please add a discrete-time analysis or experiment. For each depth, train at the largest learning rate that remains stable along the trajectory, or with a schedule that tracks the stability threshold, and report the loss against step count. Does the depth advantage of Theorem 4.5 persist?

  2. Clarification: at L = 2, does the exponent 2p/(1 + p) coincide with the two-layer result of Bordelon, Atanasov and Pehlevan (2025)? Please state the correspondence explicitly and plot the exponent predicted by Bordelon and Pehlevan (2025) in Figure 4.

  3. Clarification: for CIFAR-5M, how does p_eff change when the teacher is estimated with ridge regularization or on a held-out split, and when the smallest 10% of eigenvalues are excluded? How does the depth advantage change under online SGD with a finite training set, for example n = 50,000?

  4. Clarification: the escape time t_e scales as s^{−(L−2)} for L ≥ 3, where s is the initialization scale, and the comparisons in Figure 1 subtract this time. For a fixed total training time, at which initialization scale does the deeper network reach a lower loss than the shallower one?

  5. Suggestion: for Figure 7, please fit exponents over a late-time window with several seeds, or remove the claim of qualitative consistency.

3 comments
AuthorsOpenReview ↗

Response to Reviewer e2RJ (1/2)

We thank the reviewer for the detailed and constructive feedback. We respond to individual points from your review below.

Q1. Clarification: please add a discrete-time analysis or experiment. For each depth, train at the largest learning rate that remains stable along the trajectory, or with a schedule that tracks the stability threshold, and report the loss against step count. Does the depth advantage of Theorem 4.5 persist?

W1. Is it possible that the depth-dependent exponent could be an artifact of comparing depths at the same small learning rate?

Although our theoretical analysis is based on continuous-time gradient flow, all numerical experiments in the manuscript are performed using discrete-time optimization.

We have added experiments in Appendix I.1 (Figures 13 and 14), where we compare the loss as a function of the number of gradient descent steps. For each depth, we use a fixed learning rate chosen just below the estimated local stability threshold at the global minimum.

Our main theoretical claim concerns the improvement in the asymptotic loss-decay exponent, rather than guaranteeing that deeper networks always achieve a lower loss at the same number of gradient descent steps. Indeed, Figure 13 shows that increasing depth does not necessarily reduce the number of steps required to achieve a given loss.

Figure 14 further shows that the loss ratios relative to L=1L=1 decrease during training for p<1p<1, whereas they remain approximately constant or increase slightly for p>1p>1. Notably, this contrast is observed even when the learning rate is chosen separately for each depth. Furthermore, in the regime where gradient descent approximates gradient flow, a fixed learning rate only rescales time and does not change the depth-dependent loss-decay exponent predicted by Theorem 4.5. Thus, the observed depth-dependent effect is not merely an artifact of comparing networks at the same small learning rate.

Q2. Clarification: at L = 2, does the exponent 2p/(1 + p) coincide with the two-layer result of Bordelon, Atanasov and Pehlevan (2025)? Please state the correspondence explicitly and plot the exponent predicted by Bordelon and Pehlevan (2025) in Figure 4.

For L=2L=2, our exponent 2p/(1+p)2p/(1+p) coincides with the two-layer result of Bordelon, Atanasov and Pehlevan (2025). The spectral exponents (α,β)(\alpha,\beta) in their work and Bordelon and Pehlevan (2025) correspond to (a,p)(a,p) in our notation. We have explicitly clarified this correspondence in Appendix H.1, in the discussion of Figure 4, and added a black dotted curve representing the depth-independent exponent predicted by Bordelon and Pehlevan (2025).

Q3. Clarification: for CIFAR-5M, how does p_eff change when the teacher is estimated with ridge regularization or on a held-out split, and when the smallest 10% of eigenvalues are excluded? How does the depth advantage change under online SGD with a finite training set, for example n = 50,000?

We thank the reviewer for raising these relevant concerns. We agree that the estimated effective task exponent peffp_{\mathrm{eff}} may depend on how the teacher is constructed, particularly in directions associated with small covariance eigenvalues.

To examine this sensitivity, we have conducted additional evaluations using a held-out split for teacher estimation and excluding the teacher coefficients corresponding to the smallest 10% of covariance eigenvalues.

The resulting estimates of peffp_{\mathrm{eff}} are approximately 0.4710.471 and 0.4950.495, respectively, compared with 0.478 for the original teacher. Thus, both modifications produce relatively small changes in the estimated exponent, and both remain within the difficult-task regime (p<1p<1).

We have also conducted an additional experiment using minibatch SGD on a fixed finite training set of n=50,000n=50{,}000 samples. The results show that the depth-dependent improvement in the training-loss decay remains qualitatively consistent with the trend observed in Figure 6.

Details of these additional experiments are provided in Appendix I.2 (Figures 15–17).

AuthorsOpenReview ↗

Response to Reviewer e2RJ (2/2)

Q4. Clarification: the escape time t_e scales as s^{−(L−2)} for L ≥ 3, where s is the initialization scale, and the comparisons in Figure 1 subtract this time. For a fixed total training time, at which initialization scale does the deeper network reach a lower loss than the shallower one?

To address this question, we have added Appendix G, which develops a heuristic approximation for comparing the losses of networks with different depths while explicitly accounting for the initial escape time.

This analysis provides an approximate prediction of which depth achieves a lower loss for a given total training time and initialization scale. The predicted crossover times show broad agreement with the numerical experiments.

For example, Figure 9 compares networks with L=2L=2 and L=4L=4 at a fixed total training time of T=2000T=2000. The approximation predicts that the deeper network achieves a lower loss when the initialization scale is larger than s≈0.02s\approx 0.02. The numerical experiments exhibit a similar trend.

Although this analysis relies on heuristic approximations rather than rigorous bounds, it provides insight into the trade-off between the longer initial escape time of deeper networks and their faster subsequent loss decay.

Q5. Suggestion: for Figure 7, please fit exponents over a late-time window with several seeds, or remove the claim of qualitative consistency.

We thank the reviewer for this suggestion. The original experiments in Figure 7 used standard initialization, under which the depth-dependent trend was less clear.

We have therefore replaced Figure 7 with experiments using small initialization, which exhibit clearer qualitative consistency with the theoretical predictions for linear networks. The revised figure reports the mean loss over five random seeds for each depth, together with shaded regions indicating one standard deviation.

In Figure 7, the dashed lines indicate the decay exponents predicted by the linear theory. To provide a quantitative comparison, we also fitted the decay exponents over tˉ∈[2000,4000]\bar t\in[2000,4000] using five random seeds. For L=2,3,4L=2,3,4, the mean fitted exponents are 0.7960.796, 0.8880.888, and 0.9740.974 for p=0.6p=0.6, compared with the linear-theory predictions of 0.7500.750, 0.8180.818, and 0.8570.857, respectively. For p=1.4p=1.4, the fitted exponents are 1.4061.406, 1.4071.407, and 1.3751.375, compared with the theoretical prediction of 1.41.4 for all depths. The sample standard deviations are at most 0.0050.005 and 0.0200.020, respectively. These results support the qualitative consistency with the linear theory.

We have moved the standard-initialization experiments to Appendix I.3 (Figure 18), where we also report fitted decay exponents.

Thanks to the authors for their clarifications and additions, these are helpful and have satisfied my questions. I think the paper is nice, and I will keep my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Follow-up on Author-Reviewer Discussion

We thank the reviewers and the Area Chair for their time and consideration.

Although we have not yet received a follow-up response from reviewer 3YWv, we have addressed their concerns through additional experiments and revisions, which we believe adequately respond to the points raised. We are also pleased that reviewers e2RJ and 8sxL found our responses satisfactory and have expressed support for acceptance. Reviewer 9KW2 also acknowledged that our responses clarified questions regarding our results and their importance.

Regarding the remaining concern raised by reviewer 9KW2, we respectfully disagree with the view that our analyses of deep linear networks have limited relevance to ICLR due to their limited direct applicability to standard machine learning practices. As noted in our response, studies of deep linear networks have been consistently published at ICLR, ICML, and NeurIPS. We therefore respectfully request that, during the upcoming reviewer-AC discussion, the Area Chair and reviewers explicitly discuss whether this view of relevance is justified and should be accepted as a basis for evaluating our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.