- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper analyses deep linear networks trained by population gradient flow on power-law tasks. It shows that depth improves the training-time exponent when the task is spectrally difficult (p < 1), while the exponent is depth-independent in a reparameterised time defined by the predictor norm.
Critical strengths
- The analysis is rigorous and clearly organised into three regimes.
- The time reparameterisation gives a neat unified picture, and the paper identifies a gap in prior arguments.
Critical weaknesses
- Depth effects and scaling in deep linear networks have been studied extensively (e.g. Saxe et al. 2014, Arora et al. 2018, Bordelon & Pehlevan 2025).
- Because the end-to-end model is linear, the result is a different take on classical linear-regression learning curves. A random-matrix or deterministic-equivalent treatment would give it directly, and would also handle finite data.
- In deep linear networks, nonlinearity arises only in how individual parameters vary due to the very large matrices. It does not produce a nonlinear fitting problem. So the relationship to nonlinear networks is limited. The ReLU evidence is only qualitative.
Questions for authors
- What does this add beyond a standard random-matrix analysis of the end-to-end predictor?
- Which properties of the mechanism, if any, do you expect to carry over to nonlinear networks, where depth changes the function class?
3 comments
Response to authors
I thank the authors for their detailed responses to the posed questions. The additional information has clarified questions about the results and their importance. However, the core concern I have about this work's relevance to ICLR is the applicability to standard machine learning practices, of which deep linear networks is not often a part. The point touched upon by the authors in their final response:
"""In deep linear networks, the conservation of adjacent-layer Gram differences allows us to characterize this dependence explicitly. However, it remains unclear whether the dependence of the NTK scale on network architecture can be characterized similarly in nonlinear networks, where depth also changes the function class. Understanding how this scale evolves with network architecture would therefore be an important step toward extending our framework."""
appears to me to be the core problem of interest in a treatment like this one. Answering this question would take this approach and make it relevant to almost every machine learning practitioner. I appreciate the author responses, but I don't see a reason to raise my score.
Response to Reviewer 9KW2's Follow-up Comments
We thank the reviewer for the follow-up comments and for acknowledging that our responses have clarified their questions about our results and their importance.
However, we respectfully disagree with the view that limited direct applicability to standard machine learning practices diminishes the relevance of our work to ICLR. In fact, deep linear networks have been extensively studied at major machine learning conferences, including ICLR (Arora et al., 2019; Gissin et al., 2020; Dominé et al., 2025), ICML (Arora et al., 2018; Du & Hu, 2019; Bordelon & Pehlevan, 2025), and NeurIPS (Gidel et al., 2019; Braun et al., 2022; Kunin et al., 2024). These studies use deep linear networks as theoretical models to understand how depth and parameterization affect learning dynamics. We believe that our work makes a meaningful contribution to the ICLR community.
Response to Reviewer 9KW2
We thank the reviewer for their thoughtful feedback on our manuscript. We respond to individual points from your review below.
We agree that depth effects and scaling in deep linear networks have been extensively studied. However, to the best of our knowledge, our work is the first to resolve a fundamental, long-standing open question of whether and when increasing depth improves the asymptotic training-time scaling exponent under squared loss, by establishing that such an improvement occurs in the spectrally difficult regime (p<1).
The central contribution of our work is to explicitly separate the anisotropic structure and overall scale of the NTK using the conservation of adjacent-layer Gram differences in deep linear networks, thereby reducing its complicated dependence on the individual layer weights to a tractable expression in terms of the end-to-end predictor. This allows us to characterize the long-time feature-learning dynamics and show how the depth dependence of the NTK scale affects the training-time scaling exponent.
In the multilayer setting considered here, the gradient of each layer depends on products of the weights of the other layers, making their coupled evolution generally nontrivial to characterize and control. A standard random-matrix or deterministic-equivalent analysis does not, by itself, characterize the depth-dependent evolution of the NTK and the associated long-time feature-learning dynamics induced by this multilayer parameterization. While such methods may be useful for extending our analysis to finite-sample settings, they would still need to account for these interactions among layers.
As discussed above, our analysis shows that the dependence of the NTK scale on network depth plays a central role in determining the training-time loss-decay exponent. We expect that architecture-dependent growth of the NTK scale could similarly affect the loss-decay exponent in nonlinear networks.
In deep linear networks, the conservation of adjacent-layer Gram differences allows us to characterize this dependence explicitly. However, it remains unclear whether the dependence of the NTK scale on network architecture can be characterized similarly in nonlinear networks, where depth also changes the function class. Understanding how this scale evolves with network architecture would therefore be an important step toward extending our framework.