- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper asks when finite-step gradient descent converges globally for a deep linear residual network initialized with invertible effective factors. Its analysis separates lower and upper radial scales from logarithmic inter-layer mismatch. These coordinates yield a quantitative PL inequality, and a nested-basin argument keeps the iterates in a region where that inequality remains valid, giving linear convergence to the least-squares optimum. Further results include an explicit separation from a prior balancedness guarantee, a depth-uniform example, scalar stability obstructions, and a perturbation result near the identity. Experiments investigate certificate feasibility and compare the theoretical bounds with observed GD behavior.
Critical strengths
-
Finite-step invariance is proved rather than assumed. Combining discrete Gram-imbalance drift with summable radial motion allows the authors to preserve the PL bound along the trajectory. Controlling the candidate exit endpoint is an important part of this argument.
-
An explicit noncommuting example separates the certificate from an earlier guarantee. Appendix M.1 verifies strict inequalities and extends the example to an open neighborhood with a fixed target. This is more informative than a qualitative comparison of assumptions.
-
The stability results expose a useful distinction. Shape matters for uniform worst-case control, yet need not predict instability monotonically for individual factorizations. The inverse-pair construction and the near-identity perturbation result illustrate both sides of this point.
-
Unsuccessful certification is reported alongside positive results. Failed searches, successful training outside the certified regime, and residual-sensitivity experiments help the reader judge the limitations of the bounds.
Critical weaknesses
-
Certification requires a small initial error. Proposition 5.3 places an upper bound on the initial excess loss that cannot be relaxed simply by reducing the stepsize. In the depth-uniform example, . Neither the broader target-independent grid nor the large-matrix stress test produces a certificate. The authors report these limitations openly, but the results leave a basic question unanswered: how much does the guarantee cover beyond initializations whose predictions are already close to optimal?
-
Comparison with recent discrete-time guarantees needs more detail. Section 6 and Appendix M.1 establish an explicit separation from Arora et al. (2019a). Appendix L also identifies Nguegnang et al. (2024) as the closest discrete balancedness comparison, but does not examine whether that result covers the same separation family. A direct comparison would clarify whether the improvement concerns admissible initializations, convergence rates, or allowable stepsizes.
-
Sharpness is established for specific quantities, not for the full convergence certificate. The depth-two inverse-pair example shows that radial coordinates alone cannot control the connected one-step threshold uniformly. It does not establish the necessary dependence on shape for all-time convergence at general depth. Meanwhile, Proposition 7.2 and the matched experiments show that varying shape can leave the threshold nearly unchanged. These distinctions are stated correctly, but the gap between the scalar obstruction and the sufficient conditions in Theorem 5.2 remains substantial.
-
An irrelevant target component affects the certified stepsize. Lemma 5.1 depends on the total initial loss , even though the irreducible least-squares residual does not enter the gradient field. D5 demonstrates the resulting discrepancy: adding an orthogonal residual changes the step cap without changing GD dynamics. This is not a correctness issue, but it introduces avoidable dependence on a quantity unrelated to optimization. Applying the residual projection from Section 4 to the smoothness analysis may give a cleaner and tighter result.
Questions for authors
- Appendix N.1 notes that the archived YAML specifies 960 expected E1c cases, while the executed grid contains 1,920. Please correct the configuration documentation.
A possible mismatch concerning related to Reviewer mrMW
Thank you for taking the time to prepare your review. We would like to respectfully clarify a possible mismatch concerning Submission #35053, When Does Gradient Descent Globally Train a Deep Linear ResNet?
Our manuscript studies spectral-balance geometry, invariant-basin conditions, and global linear convergence of gradient descent for deep linear residual networks. Your report discusses RMSNorm removal, online affine surrogates, TaperNorm, GPT-OSS-style MoE models, and normalization-removal schedules, including the schedule values in Sections 4.2–4.3 and Table 5. These topics and experiments do not appear in our manuscript.
Could you please verify whether the report was intended for this submission? We are also asking the Area Chair to verify the correspondence between the report and our manuscript. If the assignment is confirmed, we would be grateful for feedback on the actual deep-linear optimization paper.
Thank you for your consideration.