acceptodds
Under review as a conference paper at ICLR 2027

When Does Gradient Descent Globally Train a Deep Linear ResNet?

Abstract

When does gradient descent globally train a deep linear residual network from a nonzero initialization? We show that finite-step trainability is governed by three distinct aspects of factorization geometry that are conflated by end-to-end conditioning: a lower radial coordinate measuring cumulative spectral contraction, an upper radial coordinate measuring cumulative spectral expansion, and a centered logarithmic shape variable measuring multiplicative inter-layer imbalance. Our first result establishes a geometry-sensitive Polyak–Lojasiewicz inequality, revealing how and control gradient transmission through depth. Our main result proves that an explicit nested spectral-balance basin, defined by simultaneous bounds on , is forward invariant under depth-normalized gradient descent. The proof exploits exact first-order cancellation in the discrete evolution of adjacent-layer Gram imbalance, converts the resulting second-order drift into logarithmic shape control, and couples it with relative-step smoothness and summable radial motion. Consequently, every trajectory initialized in the basin remains there for all time and converges linearly to a global optimum. We further show that these three geometric effects cannot in general be collapsed into a single conditioning parameter. Exactly balanced scalar families attain the predicted radial scaling laws, while a radially neutral depth-two construction, with , exhibits a pure-shape stability threshold of order where is the logarithmic inter-layer imbalance defined above. Numerical experiments recover the predicted scaling laws and show that internal factorization geometry strongly orders finite-step stability even among factorizations representing essentially the same predictor.

Primary area:
optimization
Keywords:
deep linear networks, residual networks, gradient descent, global convergence, Polyak–Łojasiewicz inequality, factorization geometry, spectral balance, optimization theory

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mrMWOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper asks when finite-step gradient descent converges globally for a deep linear residual network initialized with invertible effective factors. Its analysis separates lower and upper radial scales from logarithmic inter-layer mismatch. These coordinates yield a quantitative PL inequality, and a nested-basin argument keeps the iterates in a region where that inequality remains valid, giving linear convergence to the least-squares optimum. Further results include an explicit separation from a prior balancedness guarantee, a depth-uniform example, scalar stability obstructions, and a perturbation result near the identity. Experiments investigate certificate feasibility and compare the theoretical bounds with observed GD behavior.

Critical strengths
  • Finite-step invariance is proved rather than assumed. Combining discrete Gram-imbalance drift with summable radial motion allows the authors to preserve the PL bound along the trajectory. Controlling the candidate exit endpoint is an important part of this argument.

  • An explicit noncommuting example separates the certificate from an earlier guarantee. Appendix M.1 verifies strict inequalities and extends the example to an open neighborhood with a fixed target. This is more informative than a qualitative comparison of assumptions.

  • The stability results expose a useful distinction. Shape matters for uniform worst-case control, yet need not predict instability monotonically for individual factorizations. The inverse-pair construction and the near-identity perturbation result illustrate both sides of this point.

  • Unsuccessful certification is reported alongside positive results. Failed searches, successful training outside the certified regime, and residual-sensitivity experiments help the reader judge the limitations of the bounds.

Critical weaknesses
  1. Certification requires a small initial error. Proposition 5.3 places an upper bound on the initial excess loss that cannot be relaxed simply by reducing the stepsize. In the depth-uniform example, E0=7.2×10−7E_0 = 7.2 \times 10^{-7}. Neither the broader target-independent grid nor the large-matrix stress test produces a certificate. The authors report these limitations openly, but the results leave a basic question unanswered: how much does the guarantee cover beyond initializations whose predictions are already close to optimal?

  2. Comparison with recent discrete-time guarantees needs more detail. Section 6 and Appendix M.1 establish an explicit separation from Arora et al. (2019a). Appendix L also identifies Nguegnang et al. (2024) as the closest discrete balancedness comparison, but does not examine whether that result covers the same separation family. A direct comparison would clarify whether the improvement concerns admissible initializations, convergence rates, or allowable stepsizes.

  3. Sharpness is established for specific quantities, not for the full convergence certificate. The depth-two inverse-pair example shows that radial coordinates alone cannot control the connected one-step threshold uniformly. It does not establish the necessary dependence on shape for all-time convergence at general depth. Meanwhile, Proposition 7.2 and the matched experiments show that varying shape can leave the threshold nearly unchanged. These distinctions are stated correctly, but the gap between the scalar obstruction and the sufficient conditions in Theorem 5.2 remains substantial.

  4. An irrelevant target component affects the certified stepsize. Lemma 5.1 depends on the total initial loss L0L_0, even though the irreducible least-squares residual does not enter the gradient field. D5 demonstrates the resulting discrepancy: adding an orthogonal residual changes the step cap without changing GD dynamics. This is not a correctness issue, but it introduces avoidable dependence on a quantity unrelated to optimization. Applying the residual projection from Section 4 to the smoothness analysis may give a cleaner and tighter result.

Questions for authors
  • Appendix N.1 notes that the archived YAML specifies 960 expected E1c cases, while the executed grid contains 1,920. Please correct the configuration documentation.
1 comment
AuthorsOpenReview ↗

A possible mismatch concerning related to Reviewer mrMW

Thank you for taking the time to prepare your review. We would like to respectfully clarify a possible mismatch concerning Submission #35053, When Does Gradient Descent Globally Train a Deep Linear ResNet?

Our manuscript studies spectral-balance geometry, invariant-basin conditions, and global linear convergence of gradient descent for deep linear residual networks. Your report discusses RMSNorm removal, online affine surrogates, TaperNorm, GPT-OSS-style MoE models, and normalization-removal schedules, including the schedule values in Sections 4.2–4.3 and Table 5. These topics and experiments do not appear in our manuscript.

Could you please verify whether the report was intended for this submission? We are also asking the Area Chair to verify the correspondence between the report and our manuscript. If the assignment is confirmed, we would be grateful for feedback on the actual deep-linear optimization paper.

Thank you for your consideration.

Reviewer_obGSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the convergence of deep linear ResNets to a least square solution under gradient descent. The authors develop the analysis using an implicit non-zero initialization and obtain exponential convergence rate under this setting. They also propose three components to characterize the convergence region.

Critical strengths
  • The main technical contribution of this paper is a convergence result towards the least square solution, which is theoretically interesting.
  • The authors also factorize the elements that affect the convergence to minimum and try to establish a new geometry, although such a story is not fully interpretable (see below).
Critical weaknesses
  • The presentation of this paper needs to be improved. The current version is very technical, with little interpretation of many technical conditions and results.
  • The newly proposed spectral-balance geometry through three ensembles is interesting. However, it is very implicit. The explanation and results discussing it are even more implicit. The whole story is not reader friendly.
Questions for authors
  • Theorem 5.2: the conditions are hard to interpret. I wonder when does the requirements of the assumptions for \rho__,\rho_+,\theta_1 are satisfied. Especially, I wonder how I can understand the initialization.
  • I don’t quite understand η0,conn(F)\eta_{0,conn}(F) in the theoretical setting and the Figure 1 is not readable.
7 comments
AuthorsOpenReview ↗

Our response to weaknesses and questions

Thank you for recognizing our convergence contribution and geometric perspective. The revision makes the geometry interpretable and the certificate checkable.

1. Geometry and initialization. We measure effective factors Fℓ=I+WℓF_\ell=I+W_\ell (identity II, residual weights WℓW_\ell), not just P(F)=FL⋯F1P(F)=F_L\cdots F_1. For invertible factors at depth LL:

  • Lower radial scale: r−=∑ℓlog⁡σmin⁡(Fℓ)r_-=\sum_\ell\log\sigma_{\min}(F_\ell) controls gradient transmission.
  • Upper radial scale: r+=∑ℓlog⁡∥Fℓ∥2r_+=\sum_\ell\log\lVert F_\ell\rVert_2 enters the curvature bound.
  • Logarithmic shape: Θsh=L∑ℓ=1L−1∥Zℓ+1−Zℓ∥2\Theta_{\mathrm{sh}}=L\sum_{\ell=1}^{L-1}\lVert Z_{\ell+1}-Z_\ell\rVert_2 measures adjacent multiplicative imbalance.

For shape, start with C0=IC_0=I and take the right polar decomposition FℓCℓ−1=CℓSℓF_\ell C_{\ell-1}=C_\ell S_\ell at each layer. CℓC_\ell is orthogonal, SℓS_\ell is positive definite, Zℓ=log⁡SℓZ_\ell=\log S_\ell is the principal matrix logarithm, and ∥⋅∥2\lVert\cdot\rVert_2 is the spectral norm.

At zero target, the scalar pair (a,a−1)(a,a^{-1}), a≥1a\ge1, has product one and r−=r+=0r_-=r_+=0, yet shape 4log⁡a4\log a. Its one-step threshold decays as 4e−Θsh/24e^{-\Theta_{\rm sh}/2} for large shape. This obstruction is invisible to product conditioning, but is not universal across instances. Sections 3–5 explain the proof: descent controls accumulated Gram drift and logarithmic shape; summable relative motion controls radial drift. Together these prevent basin exit.

2. When Theorem 5.2 applies. The basin B(ρ−,ρ+,θ)\mathcal B(\rho_-,\rho_+,\theta) contains invertible tuples with r−≥−ρ−r_-\ge-\rho_-, r+≤ρ+r_+\le\rho_+ and Θsh≤θ\Theta_{\rm sh}\le\theta. Section 5 now gives a recipe. Measure initial geometry and excess E0=L(F(0))−L⋆\mathcal E_0=\mathcal L(F^{(0)})-\mathcal L^\star, where L⋆\mathcal L^\star is the optimal least-squares loss. At zero excess GD is stationary; otherwise:

  1. Choose containing inner budgets (index 0), strictly larger outer budgets (index 1), and relative-step allowance 0<ξ<10<\xi<1.
  2. Check ρ±(1)−ρ±(0)>Crad(1)E0\rho_\pm^{(1)}-\rho_\pm^{(0)}>C_{\rm rad}^{(1)}\sqrt{\mathcal E_0} for both signs; Crad(1)C_{\rm rad}^{(1)} bounds accumulated radial motion (Section 5).
  3. Require static shape slack θ1m1,ξ2>M02θ0\theta_1m_{1,\xi}^2>M_0^2\theta_0, with initial factor-norm bound M0M_0 and endpoint singular-value floor m1,ξm_{1,\xi}.
  4. Choose a positive normalized step η0\eta_0 satisfying (13) and η0<(θ1m1,ξ2−M02θ0)/(2E0)\eta_0<(\theta_1m_{1,\xi}^2-M_0^2\theta_0)/(2\mathcal E_0). The raw step is η0/L\eta_0/L.

No future iterates are needed. Strict radial and static shape slack ensure a small positive step exists; reducing the step cannot repair radial failure. Proposition 5.3 gives the radial ceiling; Appendix M.3 gives exact joint feasibility for fixed inner budgets and ξ\xi, not a convergence boundary.

3. Explicit feasible starts. Section 6 and Appendix M.2 give fixed data A=diag⁡(1,1.2,1.1)A=\operatorname{diag}(1,1.2,1.1), B=diag⁡(1,1.1,1.2)B=\operatorname{diag}(1,1.1,1.2), X=[.6I3,.8I3]X=[.6I_3,.8I_3], P⋆=I3+10−3e1e2⊤P_\star=I_3+10^{-3}e_1e_2^\top, Y=BP⋆AX+.1[−.8I3,.6I3]Y=BP_\star AX+.1[-.8I_3,.6I_3] (eie_i are standard basis vectors). Initialize Fℓ=CℓCℓ−1⊤F_\ell=C_\ell C_{\ell-1}^\top with orthogonal gauges and C0=CL=I3C_0=C_L=I_3. The product is identity and all coordinates vanish, although WℓW_\ell need not vanish. The optimal loss is .015 and initial excess is 7.2×10−77.2\times10^{-7}. For every L≥2L\ge2, the same inner/outer budgets (0,0,0)(0,0,0) and (.2,.2,.1)(.2,.2,.1), ξ=.1\xi=.1, and η0=.05\eta_0=.05 satisfy all conditions: radial bound below .00774, shape bound below 1.15×10−71.15\times10^{-7}, and all step caps verified in M.2. Thus Theorem 6.1 gives Et≤.968119tE0\mathcal E_t\le .968119^t\mathcal E_0 (excess at iteration tt). M.1 also certifies a noncommuting start with shape .109121 outside Arora et al.'s quantitative balancedness regime. Neither example guarantees arbitrary random-start convergence.

4. Connected threshold and Figure 1. Let Fu+F_u^+ be one GD update from FF with raw step u/Lu/L, using all gradients at FF. Section 7 defines η0,conn(F)=sup⁡{s≥0:L(Fu+)≤L(F) ∀u∈[0,s]}.\eta_{0,\mathrm{conn}}(F)=\sup\{s\ge0:\mathcal L(F_u^+)\le\mathcal L(F)\ \forall u\in[0,s]\}. This is the right endpoint of the loss-nonincreasing interval issuing from zero, not the largest stable step across disconnected intervals. For the inverse pair, put C=cosh⁡(Θsh/2)C=\cosh(\Theta_{\rm sh}/2). The corrected endpoint is 4C4C for 1≤C≤21\le C\le\sqrt2, and 2C−2C2−22C-2\sqrt{C^2-2} for C>2C>\sqrt2. Tangency does not disconnect the interval; the tested range and asymptotic law are unchanged. Finite searches estimate rather than certify the first boundary. Appendix M.4 shows convergence despite first-step loss increase, so this is not an all-time threshold.

Revised Figure 1 separates estimated one-step thresholds versus shape (left, same-predictor factorizations) from the numerical joint initial-error frontier (right, varying depth and shape). The latter concerns certificate feasibility, not convergence. The four original matrix panels are enlarged in Appendix K (Figures 3–6). Figures 9–10 show common-step iterations and certified loss traces.

We thank the reviewer once again for their thoughtful engagement and constructive feedback throughout the review process. Addressing your questions on the full-access motivation and practical decision utility has substantially strengthened our manuscript. Please let us know if any further clarifications would be helpful before the discussion period concludes.

Thank you again for your time and consideration!

I would like to thank the authors for their efforts to address my questions. The responses provide a clearer technical explanation, but the current version of the paper and the responses, in my opinion, are still far from being understood by even experts in this field. I will maintain my score. However, here are some suggestions that the authors might consider to further improve the presentation and readability of their paper and their story:

  • Add an illustrative figure showing what the three geometric terms represent, how they interact, and how they affect training
  • Add intuitive explanations of each result and definitions, instead of introducing more technical details.
  • Significantly reduce the number of symbols used in the paper.
Reviewer_YyhcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper establishes sufficient spectral and balance conditions for gradient descent to converge linearly to the global least-squares loss in deep linear ResNets. It also provides stability obstructions and a near-identity robustness result, distinguishing worst-case effects of inter-layer imbalance from instance-wise behavior.

Critical strengths
  1. The analysis separates distinct geometric effects on optimization. The lower radial scale, upper radial scale, and logarithmic inter-layer imbalance distinguish gradient transmission from finite-step stability. Explicit scalar constructions illustrate why radial scale alone cannot capture all stability obstructions.

  2. The paper carefully distinguishes worst-case guarantees from instance-wise behavior. The inverse-pair obstruction and the near-identity second-order robustness result provide complementary perspectives on imbalance. The matched experiments further support the appropriately qualified conclusion that shape is relevant to worst-case stability but is not a universal predictor of individual-instance instability.

Critical weaknesses
  1. The advance over prior convergence results is unclear. Prior work already establishes global linear convergence of discrete gradient descent by controlling approximate balancedness and layer norms throughout training. Although this paper introduces new spectral coordinates and a corresponding invariant region, it does not demonstrate that its conditions cover important initializations excluded by previous results or yield better admissible step sizes or convergence rates.

  2. The applicability of the main theorem is insufficiently demonstrated. The convergence guarantee requires simultaneous constraints on invertibility, spectral geometry, initial suboptimality, basin margins, and step size. In particular, the radial margin condition cannot be satisfied merely by reducing the learning rate, while enlarging the outer basin also worsens the associated constants. Concrete, nontrivial examples satisfying all conditions would help establish the practical scope of the guarantee.

Questions for authors
  1. Can you provide an explicit family of initializations that satisfies Theorem 5.2 but falls outside the assumptions of prior convergence guarantees, particularly Arora et al. (2019)? Alternatively, can you demonstrate an improved admissible step size or convergence rate in a shared setting?

  2. For a representative nontrivial regression problem, can you exhibit concrete basin parameters and a step size satisfying all conditions of Theorem 5.2? How restrictive is the resulting bound on initial suboptimality as depth increases?

  3. Beyond one-step loss nonincrease, what do the obstruction examples imply about long-term convergence? Can they establish the necessity or near-optimality of any conditions in the main convergence theorem?

8 comments
AuthorsOpenReview ↗

Our responses to the weaknesses

Thank you for recognizing the soundness of our analysis and its distinction between worst-case and instance-wise stability. The revision addresses both weaknesses through new mathematical results and controlled experiments. Our companion comment gives explicit answers to Q1–Q3.

Weakness 1: Advance over prior convergence results.

Global linear convergence and first-order Gram cancellation are not themselves new. Our contribution is a different, explicitly checkable sufficient regime: recursive polar geometry separates cumulative spectral contraction, spectral expansion, and inter-layer shape, and converts discrete Gram drift into logarithmic shape control. Coupling this control with relative-step smoothness and summable radial motion gives a forward-invariant basin without requiring the particular small additive Gram-defect tolerance used by Arora et al. (2019, Theorem 1).

The revision now establishes a strict separation of sufficient regimes. Section 6 and Appendix M.1 give a rational noncommuting initialization at depth four, with a positive deficiency margin, satisfying all conditions of Theorem 5.2. Its adjacent Gram defect is about 1,194 times the largest tolerance allowed by the prior theorem. All relevant inequalities have strict slack, so the separation holds on an open neighborhood of initializations for the same fixed target. This establishes a genuinely additional certified regime.

We also give a separate exactly balanced instance satisfying both initialization regimes. For the same objective and simultaneous GD update, our theorem certifies a raw step of .03885, whereas the published sufficient cap of Arora et al. is approximately 3.1254×10−73.1254\times10^{-7}, a factor of about 1.24×1051.24\times10^5. This compares sufficient stepsize guarantees, not actual maximal stable steps or training speeds.

The constructed comparison suite (Table 3) provides complementary evidence: among 901 configurations, 670 have our certificate only, 224 satisfy both, seven have neither, and zero have the prior certificate only. Among 180 perturbations of the analytic anchor with its target fixed, 179 retain the separation. Here a failed search is not a nonexistence proof. This constructed suite tests separation, not population coverage or universal theorem dominance.

Weakness 2: Applicability and restrictiveness of the main theorem.

As you note, reducing the step cannot repair a failed radial margin. We now characterize this restriction explicitly.

First, an analytic all-depth construction. Section 6 and Appendix M.2 specify a noninterpolating regression problem with fixed data, nonzero optimal loss .015, initial excess 7.2×10−77.2\times10^{-7}, and orthogonal gauge initializations with possibly nonzero residual weights. The same inner/outer basin budgets (0,0,0)(0,0,0) and (.2,.2,.1)(.2,.2,.1), relative-step allowance ξ=.1\xi=.1, and normalized step η0=.05\eta_0=.05 satisfy every condition of Theorem 5.2 for every depth L≥2L\ge2. The certified excess loss satisfies Et≤.968119tE0\mathcal E_t\le .968119^t\mathcal E_0, where Et=L(F(t))−L⋆\mathcal E_t=\mathcal L(F^{(t)})-\mathcal L^\star. This is a depth-uniform iteration guarantee for the constructed family, not an extrapolation or a depth-independent computational-cost claim.

Second, an exact certificate-feasibility characterization. Proposition 5.3 derives an explicit radial initial-excess ceiling, including an optimistic ceiling independent of basin choices and ξ\xi. Appendix M.3 then incorporates static shape feasibility and proves an iff criterion: for a positive-excess start in a fixed inner basin, at fixed ξ\xi, some outer basin and positive step satisfy Theorem 5.2 exactly when the initial excess is below the joint frontier EjointE_{\rm joint}. This resolves existence for the specified certificate; neither ceiling is a GD convergence boundary.

Third, controlled scope and dynamics evidence. D1a finds 4,659/7,776 certificates with matched predictor, target and initial excess across depth. D1b confirms the fixed-protocol radial ceiling; D6 finds 480/480 initial certificates at depths 256, 512 and 1024. We retain the negative grids (E1a: 0/1,920; E6: 0/180) and all 2,160 free-radially excluded E3 replay cases. D3 detects no violations within tolerance on 1,920 certified-step trajectories. Conversely, all 300 sampled free-radially excluded D4 sources reach relative excess at most 10−610^{-6} in 2,000 steps at η0=.01\eta_0=.01. This demonstrates conservativeness for finite-horizon optimization, not asymptotic convergence outside the certificate.

Together, these results establish a new sufficient regime, an exact feasibility characterization, and a nontrivial all-depth guarantee.

AuthorsOpenReview ↗

Our responses to the questions

Write Et=L(F(t))−L⋆\mathcal E_t=\mathcal L(F^{(t)})-\mathcal L^\star, LL for depth, η0\eta_0 for the normalized step (raw step η=η0/L\eta=\eta_0/L), and ξ\xi for the relative-step allowance. Basin triples are (ρ−,ρ+,θ)(\rho_-,\rho_+,\theta).

Q1. Explicit separation and improved sufficient steps. Section 6 and Appendix M.1 give both comparisons with Arora et al. (2019, Theorem 1). Set A=B=X=I2A=B=X=I_2, L=4L=4.

Let D=diag⁡(1.04,1.02)D=\operatorname{diag}(1.04,1.02), H=(01;10)H=\left(\begin{smallmatrix}0&1; 1&0\end{smallmatrix}\right), S1=S4=DS_1=S_4=D, S2=D+.01HS_2=D+.01H, S3=D−.004HS_3=D-.004H. Set C0=I2C_0=I_2, Cℓ=R(uℓ)C_\ell=R(u_\ell), where R(u)=11+u2(1−u2−2u;2u1−u2),(u1,u2,u3,u4)=(1/20,−1/15,1/12,−1/10).R(u)=\frac1{1+u^2}\begin{pmatrix}1-u^2&-2u;2u&1-u^2\end{pmatrix},\quad (u_1,u_2,u_3,u_4)=(1/20,-1/15,1/12,-1/10). Initialize Fℓ=CℓSℓCℓ−1⊤F_\ell=C_\ell S_\ell C_{\ell-1}^\top and fix Y=P0+.01(1.2;−.1−.7)Y=P_0+.01\left(\begin{smallmatrix}1&.2; -.1&-.7\end{smallmatrix}\right), where P0=F4⋯F1P_0=F_4\cdots F_1. The SℓS_\ell are positive definite, S2,S3S_2,S_3 do not commute, and E0=7.7×10−5\mathcal E_0=7.7\times10^{-5}.

Choose inner/outer basins (0,.2,.2)(0,.2,.2) and (.5,.7,.8)(.5,.7,.8), ξ=.25\xi=.25, η0=.05\eta_0=.05. The radial bound .244522 is below both margins .5; the shape bound .734150 is below .8. The selected-basin step cap is .119334, and all step conditions hold strictly. Despite a positive deficiency margin, the adjacent Gram defect ∥F3⊤F3−F2F2⊤∥F≈.0407861\|F_3^\top F_3-F_2F_2^\top\|_F\approx.0407861 exceeds the largest admissible tolerance of Arora et al. (2019, Theorem 1), 3.416662×10−53.416662\times10^{-5}, by about 1,194-fold. Strict slack yields an open family with YY fixed. M.1 verifies these inequalities analytically.

For the shared-regime comparison, instead take all Sℓ=exp⁡(diag⁡(.04,.02))S_\ell=\exp(\operatorname{diag}(.04,.02)), rotations CℓC_\ell through .12sin⁡(1.37ℓ).12\sin(1.37\ell), and target Y=P0+.005(1.2−.1−.7)Y=P_0+.005\left(\begin{smallmatrix}1&.2\\-.1&-.7\end{smallmatrix}\right). All adjacent Gram defects vanish, and the prior deficiency margin is positive. With basins (0,.2,.01)(0,.2,.01) and (.5,.7,.1)(.5,.7,.1), ξ=.25\xi=.25, our certified normalized step .1554 (raw .03885) is strictly admissible. The prior raw-step cap is 3.125412×10−73.125412\times10^{-7}. The ratio is 1.24×1051.24\times10^5 for sufficient steps, not measured speedup.

Q2. Concrete feasibility and depth dependence. Appendix M.2 gives fixed data A=diag⁡(1,1.2,1.1),B=diag⁡(1,1.1,1.2),X=[.6I3,.8I3].A=\operatorname{diag}(1,1.2,1.1),\quad B=\operatorname{diag}(1,1.1,1.2),\quad X=[.6I_3,.8I_3]. Let P⋆=I3+10−3e1e2⊤P_\star=I_3+10^{-3}e_1e_2^\top (standard basis vectors), Y=BP⋆AX+.1[−.8I3,.6I3]Y=BP_\star AX+.1[-.8I_3,.6I_3]. For each L≥2L\ge2, initialize Fℓ=CℓCℓ−1⊤F_\ell=C_\ell C_{\ell-1}^\top with orthogonal CℓC_\ell, C0=CL=I3C_0=C_L=I_3. Then P0=I3P_0=I_3, all three geometric coordinates vanish, L⋆=.015\mathcal L^\star=.015, and E0=7.2×10−7\mathcal E_0=7.2\times10^{-7}. Use basins (0,0,0)(0,0,0) and (.2,.2,.1)(.2,.2,.1), ξ=.1\xi=.1, η0=.05\eta_0=.05. Uniform bounds are: radial motion <.00774<.2<.00774<.2, shape <1.15×10−7<.1<1.15\times10^{-7}<.1, smoothness cap >1/4.93>1/4.93, relative-step cap >45>45, contraction cap ≥1\ge1. Hence Theorem 6.1 gives Et≤.968119tE0\mathcal E_t\le .968119^t\mathcal E_0 at every depth.

As you note, reducing the step cannot repair radial failure. For the inner budgets, set H0=(2−1/L)ρ−(0)+(1−1/L)ρ+(0)+4(L−1)θ0/L2H_0=(2-1/L)\rho_-^{(0)}+(1-1/L)\rho_+^{(0)}+4(L-1)\theta_0/L^2. With data constants α,β\alpha,\beta (Section 3), Proposition 5.3 requires E0<Erad:=α4(1−ξ)2exp⁡(−2H0)8exp⁡(2)β2(3−2/L)2.\mathcal E_0<E_{\rm rad}:=\frac{\alpha^4(1-\xi)^2\exp(-2H_0)}{8\exp(2)\beta^2(3-2/L)^2}. For fixed data, inner budgets and ξ\xi, its depth limit is α4(1−ξ)2exp⁡(−4ρ−(0)−2ρ+(0))/[72exp⁡(2)β2]\alpha^4(1-\xi)^2\exp(-4\rho_-^{(0)}-2\rho_+^{(0)})/[72\exp(2)\beta^2]. Thus depth alone need not destroy feasibility; geometry and excess still matter. Appendix M.3 proves that for a positive-excess start in the fixed inner basin and fixed ξ\xi, existence of an outer basin and positive certified step is equivalent to E0<Ejoint\mathcal E_0<E_{\rm joint}, the joint radial–shape frontier. This is exact for the certificate, not for GD convergence.

Q3. All-time behavior and sharpness. For balanced scalar zero-target starts fℓ=a>0f_\ell=a>0, the common factor obeys zt+1=zt(1−ηzt2L−2)z_{t+1}=z_t(1-\eta z_t^{2L-2}), z0=az_0=a. Appendix M.4 proves the exact normalized boundary 2La−2(L−1)2La^{-2(L-1)}: smaller positive steps converge to zero loss, equality gives a nonzero two-cycle, and larger steps diverge. For depth-two inverse pairs (a,a−1)(a,a^{-1}), a≥1a\ge1, Θsh=4log⁡a\Theta_{\rm sh}=4\log a, the box ∣fi∣≤a|f_i|\le a is invariant with nonincreasing magnitudes for 0<η0<4a−20<\eta_0<4a^{-2}; their product tends to zero. Equality also works for a>1a>1. This matches the one-step exponent e−Θsh/2e^{-\Theta_{\rm sh}/2}, not an exact unbalanced convergence threshold. Theorem 7.1 also attains the depth-normalized PL factor e−2(L−1)ρ/Le^{-2(L-1)\rho/L} exactly for balanced scalar contraction fℓ=e−ρ/Lf_\ell=e^{-\rho/L}, ρ>0\rho>0, target 1. These establish sharp scales in the specified settings, not necessity of all basin conditions. Indeed, (2,1/2)(2,1/2) at raw step 3/53/5 gives (1.7,−.7)(1.7,-.7), loss ratio 1.4161, then satisfies ηmax⁡i∣fi∣2<2\eta\max_i|f_i|^2<2 and converges to zero loss by the box argument. First-step nonincrease is therefore not necessary for convergence.

We sincerely thank the reviewer once again for the insightful and constructive comments, which have enabled us to significantly improve the manuscript. We hope that the newly added theoretical results, experimental evidence, and clarifications on problem relevance and decision utility fully address your concerns. We would greatly appreciate your favorable reconsideration of our revised work and welcome any additional feedback during the remainder of the discussion period.

Thank you again for your time and thoughtful evaluation.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Consolidated Author Response for Substantive Theoretical Improvement and Resolution of Reviewer Concerns

Dear Program Chairs, Senior Area Chairs, Area Chairs, and Reviewers,

Thank you for your careful evaluation of Submission #35053, When Does Gradient Descent Globally Train a Deep Linear ResNet? Reviewer Yyhc reports no substantive soundness concerns and highlights our separation of geometric effects and worst-case versus instance-wise stability. Reviewer obGS recognizes the theoretical convergence contribution. The revision addresses their concerns with four substantive advances.

1. Concrete advances over prior sufficient guarantees. Section 6 and Appendix M.1 provide both comparisons requested by Yyhc. An explicit rational, noncommuting initialization satisfies all conditions of Theorem 5.2 but violates the quantitative balancedness condition of Arora et al. (2019, Theorem 1), despite a positive deficiency margin. Its adjacent Gram defect is 0.0407861, approximately 1,194 times the largest permitted tolerance. Strict slack establishes an open family with the target fixed.

On a separate exactly balanced instance satisfying both initialization regimes, our certified raw GD step is 0.03885 versus the prior published sufficient cap of 3.1254 × 10⁻⁷, more than five orders of magnitude larger for the same loss and GD update. Table 3 complements the analytic separation: among 901 constructed cases, 670 are ours-only, 224 satisfy both, zero are prior-only, and seven have neither a found certificate nor satisfaction of the prior conditions. These are comparisons of sufficient guarantees, not maximal stable steps, measured speedups, or dominance of actual convergence sets. Our advance is joint finite-step control of radial motion and logarithmic shape, not the already-known first-order Gram cancellation alone.

2. Feasibility is now characterized, not merely assumed. Proposition 5.3 derives a necessary initial-excess ceiling for radial certification, including a bound independent of basin and relative-step allowance choices. Appendix M.3 gives an exact necessary-and-sufficient characterization of whether an outer basin and positive certified step exist for a fixed inner basin containing the initialization, fixed relative-step allowance, and positive initial excess. This explains why shrinking the learning rate cannot repair a failed radial margin. Section 5 now gives a pre-training certificate procedure based on initial geometry and loss, radial margins, static shape slack, and explicit step bounds.

Section 6 and Appendix M.2 establish a fixed noninterpolating regression with optimal loss 0.015 and initial excess 7.2 × 10⁻⁷. The same inner/outer basin budgets and normalized step 0.05 satisfy all conditions for every depth L ≥ 2. Theorem 6.1 gives a uniform excess-loss contraction factor below 0.968119, with raw step 0.05/L. This is an analytic all-depth guarantee, not extrapolation from a numerical sweep. The matched fixed-error sweep finds 4,659/7,776 certificates through depth 256; all 480 additional structured cases certify at depths 256, 512, and 1024.

3. All-time dynamics and the limits of sufficient conditions. Appendix M.4 proves an exact convergence/two-cycle/divergence boundary for balanced scalar zero-target starts, an all-time sufficient inverse-pair interval matching the one-step exponential shape scale, and an exact counterexample to the necessity of first-step nonincrease: the loss increases by 41.61% at the first update, yet converges to zero.

The new experiments provide extensive evidence supporting the theoretical results and clarifying the scope of our certificate. In D3, all 2,560 matched matrix jobs achieve terminal relative excess at most 10⁻⁶. Among them, 1,920 use certified steps, with all 3,969,920 recorded loss points and 192,000 geometry checkpoints satisfying the audited theoretical bounds within numerical tolerances. These results demonstrate the consistency of the predicted convergence and geometric stability across the tested depths.

Complementarily, D4 reveals an important distinction between certification and trainability: all 300 sampled initializations analytically excluded by the free radial ceiling nevertheless achieve the same small-excess criterion at normalized step 0.01 within 2,000 iterations. Together, these experiments establish the empirical reliability of our sufficient certificate while identifying regimes where successful optimization extends beyond its guarantees, providing a concrete basis for developing tighter instance-adaptive convergence theory.

4. Interpretation and presentation. We now explain how the three coordinates are computed from effective factors, give explicit feasible initializations, define the connected one-step threshold precisely, and correct its inverse-pair tangency case. We have improved all main content in the revised manuscript from the reviewers' suggestions; Appendices M–O contain the new proofs and supplementary evidence.

Thank you for your consideration, and we look forward to further discussion.

Dear Program Chairs, Senior Area Chairs, Area Chairs, and Reviewers,

We sincerely thank the reviewers for their constructive feedback throughout the discussion. In response to the concerns about readability and accessibility, we have substantially revised the main manuscript, focusing on communicating the central scientific contributions more clearly rather than introducing additional technical complexity.

The revised version incorporates three major improvements:

  1. A new illustrative Figure 1 (p. 2) explains the three geometric effects—contraction, expansion, and inter-layer imbalance—and visually connects them to gradient transmission, forward invariance, and global convergence.

  2. A clearer and more intuitive theoretical narrative. We rewrote the Introduction and strengthened the explanations throughout Sections 3–7, emphasizing the motivation, interpretation, and significance of each main result.

  3. Simplified notation and manuscript organization. We streamlined the mathematical exposition and separated the main paper (10 pages of content plus references) from the Supplementary Material, making the central arguments easier to follow while preserving the complete technical details.

Importantly, these are improvements to the presentation, not changes to our mathematical claims or convergence guarantees. The substantive theoretical advances and supporting experimental evidence remain intact.

We believe the revised manuscript now communicates its contributions much more effectively while maintaining mathematical rigor. We respectfully invite the reviewers and chairs to revisit the updated version and reconsider their assessments in light of both the theoretical advances and these substantial presentation improvements.

Thank you again for your time, constructive feedback, and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.