- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies width–depth limits during training in linear residual networks. It shows that ordinary gradient descent admits a joint limit in probability but can fail to converge in . With gradient clipping, it establishes convergence with explicit width and depth rates.
Critical strengths
The comparison after optimizing over rescaling is useful, since arbitrary parameter scaling can otherwise obscure differences between interpolators. The extension beyond ReLU and quadratic activations is also of interest.
Critical weaknesses
I think the paper has two main shortcomings, outlined below.
- Correctness: Proposition 3 is false as stated for ReLU. Taking satisfies the restrictions on the Gaussian parameters in Assumption 2, but , contradicting the required constant lower bound. The proof also contains substantive errors: the mean lower bound on page 42 does not account for cancellation between the two branches, and applying it to the derivative on page 43 is unjustified. The conditional variance bound is subsequently used in Lemmas 9–10 in the proof of Theorem 2. Thus, the claimed coverage of ReLU targets is not established. It is unclear whether the subsequent results can be recovered for these targets under the current assumptions.
- Presentation: The theoretical analysis is very technical and difficult to follow. It would help to explain the main ideas and the scope of each result more clearly.
Questions for authors
- Can Proposition 3 be corrected to cover ReLU targets, and which downstream results remain valid after the correction? In particular, does the conditional variance argument require stronger assumptions?
- Does the lower bound on in Eq. (1) hold uniformly over data-dependent directions? This seems necessary to apply the generalization result to all flattest interpolators.