Large Learning Rates for Deep ReLU Networks: Optimization and Generalization

Large constant learning rates can improve the convergence rate of gradient descent in separable linear and two-layer models. We establish optimization and high-probability population risk bounds for deep ReLU networks trained by gradient descent with an arbitrary constant learning rate. For logistic loss, we assume finite-width neural tangent kernel separability with margin $\gamma$ and paired Gaussian initialization, and train all hidden layers while fixing the output layer. We control the full, possibly nonmonotone trajectory and use the resulting radius to bound the complexity of a function class containing all iterates, connecting optimization to population risk analysis. The bounds hold over a finite training budget under explicit width conditions depending on depth, learning rate and budget, and yield two consequences under different learning rate choices. First, choosing a large learning rate $\eta =\Theta ({\gamma }^{2}T)$ gives final training loss $\mathcal{O}(1/({\gamma }^4{T}^2))$ under the stated budget conditions, without momentum or a schedule. Second, choosing $\eta =\Theta (1)$ and $T\asymp n$ gives population risk averaged over iterates of order $\tilde{\mathcal{O}}(1/({\gamma }^{2}n)+L^2/(\gamma n))$ at fixed confidence, where $L$ is the number of hidden layers and $n$ is the sample size. The last iterate has the same rate if the stable phase is reached with a constant fraction of the training budget remaining. At fixed depth and confidence, the bound has the same dependence on sample size and margin as the SVM-type benchmark. Extensions to a class of convex decreasing losses distinguish the conditions for trajectory control, stable-phase convergence and population risk bounds. Experiments on noisy 2-XOR and Fashion-MNIST examine the optimization and statistical effects of larger learning rates.