acceptodds
Under review as a conference paper at ICLR 2027

Large Learning Rates for Deep ReLU Networks: Optimization and Generalization

Abstract

Large constant learning rates can improve the convergence rate of gradient descent in separable linear and two-layer models. We establish optimization and high-probability population risk bounds for deep ReLU networks trained by gradient descent with an arbitrary constant learning rate. For logistic loss, we assume finite-width neural tangent kernel separability with margin and paired Gaussian initialization, and train all hidden layers while fixing the output layer. We control the full, possibly nonmonotone trajectory and use the resulting radius to bound the complexity of a function class containing all iterates, connecting optimization to population risk analysis. The bounds hold over a finite training budget under explicit width conditions depending on depth, learning rate and budget, and yield two consequences under different learning rate choices. First, choosing a large learning rate gives final training loss under the stated budget conditions, without momentum or a schedule. Second, choosing and gives population risk averaged over iterates of order at fixed confidence, where is the number of hidden layers and is the sample size. The last iterate has the same rate if the stable phase is reached with a constant fraction of the training budget remaining. At fixed depth and confidence, the bound has the same dependence on sample size and margin as the SVM-type benchmark. Extensions to a class of convex decreasing losses distinguish the conditions for trajectory control, stable-phase convergence and population risk bounds. Experiments on noisy 2-XOR and Fashion-MNIST examine the optimization and statistical effects of larger learning rates.

Primary area:
optimization
Keywords:
Large learning rate, optimization, generalization, deep neural network

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_RT6ZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper considers deep ReLU networks trained with full batch GD for a fixed number of TT steps, initialized by a concrete algorithm, and proves two main theorems under a set of assumptions. The most important assumption is a realizability assumptions regarding the training sample, the initialization and the resulting tangent feature space of the network. The first theorem intuitively states that for wide enough networks, the GD trajectories have some nice properties with high probability. The second theorem uses the first one to establish upper bounds on the true error (population risk measured via logistic loss) of the network both on average and for the last GD iterate, also with high probability. The proof technique heavily relies on prior work, mainly Wu et al. (2024) and Li et al. (2026), and, in my opinoin, can be viewed as a nontrival combination of existing techniques in order to generalize very similar results for shallow networs to deep ones, and also Li et al.'s deep network results to the case of arbitrary constant learning rates. The paper also contains a set of experiments in relation with the theorems.

Critical strengths
  • The paper contains rigorosuly proven theoretical contributions that is well motivated and properly positioned in the literature. The set of premises / assumptions is clearly stated, the resulting theorems also precisely stated, there are no informal theorems. The proof of every principal statement is fully written in the Appendix (some Lemmas are cited as is). During the limited review period I could not find any serious mistake in the proof.

  • In my opinion the results in this paper are (somewhat modest*), but worthwhile theoretical contributions to the field. *depending on the NTK assumption and the width constraints

  • The results deal with deep ReLU models, with mostly standard assumptions (see my questions regarding the NTK separability assumption though), there is no direct data separability, monotone training loss, etc.

  • The full proof can be found in the appendix, and a detailed discussion on the proof technique is present in the main text (Section 4.4), in which the proof technique is compared to existing literature.

  • The paper is very easy to read.

Critical weaknesses
  • I did not find satisfactory discussion on the realisticness / standardness of the assumptions, only previous use cases of the assumptions are cited. In particular, it is very unclear why Assumption 3 (NTK sep. ) is important, how realistic it is, how easy it is to check, what motivates it and what are its disadvantages. Same for Assumption 1, although that may be more natural. I can see some related arguments here and there in Section 4.4, but not an explicit discussion.

  • The width requirements in the theorems are given implicitly in mm, i.e. in Theorem 1 we have sth like m≥Clog(m)4m \geq C log(m)^4, and the constants (e.g. R) are compacted so it is not clear how the required value of mm depends on the constants that are essential in the learning framework of the paper. For example as long as my calculations are correct, eq (2) unfolds into

m \ge \frac{C_2 L^{13}(\log m)^4\log(nL/\delta)}{\gamma^8} \left[{\max\{\gamma+4\log(1+2 \eta T)+\eta \left(4+2\sqrt2\gamma+\frac{\gamma^2}{2}\right),\sqrt2}}\right]^8

This means that among others, TT, γ\gamma and η\eta enter the lower bound in some way. It is not easy to see how and I find the existing discussion (lines 309-314 and Remark 2) not comprenehsive enough. Same goes for Theorem 2.

  • Same goes for the bounding constants in eq (3)-(5) and (7)-(11). The discussion on these bounds is not detailed enough, in particular their dependence on important terms, like TT, η\eta, γ\gamma.

Remark 1 and 2 are useful, but should be extended.

  • The width requirement seems very unrealistic. I may be mistaken here, but for accelerated optimization with fixed margin and depth, we have sth like m=Ω(T8)m = \Omega (T^8). Is this true, so the paper is practically in the NTK regime? As I mentioned earlier, Assumption 3 is not properly discussed, the width requirement should also be discussed not just in terms of the constant dependences (my previous points), but in terms of this point. I see Table 1 (b) with a sufficient width condition, maybe that is what should be discussed in more details?
Questions for authors
  1. Please discuss Assumption 3 (and maybe Assumption 1 a little? or is it that standard?). What does it have to do with lazy / NTK regimes? Is it realistic? Can we check it for concrete training sets and initialized networks? Is there an optimal margin, can it be computed? I get that it is used in other papers, but unless I start reading those other papers, I get no insight on why this assupmtion arises and what are its properties, pros and cons, etc.

  2. Similarly, please discuss the width requirement in the Theorems. Why the implicit inequality? What does it mean in terms of the other hyperparameter of the framework, e.g. LL, TT, η\eta, γ\gamma, etc.? There is Ω(L13γ−8+L11(d+n3L2)γ−2)~\tilde{\Omega(L^13\gamma^{-8} + L^{11} (d + n^3 L^2)\gamma^{-2})} in Table (1), how does that arise? I'm sure it can be seen from the proof, but some insights in the main text would be desirable to get the intuition. I guess this is unrealistic width, but that is okay in NTK papers. Is it the case here, too? Can there be feature learning during GD under your framework?

  3. Similarly, please discuss the bounding constants in the Theorem statements.

  4. In my understanding of Section 4.4, the proof is a (straightforward?) generalization of the techniques in Wu et al and Li et al. Could you point out exactly which Lemmas in the appendix contain new tricks / proof ideas / calculations that have not been seen in those papers and which are not straightforward adaptations? Based on Sec 4.4, I'd say it is Lemma 10, and the contribution of Lemmas 8-9 are a bit unclear.

  5. The experiments does not seem to present anything that has not been presented before. They do not concern with the artifacts of the theorem, but they illustrate certain related phenomena on toy examples. For example, we see how the loss curves change as depth, sample size, etc. vary, but we don't see how these effects are connected to the constants in the theorems. As far as I understand, the motivating empirical behavior has been established before and the theorems are intuitively in line with those empirical findings, I just don't see how the experiments connect to the actual contribution of this paper. Do any of the experiment contain any empirical findings related to the actual results? What is the added value of these experiments? Is this what Appendix E provides? My personal opinion is that the theoretical contribution of the paper is potentially solid enough (as long as my points above can be properly addressed) without these experiments.

1 comment
Reviewer_RT6ZOpenReview ↗

Response the the Authors

I thank the Authors for their hard work and addressing all the points in my review. Based on their response I think my original score of the paper is correct. Namely, the paper indeed contains a worthwhile theoretical contribution in the form of Lemma 10, and the proper adaptation of existing techniques around it to obtain the main theorems. The problem is that this contribution is well within the NTK regime with its set of assumptions, especially the sufficient condition Ω(T8)\Omega(T^8) on the width for the accelerated LR choice at fixed depth, thus the significance / relevance of the contribution to the field (and not just the NTK subfield of ML theory) is still a concern to me.

I think that a theoretical paper within the NTK regime should satisfy at least one of the following,

  1. make proper effort to move the NTK framework closer to reality,
  2. contain a serious mathematical novelty (like new techniques, tools from other mathematical subdomains that have not been applied in NTK regime before, etc.),
  3. reveal / point out something about the learning problem that has not been suggested before.

In contrast, the paper at hand

  1. stays within the usual NTK assumptions,
  2. the mathematical tools used in Lemma 10 are very close to Lemma 27 in the Wu paper and Lemmas 20-21 in the Li paper,
  3. the most important intuitive consequences of the theorems that could reveal something interesting about the real-world properties of deep learning and could be applied in practice (e.g. the connection between width and LR could be something practically applicable even without fully worked out theory) were suggested by earlier related work (the LR-width example was suggested by the Wu paper).

I am open to further discussion about the points above, but only if the authors have a firm understanding that I am mistaken in any of these points above and can support their arguments with concrete references to their paper and related work. Just for the sake of trying, please do not take each other's time.

While I sincerely hope this paper succeeds at the current conference, in case it does not, I suggest TMLR / JMLR type venues for this kind of work. TMLR website explicitly says ``TMLR emphasizes technical correctness over subjective significance, to ensure that we facilitate scientific discourse on topics that may not yet be accepted in mainstream venues but may be important in the future.''

Reviewer_rPg1OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the optimization and generalization properties of full-batch gradient descent (GD) with arbitrarily large constant learning rates for deep, fully connected ReLU networks. Building upon previous results for linear models and two-layer neural networks, the authors extend the analysis to deep ReLU networks under finite-width neural tangent kernel (NTK) separability, paired Gaussian initialization, and sufficiently large network width.

The paper establishes

  1. Optimization guarantees for arbitrary constant learning rates.
  2. Accelerated optimization.
  3. High-probability population risk bounds.
Critical strengths

The paper is well-written and easy-to-follow. It establishes the result based on prior observations on logistic linear regressions, and establishes convergence without assuming the classical gradient-descent-lemma type of monotonicity bounds.

Critical weaknesses
  1. The contribution of the paper is limited. The major concern is that the paper requires an NTK regime of deeper networks to support the dynamics, and therefore on the technical side the result of the paper is a natural extension of the previous result on logistic regression without stand-out novelty. The implications of the results are also not surprising: a. large learning rate admits late-phase training convergence (with monotonic loss descent): the result is known for homogeneous networks on losses with exponential tails (e.g. from the study of https://arxiv.org/abs/1906.05890) b. large learning rate can enter late-phase: implied by previous work on logistic regressions c. large width admits NTK regime: well known d. generalization bounds for separable data with finite margin: also well-established results. It is also a mismatch that the optimization and generalization results are developed in different learning rate regimes. Therefore, a combination of these results does not establish a strong contribution for this paper.
  2. The experiments in the paper is limited and not suited for modern machine learning. The NTK regime is also not a practical choice for modern machine learning dynamics. The experiment evaluation also fails to distinguish statistical improvements caused by additional training data from improvements caused by longer optimization.
Questions for authors

Regarding the weakness: What is the most significant technical novelty of the paper? Why cannot the logistic regression argument be directly applied to a generalized regression task with kernels?

Reviewer_xceROpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper extends the large stepsize NTK logistic regression analysis of Wu et al 2024 to the case of deep homogeneous ReLU networks with gaussian initialization, showing how progress occurs in the unstable and stable phases of training, giving a sufficient condition on the transition from unstable to stable, and showing that accelerated (1/T^2) optimization is possible with large stepsizes. The proof combines the split comparator idea from Wu et al 2024 with the deep homogeneous network optimization trajectory analysis of Li et al 2026. It also uses the control of the size of parameters from this analysis to control the Rademacher averages of the trajectory and give upper bounds on the expected logistic loss of the network. Particular choices for stepsize and number of iterations give fast statistical rates. (But the stepsize choice for fast optimization rates give statistical bounds that are too loose to give consistency.)

Critical strengths

An important extension of previous work to extend our understanding of large stepsize logistic regression to the NTK regime for deep homogeneous ReLU networks. The presentation is excellent.

Critical weaknesses

None

Questions for authors

Minor comments:

Because it relies on homogeneity, the papers results are restricted to a very specific parameterized family: ReLU networks without bias parameters. This is a weakness shared by a long line of prior work.

The introduction cites Cai et al 2024 for nearly-homogeneous two-layer networks, but Cai et al 2025 gave arbitrary stepsize and acceleration results for nearly-homogeneous deep networks in the stable phase. It would be worth mentioning this and including it also in Table 1. The description of Cai et al 2025 in the related work section also omits these results, mentioning only the (somewhat unrelated) implicit bias result from that paper.

It would be worth explicitly comparing the width conditions for this analysis and Theorem 6 in Wu et al's 2024 paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.