- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies gradient descent (GD) and stochastic gradient descent (SGD) under the -smoothness condition introduced in Definition 1, and theoretically investigates how learning rate warmup can accelerate the convergence of GD and SGD. The paper provides convergence analyses for GD and SGD under several assumptions, including the -smoothness condition, variance boundedness, and the ABC condition.
Critical strengths
The main strengths of the paper are:
-
The paper provides convergence analyses of GD (Theorem 1) under Assumption 2 (-smoothness) and SGD (Theorem 4) under Assumptions 1 and 3 (variance boundedness). These results provide a theoretical framework for studying learning rate schedules under a generalized smoothness condition.
-
The paper also provides a convergence analysis of SGD (Theorem 6) under Assumptions 2 and 4 (the ABC condition), which extends the analysis beyond the standard bounded-variance setting.
Critical weaknesses
The main weaknesses of the paper are:
-
Some of the mathematical arguments appear insufficiently justified, particularly those involving Lemma 2. In particular, the claimed locally Lipschitz continuity property of following from Lemma 2 appears problematic because the proposed Lipschitz constant depends on the function value gap . This issue also raises concerns about the validity of subsequent proofs that rely on Lemma 2.
-
The main theorems do not appear to establish that learning rate warmup accelerates the convergence of GD or SGD. For example, the convergence bound in Theorem 1 seems to remain after using , rather than yielding an rate. Thus, the claimed acceleration from warmup does not seem to follow from the theorem as currently stated.
Questions for authors
1. Mathematical validity of Lemma 2: The paper states that Lemma 2 implies the locally Lipschitz continuity of (Lines 205--206). However, the quantity denoted by in Lemma 2 depends on through . This seems inconsistent with the usual notion of a Lipschitz constant, which should be fixed over the region under consideration rather than depending on the particular point . Could the authors clarify precisely what is meant by the locally Lipschitz continuity of in this statement? In particular, how is Lemma 2 used to obtain the inequality required in the subsequent convergence analysis? This issue is important because several subsequent proofs appear to rely on Lemma 2, and therefore the validity of the main theoretical results may depend on resolving this point.
2. Does Theorem 1 actually establish acceleration from warmup?: Theorem 1 does not seem to establish an accelerated convergence rate for GD. In particular, the theorem gives a bound of the form \begin{align*} \min_{t<T}\Vert \nabla f(w_t) \Vert^2 \leq O\left( \frac{K_0\Delta_0}{T} + \frac{K_\rho\Delta_0\sum_{t=0}^{T-1}\Delta_t^\rho}{T^2} \right). \end{align*} Since , this implies \begin{align*} \frac{K_\rho\Delta_0\sum_{t=0}^{T-1}\Delta_t^\rho}{T^2} \leq \frac{K_\rho\Delta_0^{1+\rho}}{T}, \end{align*} and hence the overall bound remains \begin{align*} \min_{t<T}\Vert \nabla f(w_t) \Vert^2 = O \left( \frac{1}{T} \right). \end{align*} Therefore, it seems that Theorem 1 does not yield an convergence rate, nor does it by itself establish an acceleration compared with constant learning rates. Could the authors clarify how Theorem 1 is intended to demonstrate acceleration due to warmup?
3. Relation between the theoretical learning rate and practical warmup schedules: The learning rate condition required by the theoretical analysis appears to be \begin{align*} \eta_t = \frac{1}{4\sqrt{2}+4} \min\left\lbrace \frac{1}{K_0}, \frac{1}{3^\rho K_\rho}\Delta_t^{-\rho} \right\rbrace. \end{align*} Could the authors clarify whether commonly used practical warmup schedules, such as a linear warmup schedule , satisfy this condition? If so, could the authors provide a formal argument? If not, it is unclear to what extent the theoretical results explain the acceleration observed with practical warmup schedules.
4. What is the additional theoretical insight provided by the generalized smoothness condition?: Under the standard -smoothness assumption, acceleration from an increasing learning rate can already be obtained relatively directly. In particular, for GD with , one can derive \begin{align*} \min_{t<T} \Vert \nabla f(w_t) \Vert^2 \leq \frac{2(f(w_0) - f*)}{(2-L\eta)\sum_{t=0}^{T-1}\eta_t}. \end{align*} Thus, for a linear warmup schedule with an appropriate upper bound on , we have \begin{align*} \sum_{t=0}^{T-1}\eta_t = \Omega (T^2), \end{align*} and consequently \begin{align*} \min_{t<T} \Vert \nabla f(w_t) \Vert^2 = O \left(\frac{1}{T^2} \right). \end{align*} This raises a more fundamental question about the role of the proposed -smoothness condition. What is additional theoretical insight about learning rate warmup obtained from this generalized smoothness framework that cannot already be obtained under the standard -smoothness assumption? In particular, does the proposed analysis explain a phenomenon specific to -smooth functions, or does it provide an acceleration result that is qualitatively stronger than the standard smoothness analysis? Clarifying this point is important for assessing the theoretical novelty and significance of the paper.
10 comments
Response to Reviewer 3ZND (Part 2/3)
W2 & Q2. What acceleration is established by the theorems?
W2: The main theorems do not appear to establish that learning rate warmup accelerates the convergence of GD or SGD. For example, the convergence bound in Theorem 1 seems to remain after using , rather than yielding an rate.
Q2: Therefore, it seems that Theorem 1 does not yield an convergence rate, nor does it by itself establish an acceleration compared with constant learning rates. Could the authors clarify how Theorem 1 is intended to demonstrate acceleration due to warmup?
Theorem 2 establishes explicit convex complexity gains, while Theorem 1 gives a nonconvex guarantee that retains the curvature improvement along the optimization trajectory.
Under convexity, Theorem 2 gives, for ,
\le O_\rho\!\left[ \frac{K_0D_0^2}{T} +\left(\frac{K_\rho D_0^2}{T}\right)^{1/(1-\rho)} \right], \qquad D_0=\|w_0-w^\star\|.$$ The stated constant-step guarantee is $$O_\rho\!\left[ \frac{K_0D_0^2}{T} +\frac{K_\rho D_0^2\Delta_0^\rho}{T} \right].$$ Thus the loss-dependent curvature contribution has a faster decay in $T$ under the analyzed warmup schedule. For $\rho\ge1$, Theorem 2 instead gives the following iteration bounds for function error at most $\epsilon$: $$\begin{aligned} N_{\mathrm{warm}}(\epsilon) &=O_\rho\!\left[ \frac{K_0D_0^2}{\epsilon} +K_\rho D_0^2\Delta_0^{\rho-1} \right],\\ N_{\mathrm{const}}(\epsilon) &=O_\rho\!\left[ \frac{K_0D_0^2}{\epsilon} +\frac{K_\rho D_0^2\Delta_0^\rho}{\epsilon} \right]. \end{aligned}$$ Warmup converts the initialization-dependent curvature penalty into an initial-phase cost, removing its multiplication by $1/\epsilon$. For nonconvex objectives, Theorem 1 replaces the constant-step curvature factor $\Delta_0^\rho$ by the trajectory average $T^{-1}\sum_{t<T}\Delta_t^\rho$. Substituting $\Delta_t\leq\Delta_0$ discards precisely this improvement and recovers the $O(T^{-1})$ worst-case order. The theorem quantifies the gain when the gap decreases along the trajectory. Example 4 additionally demonstrates a concrete iteration-count advantage on a river-valley objective. ## Q3. Relation to practical linear warmup > Could the authors clarify whether commonly used practical warmup schedules, such as a linear warmup schedule $\eta_t=at+b$, satisfy this condition? If so, could the authors provide a formal argument? If not, it is unclear to what extent the theoretical results explain the acceleration observed with practical warmup schedules. We would like to provide a more detailed explanation for this application to a broader range of learning rates. In the proof of Theorem 1, we see that as long as $\eta_t \leq \frac{1}{4\sqrt{2}+4} \min\left(\frac{1}{K_0}, \frac{1}{3^\rho K_\rho}\Delta_t^{-\rho}\right) =: \zeta(\Delta_t)$, we have that $$\Delta_{t+1} \leq \Delta_t-\frac{\eta_t}{2}\left\|\nabla f\left(\mathbf{w}_t\right)\right\|^2,$$ and therefore $$\min_{0 \leq t \leq T-1} \lVert \nabla f(\mathbf{w}_t) \rVert^2 \leq \frac{2\Delta_0}{\sum_{t=0}^{T-1}\eta_t}.$$ Thus, an increasing schedule satisfying $0<\eta_t\leq\zeta(\Delta_t)$ has a tighter cumulative-step guarantee than a constant schedule with the same initial learning rate. Now consider a linear warmup schedule $$\eta_t=a+bt,\qquad a=\eta_0,\quad b>0.$$ Provided that its iterates satisfy $$0<\eta_t\le\zeta(\Delta_t), \qquad \forall\,0\le t<T,$$ Theorem 1 implies the nonconvex stationarity guarantee $$\begin{aligned} \min_{0\le t<T}\|\nabla f(w_t^{\mathrm{linear}})\|^2 &\le\frac{2\Delta_0}{\sum_{t=0}^{T-1}\eta_t}\\ &=\frac{2\Delta_0}{T\eta_0+\frac{bT(T-1)}{2}}\\ &<\frac{2\Delta_0}{T\eta_0},\qquad T>1. \end{aligned}$$ In contrast, GD with the constant learning rate $\eta_t\equiv\eta_0$ has the guarantee $$\min_{0\le t<T}\|\nabla f(w_t^{\mathrm{const}})\|^2 \le \frac{2\Delta_0}{T\eta_0}.$$ Thus, whenever the linear schedule remains admissible and $b>0$, its guarantee is strictly tighter than the same-initial-rate constant-step bound above. This applies over a finite warmup horizon: when $K_0>0$, the cap $\zeta(\Delta_t)\leq c/K_0$ precludes indefinite growth with a fixed positive slope. The choice $\eta_t=\zeta(\Delta_t)$ is the particular adaptive schedule analyzed in our theorems. We will include this admissibility condition and comparison to clarify the connection to practical linear warmup.Response to Reviewer 3ZND (Part 3/3)
Q4. What generalized smoothness adds beyond a fixed global bound
Under the standard -smoothness assumption, acceleration from an increasing learning rate can already be obtained relatively directly. […] What is additional theoretical insight about learning rate warmup obtained from this generalized smoothness framework that cannot already be obtained under the standard -smoothness assumption?
The reviewer's descent inequality under standard -smoothness is valid.
Note that the restriction must remain in force when evaluating the cumulative step size. Under this fixed cap,
and the reviewer's displayed guarantee satisfies
\geq \frac{2\Delta_0}{(2-L\bar{\eta})T\bar{\eta}},$$ where the left is the warmup bound, and the right is the constant-step bound. Thus every such warmup schedule has a guarantee no better than the one obtained by using the largest constant step permitted by the same chosen cap, $\eta_t\equiv\bar\eta$. A linear schedule with a fixed positive slope cannot respect this cap for arbitrarily large $T$. Since we need $\eta_t\leq \bar{\eta}<2/L, \forall t$, the linear schedule is $$\eta_t=\eta_0 + \frac{\bar{\eta}-\eta_0}{T-1}t.$$ Then $$\sum_{t=0}^{T-1}\eta_t =\Theta(T).$$ Therefore, the asserted $\Omega(T^2)$ cumulative step size and uniform asymptotic $O(T^{-2})$ conclusion do not follow under the stated constraint. The fixed-$L$ argument provides no advantage for warmup over the constant-step comparison above. For $(\rho,K_0,K_\rho)$-smooth functions, the certified local smoothness bound decreases as $\Delta_t$ decreases, allowing the admissible step size to grow. Our descent proof applies to any positive learning-rate schedule satisfying $$0<\eta_t \leq \frac{1}{4 \sqrt{2}+4} \min \left(\frac{1}{K_0}, \frac{1}{3^\rho K_\rho \Delta_t^\rho}\right):=\zeta(\Delta_t).$$ As optimization reduces the gap $\Delta_t$ and its associated smoothness bound, the largest step certified by this sufficient condition becomes larger. The baseline for GD with constant learning rate in Theorem 1 is $\eta=\zeta\left(\Delta_0\right)$. A constant step calibrated to the initial curvature cannot exploit that change, whereas an admissible warmup schedule can. This is precisely the information missing from a comparison based only on one fixed global $L$. This mechanism is consistent with empirical observations of warmup. [3] show examples in which warmup guides training toward regions of lower curvature that tolerate larger learning rates. [4] identify the ability to use larger target learning rates as a central benefit of warmup in their SGD and Adam experiments. Our theory formalizes how a decreasing loss-dependent curvature bound enlarges the permitted learning rate and gives a theoretical perspective for the benefits of learning rate warmup. --- ## References [1] Zhang et al. (2020a). Bohang Zhang, Jikai Jin, Cong Fang, and Liwei Wang. *Improved Analysis of Clipping Algorithms for Non-convex Optimization.* NeurIPS 2020. [2] Li et al. (2023a). Haochuan Li, Jian Qian, Yi Tian, Alexander Rakhlin, and Ali Jadbabaie. *Convex and Non-convex Optimization Under Generalized Smoothness.* NeurIPS 2023. [3] Gilmer et al. (2022). Justin Gilmer, Behrooz Ghorbani, Ankush Garg, Sneha Kudugunta, Behnam Neyshabur, David Cardoze, George Dahl, Zack Nado, and Orhan Firat. *A Loss Curvature Perspective on Training Instability in Deep Learning.* ICLR 2022. [4] Kalra and Barkeshli (2024). Dayal Singh Kalra and Maissam Barkeshli. *Why Warmup the Learning Rate? Underlying Mechanisms and Improvements.* NeurIPS 2024.
Response to Reviewer 3ZND (Part 1/3)
We thank the reviewer for the effort in reading our paper and for the valuable feedback. We provide the following responses to the points.
W1 & Q1. Local Lipschitz continuity and the descent argument
A local Lipschitz constant may depend on the chosen neighborhood, provided that it is fixed for all pairs of points within that neighborhood. Fix an anchor x and write Δx=f(x)−f⋆,rx=r(Δx), and Lx=L(Δx). The neighborhood interpretation intended in Section 2.4 is
∥∇f(u)−∇f(v)∥≤Lx∥u−v∥,u,v∈B(x,rx).
Here Lx is fixed after choosing x; it does not vary with u or v. Therefore, as long as both points u,v lie in B(x,rx), we can apply the descent lemma to these two points (Lemma 2), just as in the globally Lipschitz case.
Consider the GD update
wt+1=wt−ηt∇f(wt),
where the learning rate is chosen as
\frac{1}{K_0}, \frac{1}{3^\rho K_\rho \Delta_t^\rho} \right), \qquad c = \frac{1}{4\sqrt{2}+4}.$$ For the actual GD update, Appendix F.1 uses a segment bound. Set $g_t=\nabla f(w_t)$, $d_t=-\eta_tg_t$, and $a_t=K_0+3^\rho K_\rho\Delta_t^\rho$. The gradient bound gives $\|g_t\|^2\leq4a_t\Delta_t$, while $\eta_ta_t\leq2c$. Hence $$a_t\|d_t\|^2+\|g_t\|\|d_t\| =\eta_t\|g_t\|^2(1+a_t\eta_t) \leq8c(1+2c)\Delta_t =\Delta_t\leq2\Delta_t.$$ The segment lemma in Appendix E.3, applied with $m=2\Delta_t$, therefore bounds the curvature by $a_t$ throughout $[w_t,w_{t+1}]$ and yields $$f(w_{t+1})\leq f(w_t)-\eta_t\left(1-\frac{a_t\eta_t}{2}\right)\|g_t\|^2 \leq f(w_t)-\frac{\eta_t}{2}\|g_t\|^2.$$ This directly verifies the local descent inequality used in Theorem 1. The same argument holds for every positive step below the displayed cap; if $\Delta_t=0$, the gradient is already zero. The local Lipschitz continuity of $\nabla f$ is also common in other generalized smoothness settings. For example, the $(L_0,L_1)$-smoothness defined in [1], gives $$\|\nabla f(y)-\nabla f(x)\| \le \bigl[2L_0+(e-1)L_1\|\nabla f(x)\|\bigr]\|y-x\|, \qquad \|y-x\|\le L_1^{-1}.$$ The coefficient depends on the fixed reference point $x$. Thus, $(L_0,L_1)$-smoothness provides a local Lipschitz guarantee in general; it does not imply that one finite Lipschitz constant applies over the entire domain. [2] explicitly formalize the corresponding property for all pairs of points within a fixed neighborhood.