acceptodds
Under review as a conference paper at ICLR 2027

Theoretical Analysis on How Learning Rate Warmup Accelerates Convergence

Abstract

Learning rate warmup is a popular and practical technique in training large-scale deep neural networks. Despite the huge success in practice, the theoretical advantages of this strategy of gradually increasing the learning rate at the beginning of the training process have not been fully understood. To resolve this gap between theory and practice, we first propose a novel family of generalized smoothness assumptions, and validate its applicability both theoretically and empirically. Under the novel smoothness assumption, we study the convergence properties of gradient descent (GD) in both deterministic and stochastic settings. It is shown that learning rate warmup consistently accelerates GD, and GD with warmup can converge at most times faster than with a non-increasing learning rate schedule in some specific cases, providing insights into the benefits of this strategy from an optimization theory perspective.

Primary area:
optimization
Keywords:
optimization, learning rate warmup, convergence rate, theoretical analysis, generalized smoothness

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3ZNDOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies gradient descent (GD) and stochastic gradient descent (SGD) under the (ρ,K0,Kρ)(\rho,K_0,K_\rho)-smoothness condition introduced in Definition 1, and theoretically investigates how learning rate warmup can accelerate the convergence of GD and SGD. The paper provides convergence analyses for GD and SGD under several assumptions, including the (ρ,K0,Kρ)(\rho,K_0,K_\rho)-smoothness condition, variance boundedness, and the ABC condition.

Critical strengths

The main strengths of the paper are:

  • The paper provides convergence analyses of GD (Theorem 1) under Assumption 2 ((ρ,K0,Kρ)(\rho,K_0,K_\rho)-smoothness) and SGD (Theorem 4) under Assumptions 1 and 3 (variance boundedness). These results provide a theoretical framework for studying learning rate schedules under a generalized smoothness condition.

  • The paper also provides a convergence analysis of SGD (Theorem 6) under Assumptions 2 and 4 (the ABC condition), which extends the analysis beyond the standard bounded-variance setting.

Critical weaknesses

The main weaknesses of the paper are:

  • Some of the mathematical arguments appear insufficiently justified, particularly those involving Lemma 2. In particular, the claimed locally Lipschitz continuity property of ∇f\nabla f following from Lemma 2 appears problematic because the proposed Lipschitz constant depends on the function value gap Δ(x)=f(x)−f∗\Delta (x) = f(x)-f^*. This issue also raises concerns about the validity of subsequent proofs that rely on Lemma 2.

  • The main theorems do not appear to establish that learning rate warmup accelerates the convergence of GD or SGD. For example, the convergence bound in Theorem 1 seems to remain O(1/T)O(1/T) after using Δt≤Δ0\Delta_t \leq \Delta_0, rather than yielding an O(1/T2)O(1/T^2) rate. Thus, the claimed acceleration from warmup does not seem to follow from the theorem as currently stated.

Questions for authors

1. Mathematical validity of Lemma 2: The paper states that Lemma 2 implies the locally Lipschitz continuity of ∇f\nabla f (Lines 205--206). However, the quantity denoted by L(Δ)L(\Delta) in Lemma 2 depends on xx through Δ(x)=f(x)−f∗\Delta (x) = f(x)-f^*. This seems inconsistent with the usual notion of a Lipschitz constant, which should be fixed over the region under consideration rather than depending on the particular point xx. Could the authors clarify precisely what is meant by the locally Lipschitz continuity of ∇f\nabla f in this statement? In particular, how is Lemma 2 used to obtain the inequality required in the subsequent convergence analysis? This issue is important because several subsequent proofs appear to rely on Lemma 2, and therefore the validity of the main theoretical results may depend on resolving this point.

2. Does Theorem 1 actually establish acceleration from warmup?: Theorem 1 does not seem to establish an accelerated convergence rate for GD. In particular, the theorem gives a bound of the form \begin{align*} \min_{t<T}\Vert \nabla f(w_t) \Vert^2 \leq O\left( \frac{K_0\Delta_0}{T} + \frac{K_\rho\Delta_0\sum_{t=0}^{T-1}\Delta_t^\rho}{T^2} \right). \end{align*} Since Δt≤Δ0\Delta_t \leq \Delta_0, this implies \begin{align*} \frac{K_\rho\Delta_0\sum_{t=0}^{T-1}\Delta_t^\rho}{T^2} \leq \frac{K_\rho\Delta_0^{1+\rho}}{T}, \end{align*} and hence the overall bound remains \begin{align*} \min_{t<T}\Vert \nabla f(w_t) \Vert^2 = O \left( \frac{1}{T} \right). \end{align*} Therefore, it seems that Theorem 1 does not yield an O(1/T2)O(1/T^2) convergence rate, nor does it by itself establish an acceleration compared with constant learning rates. Could the authors clarify how Theorem 1 is intended to demonstrate acceleration due to warmup?

3. Relation between the theoretical learning rate and practical warmup schedules: The learning rate condition required by the theoretical analysis appears to be \begin{align*} \eta_t = \frac{1}{4\sqrt{2}+4} \min\left\lbrace \frac{1}{K_0}, \frac{1}{3^\rho K_\rho}\Delta_t^{-\rho} \right\rbrace. \end{align*} Could the authors clarify whether commonly used practical warmup schedules, such as a linear warmup schedule ηt=at+b\eta_t = at+b, satisfy this condition? If so, could the authors provide a formal argument? If not, it is unclear to what extent the theoretical results explain the acceleration observed with practical warmup schedules.

4. What is the additional theoretical insight provided by the generalized smoothness condition?: Under the standard LL-smoothness assumption, acceleration from an increasing learning rate can already be obtained relatively directly. In particular, for GD with ηt≤η<2/L\eta_t \leq \eta < 2/L, one can derive \begin{align*} \min_{t<T} \Vert \nabla f(w_t) \Vert^2 \leq \frac{2(f(w_0) - f*)}{(2-L\eta)\sum_{t=0}^{T-1}\eta_t}. \end{align*} Thus, for a linear warmup schedule ηt=at+b\eta_t = at+b with an appropriate upper bound on ηt\eta_t, we have \begin{align*} \sum_{t=0}^{T-1}\eta_t = \Omega (T^2), \end{align*} and consequently \begin{align*} \min_{t<T} \Vert \nabla f(w_t) \Vert^2 = O \left(\frac{1}{T^2} \right). \end{align*} This raises a more fundamental question about the role of the proposed (ρ,K0,Kρ)(\rho,K_0,K_\rho)-smoothness condition. What is additional theoretical insight about learning rate warmup obtained from this generalized smoothness framework that cannot already be obtained under the standard LL-smoothness assumption? In particular, does the proposed analysis explain a phenomenon specific to (ρ,K0,Kρ)(\rho,K_0,K_\rho)-smooth functions, or does it provide an acceleration result that is qualitatively stronger than the standard smoothness analysis? Clarifying this point is important for assessing the theoretical novelty and significance of the paper.

10 comments
AuthorsOpenReview ↗

Response to Reviewer 3ZND (Part 1/3)

We thank the reviewer for the effort in reading our paper and for the valuable feedback. We provide the following responses to the points.

W1 & Q1. Local Lipschitz continuity and the descent argument

W1: Some of the mathematical arguments appear insufficiently justified, particularly those involving Lemma 2. In particular, the claimed locally Lipschitz continuity property of ∇f\nabla f following from Lemma 2 appears problematic because the proposed Lipschitz constant depends on the function value gap Δ(x)=f(x)−f∗\Delta(x)=f(x)-f^*. This issue also raises concerns about the validity of subsequent proofs that rely on Lemma 2.

Q1: The quantity denoted by L(Δ)L(\Delta) in Lemma 2 depends on xx through Δ(x)=f(x)−f∗\Delta(x)=f(x)-f^*. This seems inconsistent with the usual notion of a Lipschitz constant, which should be fixed over the region under consideration rather than depending on the particular point xx. Could the authors clarify precisely what is meant by the locally Lipschitz continuity of ∇f\nabla f in this statement? In particular, how is Lemma 2 used to obtain the inequality required in the subsequent convergence analysis?

A local Lipschitz constant may depend on the chosen neighborhood, provided that it is fixed for all pairs of points within that neighborhood. Fix an anchor xx and write Δx=f(x)−f⋆,rx=r(Δx)\Delta_x=f(x)-f^{\star}, r_x=r\left(\Delta_x\right), and Lx=L(Δx)L_x=L\left(\Delta_x\right). The neighborhood interpretation intended in Section 2.4 is

∥∇f(u)−∇f(v)∥≤Lx∥u−v∥,u,v∈B(x,rx).\|\nabla f(u)-\nabla f(v)\| \leq L_x\|u-v\|, \quad u, v \in B\left(x, r_x\right) .

Here LxL_x is fixed after choosing xx; it does not vary with uu or vv. Therefore, as long as both points u,vu,v lie in B(x,rx)B(x,r_x), we can apply the descent lemma to these two points (Lemma 2), just as in the globally Lipschitz case.

Consider the GD update

wt+1=wt−ηt∇f(wt),w_{t+1}=w_t-\eta_t\nabla f(w_t),

where the learning rate is chosen as

\frac{1}{K_0}, \frac{1}{3^\rho K_\rho \Delta_t^\rho} \right), \qquad c = \frac{1}{4\sqrt{2}+4}.$$ For the actual GD update, Appendix F.1 uses a segment bound. Set $g_t=\nabla f(w_t)$, $d_t=-\eta_tg_t$, and $a_t=K_0+3^\rho K_\rho\Delta_t^\rho$. The gradient bound gives $\|g_t\|^2\leq4a_t\Delta_t$, while $\eta_ta_t\leq2c$. Hence $$a_t\|d_t\|^2+\|g_t\|\|d_t\| =\eta_t\|g_t\|^2(1+a_t\eta_t) \leq8c(1+2c)\Delta_t =\Delta_t\leq2\Delta_t.$$ The segment lemma in Appendix E.3, applied with $m=2\Delta_t$, therefore bounds the curvature by $a_t$ throughout $[w_t,w_{t+1}]$ and yields $$f(w_{t+1})\leq f(w_t)-\eta_t\left(1-\frac{a_t\eta_t}{2}\right)\|g_t\|^2 \leq f(w_t)-\frac{\eta_t}{2}\|g_t\|^2.$$ This directly verifies the local descent inequality used in Theorem 1. The same argument holds for every positive step below the displayed cap; if $\Delta_t=0$, the gradient is already zero. The local Lipschitz continuity of $\nabla f$ is also common in other generalized smoothness settings. For example, the $(L_0,L_1)$-smoothness defined in [1], gives $$\|\nabla f(y)-\nabla f(x)\| \le \bigl[2L_0+(e-1)L_1\|\nabla f(x)\|\bigr]\|y-x\|, \qquad \|y-x\|\le L_1^{-1}.$$ The coefficient depends on the fixed reference point $x$. Thus, $(L_0,L_1)$-smoothness provides a local Lipschitz guarantee in general; it does not imply that one finite Lipschitz constant applies over the entire domain. [2] explicitly formalize the corresponding property for all pairs of points within a fixed neighborhood.
AuthorsOpenReview ↗

Response to Reviewer 3ZND (Part 2/3)

W2 & Q2. What acceleration is established by the theorems?

W2: The main theorems do not appear to establish that learning rate warmup accelerates the convergence of GD or SGD. For example, the convergence bound in Theorem 1 seems to remain O(1/T)O(1/T) after using Δt≤Δ0\Delta_t\leq\Delta_0, rather than yielding an O(1/T2)O(1/T^2) rate.

Q2: Therefore, it seems that Theorem 1 does not yield an O(1/T2)O(1/T^2) convergence rate, nor does it by itself establish an acceleration compared with constant learning rates. Could the authors clarify how Theorem 1 is intended to demonstrate acceleration due to warmup?

Theorem 2 establishes explicit convex complexity gains, while Theorem 1 gives a nonconvex guarantee that retains the curvature improvement along the optimization trajectory.

Under convexity, Theorem 2 gives, for 0<ρ<10<\rho<1,

\le O_\rho\!\left[ \frac{K_0D_0^2}{T} +\left(\frac{K_\rho D_0^2}{T}\right)^{1/(1-\rho)} \right], \qquad D_0=\|w_0-w^\star\|.$$ The stated constant-step guarantee is $$O_\rho\!\left[ \frac{K_0D_0^2}{T} +\frac{K_\rho D_0^2\Delta_0^\rho}{T} \right].$$ Thus the loss-dependent curvature contribution has a faster decay in $T$ under the analyzed warmup schedule. For $\rho\ge1$, Theorem 2 instead gives the following iteration bounds for function error at most $\epsilon$: $$\begin{aligned} N_{\mathrm{warm}}(\epsilon) &=O_\rho\!\left[ \frac{K_0D_0^2}{\epsilon} +K_\rho D_0^2\Delta_0^{\rho-1} \right],\\ N_{\mathrm{const}}(\epsilon) &=O_\rho\!\left[ \frac{K_0D_0^2}{\epsilon} +\frac{K_\rho D_0^2\Delta_0^\rho}{\epsilon} \right]. \end{aligned}$$ Warmup converts the initialization-dependent curvature penalty into an initial-phase cost, removing its multiplication by $1/\epsilon$. For nonconvex objectives, Theorem 1 replaces the constant-step curvature factor $\Delta_0^\rho$ by the trajectory average $T^{-1}\sum_{t<T}\Delta_t^\rho$. Substituting $\Delta_t\leq\Delta_0$ discards precisely this improvement and recovers the $O(T^{-1})$ worst-case order. The theorem quantifies the gain when the gap decreases along the trajectory. Example 4 additionally demonstrates a concrete iteration-count advantage on a river-valley objective. ## Q3. Relation to practical linear warmup > Could the authors clarify whether commonly used practical warmup schedules, such as a linear warmup schedule $\eta_t=at+b$, satisfy this condition? If so, could the authors provide a formal argument? If not, it is unclear to what extent the theoretical results explain the acceleration observed with practical warmup schedules. We would like to provide a more detailed explanation for this application to a broader range of learning rates. In the proof of Theorem 1, we see that as long as $\eta_t \leq \frac{1}{4\sqrt{2}+4} \min\left(\frac{1}{K_0}, \frac{1}{3^\rho K_\rho}\Delta_t^{-\rho}\right) =: \zeta(\Delta_t)$, we have that $$\Delta_{t+1} \leq \Delta_t-\frac{\eta_t}{2}\left\|\nabla f\left(\mathbf{w}_t\right)\right\|^2,$$ and therefore $$\min_{0 \leq t \leq T-1} \lVert \nabla f(\mathbf{w}_t) \rVert^2 \leq \frac{2\Delta_0}{\sum_{t=0}^{T-1}\eta_t}.$$ Thus, an increasing schedule satisfying $0<\eta_t\leq\zeta(\Delta_t)$ has a tighter cumulative-step guarantee than a constant schedule with the same initial learning rate. Now consider a linear warmup schedule $$\eta_t=a+bt,\qquad a=\eta_0,\quad b>0.$$ Provided that its iterates satisfy $$0<\eta_t\le\zeta(\Delta_t), \qquad \forall\,0\le t<T,$$ Theorem 1 implies the nonconvex stationarity guarantee $$\begin{aligned} \min_{0\le t<T}\|\nabla f(w_t^{\mathrm{linear}})\|^2 &\le\frac{2\Delta_0}{\sum_{t=0}^{T-1}\eta_t}\\ &=\frac{2\Delta_0}{T\eta_0+\frac{bT(T-1)}{2}}\\ &<\frac{2\Delta_0}{T\eta_0},\qquad T>1. \end{aligned}$$ In contrast, GD with the constant learning rate $\eta_t\equiv\eta_0$ has the guarantee $$\min_{0\le t<T}\|\nabla f(w_t^{\mathrm{const}})\|^2 \le \frac{2\Delta_0}{T\eta_0}.$$ Thus, whenever the linear schedule remains admissible and $b>0$, its guarantee is strictly tighter than the same-initial-rate constant-step bound above. This applies over a finite warmup horizon: when $K_0>0$, the cap $\zeta(\Delta_t)\leq c/K_0$ precludes indefinite growth with a fixed positive slope. The choice $\eta_t=\zeta(\Delta_t)$ is the particular adaptive schedule analyzed in our theorems. We will include this admissibility condition and comparison to clarify the connection to practical linear warmup.
AuthorsOpenReview ↗

Response to Reviewer 3ZND (Part 3/3)

Q4. What generalized smoothness adds beyond a fixed global bound

Under the standard LL-smoothness assumption, acceleration from an increasing learning rate can already be obtained relatively directly. […] What is additional theoretical insight about learning rate warmup obtained from this generalized smoothness framework that cannot already be obtained under the standard LL-smoothness assumption?

The reviewer's descent inequality under standard LL-smoothness is valid.

Note that the restriction ηt≤ηˉ<2/L\eta_t\le\bar\eta<2/L must remain in force when evaluating the cumulative step size. Under this fixed cap, ∑t<Tηt≤Tηˉ,\sum_{t<T}\eta_t\le T\bar\eta,

and the reviewer's displayed guarantee satisfies

\geq \frac{2\Delta_0}{(2-L\bar{\eta})T\bar{\eta}},$$ where the left is the warmup bound, and the right is the constant-step bound. Thus every such warmup schedule has a guarantee no better than the one obtained by using the largest constant step permitted by the same chosen cap, $\eta_t\equiv\bar\eta$. A linear schedule with a fixed positive slope cannot respect this cap for arbitrarily large $T$. Since we need $\eta_t\leq \bar{\eta}<2/L, \forall t$, the linear schedule is $$\eta_t=\eta_0 + \frac{\bar{\eta}-\eta_0}{T-1}t.$$ Then $$\sum_{t=0}^{T-1}\eta_t =\Theta(T).$$ Therefore, the asserted $\Omega(T^2)$ cumulative step size and uniform asymptotic $O(T^{-2})$ conclusion do not follow under the stated constraint. The fixed-$L$ argument provides no advantage for warmup over the constant-step comparison above. For $(\rho,K_0,K_\rho)$-smooth functions, the certified local smoothness bound decreases as $\Delta_t$ decreases, allowing the admissible step size to grow. Our descent proof applies to any positive learning-rate schedule satisfying $$0<\eta_t \leq \frac{1}{4 \sqrt{2}+4} \min \left(\frac{1}{K_0}, \frac{1}{3^\rho K_\rho \Delta_t^\rho}\right):=\zeta(\Delta_t).$$ As optimization reduces the gap $\Delta_t$ and its associated smoothness bound, the largest step certified by this sufficient condition becomes larger. The baseline for GD with constant learning rate in Theorem 1 is $\eta=\zeta\left(\Delta_0\right)$. A constant step calibrated to the initial curvature cannot exploit that change, whereas an admissible warmup schedule can. This is precisely the information missing from a comparison based only on one fixed global $L$. This mechanism is consistent with empirical observations of warmup. [3] show examples in which warmup guides training toward regions of lower curvature that tolerate larger learning rates. [4] identify the ability to use larger target learning rates as a central benefit of warmup in their SGD and Adam experiments. Our theory formalizes how a decreasing loss-dependent curvature bound enlarges the permitted learning rate and gives a theoretical perspective for the benefits of learning rate warmup. --- ## References [1] Zhang et al. (2020a). Bohang Zhang, Jikai Jin, Cong Fang, and Liwei Wang. *Improved Analysis of Clipping Algorithms for Non-convex Optimization.* NeurIPS 2020. [2] Li et al. (2023a). Haochuan Li, Jian Qian, Yi Tian, Alexander Rakhlin, and Ali Jadbabaie. *Convex and Non-convex Optimization Under Generalized Smoothness.* NeurIPS 2023. [3] Gilmer et al. (2022). Justin Gilmer, Behrooz Ghorbani, Ankush Garg, Sneha Kudugunta, Behnam Neyshabur, David Cardoze, George Dahl, Zack Nado, and Orhan Firat. *A Loss Curvature Perspective on Training Instability in Deep Learning.* ICLR 2022. [4] Kalra and Barkeshli (2024). Dayal Singh Kalra and Maissam Barkeshli. *Why Warmup the Learning Rate? Underlying Mechanisms and Improvements.* NeurIPS 2024.
Reviewer_MZ73OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces (K0,Kρ,ρ)(K_0,K_\rho,\rho)-generalized smoothness assumption of the form ∣∇2f(w)∣≤K0+Kρ(f(w)−f⋆),\\|\nabla^2f(w)\\|\le K_0 + K_\rho(f(w) -f^\star), where f⋆=min⁡wf(w)f^\star=\min_w f(w), and ρ>0\rho>0. The authors study this assumption in connection with learning-rate warmup, typically used in deep learning.

The summary of the results:

  • In Lemma 1, they demonstrate that any (L0,L1,ρ)(L_0,L_1,\rho)-smooth function is also (K0,Kα,α)(K_0,K_{\alpha},\alpha)-smooth with α=ρ2−ρ\alpha=\frac{\rho}{2-\rho}. Here, (L0,L1,ρ)(L_0,L_1,\rho)-smoothness means ∣∇2f(w)∣≤L0+Lρ∣∇f(w)∣.\\|\nabla^2f(w)\\|\le L_0 + L_\rho\\|\nabla f(w)\\|. This implies that the proposed class is more general than the previously studied assumption with a gradient norm.
  • They also show that there are some examples of (K0,Kρ,ρ)(K_0,K_\rho,\rho)-smooth functions that are not (L0,L1,ρ)(L_0,L_1,\rho) (in Example 1 it is not (L0,L1,ρ)(L_0,L_1,\rho) for any ρ>0\rho>0, while in Examples 2 and 3 it is not (L0,L1,ρ)(L_0,L_1,\rho) for any 0≤ρ<20\le\rho<2). These examples show that the proposed assumption applies better to simple neural networks (Examples 2 and 3) and to functions with multiple local minima (Example 1).
  • The authors provide convergence under the proposed condition in the general non-convex and convex settings. They demonstrate that the learning rate schedule implied by the assumption follows some sort of learning-rate warmup and leads to possibly faster convergence than GD with a constant learning rate. Example 4 shows that, on a river-valley-like function satisfying the proposed assumption, GD with the theoretical schedule converges faster.
  • The convergence results are generalized to the stochastic setting a.s. bounded variance and a.s. (A,B,σ2)(A,B,\sigma^2)-bounded variance in the general non-convex regime.
  • The authors demonstrate that the proposed theoretical schedule can improve convergence over a constant learning rate schedule and be competitive with a linear warmup in practice in toy and realistic small settings.
Critical strengths
  • The introduction of smoothness is reasonable. The examples show when the gradient-based smoothness condition fails to be satisfied for some neural networks, which motivates the design of a new condition. Based on the examples, the proposed condition can have a potential to be a practical condition to some extent.
  • The warm-up schedule derived from the condition can be competitive with linear warm-up (though it is not clear whether a linear warm-up length was tuned).
  • The authors provide convergence analysis under the proposed condition. It is provably faster in the convex case (for the non-convex case, see weaknesses).
  • The authors provide a lower bound in the ρ=1\rho=1 case to demonstrate a provable improvement from the use of theoretical warm-up.
  • The convergence results are extended to a.s. bounded variance and a.s. (A,B,σ2)(A,B,\sigma^2) bounded variance (though not sure why the authors separate two cases since the former is a special case of the latter condition).
  • I verified the analysis and do not spot any issues, but some statements should be clarified (see weaknesses).
Critical weaknesses
  • The provided examples are limited to very toy cases where the networks have a few parameters. The authors also don't discuss the potential generalization of the examples to more realistic settings.
  • Some claims are not precise enough:
    • "Since the primary role of warm-up is to suppress sharpness to allow the network to tolerate larger learning rate,...": this claim is not justified; the authors do not provide enough evidence for this claim, so it cannot be used as a motivation. The authors should work on the wording or provide references that prove such claims.
    • "better at capturing the properties of functions with multiple stationary points or local minima, which is a common case in deep neural network training." Again, to make such a claim, the authors should provide evidence that Example 1 is realistic. An arbitrary (to some extent) non-convex function with multiple stationary points/local minima cannot be used to justify that it is close to neural network training. I also believe that warm-up does not help in minimizing an arbitrary non-convex function, even if it satisfies the proposed condition. Finally, I do not see local minima in the provided empirical measurements of the smoothness.
  • The authors do not discuss the works that propose a similar condition in the main part (also given that the papers are online for a long time). The whole discussion is moved to the appendix. This makes the understanding of the significance and novelty of the results less clear. For example, the authors should discuss
    • how the examples in [Alimisis et al., 2025] of networks satisfying a similar condition differ from the examples in this work. I find that the examples in [Alimisis et al., 2025] are more realistic: they cover MLP and Transformer under the balancedness condition, which is significantly more general than the examples in this work.
    • The experiments in [Alimisis et al., 2025, Riabinin et al., 2026], comparing the proposed warm-up schedules against linear warm-up,-up, are conducted in a more realistic setting.
    • [Alimisis et al., 2025] also provide an example when the gradient-based condition is not satisfied, and prove that the condition is closed under summation and linear transformation. The authors claim that "our work focuses on demonstrating that the $(\rho,K_0,K_\rho) $- smoothness is more suited than (ρ,L0,Lρ)(\rho, L_0, L_\rho)-smoothness for the characterization of the loss landscape. Since this work does not provide any other example beyond toy examples, I find this claim to be not justified.
  • Experiments on the ResNet-18 model require a significant amount of tuning: the authors propose to grid search K0K_0 and KρK_\rho, while one can simply tune the length of linear warm-up instead. This raises questions regarding the practicality of the proposed warm-up. The authors claim that this is not the main goal of this work, but it still demonstrates the limitations of the proposed schedule.
  • The improved convergence in the general non-convex case is not proven. In this case, the derived warm-up schedule implies the convergence rate K0Δ0T+KρΔ0∑t=0T−1ΔtρT2.\frac{K_0\Delta_0}{T} + \frac{K_\rho\Delta_0\sum_{t=0}^{T-1}\Delta_t^\rho}{T^2}. Since ∑t=0T−1Δtρ≤TΔ0ρ\sum_{t=0}^{T-1}\Delta_t^\rho \le T\Delta_0^\rho (line 255), this gives the bound K0Δ0T+KρΔ0ρ+1T.\frac{K_0\Delta_0}{T} + \frac{K_\rho\Delta_0^{\rho+1}}{T}. Therefore, I do not see the improvement over a constant learning-rate analysis.
  • The stochastic analysis is done using strong a.s. noise assumptions, limiting the applicability of the results. Relaxing the assumption to sub-Gaussian or bounded-variance noise conditions would improve the results of the work.
  • The authors should clarify the statement of Theorem 3. Currently, it says "for GD with any non-increasing learning rate sequence ηt\\{\eta_t\\}. However, I can choose ηt\eta_t to be the one from Theorem 2. I believe this implies a contradiction, and the authors should clarify that ηt\eta_t should not depend on ff.
  • The authors provide the improved convergence results only for the case ρ∈(0,1]\rho\in(0,1]. Moreover, for any ρ∈(0,1)\rho\in(0,1) we can bound (f(x)−f∗)ρ(f(x)-f^*)^\rho by A+B(f(x)−f∗)A + B(f(x)-f^*). This means that the improvements are already covered in [Alimimis et al., 2025] to some extent, while for ρ>1\rho>1 the authors provide no improvement.
Questions for authors
  • Please update the figures. When printing the paper, the legends and labels are not visible. The authors are encouraged to pay more attention to the style of the figures to make the paper more readable.
  • In Lemma 7, it is sufficient to check it for m=2m=2. The case m>2m>2 follows by induction on mm. This makes the lemma easier to read.
  • Why do the authors plot Figure 1 on a log-log scale? If we take the log on both sides of Definition 1, I do not see what we should expect in the experiments. Do the authors use clipping in Figure 1? What happens in the training continues in Figure 1?
  • In Lemma 2, can the condition ∣y−x∣≤r(Δ)\\|y-x\\|\le r(\Delta) be removed?
  • Do the authors have an intuition for how to prove the lower bound in the ρ≠1\rho\neq1 case?
  • Can the stochastic analysis be analysed under sub-Gaussian noise or other more relaxed assumptions?
  • In line 280, ρ≥1\rho\ge1 seems to be a typo. It should be ρ∈(0,1]\rho\in(0,1].
17 comments
AuthorsOpenReview ↗

Response to Reviewer MZ73 (Part 1/6)

We thank the reviewer for the careful review and the many comments. We respond below in the order of the review, including each subpoint.

W1 & W3.1. Neural-network examples beyond scalar models

W1: The provided examples are limited to very toy cases where the networks have a few parameters. The authors also don't discuss the potential generalization of the examples to more realistic settings.

W3.1: I find that the examples in [Alimisis et al., 2025] are more realistic: they cover MLP and Transformer under the balancedness condition, which is significantly more general than the examples in this work.

The extension below covers regularized deep linear networks of arbitrary finite width and depth, without balancedness. Our original examples additionally establish a mathematical separation between (ρ,K0,Kρ)(\rho,K_0,K_\rho)-smoothness and (ρ,L0,Lρ)(\rho,L_0,L_\rho)-smoothness. For example, Examples 2 and 3 consider regularized feedforward and recurrent networks. We proved that these losses satisfy (ρ,K0,Kρ)(\rho,K_0,K_\rho)-smoothness for ρ≥3\rho\ge3, while failing to satisfy (ρ,L0,Lρ)(\rho,L_0,L_\rho)-smoothness for 0≤ρ<20\le \rho<2.

The examples in [5] provide broader architectural evidence: deep linear networks under balancedness, regularized two-layer nonlinear networks, and a simplified single-attention-layer Transformer under their stated regularization and distributional assumptions. Since their (H0,H1)(H_0,H_1) condition is the ρ=1\rho=1 case of the (ρ,K0,Kρ)(\rho,K_0,K_\rho)-smoothness, these results also establish applicability of (ρ,K0,Kρ)(\rho,K_0,K_\rho)-smoothness beyond our examples.

Below we discuss a more general example. Consider a deep linear network without biases, with any fixed depth D≥2D\ge2 and arbitrary finite layer widths. Let ww collect all trainable weights, and define its logits and regularized empirical loss on a finite dataset by

\qquad f(w)=\frac1N\sum_{i=1}^N\ell_{y_i}(z_i(w)) +\frac{\lambda}{2}\|w\|^2, \quad \lambda>0,$$ where $\ell_y(z)=\log(\sum_{j=1}^c e^{z_j})-z_y$ is softmax cross-entropy. We show that $f$ satisfies our condition globally with $\rho=D-1$. First, write $r=\|w\|_2$. Each logit is a homogeneous polynomial of degree $D$ in the weights. Its first and second derivatives therefore have degrees $D-1$ and $D-2$. By homogeneity and boundedness on the unit sphere, there are finite constants $a_i,b_i$ such that $$\|J_i(w)\|\le a_i r^{D-1}, \qquad \max_j\|\nabla_w^2z_{ij}(w)\| \le b_i r^{D-2},$$ where $J_i=\partial z_i/\partial w$ is the logit Jacobian. For $D=2$, the second bound is a constant. The constants depend on the fixed architecture and sample, not on $w$. Second, for $s=\operatorname{softmax}(z)$, cross-entropy satisfies $\nabla_z\ell_y=s-e_y$ and $\nabla_z^2\ell_y=\operatorname{diag}(s)-ss^\top$, so $$\|\nabla_z\ell_y(z)\|_1\le2, \qquad \|\nabla_z^2\ell_y(z)\|\le1.$$ The Hessian chain rule gives $$\nabla_w^2\ell_{y_i}(z_i(w)) =J_i^\top\nabla_z^2\ell_{y_i}(z_i)J_i +\sum_{j=1}^c\partial_j\ell_{y_i}(z_i)\,\nabla_w^2z_{ij}(w).$$ Combining these bounds, averaging over the finite dataset, and adding the regularization Hessian $\lambda I$ yields finite constants $A,B,C$ with $$\|\nabla^2f(w)\| \le\lambda+A r^{2D-2}+B r^{D-2} \le C(1+r^{2D-2}).$$ The final inequality follows from $r^{D-2}\le1+r^{2D-2}$ for $D\ge2$. Finally, writing $\Delta(w)=f(w)-f^\star$, we have $$r^2\le\frac{2f(w)}{\lambda} =\frac{2}{\lambda}\bigl(\Delta(w)+f^\star\bigr), \qquad 0\le f^\star\le f(0)=\log c.$$ Set $p=D-1\ge1$. Applying $(a+b)^p\le2^{p-1}(a^p+b^p)$ gives $$\begin{aligned} \|\nabla^2f(w)\| &\le C\left[1+\left(\frac2\lambda\right)^p \bigl(\Delta(w)+f^\star\bigr)^p\right]\\ &\le K_0+K_p\Delta(w)^p, \end{aligned}$$ where we can choose $$K_p=C\,2^{p-1}\left(\frac2\lambda\right)^p, \qquad K_0=C+K_p(f^\star)^p.$$ Thus, the proposed condition holds with $\rho=D-1$ for arbitrary finite widths, with constants independent of $w$. We will add this extension and proof to the discussion. For the regularized cross-entropy objective specified above, this result holds without balancedness. ## W2.1. Evidence for the curvature-based motivation > "Since the primary role of warm-up is to suppress sharpness to allow the network to tolerate larger learning rate,...": this claim is not justified; the authors do not provide enough evidence for this claim, so it cannot be used as a motivation. The authors should work on the wording or provide references that prove such claims. This motivation is supported by empirical evidence in [1,2], both cited and discussed in Section 1. They document a mechanism by which warmup enables larger target learning rates. Specifically, [1] identify the reduction of sharpness as a mechanism that enables training at larger learning rates. [2] study how warmup guides training toward lower-curvature regions that support larger stable learning rates. Our analysis then models this mechanism through a loss-dependent curvature bound.
AuthorsOpenReview ↗

Response to Reviewer MZ73 (Part 2/6)

W2.2. Stationary points, local minima, and the purpose of Example 1

"better at capturing the properties of functions with multiple stationary points or local minima, which is a common case in deep neural network training." Again, to make such a claim, the authors should provide evidence that Example 1 is realistic. I also believe that warm-up does not help in minimizing an arbitrary non-convex function, even if it satisfies the proposed condition. Finally, I do not see local minima in the provided empirical measurements of the smoothness.

The occurrence of multiple local minima in neural-network loss landscapes is well documented, although their prevalence and quality depend on the architecture and training regime [3,4]. We will add these references in our manuscript. These references motivate considering landscapes with multiple stationary points. The purpose of Example 1 is to isolate a structural difference between (ρ,K0,Kρ)(\rho,K_0,K_\rho)-smoothness and (ρ,L0,Lρ)(\rho,L_0,L_\rho)-smoothness. At a local minimum wiw_i, the proposed condition permits the curvature bound

\le K_0+K_\rho\bigl(f(w_i)-f^\star\bigr)^\rho$$ to depend on the objective value at that minimum. Thus, minima with different objective values may have different admissible curvature bounds. In contrast, under $(L_0,L_1)$-smoothness, $\nabla f(w_i)=0$ gives $$\|\nabla^2 f(w_i)\|\le L_0$$ at every stationary point. The distinction is between a loss-dependent bound and a common bound at stationary points; $(\rho, L_0,L_\rho)$-smoothness does not require the actual Hessian norms at different minima to be equal. The convergence claims are precise: Theorem 1 quantifies the trajectory-dependent improvement in a stationarity bound, and Theorem 2 establishes convex complexity gains. A general belief about arbitrary nonconvex minimization does not identify a counterexample to either result; the relevant bounds are given in W5 below. The empirical plots test the loss–smoothness relationship along training trajectories. Example 1 establishes the mathematical distinction at stationary points described above. ## W3. Contribution relative to related warmup analyses > The authors do not discuss the works that propose a similar condition in the main part (also given that the papers are online for a long time). The whole discussion is moved to the appendix. This makes the understanding of the significance and novelty of the results less clear. We will move the essential comparison to the main text. Alimisis et al. [5] explicitly discuss the general-power framework as concurrent work. Our contribution combines the general-$\rho$ smoothness framework and separating examples, explicit convex and nonconvex trajectory bounds, high-probability SGD analysis under the stated noise conditions without PL/aiming or interpolation assumptions, and a lower bound covering predetermined non-increasing sequences. The stochastic finite-sum result in [5] assumes aiming and interpolation; [6] analyzes norm-constrained, LMO-based updates. These distinctions complement the overlap in loss-dependent smoothness. ### W3.1. Comparison with the network examples in [5] The network comparison is given in W1 above: our original examples establish separation from gradient-based smoothness, and the extension covers regularized deep linear networks of arbitrary finite width and depth without balancedness. We will include this extension alongside the discussion of [5] in the main text. ### W3.2. Comparison with the warmup experiments in [5,6] > The experiments in [Alimisis et al., 2025, Riabinin et al., 2026], comparing the proposed warm-up schedules against linear warm-up,-up, are conducted in a more realistic setting. Our experiments examine the loss–smoothness relationship on ResNet18 and NanoGPT, and show that the derived rule exhibits warmup behavior on ResNet18 with performance comparable to linear warmup across six runs (Appendix C). These measurements directly test the mechanism studied here and complement the larger-scale evaluations in [5,6].
AuthorsOpenReview ↗

Response to Reviewer MZ73 (Part 3/6)

W3.3. Separating examples and closure properties

[Alimisis et al., 2025] also provide an example when the gradient-based condition is not satisfied, and prove that the condition is closed under summation and linear transformation. Since this work does not provide any other example beyond toy examples, I find this claim to be not justified.

Our evidence includes neural-network losses as well as the abstract function in Example 1. Specifically, Examples 2 and 3 consider regularized feedforward and recurrent networks. We prove that these losses satisfy (ρ,K0,Kρ)(\rho,K_0,K_\rho)-smoothness for ρ≥3\rho\ge3, while failing to satisfy (ρ,L0,Lρ)(\rho,L_0,L_\rho)-smoothness for 0≤ρ<20\le \rho<2. Lemma 1 additionally establishes the general exponent-dependent implication from (ρ,L0,Lρ)(\rho,L_0,L_\rho)-smoothness to (ρ,K0,Kρ)(\rho,K_0,K_\rho)-smoothness.

The distinction at stationary points is explained in W2.2 above.

The manuscript includes closure under finite sums and affine transformations in Appendix E.4, with a pointer in the main text. Together with the examples above, these results establish both structural coverage and stability under common operations.

W4. Hyperparameter tuning and the practical schedule

Experiments on the ResNet-18 model require a significant amount of tuning: the authors propose to grid search K0K_0 and KρK_\rho, while one can simply tune the length of linear warm-up instead. This raises questions regarding the practicality of the proposed warm-up.

The parameters have a useful interpretation that clarifies the comparison. With ρ=1\rho=1, our rule can be written as

\qquad \eta_{\max} = \frac{1}{(4\sqrt{2}+4)K_0}, \qquad \Delta_{\mathrm{switch}} = \frac{K_0}{3K_1}.$$ Thus, $K_0$ controls the peak learning rate, which also needs to be chosen for linear warmup. At a fixed peak rate, the remaining parameter determines a loss threshold instead of a prescribed warmup duration. Thus, after choosing the peak rate, both methods have one warmup-specific control: a loss threshold for our rule or a duration for linear warmup. Our experiment searches a $4\times4$ grid for the theoretical rule, uses ten epochs of linear warmup, and matches peak learning rates. The six-run accuracies, $85.89\%\pm0.23\%$ and $85.77\%\pm0.23\%$, support comparable performance under this protocol. The schedule comparison illustrates the practical behavior predicted by the theory; the protocol measures performance at matched peak rates, not tuning efficiency. ## W5. Nonconvex trajectory bounds and convex acceleration > The improved convergence in the general non-convex case is not proven. […] Therefore, I do not see the improvement over a constant learning-rate analysis. The substitution $\sum_{t<T}\Delta_t^\rho\leq T\Delta_0^\rho$ discards the trajectory information retained by Theorem 1. Its bound improves as the average gap decreases. Theorem 2 additionally gives explicit convex complexity gains, as follows. Under convexity, Theorem 2 gives, for $0<\rho<1$, $$\Delta_{T-1} \le O_\rho\!\left[ \frac{K_0D_0^2}{T} +\left(\frac{K_\rho D_0^2}{T}\right)^{1/(1-\rho)} \right], \qquad D_0=\|w_0-w^\star\|.$$ The stated constant-step guarantee is $$O_\rho\!\left[ \frac{K_0D_0^2}{T} +\frac{K_\rho D_0^2\Delta_0^\rho}{T} \right].$$ Thus the loss-dependent curvature contribution has a faster decay in $T$ under the analyzed warmup schedule. For $\rho\ge1$, Theorem 2 instead gives the following iteration bounds for function error at most $\epsilon$: $$\begin{aligned} N_{\mathrm{warm}}(\epsilon) &=O_\rho\!\left[ \frac{K_0D_0^2}{\epsilon} +K_\rho D_0^2\Delta_0^{\rho-1} \right],\\ N_{\mathrm{const}}(\epsilon) &=O_\rho\!\left[ \frac{K_0D_0^2}{\epsilon} +\frac{K_\rho D_0^2\Delta_0^\rho}{\epsilon} \right]. \end{aligned}$$ Warmup converts the initialization-dependent curvature penalty into an initial-phase cost, removing its multiplication by $1/\epsilon$. For the nonconvex result, the relevant comparison is $T^{-1}\sum_{t<T}\Delta_t^\rho$ versus $\Delta_0^\rho$, as explained above; both have $O(T^{-1})$ worst-case order after discarding this trajectory dependence. Example 4 additionally demonstrates a concrete iteration-count advantage on a river-valley objective.
Reviewer_8LuuOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper defines a smoothness notion for functions that bound local smoothness with a polynomial of the optimality gap.

The paper shows why this smoothness notion is a relaxation from previous notion that bounds local smoothness with the gradient norm - which cannot capture simple neural networks.

Finally, the paper proves convergence theorems for the new smoothness definition - showing that if the learning rate is small enough, the optimization converges.

Critical strengths

The paper is well organized and well written. The framework is clear and clean, and the new convergence theorem generalizes the existing ones.

Critical weaknesses
  • Going from norm based local smoothness to optimality gap based one (Lemma 1), the polynomial constants can become really large when rho is close to 2 - creating a loose upper bound for the local smoothness.

  • It is not very clear what happens when the bound is loose (say the optimality gap is high), around a flat area / bad local minima - The theoretical LR is kept small, which might mean convergence to high loss or extremely slow training.

  • While I understand that the paper is theory based, a few more experiments beyond simple synthetic settings could benefit the reader especially since 1) there is a bit of tension between edge of stability body of work (which claims sharpness does not necessarily exhibit a monotonic decline) and 2) the paper aims to explain practical methods, and claim that the new smoothness framework expands over previous ones to include real neural networks architectures.

Questions for authors

Please see weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 8Luu (Part 1/2)

We thank the reviewer for highlighting the tightness of the smoothness bound and the connection between our theory and practical training. We address the three concerns below.

W1. Tightness of the gradient-to-gap smoothness conversion

Going from norm based local smoothness to optimality gap based one (Lemma 1), the polynomial constants can become really large when rho is close to 2 - creating a loose upper bound for the local smoothness.

Lemma 1 establishes coverage of the gradient-based class; our convergence analysis directly uses the gap-based condition. The divergence of the required gap-based exponent as ρ→2−\rho\to2^- is unavoidable in general. Below we use ρ\rho for the gradient-based exponent and α\alpha for the gap-based exponent.

Consider a C∞C^\infty, lower-bounded example. For ρ∈[3/2,2)\rho\in[3/2,2), let

\qquad f_\rho(x)=\int_{-\infty}^{x}\bigl(e^{-2t}+\sin^2t\bigr)^{-p}\,dt, \qquad x\in\mathbb R.$$ This function is $C^\infty$ on $\mathbb R$ and has $f_\rho^\star=0$. Let $h(x)=e^{-2x}+\sin^2x$. Direct differentiation gives $$f_\rho'(x)=h(x)^{-p}, \qquad f_\rho''(x)=\frac{2p\bigl(e^{-2x}-\sin x\cos x\bigr)}{h(x)^{p+1}},$$ and, for every $x\in\mathbb R$, $$|f_\rho''(x)|\leq 4+4|f_\rho'(x)|^\rho.$$ Thus, the gradient-based constants can be chosen as **$L_0=L_\rho=4$, uniformly as $\rho\to2^-$**. Nevertheless, at $x_n=n\pi+e^{-n\pi}$, integrating around successive multiples of $\pi$ yields, for each fixed $p>1/2$, $$f_\rho(x_n)=\Theta_p\!\left(e^{(2p-1)n\pi}\right), \qquad |f_\rho''(x_n)|=\Theta_p\!\left(e^{(2p+1)n\pi}\right),$$ where the implicit constants may depend on $p$, but not on $n$. Consequently, any global inequality $$|f_\rho''(x)|\leq K_0+K_\alpha\bigl(f_\rho(x)-f_\rho^\star\bigr)^\alpha$$ with finite constants $K_0,K_\alpha$ necessarily requires $$\alpha\geq\frac{2p+1}{2p-1} =\frac{\rho}{2-\rho} \longrightarrow\infty \qquad\text{as }\rho\to2^-.$$ This matches the exponent in Lemma 1 and establishes its worst-case necessity. The example establishes exponent tightness; it does not assert optimality of every numerical conversion constant. More importantly, the primary purpose of Lemma 1 is to establish the broader coverage of our $(\rho,K_0,K_\rho)$-smoothness family relative to the $(\rho,L_0,L_\rho)$-smoothness family with $\rho < 2$. Together with the separating example in Section 2.1 and the neural-network examples in Section 2.2, it establishes strictly broader coverage. ## W2. Large objective gaps in flat regions > It is not very clear what happens when the bound is loose (say the optimality gap is high), around a flat area / bad local minima - The theoretical LR is kept small, which might mean convergence to high loss or extremely slow training. The proposed rule uses a certified curvature upper bound: as the gap decreases, it permits larger steps. In a flat region with a large gap, this sufficient bound can be conservative; the descent and stationarity guarantees still apply. In the early-training regime motivating our analysis, the loss is initially high but decreases rapidly. As the optimality gap shrinks, the $(\rho,K_0,K_\rho)$-smoothness upper bound decreases accordingly, permitting progressively larger learning rates. Warmup exploits this evolution instead of retaining a conservative fixed learning rate dictated by the initial high-loss state. Consequently, when the optimality gap decreases substantially, our trajectory-dependent analysis yields an improved convergence guarantee over using this conservative fixed learning rate. The nonconvex guarantee concerns first-order stationarity; under the additional convexity and level-boundedness assumptions, Theorem 2 quantifies the improvement in objective-error complexity.
AuthorsOpenReview ↗

Response to Reviewer 8Luu (Part 2/2)

W3. Neural-network experiments and edge-of-stability behavior

While I understand that the paper is theory based, a few more experiments beyond simple synthetic settings could benefit the reader especially since 1) there is a bit of tension between edge of stability body of work (which claims sharpness does not necessarily exhibit a monotonic decline) and 2) the paper aims to explain practical methods, and claim that the new smoothness framework expands over previous ones to include real neural networks architectures.

The neural-network experiments examine both the loss–smoothness relationship and the warmup trajectory of the derived rule, with performance comparable to linear warmup across six ResNet18 runs.

The current manuscript does include experiments beyond synthetic settings: smoothness investigations on ResNet18/CIFAR-10 and NanoGPT/TinyShakespeare in Section 2.3 and Appendix C.1, and a warmup comparison on ResNet18/CIFAR-10 in Appendix C.2. The latter uses 100 epochs and six independent runs, with the same target learning rate and cosine decay after the initial phase. The reported test accuracies are:

ScheduleTest accuracy, mean ± standard deviation
Theoretical warmup85.89%±0.23%85.89\%\pm0.23\%
Linear warmup85.77%±0.23%85.77\%\pm0.23\%
No warmup85.62%±0.19%85.62\%\pm0.19\%

Regarding EoS, a decreasing curvature upper bound is compatible with an initial increase in actual sharpness below that bound. The empirical discussion in Section 3.1 and Appendix B follows [1]: training can exhibit progressive sharpening, followed by sharpness reduction as warmup increases the learning rate and lowers the approximate GD stability threshold 2/ηt2/\eta_t.

These empirically documented regimes motivate our analysis. Our theorems quantify how the loss-dependent curvature bound permits larger steps, connecting directly to the benefit of larger target learning rates observed in [1].

Reference

[1] Dayal Singh Kalra and Maissam Barkeshli. Why Warmup the Learning Rate? Underlying Mechanisms and Improvements. Advances in Neural Information Processing Systems 37 (NeurIPS 2024). arXiv:2406.09405.

Details for the Example in W1

It is not hard to show that ∣fρ′′(x)∣≤4+4∣fρ′(x)∣ρ|f_\rho''(x)|\leq4+4|f_\rho'(x)|^\rho. Next we show the gap-based bound.

Let εn=e−nπ,xn=nπ+εn.\varepsilon_n=e^{-n\pi}, x_n=n\pi+\varepsilon_n.

As n→∞n\to\infty, we have that h(xn)=εn2e−2εn+sin⁡2εn∼2εn2h(x_n)=\varepsilon_n^2e^{-2\varepsilon_n}+\sin^2\varepsilon_n\sim2\varepsilon_n^2 and e−2xn−sin⁡xncos⁡xn=εn2e−2εn−sin⁡εncos⁡εn∼−εn.e^{-2x_n}-\sin x_n\cos x_n=\varepsilon_n^2e^{-2\varepsilon_n}-\sin\varepsilon_n\cos\varepsilon_n\sim-\varepsilon_n.

It follows that fρ′′(xn)∼−p 2−pεn−(2p+1)f_\rho''(x_n)\sim-p\,2^{-p}\varepsilon_n^{-(2p+1)} and consequently

=\Theta_p\!\left(e^{(2p+1)n\pi}\right).$$ Next, consider the interval $$I_k=[k\pi-\pi/2,k\pi+\pi/2], \qquad\varepsilon_k=e^{-k\pi}.$$ For $|s|\leq\pi/2$, we have that $h(k\pi+s)=\varepsilon_k^2e^{-2s}+\sin^2s$. Using $e^{-\pi}\leq e^{-2s}\leq e^\pi$ and $\frac{4}{\pi^2}s^2\leq\sin^2s\leq s^2$, we obtain constants $c,C>0$, independent of $k$, such that $$c(\varepsilon_k^2+s^2) \leq h(k\pi+s) \leq C(\varepsilon_k^2+s^2).$$ For fixed $p>1/2$, substituting $s=\varepsilon_k u$ therefore gives $$\int_{I_k}h(t)^{-p}\,dt \asymp_p \varepsilon_k^{1-2p} \int_{-\pi/(2\varepsilon_k)}^{\pi/(2\varepsilon_k)}(1+u^2)^{-p}\,du \asymp_p\varepsilon_k^{1-2p}.$$ The final comparison holds because $\int_{\mathbb R}(1+u^2)^{-p}\,du<\infty$ for $p>1/2$, while the displayed integrals also have a strictly positive lower bound. Here $\asymp_p$ denotes two-sided inequalities with positive constants that may depend on $p$ but not on $k$. For the upper bound on $f_\rho(x_n)$, sum the contributions of $I_1,\ldots,I_n$ and the finite initial contribution up to $\pi/2$. Since $2p-1>0$, this is a geometric sum: $$f_\rho(x_n) \leq f_\rho(\pi/2)+C_p\sum_{k=1}^{n}e^{(2p-1)k\pi} =O_p\!\left(e^{(2p-1)n\pi}\right).$$ For the lower bound, integrate only over $[n\pi,n\pi+\varepsilon_n]$. On this interval $h(t)\leq2C\varepsilon_n^2$, so $$f_\rho(x_n) \geq\int_{n\pi}^{n\pi+\varepsilon_n}h(t)^{-p}\,dt \geq(2C)^{-p}\varepsilon_n^{1-2p}.$$ Thus, $$f_\rho(x_n)-f_\rho^\star =f_\rho(x_n) =\Theta_p\!\left(e^{(2p-1)n\pi}\right).$$ Now suppose the global gap-based bound holds for finite $K_0,K_\alpha\geq0$. Evaluating at $x_n$ and using the estimates above gives $c_p e^{(2p+1)n\pi}\leq K_0+K_\alpha C_p^\alpha e^{\alpha(2p-1)n\pi}$ for all sufficiently large $n$. If $\alpha(2p-1)<2p+1$, dividing by $e^{(2p+1)n\pi}$ makes the right-hand side tend to zero while the left-hand side remains positive. This contradiction proves $$\alpha\geq\frac{2p+1}{2p-1} =\frac{\rho}{2-\rho}.$$

Thank you for your answers, which I find satisfactory. I'll therefore raise my score.

We offer our sincere appreciation to the reviewer for carefully checking our results and recognizing our paper. Thank you!

Comments on the paper

1 comment
AuthorsOpenReview ↗

A Brief Summary of Rebuttal

Dear AC and reviewers,

Thank you so much for the careful review and discussion. We sincerely appreciate the time and effort the AC and reviewers have devoted to our submission. We are pleased to have had a productive rebuttal, during which we reached substantial agreement on the paper's central contributions and provided additional results to address the remaining questions. We summarize the main points of agreement and our latest responses below.

Points clarified and acknowledged

  • Meaningful theoretical improvements (Reviewer 3ZND, Reviewer MZ73). Both reviewers acknowledged that Theorem 2 gives a provably better convex convergence bound than constant steps. Reviewer 3ZND also recognized the trajectory-dependent nonconvex improvement in Theorem 1 as meaningful and potentially substantial when the loss gap decreases sufficiently rapidly, despite the unchanged worst-case order. Reviewer MZ73 likewise acknowledged that faster convergence is possible in suitable nonconvex regimes. These results support the paper's central contribution: warmup offers provable benefits in concrete settings, with the improvement quantified through the loss trajectory and smoothness parameters.

  • Noise assumptions and other technical clarifications (Reviewer MZ73). We clarified the existing norm-sub-Gaussian guarantees, the consistency of Theorems 2 and 3, and the improvement for ρ>1\rho>1. We reached consensus on these technical questions. We also pointed to the closure properties under finite sums and affine transformations in Appendix E.4 (W3.3), and this specific issue was not raised again in the subsequent replies.

  • A fair constant-step comparison (Reviewer 3ZND). The reviewer confirmed that our baseline is the largest admissible constant step under the same condition, and recognized the separation from the optimal constant learning rate in Appendix F.4.

  • An empirical connection to neural-network training (Reviewer 3ZND, Reviewer 8Luu). Reviewer 3ZND acknowledged that the ResNet18/CIFAR-10 results show similar warmup behavior and comparable accuracy to linear warmup, providing a useful connection to practice. We also addressed Reviewer 8Luu's related questions about neural-network evidence and explained why a decreasing curvature upper bound is compatible with progressive sharpening.

We also sincerely appreciate Reviewer 8Luu's recognization of our paper and rebuttal and the positive assessment of score 4.

Remaining concerns and our latest responses

  • Reviewer 3ZND: the theoretical link to standard linear warmup. The reviewer's remaining concern was whether our mechanism explains practical warmup schedules. In our last reply, we added Theorem 2a and Appendix J, directly analyzing linear warmup in the convex setting of Theorem 2, proving convergence guarantee improvements over constant learning rates in the scale of O(Δ0/ϵ)\mathcal{O}(\sqrt{\Delta_0/\epsilon}) in the generalized smoothness KρK_\rho related terms. This provides direct provable benefits of the practical linear warmup schedule upon constant learning rates under our theoretical settings in the paper, which we think should be good to address the reviewer's concern.

  • Reviewer MZ73: the comparison with concurrent work. Based on the latest reply, this appears to be the reviewer's main remaining concern. We have clarified that the cited warmup studies, including Alimisis et al., are concurrent and explained the technical distinctions. We believe overlap with concurrent work should not, by itself, count against our independent contributions. Our latest reply also supplied the requested nonlinear matrix example, showing that global ρ=3\rho=3 smoothness can hold while ρ=1\rho=1 fails, and clarified how loss descent yields the nonconvex bound improvement. Thus, we would like to note that even in a direct comparison with concurrent work, the general-ρ\rho framework gives meaningful additional coverage and convergence improvements for ρ>1\rho>1.

We have not yet received responses to these latest additions. We fully understand the reviewers' busy schedules and sincerely appreciate their time, careful reading, and helpful feedback throughout the discussion. We believe these results and clarifications help address the remaining concerns. We hope these additions, together with the contributions already acknowledged above, will be helpful to the AC in the final assessment.

Thank you so much for the time and effort spent reviewing our work and for the careful consideration of our submission.

Authors of Submission 16320

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.