acceptodds
Under review as a conference paper at ICLR 2027

Closing the Bias-Correction Gap in Discounted Regret Analysis of Adam

Abstract

Recent discounted-to-nonconvex analyses of Adam establish convergence guarantees by bounding the discounted regret of an associated online learner. However, these analyses typically remove Adam's original bias-correction factors by absorbing them into the learning-rate schedule or clipping radius, and therefore prove guarantees for a reparameterized update. To accurately understand its convergence behavior, we analyze clipped Adam while retaining the original bias correction in this paper. The key difficulty is that bias correction introduces nonmonotone time-dependent coefficients into the regret analysis: the relevant coefficient sequence first decreases and then increases, forming a single-valley structure. Under standard stochastic cost-vector assumptions and the mild parameter coupling, we prove that the effective step sizes remain automatically monotone and develop a summation argument showing that the transient valley cost is controlled by sufficiently many post-valley terms. The resulting discounted-regret guarantees match the known dependence of analyses without retained bias correction, and hence preserve Adam's original bias-corrected parameterization without worsening the regret order.

Primary area:
optimization
Keywords:
Adam, discounted-to-nonconvex conversion, discounted regret, non-convex optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YRn1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies discounted regret for Adam while retaining its original bias-correction factors. Under clipped updates and β1<β2\beta_1<\beta_2, it establishes monotonicity of the effective step sizes, characterizes the single-valley structure of the Abel-summation coefficients, and derives pointwise and cumulative discounted-regret bounds once explicit thresholds are reached. Under fixed proportional coupling 1−β2=r(1−β1)1-\beta_2=r(1-\beta_1), these bounds retain the existing (1−β1)−1/2(1-\beta_1)^{-1/2} parameter dependence. The paper further uses the existing discounted-to-nonconvex conversion to derive nonconvex stationarity guarantees and a corollary for coordinate-wise clipped Adam, with random step-size scaling and model EMA.

Critical strengths
  1. The research objective is specific, and retaining bias correction has technical value. The paper directly handles the original correction factors under a fixed external learning rate. It distinguishes the monotonicity of the effective step sizes from the nonmonotonicity of the subsequent summation coefficients and addresses them separately.

  2. The core regret analysis is largely complete. The single-valley structure, terminal-term control, and absorption of the cumulative remainder are supported by detailed proofs, with explicit constants and thresholds. Proposition 6.4 provides additional insight by characterizing the coefficient required for terminal-term control, while explicitly limiting this characterization to the proof framework. The separate treatment of early rounds in the stationarity derivation appropriately avoids invoking the larger cumulative threshold.

Critical weaknesses
  1. The scope and substantive gains of the new guarantees remain limited. The paper primarily recovers existing leading-order rates while retaining bias correction. Theorem 7.1 requires β1<β2\beta_1<\beta_2, which is stricter than β1≤β2\beta_1\leq\sqrt{\beta_2} (Theorem 1, Nguyen 2025), and also excludes the choice β2=β12\beta_2=\beta_1^2 used by Ahn & Cutkosky. Moreover, the matching pointwise and cumulative bounds apply only once T0T_0 and max⁡(T0,T1)\max(T_0,T_1) are reached, respectively. Let δ=1−β1\delta=1-\beta_1. Under fixed proportional coupling as δ↓0\delta\downarrow0, T1=Θr(δ−2log⁡(1/δ))T_1=\Theta_r(\delta^{-2}\log(1/\delta)), a restriction that cannot be ignored when β1\beta_1 is close to 11. The tightness established in Proposition 6.4 concerns only the terminal-term Abel envelope; it does not characterize the actual early-round regret cost for realizable gradient sequences. Although the paper acknowledges these limitations, its explanation of the actual effect of bias correction remains limited, and the significance of the new guarantees requires fuller justification.

  2. The stationarity complexity still incurs an additional logarithmic loss in some parameter regimes. Let A=G+σ+ϵA=G+\sigma+\epsilon. For any λ>0\lambda>0 and εstat∈(0,A]\varepsilon_{\mathrm{stat}}\in(0,A], Theorem 7.2 displays only T=Ωr(ΔλA2εstat−7/2)T=\Omega_r(\Delta\sqrt{\lambda}A^2\varepsilon_{\mathrm{stat}}^{-7/2}), whereas the sufficient condition actually given in Appendix I is

\Delta\sqrt{\lambda}A^2\varepsilon_{\mathrm{stat}}^{-7/2}, A^3\varepsilon_{\mathrm{stat}}^{-3}\log\frac{A}{\varepsilon_{\mathrm{stat}}}, A^2\varepsilon_{\mathrm{stat}}^{-2} \}$$ The paper notes that lower-order and logarithmic terms are omitted from the main text, but these terms are not lower order throughout the stated range. For example, when $\lambda=\Theta(\varepsilon_{\mathrm{stat}})$ and the other positive problem parameters are fixed, the current proof yields a complexity of $O_r(\varepsilon_{\mathrm{stat}}^{-3}\log(1/\varepsilon_{\mathrm{stat}}))$, whereas the full guarantee in Ahn & Cutkosky (2024), Corollary 12 / Eq. (13) is $O(\varepsilon_{\mathrm{stat}}^{-3})$. Thus, the current analysis has not recovered the optimal order of the compared result in this regime. I recommend retaining the complete conditions in Theorem 7.2 and Corollary 7.3 and explicitly identifying the parameter regimes in which the existing complexity is matched exactly. Covering the case above would require further control of the logarithmic term arising from the early rounds. **Minor issues.** Appendix J should specify the shared scalar learning rate. Coordinate-wise optimization generally yields different values of $\eta$; summing the unoptimized scalar bounds before choosing a common $\eta$ would complete the argument consistently with the algorithm definition. This addition would not change the claimed order. In addition, Remark 4.2 should describe $T_0$ as a sufficient threshold, rather than conclude from it that terminal-term control is unavailable for every $t<T_0$. The intuitive explanation on page 5 that $\sqrt{b_s}+\epsilon$ can decrease at most at rate $\sqrt{\beta_2}$ also needs correction; the formal proof in Appendix B accounts for the time-varying bias-correction factor.
Questions for authors
  1. Does the stronger restriction β1<β2\beta_1<\beta_2 reflect a genuine obstacle caused by retaining bias correction, or a limitation of the current growth-event and Abel-summation analysis? Please clarify how the covered parameter range differs from that of prior results.
  2. When λ=Θ(εstat)\lambda=\Theta(\varepsilon_{\mathrm{stat}}), can the current analysis eliminate the additional logarithmic term? If not, to which parameter regimes should the claim of “optimal complexity” be restricted?
3 comments

We thank the reviewer for the careful assessment. Your comments helped us improve the analysis. The revision removes the stronger coupling restriction and the additional logarithmic iteration factor.


W1. Parameter restriction

The restriction β1<β2\beta_{1}<\beta_{2} was a limitation of our proof. The revised Theorem 7.1 covers 0<β1,β2<10<\beta_{1},\beta_{2}<1 with β12≤β2.\beta_{1}^{2}\le\beta_{2}.

The extension uses two refinements of the existing analysis:

  • 1.1. Sharper parameter comparisons. The previous proof imposed stronger parameter comparisons than necessary in the step-size and movement estimates.
    • Effective-step monotonicity (Proposition 4.3). We need the numerator of the effective step size to decrease at least as fast as its denominator can decrease. With bias correction retained, this requires

    \qquad \phi_k(b):=\frac{b(1-b^k)}{1-b^{k+1}}.$$ Previously, we used the stronger comparison $\phi_k(\beta_1)\le\phi_k(\beta_2)$. The sharper inequality $$\phi_k(\beta_1)^2\le\phi_k(\beta_1^2)\le\phi_k(\beta_2)$$ establishes the required result under $\beta_1^2\le\beta_2$, since $\phi_k$ is increasing. * > **Movement bound (Proposition 5.1).** We retain the existing geometric-growth argument under clipping, but define growth event using $\beta_2^{-s}\|v_s\|^2$ to match the second-moment weights. Taking square roots changes the comparison factor from $(\beta_2/\beta_1)^s$ to $(\sqrt{\beta_2}/\beta_1)^s$. The latter is nondecreasing under $\beta_1^2\le\beta_2$, preserving the $\sqrt2$ growth that bounds the selected sum by a constant times its largest term, including at equality.
  • 1.2. Explicit control of the Abel remainder.
    • Previously, the pointwise bound waited for the remainder R(T)≤0,R(T)\le0, while the cumulative bound paid for the number of declining steps of Hs.H_{s}.
      We now allow a positive remainder and explicitly control its contribution through the relative decreases below.

    • Specifically, we now retain the magnitude of each relative decrease. Lemma 6.3(v) gives d_{s}:=\max\left\\{1-\frac{H_{s+1}}{H_{s}},0\right\\},\qquad\sum_{s=2}^{\infty}d_{s}\le\log\frac{1}{1-\beta_{1}}. Here, dsd_s measures the relative decrease of the Abel coefficient HsH_s from step ss to s+1s+1, and is zero when Hs+1≥HsH_{s+1}\ge H_s. We therefore pay for the size of each decrease rather than counting every declining step equally.
      The expected discounted remainder is at most CHη(G+σ)∑s=2T−1β1T−sdsC_{\mathrm{H}}\eta(G+\sigma)\sum_{s=2}^{T-1}\beta_{1}^{T-s}d_{s} (proof of Proposition 6.5). This gives bounds for every horizon.

The next response explains why this logarithmic transient does not introduce an additional logarithmic factor in the stationarity complexity.


W2. Complete complexity and the additional logarithm

The reviewer is correct that the omitted terms in the submitted theorem were not uniformly lower order. The revised analysis establishes the complete sufficient condition T=\Omega_{r}\left(\max\left\\{\Delta\sqrt{\lambda}A^{2}\varepsilon_{\mathrm{stat}}^{-7/2}, A^{3}\varepsilon_{\mathrm{stat}}^{-3}\right\\}\right),\qquad A=G+\sigma+\epsilon, for the fixed ratio defined in Section 7. In particular, when λ=Θ(εstat)\lambda=\Theta(\varepsilon_{\mathrm{stat}}) and the other problem parameters are fixed, Or(εstat−3)O_{r}(\varepsilon_{\mathrm{stat}}^{-3}) iterations suffice, with no additional logarithm, matching the dependence in [Ahn and Cutkosky (2024)].

The remainder bound above lets us use the weighted regret combination β1RegretT+(1−β1)∑t=1TRegrett\beta_{1}\mathrm{Regret}_{T}+(1-\beta_{1})\sum_{t=1}^{T}\mathrm{Regret}_{t} directly for every horizon, without separately bounding the rounds before recovery. In this combination, each dsd_{s} has total weight β1,\beta_{1}, since β1T−s+1+(1−β1)∑t=s+1Tβ1t−s=β1(2≤s<T).\beta_{1}^{T-s+1}+(1-\beta_{1})\sum_{t=s+1}^{T}\beta_{1}^{t-s}=\beta_{1}\qquad(2\le s<T).

Consequently, Lemma 6.3(v) bounds their total contribution by β1log⁡(1/(1−β1)).\beta_{1}\log(1/(1-\beta_{1})). After choosing η\eta and dividing by DT,DT, the finite-horizon contribution has order A[1+log⁡(1/(1−β1))]T1−β1=O(AT(1−β1)),\frac{A[1+\log(1/(1-\beta_{1}))]}{T\sqrt{1-\beta_{1}}}=O\left(\frac{A}{T(1-\beta_{1})}\right), because 1−β1[1+log⁡(1/(1−β1))]≤2/e.\sqrt{1-\beta_{1}}[1+\log(1/(1-\beta_{1}))]\le2/\sqrt{\mathrm{e}}. The logarithm remains in the explicit regret transients, but is absorbed into the existing A/[T(1−β1)]A/[T(1-\beta_{1})] term in the stationarity conversion (Eq. (7)).

Thus no additional logarithmic iteration factor is needed. Theorem 7.2 and Corollary 7.3 now display the full complexity, and the lower-bound comparison states its applicable regime. Appendix I establishes the complete stationarity complexity.

Corrections to minor issues

Shared learning rate. Appendix J now sums the unoptimized scalar bounds before selecting the common η.\eta.

Recovery threshold. Remark 4.2 now describes T0T_{0} as a sufficient threshold.

Denominator comparison. We have corrected the explanation to retain the time-varying bias correction.

Dear Reviewer,

Thank you again for taking the time to review our paper and share your feedback. We have provided a detailed response addressing your comments and concerns.

When you have a chance, we would greatly appreciate your thoughts on our response and would be happy to discuss any remaining questions or concerns.

Thank you again for your time and consideration.

Reviewer_2aV5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to provide a discounted-to-nonconvex analyses of Adam, addressing the issue of the existing analysis that ignores the bias correction factor.

Then it uses the derived discounted-regret bound to derive the stationarity of Adam and coordinate-wise clipped Adam.

This is a theoretical paper and hence no experiments are provided.

Critical strengths
  1. It provides discounted regret bounds for the original adam with the bias correction term, which is missing in earlier papers

  2. Its analysis leads to stationarity convergence guarantee of Adam without smoothness assumption.

Critical weaknesses
  1. while providing analysis to the exact Adam is interesting, it is unclear why bias correction is important in practice.

  2. The result in Theorem 7.2 needs more clarification. It is unclear what does the notation ∥∇F(xˉt)∥[λ]\|\nabla F(\bar x_t)\|^{[\lambda]} mean here. since you are not assuming smoothness, then ∇F(xˉt)\nabla F(\bar x_t) is not necessarily defined. What is exactly λ\lambda here? The paper says ϵstat−3.5\epsilon_{stat}^{-3.5} matches the known lower bound, which is quite different from the standard lower bound of ϵ−4\epsilon^{-4} for non-convex smooth optimization. The paper should have more discussions here.

  3. Can the complexity result recover the standard complexity of O(ϵ−4)O(\epsilon^{-4}) for the smooth case?

  4. The coordinate-wise clipped Adam is not clearly explained in the paper. The authors should make it clear.

Questions for authors

see weakness section.

2 comments

We thank the reviewer for these helpful questions. We have clarified the practical motivation and the stationarity results as follows.


W1. Practical importance of bias correction

Thank you for raising this important clarity issue.

Adam initializes its moment estimates at zero, which biases the early estimates toward zero. Bias correction compensates for this effect and can substantially change the early update magnitudes.

This effect has two complementary interpretations: practical scaling\textit{practical scaling} and normalized averaging\textit{normalized averaging}.

  • Practical scaling.

    • For a concrete example, set ϵ=0\epsilon=0 to isolate the correction factors. After kk observed gradients, the corrected update is the uncorrected update multiplied by ck=1−β2k1−β1k,c_{k}=\frac{\sqrt{1-\beta_{2}^{k}}}{1-\beta_{1}^{k}}, before clipping and for the same moment history. With (β1,β2)=(0.9,0.999),(\beta_{1},\beta_{2})=(0.9,0.999), this factor is 0.3160.316 at k=1k=1 and 0.1520.152 at k=12:k=12: omitting correction makes these updates about 3.163.16 and 6.576.57 times larger, respectively. The factor exceeds 0.90.9 only at k=1660,k=1660, so the effect persists well beyond the first few steps.

    • This calculation illustrates the size and duration of the scaling effect. For empirical evidence, Kingma and Ba (2015, Section 6.4, Figure 4) report training instability without correction in their experiments when β2\beta_{2} is close to one.

  • Normalized averaging (convex combination).

    • Bias correction makes the first and second moments convex combinations of the observed gradients and squared gradient norms, respectively.

    • For example, for t≥2,t\ge2, bt=∑s=1t−1(1−β2)β2t−1−s1−β2t−1∥vs∥2,∑s=1t−1(1−β2)β2t−1−s1−β2t−1=1.b_{t}=\sum_{s=1}^{t-1}\frac{(1-\beta_{2})\beta_{2}^{t-1-s}}{1-\beta_{2}^{t-1}}\|v_{s}\|^{2},\qquad\sum_{s=1}^{t-1}\frac{(1-\beta_{2})\beta_{2}^{t-1-s}}{1-\beta_{2}^{t-1}}=1.

    • Without correction, the observed weights sum to 1−β2t−1,1-\beta_{2}^{t-1}, with the remaining weight on the initial zero. Removing this initialization effect calibrates the moments to the observed gradients from the start: if all observed gradients equal g,g, the corrected moments are exactly gg and ∥g∥2\|g\|^{2} at every step. Thus, even under a constant gradient, the update scale is not distorted merely because few iterations have elapsed. This explains the role of initialization correction discussed by Kingma and Ba (2015, Section 3).

Our introduction now explains the initialization effect and cites the empirical evidence above.


W2. Stationarity measure

We agree that placing the stationarity definition only in the appendix made the convergence results harder to interpret.

We use the established stationarity criterion from this line of nonsmooth nonconvex analysis, as in [Ahn and Cutkosky (2024, Definition 3)]. We originally placed its formal definition in Appendix I to save space, and we have now moved the definition into Section 7.1: \|\nabla F(x)\|^{[\lambda]}=\inf_{\mathrm{E}[y]=x}\left\\{\|\mathrm{E}[\nabla F(y)]\|+\lambda\mathrm{E}\|y-x\|^{2}\right\\}.

We call xx (λ,εstat)(\lambda,\varepsilon_{\mathrm{stat}})-stationary when this measure is at most εstat.\varepsilon_{\mathrm{stat}}. The first term measures the norm of an averaged gradient; the second penalizes the spread of the averaging distribution. Intuitively, xx is the mean of a sufficiently concentrated distribution of nearby points whose average gradient has small norm.

A larger λ\lambda imposes stronger concentration. We have added this interpretation to the main text as well.


W3. Recovering the smooth complexity

Yes, the derived complexity result in our work can recover the standard complexity of O(ε−4)O(\varepsilon^{-4}) for the smooth case.

If ∇F\nabla F is LL-Lipschitz with L>0,L>0, the stationarity definition yields the following inequality; Appendix I gives the derivation: ∥∇F(x)∥≤∥∇F(x)∥[λ]+L24λ.\|\nabla F(x)\|\le\|\nabla F(x)\|^{[\lambda]}+\frac{L^{2}}{4\lambda}.

Choosing λ=L2/εstat\lambda=L^{2}/\varepsilon_{\mathrm{stat}} makes the additional term equal to εstat/4.\varepsilon_{\mathrm{stat}}/4. Since Theorem 7.2 bounds the expected generalized stationarity measure by O(εstat),O(\varepsilon_{\mathrm{stat}}), the expected gradient norm is also O(εstat).O(\varepsilon_{\mathrm{stat}}). Substituting this choice into Theorem 7.2 yields the sufficient condition T\gtrsim\max\left\\{\Delta LA^{2}\varepsilon_{\mathrm{stat}}^{-4}, A^{3}\varepsilon_{\mathrm{stat}}^{-3}\right\\},\qquad A=G+\sigma+\epsilon.

This recovers the standard fourth-power accuracy dependence. We have added this consequence after Theorem 7.2 and its derivation in Appendix I.

W4. Coordinate wise clipped Adam

We have expanded the discussion before Corollary 7.3 to explain how the coordinate-wise update yields its regret and stationarity bounds. Appendix J gives the full proof.

  • Coordinate update.

    Each coordinate is a one-dimensional copy of the learner in Section 3.1. For t≥2,t\ge2, zt[i]=clip⁡D(−αt[i]∑s=1t−1β1−svs[i]),z1[i]=0,z_{t}[i]=\operatorname{clip}_{D}(-\alpha_{t}[i]\sum_{s=1}^{t-1}\beta_{1}^{-s}v_{s}[i]),\qquad z_{1}[i]=0, αt[i]=η(1−β1)β1t−1(1−β1t−1)(bt[i]+ϵ),bt[i]=1−β21−β2t−1∑s=1t−1β2t−1−svs[i]2.\alpha_{t}[i]=\eta\frac{(1-\beta_{1})\beta_{1}^{t-1}}{(1-\beta_{1}^{t-1})(\sqrt{b_{t}[i]}+\epsilon)},\qquad b_{t}[i]=\frac{1-\beta_{2}}{1-\beta_{2}^{t-1}}\sum_{s=1}^{t-1}\beta_{2}^{t-1-s}v_{s}[i]^{2}. Both corrections are retained, with each coordinate clipped to [−D,D].[-D,D].

  • Regret aggregation.

    The linear loss separates, so Regrett[β](ut)=∑i=1dRegrett,i[β](ut[i]).\mathrm{Regret}_{t}^{[\beta]}(u_{t})=\sum_{i=1}^{d}\mathrm{Regret}_{t,i}^{[\beta]}(u_{t}[i]). Under Assumption J.1, we sum the scalar bounds before choosing the common η=D/[2CH(1−β1)].\eta=D/[2\sqrt{C_{\mathrm{H}}(1-\beta_{1})}]. This replaces the scalar cost bound by A1=∑i=1d(Gi+σi+ϵ)A_{1}=\sum_{i=1}^{d}(G_{i}+\sigma_{i}+\epsilon) and gives both regret bounds in Corollary 7.3; coordinate independence is not required.

  • Stationarity conversion.

    Under Assumption J.2, the L1L_{1} discounted-to-nonconvex conversion uses this aggregated weighted regret. Only the averaged-gradient norm changes to L1;L_{1}; the dispersion remains squared Euclidean distance. Coordinate clipping gives ∥zt∥2≤dD2,\|z_{t}\|^{2}\le dD^{2}, so the localization term has order λdD2/(1−β1)2.\lambda dD^{2}/(1-\beta_{1})^{2}. The parameter choices in Appendix J then give the stationarity complexity in Corollary 7.3, including its d\sqrt{d} factor.

Reviewer_nWEFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In summary, the paper studies the Adam algorithm in a discounted regret analysis setting. Concisely, it contributes the knowledge of how to retain the original bias correction mechanism of Adam and still obtain the known bounds through detailed analysis of certain terms temporal evolution. The validation of the approaches are explained throughout the paper through theoretical findings and proofs with additional supplementation in the appendix.

Critical strengths

The submission is clear, the flow of the paper was well designed and it was easy to follow starting from the first page.

The submission appears technically correct, though for a detailed analysis delving into the appendix is a must.

Critical weaknesses

The submission is not experimentally rigorous and reproducible, since it is a pure theory paper.

The submission may present novel findings both theoretically (through regret analyses) and algorithmically (through Adam parametrization), though the significance of the findings (e.g., whether they are incremental) is not completely obvious.

Questions for authors

Line 15: why is reparameterization a problem?

Line 61: consequence?

Line 130: specify the exact bias correction here, i.e, what you retained compared to the past works.

Line 240: so the denominator decrease faster, wouldn't that make it increasing? I think this sentence should be rewritten.

Proposition 5.1: explain here how the O(T) is sufficient.

Line 323: specify I_t>=0 as necessary.

Theorem 7.2: what is delta?

2 comments

We thank the reviewer for recognizing the contribution and for raising important questions.


Line 15 Reparameterization

Thank you for raising this important clarity issue.

  • Understanding bias correction.
    • Treating the absorbed factors as an unspecified schedule leaves their interaction with discounting and adaptive normalization unexplained.

    • Our analysis keeps this dependence explicit: effective step sizes remain nonincreasing, while the regret coefficient can decrease, producing an Abel remainder that must be bounded. This identifies how a core component affects Adam's guarantees and may inform future optimizer design.

Additionally,

  • Coverage of existing guarantees.
    • Reparameterization is algebraically valid, but the existing discounted-regret analyses do not directly yield our target bounds with both bias corrections retained and a constant external learning rate across the stated parameter range.

    • For example, even with ϵ=0,\epsilon=0, the induced schedule is nonmonotone at (β1,β2)=(0.9,0.999),(\beta_{1},\beta_{2})=(0.9,0.999), violating the nonincreasing-schedule assumption of [Nguyen (2025)]. One must therefore check both the theorem's applicability and the quality of the resulting bound.

We have clarified both motivations in the introduction.


Clarifications and corrections

Line 61. We have replaced “consequence” with “stationarity guarantee”.

Line 130. We have added the comparison directly after the increment in Section 3.1: our update retains the first- and second-moment divisors 1−β1t−11-\beta_{1}^{t-1} and 1−β2t−1,1-\beta_{2}^{t-1}, whereas the displayed updates in Ahn and Cutkosky (2024, Section 6.1) and Nguyen (2025, Eq. 1) omit them by absorbing their effects into the scalar scale.

Line 240. We have corrected the intuitive explanation: the numerator decreases at least as fast as the denominator can decrease, so their ratio is nonincreasing.

Proposition 5.1. The conversion multiplies the cumulative bound by (1−β1)/(DT),(1-\beta_{1})/(DT), so its O(D(G+σ)T/1−β1)O(D(G+\sigma)T/\sqrt{1-\beta_{1}}) contribution becomes O((G+σ)1−β1).O((G+\sigma)\sqrt{1-\beta_{1}}). The factor TT cancels, and choosing β1\beta_{1} close to one makes this term small; we have added this explanation after the proposition.

Line 323. We have explicitly specified It≥0I_{t}\ge0 there.

Theorem 7.2. We have defined Δ=F(x0)−inf⁡xF(x),\Delta=F(x_{0})-\inf_{x}F(x), the initial objective gap, directly in the theorem.

Thank you for your answers. My concerns have been addressed. I will keep my recommendation for now.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.