acceptodds
Under review as a conference paper at ICLR 2027

LOVA: Combating Local Variance Dominance for Efficient Neural Network Training

Abstract

Adam is one of the most dominant optimizers for training neural networks. Nevertheless, it has been challenged by matrix-based optimizers recently. In this work, we identify that the signal is often dominated by the random noise in Adam, an issue that becomes increasingly severe in large-batch training. The consequence is catastrophic—Adam overdampens the updates and heavily impedes the training efficiency. To address this issue, we propose Lova (Local Variance Adaptation), a memory-efficient optimizer that preserves the signal drift while retaining the adaptivity of coordinate-wise learning rate. It should be highlighted that Lova tracks only the first-moment buffer, and the memory footprint is as low as stochastic gradient descent (SGD) with momentum. While Adam is not guaranteed to converge for smooth nonconvex functions, theoretical analysis suggests that Lova, under standard assumptions, achieves a convergence rate of . Importantly, this rate holds for an arbitrary constant and eliminates the polynomial dependence on the reciprocal of a small constant for numerical stability, . Extensive experiments on various architectures, datasets, and tasks demonstrate the superiority of Lova over AdamW and other popular optimizers.

Primary area:
optimization
Keywords:
deep learning, adaptive algorithm, stochastic gradient decent, large-batch training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fgG3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes Lova (Local Variance Adaptation), a memory-efficient adaptive optimizer motivated by what the authors call "local variance dominance" in Adam. Lova maintains a first-moment buffer and constructs a memoryless denominator from the squared momentum and squared gradient–momentum residual:

v_t^2 = \beta_2 m_t^2 + (1-\beta_2)(g_t-m_t)^2.$$ This eliminates the persistent second-moment buffer while retaining coordinate-wise adaptive scaling. The authors present asymptotic orthogonality results for the momentum and residual, an $O(\sqrt{T})$ convex regret bound for a maximum-tracking variant, and an $O(\log T/T^{1/4})$ nonconvex stationarity bound under time-varying momentum and learning-rate schedules. The reported nonconvex bound avoids polynomial dependence on $1/\epsilon$. Experiments cover image classification, learning with noisy labels, language-model pretraining, GLUE fine-tuning, diffusion-transformer training, and Atari reinforcement learning. The paper reports improvements over the evaluated baselines, together with reduced memory use and increased throughput in the tested language-model pretraining setting.
Critical strengths

1. A simple, memory-efficient update with a useful structural guarantee.

Lova requires only one persistent momentum buffer and avoids maintaining a historical second moment. Its denominator satisfies vt,i2≥β2mt,i2v_{t,i}^2 \geq \beta_2 m_{t,i}^2, which directly implies

\leq \frac{1}{\sqrt{\beta_2}}.$$ Thus, the coordinate-wise update is bounded without relying on the numerical stabilizer to control its magnitude. This is an appealing combination of implementation simplicity, reduced optimizer-state storage, and controlled updates, independently of the stronger mechanistic claims (Algorithm 1 and Section 3.2). **2. Broad empirical evaluation with encouraging results.** The evaluation extends beyond a single architecture or task. The classification and GLUE tables report improvements across multiple settings, and the diffusion experiments show substantial reductions in FID for both evaluated model sizes (Tables 1, 3, and 4). The classification experiments use three random seeds, while the reinforcement-learning results include confidence intervals over ten trials (Section 5.1 and Figure 5). This breadth provides meaningful evidence that the update is worth investigating, although the comparison protocol limits the strength of the superiority claim.
Critical weaknesses

1. The main comparisons do not establish an advantage over appropriately tuned baselines.

The ImageNet experiments use a global batch size of 20,480, and the caption of Table 1 explicitly states that no learning-rate scaling is applied to illustrate AdamW's deterioration in large-batch training. This evaluates robustness to a particular unchanged configuration, but does not establish superiority over a baseline configured for that batch size. Increasing batch size at a fixed epoch budget also reduces the number of parameter updates, so the experiment does not isolate the proposed variance-related explanation.

Similarly, the language-model experiments fix the peak learning rate and weight decay across optimizers without reporting an optimizer-specific search (Section 5.2). Identical nominal hyperparameters are not necessarily inappropriate, but the paper needs to establish that they produce competitive configurations for each method. Otherwise, the observed differences may reflect update scaling or hyperparameter sensitivity rather than the proposed normalization mechanism. AdamS is included in the classification and fine-tuning tables but not in the language-model pretraining comparison; including this comparator would also strengthen the evidence.

2. The theoretical justification of the central mechanism is not established as written.

In the proof of Theorem 3.4, page 14, the authors write

E[mt−1]=∑k=1t−1wk,t−1∇f(θk−1),\mathbb{E}[m_{t-1}] = \sum_{k=1}^{t-1} w_{k,t-1}\nabla f(\theta_{k-1}),

and consequently identify mt−1−E[mt−1]m_{t-1}-\mathbb{E}[m_{t-1}] with a weighted sum of stochastic noise terms. However, the historical iterates are random, so the unconditional expectation should instead be

E[mt−1]=∑k=1t−1wk,t−1E[∇f(θk−1)].\mathbb{E}[m_{t-1}] = \sum_{k=1}^{t-1} w_{k,t-1}\mathbb{E}[\nabla f(\theta_{k-1})].

The displayed decomposition therefore does not hold as stated. This is not merely a naming issue: the later argument uses the properties of a deterministic expectation to reduce a cross term to a negative variance (page 16). The proof of Theorem 3.6 explicitly reuses this argument and inherits the gap.

There is also a distinction between the stated orthogonality result and the interpretation used to motivate the optimizer. A vanishing expected global inner product does not establish a coordinate-wise, instantaneous decomposition of squared gradients: cross terms can cancel across coordinates or in expectation. Moreover, gt−mtg_t-m_t includes momentum tracking error as well as stochastic noise. Consequently, the diagnostics do not by themselves establish that the observed residual dominance is specifically noise dominance (Sections 3.1 and 3.2).

Questions for authors

1. Clarification of tuning and available comparison evidence.

What learning-rate, weight-decay, momentum, and normalization settings were considered for each optimizer, and how were the final configurations selected? In particular, do the main ImageNet and language-model advantages persist with comparable optimizer-specific tuning budgets? Please provide any available evidence and clarify the omission of AdamS from the pretraining comparison.

2. Clarification and correction of the orthogonality proof.

How should the expectation decomposition on page 14 be interpreted given that the historical iterates are random? Can the subsequent cross-term argument be corrected while retaining the assumptions and rates of Theorems 3.4 and 3.6? A corrected derivation would materially affect my assessment.

3. Clarification of the mechanism and its practical scope.

How is the ratio in Figure 1(a) aggregated, and what evidence distinguishes stochastic noise from momentum lag or changing true gradients? Please also clarify the scope of the orthogonality claim for the practical optimizer: Algorithm 1 uses constant momentum, whereas the orthogonality theorems require β1,t→1\beta_{1,t}\to 1. Are the Figure 2 diagnostics obtained with constant or time-varying momentum, and what conclusion do they support about coordinate-wise residual scaling?

1 comment

Many thanks for the valuable comments, and they are addressed one by one as follows.

(Q1): ImageNet experiments use a global batch size of 20,480 and no learning-rate scaling is applied.

(R1): In the revision (Table 1), we have included the results using a root square learning rate scaling, see Table below:

AdamWMARSAdamSSIGNUMAdaBeliefLova
ResNet-1867.16 ±\pm 0.1466.69 ±\pm 0.2267.34$\pm$ 0.3067.65 ±\pm 0.1167.08 ±\pm 0.1468.69 ±\pm 0.05
ResNet-5073.79 ±\pm 0.1873.13 ±\pm 0.4174.56 ±\pm 0.2874.09 ±\pm 0.3573.97 ±\pm 0.1475.32$\pm$ 0.17

We can observe that Lova performs even comparable to using small batch sizes (approximately at 76% for batch size=256).

(Q2): Does language-model advantages persist with comparable optimizer-specific tuning budgets? Please provide any available evidence and clarify the omission of AdamS from the pretraining comparison.

(R2): Sorry for the missing details. In the revision, we have included all training details in Appendix A.4. In particular, we have added AdamS and Lion in LLM training (see Figure 4). We can observe that the superoirty of Lova is retained, particularly for over-training. Moreover, Lion and AdamS also perform better than AdamW on this small-scale model. But the gap between Lion and AdamW quickly diminishes as more tokens are progressed.

(Q3): How should the expectation decomposition on page 14 be interpreted given that the historical iterates are random? Can the subsequent cross-term argument be corrected while retaining the assumptions and rates of Theorems 3.4 and 3.6? A corrected derivation would materially affect my assessment.

(R3): Thanks for pointing out this flaw. Fortunately, this issue can be easily fixed under same assumptions and the optimal rate is still 1/2. The ctritical step is to bound the gradient–noise covariance. Please see Appendix A.5 for the corrected proofs of Theorems 3.4 and 3.6.

(Q4): How is the ratio in Figure 1(a) aggregated, and what evidence distinguishes stochastic noise from momentum lag or changing true gradients?

(R4): The ratio in Figure 1(a) actually is the ratio between ∥gt−mt∥2\|\boldsymbol{g} _ t - \boldsymbol{m} _ t\| ^ 2 and ∥mt∥2\|\boldsymbol{m} _ t\| ^ 2 and is calculated at the end of each epoch and then averaged over all epochs.
To distinguish stochastic noise from momentum lag, note that the residual gt−mt\textbf{g} _ t - \textbf{m} _ t is actually the sum of the sampling noise gt−∇f(θt)\textbf{g} _ t - \nabla f(\boldsymbol{\theta} _ t) and the momentum tracking error ∇f(θt)−mt\nabla f(\boldsymbol{\theta} _ t) - \textbf{m} _ t. However, as shown in Figure 7 (Appendix A.1), for most coordinates the sampling noise dominates and the tracking error is negligible. This implies that mt\textbf{m} _ t estimates the expected gradient ∇f(θt)\nabla f(\boldsymbol{\theta} _ t) in the long run and that we refer to gt−mt\textbf{g} _ t - \textbf{m} _ t as the local variance is approximately correct. To avoid ambiguity, we added a footnote to local variance.

(Q5): Please also clarify the scope of the orthogonality claim for the practical optimizer: Algorithm 1 uses constant momentum, whereas the orthogonality theorems require β1,t→1\beta_{1,t}\to 1. Are the Figure 2 diagnostics obtained with constant or time-varying momentum, and what conclusion do they support about coordinate-wise residual scaling?

(R5): We want to clarify that the Figure 2 diagnostics are obtained with constant momentum. But when a varying β1,t\beta _ {1,t} is used, we also can observe the same trend. To verify this, we trained a ResNet-56 on CIFAR-10 for 200 epochs using AdamW and Lova. The initial learning rate is 3.0e-4 and decayed according to the cosine scheduler. In particular, we set β1,t=1−1/t\beta _ {1,t}=1-1/\sqrt{t} while fixing β2\beta _ 2. Figure 8 (see Appendix A.2) suggests that the inner-product indeed stays around zero and Lova decays in a faster rate. Moreover, Figure 7 (see Appendix A.1) also suggests for most coordinates the sampling noise dominates and the tracking error is negligible. Therefore, the decomposition gt2≈mt−12+(gt−mt−1)2\boldsymbol{g} _ t ^ 2 \approx \boldsymbol{m} _ {t-1} ^ 2 + (\boldsymbol{g} _ t - \boldsymbol{m} _ {t-1}) ^ 2 is practically plausible.

At last, many thanks for your time and effort. Please let us know if you have further questions.

Reviewer_CTdMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces LOVA, an adaptive optimizer that retains one momentum buffer and constructs its denominator from a weighted combination of squared momentum and the squared gradient–momentum residual. The rule of LOVA interpolates between signum (β2=1\beta_2=1) and an AdaBelief-like update (β2=0\beta_2=0). The theory proves an O(T)O(\sqrt{T}) regret bound with an AMSGrad-style maximum and an O(T−1/4ln⁡T)O(T^{-1/4}\ln T) rate for any β2\beta_2. Experiments include image classification, language model pretraining and fine-tuning, diffusion transformers, and rl, with reported gains over other optimizers.

Critical strengths
  • The optimizer is simple, Algorithm 1 eliminates the second-moment buffer while preserving coordinatewise scaling.

  • Experiments show meaningful improvements of proposed optimizer in a wide range of tasks.

Critical weaknesses
  • AdamS (possibly the closest prior work) already uses a memoryless denominator, the root of a weighted sum of squares of the momentum and the current gradient with the memory of SGD with momentum. Lova differs only by centering the gradient term in the manner of AdaBelief. The experiments use AdamS as a baseline in two tables but the paper didn't discuss the relation, and AdamS, signum and Lion do not appear in the llm-pretraining and DiT experiments where the strongest claims are made.

  • The statement in the abstract that Adam is not guaranteed to converge is somewhat misleading. There are many prior works proving its convergence even without modification, including [1, 2]. Also the orthogonality theorems require β1,t→1\beta_{1,t}\to1 and the proof treats E[mt−1]=∑kwk,t−1∇f(θk−1)\mathbb{E}[m_{t-1}] = \sum_k w_{k,t-1} \nabla f(\theta_{k-1}) as deterministic although the iterates are stochastic.

[1] Haochuan Li, Alexander Rakhlin, and Ali Jadbabaie. "Convergence of adam under relaxed assumptions." Advances in Neural Information Processing Systems 36 (2023): 52166-52196.

[2] Alexandre Défossez, Léon Bottou, Francis Bach, and Nicolas Usunier. "A simple convergence proof of adam and adagrad." arXiv preprint arXiv:2003.02395 (2020).

Questions for authors
  • Please report the learning rates used for every optimizer in Tables 1–3 and Figure 5.

  • Could you discuss the relation to AdamS explicitly and add AdamS and signum/Lion to the llm/DiT experiments?

  • Figure 6 shows ResNet-56 on CIFAR-100 at 55-62%, but Table 1 reports 69.1%. Why is that?

5 comments

Many thanks for the valuable comments, and they are addressed one by one as follows.

(Q1): The statement in the abstract that Adam is not guaranteed to converge is somewhat misleading.

(R1): Agree. While the vanilla Adam is not guaranteed to converge, the references indicate that Adam indeed can converge though with very strong assumptions. To avoid ambiguity, we have removed this statement.

(Q2): The proof treats E[mt]\mathbb{E}[\boldsymbol{m} _ t] as deterministic although the iterates are stochastic.

(R2): Thanks for pointing out this flaw. Fortunately, this issue can be easily fixed under same assumptions and the optimal rate is still 1/2. The ctritical step is to bound the gradient–noise covariance. Please see Appendix A.5 for the corrected proof.

(Q3): Please report the learning rates used for every optimizer in Tables 1–3 and Figure 5.

(R3): Sorry for the missing details. In the revision, we have included all training details in Appendix A.4. Please check it out.

(Q4): Could you discuss the relation to AdamS explicitly?

(R4): In the revision, we discussed AdamS in Section 3.2. Actually, AdamS also uses a memoryless denominator to reduce memory footprint, where the second-order statistic is defined as vt2=β2mt−12+(1−β2)gt2\boldsymbol{v} _ t ^ 2=\beta _ 2 \boldsymbol{m} _ {t-1} ^ 2 + (1-\beta _ 2) \boldsymbol{g} _ t ^ 2 instead. Different from Lova, AdamS is proposed to mimic AdamW by replacing the square of historical gradients with mt−12\boldsymbol{m} _ {t-1} ^ 2. However, these two terms could dramatically differ in magnitudes since mt−1\boldsymbol{m} _ {t-1} approaches to the expected gradient E[gt]\mathbb{E}[\boldsymbol{g} _ t]. For example, when E[gt]\mathbb{E}[\boldsymbol{g} _ t] is very close to zero towards the end of training, the stochastic noise can still be relatively large.

(Q5): Could you add AdamS and signum/Lion to the llm/DiT experiments?

(R5): We have included AdamS and Lion in LLM pretraining, but we only add AdamS in DiT model because Lion/SIGNUM abruptly explodes towards the end of training. The results suggest that AdamS performs better than AdamW, but still performs worse than Lova. For example, on DiT models, FID of AdamS are 52.57, 32.11 while FID of Lova are 49.84, 29.54. Please refer to Figure 3 and 4.

(Q6): Figure 6 shows ResNet-56 on CIFAR-100 at 55-62%, but Table 1 reports 69.1%. Why is that?

(R6): Sorry for the confusion. This is because in Figure 6, the learning rate is fixed to be 3.0e-4 and the weight decay is 0.1. While this setting is often used in LLM training, but it is sub-optimal for vision tasks. In Table 1, we do a grid search over a much wider range.

At last, many thanks for your time and effort. Please let us know if you have further questions.

Many thanks for the rebuttal. First, I respectfully disagree with your claim that "while the vanilla Adam is not guaranteed to converge, the references indicate that Adam indeed can converge though with very strong assumptions". To my understanding, [1] relies only on standard assumptions and establishes an O(ϵ−4)O(\epsilon^{-4}) complexity for vanilla Adam.

Second, I appreciate your efforts to correct the proofs. However, the revised proofs add 6 pages to the appendix, which seems to be a major revision instead of a minor fix. Given the extent of these changes, I may not be able to verify the revised proof within a short period.

Therefore I maintain my rating.

[1] Haochuan Li, Alexander Rakhlin, and Ali Jadbabaie. "Convergence of adam under relaxed assumptions." Advances in Neural Information Processing Systems 36 (2023): 52166-52196.

Thanks for your reply. There may be a misunderstanding. We are sorry for the phrase strong assumptions which is not accurate. What we mean is the convegence analysis of Adam is still heavily restricted. As Theorem 4.1 of [1] indicates, the authors choose β≤min⁡(1,c1λϵ2σ2Gι)\beta \leq \min (1, \frac{c _ 1 \lambda \epsilon ^ 2}{\sigma ^ 2 G \sqrt{\iota}}) and η≤c2min⁡(rλG,σλβLGι,λ3/2βLG)\eta \leq c _ 2 \min(\frac{r \lambda}{G}, \frac{\sigma \lambda \beta}{L G \sqrt{\iota}}, \frac{\lambda ^ {3/2} \beta}{L \sqrt{G}} ) to attain the final converge result. Since ϵ=10−8\epsilon=10 ^ {-8} or even smaller, they are actually unrealstic settings. And it's impossible to examine the validity of the commonly used learning rate, for example, η=3.0e−4\eta=3.0e-4.

And the corrected proofs contain two Theorems, so the revision seems to be a little bit lengthy. The ctritical revision lies in Step 4 (Line 972-989) while other steps generally follow the previous version. We have also carefully checked them with the assistance of GPT-Astra.

Reviewer_SYHmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Lova, a single-state adaptive optimizer that constructs its update denominator from the current momentum and gradient residual, eliminating the historical second-moment buffer. The authors present theoretical results and report improvements across image classification, language modeling, diffusion models, and reinforcement learning.

Critical strengths
  1. Simple design with low optimizer-state memory. Lova stores only one momentum buffer while retaining coordinate-wise magnitude adaptation. Its denominator also bounds coordinate-wise update magnitudes, making the method practically appealing.
  2. Evaluation across several tasks. Results on classification, language modeling, generation, and reinforcement learning support its practical potential. Table 4 reports improved diffusion-model FID, while Figure 1 reports memory and throughput benefits.
Critical weaknesses
  1. The proposed mechanism is not sufficiently established. Gradient residuals contain both sampling noise and momentum tracking error. The increasing residual ratio in Figure 1 therefore does not directly establish greater noise dominance. Likewise, the vanishing aggregate inner product in Figure 2 does not establish a coordinate-wise second-moment approximation. The experiments do not isolate whether the gains come from noise adaptation, momentum tracking, or behavior closer to sign-based updates. Theorem 4.3 also uses time-varying momentum and a specific learning-rate schedule, unlike the fixed momentum and cosine schedule used experimentally.
  2. The experiments do not sufficiently isolate the contribution of the new design. The ImageNet experiments use a global batch size of 20,480 without scaling the learning rate. The language-model experiments use the same learning rate and weight decay across optimizers, but do not adequately describe tuning ranges or update-scale matching. Baseline configurations may therefore contribute to the reported gaps.
Questions for authors
  1. How do you distinguish sampling noise from momentum tracking error in the gradient residual? What evidence attributes the gains primarily to noise adaptation?
  2. What hyperparameter ranges were searched for the baselines, and how were Muon’s update scales matched? Do the gains hold under comparable tuning budgets?
  3. Can you provide matched AdamS pretraining results and ablations separating momentum timing from residual-based scaling?
1 comment

Many thanks for the valuable comments, and they are addressed one by one as follows.

(Q1): ImageNet experiments use a global batch size of 20,480 without scaling the learning rate.

(R1): In the revision (Table 1), we have included the results using a root square learning rate scaling, see Table below:

AdamWMARSAdamSSIGNUMAdaBeliefLova
ResNet-1867.16 ±\pm 0.1466.69 ±\pm 0.2267.34$\pm$ 0.3067.65 ±\pm 0.1167.08 ±\pm 0.1468.69 ±\pm 0.05
ResNet-5073.79 ±\pm 0.1873.13 ±\pm 0.4174.56 ±\pm 0.2874.09 ±\pm 0.3573.97 ±\pm 0.1475.32$\pm$ 0.17

We can observe that Lova performs even comparable to using small batch sizes (approximately at 76% for batch size=256).

(Q2): LLM experiments do not describe tuning ranges or update-scale matching.

(R2): In the revision (Figure 4), we have included the tuned results and the search range is described in Appendix A.4. The superoirty of Lova is retained, particularly for over-training.

(Q3): How do you distinguish sampling noise from momentum tracking error in the gradient residual?

(R3): The residual gt−mt\textbf{g} _ t - \textbf{m} _ t is actually the sum of the sampling noise gt−∇f(θt)\textbf{g} _ t - \nabla f(\boldsymbol{\theta} _ t) and the momentum tracking error ∇f(θt)−mt\nabla f(\boldsymbol{\theta} _ t) - \textbf{m} _ t. However, as shown in Figure 7 (Appendix A.1), for most coordinates the sampling noise dominates and the tracking error is negligible. This implies that mt\textbf{m} _ t estimates the expected gradient ∇f(θt)\nabla f(\boldsymbol{\theta} _ t) in the long run and that we refer to gt−mt\textbf{g} _ t - \textbf{m} _ t as the local variance is approximately correct. To avoid ambiguity, we added a footnote to local variance.

(Q4): What evidence attributes the gains primarily to noise adaptation?

(R4): Sign-based optimizers such as SIGNUM and Lion by themselves are very effcient optimizers. But, they often diverge or explode and therfore are not quite widely used in practical training. Lova introduces (gt−mt)2(\textbf{g} _ t - \textbf{m} _ t) ^ 2 in the denominator to stablize the training process. To show this noise adaptation is essential, we trained a ResNext29-32x4d on CIFAR-100, ranging the learning rate from 3.0e-4 to 5.0e-3 while using a large weight decay λ=(0.1,0.5,1.0)\lambda = (0.1, 0.5, 1.0).

Lova achieves the best test accuracy of 78.22%, while SIGNUM only attains an accuracy of 77.73%. Furthermore, Figure 9 (Appendix A.3) demonstrates that the performance of SIGNUM quickly deteriorates when the learning rate is relatively large. This is strikingly different from Lova. Therefore, the noise adaptation is very important for stable training and better generalization.

(Q5): What hyperparameter ranges were searched for the baselines, and how were Muon’s update scales matched? Do the gains hold under comparable tuning budgets?

(R5): Sorry for the missing details. In the revision, we have included all training details in Appendix A.4. Particularly, since AdamW has the property of keeping its RMS updates around 0.2-0.4 in the practice of LLM training, we match Muon’s update to a fixed value of 0.2, as suggested by (Semenov et al., 2025). Moreover, since we only need to tune one hyperparamter β2\beta _ 2 of Lova, it is quite effcient to do grid search. Actually, we often use β2=0.99\beta _ 2 = 0.99 by default for most experiments. Therefore, the performance gains generally hold under comparable tuning budgets because we only need to adjust the learning rate and weight decay as AdamW.

Semenov et al., Benchmarking Optimizers for Large Language Model Pretraining, arxiv, 2025.

(Q6): Can you provide matched AdamS pretraining results and ablations separating momentum timing from residual-based scaling?

(R6): In the revision, we have included AdamS in LLM pretraining and DiT. The results suggest that AdamS performs better than AdamW, but still performs worse than Lova. For example, on DiT models, FID of AdamS are 52.57, 32.11 while FID of Lova are 49.84, 29.54.

To ablate momentum timing, we consider a variant, Lova$^-$, defined as

mt←β1mt−1+(1−β1)gt\boldsymbol{m} _ t \leftarrow \beta _ 1 \boldsymbol{m} _ {t-1} + (1 - \beta _ 1) \boldsymbol{g} _ t

vt2←β2mt−12+(1−β2)(gt−mt−1)2\boldsymbol{v} _ t ^ 2 \leftarrow \beta _ 2 \boldsymbol{m} _ {t-1} ^ 2 + (1 - \beta _ 2) (\boldsymbol{g} _ t - \boldsymbol{m} _ {t-1}) ^ 2

We trained them on CIFAR-100 using ResNext29-32x4d. The following accuracy results suggest that there is no significant difference. This is because both mt\boldsymbol{m} _ t and mt−1\boldsymbol{m} _ {t-1} converges to the expected gradient in the long run.

β2\beta _ 20.90.950.990.999
Lova77.36 ±\pm 0.3476.95 ±\pm 0.4276.87 ±\pm 0.1976.96 ±\pm 0.06
Lova −^-77.48 ±\pm 0.0477.01 ±\pm 0.3577.02 ±\pm 0.2376.88$\pm$ 0.11

At last, many thanks for your time and effort. Please let us know if you have further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.