- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes Lova (Local Variance Adaptation), a memory-efficient adaptive optimizer motivated by what the authors call "local variance dominance" in Adam. Lova maintains a first-moment buffer and constructs a memoryless denominator from the squared momentum and squared gradient–momentum residual:
v_t^2 = \beta_2 m_t^2 + (1-\beta_2)(g_t-m_t)^2.$$ This eliminates the persistent second-moment buffer while retaining coordinate-wise adaptive scaling. The authors present asymptotic orthogonality results for the momentum and residual, an $O(\sqrt{T})$ convex regret bound for a maximum-tracking variant, and an $O(\log T/T^{1/4})$ nonconvex stationarity bound under time-varying momentum and learning-rate schedules. The reported nonconvex bound avoids polynomial dependence on $1/\epsilon$. Experiments cover image classification, learning with noisy labels, language-model pretraining, GLUE fine-tuning, diffusion-transformer training, and Atari reinforcement learning. The paper reports improvements over the evaluated baselines, together with reduced memory use and increased throughput in the tested language-model pretraining setting.Critical strengths
1. A simple, memory-efficient update with a useful structural guarantee.
Lova requires only one persistent momentum buffer and avoids maintaining a historical second moment. Its denominator satisfies , which directly implies
\leq \frac{1}{\sqrt{\beta_2}}.$$ Thus, the coordinate-wise update is bounded without relying on the numerical stabilizer to control its magnitude. This is an appealing combination of implementation simplicity, reduced optimizer-state storage, and controlled updates, independently of the stronger mechanistic claims (Algorithm 1 and Section 3.2). **2. Broad empirical evaluation with encouraging results.** The evaluation extends beyond a single architecture or task. The classification and GLUE tables report improvements across multiple settings, and the diffusion experiments show substantial reductions in FID for both evaluated model sizes (Tables 1, 3, and 4). The classification experiments use three random seeds, while the reinforcement-learning results include confidence intervals over ten trials (Section 5.1 and Figure 5). This breadth provides meaningful evidence that the update is worth investigating, although the comparison protocol limits the strength of the superiority claim.Critical weaknesses
1. The main comparisons do not establish an advantage over appropriately tuned baselines.
The ImageNet experiments use a global batch size of 20,480, and the caption of Table 1 explicitly states that no learning-rate scaling is applied to illustrate AdamW's deterioration in large-batch training. This evaluates robustness to a particular unchanged configuration, but does not establish superiority over a baseline configured for that batch size. Increasing batch size at a fixed epoch budget also reduces the number of parameter updates, so the experiment does not isolate the proposed variance-related explanation.
Similarly, the language-model experiments fix the peak learning rate and weight decay across optimizers without reporting an optimizer-specific search (Section 5.2). Identical nominal hyperparameters are not necessarily inappropriate, but the paper needs to establish that they produce competitive configurations for each method. Otherwise, the observed differences may reflect update scaling or hyperparameter sensitivity rather than the proposed normalization mechanism. AdamS is included in the classification and fine-tuning tables but not in the language-model pretraining comparison; including this comparator would also strengthen the evidence.
2. The theoretical justification of the central mechanism is not established as written.
In the proof of Theorem 3.4, page 14, the authors write
and consequently identify with a weighted sum of stochastic noise terms. However, the historical iterates are random, so the unconditional expectation should instead be
The displayed decomposition therefore does not hold as stated. This is not merely a naming issue: the later argument uses the properties of a deterministic expectation to reduce a cross term to a negative variance (page 16). The proof of Theorem 3.6 explicitly reuses this argument and inherits the gap.
There is also a distinction between the stated orthogonality result and the interpretation used to motivate the optimizer. A vanishing expected global inner product does not establish a coordinate-wise, instantaneous decomposition of squared gradients: cross terms can cancel across coordinates or in expectation. Moreover, includes momentum tracking error as well as stochastic noise. Consequently, the diagnostics do not by themselves establish that the observed residual dominance is specifically noise dominance (Sections 3.1 and 3.2).
Questions for authors
1. Clarification of tuning and available comparison evidence.
What learning-rate, weight-decay, momentum, and normalization settings were considered for each optimizer, and how were the final configurations selected? In particular, do the main ImageNet and language-model advantages persist with comparable optimizer-specific tuning budgets? Please provide any available evidence and clarify the omission of AdamS from the pretraining comparison.
2. Clarification and correction of the orthogonality proof.
How should the expectation decomposition on page 14 be interpreted given that the historical iterates are random? Can the subsequent cross-term argument be corrected while retaining the assumptions and rates of Theorems 3.4 and 3.6? A corrected derivation would materially affect my assessment.
3. Clarification of the mechanism and its practical scope.
How is the ratio in Figure 1(a) aggregated, and what evidence distinguishes stochastic noise from momentum lag or changing true gradients? Please also clarify the scope of the orthogonality claim for the practical optimizer: Algorithm 1 uses constant momentum, whereas the orthogonality theorems require . Are the Figure 2 diagnostics obtained with constant or time-varying momentum, and what conclusion do they support about coordinate-wise residual scaling?
Many thanks for the valuable comments, and they are addressed one by one as follows.
(Q1): ImageNet experiments use a global batch size of 20,480 and no learning-rate scaling is applied.
(R1): In the revision (Table 1), we have included the results using a root square learning rate scaling, see Table below:
We can observe that Lova performs even comparable to using small batch sizes (approximately at 76% for batch size=256).
(Q2): Does language-model advantages persist with comparable optimizer-specific tuning budgets? Please provide any available evidence and clarify the omission of AdamS from the pretraining comparison.
(R2): Sorry for the missing details. In the revision, we have included all training details in Appendix A.4. In particular, we have added AdamS and Lion in LLM training (see Figure 4). We can observe that the superoirty of Lova is retained, particularly for over-training. Moreover, Lion and AdamS also perform better than AdamW on this small-scale model. But the gap between Lion and AdamW quickly diminishes as more tokens are progressed.
(Q3): How should the expectation decomposition on page 14 be interpreted given that the historical iterates are random? Can the subsequent cross-term argument be corrected while retaining the assumptions and rates of Theorems 3.4 and 3.6? A corrected derivation would materially affect my assessment.
(R3): Thanks for pointing out this flaw. Fortunately, this issue can be easily fixed under same assumptions and the optimal rate is still 1/2. The ctritical step is to bound the gradient–noise covariance. Please see Appendix A.5 for the corrected proofs of Theorems 3.4 and 3.6.
(Q4): How is the ratio in Figure 1(a) aggregated, and what evidence distinguishes stochastic noise from momentum lag or changing true gradients?
(R4): The ratio in Figure 1(a) actually is the ratio between ∥gt−mt∥2 and ∥mt∥2 and is calculated at the end of each epoch and then averaged over all epochs.
To distinguish stochastic noise from momentum lag, note that the residual gt−mt is actually the sum of the sampling noise gt−∇f(θt) and the momentum tracking error ∇f(θt)−mt. However, as shown in Figure 7 (Appendix A.1), for most coordinates the sampling noise dominates and the tracking error is negligible. This implies that mt estimates the expected gradient ∇f(θt) in the long run and that we refer to gt−mt as the local variance is approximately correct. To avoid ambiguity, we added a footnote to local variance.
(Q5): Please also clarify the scope of the orthogonality claim for the practical optimizer: Algorithm 1 uses constant momentum, whereas the orthogonality theorems require β1,t→1. Are the Figure 2 diagnostics obtained with constant or time-varying momentum, and what conclusion do they support about coordinate-wise residual scaling?
(R5): We want to clarify that the Figure 2 diagnostics are obtained with constant momentum. But when a varying β1,t is used, we also can observe the same trend. To verify this, we trained a ResNet-56 on CIFAR-10 for 200 epochs using AdamW and Lova. The initial learning rate is 3.0e-4 and decayed according to the cosine scheduler. In particular, we set β1,t=1−1/t while fixing β2. Figure 8 (see Appendix A.2) suggests that the inner-product indeed stays around zero and Lova decays in a faster rate. Moreover, Figure 7 (see Appendix A.1) also suggests for most coordinates the sampling noise dominates and the tracking error is negligible. Therefore, the decomposition gt2≈mt−12+(gt−mt−1)2 is practically plausible.
At last, many thanks for your time and effort. Please let us know if you have further questions.