acceptodds
Under review as a conference paper at ICLR 2027

Momentum for Variational Learning at Scale

Abstract

Momentum is well-studied for deep learning, but the same cannot be said for variational learning at scale. Here, we revisit the current practices of momentum for variational learning and propose a new momentum where an exponential moving-average of the site functions is maintained. We show that this site-momentum not only generalizes several existing (non-Bayesian) momentum approaches for deep learning (such as MARS and SGDHess) but also speeds up variational learning at scale. For instance, for training Llama 3 (1.3B) on 100B tokens from FineWeb, site-momentum yields a lower perplexity of up to 0.76 throughout compared to the state-of-the-art method. Overall, our work provides new theoretical connections as well as practical improvements for variational learning with momentum.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
variational learning, momentum, stochastic variance reduced gradient

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_KLWUOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds on variational learning of deep neural networks (IVON and EVON) by introducing a momentum in the gradient descent algorithm. The paper motivates from a theoretical gap in current variational learning algorithms, where their momentum is not really on the natural gradients. It then proposes a site-momentum modification to this variational gradient descent algorithm. It claims it is a first approach at trying variational learning on LLMs (e.g. 1.3B model) and empirically finds that the paper’s method has more stable training and lower perplexity than other variational learning methods.

Critical strengths

The paper’s approach is promising and theoretically motivated, from the point of momentum on natural gradients from existing variational learning methods. It connects well with existing literature, and experimental results suggest the paper’s method to be promising.

It also claims to be a first attempt at trying the variational learning line of work at this scale. An initial look at literature by me supports this claim. Meanwhile, I note that other works have attempted at Bayesian posterior approximation training of such scale as well, often on LoRA or just a few layers.

Critical weaknesses

The extent of empirical evaluation is a main weakness. First, confidence intervals are not shown in results. Second, the way experimental results are shown limits the implication of these empirical results. The paper is entirely situated on the variational learning line of work, which compared with default training or SFT methods, targets a number of properties offered by a probabilistic framework while maintaining / slightly improving standard perplexity / cross-entropy loss. These properties can include calibration, robustness, sensitivity to data, and generalization. These analyses are missing.

Third, the extent of evaluation is two models within one family (Llama 3) and two datasets. While two models x 2 data is not that few on its own, the lack of confidence intervals and analyses beyond the perplexity metric adds up, making the current results not adequate enough to judge the impact of the proposed method.

The empirical analyses can tie more to how the addition from the paper’s proposed method helps achieve the improvement in training stability and perplexity.

Finally, writing and presentation can be improved. For example, Line 375: “More results and ablations are in App. F.” A summary for why these experiments and the main takeaway in the main text would be good.

Overall, I consider the paper promising. The empirical analysis and presentation weaknesses make it slightly fall short of acceptance score in its current form. (Updated to weak accept after rebuttal period)

Questions for authors

My main questions would be about the "Weakness" section:

Would the method affect other properties that typically motivate probabilistic training, such as calibration, robustness, sensitivity to data, and generalization?

1 comment
AuthorsOpenReview ↗

Thanks for your review!

We thank the reviewer for their feedback. The issues raised are regarding the lack of non-variational baselines, as well as missing analysis for properties of the posterior distribution. We have provided new results to address these. There is a minor concern regarding confidence intervals, and lack of diverse models, but we do not believe this is a problem. We believe we have adequately addressed the concerns and kindly ask the reviewer to reconsider their score.

The way experimental results are shown limits the implication of these empirical results.….The paper is entirely situated on the variational learning line of work, which compared with default training or SFT methods, targets a number of properties offered by a probabilistic framework while maintaining / slightly improving standard perplexity / cross-entropy loss.

We have now added non-variational baselines in Fig. 3 which also confirm the competitiveness of our approach. We see that adding MARS to both Adam and SOAP still is much worse than adding VRM to EVON. At convergence, EVON with VRM is still better than any other method.

We also want to stress that our paper focuses on pretraining, not on SFT which is mentioned by the reviewer. We hope that there is no misunderstanding.

These properties [of variational methods] can include calibration, robustness, sensitivity to data, and generalization. These analyses are missing. Would the method affect other properties that typically motivate probabilistic training, such as calibration, robustness, sensitivity to data, and generalization?

In Fig. 4 (b), we now show that the posterior found by VRM indeed has similar properties to those found with EVON. The result shows that when sampling from the posterior the accuracy increases for VRM in the same way as EVON, but now the perplexity is also much better. We expect other metrics to be the same and will include the full results for Minut et al. (2026) in the final version of the paper.

Additionally, we also have results for zero-shot evaluation on a variety of datasets for both EVON and VRM on a variety of datasets which shows that VRM slightly improves over EVON. This result further shows that the improvements to perplexity also manifest in the improvements to downstream performance.

MethodMMLUARC-CARC-EHellaSwagOpenBookQAPIQASciQWinoGrandeAverage
EVON0.24730.39590.67970.59260.39800.71870.86200.59590.5613
VRM0.23890.39930.68180.60420.40000.72740.87500.58250.5636

confidence intervals are not shown in results. The extent of evaluation is two models within one family (Llama 3) and two datasets. While two models x 2 data is not that few on its own, the lack of confidence intervals and analyses beyond the perplexity metric adds up, making the current results not adequate enough to judge the impact of the proposed method.

We do not agree with the statement that the “current results are not adequate”. Reviewer LcZE specifically commented that “the scale of the experiment is impressive”. Showing such improvements at such a scale is rare in the Bayesian literature. Note that the result is not on just “two models within one family”. We have many such results in the App. G.2. We have improved the writing of Sec. 4 to avoid this confusion and clearly mention the nanoGPT results and the other variants of Llama3 we considered. Finally, the lack of confidence intervals is standard practice in LLM works due to the high cost of repeating experiments, e.g., see Yuan et al. (2025); Semenov et al. (2025), and we follow this standard practice, as well.

The empirical analyses can tie more to how the addition from the paper’s proposed method helps achieve the improvement in training stability and perplexity. Finally, writing and presentation can be improved. For example, Line 375: “More results and ablations are in App. F.” A summary for why these experiments and the main takeaway in the main text would be good.

We thank the reviewer for these suggestions. We have improved the writing of the Experiments section (Sec. 4) to highlight the reasons behind the improvements in training stability and perplexity. Intuitively, the reasons are the same as those for variance reduction as in SGDHess and MARS, which are both special cases of SOM and VRM, respectively. We also provide a new explanation for the better performance of VRM compared to SOM in Sec. 4. We have also reorganized Sec. 4 to better explain the summary of the experiments and the main takeaways.

Reviewer_6KGeOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces site-momentum as an alternative to Adam-like momentum for variational inference. Unlike standard momentum, which accumulates gradients of the full ELBO (log joint + entropy), site-momentum uses only the site function λ⊤T(θ)\lambda^\top T(\theta) from the Bayesian learning rule, which performs inference with an exponential family h(θ)exp⁡(λ⊤T(θ))h(\theta)\exp(\lambda^\top T(\theta)). The site-momentum construction connects several types of momentum and yields second-order and variance-reduced versions of existing momentum algorithms. The connections are derived analytically, and the new variants are shown to improve perplexity and reduce gradient-norm spikes when pretraining 1B- and 1.3B-parameter Llama 3 models on FineWeb (100B tokens), at no additional runtime cost.

Critical strengths

The paper claims it can connect six different types of variational learning through site-momentum. The connection are explicitly derived by introducing the appropriate variational family, setting a learning rates or gradient evaluation sites for the proposed site-momentum. The topic of momentum is underexplored in variational inference and the paper fills this gap with a unifying construction that opens two variants: second-order momentum and variance-reduction. The experimental results demonstrates site-momentum is not just a theoretical construction, with the second-order momentum and variance-reduction site-momentum achieves better perplexity on 1B and 1.3B Llama 3 than EVON and IVON. This is why this paper should be accepted.

Critical weaknesses

None

Questions for authors
  • Why does SOM only give marginal improvements?
  • Was MARS with γ=0.25\gamma=0.25 tested? If so, what were the results?
3 comments
AuthorsOpenReview ↗

Thanks for your review!

We thank the reviewer for the encouraging and constructive feedback. We are happy that you found both the theoretical results and connections, as well as the practical results to be useful. You raised two issues regarding marginal improvements with SOM and ablations of the γ\gamma hyperparameter. We have addressed both below.

Why does SOM only give marginal improvements?

SOM gives marginal improvements due to the (block-diagonal) Hessian approximations used in IVON and EVON. We have added a detailed explanation for this in the paper (see line 478). We can see this via a simple example of linear regression. See Eq. 72 in App. G.2. There, both SOM and VRM are equivalent when full-Hessian is used. For this case, both speed up the convergence (see an example in Fig. 8 in the revised version of our paper) but IVON with diagonal approximations gives marginal improvements. This shows that marginal improvements are most likely a result of the Hessian approximation. In general, having a good estimate of the full Hessian is a long-standing problem, while computing differences in gradients is much cheaper and more effective in practice. This is why VRM is much more effective in practice.

Q2: Was MARS with gamma=0.25 tested? If so, what were the results?

We have included these results in Fig. 4 (a) now with a full ablation for the hyperparameter γ\gamma. For Adam and SOAP a small value of γ=0.1\gamma=0.1 is the best.

Reviewer_6KGeOpenReview ↗

Response to Authors’ Rebuttal

I appreciate the authors investigating the marginal improvement and attributing it to the Hessian approximation. I have no further concerns about this paper and will keep my score as it stands.

Thanks a lot for acknowledging and also many thanks for your support!

Reviewer_LcZEOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a new momentum method for Variational Learning, while also revisiting how momentum is currently implemented in the state of the art. The authors point out that the existing BLR momentum accumulates the gradient of the full objective, including the entropy term, while in practice IVON and EVON just use an Adam-style gradient EMA.

The introduced method, called site-based momentum, is argued by the authors to be more appropriate in a variatioanl setting, and, instead of accumulating an EMA on the real loss gradients, it is done on a loss surrogate, so the entropy never enters the momentum calculation. They propose two practical variants, SOM (second-order momentum) and VRM (variance-reduced momentum), with VRM showing the most consistent improvements. The authors establish theoretical connections to the literature: STORM and MARS; and showcase their methods experimentally in LLM experiments where they achieve consistent perplexity improvements over both IVON and EVON baselines.

I note that this paper is outside my expertise, particularly regarding the mathematical details of variational inference momentum methods. My assessment is based primarily on the clarity of the exposition and the experimental results.

Critical strengths
  1. The scale of the experiments is impressive, especially given the variational setting.
  2. Compared to state of the art, having VRM not needing an overhead computation is a big win.
  3. Identifying the theoretical inconsistency in how IVON and EVON implement momentum is a valuable contribution, and the authors do a good job framing the theoretical foundations as well as connecting to previous methods.
  4. The site momentum idea is an interesting approach to the problem, and it seems to hold experimentally.
Critical weaknesses
  1. In practice the method mostly comes down to adding the MARS correction term to IVON/EVON.
  • The practical VRM drops the Hessian momentum and the curvature terms, due to memory constraints at billion-parameter scale.
  • SOM, the second-order variant of site-momentum, only improves perplexity by less than 0.05, suggesting the Hessian correction contributes little in practice. While the memory constraints justify these simplifications, they do undermine part of the theoretical contribution.
  1. The experimentation is not finished by the time of the deadline. It is expected to be finished by the camera-ready version, but it makes the claims somewhat less conclusive.
  2. The experiments are limited to LLMs, so it remains unclear how general the experimental findings are in other domains, such as imaging.
Questions for authors
  1. How does VRM compare against standard non-variational optimizers such as SOAP and SOAP+MARS with the same token budget and similar tuning effort? The paper benchmarks exclusively against variational baselines (IVON and EVON), making it difficult to assess the overall competitiveness of the approach. (I am aware the comparison may be so off that could result pointless, but nevertheless could be interesting to see)

  2. The γ\gamma parameter requires different values at different scales (0.0250.025 for small models, 0.250.25 for large ones). Is there a principled way to set or schedule γ\gamma automatically, rather than relying on empirical tuning? Or is this just a hyperparameter that depends on the problem at hand?

  3. The experiments are limited to language model pretraining. Do the authors expect similar gains in other domains such as computer vision, and are there any preliminary results?

1 comment
AuthorsOpenReview ↗

Thank you for your review!

We thank the reviewer for their constructive feedback. We are happy to see that you find our framing of theoretical foundations valuable. Two issues have been raised regarding the reasons behind the marginal improvements obtained with SOM and regarding missing comparisons to non-Bayesian baselines and other ablations. We have addressed both issues below.

SOM, the second-order variant of site-momentum, only improves perplexity by less than 0.05, suggesting the Hessian correction contributes little in practice. While the memory constraints justify these simplifications, they do undermine part of the theoretical contribution.

SOM gives marginal improvements due to the (block-diagonal) Hessian approximations used in IVON and EVON. We have added a detailed explanation for this in the paper (see line 478). We can see this via a simple example of linear regression. See Eq. 72 in App. G.2. There, both SOM and VRM are equivalent when full-Hessian is used. For this case, both speed up the convergence (see an example in Fig. 8 in the revised version of our paper) but IVON with diagonal approximations gives marginal improvements. This shows that marginal improvements are most likely a result of the Hessian approximation. In general, having a good estimate of the full Hessian is a long-standing problem, while computing differences in gradients is much cheaper and more effective in practice. This is why VRM is much more effective in practice.

The experimentation is not finished by the time of the deadline. It is expected to be finished by the camera-ready version, but it makes the claims somewhat less conclusive.

We have now included the new version of the only unfinished experiment in Fig. 1b. We see that the claim still holds as strongly as it was in the original submission: VRM consistently performs better than EVON, even at convergence. Please note that the perplexity numbers are updated in this figure as well due to an issue with the rescaling in the code by Minut et al. (2026) which we used in our experiments. This does not change the result. Just that, with this rescaling, the increase in perplexity is slightly higher (up to 0.96).

How does VRM compare against standard non-variational optimizers such as SOAP and SOAP+MARS with the same token budget and similar tuning effort? The paper benchmarks exclusively against variational baselines (IVON and EVON), making it difficult to assess the overall competitiveness of the approach. (I am aware the comparison may be so off that could result pointless, but nevertheless could be interesting to see)

Thanks for the suggestion. We have now added non-variational baselines in Fig. 3 to confirm the competitiveness of our approach. We see that adding MARS to both Adam and SOAP still is much worse than adding VRM to EVON. At convergence, too, EVON with VRM is still better than any other method.

The parameter requires different values at different scales ( for small models, for large ones). Is there a principled way to set or schedule automatically, rather than relying on empirical tuning? Or is this just a hyperparameter that depends on the problem at hand?

We have now included a full ablation for the hyperparameter γ\gamma in Fig. 4 (a). We see that γ=0.25\gamma=0.25 is the best choice for EVON, while for Adam and SOAP a small value of 0.1 is the best. We stress that we do not rely on empirical tuning bur rather use these fixed values for all of our experiments. This is written in line 475. We are happy to report that it also turns out to be the best choice.

The experiments are limited to language model pretraining. Do the authors expect similar gains in other domains such as computer vision, and are there any preliminary results?

Yes, we do, although we have not yet tried this because, similarly to LLMs, we expect the significant improvements to appear at a much larger scale. We did not have enough compute during the rebuttal period to do this but we will consider adding such a result in the next version of the paper.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response

We thank all reviewers for their thorough reading and constructive feedback. Below is the list of main concerns raised along with our attempt to address it.

  1. Lack of non-variational baselines: We have added an experiment comparing to SOAP+MARS and Adam+MARS where our method beats both of them.
  2. Marginal improvements with SOM: We show that this is due to approximate Hessians and highlight this with a new experiment on linear regression.
  3. Lack of ablations for γ\gamma: We have added an extensive ablation to justify the hyperparameter choices we used.
  4. Some minor comments about writing, presentation, and lack of intuitive explanation and confidence intervals: We have improved the writing and provided some explanations for these issues.

Many thanks for raising the score and acknowledging the rebuttal. We agree that CI is absolutely useful for performance evaluation! We have launched an experiment now to get CI for some of our experiment, at least for the ones where we can afford to do it. It is not our intention to be dismissive of CI but we rather wanted to point that they are often absent in papers due to computational reasons. We will try our best to include them in our paper, as per your suggestion.

Quick update the reviewer might be interested in:

We obtained some intermediate results for Fig. 3 on the pre-training of Llama3-350M on 30B tokens of C4. Over 3 trials, we observed the following mean and standard error for the final perplexity:

  • EVON: 16.6977 ±\pm 0.0077
  • VRM:  16.3919 ±\pm 0.0114

The standard error is below 4% of the perplexity reduction of 0.3058 points.

Thank you again for your support.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.