- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper builds on variational learning of deep neural networks (IVON and EVON) by introducing a momentum in the gradient descent algorithm. The paper motivates from a theoretical gap in current variational learning algorithms, where their momentum is not really on the natural gradients. It then proposes a site-momentum modification to this variational gradient descent algorithm. It claims it is a first approach at trying variational learning on LLMs (e.g. 1.3B model) and empirically finds that the paper’s method has more stable training and lower perplexity than other variational learning methods.
Critical strengths
The paper’s approach is promising and theoretically motivated, from the point of momentum on natural gradients from existing variational learning methods. It connects well with existing literature, and experimental results suggest the paper’s method to be promising.
It also claims to be a first attempt at trying the variational learning line of work at this scale. An initial look at literature by me supports this claim. Meanwhile, I note that other works have attempted at Bayesian posterior approximation training of such scale as well, often on LoRA or just a few layers.
Critical weaknesses
The extent of empirical evaluation is a main weakness. First, confidence intervals are not shown in results. Second, the way experimental results are shown limits the implication of these empirical results. The paper is entirely situated on the variational learning line of work, which compared with default training or SFT methods, targets a number of properties offered by a probabilistic framework while maintaining / slightly improving standard perplexity / cross-entropy loss. These properties can include calibration, robustness, sensitivity to data, and generalization. These analyses are missing.
Third, the extent of evaluation is two models within one family (Llama 3) and two datasets. While two models x 2 data is not that few on its own, the lack of confidence intervals and analyses beyond the perplexity metric adds up, making the current results not adequate enough to judge the impact of the proposed method.
The empirical analyses can tie more to how the addition from the paper’s proposed method helps achieve the improvement in training stability and perplexity.
Finally, writing and presentation can be improved. For example, Line 375: “More results and ablations are in App. F.” A summary for why these experiments and the main takeaway in the main text would be good.
Overall, I consider the paper promising. The empirical analysis and presentation weaknesses make it slightly fall short of acceptance score in its current form. (Updated to weak accept after rebuttal period)
Questions for authors
My main questions would be about the "Weakness" section:
Would the method affect other properties that typically motivate probabilistic training, such as calibration, robustness, sensitivity to data, and generalization?
Thanks for your review!
We thank the reviewer for their feedback. The issues raised are regarding the lack of non-variational baselines, as well as missing analysis for properties of the posterior distribution. We have provided new results to address these. There is a minor concern regarding confidence intervals, and lack of diverse models, but we do not believe this is a problem. We believe we have adequately addressed the concerns and kindly ask the reviewer to reconsider their score.
We have now added non-variational baselines in Fig. 3 which also confirm the competitiveness of our approach. We see that adding MARS to both Adam and SOAP still is much worse than adding VRM to EVON. At convergence, EVON with VRM is still better than any other method.
We also want to stress that our paper focuses on pretraining, not on SFT which is mentioned by the reviewer. We hope that there is no misunderstanding.
In Fig. 4 (b), we now show that the posterior found by VRM indeed has similar properties to those found with EVON. The result shows that when sampling from the posterior the accuracy increases for VRM in the same way as EVON, but now the perplexity is also much better. We expect other metrics to be the same and will include the full results for Minut et al. (2026) in the final version of the paper.
Additionally, we also have results for zero-shot evaluation on a variety of datasets for both EVON and VRM on a variety of datasets which shows that VRM slightly improves over EVON. This result further shows that the improvements to perplexity also manifest in the improvements to downstream performance.
We do not agree with the statement that the “current results are not adequate”. Reviewer LcZE specifically commented that “the scale of the experiment is impressive”. Showing such improvements at such a scale is rare in the Bayesian literature. Note that the result is not on just “two models within one family”. We have many such results in the App. G.2. We have improved the writing of Sec. 4 to avoid this confusion and clearly mention the nanoGPT results and the other variants of Llama3 we considered. Finally, the lack of confidence intervals is standard practice in LLM works due to the high cost of repeating experiments, e.g., see Yuan et al. (2025); Semenov et al. (2025), and we follow this standard practice, as well.
We thank the reviewer for these suggestions. We have improved the writing of the Experiments section (Sec. 4) to highlight the reasons behind the improvements in training stability and perplexity. Intuitively, the reasons are the same as those for variance reduction as in SGDHess and MARS, which are both special cases of SOM and VRM, respectively. We also provide a new explanation for the better performance of VRM compared to SOM in Sec. 4. We have also reorganized Sec. 4 to better explain the summary of the experiments and the main takeaways.