- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how Polyak and Nesterov momentum affect large-batch training in the one-pass regime. Using power-law kernel (PLK) regression, the authors derive three layers of results: critical learning rates, risk-dynamics scaling laws, and a three-regime batch-size phase diagram of the optimal data-scaling exponent. The core insight is a clean separation of mechanisms: Polyak only enlarges the admissible region without changing the signal–noise tradeoff itself, so it enlarges the critical batch size but not the optimal exponent. Nesterov's look-ahead induces curvature-adaptive damping, lowering the noise floor from to , so it strictly improves data efficiency at large batch. PLK simulations quantitatively validate all claims. ImageNet/CIFAR-10 qualitatively reproduce the phase structure.
But still, there are some concerns and thus I lean to weak acceptance for the current manuscript. I'd like to raise it to clear acceptance as long as the questions can be answered well.
Critical strengths
- Clear mechanism separation. The paper gives a precise answer to when and why momentum helps at large batch, organized in a clean hierarchy: stability -> risk dynamics -> optimal data scaling.
- The Nesterov analysis is genuinely new. The stability scaling comes from a power-law spectral sum , and the noise suppression is essentially an effective spectral truncation. The mechanism story and the algebra reinforce each other.
- Thorough PLK experiments. Five settings, multi-run averaging, and fitting only constants of the predicted functional forms — the experiments test the theory, not the tuning ability.
- Well written, easy to follow.
Critical weaknesses
-
Theory is confined to case study. The PLK setting assumes a quadratic (least-squares kernel) objective, Gaussian features. In fact, the PLK setting has carried strong assumptions including convexity, smoothness, kernel spectrum and data distribution. There is no theoretical discussion of transfer to modern deep learning (feature learning, non-Gaussian features, classification losses) or even the general kernel methods (general spectral decay rate).
-
Deep-learning experiments are weak. ImageNet uses a single architecture (ResNet-18 + GroupNorm), 3 epochs, and a single seed run without no error bars, which is relatively weak than PLK experiments.
-
The LLM motivation is not closed. This theoretical research is motivated by modern LLM training, but I cannot find the related experimental results, which may weaken the theoretical implications. Personally, a small-scale test (e.g., small language model ranging from 100M to 2B is also okay) would substantially strengthen the paper.
-
(Minor point) No discussion related to the optimizer, like Adam and Muon, which are more frequently used today. But this point does not hurt the contributions of this paper.
Questions for authors
-
How many random seeds were used for the ImageNet results in Fig. 9? If one, can you provide multi-seed error bars for at least a subset of batch sizes?
-
Have you tried different setting for ImageNet experiments (e.g., more epochs, LayerNorm, ResNet34)?
-
Can the similar mechanism be observed in the language model (training a small model is also acceptable)?
-
Can you give a discussion how the PLK setting (in what mild conditions) be extended into deep learning, and how the current results related to Adam and Muon optimizers?
3 comments
Response to the rebuttal
Thank you for your detailed response. My concerns were partially resolved at this stage. But the remaining issues (e.g., language-model evidence) may not affect the positive evaluation. I'd like to keep my score at 3, because there is no "3.5" this year. I'm looking forward to the subsequent work, e.g., related theoretical analysis about Adam and some second-order methods.
Thank you very much for your kind follow-up. We are pleased to hear that you found our rebuttal helpful. We sincerely appreciate your encouraging feedback and your thoughtful evaluation of our work.
Response to Reviewer 4cqW
We sincerely thank the reviewer for suggesting additional seed and language-model experiments and asking how PLK relates to deep learning and modern optimizers. Below are our responses to the questions raised by the reviewer.
The original experiments in Fig. 9 use a single random seed. We have added runs with two additional random seeds, and the main batch-size dependence remain consistent; see Global Response (c).
Due to time constraints, we evaluate ViT-Small only on CIFAR-10. As summarized in Global Response (e), these experiments show batch-size dependence similar to that in Fig. 1b.
Thank you for this helpful suggestion. Due to the limited rebuttal time and computational resources, we are unable to provide language-model results at this stage. A systematic comparison requires tuning learning rate and momentum separately at each batch size (a three-dimensional hyperparameter search). We will include these experiments in a subsequent version of the paper.
Although we do not present language-model experiments here, previous work on language models provides qualitative experimental support for our findings on batch-size dependence. In [1] (Section 4.4, Fig. 4(b)), momentum and Nesterov extend the useful batch-size range for Transformer training on LM1B. In [2] (Section 2.2, Fig. 1), momentum improves robustness to hyperparameter choices at large batch sizes.
We have also strengthened our neural-network evaluation with ViT-Small on CIFAR-10. These experiments test the robustness of the qualitative trends across architectures.
Power-law kernel regression (PLK). Our goal is to understand how momentum affects stability and data efficiency as batch size increases. We adopt PLK as a tractable phenomenological model, following a line of theoretical studies of scaling laws [3–8]. Its power-law spectrum spans a broad range of learning timescales, allowing us to isolate how momentum changes signal learning and noise accumulation for directions with different sharpness (e.g. overdamped and underdamped directions). This structure makes it possible to derive explicit predictions for stability, loss dynamics, and batch-size-dependent data efficiency.
Empirical evidence beyond the PLK setting. Previous work shows that FSLs derived from PLK can capture language-model loss trajectories across learning-rate schedules and batch sizes [6][8]. These results provide empirical evidence that PLK can reveal training mechanisms and predict macroscopic behavior beyond the idealized kernel setting. Our ImageNet experiments and the additional ViT-Small experiments provide analogous evidence for momentum: the relative benefits of the optimizers change with batch size in ways qualitatively consistent with our predictions.
Our analysis provides a starting point for understanding the momentum component of these optimizers. Extending our results to Adam and Muon requires combining momentum with coordinatewise second-moment normalization and matrix orthogonalization, respectively. In the SignSGD FSL analysis [8], sign normalization induces effective preconditioning under diagonal-feature and noise–curvature alignment assumptions. A joint analysis must additionally account for Adam's evolving second-moment estimator and Muon's matrix coupling to characterize their loss dynamics and stability. We leave this to future work.
References
[1] Measuring the Effects of Data Parallelism on Neural Network Training. JMLR 2019.
[2] Is your batch size the problem? Revisiting the Adam-SGD gap in language modeling. arXiv preprint, 2025.
[3] 4+3 phases of compute-optimal neural scaling laws. NeurIPS 2024.
[4] Scaling laws in linear regression: Compute, parameters, and data. NeurIPS 2024.
[5] How Does Critical Batch Size Scale in Pre-training? ICLR 2025.
[6] Functional Scaling Laws in Kernel Regression: Loss Dynamics and Learning Rate Schedules. NeurIPS 2025.
[7] Fast Catch-Up, Late Switching: Optimal Batch Size Scheduling via Functional Scaling Laws. ICLR 2026.
[8] When and How SignSGD Outperforms SGD? A Functional Scaling Law Perspective. NeurIPS 2026.
[9] A Defense of the Quadratic Model. arXiv preprint, 2026.
[10] Predicting Large Model Test Losses with a Noisy Quadratic System. ICML 2026.