acceptodds
Under review as a conference paper at ICLR 2027

Momentum in large-batch training: Polyak enlarges the critical batch size, Nesterov improves data efficiency

Abstract

We study when and how momentum improves large-batch training in the one-pass regime, using power-law kernel regression as a tractable setting. We first characterize risk stability through the critical learning rate, defined as the largest learning rate for stable training, and obtain , , and , where is the batch size, is the momentum factor, and is the capacity exponent. Within this admissible region, we derive scaling laws for the full risk dynamics, capturing the progression from an early transient, through power-law decay, to a noise floor. We then minimize the final-step risk over the admissible learning rates and momentum factors under a fixed data budget, yielding a three-regime batch-size phase diagram that reveals how the role of momentum changes with batch size. Notably, Polyak enlarges the critical batch size, the largest batch size preserving the best small-batch data-scaling exponent, thereby enabling greater parallelism without sacrificing data efficiency. In contrast, Nesterov achieves better data efficiency in the large-batch regime because its look-ahead mechanism suppresses noise accumulation. Numerical experiments validate the predicted stability boundaries, risk dynamics, and batch-size phase diagram.

Primary area:
learning theory
Keywords:
Momentum methods, Large-batch training, Scaling laws, Risk stability, Stochastic Optimization.

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_4cqWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how Polyak and Nesterov momentum affect large-batch training in the one-pass regime. Using power-law kernel (PLK) regression, the authors derive three layers of results: critical learning rates, risk-dynamics scaling laws, and a three-regime batch-size phase diagram of the optimal data-scaling exponent. The core insight is a clean separation of mechanisms: Polyak only enlarges the admissible region without changing the signal–noise tradeoff itself, so it enlarges the critical batch size but not the optimal exponent. Nesterov's look-ahead induces curvature-adaptive damping, lowering the noise floor from σ2ηβ/B\sigma^2\eta_\beta/B to σ2ηβ1/q/B\sigma^2\eta_\beta^{1/q}/B, so it strictly improves data efficiency at large batch. PLK simulations quantitatively validate all claims. ImageNet/CIFAR-10 qualitatively reproduce the phase structure.

But still, there are some concerns and thus I lean to weak acceptance for the current manuscript. I'd like to raise it to clear acceptance as long as the questions can be answered well.

Critical strengths
  1. Clear mechanism separation. The paper gives a precise answer to when and why momentum helps at large batch, organized in a clean hierarchy: stability -> risk dynamics -> optimal data scaling.
  2. The Nesterov analysis is genuinely new. The BqB^q stability scaling comes from a power-law spectral sum ∑iηλi1−β+ηλi∼(η/(1−β))1/q\sum_i \frac{\eta\lambda_i}{1-\beta+\eta\lambda_i} \sim (\eta/(1-\beta))^{1/q}, and the ηβ1/q\eta_\beta^{1/q} noise suppression is essentially an effective spectral truncation. The mechanism story and the algebra reinforce each other.
  3. Thorough PLK experiments. Five (s,q)(s,q) settings, multi-run averaging, and fitting only constants of the predicted functional forms — the experiments test the theory, not the tuning ability.
  4. Well written, easy to follow.
Critical weaknesses
  1. Theory is confined to case study. The PLK setting assumes a quadratic (least-squares kernel) objective, Gaussian features. In fact, the PLK setting has carried strong assumptions including convexity, smoothness, kernel spectrum and data distribution. There is no theoretical discussion of transfer to modern deep learning (feature learning, non-Gaussian features, classification losses) or even the general kernel methods (general spectral decay rate).

  2. Deep-learning experiments are weak. ImageNet uses a single architecture (ResNet-18 + GroupNorm), 3 epochs, and a single seed run without no error bars, which is relatively weak than PLK experiments.

  3. The LLM motivation is not closed. This theoretical research is motivated by modern LLM training, but I cannot find the related experimental results, which may weaken the theoretical implications. Personally, a small-scale test (e.g., small language model ranging from 100M to 2B is also okay) would substantially strengthen the paper.

  4. (Minor point) No discussion related to the optimizer, like Adam and Muon, which are more frequently used today. But this point does not hurt the contributions of this paper.

Questions for authors
  1. How many random seeds were used for the ImageNet results in Fig. 9? If one, can you provide multi-seed error bars for at least a subset of batch sizes?

  2. Have you tried different setting for ImageNet experiments (e.g., more epochs, LayerNorm, ResNet34)?

  3. Can the similar mechanism be observed in the language model (training a small model is also acceptable)?

  4. Can you give a discussion how the PLK setting (in what mild conditions) be extended into deep learning, and how the current results related to Adam and Muon optimizers?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 4cqW

We sincerely thank the reviewer for suggesting additional seed and language-model experiments and asking how PLK relates to deep learning and modern optimizers. Below are our responses to the questions raised by the reviewer.

[W2, Q1, Q2] Could you clarify the number of seeds used for Fig. 9, provide multi-seed error bars, and test other ImageNet settings?

The original experiments in Fig. 9 use a single random seed. We have added runs with two additional random seeds, and the main batch-size dependence remain consistent; see Global Response (c).

Due to time constraints, we evaluate ViT-Small only on CIFAR-10. As summarized in Global Response (e), these experiments show batch-size dependence similar to that in Fig. 1b.

[W3, Q3] Given the paper's motivation from language-model training, can the proposed mechanism also be observed in this setting?

Thank you for this helpful suggestion. Due to the limited rebuttal time and computational resources, we are unable to provide language-model results at this stage. A systematic comparison requires tuning learning rate and momentum separately at each batch size (a three-dimensional hyperparameter search). We will include these experiments in a subsequent version of the paper.

Although we do not present language-model experiments here, previous work on language models provides qualitative experimental support for our findings on batch-size dependence. In [1] (Section 4.4, Fig. 4(b)), momentum and Nesterov extend the useful batch-size range for Transformer training on LM1B. In [2] (Section 2.2, Fig. 1), momentum improves robustness to hyperparameter choices at large batch sizes.

We have also strengthened our neural-network evaluation with ViT-Small on CIFAR-10. These experiments test the robustness of the qualitative trends across architectures.

[W1, Q4(1)] Can you discuss how the PLK setting be extended into deep learning?

  • Power-law kernel regression (PLK). Our goal is to understand how momentum affects stability and data efficiency as batch size increases. We adopt PLK as a tractable phenomenological model, following a line of theoretical studies of scaling laws [3–8]. Its power-law spectrum spans a broad range of learning timescales, allowing us to isolate how momentum changes signal learning and noise accumulation for directions with different sharpness (e.g. overdamped and underdamped directions). This structure makes it possible to derive explicit predictions for stability, loss dynamics, and batch-size-dependent data efficiency.

  • Empirical evidence beyond the PLK setting. Previous work shows that FSLs derived from PLK can capture language-model loss trajectories across learning-rate schedules and batch sizes [6][8]. These results provide empirical evidence that PLK can reveal training mechanisms and predict macroscopic behavior beyond the idealized kernel setting. Our ImageNet experiments and the additional ViT-Small experiments provide analogous evidence for momentum: the relative benefits of the optimizers change with batch size in ways qualitatively consistent with our predictions.

[W4, Q4(2)] How do the results relate to Adam and Muon, and what would be needed to extend the analysis to these optimizers?

Our analysis provides a starting point for understanding the momentum component of these optimizers. Extending our results to Adam and Muon requires combining momentum with coordinatewise second-moment normalization and matrix orthogonalization, respectively. In the SignSGD FSL analysis [8], sign normalization induces effective preconditioning under diagonal-feature and noise–curvature alignment assumptions. A joint analysis must additionally account for Adam's evolving second-moment estimator and Muon's matrix coupling to characterize their loss dynamics and stability. We leave this to future work.

References

[1] Measuring the Effects of Data Parallelism on Neural Network Training. JMLR 2019.

[2] Is your batch size the problem? Revisiting the Adam-SGD gap in language modeling. arXiv preprint, 2025.

[3] 4+3 phases of compute-optimal neural scaling laws. NeurIPS 2024.

[4] Scaling laws in linear regression: Compute, parameters, and data. NeurIPS 2024.

[5] How Does Critical Batch Size Scale in Pre-training? ICLR 2025.

[6] Functional Scaling Laws in Kernel Regression: Loss Dynamics and Learning Rate Schedules. NeurIPS 2025.

[7] Fast Catch-Up, Late Switching: Optimal Batch Size Scheduling via Functional Scaling Laws. ICLR 2026.

[8] When and How SignSGD Outperforms SGD? A Functional Scaling Law Perspective. NeurIPS 2026.

[9] A Defense of the Quadratic Model. arXiv preprint, 2026.

[10] Predicting Large Model Test Losses with a Noisy Quadratic System. ICML 2026.

Reviewer_4cqWOpenReview ↗

Response to the rebuttal

Thank you for your detailed response. My concerns were partially resolved at this stage. But the remaining issues (e.g., language-model evidence) may not affect the positive evaluation. I'd like to keep my score at 3, because there is no "3.5" this year. I'm looking forward to the subsequent work, e.g., related theoretical analysis about Adam and some second-order methods.

Thank you very much for your kind follow-up. We are pleased to hear that you found our rebuttal helpful. We sincerely appreciate your encouraging feedback and your thoughtful evaluation of our work.

Reviewer_jMtXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors study the theoretical impact of two different types of momentum, Polyak and Nesterov, on the large-batch training performance using Power-Law Kernel (PLK) regression. The results can be summarized as 'Nesterov enables SGD to use large batch sizes without sacrificing training stability'. Throughout extensive theoretical analyses, the authors result in deriving scaling laws for the full risk dynamics and three-regime batch-size phase diagram. The numerical results include PLK regression experiments with varying q and s settings in addition to ImageNet-1K benchmark results.

Critical strengths
  1. The paper thoroughly analyze the impact of two representative momentum techniques on the stability of large-batch training. While this problem setup is not that special, this study provides new and insightful results regarding the critical learning rate for stable training as well as batch size scalability.

  2. The numerical results include not only PLK regression results but also large-scale benchmark (ImageNet-1K) results, making the theoretical results more convincing.

  3. The paper clearly explains the difference from concurrent work (Morwani et al., 2026). Particularly, this work provides a complete phase diagram, showing the large-batch regime in where Nesterov achieves lower validation loss within the same training budget than Polyak.

Critical weaknesses

Overall, I appreciate thorough theoretical analyses. I learned a lot from the results. However, I have some critical concerns as follows.

  1. Figure 1 appears on the very first page while it is referenced on the third page. It is a common convention that figures come after paragraphs where they are referenced. It also hurts readability because readers have to go back to the first page while reading the third page. I recommend relocating the figure near to the end of Section 1.

  2. The noise floors in Theorem 5.2 appear to be based on the same information as the critical learning rates in Theorem 4.3. By setting ηβ=η/(1−β)\eta_\beta = \eta/(1-\beta), Eq. (10) becomes ηβ≲B\eta_\beta \lesssim B and Eq. (11) becomes ηβ≲Bq\eta_\beta \lesssim B^q. The noise floors can then be written as σ2(ηβ/Bq)1/q\sigma^2 (\eta_\beta / B^q)^{1/q}, so each of them becomes \simga2\simga^2 exactly at the corresponding critical learning rate. Could authors clarify whether the results in Section 4 and 5 are independently derived or follow from the same underlying lemma? This determines how much Section 5 adds beyond Section 4.

  3. In Section 6, the authors should explain how three batch sizes, b1b_1, b2b_2, and b3b_3, are defined. It seems like they are defined based on the noise floors, but the authors should have explained it so that Section 6 is self-contained.

  4. Appendix~E.1.1 shows the risk stability results with varying (s,q)(s,q) settings, especially with q∈{1.5,2,2.5}q \in \{1.5, 2, 2.5\}. However, the risk-dynamics experiment shown in Figure 2 uses (s,q)=(0.3,4)(s,q) = (0.3, 4), a value of qq outside that range and the largest one appearing in this paper. Since the gap between the Polyak and Nesterov noise floors increases with qq, the largest value is used precisely where the advantage of Nesterov is better displayed. Could the authors report the risk dynamics for the values of qq already used in Figure 4?

  5. The numerical results shown in Section 7.2. look odd. First, even without momentum, it is strange that the SGD's validation loss starts to increase between B=10B=10 and B=100B=100. It is widely known that the batch size of 128128 gives good results and does not exhibit a noticeable performance drop against the batch size of 3232. I guess the training is not performed for a sufficiently large number of iterations. Appendix also does not explain how long the training was conducted. Could authors provide more information regarding the hyperparameter settings as well as the total number of iterations? Second, the loss values look too high. In general, the convergence loss is known to be around 1, rather than 3 or 4. Which loss function was used? How was the learning rate decayed? The authors should provide detailed experimental settings.

  6. The authors used ResNet18 and ImageNet-1K only in the main text. Appendix shows CIFAR-10 results additionally, however the model is still the same ResNet18. Can we expect the same or similar results when using different (especially with a larger model) models?

Questions for authors

My questions and concerns are all included in the above weakness part. Please carefully address them.

2 comments
AuthorsOpenReview ↗

Response to Reviewer jMtX

We sincerely thank the reviewer for the questions about stability, risk dynamics, and batch-size transitions, and for the suggestions on experiments and presentation. Below are our responses to the questions and suggestions raised by the reviewer.

[W1] Could Figure 1 be moved closer to its first reference?

Thank you for this suggestion. We have moved the first reference to Figure 1 earlier in the Introduction.

[W2] How does Section 5's noise-floor analysis relate to Section 4's stability results?

We first clarify the stability condition and the noise floor. We use Nesterov as an example, and the same reasoning applies to Polyak. The bound ηβ≲Bq\eta_\beta\lesssim B^q captures only part of the stability condition. It is important to retain the order-one ceiling η≲1\eta\lesssim1 in Eqs. (10)–(11). Writing ηβ=η/(1−β)\eta_\beta=\eta/(1-\beta), we obtain ηβcrit≂min⁡{(1−β)−1,Bq}\eta_{\beta}^{\mathrm{crit}}\eqsim\min\left\lbrace(1-\beta)^{-1},B^q\right\rbrace.

The bound η≲1\eta\lesssim1 is important because, in the optimal large-batch regime of Section 6, Nesterov’s optimal parameters (η,β)(\eta,\beta) satisfy η≂1\eta\eqsim1 and Bq(1−β)≫1B^q(1-\beta)\gg1. Its noise floor therefore scales as σ2/[Bq(1−β)]1/q≪σ2\sigma^2/[B^q(1-\beta)]^{1/q}\ll\sigma^2.

We then clarify the relationship between Sections 4 and 5. Section 5 builds directly on Section 4. The stability conditions in Section 4 determine the admissible range of hyperparameters, and Section 5 derives the scaling laws within that range. Only when the learning rate is below the stability boundary, can we combine the bound on multiplicative noise with estimates of signal learning and label-noise accumulation to derive the scaling law in Section 5.

[W3] How are b1,b2,b3b_1,b_2,b_3 defined, and how do they relate to the noise floors?

Equation (17) defines b1=s/(s+1)b_1=s/(s+1), b2=(2s+1/q)/(2s+1+1/q)b_2=(2s+1/q)/(2s+1+1/q), and b3=(1+b1)/2b_3=(1+b_1)/2 for batch sizes of order DbD^b. Although their derivation involves balancing signal and noise terms, their definitions depend only on ss and qq, independent of the noise. Remark 1.1 and the “Mechanism” paragraph of Section 6 explain their roles. Briefly, SGD maintains the optimal per-sample learning rate η/B\eta/B by increasing the learning rate proportionally with batch size, until it reaches the stability ceiling at b1b_1. Polyak maintains the optimal η/[B(1−β)]\eta/[B(1-\beta)] by increasing momentum until transient decay becomes limited by the training horizon at b3b_3. Nesterov's lower noise term improves the data-scaling exponent and it peaks at b2b_2.

[W4] Does Nesterov's advantage persist at q=1.5,2,2.5q=1.5,2,2.5?

As summarized in Global Response (a), we add loss-dynamics experiments at q=1.5,2,2.5q=1.5,2,2.5 and stability experiments covering q=4q=4. The late-training advantage of Nesterov over Polyak persists at all three smaller qq values.

[W5] Why does SGD's loss rise at moderate batch sizes, and why are losses high? Please clarify the training settings.

  • Clarification of the few-pass training setting. Our study is motivated by large-scale model training, where one or a few passes over the data are common. Our theory analyzes one-pass training, and therefore our ImageNet experiments focus on the few-pass regime (3 epochs). This differs from classical multi-pass training (∼\sim 100 epochs). In the few-pass regime, the optimization process is important, and typically Ltrain≈LtestL_{train} \approx L_{test}. In the multi-pass regime, the generalization error dominates, and usually Ltrain≪LtestL_{train} \ll L_{test}.

    Few-pass training has two consequences. First, the batch size cannot be too large, since that would leave too few optimizer updates. Second, the final loss tends to be higher. These two points help explain why our results differ from the reviewer’s expectations, which we suspect are based on multi-pass training [1][2][3][4].

    Another reason for the higher final loss may be that we use a constant learning rate, consistent with our theoretical analysis. In practice, the training typically uses learning-rate decay.

  • Detailed experimental settings. We train ResNet-18 on ImageNet-1K using cross-entropy loss and a constant learning rate. We fix the sample budget at three epochs, giving N=3D/BN=3D/B updates at batch size BB. For each optimizer and batch size, we report the lowest final validation cross-entropy loss among stable configurations in the learning rate and momentum grids.

[W6] Do the trends persist with different architectures or larger models?

We add ViT-Small experiments on CIFAR-10 (Global Response (e)).

References

[1] Blurs Behave Like Ensembles: Spatial Smoothings to Improve Accuracy, Uncertainty, and Robustness. ICML 2022.

[2] Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors. ICML 2020.

[3] Why Does Adaptive Batching Help LLM Pretraining? A Perspective from Unbounded Variance. arXiv preprint, 2026.

[4] Training Independent Subnetworks for Robust Prediction. ICLR 2021.

Please engage with the authors during the discussion period.

Reviewer_Xj4NOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how Polyak and Nesterov momentum affect large-batch training under a fixed data budget. Using power-law kernel (PLK) regression as a tractable theoretical setting, the authors derive scaling laws for critical learning rates, risk dynamics, and optimal data-scaling exponents as the batch size increases. The analysis identifies three batch-size regimes and predicts that Polyak momentum extends the range of batch sizes over which SGD-like data efficiency can be maintained, whereas Nesterov momentum achieves better data efficiency in the large-batch regime by reducing noise accumulation through its look-ahead mechanism. The theoretical results are supported by numerical experiments on synthetic PLK regression, which examine stability boundaries, transient dynamics, and batch-size-dependent scaling behavior. Additional experiments using ResNet-18 on CIFAR-10 and ImageNet-1K provide qualitative evidence that the predicted trends also appear in neural-network training. The paper's main contribution is a unified theoretical characterization of the relationship between momentum, learning-rate stability, batch size, and data efficiency.

Critical strengths
  1. A coherent theoretical characterization of momentum across batch-size regimes. The paper connects critical learning-rate scaling, transient optimization dynamics, stochastic noise accumulation, and final risk under a fixed data budget. Rather than studying learning rate, momentum, and batch size independently, it derives a unified phase diagram that explains when Polyak and Nesterov momentum provide different benefits. This is a meaningful conceptual contribution to understanding large-batch optimization.
  2. Complementary theoretical and empirical validation. The authors validate their predictions through multiple experiments on synthetic PLK regression, including stability-boundary estimates, risk-dynamics comparisons, and data-scaling experiments under different parameter settings. Additional CIFAR-10 and ImageNet-1K experiments provide an initial connection between the theoretical analysis and neural-network training. The extensive appendices also improve the transparency of the mathematical analysis and experimental procedures.
Critical weaknesses
  1. The practical validation is limited relative to the breadth of the conclusions. The ImageNet-1K experiments use ResNet-18 with a fixed sample budget equivalent to only three training epochs, a constant learning rate, and no warmup, weight decay, or label smoothing. The appendix states that all ImageNet runs use the same random seed. Furthermore, the reported result for each method and batch size is the lowest validation loss among the evaluated stable hyperparameter configurations. If the same validation set is used for both hyperparameter selection and final reporting, this procedure may introduce selection bias. Consequently, the experiments do not establish whether the observed advantages are robust across independent runs, longer training horizons, larger architectures, or more representative modern training configurations. This limitation is particularly relevant when interpreting the results as evidence for improved data efficiency in large-scale language-model pretraining.
  2. The connection between the idealized theoretical setting and practical training remains incomplete. The main theoretical results rely on the assumptions of power-law kernel regression, Gaussian features, and specified power-law spectral and source conditions. The resulting scaling laws are asymptotic and are generally stated up to constant factors. Although the neural-network experiments exhibit qualitatively consistent trends, they do not establish that the precise theoretical scaling exponents or stability boundaries transfer to practical deep-learning models. In addition, the theoretical definition of risk stability concerns bounded expected risk over an infinite training horizon, whereas the CIFAR-10 experiments assess stability over 2,000 updates using a finite-horizon heuristic. The distinction between these two notions should be made clearer to establish how strongly the empirical results validate the formal stability analysis.
Questions for authors

Question 1 — Clarification of the ImageNet evaluation protocol. The ImageNet experiments report the best validation loss obtained from a hyperparameter grid, while the appendix states that all runs use the same random seed. Is the full validation set used both for hyperparameter selection and for reporting the final results? If so, how do the authors account for potential selection bias? More importantly, how robust are the relative rankings of SGD, Polyak, and Nesterov across independent random seeds? These details would materially affect the strength of the empirical evidence supporting the claimed data-efficiency advantage. Question 2 — Relationship between formal and empirical stability. The theoretical analysis defines stability in terms of bounded expected risk over an infinite horizon, whereas the CIFAR-10 experiments use a finite horizon of 2,000 updates and a heuristic loss-based criterion. How sensitive are the estimated critical learning-rate boundaries and their fitted scaling relationships to the choice of training horizon and stability criterion? Clarifying this relationship would help determine how directly the neural-network experiments validate the theoretical stability results.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Xj4N

We sincerely thank the reviewer for raising concerns about selection bias, robustness across seeds, and the distinction between theoretical and empirical stability. Below are our responses to the questions raised by the reviewer.

[W1, Q1] Does using the same validation set for tuning and reporting introduce selection bias, and are the optimizer rankings robust across seeds and training settings?

In the original experiments, we use the full 50,000-image ImageNet-1K validation set both to select hyperparameters and to report loss. To address potential selection bias, we repeat the comparison with separate validation sets for hyperparameter selection and loss reporting (Global Response (b)). We also repeat the ImageNet experiments with two additional random seeds (Global Response (c)). The main batch-size dependence remains consistent across the separate validation subsets and additional random seeds.

[W2, Q2] How does empirical stability relate to theoretical risk stability, and how sensitive are the estimated boundaries and fitted scalings to the training horizon and stability criterion?

The stability boundaries play different roles in theory and in experiments. In the theoretical analysis of PLK, an algorithm is risk stable if its expected population risk remains uniformly bounded over all iterations for every admissible initialization with finite population risk. To validate our findings in experiments, we use an empirical proxy. In the original setup, we train for 2,000 updates and use a loss-based criterion to determine whether each run is stable (Appendix E.3.2).

In the additional experiments, we first shorten the training horizon from 2,000 to 1,000 updates and repeat the stability check using the original criterion. Only 30 out of 2,598 runs (1.2%) change stability classification (see Global Response (d)). At 2,000 updates, we also test a stricter criterion (125 of 2,598 runs changed, 4.8%) and a looser criterion (36 of 2,598 runs changed, 1.4%). Across all these configurations, the stability boundaries show the same qualitative dependence on batch size and momentum damping.

Thank you for the detailed response. I appreciate the authors' efforts to address the review comments. These revisions clarify some of my concerns and improve the completeness of the paper. However, considering the paper's contributions, experimental quality, and remaining limitations, I decide to maintain my original rating.

We sincerely appreciate your thoughtful follow-up. Thank you again for the time and care you devoted to reviewing our manuscript and considering our rebuttal.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

Dear Area Chair and reviewers,

We sincerely thank the Area Chair and all reviewers for their time and constructive feedback. Below, we summarize our main findings and describe the experiments we add to the paper's appendix in response to your comments.

Summary

We study how Polyak and Nesterov momentum affect data efficiency as batch size increases. In one-pass power-law kernel (PLK) regression, we derive critical learning-rate bounds and risk-dynamics scaling laws, then use them to characterize optimal data scaling across batch-size regimes. Polyak retains SGD's small-batch data-scaling rate over a wider range of batch sizes. Nesterov's look-ahead reduces noise accumulation and improves the optimal data-scaling exponent in the large-batch regime. The synthetic experiments test the quantitative predictions, and the CIFAR-10 and ImageNet-1K experiments examine the corresponding qualitative trends.

Additional experiments

We add five sets of experiments to the appendix. They cover (a) PLK stability and loss dynamics, (b) ImageNet-1K evaluation using separate validation subsets for hyperparameter selection and loss reporting, (c) ImageNet-1K runs with additional random seeds, (d) the sensitivity of CIFAR-10 stability boundaries to the training horizon and stability criterion, and (e) CIFAR-10 experiments with a different architecture, ViT-Small.

(a) PLK stability and loss dynamics

We add loss-dynamics experiments at q=1.5,2,2.5q=1.5,2,2.5 (Appendix E.1.2), complementing the original stability experiments at the same values of qq, and we also add batch-size stability-boundary experiments at q=3,4,5q=3,4,5 (Appendix E.1.1), complementing the original loss-dynamics experiment at q=4q=4.

(b) Evaluation with separate validation subsets

To address potential selection bias, we repeat the ImageNet comparison in Section 7.2 with the ImageNet-1K validation set split into two disjoint halves of 25,000 images each (Appendix E.2.3). We select the optimal learning rate and momentum parameters on one half and report the selected configuration's loss on the other.

The split-validation comparison preserves the main batch-size trends: Polyak maintains low loss over a wider range of batch sizes than SGD, while Nesterov achieves the lowest loss in the large-batch regime.

(c) Additional random seeds

We supplement the ImageNet experiments in Section 7.2 with results of two additional random seeds, confirming the robustness of our findings (Appendix E.2.2).

(d) Sensitivity of stability boundaries

To assess the sensitivity of the CIFAR-10 stability boundaries, we repeat the analysis with different training horizons and stability criteria (Appendix E.3.2). The momentum stability boundaries show the same trend across all settings.

(e) Different architecture

We add CIFAR-10 experiments using ViT-Small (Appendix E.3.1). These results provide evidence that the batch-size dependence in Fig. 1b persists with a different architecture.

We have updated the manuscript to include the additional experiments described above. We thank the Area Chair and all reviewers again for their time and constructive feedback.

Best regards,

Authors of Submission 31636

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.