- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies how batch size affects hyperparameter scaling and optimizer rankings. It derives two scaling prescriptions for Muon, evaluates their transfer across training regimes, and reports optimizer ranking reversals after per-batch hyperparameter tuning. It further investigates these observations through noisy quadratic models (NQMs) and directional interventions in language-model pretraining.
Critical strengths
-
The paper derives Muon scaling prescriptions through two approaches: matching an SDE approximation and minimizing a convergence upper bound. These complementary perspectives are theoretically useful.
-
The experiments show that the tested hyperparameter scaling prescriptions do not transfer consistently across different training regimes. This highlights an important limitation of applying a single scaling recipe broadly.
-
The paper demonstrates that batch size can change optimizer rankings despite substantial per-batch retuning. This observation is relevant to optimizer evaluation and practical training.
-
The directional intervention shows that retaining small-batch updates in a small subspace of sharp directions can substantially reduce the large-batch branch penalty. This provides an interesting direction for improving large-batch optimization.
Critical weaknesses
-
The empirical novelty should be positioned more precisely. Prior work has already demonstrated batch-dependent optimizer advantages [1–3] and compared modern LLM optimizer rankings across batch sizes [4]. The independently tuned crossover among preconditioned optimizers is useful, but the main text should explain more explicitly what additional understanding it provides.
-
The upper-bound-based Muon scaling derivation needs a more self-contained explanation in the main text. Although Appendix B.3 provides technical details, the assumptions, the Frank–Wolfe gap being bounded, and the quantities held fixed when varying batch size deserve clearer presentation. The connection between minimizing this bound and selecting a learning rate for terminal training performance also needs further explanation.
-
The NQM analysis does not fully account for the main empirical findings. It explains square-root-to-linear learning-rate scaling, but the formal crossover result compares SGD and Newton rather than KL-SOAP and Shampoo. Appendix D provides useful matrix-model experiments, but a closer qualitative agreement with the principal pretraining crossover would strengthen the explanation.
-
Coordinate descent provides substantial tuning effort but does not establish globally optimal hyperparameters. Interactions among learning rate, momentum, and weight decay may leave some optimizers at less favorable configurations. This does not invalidate the observed crossover, but its robustness to the tuning procedure deserves further evidence, and empirical claims should distinguish the best configurations found from true optimizer-family optima.
-
Section 6 provides evidence that retaining small-batch updates in high-curvature directions reduces the branch penalty, but its connection to the proposed CNR explanation remains unclear. CNR depends on both directional curvature and gradient-noise variance, whereas the experiment selects directions using curvature alone and does not measure their noise variances. The comparison with a random subspace therefore shows that the selected directions matter, but does not establish whether CNR explains their importance beyond curvature alone. Measuring directional noise and comparing curvature-based versus CNR-based direction selection would strengthen this connection.
-
The practical implications remain somewhat unclear. The headline crossover occurs at the largest tested batch size, where validation loss has already deteriorated relative to smaller-batch runs. Large batches may compensate through higher throughput, but this trade-off is not quantified. It is therefore unclear when practitioners should change optimizers, reduce batch size, or adopt a different scaling or update strategy.
Questions for authors
-
Could the authors state more explicitly in the main text the incremental contributions and their significance relative to prior work on batch-dependent optimizer performance?
-
Could the authors provide more intuition for the upper-bound-based scaling derivation, including why the chosen bound is informative for this purpose and when its minimizer should be expected to track a useful learning rate?
-
Can the NQM framework be extended to analyze a batch-dependent crossover involving Shampoo and KL-SOAP, or Muon, rather than only SGD and Newton? A result that qualitatively matches the pretraining observations would strengthen the theoretical connection.
-
Could the authors better justify the coordinate-descent tuning procedure or validate its robustness against a broader joint hyperparameter grid on a smaller pretraining setup, such as a 30M model? I recognize that exhaustive tuning at the main experimental scale would be expensive; even a restricted joint grid at smaller scale could help assess hyperparameter interactions.
-
Could the authors measure gradient-noise variance along the selected directions and assess whether CNR explains the benefit of retaining small-batch updates beyond curvature alone? For example, comparing curvature-based and CNR-based subspace selection would help clarify this connection.
-
Could the authors summarize the practical guidance supported by their findings? In particular, when should practitioners favor large-batch training, and what changes to optimizer design or scaling prescriptions are suggested by the results?
Overall, I like this paper and currently lean toward weak acceptance. The paper combines useful empirical observations with theoretical analyses and an interesting directional intervention. Although the novelty positioning, the connection between theory and practice, and the robustness of the tuning-based comparisons could be strengthened, I find the overall investigation valuable. I would welcome further discussion with the authors and will reconsider my rating based on their response and our subsequent discussion.
References:
[1] Zhang et al. “Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model.” NeurIPS 2019.
[2] Marek et al. “Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wasteful.” NeurIPS 2025.
[3] Srećković et al. “Is your batch size the problem? Revisiting the Adam-SGD gap in language modeling.” arXiv:2506.12543, 2025.
[4] Semenov et al. “Benchmarking Optimizers for Large Language Model Pretraining.” arXiv:2509.01440, 2025.
4 comments
Q1 / W1. Positioning.
Prior work shows batch-dependent advantages mainly between Adam and SGD (Zhang et al., 2019; Marek et al., 2025; Srećković et al., 2025), and benchmarks modern optimizers at several batch sizes with hyperparameters transferred from one batch size (Semenov et al., 2025). We add four things. First, we retune five optimizers independently at every batch size, and the ranking among the three preconditioned methods changes with batch size under two weight-norm controls. Second, we derive two scaling rules for Muon, test whether each transfers, and also sweep a family of candidate scaling rules for Muon to see how they perform in different settings. Third, we analyze both the optimizer crossover and the batch-size scaling rules from a curvature-to-noise perspective in noisy quadratic models. Fourth, we show that a few sharp, high-SNR directions carry a disproportionate share of the local large-batch penalty. We will make these points explicit in the paper.
Q2. Intuition for the bound-based rule.
We bound the expected Frank–Wolfe gap of the iterates, a measure of stationarity over the spectral-norm ball: , where is the effective step size and is the token budget, with momentum and weight decay held fixed (Section 3, Candidate 2). The first term spreads the initial objective gap over the updates, and the second is the smoothness cost of the step, so the minimizing step size grows as . As in prior bound-minimization work (Shulgin et al., 2026; Bu et al., 2026; Schaipp et al., 2025), we take the step size that minimizes the convergence bound as a proxy for the step size that minimizes the loss; the proxy is useful when these two terms capture how the loss depends on the step size and the batch size.
Q6 / W6. Practical guidance and throughput.
The main practical guidance of our work is that comparison between optimizers should be performed at the target batch size and results from one batch size may not generalize to other batch sizes. Large batches matter for throughput. In large-scale pretraining, many GPUs are available, and the batch size is usually set as large as data parallelism across them allows. The optimizer therefore has to work well at that batch size, which is the regime our comparison targets. The throughput gain depends on the hardware and the implementation, so we give one data point: on a single 8×H100 node at the 3× budget, increasing the batch from 512K to 2M tokens shortens Shampoo's training by 2.3× (Fig. R3). This measurement fixes the number of devices. At scale, a larger batch also lets more devices run in parallel.
In addition, when the batch size changes, retuning at least the learning rate and weight decay is advisable, since none of the scaling rules we tested transferred reliably in every setting (Fig. 2).
Please engage with the authors during the discussion period.
Thank you for the response. I will maintain my current score.
We thank the reviewer for the constructive suggestions and for finding the overall investigation valuable. We are glad the reviewer considers the two derivations of Muon scaling rules theoretically useful, and the change of optimizer rankings under per-batch retuning relevant to optimizer evaluation and practical training. We ran each of the proposed experiments.
Q4 / W4. Robustness to the tuning procedure.
We ran the suggested check on a 30M model trained on 300M tokens, for KL-SOAP and Shampoo at 128K and 1M tokens (App. R.4, Tab. R6, Fig. R6). All searches start from the same configuration and cover learning rate, momentum and weight decay.
A wide joint grid of 75 points with factor-2 spacing, with its 5 best points rerun for 3 seeds, finds no better configuration than coordinate descent. Its best point is within two seed standard deviations of the coordinate-descent endpoint in three cells and worse in the fourth (joint − CD = (+3.5,−0.8,+1.8,+5.8)×10−3; seed s.d. 0.8×10−3 to 2.8×10−3; the acceptance threshold of our coordinate descent is 2×10−3, about one to two seed standard deviations).
A local joint grid of all 26 neighbours of the endpoint, at the 2 resolution of coordinate descent, finds no neighbour better by more than two seed standard deviations. The endpoint is therefore a joint local optimum: no diagonal move in learning rate, momentum and weight decay improves it. Following the reviewer's suggestion, Section 4 now describes our results as the best configurations found by this protocol and refers to this check.
Q5 / W5. Curvature vs. CNR.
We thank the reviewer for pointing out the relation among high CNR, gradient-noise variance and curvature in choosing the sharp directions. We measured the gradient-noise variance along the selected directions (App. R.6). The sharp directions have a 2.8–44× higher signal-to-noise ratio than the remaining directions. In Muon's geometry, CNR and curvature rankings are strongly correlated (Spearman 0.81–0.90), and their top-128 sets share 111–116 directions. Selecting by CNR therefore gives almost the same recovery as selecting by curvature (17/24/11/3% vs. 19/24/11/3%; Tab. R7).
Q3 / W3. A noisy quadratic model for KL-SOAP and Shampoo.
We added KL-SOAP to the matrix models of Appendix D with the same tuning protocol. KL-SOAP is best at small batches and Shampoo at large ones in all six settings. In two of the three tasks, a higher noise level moves the crossover to larger batches (Tab. R9, Fig. R10).