acceptodds
Under review as a conference paper at ICLR 2027

The Best Optimizer Depends on Batch Size

Abstract

A plethora of new adaptive optimizers are designed to efficiently estimate and use minibatch gradient statistics to shape parameter updates, but they are typically benchmarked at a single batch size. Hyperparameter scaling rules promise to preserve performance as batch size and gradient noise change, suggesting that the best optimizer at one batch size should remain the best at another. We challenge this approach to developing and evaluating optimizers by showing: (1) no principled scaling rule for Muon works consistently across training settings, and (2) the best optimizer for language model pretraining changes with batch size even after extensive hyperparameter tuning. To understand these failures, we analyze a noisy quadratic model and show that the critical batch size beyond which a scaling rule can break down varies across directions in the loss landscape. We further prove that gradient noise drives a bias–variance tradeoff in preconditioning that can reverse optimizer rankings across batch sizes, even under optimal learning rate and momentum tuning. Taken together, our results show that principled optimizer design must account for batch size and empirical benchmarking should be performed across batch sizes.

Primary area:
optimization
Keywords:
batch size, pretraining, hyper-parameter scaling, optimizers

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_2AoVOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how batch size affects hyperparameter scaling and optimizer rankings. It derives two scaling prescriptions for Muon, evaluates their transfer across training regimes, and reports optimizer ranking reversals after per-batch hyperparameter tuning. It further investigates these observations through noisy quadratic models (NQMs) and directional interventions in language-model pretraining.

Critical strengths
  1. The paper derives Muon scaling prescriptions through two approaches: matching an SDE approximation and minimizing a convergence upper bound. These complementary perspectives are theoretically useful.

  2. The experiments show that the tested hyperparameter scaling prescriptions do not transfer consistently across different training regimes. This highlights an important limitation of applying a single scaling recipe broadly.

  3. The paper demonstrates that batch size can change optimizer rankings despite substantial per-batch retuning. This observation is relevant to optimizer evaluation and practical training.

  4. The directional intervention shows that retaining small-batch updates in a small subspace of sharp directions can substantially reduce the large-batch branch penalty. This provides an interesting direction for improving large-batch optimization.

Critical weaknesses
  1. The empirical novelty should be positioned more precisely. Prior work has already demonstrated batch-dependent optimizer advantages [1–3] and compared modern LLM optimizer rankings across batch sizes [4]. The independently tuned crossover among preconditioned optimizers is useful, but the main text should explain more explicitly what additional understanding it provides.

  2. The upper-bound-based Muon scaling derivation needs a more self-contained explanation in the main text. Although Appendix B.3 provides technical details, the assumptions, the Frank–Wolfe gap being bounded, and the quantities held fixed when varying batch size deserve clearer presentation. The connection between minimizing this bound and selecting a learning rate for terminal training performance also needs further explanation.

  3. The NQM analysis does not fully account for the main empirical findings. It explains square-root-to-linear learning-rate scaling, but the formal crossover result compares SGD and Newton rather than KL-SOAP and Shampoo. Appendix D provides useful matrix-model experiments, but a closer qualitative agreement with the principal pretraining crossover would strengthen the explanation.

  4. Coordinate descent provides substantial tuning effort but does not establish globally optimal hyperparameters. Interactions among learning rate, momentum, and weight decay may leave some optimizers at less favorable configurations. This does not invalidate the observed crossover, but its robustness to the tuning procedure deserves further evidence, and empirical claims should distinguish the best configurations found from true optimizer-family optima.

  5. Section 6 provides evidence that retaining small-batch updates in high-curvature directions reduces the branch penalty, but its connection to the proposed CNR explanation remains unclear. CNR depends on both directional curvature and gradient-noise variance, whereas the experiment selects directions using curvature alone and does not measure their noise variances. The comparison with a random subspace therefore shows that the selected directions matter, but does not establish whether CNR explains their importance beyond curvature alone. Measuring directional noise and comparing curvature-based versus CNR-based direction selection would strengthen this connection.

  6. The practical implications remain somewhat unclear. The headline crossover occurs at the largest tested batch size, where validation loss has already deteriorated relative to smaller-batch runs. Large batches may compensate through higher throughput, but this trade-off is not quantified. It is therefore unclear when practitioners should change optimizers, reduce batch size, or adopt a different scaling or update strategy.

Questions for authors
  1. Could the authors state more explicitly in the main text the incremental contributions and their significance relative to prior work on batch-dependent optimizer performance?

  2. Could the authors provide more intuition for the upper-bound-based scaling derivation, including why the chosen bound is informative for this purpose and when its minimizer should be expected to track a useful learning rate?

  3. Can the NQM framework be extended to analyze a batch-dependent crossover involving Shampoo and KL-SOAP, or Muon, rather than only SGD and Newton? A result that qualitatively matches the pretraining observations would strengthen the theoretical connection.

  4. Could the authors better justify the coordinate-descent tuning procedure or validate its robustness against a broader joint hyperparameter grid on a smaller pretraining setup, such as a 30M model? I recognize that exhaustive tuning at the main experimental scale would be expensive; even a restricted joint grid at smaller scale could help assess hyperparameter interactions.

  5. Could the authors measure gradient-noise variance along the selected directions and assess whether CNR explains the benefit of retaining small-batch updates beyond curvature alone? For example, comparing curvature-based and CNR-based subspace selection would help clarify this connection.

  6. Could the authors summarize the practical guidance supported by their findings? In particular, when should practitioners favor large-batch training, and what changes to optimizer design or scaling prescriptions are suggested by the results?

Overall, I like this paper and currently lean toward weak acceptance. The paper combines useful empirical observations with theoretical analyses and an interesting directional intervention. Although the novelty positioning, the connection between theory and practice, and the robustness of the tuning-based comparisons could be strengthened, I find the overall investigation valuable. I would welcome further discussion with the authors and will reconsider my rating based on their response and our subsequent discussion.

References:

[1] Zhang et al. “Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model.” NeurIPS 2019.

[2] Marek et al. “Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wasteful.” NeurIPS 2025.

[3] Srećković et al. “Is your batch size the problem? Revisiting the Adam-SGD gap in language modeling.” arXiv:2506.12543, 2025.

[4] Semenov et al. “Benchmarking Optimizers for Large Language Model Pretraining.” arXiv:2509.01440, 2025.

4 comments

We thank the reviewer for the constructive suggestions and for finding the overall investigation valuable. We are glad the reviewer considers the two derivations of Muon scaling rules theoretically useful, and the change of optimizer rankings under per-batch retuning relevant to optimizer evaluation and practical training. We ran each of the proposed experiments.

Q4 / W4. Robustness to the tuning procedure.

We ran the suggested check on a 30M model trained on 300M tokens, for KL-SOAP and Shampoo at 128K and 1M tokens (App. R.4, Tab. R6, Fig. R6). All searches start from the same configuration and cover learning rate, momentum and weight decay.

A wide joint grid of 75 points with factor-2 spacing, with its 5 best points rerun for 3 seeds, finds no better configuration than coordinate descent. Its best point is within two seed standard deviations of the coordinate-descent endpoint in three cells and worse in the fourth (joint − CD = (+3.5,−0.8,+1.8,+5.8)×10−3(+3.5, -0.8, +1.8, +5.8)\times 10^{-3}; seed s.d. 0.8×10−30.8\times 10^{-3} to 2.8×10−32.8\times 10^{-3}; the acceptance threshold of our coordinate descent is 2×10−32\times 10^{-3}, about one to two seed standard deviations).

A local joint grid of all 26 neighbours of the endpoint, at the 2\sqrt{2} resolution of coordinate descent, finds no neighbour better by more than two seed standard deviations. The endpoint is therefore a joint local optimum: no diagonal move in learning rate, momentum and weight decay improves it. Following the reviewer's suggestion, Section 4 now describes our results as the best configurations found by this protocol and refers to this check.

Q5 / W5. Curvature vs. CNR.

We thank the reviewer for pointing out the relation among high CNR, gradient-noise variance and curvature in choosing the sharp directions. We measured the gradient-noise variance along the selected directions (App. R.6). The sharp directions have a 2.8–44× higher signal-to-noise ratio than the remaining directions. In Muon's geometry, CNR and curvature rankings are strongly correlated (Spearman 0.81–0.90), and their top-128 sets share 111–116 directions. Selecting by CNR therefore gives almost the same recovery as selecting by curvature (17/24/11/3% vs. 19/24/11/3%; Tab. R7).

Q3 / W3. A noisy quadratic model for KL-SOAP and Shampoo.

We added KL-SOAP to the matrix models of Appendix D with the same tuning protocol. KL-SOAP is best at small batches and Shampoo at large ones in all six settings. In two of the three tasks, a higher noise level moves the crossover to larger batches (Tab. R9, Fig. R10).

Q1 / W1. Positioning.

Prior work shows batch-dependent advantages mainly between Adam and SGD (Zhang et al., 2019; Marek et al., 2025; Srećković et al., 2025), and benchmarks modern optimizers at several batch sizes with hyperparameters transferred from one batch size (Semenov et al., 2025). We add four things. First, we retune five optimizers independently at every batch size, and the ranking among the three preconditioned methods changes with batch size under two weight-norm controls. Second, we derive two scaling rules for Muon, test whether each transfers, and also sweep a family of candidate scaling rules for Muon to see how they perform in different settings. Third, we analyze both the optimizer crossover and the batch-size scaling rules from a curvature-to-noise perspective in noisy quadratic models. Fourth, we show that a few sharp, high-SNR directions carry a disproportionate share of the local large-batch penalty. We will make these points explicit in the paper.

Q2. Intuition for the bound-based rule.

We bound the expected Frank–Wolfe gap of the iterates, a measure of stationarity over the spectral-norm ball: E G(W)≤c1B/(τγ)+c2γ+r(B)\mathbb{E}\,G(W) \le c_1 B/(\tau\gamma) + c_2\gamma + r(B), where γ=ηλ\gamma = \eta\lambda is the effective step size and τ\tau is the token budget, with momentum and weight decay held fixed (Section 3, Candidate 2). The first term spreads the initial objective gap over the τ/B\tau/B updates, and the second is the smoothness cost of the step, so the minimizing step size grows as B\sqrt{B}. As in prior bound-minimization work (Shulgin et al., 2026; Bu et al., 2026; Schaipp et al., 2025), we take the step size that minimizes the convergence bound as a proxy for the step size that minimizes the loss; the proxy is useful when these two terms capture how the loss depends on the step size and the batch size.

Q6 / W6. Practical guidance and throughput.

The main practical guidance of our work is that comparison between optimizers should be performed at the target batch size and results from one batch size may not generalize to other batch sizes. Large batches matter for throughput. In large-scale pretraining, many GPUs are available, and the batch size is usually set as large as data parallelism across them allows. The optimizer therefore has to work well at that batch size, which is the regime our comparison targets. The throughput gain depends on the hardware and the implementation, so we give one data point: on a single 8×H100 node at the 3× budget, increasing the batch from 512K to 2M tokens shortens Shampoo's training by 2.3× (Fig. R3). This measurement fixes the number of devices. At scale, a larger batch also lets more devices run in parallel.

In addition, when the batch size changes, retuning at least the learning rate and weight decay is advisable, since none of the scaling rules we tested transferred reliably in every setting (Fig. 2).

Please engage with the authors during the discussion period.

Thank you for the response. I will maintain my current score.

Reviewer_knVTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how batch size affects optimizer performance and hyperparameter transfer under a fixed training data budget. After tuning each optimizer independently at different batch sizes, the results show that KL-SOAP performs best at smaller batches, while Shampoo performs best at the largest batch tested. The paper also uses noisy quadratic models and directional experiments to study the role of curvature and gradient noise.

Critical strengths
  • The paper provides extensive experiments and hyperparameter tuning to show that the best optimizer can change with batch size, even under a fixed data budget. This is a useful finding for practical LLM training, suggesting that changing batch size may also require reconsidering the choice of optimizer.
  • The sharp-subspace experiment is interesting. Keeping small-batch updates in only 0.001% of the sharpest directions recovers up to 59.5% of the local loss gap caused by increasing batch size. This provides useful insight into how batch size affects optimization in different directions.
Critical weaknesses
  • The training data budget is relatively small. Validation loss increases monotonically with batch size across the tested range, and the largest batch size (2M tokens) has only about 800 optimizer steps. The observed optimizer crossover may partly reflect insufficient training at larger batches. It is unclear whether the conclusions hold for longer pretraining runs.
  • The connection between the proposed curvature-to-noise mechanism and the empirical results is not sufficiently established. Theorem 5.1 proves an optimizer crossover between SGD and Newton on a constructed two-dimensional quadratic, but does not explain why practical optimizers such as Muon, KL-SOAP, and Shampoo respond differently to batch size. The directional experiments focus on sharp directions without directly measuring gradient noise.
Questions for authors
  • Do the optimizer rankings and hyperparameter transfer results hold under larger training data budgets? The current 1.7B-token budget seems rather small, especially for the 2M-token batch size with fewer than 1,000 optimizer steps.
  • Can the authors further explain how different optimizers respond to directional curvature and gradient noise as batch size changes? In particular, can the proposed mechanism explain the observed crossover between KL-SOAP and Shampoo?
1 comment

We thank the reviewer for the positive assessment. We are glad the reviewer finds our main result, that the best optimizer can change with batch size under a fixed data budget, useful for practical LLM training, and the sharp-subspace experiment interesting. We answer both questions with new experiments.

W1 / Q1. Do the conclusions hold with a larger data budget?

We ran the suggested experiment: the 124M setting with 3× the token budget (5.1B tokens; 2,438 steps at 2M instead of 813). We retuned Muon, Shampoo and KL-SOAP at 512K, 1M, 2M and 4M tokens, under both decoupled weight decay and HyperBall (316 runs, plus second seeds; App. R.1).

We observe crossovers at both budgets. The training length changes the critical batch size (Merrill et al., 2025), and therefore the point of crossover also changes. At 3×, Muon and Shampoo swap order between 512K and 1M under HyperBall (Fig. R2), and between 2M and 4M under decoupled weight decay (Fig. R1; both seeds agree). At 3×, KL-SOAP has the lowest loss at every batch size we tested (Tabs. R2–R3).

We revised Section 4 accordingly. With a longer budget, optimizers still need to be compared across batch sizes, and the batch size at which their ranking changes depends on the budget.

W2 / Q2. How do the optimizers respond to curvature and noise, and does the mechanism explain the KL-SOAP/Shampoo crossover?

In the matrix models of Appendix D with KL-SOAP added, KL-SOAP is best at small batches and Shampoo at large ones in all six settings, with the crossover between batch 8 and 512 (Tab. R9, Fig. R10). This matches the mechanism of our noisy quadratic analysis, in which a preconditioner that reduces the initialization bias faster also amplifies gradient noise more, so the ranking of two preconditioners can reverse as the batch size changes.

In the language model, we measured gradient noise along the sharp directions. Their signal-to-noise ratio is 2.8–44× higher than in the remaining directions (Tab. R8). Ranking directions by curvature-to-noise ratio instead of curvature selects nearly the same directions and gives the same recovery (Tab. R7).

Reviewer_VbRROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the interaction of batch size and optimizer. The two questions that this work addresses are

  • Can a hyperparameter scaling rule consistently work in different training settings?
  • Do optimizer rankings persist across batch sizes after extensive retuning?

The paper derives hyperparameter scaling rules for Muon using two prior approaches:

  • SDE analysis preserving training dynamics
  • minimization of convergence bounds

The authors report that the best scaling rule differs between tasks and batch sizes. This validation is performed on the Modded-NanoGPT benchmark and CIFAR-5M with Resnet model.

Next, the paper compares algorithms (SOAP, Shampoo, Muon, Adam, Lion) at different batch sizes with independent hyperparameter tuning (based on coordinate descent) using the Modded-NanoGPT benchmark setting and CIFAR-5M with Resnet model. The authors report optimzer crossover phenomenon, which means that the optimizer rankings can change with batch size.

Finally, the authors study how sharpness interacts with the noise through the curvature-to-noise ratio. They show that the complex interaction between batch size and optimizer is linked to CNR. In particular, when a scaling rule transfers hyperparameters to a larger batch size, the resulting loss increase is concentrated in the sharp directions.

Critical strengths
  • The authors demonstrate that the optimizer rankings depend on the batch size. They show empirically that there can be optimizer crossover, when one optimizer is better at smaller batch sizes, but another becomes better at larger batch sizes.

  • The authors derive new scaling rules for the Muon algorithm based on SDE preservation and convergence bound minimization approaches, known in the literature. They show that the optimal scaling rule is setting-dependent.

  • The authors also study the curvature-to-noise ratio (CNR) on noisy quadratics. Based on this toy model, they demonstrate that the quality of the scaling rule depends on the reference and target batch sizes (how far they are from each other). They also show that changing CNR can affect the optimal batch size scaling.

  • The authors study how sharpness interacts with the noise through the curvature-to-noise ratio. They show that the complex interaction between batch size and optimizer is linked to CNR. In particular, when a scaling rule transfers hyperparameters to a larger batch size, the resulting loss increase is concentrated in the sharp directions.

  • The authors provide extensive empirical results to study the effect of batch size and scaling rules on the optimizers' rankings.

  • The empirical results are also observed in the HyperBall setting, where weight decay is replaced by the hard projection.

  • The authors provide the analysis for Muon with SDE and discrete time analysis.

Critical weaknesses
  • [major] The authors submit a 106-page paper. I believe that this is too much for the conference format. Either the authors should have considered keeping the main results or found a better venue, like a journal.

  • [major] I have two concerns regarding the SDE approach presented in this work:

    • The notion of invariant preservation needs further clarification. The authors describe the goal as preserving training properties, such as an approximate parameter trajectory, but it is unclear in what sense this trajectory is preserved. For example, increasing the batch size by a factor of four leads to a twofold increase in the learning rate under the proposed rule. This means that every step is two times larger, so the optimizer cannot have the same trajectory.
    • The derived SDE for Muon is not validated empirically. The authors should assess whether it approximates Muon’s actual optimization dynamics in realistic settings and identify the regimes in which the approximation is accurate.
  • The discussion of prior work contains several potentially inaccurate characterizations that the authors should address.

    • [1] [minor]: This paper similarly finds that the preferred optimizer depends on batch size and calls the transition point the crossover batch size. Rather than comparing many optimizers, it studies the Adam–SGD gap across image, vision, regression, and graph tasks with hyperparameter tuning. It also provides theoretical evidence that this crossover is an optimization phenomenon governed by problem geometry and problem-specific constants (Section 5). This perspective may help interpret the present experiments, since the tested optimizers, in some sense, exploit different geometries. I encourage the authors to discuss this connection. However, [1] appeared online in July and should therefore not be used to judge the submission.
    • [2] [major]: Figure 2 of [2] already shows that spectral algorithms become more competitive at larger batch sizes. The present work also focuses largely on this regime, using approximately 1,024 or more sequences per iteration. Including optimizers such as SOAP and Shampoo extends the comparison, but the additional insight appears limited given their related designs with Muon. Experiments at smaller batch sizes would be more beneficial to see the optimizers' rankings with tuned batch sizes.
  • [minor] The recommendation to “perform extensive hyperparameter tuning across several batch sizes” is too general to provide actionable guidance. Moreover, the batch-size search in this work appears insufficient to identify the optimal batch size for each optimizer considered, limiting the empirical support for this recommendation.

  • [major] Statements such as “Because the scaling rules are unreliable…” leave the paper’s motivation unclear. If these rules are unreliable, the authors should explain what their comparison aims to establish and what useful conclusions readers can draw from it.

  • [major] Some modeling choices underlying the derivations are insufficiently justified. For example, Candidate 1 is derived under the assumption that ω\omega remains fixed, but the authors do not explain why this restriction is appropriate. This choice may partly account for the poor empirical performance observed in Figure 1. Similarly, momentum and weight decay are held fixed without justification. These restrictions may lead to suboptimal scaling rules, so the authors should explain the rationale behind each choice and discuss its implications. Without this discussion, it is difficult to assess whether the reported limitations reflect the scaling approaches themselves or the assumptions imposed in their derivation.

  • [major] I have several concerns about the results of Figure 3.

    • First, the results are not surprising and somewhat expected. It is clear that if the reference batch size is close to the dataset size, then we should not expect the theory to work, since most of the assumptions they rely on are violated. The setting becomes close to deterministic, and therefore, in that regime, we do not have 1/B1/B scaling of batch size. It seems that these findings are also known in the literature. For example, [3] report that complete(d)P scaling rules are actionable when the batch size is smaller than the critical one.
    • Second, the low-dimensional Gaussian experiments also provide limited insight into realistic settings: they omit interactions between coordinates, may not capture the structure or distribution of gradient noise, and do not account for a finite data budget, which is important in practice.
    • Third, evaluating the relationship between reference and target batch sizes on a more realistic task, such as training a CNN on CIFAR-10, would strengthen the empirical results.
  • [major] I also have some concerns regarding the validity of experiments and claims in Section 6. A selected sharp subspace is indeed especially sensitive to replacing the small-batch update procedure with the tested large-batch procedure. However, the interventions change several things at the same time.

    • First, it is not very clear how the orthogonalization is performed in this hybrid setting. The paper never defines the update ut(B)u_t^{(B)}.
    • Second, there seems to be an ambiguity in the explanation of the observed phenomenon. Selected directions get updated 16 times more than the rest. Therefore, the selected directions have more room for changes in the landscape and noise. A larger learning rate for unselected directions can compensate for the movement distance, but it cannot adapt to changes in the loss landscape and noise.
    • Third, there is another problem with the momentum term. Small-batch and large-batch updates have a different contribution to the momentum. For one, the history from the last (1−0.95)16(1-0.95)^{16} iterations is preserved, while in the other it's 1−0.951-0.95.
    • Finally, it is also unclear how weight decay affects the results since, for one group of directions, we apply it more frequently. Therefore, I am not sure whether it is that easy to distinguish which of the changes has the largest contribution to the observed results.
  • While SDE analysis seems to be novel, the discrete time has limited novelty. Most of the steps are closely aligned with the proofs in the literature. In the discrete time, the novelty is mostly in adding Nesterov momentum to the analysis, which changes a small part in the whole proof.

[1] Zhang et al., Revisiting the Adam-SGD Gap Beyond Single Factors, NeurIPS 2026

[2] Pethick et al., Training deep learning models with norm-constrained lmos, ICML 2025

[3] Bethune et al., The Design Space of Tri-Modal Masked Diffusion Models, arXiv preprint arXiv:2602.21472 (2026)

Questions for authors
  • Can the authors provide the results for smaller batch sizes in Figure 1?

  • The notation βa\beta^a is confusing. Without looking at Table 1, a reader cannot understand this. It's worth working with 1−β1-\beta instead of β\beta and providing the scaling for 1−β1-\beta. Reading something like μ(κ)=μκ\mu^{(\kappa)}=\mu^{\kappa} is really odd.

  • It is worth highlighting somewhere in the text that some works are extendable to scalings w.r.t. token budget. For example, Power Lines explains how preserving the averaging window can be extended to such a setting.

  • Can the authors explain in more detail how the SDE in (1)-(2) is derived? Why do we need to normalize the hyperparameters per sample? Why do we need to keep ω\omega fixed while varying only μ\mu? Why should it not be adjusted as well? Is this a limitation of the SDE analysis or some deliberate choice?

  • What is the optimizer in Figure 2 (a,b)? It's written Muon-AdamW; what does it mean? I believe it means using Muon for 2D layers and AdamW for the rest. I suggest clarifying this, since Muon by default means that AdamW is used for 1D parameters.

  • Do the authors tune hyperparameters separately for 1D and 2D parameters when using Muon (with AdamW)? Can this change the rankings? How are the scaling rules used in this case: one rule for all hyperparameters or separate rules for each of them? Please provide a discussion around this concern.

  • How good is the tuning obtained by coordinate descent instead of direct independent/sequential tuning?

  • What does "empirical LLM rule" mean? Can the authors provide a detailed rule?

  • What is the difference between the approach in this work and the one in [Shulgin et al., 2026]? What are the main differences? Is the difference in considering weight decay and Nesterov momentum in the analysis, or is there something else? Why do the authors minimize the Frank-Wolfe gap, not the functional gap, as done in [Islamov et al., 2026]?

  • There are unclear rules like "empirically best rule" that are not supported by the definition.

  • In Section 4, why do the authors report the results as tables, not figures? Please add a figure that shows the performance in the HyperBall setting.

  • Does [Meterez et al, 2026] actually show that the quadratic model is good? Their experiments show that it becomes good enough at the end of training, where the learning rate following a cosine schedule becomes small. Of course, if the model does not move much, then the quadratic model is good. To support such claims, it would be much better to demonstrate the gap between the real loss and the quadratic approximation using a fixed learning rate. Moreover, I do not see the reason why having a matched loss means that the dynamics of optimizers on the loss will be the same. Since Section 5 is based on this work, I would like the authors to clarify these points to justify the use of quadratic approximation.

  • Is CNR connected to under- or over-training regimes?

  • I also did not find any SDE in complete(d)P paper. How is it different from the one derived here or other prior work?

11 comments

We thank the reviewer for the detailed review. We appreciate that the reviewer recognizes the extensive experiments showing that optimizer rankings depend on batch size, also under HyperBall, and the scaling rules for Muon derived from the SDE and from convergence bounds. We address each concern below.

We respectfully disagree. Our paper includes extensive theoretical and empirical results, which accounts for its length. Several papers of about 100 pages have been accepted at similar conferences [1,2,3].

[1] Fantastic Pretraining Optimizers and Where to Find Them, ICLR 2026 (108 pages)

[2] Language Modeling by Language Models, NeurIPS 2025 (93 pages)

[3] Are Greedy Task Orderings Better Than Random in Continual Linear Regression?, NeurIPS 2025 (102 pages)

In what sense does the SDE rule preserve the trajectory? The rule preserves the distribution of (W,M)(W, M) at a matched number of samples processed. Four steps of size η\eta at batch BB and one step of size 4η4\eta at batch 4B4B use the same samples, and their distributions agree to first order in η\eta. A 4× larger batch therefore takes a 4× larger step, a quarter as often. The reviewer's example of a 2× larger step is the square-root rule, which is not the rule we propose in the paper.

Empirical validity of the SDE. Due to the time constraint, we validate the SDE on a noisy 16×16 matrix quadratic. With 2,048 independent paths, we transport discrete MuonW from batch 1 to batch κ∈{1,…,64}\kappa \in \{1, \dots, 64\} with the SDE rule and compare it with the Euler–Maruyama solution of Eqs. (1)–(2). The two agree within about 1% while κη0≤0.064\kappa\eta_0 \le 0.064. The error is 2–4% at κη0=0.16\kappa\eta_0 = 0.16 and 15% at κη0=0.64\kappa\eta_0 = 0.64. The rule is thus accurate when the step at the target batch is small compared with the curvature, which is the regime the SDE approximation assumes. In the language-model experiments of Fig. 2, the largest target batches are outside this regime. On a 30M language model, we also test a consequence of the SDE: runs transported by the SDE rule should follow the same loss curve in tokens. We transport the MuonW configuration of our main experiments at 128K tokens to 64K and 32K. The final losses at the three batch sizes differ by 0.015. With all learning rates divided by 4, they differ by 0.012 under the SDE rule and by 0.087 under the square-root rule (Tab. R11).

Justification of scaling of ω\omega, momentum and weight decay in Candidate 1. For Candidate 1, the goal of the scaling rule is to keep the corresponding Muon SDE unchanged and therefore all hyperparameters are normalized per sample because the SDE's time variable is the number of samples processed. This exact invariance induces all hyperparameter scaling (including forcing ω\omega to be fixed). Intuitively, the Nesterov momentum coefficient ω\omega mixes the current gradient and the momentum, which is the average of gradient over a fixed continual time interval in the corresponding SDE. To keep the SDE constant, ω\omega has to stay constant so that the mixing ratio stays constant when the batch size changes. For the same motivation, the SDE rule sets the momentum μ→μκ\mu \to \mu^{\kappa}, which keeps the decay per sample constant. It keeps λ\lambda because the SDE depends on λ\lambda only through (η/B)λ(\eta/B)\lambda, which the rule keeps fixed.

Fixed ω\omega does not explain why performance is poor. Our MuonW implementation ties ω\omega to μ\mu. This is stated in Section 3; the revision adds a check that the two choices differ by less than 1.5% (App. R.8).

Possible confounds in Section 6. We ran a control for each (App. R.5, Tab. R7).

  1. Definition of ut(B)u_t^{(B)}. Both updates are defined in the paragraph "Directional branching" of Section 6. The small-batch copy computes ut(B)u_t^{(B)} from each 128K-token gradient. The large-batch copy computes ut(κB)u_t^{(\kappa B)} from the mean of the last 16 gradients, once every 16 steps, with its own momentum buffer. Each copy orthogonalizes its own momentum, and the projection QQ is applied afterwards.

  2. Update frequency and landscape change. The random-subspace control splits the directions with exactly the same cadence and recovers nothing (−8% to 0%). Update frequency alone therefore cannot produce the effect.

  3. Momentum memory. We set the large-batch momentum to μ16\mu^{16}, so that both channels average over the same number of tokens. The sharp hold still removes 28–34% of the local penalty, and a random subspace 3%.

  4. Weight decay. Weight decay acts on the full weights once per base step in every branch, including the control and the fully scaled branch. The decay per token is therefore the same for all directions and branches.

Concern 1 about Figure 3: scaling rule is expected to fail at large batch size. We agree that the theoretical rules are expected to fail when the reference batch becomes very large. However, the batch sizes in our experiments are realistic ones that are already used in practice. It is therefore of both empirical and theoretical interest to study robust scaling rules in this regime, and the NQM provides a simple setting in which to study it theoretically.

Concern 2 about Figure 3: Noisy Quadratic Model may not be realistic and doesn't capture interaction between coordinates. We agree that the NQM is a simplified model that captures only part of the training dynamics. However, Section 6 validates that the NQM captures at least some local properties of batch-size scaling in pretraining. We also added matrix-based synthetic tasks (Fig. R10), in which we reproduce the crossover between KL-SOAP and Shampoo.

Concern 3 about Figure 3: we should add empirical experiments on CIFAR. We already include results for a ResNet trained on CIFAR-5M in the original paper (Fig. 2b).

Prior work. We thank the reviewer for correcting details about the prior work.

We agree that Zhang et al. (NeurIPS 2026) also observe a crossover batch size for the Adam–SGD gap. Its geometric explanation complements our crossover among preconditioned optimizers, and we will discuss it in the paper.

Regarding Figure 2 of Pethick et al., Scion’s batch size experiments are relevant prior work, but our goal is broader than assessing any individual optimizer. Our work is complementary, with a focus on what these comparisons imply for optimizer benchmarking. Whereas Scion’s Fig. 2 emphasizes Scion/uScion’s robustness at larger batches, our Fig. 1 shows that the best optimizer can change with batch size.

We added this comparison to the related work of the revised paper ("Comparing optimizers").

Regarding scale, our smallest batch is 128K tokens, which is 128 sequences of 1,024 tokens. We added a batch of 64K tokens (64 sequences). The ranking there is the same as at 128K: KL-SOAP, Shampoo, Muon, AdamW (App. R.2).

Motivation and recommendation. We rewrote the motivation of Section 4: no single scaling rule transfers reliably across settings, so a fair comparison across batch sizes has to retune each optimizer at each batch size. Our recommendation is to compare optimizers at several batch sizes and training budgets, retuning at least the learning rate and weight decay at each. We do not claim to find the optimal batch size of each optimizer. Our claim concerns the ranking at a given batch size, which is the decision practitioners face for a given hardware budget.

Discrete-time analysis. We agree that the main novelty of the SDE analysis is the treatment of Muon and Nesterov momentum. We view the main contribution of our work as the empirical validation of the different scaling rules.

Smaller batches in Fig. 1. We added 64K tokens for KL-SOAP, Shampoo, Muon and AdamW; the ranking is the same as at 128K (App. R.2).

Notation βa\beta^a. We agree and will state the rules in terms of 1−β1-\beta in the camera-ready version.

Power Lines and the token budget. We agree; Section 2 now notes that Power Lines also prescribes how λ\lambda scales with the token budget.

Derivation of Eqs. (1)–(2), per-sample normalization, ω\omega. See above.

Muon–AdamW in Fig. 2. Yes, Muon updates the 2-D hidden matrices and AdamW updates the embeddings, output head and scalars, as defined in Section 3. We will repeat this in the caption.

Separate tuning of 1-D and 2-D parameters. The AdamW learning rates, β1\beta_1 and β2\beta_2 are coordinate-descent coordinates separate from the matrix ones. The scaling rules transfer the matrix and AdamW groups with separate rules (the columns of Tab. 1).

Quality of coordinate descent. Our coordinate descent sweeps every hyperparameter in parallel around the current configuration, accepts the best move, and repeats until no move improves the loss. Tuning each hyperparameter independently, or once in sequence, is a weaker search than this, because it ignores the interactions between hyperparameters or stops after one pass. On a 30M model at a batch size of 1M tokens, both procedures, run from the same starting configuration and candidate values as coordinate descent, end above it in final loss: 3.882 vs. 3.858 for KL-SOAP, and 3.917 (independent) and 3.906 (sequential) vs. 3.887 for Shampoo. We also compare coordinate descent with a stronger baseline, a joint search. On a 30M model, coordinate descent matches a 75-point joint grid and a 27-point local joint grid within seed noise in all four cells (App. R.4).

"Empirical LLM rule" and "empirically best rule". The empirical LLM rule keeps η\eta and μ\mu fixed and scales λ\lambda linearly; for AdamW it keeps η\eta and β1\beta_1 and sets β2→β2κ\beta_2 \to \beta_2^{\kappa} (Tab. 1). It is the best of the 216 rules at 2M in our sweep. "Empirically best rule" means the best rule of that sweep at a given target batch. Section 3 now names the empirical rules.

Comparison with Shulgin et al. (2026). The main difference is the consideration of weight decay. When weight decay is considered, the Frank-Wolfe gap is a natural subject given the connection between Frank-Wolfe method and weight decay. This makes our theoretical analysis simpler.

Figures for the tables of Section 4. Added for both weight-decay variants (Fig. R12).

Meterez et al. and the quadratic model. Meterez et al. show that, in the second half of training with a cosine schedule, the local quadratic model already yields loss curves similar to those on the original loss landscape. One important point is that this similarity is established by evaluating the original loss function along the trajectories obtained by minimizing the quadratic model, which strongly supports the connection between the two landscapes. On the other hand, our Section 5 is motivated by Meterez et al., but its goal is to show that a batch-size crossover can occur in a simple analytical setting, which does not depend on the full validity of their result.

CNR and under- or over-training. CNR is a property of each direction, not of the training regime as a whole, but it changes during training. For MuonW, the signal-to-noise ratio of the sharp subspace falls from 8.3 to 0.46 between steps 1K and 9K (Tab. R8), so more directions are dominated by noise later in training.

SDE in complete(d)P. In complete(d)P (https://arxiv.org/abs/2512.22382), the authors mention that they derive an "SDE-inspired scaling rule". The main difference with our work is that we derive an SDE for MuonW and state it explicitly (Eqs. (1)–(2)). Complete(d)P extends the AdamW parameterisation of Malladi et al. (2022), which has a consistent SDE limit, and obtains the square-root rule for the learning rate and weight decay. MuonW has a different SDE, so its rule differs: the learning rate scales linearly with the batch size and the momentum as μ→μκ\mu \to \mu^{\kappa}.

Reviewer_BaMJOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how batch size affects hyperparameter scaling rules and optimizer rankings after independent tuning. Two theoretically motivated scaling rules for Muon are evaluated on language modeling and image classification, showing that the preferred rule depends on the training setting. Under a fixed token budget, KL-SOAP leads at smaller batches, while Shampoo performs best at the largest batch. A noisy quadratic analysis examines direction-dependent scaling and preconditioning tradeoffs, while directional interventions show that a few sharp directions account for much of the local batch-size penalty.

Critical strengths

1.The paper highlights the limitations of optimizer evaluation at a single batch size and provides evidence for comparing optimizers across batch sizes. 2.The analysis connects directional curvature and gradient noise to learning-rate scaling and optimizer rankings, offering useful insights into the benefits and costs of preconditioning. 3.Systematic comparisons and targeted interventions provide evidence beyond aggregate loss, helping identify which directions contribute to the batch-size penalty.

Critical weaknesses

1.The connection between the proposed mechanism and the optimizer crossover remains incomplete. The language-model intervention selects directions by preconditioned curvature without measuring directional noise, although the theory centers on the curvature-to-noise ratio. It also uses square-root learning-rate scaling without retuning. Thus, it shows that the penalty under this scaling protocol is concentrated in sharp directions, but does not establish that these directions explain the KL-SOAP–Shampoo crossover after retuning. 2.The independently retuned comparison is limited to one relatively small language model and a roughly 1.7B-token budget. The nanochat experiments broaden the evidence but use hyperparameter transfer, leaving the robustness of the crossover across model scales and training horizons unclear.

Questions for authors

1.After retuning the large-batch learning rate at each branching checkpoint, does retaining small-batch updates in the top-768 directions still recover a substantial fraction of the loss gap? 2.Do the sharp directions have distinctive gradient-noise levels? Would selecting subspaces by CNR better predict the recovery than curvature alone, especially late in training? 3.Does the crossover persist with the original SOAP implementation, larger models, or longer training horizons? What evidence identifies the factors that determine its location?

2 comments

We thank the reviewer for the careful review. We are glad the reviewer finds that the paper shows the limitations of evaluating optimizers at a single batch size, and that the analysis connecting directional curvature and gradient noise to learning-rate scaling and optimizer rankings gives useful insight into the benefits and costs of preconditioning. We answer the reviewer's questions with new experiments.

W1 / Q1. Does the intervention result hold after retuning the large-batch learning rate?

The connection between the proposed mechanism and the optimizer crossover remains incomplete. We agree with the reviewer. Our directional CBS analysis mainly focuses on the local training dynamics, similar to prior work that uses the noisy quadratic model (https://arxiv.org/abs/2607.21716). We also observe this in a newly added 30M experiment, where the local penalty is reduced but the effect on the final loss is much milder.

Therefore, we do not think that this interpretation fully explains the crossover phenomenon.

Do the directional CBS results hold when we retune the learning rate? It is known that, in pretraining, reducing the learning rate typically makes the loss drop sharply in the short term (https://arxiv.org/abs/2410.05192v3). Choosing the learning rate that minimizes the local batch-size penalty is therefore biased towards lower learning rates. We designed the following experiment on a 30M model (App. R.5, Tab. R7, Figs. R7–R8).

The parent run is MuonW at 128K tokens, the large batch is 16× larger, and the held subspace is spanned by the top 128 directions of P1/2GP1/2P^{1/2} G P^{1/2}. We ran the branches to the end of training to select the best learning rate after scaling. The best learning rate is 1 or 1/2 of the value given by the square-root rule, so the rule is close to optimal.

In this setting, the advantage of holding the sharp directions is present locally, on the branches run for 8% of training. It decreases over the training steps when the learning rate decays: at the end of training, the hold removes 0–4% of the final penalty (Tab. R7). The main cause of this shrinking advantage is learning-rate decay: when we freeze the learning rate at its value at the branch point, the advantage persists to the end of training (0.017 nats at the start of the branch, 0.019 at the end).

In conclusion, the square-root scaling rule is a realistic choice for preserving long-term performance when the batch size is scaled. The scaling motivated by the directional CBS analysis mostly reduces the local batch-size penalty, and learning-rate decay reduces its effect. We have updated Section 6 to make this clearer.

Q2. Do sharp directions have distinctive noise, and does selecting by CNR help?

We found that the largest-CNR subspace roughly corresponds to the highest-curvature subspace. On the 124M model, the sharpest 16 directions have a 2.8–44× higher per-batch signal-to-noise ratio than the remaining ~85M directions (Tab. R8, Fig. R9). The gap narrows during training.

In Muon's preconditioned geometry, the curvature-to-noise ratio and the curvature are strongly rank-correlated (Spearman 0.81–0.90 over the top 256 directions of the 30M model). The top 128 directions under the two rankings share 111–116 directions. Selecting by CNR gives the same recovery as selecting by curvature at every branch point, including late ones (17/24/11/3% vs. 19/24/11/3%; Tab. R7). We conclude that curvature is a good proxy for CNR.

W2 / Q3. Larger models, longer training, and what sets the crossover location.

Longer training. We retuned Muon, Shampoo and KL-SOAP at 3× the token budget, at 512K–4M tokens, under both weight-norm controls (App. R.1). We observe crossovers at both budgets. The training length changes the critical batch size (Merrill et al., 2025), and therefore the point of crossover also changes. At 3×, Muon and Shampoo swap order between 512K and 1M under HyperBall (Fig. R2), and between 2M and 4M under decoupled weight decay (Fig. R1). At 3×, KL-SOAP has the lowest loss at every batch size we tested (Tabs. R2–R3).

Larger model. On a 350M model at the 1× budget, Muon and Shampoo are tied at 512K and Muon is ahead at 2M (App. R.3). Due to the limited compute budget, we retuned only the matrix learning rate at this scale.

Crossover location. Our results point to two factors: the training budget, through the critical batch size, and the noise level relative to the curvature. In the matrix models, a higher noise level moves the crossover to larger batches in two of three tasks (Tab. R9). A predictive account of the location remains open.

Please engage with the authors during the discussion period.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Discussion Summary for the Area Chairs

We would like to briefly summarize the discussion period for the benefit of the area chairs. Our paper studies how batch size changes the ranking of modern optimizers after retuning, how well scaling rules transfer those settings, and how curvature and gradient noise help explain the results. All four reviewers identified our main observation, that the ranking of optimizers changes with batch size even when every optimizer is retuned at every batch size, as a strength of the paper.

During the discussion, we added several experiments in response to the reviewers’ questions:

  • At 124M parameters and 5.1B tokens, Muon and Shampoo still swap order across batch sizes after independent retuning under both HyperBall and decoupled weight decay. KL-SOAP performs best at every batch size tested at this training budget.
  • On a 30M model, we ran joint sweeps over learning rate, momentum, and weight decay. These sweeps show that coordinate descent is competitive with joint search: none of the configurations we tested improved on it by more than two seed standard deviations.

Reviewer BaMJ: The reviewer asked about (1) how to interpret the sharp direction intervention in Section 6 and (2) validation on larger models. Regarding (1), we validate that the batch size scaling we used for studying batch size schedule is reasonable by performing a tuning on a small scale proxy, and sharpen our claims in Section 6 to be about local penalty rather than final loss. Regarding (2), we reproduced the batch size crossover phenomenon on a 350M model setting.

Reviewer knVT: The reviewer asked (1) whether our results would hold with more training, (2) why we consider the sharpest subspace in Section 6 instead of the space with highest curvature-to-noise (CNR). For (1) We upscale to 5.1B tokens and retuned Muon, Shampoo, and KL-SOAP at four batch size and find Muon and Shampoo still swap order. For (2), we find that the sharpest subspace also has much higher CNR compared to the other directions. Ranking directions by curvature-to-noise ratio instead of curvature selects nearly the same directions and gives the same recovery.

Reviewer 2AoV: The reviewer asked for a validation of our coordinate descent tuning method. We tested joint searches over learning rate, momentum, and weight decay. Neither found a setting more than two seed standard deviations better than coordinate descent in any setting. The reviewer also shared a similar question to question (2) of Reviewer knVT.

Reviewer VbRR:

SDE derivation & validation. The reviewer has many questions on our SDE derivations, centered around (1) what the scaling rule preserves, (2) how to scale some hyperparameters such as Nesterov momentum and weight decay following SDE, and (3) how to validate the SDE. For (1), we explained that the rule preserves the distribution of weights and momentum at a matched number of samples processed. We also restate how we derive (2). For (3), We compared discrete Muon directly with the SDE on a noisy matrix quadratic (Appendix R.8).

We also rederive the SGD SDE to clarify why linear scaling, rather than the square-root scaling suggested by the reviewer, is appropriate. Despite extensive back-and-forth on these points, we maintain that our derivations and interpretations are correct. We believe much of the disagreement arises because the reviewer's interpretation of the SDE framework differs from the standard formulation in the literature, rather than from substantive issues with our analysis. We hope our detailed mathematical explanations and additional empirical validation sufficiently address these concerns.

Section 6 controls. The reviewer asked whether the sharp-direction effect could be explained by update frequency, momentum memory, or weight decay. We addressed these concerns with a random-subspace control at the same update frequency, matched momentum memory in tokens, and the same weight decay per token across branches. The local effect persisted.

Tuning and smaller batches. The reviewer asked how coordinate descent compares with other tuning procedures and whether the rankings in Figure 1 hold at smaller batch sizes. We found that coordinate descent is comparable to joint search and better than the independent tuning and sequential tuning suggested by the reviewer. We also added a smaller batch size and found the same ranking.

Positioning and presentation. The reviewer asked us to distinguish our contribution from prior work and make the paper easier to follow. We clarified that many of the empirical observations on this paper are novel and can potentially lead to better optimizer design in large batch size regimes. Meanwhile, we agreed that we should improve the readability of the appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.