acceptodds
Under review as a conference paper at ICLR 2027

Towards joint scaling laws with optimal batch size schedules

Abstract

Modern deep learning typically keeps the batch size static throughout training, thus overlooking the joint effect of learning rate and batch size on the training dynamics. In this paper, we study the deep learning dynamics through the lens of convex optimization and derive a joint characterization of loss in terms of both schedules, applicable to general optimizers and model architectures. This characterization yields a closed-form optimal batch size schedule for any prescribed learning rate schedule, and further leads to joint scaling laws that consistently outperform static batch size baselines, highlighting the significance of dynamic batch size schedule in large language model training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM pretraining, scaling law, batch size

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_GsC4OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper derives a batch-size schedule for a prescribed learning-rate trajectory, training horizon, and token budget. Starting from a convex-SGD bound and a gradient-variance model, it constructs a continuous loss surrogate and obtains a formula for batchsize. The schedule is combined with training-horizon scaling rules for learning rate and weight decay. Experiments cover dense and MoE language-model pretraining, vision-language training, and mathematical fine-tuning, reporting approximately 6–15% improvements in FLOP-based efficiency over static batching. The contribution is a specific formulation of batch allocation: distribute a fixed sample budget across a fixed number of updates to minimize a predicted final loss. This differs conceptually from Seesaw’s objective of preserving reference performance while reducing sequential updates. The proposed formulation is useful, but its mathematical justification and comparative evaluation need strengthening.

Critical strengths

1 A simple and actionable scheduling rule.

The method produces explicit schedules without online gradient-noise estimation or additional task-specific fitting. Its predictions depend meaningfully on the learning-rate shape: constant learning rate leads to increasing batches, linear decay leads to constant batches, and WSD leads to increasing batches during the stable phase followed by constant batches during cooldown.

2 Meaningful empirical breadth.

The experiments include dense models up to approximately 7B parameters, MoE models, eight language benchmarks, and two post-training settings. Downstream improvements provide evidence beyond training loss: Table 1 reports gains on eight of eight tasks for the dense model and seven of eight for the MoE model.

Critical weaknesses

1 The discrete-to-continuous derivation contains unresolved endpoint inconsistencies.

  1. The optimality claim is stronger than the demonstrated result.

Theorem 3 correctly solves the relaxed continuous allocation problem. However, minimizing a surrogate derived from an upper bound does not establish that the resulting schedule minimizes actual training loss.

Questions for authors

In Eq. (2.6), setting the final learning rate to zero makes the penultimate term’s denominator zero when applied that to the last training iteration. How are the discrete and continuous endpoint issues resolved, and do the corrected expressions preserve the proposed schedule and convergence claims?

5 comments

We thank the reviewer for thoroughly reading our paper and for the valuable suggestions! Bellow are our responses. Please kindly raise the score if we have addressed your questions.

Summary

We explicitly require positive learning rate and delete Eq. (2.6) which diverges when learning rate is zero. We directly connect Eq. (2.5) to (3.1) and add Appendix A.3 to justify. We add wall-clock time in Figure 1 to demonstrate the advantage of our batch size schedule. We significantly extend Appendix A.2 to apply our method to adaptive gradient descent like AdamW and Muon. All new content is presented by blue text.

Weakness 1. The discrete-to-continuous derivation contains unresolved endpoint inconsistencies.

Response: see our respose to the question section below.

Weakness 2. The optimality claim is stronger than the demonstrated result. Theorem 3 correctly solves the relaxed continuous allocation problem. However, minimizing a surrogate derived from an upper bound does not establish that the resulting schedule minimizes actual training loss.

Thank you for making this distinction. Theorem 3 establishes optimality for the allocation objective in Eq. (3.3), for a prescribed LR schedule, horizon, and budget. It yields an analytical schedule whose shape requires no additional fitting: the batch-independent terms drop out, and the fixed positive noise coefficient cancels upon normalization. The experiments demonstrate the practical value of this prescription through 6–15% FLOP-efficiency gains against static batching.

We do not interpret this fit as proving that the theoretical bound is exact for arbitrary nonconvex training. We believe that FLOP-efficiency gains are enough to show the upper bound surrogate's effectiveness.

Question. In Eq. (2.6), setting the final learning rate to zero makes the penultimate term’s denominator zero when applied that to the last training iteration. How are the discrete and continuous endpoint issues resolved, and do the corrected expressions preserve the proposed schedule and convergence claims?

The reviewer is correct about the original substitution. We deleted Eq. (2.6); at every finite horizon, Eq. (2.5) retains positive discrete learning rates and the terminal contribution ηT(G2+X/BT)/2\eta_T(G^2+X/B_T)/2. This makes each finite tail denominator positive.

In fact, we can directly connect Eq. (2.5) to Eq. (3.1) by showing the difference of equations vanishes as T goes to infinity. For the linear example in Appendix A.3, the precise comparison is on ηtdisc=T−1/2(1−tT+1),\eta_t^{\mathrm{disc}}=T^{-1/2}\left(1-\frac{t}{T+1}\right), the displayed calculation gives ∣FTdisc−FTcont∣=O(log⁡T/T3/2)|F_T^{\mathrm{disc}}-F_T^{\mathrm{cont}}|=O(\log T/T^{3/2}), including the terminal contribution. The final discrete LR remains positive at every finite horizon. This shows a slight discrepancy to Table 2, where we use T−1/2(1−tT)T^{-1/2}\left(1-\frac{t}{T}\right) for the sake of simplicity. This is justifiable as Eq. (3.1) with either linear schedules is asymptotically equivalent.

Dear Reviewer GsC4,

Thank you again for your constructive feedback. We have provided detailed responses to your questions and concerns, along with additional clarifications and revisions.

As the discussion period is approaching its end, we would greatly appreciate it if you could take a look at our responses and let us know whether any concerns remain. We would be happy to provide further clarification.

Thank you for your time and consideration!

Hi authors, thanks for the feedback. Removing (2.6) and connecting (2.5) to (3.1) only moves the singularity point but does not eliminate it. Particularly (3.1) seems to diverge for constant LR scheduling. And if you look at Table 3, L_dynamic - L_static would also become infinite. So the whole modeling seems have some fundamental inconsistencies to be addressed. Please check it carefully and see if they could be fixed.

Reviewer_T5sjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper derives a batch-size schedule from a convex SGD loss bound, for a fixed learning-rate schedule and token budget. It combines this schedule with learning-rate and weight-decay scaling rules. Experiments across language models and post-training tasks show gains over static batch sizes.

Critical strengths
  1. A simple, useful scheduling rule with no extra tuning of its shape. A clear attempt to study learning rate and batch size together under a fixed budget.
  2. Broad experiments across dense models, MoE models, and vision-language tasks, using both AdamW and Muon. Improvements in validation loss and several downstream tasks.
Critical weaknesses
  1. The “optimal” claim goes beyond the evidence. The schedule minimizes a convex SGD bound with fixed gradient and noise constants, rather than the actual neural-network loss. Fitting this expression to observed loss curves does not establish that the bound is tight or that the dynamics become convex. Tests on unseen schedules with shared fitted constants needed.
  2. A problem with the zero-learning-rate endpoint. Equation 2.6 sets ηT=0\eta_T=0, but its t=T−1t=T-1 term divides by ηT\eta_T, making the expression divergent when the numerator is nonzero. Replacing sums with integrals does not resolve the discrete formula. A corrected endpoint treatment and its effect on the schedule needed.
  3. Limited comparison with other batch-size schedules. Most results compare against static batches. Comparisons with simple increasing schedules and existing adaptive rules needed to establish the value of the specific proposed shape.
  4. The efficiency claims lack wall-clock evidence. Reported gains mainly measured in tokens, FLOPs, or iterations. Changing batch size also changes hardware use and communication costs. The general cost-based extension is not tested with measured training times, and practical batch-size limits and rounding need clearer treatment.
Questions for authors

N/A

3 comments

We thank the reviewer for thoroughly reading our paper and for the valuable suggestions! Bellow are our responses. Please kindly raise the score if we have addressed your questions.

Summary

We explicitly require positive learning rate and delete Eq. (2.6) which diverges when learning rate is zero. We directly connect Eq. (2.5) to (3.1) and add Appendix A.3 to justify. We add wall-clock time in Figure 1 to demonstrate the advantage of our batch size schedule. We significantly extend Appendix A.2 to apply our method to adaptive gradient descent like AdamW and Muon. All new content is presented by blue text.

  1. The “optimal” claim goes beyond the evidence. The schedule minimizes a convex SGD bound with fixed gradient and noise constants, rather than the actual neural-network loss. Fitting this expression to observed loss curves does not establish that the bound is tight or that the dynamics become convex. Tests on unseen schedules with shared fitted constants needed.

Thank you for bringing up the distinction between surrogate optimality and practical optimality. Theorem 3 establishes optimality for the allocation objective in Eq. (3.3), given the LR schedule, horizon, and budget. Figure 2 demonstrates the empirical agreement of the resulting loss model with observed neural-network training trajectories. Convex analysis has also provided useful guidance for practical LR scheduling in prior work [1, 2]. Our experiments evaluate the derived prescription and show 6–15% compute-efficiency improvements over the static baselines in Figure 5.

Crucially, the LR schedule and budget determine the batch schedule analytically. Under the fixed-coefficient model, the L∗,D2,G2L_*,D^2,G^2 terms do not depend on batch allocation, and fixed positive XX cancels from the normalized optimizer. We can therefore prescribe the schedule for an unseen run directly, independently of the fitted coefficient values. This separates schedule design from predicting an entire unseen loss curve with transferred coefficients.

  1. A problem with the zero-learning-rate endpoint. Equation 2.6 sets (\eta_T=0), but its (t=T-1) term divides by (\eta_T), making the expression divergent when the numerator is nonzero. Replacing sums with integrals does not resolve the discrete formula. A corrected endpoint treatment and its effect on the schedule needed.

Thank you for identifying this error. We have deleted the original Eq. (2.6) and retained the complete finite Eq. (2.5), including ηT(G2+X/BT)/2\eta_T(G^2+X/B_T)/2, with positive discrete learning rates. Every finite tail denominator is then positive. In fact, we can directly connect Eq. (2.5) to Eq. (3.1) by showing the difference of equations vanishes as T goes to infinity. See Appendix A.3 for examples of linearly-decaying-learning-rate.

  1. Limited comparison with other batch-size schedules. Most results compare against static batches. Comparisons with simple increasing schedules and existing adaptive rules needed to establish the value of the specific proposed shape.

Our experiments establish improvements over matched static-batch baselines. In the scaling-law experiments, the dynamic schedule uses the same LR and weight decay tuned for static batching, and the sequence length is fixed within each model family. We believe that obtaining the reported 6–15% compute-efficiency gains without retuning these hyperparameters is already a useful contribution. The analytical result also explains why the batch schedule depends on the LR schedule: linear decay gives constant batching, whereas cosine and WSD give different allocations.

We appreciate the value of comparisons with tuned dynamic alternatives. Appendix E documents existing practice, whose training recipes differ in data, LR, and potentially sequence-length schedules. A controlled comparison requires matching these conditions and retuning the competing schedules, which is outside the scope of this rebuttal.

  1. The efficiency claims lack wall-clock evidence. Reported gains mainly measured in tokens, FLOPs, or iterations. Changing batch size also changes hardware use and communication costs. The general cost-based extension is not tested with measured training times, and practical batch-size limits and rounding need clearer treatment.

Our primary objective is loss at a fixed token/FLOP budget, and we now report actual training time separately. Figure 1 gives complete-run timings on eight H200 GPUs: static/dynamic-baseline/large-batch-dynamic runs took 23.2/22.0/18.1 hours for Llama3 and 23.1/18.1/17.4 hours for Qwen3 MoE. These measurements show that the reported improvements were obtained with shorter total training times in these configurations.

The general cost extension provides an analytical formulation for allocation under a chosen cost function, complementing the timing evidence for our existing FLOP-budget experiments. We consider these extensions as future work that is out of scope of this paper.

References

[1] Aaron Defazio, Ashok Cutkosky, Harsh Mehta, and Konstantin Mishchenko. Optimal Linear Decay Learning Rate Schedules and Further Refinements. arXiv:2310.07831, 2024 revision.

[2] Fabian Schaipp, Alexander Hägele, Adrien Taylor, Umut Şimşekli, and Francis Bach. The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Training. 2025.

Dear Reviewer T5sj,

Thank you again for your constructive feedback. We have provided detailed responses to your questions and concerns, along with additional clarifications and revisions.

As the discussion period is approaching its end, we would greatly appreciate it if you could take a look at our responses and let us know whether any concerns remain. We would be happy to provide further clarification.

Thank you for your time and consideration!

Reviewer_kiUGOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies learning-rate and batch-size schedules jointly for large-scale neural network training. Starting from a stochastic convex optimization bound, the authors model the mini-batch gradient second moment as G2+X/BtG^2+X/B_t and derive a sequence-to-sequence characterization of training loss as a function of both the learning-rate schedule ηt\eta_t and batch-size schedule BtB_t. Optimizing the resulting continuous surrogate under fixed data/compute budgets yields a closed-form batch-size schedule that depends on the shape of the learning-rate schedule but not its peak value. The paper further combines this schedule with 1/T1/\sqrt{T} scaling of peak learning rate and weight decay to propose joint scaling laws. Extensive experiments across Llama and Qwen MoE models, Muon and AdamW, different learning-rate schedules, model scales up to 7B, VLM training, and math fine-tuning generally show advantages over static batch-size training.

Critical strengths
  1. The paper provides a simple and useful theoretical framework for jointly reasoning about learning-rate and batch-size schedules. Rather than treating batch size as a fixed hyperparameter or coupling it heuristically to the learning rate, the authors formulate an explicit constrained optimization problem and obtain a closed-form schedule. An appealing property is that, under the proposed model, the schedule depends only on the learning-rate shape and total data/compute budget, and not on the peak learning rate. This makes the proposed rule easy to transfer and avoids an additional hyperparameter sweep.

  2. The empirical validation is unusually broad and generally supports the proposed scheduling principle. The authors evaluate different learning-rate schedules, base batch sizes, learning rates, optimizers, model families, dense and MoE architectures, model scales up to 7B, and both pre-training and post-training settings. The reported compute-efficiency gains of approximately 66--1515% in scaling experiments are practically meaningful. I also find the controlled experiment showing that identical ηt/Bt\eta_t/B_t trajectories do not imply identical training dynamics particularly informative.

Critical weaknesses
  1. The central "optimal" batch-size schedule relies on treating the gradient-noise term as time-independent, and it is unclear how robust the main conclusion is when this assumption is relaxed.

The key assumption is

E∣gt∣2≤G2+X/Bt\mathbb{E}|g_t|^2 \leq G^2 + X/B_t,

where GG and XX are global upper bounds. The subsequent optimization effectively treats XX as a constant, which is why it cancels from the optimizer and yields a schedule determined only by the learning-rate shape. The paper further relates X/G2X/G^2 to the critical batch size and treats it as time-independent. This issue is important because the central result changes if the covariance term is allowed to vary with time. For example, replacing XX with XtX_t leads to an objective proportional to

∫0TXtbt2/Btdt\int_0^T X_t b_t^2/B_t dt,

whose solution under the same token constraint is generally

Bt∝btXtB_t \propto b_t\sqrt{X_t},

rather than simply Bt∝btB_t\propto b_t. Therefore, the claim that the optimal schedule is independent of model, optimizer, and training state appears stronger than what the current theoretical model establishes.

  1. The theoretical and practical generality claims extend beyond what is currently justified, particularly for adaptive optimizers and real training systems. The main theorem is derived for convex objectives and SGD. Appendix A.2 notes that for a generic adaptive map ϕ(gt)\phi(g_t) one may obtain a similar second-moment expansion with an additional O(1/Bt2)O(1/B_t^2) term, but the paper does not provide optimizer-specific conditions or a full analysis for AdamW or Muon, whose updates depend on historical optimizer states. The empirical evidence with these optimizers is valuable, but it does not fully justify statements that the schedule is optimizer-independent in general. There are also several practical issues that are underexplored. The continuous optimum can produce extreme endpoint behavior: for cosine decay the optimal batch size approaches zero, while for constant learning rate it grows very large near the end of training. The paper does not clearly specify how these schedules are discretized, rounded, or constrained by feasible minimum/maximum global batch sizes in the experiments.
Questions for authors
  1. Could the authors measure XtX_t, ∣Egt∣2| \mathbb{E}g_t|^2, or an empirical critical-batch/noise-scale proxy throughout representative LLM training runs? How constant is Xt/Gt2X_t/G_t^2 in the regime where Eq. (2.5) is claimed to be accurate? If XX is replaced by a time-varying XtX_t, the same variational argument appears to yield approximately Bt∝btXtB_t\propto b_t\sqrt{X_t}. Does this materially change the proposed schedule?

  2. Can the authors provide a more precise theoretical statement for AdamW or Muon? In particular, what assumptions on the state-dependent mapping ϕt(gt)\phi_t(g_t) are needed for the claimed G2+X/Bt+O(1/Bt2)G^2+X/B_t+O(1/B_t^2) form, and under what conditions does the resulting optimal schedule remain independent of optimizer state?

  3. Could the authors report actual wall-clock time, throughput, and hardware utilization for dynamic versus static batch sizes? A direct comparison with a recent principled dynamic-batch method such as Seesaw under matched model, data, FLOPs, and hardware would help establish whether the proposed compute-efficiency gains translate into systems-level speedups.

  4. Appendix C.3 shows approximately 2.52.5--2.62.6% improvement under FP32 and mixed FP32/BF16, but only around 0.10.1% under pure BF16. Could the authors explain why the benefit nearly disappears in this setting? Does this suggest that the mechanism depends on numerical precision, and what should be expected for common BF16/FP8 large-scale training pipelines?

4 comments

We thank the reviewer for thoroughly reading our paper and for the valuable suggestions! Bellow are our responses. Please kindly raise the score if we have addressed your questions.

Summary

We explicitly require positive learning rate and delete Eq. (2.6) which diverges when learning rate is zero. We directly connect Eq. (2.5) to (3.1) and add Appendix A.3 to justify. We add wall-clock time in Figure 1 to demonstrate the advantage of our batch size schedule. We significantly extend Appendix A.2 to apply our method to adaptive gradient descent like AdamW and Muon. All new content is presented by blue text.

Weakness 1. The central "optimal" batch-size schedule relies on treating the gradient-noise term as time-independent, and it is unclear how robust the main conclusion is when this assumption is relaxed.

A common upper bound allows the actual covariance to vary throughout training. Using a fixed fitted coefficient in the loss model is a separate approximation, whose usefulness is supported by the reported 6–15% compute-efficiency gains over static batching. For prescribed time-varying bounds held fixed across candidate schedules, the continuous optimizer becomes B∗(t)∝b(t)X(t)B^*(t)\propto b(t)\sqrt{X(t)}. Our answer to Question 1 gives the formulas and the coefficient-stability diagnostic.

Weakness 2. The theoretical and practical generality claims extend beyond what is currently justified, particularly for adaptive optimizers and real training systems. The main theorem is derived for convex objectives and SGD. (...) but the paper does not provide optimizer-specific conditions or a full analysis for AdamW or Muon, whose updates depend on historical optimizer states.

We use convex SGD to obtain an analytically solvable allocation model. Wen et al. [1] provide precedent for explaining training through a structured landscape, while our fits and scheduling experiments assess the usefulness of our model in the studied settings. The empirical results demonstrate the effectiveness of the resulting schedule with AdamW and Muon.

For adaptive optimizers, Appendix A.2, Remark 2 identifies explicit bounded-curvature assumptions preserving the Gϕ2+Xϕ/BtG_\phi^2+X_\phi/B_t moment bound. Applying this result to a loss guarantee additionally requires the stated mean-update condition. Our answer to Question 2 gives the conditional formulation that accounts for optimizer history.

Question 1. Could the authors measure XtX_t, ∥Egt∥2\|\mathbb{E}g_t\|^2, or an empirical critical-batch/noise-scale proxy throughout representative LLM training runs? How constant is Xt/Gt2X_t/G_t^2 in the regime where Eq. (2.5) is claimed to be accurate? If XX is replaced by a time-varying XtX_t, the same variational argument appears to yield approximately Bt∝btXtB_t\propto b_t\sqrt{X_t}. Does this materially change the proposed schedule?

Unfortunately measuring XtX_t, EgtEg_t is (1) very noisy and (2) not particularly useful in the representative LLM training runs. For models with billions of parameters, even large batch size with millions of samples cannot measure these quantities reliably. In addition, for LLM training, we need adaptive gradient methods wt+1=wt−ηtϕ(gt)w_{t+1}=w_t-\eta_t \phi(g_t) that follow Appendix A.3, instead of Eq. (2.5). The formula is the same as Eq. (2.5) if we map [L⋆,MϕD,MϕGϕ,MϕXϕ][L_\star,M_\phi D,M_\phi G_\phi, M_\phi X_\phi] to [L⋆,D,G,X][L_\star,D,G, X], but the notations mean different quantities. Hence we need to measure pre-conditioned gradient norm and covariance, which is quite difficult and out of scope of this rebuttal.

For prescribed positive X(t)X(t), held fixed and valid uniformly across candidate schedules, and a finite positive normalizing integral,

\frac{K\,b(t)\sqrt{X(t)}}{\int_0^T b(s)\sqrt{X(s)}\,ds}, \qquad b(t)=\frac{\eta(t)}{\sqrt{\int_t^T\eta(s)\,ds}}.$$ The finite version is $B_t^*\propto\sqrt{a_tX_t}$, with $a_t=\eta_t^2/\sum_{k>t}\eta_k$ for $t<T$ and $a_T=\eta_T$. Whether this materially changes the objective depends on **the variation of $\sqrt X$ under the original allocation**. Let $\rho=b/\int b$, $B_0=K\rho$, and $J_X(B)=\int b^2X/B$. If $X>0$ almost everywhere and $\mathbb E_\rho X<\infty$, then $$\frac{J_X(B_0)}{J_X(B_X^*)} =1+\frac{\operatorname{Var}_\rho(\sqrt X)} {(\mathbb E_\rho\sqrt X)^2}.$$ Thus constant $X$ leaves the schedule unchanged, while variation gives a quantified penalty within the allocation model. The constant-coefficient schedule already improves the reported baselines. Our planned retrospective fits assess coefficient stability; **using online noise estimates to adapt the schedule** remains a future direction.

Question 2. Can the authors provide a more precise theoretical statement for AdamW or Muon? In particular, what assumptions on the state-dependent mapping ϕt(gt)\phi_t(g_t) are needed for the claimed G2+X/Bt+O(1/Bt2)G^2+X/B_t+\mathcal{O}(1/B_t^2) form, and under what conditions does the resulting optimal schedule remain independent of optimizer state?

Appendix A.2, Remark 2 replaces the informal higher-order expansion with a nonasymptotic 1/Bt1/B_t upper bound under explicit regularity assumptions. Below we state its conditional form to account for optimizer history and identify the additional assumption needed for a loss guarantee.

Let Ft−1\mathcal F_{t-1} contain the pre-batch history, including the chosen BtB_t, set μt=E[gt∣Ft−1]\mu_t=\mathbb E[g_t\mid\mathcal F_{t-1}], and define ht(g)=∥ϕt(g)∥2h_t(g)=\|\phi_t(g)\|^2. Assume, uniformly over relevant histories, candidate schedules, and gradient segments,

\le\frac{X}{B_t}, \qquad h_t(\mu_t)\le G_\phi^2, \qquad \|\nabla^2h_t(g)\|_{\mathrm{op}}\le C_\phi.$$ Taylor's inequality and the zero conditional mean of $g_t-\mu_t$ then give **the bound directly, without an unspecified remainder**: $$\mathbb E[\|\phi_t(g_t)\|^2\mid\mathcal F_{t-1}] \le G_\phi^2+\frac{C_\phi X}{2B_t} =G_\phi^2+\frac{X_\phi}{B_t}.$$ Conditioning fixes historical state while **retaining the current gradient's effect on the update**. The map includes Adam's moment updates and normalization, or Muon's momentum, normalization, and finite Newton–Schulz iterations. The derivative bounds concern their real-arithmetic maps on regular operating regions. Applying the moment bound to a loss guarantee also requires the mean-update condition in Remark 2 to hold conditionally on the pre-batch history, with a common deterministic $M_\phi>0$. The normalized allocation is independent of optimizer state when **the coefficients are fixed bounds shared across time and candidate batch schedules**: a common positive $X_\phi$ cancels from normalization. > Question 3. Could the authors report actual wall-clock time, throughput, and hardware utilization for dynamic versus static batch sizes? A direct comparison with a recent principled dynamic-batch method such as Seesaw under matched model, data, FLOPs, and hardware would help establish whether the proposed compute-efficiency gains translate into systems-level speedups. Figure 1 in the revised draft reports **shorter complete-run times on the tested hardware**. On eight H200 GPUs, static/dynamic-baseline/large-batch-dynamic runs took 23.2/22.0/18.1 hours for Llama3 and 23.1/18.1/17.4 hours for Qwen3 MoE. Our primary objective is loss at a fixed compute budget; these runtime measurements complement that result by showing shorter total training times in the reported configurations. We appreciate the suggestion to compare with Seesaw. Seesaw jointly adjusts LR and batch size to reduce serial steps [2], whereas our allocation uses **a prescribed LR schedule**. A controlled comparison requires matched model, data, FLOPs, and hardware, together with retuning, and is outside the scope of this rebuttal. Our scaling-law experiments demonstrate 6–15% compute-efficiency gains using the same LR and weight decay tuned for static batching. > Question 4. Appendix C.3 shows approximately 2.5 -- 2.6 improvement under FP32 and mixed FP32/BF16, but only around 0.1 under pure BF16. Could the authors explain why the benefit nearly disappears in this setting? Does this suggest that the mechanism depends on numerical precision, and what should be expected for common BF16/FP8 large-scale training pipelines? Figure 12 reports relative test-perplexity changes of −2.6%, −2.5%, and +0.1%, respectively: **the tested mixed-precision pipeline retains the benefit, while pure BF16 shows essentially none**. A plausible explanation is that rounding of small updates or optimizer-state computations can obscure the sampling-noise effect described by the loss model. The results show that **the observed benefit depends on the precision configuration**. BF16 pipelines differ in weight storage, accumulation, optimizer states, and update arithmetic, so the tested mixed-precision result is more informative than the datatype label alone. Understanding how these arithmetic choices interact with batch scheduling is a useful direction for extending the method to lower-precision training. ### References [1] Wen, Kaiyue, et al. "Understanding warmup-stable-decay learning rates: A river valley loss landscape perspective." arXiv preprint arXiv:2410.05192 (2024). [2] Alexandru Meterez, Depen Morwani, Jingfeng Wu, Costin-Andrei Oncescu, Cengiz Pehlevan, and Sham Kakade. *[Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Scheduling](https://arxiv.org/abs/2510.14717v2)*. arXiv:2510.14717, 2026 revision.

Dear Reviewer kiUG,

Thank you again for your constructive feedback. We have provided detailed responses to your questions and concerns, along with additional clarifications and revisions.

As the discussion period is approaching its end, we would greatly appreciate it if you could take a look at our responses and let us know whether any concerns remain. We would be happy to provide further clarification.

Thank you for your time and consideration!

Reviewer_kiUGOpenReview ↗

Response to Submission26827

Thank you for your detailed response. Your clarifications have addressed most of my concerns. After carefully considering your rebuttal, I have decided to maintain my original rating.

Reviewer_55BaOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the joint effects of learning-rate and batch-size schedules on deep learning training dynamics. Building on stochastic convex optimization, the authors derive a loss characterization involving both schedules and obtain a closed-form batch-size schedule that minimizes the resulting objective under a fixed data or compute budget. The proposed schedule depends on the shape of the learning-rate schedule but not its peak value. Combined with learning-rate and weight-decay scaling rules, the method demonstrates improved training efficiency across dense language models, MoE architectures, vision-language models, and mathematical fine-tuning.

Critical strengths

1.The paper addresses a practically important problem by treating batch size as a dynamically scheduled hyperparameter rather than a fixed constant. Formulating batch-size scheduling as a constrained resource allocation problem provides a useful perspective on the interaction between learning rate, gradient noise, and training efficiency. 2.The proposed batch-size schedule has a simple and interpretable closed-form solution. Its dependence on the learning-rate trajectory, rather than the peak learning rate, makes the approach convenient to apply and potentially transferable across different training configurations. 3.The empirical evaluation covers different learning-rate schedules, model scales, dense and MoE architectures, and both pre-training and fine-tuning. The improvements over static batch-size baselines provide encouraging evidence for the practical usefulness of dynamic batch sizing.

Critical weaknesses

1.The theoretical optimality of the proposed schedule does not directly establish its optimality for practical deep learning. The derivation relies on a convex optimization bound with a time-independent upper bound on gradient covariance. Equation (2.5) further approximates this bound as an equality, primarily supported by regression fits to observed training trajectories in Figure 2. However, accurate fitting of observed trajectories does not establish predictive accuracy for unseen batch-size schedules. Moreover, gradient covariance may change throughout training, potentially altering the optimal batch-size allocation. The experiments support the effectiveness of the proposed schedule, but provide limited evidence that it is close to optimal for non-convex networks trained with adaptive optimizers. 2.The comparisons do not sufficiently establish the advantage of the theoretically derived schedule over existing dynamic batch-size strategies. The main experiments primarily compare the proposed method against static batch-size baselines. However, Appendix E documents that several large-scale models already employ heuristic increasing or stepwise batch-size schedules. Without direct comparisons against these strategies under matched training budgets and learning-rate schedules, it remains unclear how much of the reported improvement comes from dynamic batch sizing itself rather than the specific analytical form of the proposed schedule. 3.The treatment of constant learning rates and practical batch-size constraints requires further clarification. Equation (2.6), which motivates the subsequent optimization, assumes that the terminal learning rate is zero, whereas Table 2 and Figure 4 also consider constant learning rates. In this setting, the terminal-gradient contribution cannot be directly discarded, and the continuous optimal batch-size expression diverges near the training endpoint. The paper does not sufficiently clarify how the endpoint is handled in the derivation and implementation, or how finite batch-size constraints affect the claimed optimality. 4.The practical efficiency and robustness claims require stronger empirical support. The reported 6–15% compute-efficiency improvements are primarily based on FLOP-oriented comparisons rather than end-to-end training time. Dynamic batch sizes may affect hardware utilization, memory requirements, and communication efficiency, which are not fully captured by the theoretical compute objective. Furthermore, Appendix C.3 shows that the advantage of dynamic batch sizing disappears under pure BF16 training. The results also lack systematic uncertainty estimates across independent runs, leaving the robustness of the improvements across training configurations insufficiently established.

Questions for authors

1.Can the coefficients fitted in Figure 2 using one set of learning-rate and batch-size schedules accurately predict the losses of unseen schedules? How sensitive is the proposed optimal schedule to time-varying gradient covariance? 2.How does the proposed schedule compare against tuned heuristic dynamic batch-size strategies, such as linear growth or stepwise doubling, under identical token budgets, training horizons, and learning-rate schedules? How much improvement remains beyond these baselines? 3.How is the constant-learning-rate case reconciled with the zero-terminal-learning-rate assumption in Eq. (2.6)? What constraints are imposed on minimum and maximum batch sizes during implementation, and do these constraints affect the derived optimality? 4.At a matched target validation loss, how do dynamic and static batch-size schedules compare in actual GPU-hours, training throughput, and peak memory usage? Can the authors explain the disappearance of the improvement under pure BF16 training and provide results across multiple random seeds?

6 comments

We thank the reviewer for thoroughly reading our paper and for the valuable suggestions! Bellow are our responses. Please kindly raise the score if we have addressed your questions.

Summary

We explicitly require positive learning rate and delete Eq. (2.6) which diverges when learning rate is zero. We directly connect Eq. (2.5) to (3.1) and add Appendix A.3 to justify. We add wall-clock time in Figure 1 to demonstrate the advantage of our batch size schedule. We significantly extend Appendix A.2 to apply our method to adaptive gradient descent like AdamW and Muon. All new content is presented by blue text.

Weakness 1

Weakness 1. The theoretical optimality of the proposed schedule does not directly establish its optimality for practical deep learning. The derivation relies on a convex optimization bound with a time-independent upper bound on gradient covariance. Equation (2.5) further approximates this bound as an equality, primarily supported by regression fits to observed training trajectories in Figure 2. However, accurate fitting of observed trajectories does not establish predictive accuracy for unseen batch-size schedules. Moreover, gradient covariance may change throughout training, potentially altering the optimal batch-size allocation. The experiments support the effectiveness of the proposed schedule, but provide limited evidence that it is close to optimal for non-convex networks trained with adaptive optimizers.

The theoretical optimality of the proposed schedule does not directly establish its optimality for practical deep learning.

Thank you for bringing up the distinction between surrogate optimality and practical optimality. We agree that the theoretical result does not directly establish optimality for practical deep learning. Specifically, Theorem 3 establishes optimality for the allocation objective in Eq. (3.3), for a prescribed LR schedule, horizon, and budget. The practical value of this analytical prescription is then evaluated through the training experiments.

The derivation relies on a convex optimization bound with a time-independent upper bound on gradient covariance.

Yes, the derivation starts from a convex-SGD bound. In that bound, a constant upper bound does not require constant gradient covariance: the covariance may vary throughout training while remaining below the bound. Treating the fitted coefficient as constant in the empirical loss model is a separate approximation, whose sensitivity we address below.

Convex analysis also has empirical precedent for guiding deep-learning schedules. Defazio et al. derive LR schedules from optimization theory and evaluate them on deep-learning tasks [1]; Schaipp et al. study agreement between convex bounds and large-model LR scheduling [2]. These works support using such analysis to design schedules, without establishing that neural-network objectives are globally convex.

Equation (2.5) further approximates this bound as an equality, primarily supported by regression fits to observed training trajectories in Figure 2.

For nonconvex networks and adaptive optimizers, we use Eq. (2.5) as an empirically evaluated loss model derived from the convex-SGD bound. Figure 2 supports its agreement with the observed trajectories. We do not interpret this fit as proving that the theoretical bound is exact for arbitrary nonconvex training.

However, accurate fitting of observed trajectories does not establish predictive accuracy for unseen batch-size schedules.

We agree that Figure 2 does not establish this prediction property. However, constructing the proposed schedule does not require transferring fitted coefficients to an unseen run. Under the fixed-coefficient model, the L∗,D2,G2L_*,D^2,G^2 terms are independent of batch allocation, and fixed positive XX cancels from the normalized optimizer. The LR schedule and budget therefore determine the batch schedule analytically; predicting unseen loss curves with shared fitted coefficients is a separate question.

Moreover, gradient covariance may change throughout training, potentially altering the optimal batch-size allocation.

For prescribed time-varying XtX_t, the allocation acquires a Xt\sqrt{X_t} factor, as shown in the finite and continuous formulas in our answer to Question 1. The constant-coefficient schedule already improves the reported main baselines. We will visualize fitted effective coefficients over existing runs to assess their stability; using online estimates of XtX_t to adapt the schedule remains a further direction.

The experiments support the effectiveness of the proposed schedule, but provide limited evidence that it is close to optimal for non-convex networks trained with adaptive optimizers.

The experiments establish effectiveness against the tested baselines, including the 6–15% compute-efficiency gains in Figure 5. We do not infer a quantified distance to the best possible nonconvex training schedule, which is largely under-explored in the literature too. Our contribution combines an analytically optimal allocation within the stated model with direct evidence that the resulting schedule improves the reported training comparisons.

Weakness 2

Weakness 2. The comparisons do not sufficiently establish the advantage of the theoretically derived schedule over existing dynamic batch-size strategies. The main experiments primarily compare the proposed method against static batch-size baselines. However, Appendix E documents that several large-scale models already employ heuristic increasing or stepwise batch-size schedules. Without direct comparisons against these strategies under matched training budgets and learning-rate schedules, it remains unclear how much of the reported improvement comes from dynamic batch sizing itself rather than the specific analytical form of the proposed schedule.

The comparisons do not sufficiently establish the advantage of the theoretically derived schedule over existing dynamic batch-size strategies. The main experiments primarily compare the proposed method against static batch-size baselines.

The reported gains are relative to matched static-batch baselines. In the scaling-law experiments, the dynamic schedule uses the same LR and weight decay tuned for static batching, yielding 6–15% compute-efficiency gains without retuning these hyperparameters. The analytical result also supplies a specific schedule for each LR choice: linear decay gives constant batching, while cosine and WSD give different allocations, without additional tuning of the batch-schedule shape.

However, Appendix E documents that several large-scale models already employ heuristic increasing or stepwise batch-size schedules.

Appendix E documents existing practice and motivates studying batch schedules systematically. Those reports use different training recipes, including different data and potentially different sequence-length schedules. A controlled comparison requires matching these conditions and retuning each alternative, which is beyond the scope of this rebuttal.

Without direct comparisons against these strategies under matched training budgets and learning-rate schedules, it remains unclear how much of the reported improvement comes from dynamic batch sizing itself rather than the specific analytical form of the proposed schedule.

We appreciate the value of this comparison. Reproducing and retuning those schedules under common model, data, LR, sequence-length, horizon, and budget conditions is outside the scope of this rebuttal. Our answer to Question 2 provides an exact comparison within the allocation objective, explaining the role of the analytical shape, while the experiments establish its benefit over matched static batching.

Weakness 3

Weakness 3. The treatment of constant learning rates and practical batch-size constraints requires further clarification. Equation (2.6), which motivates the subsequent optimization, assumes that the terminal learning rate is zero, whereas Table 2 and Figure 4 also consider constant learning rates. In this setting, the terminal-gradient contribution cannot be directly discarded, and the continuous optimal batch-size expression diverges near the training endpoint. The paper does not sufficiently clarify how the endpoint is handled in the derivation and implementation, or how finite batch-size constraints affect the claimed optimality.

See our response to Question 3.

Weakness 4

Weakness 4. The practical efficiency and robustness claims require stronger empirical support. The reported 6–15% compute-efficiency improvements are primarily based on FLOP-oriented comparisons rather than end-to-end training time. Dynamic batch sizes may affect hardware utilization, memory requirements, and communication efficiency, which are not fully captured by the theoretical compute objective. Furthermore, Appendix C.3 shows that the advantage of dynamic batch sizing disappears under pure BF16 training. The results also lack systematic uncertainty estimates across independent runs, leaving the robustness of the improvements across training configurations insufficiently established.

Our primary objective is loss at a fixed token/FLOP budget, consistent with the compute-budget formulation in previous scaling-law studies [3]. The 6–15% result measures this objective. Our answer to Question 4 provides the existing-run timings, discusses precision dependence, and describes the ongoing multi-seed evaluations.

Question 1

Question 1. Can the coefficients fitted in Figure 2 using one set of learning-rate and batch-size schedules accurately predict the losses of unseen schedules? How sensitive is the proposed optimal schedule to time-varying gradient covariance?

Figure 2 does not establish held-out prediction with shared coefficients. Crucially, the schedule requires no fitted-coefficient transfer: batch-independent terms drop out, and fixed positive XX cancels upon normalization. Extrapolating the loss model is therefore a separate validation question, not a prerequisite for computing the proposed schedule.

For prescribed time-varying gradient covariance, assume E∥gt∥2≤Gt2+Xt/Bt\mathbb E\|g_t\|^2\le G_t^2+X_t/B_t, with finite bounds held fixed and valid uniformly across candidate schedules. The continuous allocation has the same square-root noise weighting. With b(t)=η(t)/∫tTη(s) dsb(t)=\eta(t)/\sqrt{\int_t^T\eta(s)\,ds} and a finite positive normalizer, BX∗(t)=K b(t)X(t)∫0Tb(s)X(s) ds.B_X^*(t)=\frac{K\,b(t)\sqrt{X(t)}}{\int_0^T b(s)\sqrt{X(s)}\,ds}. Notice that prescribed Gt2G_t^2 changes the loss prediction but not the allocation.

Question 2

Question 2. How does the proposed schedule compare against tuned heuristic dynamic batch-size strategies, such as linear growth or stepwise doubling, under identical token budgets, training horizons, and learning-rate schedules? How much improvement remains beyond these baselines?

A controlled empirical comparison requires matching the training conditions and retuning each heuristic, which is outside the scope of this rebuttal. Within Eq. (3.3), we can give an exact comparison of allocation objectives. Let J(B)=∫0Tb(t)2/B(t) dtJ(B)=\int_0^T b(t)^2/B(t)\,dt, Z=∫0Tb(t) dtZ=\int_0^T b(t)\,dt, and B∗=Kb/ZB^*=Kb/Z. Any positive feasible allocation with finite objective satisfies

=\frac{Z^2}{K^2}\int_0^T\frac{(B(t)-B^*(t))^2}{B(t)}\,dt \ge0.$$ This identifies the cost of a different schedule within the model without tuning the proposed shape. The empirical result is **6–15% compute-efficiency improvement over the static baselines in the scaling-law experiments**, using the same LR and weight decay tuned for those baselines. ### Question 3 > Question 3. How is the constant-learning-rate case reconciled with the zero-terminal-learning-rate assumption in Eq. (2.6)? What constraints are imposed on minimum and maximum batch sizes during implementation, and do these constraints affect the derived optimality? To clear the confusion, we have removed Eq. (2.6) which was a sub-case of Eq. (2.5). In fact, we can directly connect Eq. (2.5) to Eq. (3.1) by showing the difference of equations vanishes as T goes to infinity. See Appendix A.3 for examples of linearly-decaying-learning-rate. We don't impose constraints on minimum and maximum batch size during implementation, except that we round the batch size to multiples of per-GPU microbatch. As microbatch increases, the batch size schedule is more coarse (and slightly deviated from optimal), which results in zigzag in loss curves, such as Figure 4. ### Question 4 > Question 4. At a matched target validation loss, how do dynamic and static batch-size schedules compare in actual GPU-hours, training throughput, and peak memory usage? Can the authors explain the disappearance of the improvement under pure BF16 training and provide results across multiple random seeds? Figure 1 reports **complete-run measurements** on eight H200 GPUs, rather than matched-target timings. GPU-hours equal eight times elapsed hours: | Model | Configuration | Hours | GPU-hours | |---|---|---:|---:| | Llama3 | Static | 23.2 | 185.6 | | Llama3 | Dynamic baseline | 22.0 | 176.0 | | Llama3 | Dynamic, large batch | 18.1 | 144.8 | | Qwen3 MoE | Static | 23.1 | 184.8 | | Qwen3 MoE | Dynamic baseline | 18.1 | 144.8 | | Qwen3 MoE | Dynamic, large batch | 17.4 | 139.2 | Figure 12 reports relative test-perplexity changes of −2.6% in FP32, **−2.5% in mixed FP32/BF16**, and +0.1% in pure BF16. Rounding of small updates or optimizer-state computations could obscure the sampling-noise benefit, but the current ablation does not isolate that mechanism. It still supports a precision-dependent empirical conclusion, including a gain in the tested mixed-precision training. We are working on **additional evaluations across independent random seeds** and will include the results in the camera-ready version. ### References [1] Aaron Defazio, Ashok Cutkosky, Harsh Mehta, and Konstantin Mishchenko. *[Optimal Linear Decay Learning Rate Schedules and Further Refinements](https://arxiv.org/abs/2310.07831v2).* arXiv:2310.07831, 2024 revision. [2] Fabian Schaipp, Alexander Hägele, Adrien Taylor, Umut Şimşekli, and Francis Bach. *[The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Training](https://arxiv.org/abs/2501.18965).* 2025. [3] Jordan Hoffmann et al. *[Training Compute-Optimal Large Language Models](https://arxiv.org/abs/2203.15556).* 2022.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.