- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper derives a batch-size schedule for a prescribed learning-rate trajectory, training horizon, and token budget. Starting from a convex-SGD bound and a gradient-variance model, it constructs a continuous loss surrogate and obtains a formula for batchsize. The schedule is combined with training-horizon scaling rules for learning rate and weight decay. Experiments cover dense and MoE language-model pretraining, vision-language training, and mathematical fine-tuning, reporting approximately 6–15% improvements in FLOP-based efficiency over static batching. The contribution is a specific formulation of batch allocation: distribute a fixed sample budget across a fixed number of updates to minimize a predicted final loss. This differs conceptually from Seesaw’s objective of preserving reference performance while reducing sequential updates. The proposed formulation is useful, but its mathematical justification and comparative evaluation need strengthening.
Critical strengths
1 A simple and actionable scheduling rule.
The method produces explicit schedules without online gradient-noise estimation or additional task-specific fitting. Its predictions depend meaningfully on the learning-rate shape: constant learning rate leads to increasing batches, linear decay leads to constant batches, and WSD leads to increasing batches during the stable phase followed by constant batches during cooldown.
2 Meaningful empirical breadth.
The experiments include dense models up to approximately 7B parameters, MoE models, eight language benchmarks, and two post-training settings. Downstream improvements provide evidence beyond training loss: Table 1 reports gains on eight of eight tasks for the dense model and seven of eight for the MoE model.
Critical weaknesses
1 The discrete-to-continuous derivation contains unresolved endpoint inconsistencies.
- The optimality claim is stronger than the demonstrated result.
Theorem 3 correctly solves the relaxed continuous allocation problem. However, minimizing a surrogate derived from an upper bound does not establish that the resulting schedule minimizes actual training loss.
Questions for authors
In Eq. (2.6), setting the final learning rate to zero makes the penultimate term’s denominator zero when applied that to the last training iteration. How are the discrete and continuous endpoint issues resolved, and do the corrected expressions preserve the proposed schedule and convergence claims?
5 comments
Dear Reviewer GsC4,
Thank you again for your constructive feedback. We have provided detailed responses to your questions and concerns, along with additional clarifications and revisions.
As the discussion period is approaching its end, we would greatly appreciate it if you could take a look at our responses and let us know whether any concerns remain. We would be happy to provide further clarification.
Thank you for your time and consideration!
Hi authors, thanks for the feedback. Removing (2.6) and connecting (2.5) to (3.1) only moves the singularity point but does not eliminate it. Particularly (3.1) seems to diverge for constant LR scheduling. And if you look at Table 3, L_dynamic - L_static would also become infinite. So the whole modeling seems have some fundamental inconsistencies to be addressed. Please check it carefully and see if they could be fixed.
We thank the reviewer for thoroughly reading our paper and for the valuable suggestions! Bellow are our responses. Please kindly raise the score if we have addressed your questions.
Summary
We explicitly require positive learning rate and delete Eq. (2.6) which diverges when learning rate is zero. We directly connect Eq. (2.5) to (3.1) and add Appendix A.3 to justify. We add wall-clock time in Figure 1 to demonstrate the advantage of our batch size schedule. We significantly extend Appendix A.2 to apply our method to adaptive gradient descent like AdamW and Muon. All new content is presented by blue text.
Response: see our respose to the question section below.
Thank you for making this distinction. Theorem 3 establishes optimality for the allocation objective in Eq. (3.3), for a prescribed LR schedule, horizon, and budget. It yields an analytical schedule whose shape requires no additional fitting: the batch-independent terms drop out, and the fixed positive noise coefficient cancels upon normalization. The experiments demonstrate the practical value of this prescription through 6–15% FLOP-efficiency gains against static batching.
We do not interpret this fit as proving that the theoretical bound is exact for arbitrary nonconvex training. We believe that FLOP-efficiency gains are enough to show the upper bound surrogate's effectiveness.
The reviewer is correct about the original substitution. We deleted Eq. (2.6); at every finite horizon, Eq. (2.5) retains positive discrete learning rates and the terminal contribution ηT(G2+X/BT)/2. This makes each finite tail denominator positive.
In fact, we can directly connect Eq. (2.5) to Eq. (3.1) by showing the difference of equations vanishes as T goes to infinity. For the linear example in Appendix A.3, the precise comparison is on ηtdisc=T−1/2(1−T+1t), the displayed calculation gives ∣FTdisc−FTcont∣=O(logT/T3/2), including the terminal contribution. The final discrete LR remains positive at every finite horizon. This shows a slight discrepancy to Table 2, where we use T−1/2(1−Tt) for the sake of simplicity. This is justifiable as Eq. (3.1) with either linear schedules is asymptotically equivalent.