We develop a finite-horizon statistical theory of dynamic mini-batching under warmup–stable–decay learning rates (WSD) for sketched linear regression with power-law spectral and source conditions. WSD induces two effective time scales: deterministic learning is governed by the full horizon γJ, whereas stochastic noise retains only a shorter terminal memory γJ/logJ. Although a dynamic batch schedule has one degree of freedom per update, its leading stochastic contribution to final risk collapses to an influence-weighted scalar summary: Beff for one-pass WSD-SGD and ρeff under data reuse. This compression also reveals a structural distinction. One-pass batching controls centered SGD variance, while multi-pass batching controls only fluctuation around the full-batch WSD-GD reference; approximation, GD bias, and GD variance remain schedule-independent at a fixed update horizon. We obtain matching one-pass variance bounds and nearly matching multi-pass fluctuation bounds. As a consequence, minimizing the influence-weighted stochastic terms yields the oracle allocation Bt∝κt.
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.Reviewer_s91ROpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary
This paper studies the dynamic batch size in linear regression, including SGD with and without data reuse. In both cases, the batch size affects the excess risk through the effective batch size. To maximize the effective batch size, influential iterates should receive larger batch sizes.
Critical strengths
The paper gives an insightful observation: Influential iterates should receives large batch sizes.
The presentation is overall easy to follow.
Critical weaknesses
The manuscript directly applies the learning rate safeguard in [1]. However, [1] mainly considers the geometric schedule. The main text of this manuscript does not discuss how the learning rate safeguard in [1] can be applied to the WSD schedule.
The analysis is restricted to the WSD schedule.
The Gaussian sketch concentration bounds in Lemmas K.4 and K.6 omit square roots.
The following statement is a bit confusing: Line 291-293: "Its spectral factor...": The term "spectral factor" is not defined; it is unclear how the spectral factor "counts" the directions.
Although the optimal batch size schedule is given in Corollary 3.1, it would be better if they can be expressed as a function of a, b, and γt rather than κt(T).
A discussion about the application of theoretical results to commonly-used batch size schedules is lacking.
My key concerns are items 1, 2, 5, and 6. I will raise my rating if they are all addressed in the rebuttal.
Questions for authors
The optimal batch size schedule in Corollary 3.1 does not seem directly related to the WSD schedule. Does it hold for general learning rate schedules?
This paper studies the effect of dynamic mini-batch schedules on the final prediction risk of SGD in a sketched linear regression model with power-law spectral and source conditions. The analysis considers both one-pass and multi-pass SGD under warmup–stable–decay (WSD) learning-rate schedules.
The main theoretical scaling laws characterize the stochastic error through an effective batch size Beff for one-pass SGD and an effective fluctuation factor ρeff for multi-pass SGD. Both quantities summarize the dynamic batch-size schedule Bt using influence weights κt, which depend on the learning-rate trajectory and the sketched covariance spectrum.
They establish matching bounds for the one-pass variance and nearly matching bounds for the multi-pass fluctuation term.
Critical strengths
1. Rigorous theoretical guarantees under a standard framework
Building on the established sketched linear-regression framework and standard assumptions, the paper derives matching upper and lower bounds for the one-pass variance and nearly matching bounds for the multi-pass fluctuation term, including the finite-population correction for sampling without replacement. These results provide rigorous theoretical guarantees for time-varying mini-batch schedules.
2. Discrete-time analysis of dynamic batch-size and learning-rate schedules
The paper provides a discrete-time characterization of how dynamic batch sizes and WSD learning rates jointly affect the final prediction risk. This extends the analysis of scheduling effects beyond continuous-time approximations [1] and provides a useful theoretical treatment of dynamic batching in discrete SGD.
Reference:
[1] Li et al. (2025), Functional scaling laws in kernel regression: Loss dynamics and learning rate schedules.
Critical weaknesses
1. Limited technical novelty
The technical contribution appears somewhat incremental relative to the existing sketched linear-regression framework of Lin et al. (2024, 2025). Much of the bias–variance analysis and fluctuation decomposition follows established techniques. The main technical challenge appears to be handling time-varying batch sizes. However, the paper seems to address this largely by incorporating the iteration-dependent noise factors 1/Bt or ρt into the existing covariance recursions, while capturing the learning-rate dependence through the influence weights κt. This extension appears to require relatively little additional analysis beyond the established framework, raising concerns about the extent of its technical novelty.
2. Concerns on theoretical results and insights
Although the paper considers WSD learning rates, the dependence on dynamic batch sizes in the resulting scaling laws remains expressed through the weighted sums ∑tκt/Bt and ∑tρtκt, which can accommodate general learning-rate schedules. Consequently, the specific role of WSD is not sufficiently explicit, and it remains unclear what additional scaling predictions arise from specializing the analysis to WSD.
The main qualitative insights also appear largely anticipated by existing functional scaling law results. Li et al. (2025) [1] already characterize learning-rate-dependent noise forgetting and the distinct effective time scales for signal learning and stochastic noise under WSD, while Wang et al. (2026) study influence-based optimal batch allocation. Thus, the proposed influence weights, schedule compression, and square-root allocation rule appear closely related to established principles. Although the one-pass versus multi-pass distinction is useful, the paper does not clearly identify substantially new qualitative predictions beyond these prior results.
Reference:
[1] Li et al. (2025), Functional scaling laws in kernel regression: Loss dynamics and learning rate schedules.
Questions for authors
1. Relationship to existing theoretical scaling laws literature.
Could the authors discuss [1–4] more thoroughly in the related-work section and clarify how the present results differ from these studies? In particular, [1] already characterizes general learning-rate and batch-size schedules through a noise-forgetting kernel. Moreover, [3,4] extend theoretical scaling laws to new dimensions such as precision and data recipe. They should be the relevant literature. A more systematic discussion would help clarify the novelty and positioning of the present work within the broader theoretical scaling-law literature.
2. Necessity of the WSD restriction.
Since the influence weights κt are defined for general time-varying learning rates, which parts of the main theoretical results fundamentally require the WSD structure? Could the influence-weighted risk bounds be established for general deterministic learning-rate schedules, with WSD-specific rates derived as a special case? More importantly, what additional theoretical insights or explicit scaling predictions does the WSD specialization provide beyond those obtainable under general learning-rate schedules?
References:
[1] Functional scaling laws in kernel regression: Loss dynamics and learning rate schedules.
[2] How Does Critical Batch Size Scale in Pre-training?
[3] Scaling Laws for Precision in High-Dimensional Linear Regression.
This paper studies dynamic batch-size scheduling under warmup–stable–decay learning rates in sketched linear regression. It characterizes final prediction risk in both one-pass and multi-pass settings using influence-weighted summaries of the batch schedule, and derives an oracle square-root allocation rule under a fixed update horizon and sample-processing budget.
Critical strengths
Joint consideration of learning-rate and batch-size schedules. The paper analyzes batch-size scheduling under a time-varying learning rate, accounting for how the current learning rate and subsequent optimization jointly determine each update’s contribution to final prediction risk.
Coverage of both one-pass and multi-pass training. The paper considers both fresh-data training and finite-data reuse. Its multi-pass analysis separates mini-batch-induced fluctuations from the statistical error of the full-batch GD reference, providing a useful framework for understanding what batching changes in this setting.
Critical weaknesses
1. Insufficient comparison with the Functional Scaling Law literature makes the incremental novelty difficult to assess.
The connections to the FSL series are substantially closer than the current discussion suggests. The following four points require detailed treatment in the main text.
(a) The two time scales under WSD have a direct precedent. In Functional Scaling Laws in Kernel Regression: Loss Dynamics and Learning Rate Schedules [1], Theorem 5.4 characterizes signal learning through the total intrinsic time T1+T2, while the stochastic term depends on the decay-phase intrinsic time T2. With constant-fraction stable and decay phases and a terminal learning rate equal to the peak rate divided by the training horizon, these quantities scale as γJ and γJ/logJ, respectively. This closely matches the two effective horizons emphasized here. The authors should explicitly compare the assumptions and guarantees, and explain what their discrete-time analysis adds.
(b) The logarithmic dependence is schedule-dependent and should be discussed alongside existing improvements.Optimal Learning-Rate Schedules under Functional Scaling Laws: Power Decay and Warmup–Stable–Decay [2] shows that appropriately chosen power-decay schedules remove logarithmic suboptimality in related one-pass problems, including rigorous last-iterate guarantees for discrete-time kernel SGD. Therefore, the present paper should clarify to what extent its logarithmic separation reflects the particular exponential-decay WSD schedule rather than an unavoidable property of learning and noise forgetting. It should also discuss whether alternative LR schedules improve the risk bounds and how this affects the motivation for optimizing batch sizes under the chosen schedule. This issue is separate from the additional logN gap in the multi-pass fluctuation bounds.
(c) The influence weights closely correspond to the established forgetting kernel. FSL already explicitly describes how previously injected noise survives subsequent optimization through a spectral forgetting kernel. In the present notation, a small-step approximation gives
Thus, “final-risk influence” appears to be a learning-rate-weighted discrete counterpart of the FSL forgetting kernel. The authors should explicitly establish this relationship and maintain terminological continuity, or explain the substantive distinction that warrants introducing a separate concept. This approximation does not establish equivalence of the full theorems, but it demonstrates a close conceptual connection.
(d) The square-root allocation principle is already explicit in prior optimal-batch work. Appendix A.3 of Fast Catch-Up, Late Switching: Optimal Batch Size Scheduling via Functional Scaling Laws [3] derives
B⋆(t)∝K(T−t)
under a fixed budget and handles batch-size bounds through clipping. The rule Bt⋆∝κt in Corollary 3.1 should therefore be compared directly with this result. Discussing [3] primarily through fast catch-up and late switching misses the most immediate mathematical connection. The authors should explain the additional contribution of time-varying learning rates, discrete updates, and finite-data reuse.
These connections do not negate the potential value of the new analysis, but the paper needs to distinguish previously established mechanisms and allocation principles from its new technical results.
2. The experiments provide limited evidence of practical relevance.
The experiments are numerical simulations of the assumed Gaussian linear-regression model. They are useful consistency checks, but do not establish that the insights transfer to practical large-scale training, where feature learning, adaptive optimization, and changing gradient-noise structure may alter the dynamics. Real-world experiments would strengthen the practical motivation.
Even within the synthetic setting, the evaluation focuses on stochastic error components rather than the resulting improvement in total prediction risk. It also does not directly test the claimed time-scale separation across training horizons or provide a targeted empirical demonstration of the one-pass/multi-pass distinction. These would be informative additions.
3. The allocation result solves a restricted optimization problem.
The square-root rule optimizes an influence-weighted stochastic proxy with both the update horizon and LR trajectory fixed. It does not jointly optimize training duration, learning rate, and batch sizes under a common compute budget. Since changing the update horizon also changes optimization bias, this distinction matters when interpreting the scheduling result. Moreover, the allocation uses population spectral information, and its robustness to estimated influence weights is not evaluated. The scope of the optimality claim and the gap between the oracle prescription and an implementable method should be clearer.
Questions for authors
What are the main new technical contributions relative to the FSL series? Please identify the proof obstacles and techniques needed beyond the existing forgetting-kernel framework and square-root allocation argument. A main-text comparison aligning assumptions, parameterization, budget constraints, and guarantees would be particularly helpful.
How should the logarithmic time-scale separation be interpreted in light of optimal LR scheduling? Which conclusions depend specifically on exponential-decay WSD? What changes under power-decay schedules of the type analyzed in [2], and which logarithmic factors are consequences of schedule choice versus limitations of the analysis?
What is the intuitive reason for the different role of batch size in one-pass and multi-pass training? The final-risk decompositions are mathematically clear, but the central message would benefit from more explanation. How does reusing a finite dataset change what gradient noise represents? Which aspects of the distinction are substantive consequences of data reuse, and which arise from centering around different reference trajectories? Please explain this beyond stating the final-risk theorems.
Can the experiments more directly test the proposed mechanisms? For example, can the authors vary the training horizon to test the two effective time scales, and separately measure the GD-reference error and mini-batch fluctuation under data reuse? Reporting total prediction risk would also clarify when reducing the stochastic component produces a meaningful overall benefit.
How should the scheduling baselines be interpreted relative to [3]? The simple early-small/late-large allocation is not the optimized FSL schedule. Can the authors provide an appropriately matched comparison, or explain why the different LR and horizon constraints prevent one? In particular, how much of the difference in preferred batch-schedule shape follows directly from using WSD instead of a constant learning rate?
Overall, given the insufficient discussion of closely related work and the resulting uncertainty about the paper’s incremental novelty, I currently lean toward weak rejection. However, I welcome discussion with the authors and am open to revising my assessment and score based on their response and corresponding revisions to the manuscript.
References
[1] Li et al. Functional Scaling Laws in Kernel Regression: Loss Dynamics and Learning Rate Schedules. https://arxiv.org/abs/2509.19189
[2] Li et al. Optimal Learning-Rate Schedules under Functional Scaling Laws: Power Decay and Warmup–Stable–Decay. https://arxiv.org/abs/2602.06797
[3] Wang et al. Fast Catch-Up, Late Switching: Optimal Batch Size Scheduling via Functional Scaling Laws. https://arxiv.org/abs/2602.14208
Thank you again for your detailed and constructive review.
As the discussion period is approaching its end very soon, we would like to kindly follow up on our previous response and revised manuscript, particularly the expanded FSL comparison, the general learning-rate results, and additional experiments.
We would greatly appreciate any further feedback on whether these revisions have addressed your main concerns. If any questions remain, we would be happy to provide further clarification.
Thank you very much for your time and consideration throughout the discussion. Again, we sincerely appreciate your previous constructive feedback, which has helped us improve the manuscript. We would also be happy to provide any further clarification if needed.
From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.
Dear Reviewer,
Thank you again for your detailed and constructive review.
As the discussion period is approaching its end very soon, we would like to kindly follow up on our previous response and revised manuscript, particularly the expanded FSL comparison, the general learning-rate results, and additional experiments.
We would greatly appreciate any further feedback on whether these revisions have addressed your main concerns. If any questions remain, we would be happy to provide further clarification.
Thank you for your time and consideration!