- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper asks when transferring a learning rate from a small model yields a computational benefit, rather than merely a convergent sequence of optimal learning rates. At a fixed training horizon, it develops a second-order tensor-program analysis and obtains loss-curve and optimizer fluctuations for polynomial muP MLPs. It then explains why centered finite-width fluctuations can satisfy a fast-transfer rate criterion without a systematic benefit from increasing width. For jointly growing width and horizon, a power-law sketched linear-regression model separates centered fluctuations from positive capacity bias and yields conditions for transfer to outperform direct uniform-grid tuning under a specified compute budget. The paper also discusses compute-optimal schedules and illustrates a different mechanism for stochastic gradients.
Critical strengths
- The paper identifies a meaningful limitation of using optimal-HP convergence or the ratio of transfer loss to an absolute finite-width loss gap as evidence of computational benefit. The centered-fluctuation interpretation makes this distinction precise and provides a well-motivated reason to study the joint width-horizon limit. The proposed second-order tensor-program result could also be a valuable technical contribution beyond this application.
- The growing-horizon analysis connects the stability edge, changing curvature, capacity bias and tuning cost within one tractable model. The manuscript carefully distinguishes proportional from common horizons and acknowledges that the SGD mechanism differs from population GD. The synthetic figures include substantial seed/sketch variation and validation-loss diagnostics, which help make the asymptotic mechanisms concrete.
Critical weaknesses
-
The compute-optimal equality case needs a more explicit connection to the admissible tuning protocols. Definition 1 (p.4) requires widths and grid sizes to be fixed powers of the budget, whereas Theorem 4 (pp.8-9) includes beta = 3 alpha, an asymptotically smaller proxy n = o(m), and a common training horizon. At this boundary the target horizon is of order sqrt(m). For a strictly smaller-power proxy, its stability-edge fluctuations of order n^(-1/2) can exceed the infinite-width optimum's distance from the edge, of order (log T)/T in Proposition 1. It is therefore not clear from the main statement how the selected proxy rate remains safe for an independently initialized target with probability tending to one. A logarithmically separated proxy may address this comparison, but would require clarification of the fixed-power restriction. Please give the precise allocation and boundary argument, or delimit the statement accordingly. This concern is about the endpoint and definitions; it does not by itself challenge the strict-interior regimes.
-
The scope of the fixed-horizon nonlinear result should be stated consistently. Section 3.1 (p.4) describes coverage of GD-trained MLPs with pseudo-Lipschitz activations, while Theorem 2 (p.5) obtains the entire learning-rate curve through a finite polynomial expansion and explicitly assumes polynomial activation and loss. Figure 2 uses tanh. It would help to state whether an additional result extends the split-program reduction to this non-polynomial setting, or whether the tanh experiment is empirical evidence beyond the theorem. Similarly, the broad conclusion about fixed horizons should retain the positive fluctuation-variance and nondegenerate-optimum conditions of Corollary 1. Clear separation of these cases would make the theoretical contribution easier to assess without diminishing the value of the experiment.
Questions for authors
- At beta = 3 alpha under a common horizon, what are the exact proxy width and grid-size scalings in Theorem 4? Are slowly varying factors allowed in Definition 1, and how is the probability of selecting a rate above the target stability edge controlled?
- Does the supplementary tensor-program argument establish the loss-curve CLT for tanh or other non-polynomial activations? If so, which assumptions justify moving scalar-dependent training operations into the split-program representation?
- What is the boundary behavior at rho = 1 in Theorem 3? Equation (6) lists rho < 1 and rho > 1, while Assumption 1 allows equality; a concise statement of any logarithmic correction or exclusion would be useful.
5 comments
Scope of the constant- result
Thank you for pointing this out. Our intention in using the tanh activation in Figure 2 was to illustrate that the same rate characterization is expected to hold beyond the polynomial restriction; we now remark on this in lines 289–292 of the revision. We make the following clarifications.
- Our CLT (Theorem 1) holds for nonlinearities satisfying Assumption 2, which include tanh; polynomial activations are needed only for recasting GD training of an MLP as a split program (Proposition 9). This is because, after the first update, the inputs to the matrix multiplications depend on scalars such as the loss derivative, and only for polynomials can these scalar factors be pulled out exactly. We have revised the corresponding sentence in Section 3.1 to avoid confusion (lines 212–214).
- For tanh, the recasting is still possible, but it requires extra steps: one linearizes each scalar-dependent input around the deterministic limit of its scalars. Since the scalars fluctuate at scale and the activation has a bounded second derivative, the remainder changes every average only by , below the scale of the CLT, while the linear terms can be expressed by a split program. This gives the CLT of the loss at any fixed learning rate. It does not by itself give Corollary 1 (tuned loss and optimal learning rate), which also requires uniform control of the curve and its derivatives in ; we do not carry this out for tanh, since it would only lengthen the proof, and the polynomial case already yields the conclusions of Section 3. This is why the submitted version stated Theorem 2 for polynomial activations; for tanh, Figure 2(b,c) provides empirical evidence beyond our theorems. We have added Remark 6 to the appendix (lines 2448–2479) explaining the extension, and the caption of Figure 2 (line 281) now refers to it.
- Following your suggestion, the revision points to the conditions of Corollary 1 more explicitly when stating the positive-variance result (lines 088 and 268–269). These conditions are mild: by Theorem 2, for all but at most learning rates, under conditions at initialization that hold almost surely for nondegenerate data (together with ). A nondegenerate optimum is not even needed for the conclusion: degenerate optima are covered by Theorem 5. We state this assumption in the main text only to connect with the criterion of [GWB25], which is expressed in terms of the curvature ; the extension to degenerate optima requires more notation and is orthogonal to our message, so we defer it to the appendix.
Boundary case in Theorem 3
- We omitted from Equation (6) for brevity. It is covered in the appendix: Lemma 29 (Appendix I.3) gives . We have now added it to Equation (6) in the main text.
We would be happy to answer any follow-up questions in the discussion period.
References
[GWB25] Understanding the Mechanisms of Fast Hyperparameter Transfer
Thank you for the detailed response and revisions. These address my concerns, and I will maintain my original rating.
Thank you for the careful reading and constructive feedback. We address the technical concerns below. Line numbers refer to the revised version.
Compute-optimal allocation
(Continued in the next comment.)