On fast hyperparameter transfer under fixed and diverging time horizons

Hyperparameter (HP) transfer, which avoid expensive tuning on large neural networks by reusing values tuned on small models, can substantially reduce large-scale pretraining costs. Yet existing evidence is largely qualitative and does not precisely characterize the computational gain. We study this methodology through the framework of fast transfer ghosh2025understanding, which relates computational benefits to the convergence rates of the optimal loss and HPs with model scale $n$. We first prove a second-order Tensor Program master theorem for general neural computations at a fixed training horizon $T$. For fully connected polynomial networks trained by gradient descent under the Maximal Update Parameterization ($\mu$P), this yields a sharp $n^{-1/2}$ fluctuation of the loss and optimal HPs. While this result quantifies loss and HP convergence, we argue that fixed-$T$ asymptotics do not explain the computational benefit of $\mu$-transfer, as the leading centered loss fluctuations conflict with empirically observed scaling laws and loss monotonicity. This motivates us to investigate jointly diverging model width $n$ and training horizon $T$ in a minimal tractable setting: tuning the gradient descent learning rate $\eta$ in a power-law random features model. Using a resolvent expansion, we derive sharp loss asymptotics and HP convergence, and establish necessary and sufficient spectral conditions for fast transfer at width-dependent training horizons. We further discuss the scaling behaviors at the compute-optimal frontier and the effect of stochastic gradients.