acceptodds
Under review as a conference paper at ICLR 2027

Fast Learning Rate Transfer in Shallow Linear Networks at Growing Training Horizons

Abstract

Hyperparameter transfer across model width can substantially reduce the cost of tuning large neural networks, but its behavior when the training horizon grows with width is not fully understood. Building on the framework of fast hyperparameter transfer ghosh2026understanding, which formalizes when transfer is effective, we investigate conditions that ensure fast transfer in the growing-horizon regime. Specifically, we study learning-rate transfer in a shallow linear network with a single trainable hidden matrix, trained by full-batch gradient descent. Under additional spectral assumptions, our main results are threefold. (i) We prove fast learning-rate transfer as whenever . (ii) We characterize the transfer rates through the finite-width perturbation scale, the first-order sensitivities of the loss and its learning-rate derivative to finite-width perturbations, and the local loss curvature. (iii) We derive limiting distributions for the optimal learning rate and optimized loss, governed by fluctuations associated with the extreme eigenvalues of the data Gram matrix. These results clarify how spectral structure and local loss sensitivities govern learning-rate transfer at growing horizons.

Primary area:
learning theory
Keywords:
hyperparameter transfer, learning rate transfer, maximal update parameterization, linear network

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_Lfj1OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies learning rate transfer across width in a linear MLP with a single trainable matrix and frozen input/output weights. Under some spectral assumptions, it establishes fast transfer and derives limiting distributions for the optimal learning rate and optimised loss. The analysis is sound and I have not found any major technical flaws. However, the contribution is very close to related prior work.

Critical strengths

The paper is well-written, and easy to follow. 
The analysis appears technically sound under the stated assumptions. The growing horizon is a strength since most muP papers consider a fixed horizon. Analysing the spectral view is also interesting.

Critical weaknesses

The main weaknesses is related to a significant overlap with prior work, and limited technical novelty.

  • Substantial overlap with Wen et al. (2026). That work already establishes fixed-horizon and growing-horizon analysis of learning rate transfer, and sufficient conditions for fast transfer. Both works reduce training to a quadratic objective involving the power of the covariance operator, etc. The main component that this paper adds is a random input layer. The paper should distinguish more sharply between the existing framework and its additional results.

  • Spectral view: Wen et al. analyse an infinite spectrum where the slow frequencies basically produce polynomial loss decay and (therefore) the optimal learning rate approaches the stability boundary. This paper assumes a fixed finite spectrum, which allows the exponential loss decay. The explicit growing-horizon CLT is an additional result, but the overall contribution is just a refinement of the prior techniques. Also, I’m not sure what additional insights it brings to neural-network training in this case.

Questions for authors

See weaknesses.

3 comments

Thank you for your valuable comments. As you pointed out, our work studies a setting closely related to Wen et al. (2026). However, our finite-spectrum setting allows us to obtain several additional insights.

First, we establish two counterexamples to fast transfer: (i) Proposition 3.8 demonstrates that fast transfer can fail even at a fixed training horizon due to the spectral structure of the training data, and (ii) the newly added Proposition H.2 shows that fast transfer can fail when T≍nT\asymp\sqrt{n}, even under our spectral assumptions. These counterexamples highlight that successful fast transfer depends not only on network architecture but also on the properties of the training data and the balance between the training horizon TT and width nn.

Second, by deriving explicit limiting distributions (Theorem 3.4), we characterize the directional sensitivities of the optimized loss and optimal learning rate to finite-width perturbations, revealing that these two quantities can respond differently to the same perturbation (Section 5).

Reviewer_Lfj1OpenReview ↗

Response

Thank you for your response.

I carefully went over the proofs and technical results in the paper, and I still believe that there is a substantial overlap with Wen et al. (2026) and the technical contributions are marginal. The proof techniques are mostly refinements of that previous work.

Thank you for taking another careful look at our work. We acknowledge that the overall proof strategy is closely related to Wen et al. (2026). However, our finite-spectrum setting leads to a different long-horizon behavior, where the optimal learning rate is determined by both spectral endpoints and remains strictly inside the stability region. We believe that this distinction, together with the counterexamples and explicit limiting distributions highlighted above, provides meaningful insights beyond the existing literature.

We appreciate your consideration.

Reviewer_bG1hOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies fast learning-rate transfer across width in a three-layer linear network with only the hidden layer trained. Under a spectral assumption on the data, the authors show that fast transfer holds when the training horizon grows as T=o(n)T = o(\sqrt{n}). They derive limiting distributions for the optimal learning rate and the optimized loss. They also give a fixed training-horizon counterexample in the setting where the data Gram matrix has a single positive eigenvalue and show that while the optimal learning rate converges, fast transfer fails.

Critical strengths

Limiting distributions and a counterexample. The authors derive limiting distributions of the optimal learning rate and the loss. This shows precisely how width-induced fluctuations affect both of them. They also give a fixed training-horizon counterexample in the setting where the data Gram matrix has a single positive eigenvalue and show that while the optimal learning rate converges, fast transfer fails.

Interpretable three-term analysis. The authors further analyzed the transfer rates through three training horizon-dependent quantities: the sensitivity of the loss to finite-width perturbations, the sensitivity of its learning-rate derivative, and the curvature in the learning rate. The time dependence in these terms also motivates the scaling T<nT < \sqrt{n}.

Critical weaknesses

The main weakness of this paper is the restricted setting and its consequences for realistic settings. The model is a three-layer linear network with only the hidden layer trained. Therefore, there is no feature learning in this model, which is the primary feature of parameterizations like μ\muP. The training steps T are also smaller than n\sqrt{n}, whereas in practice T≫nT \gg n. As a result, it's unclear what the implications are for realistic training.

Questions for authors
  • Can the authors explain the practical implications of their work?
1 comment

We thank the reviewer for the helpful comments.

Feature learning. We would like to briefly clarify the notion of feature learning. We agree that our model does not capture non-kernel representation learning, as its prediction dynamics reduce to gradient descent on fixed random features. However, in the original paper introducing μ\muP (Yang and Hu, 2021), feature learning is defined in terms of non-vanishing changes in hidden representations in the infinite-width limit. In fact, the authors explicitly note that a linear network with only its middle layer trained can exhibit such changes, even though its prediction dynamics remain in the kernel regime (Appendix E.1 of Yang and Hu, 2021). Our setting exhibits the same distinction.

Practical implications. We have added a new counterexample (Proposition H.2) showing that fast learning-rate transfer can fail when T≍nT\asymp\sqrt{n}, even under μ\muP-type scaling and the spectral assumptions of our positive result. This demonstrates that a parameterization supporting desirable training behavior does not automatically guarantee fast transfer when the training horizon grows with width. Thus, scaling choices that work well for other purposes need not ensure fast transfer. While our result does not directly establish a scaling rule for practical nonlinear architectures, it highlights the importance of explicitly accounting for the width–horizon relationship when designing or evaluating learning-rate transfer.

[1] Yang and Hu (2021, ICML). Tensor Programs IV: Feature Learning in Infinite-Width Neural Networks.

Comments on the paper

2 comments

We thank the reviewer for the constructive questions. We address each question below. We have incorporated the new counterexample and numerical experiments into the revised manuscript, and will incorporate the remaining clarifications in a subsequent revision.

Q1. μ\muP is essential for the normalization and fluctuation structure of our model. It ensures that the effective kernel AnA_n converges to a nontrivial limit KK and that the initial predictions vanish, with n−1/2n^{-1/2} fluctuations (Lemma 3.3). The explicit limiting distributions and convergence rates therefore depend on this initialization.

The underlying fixed-kernel perturbation mechanism is more general. Once the effective kernel and initial residual converge appropriately, the spectral characterization of the limiting optimum (Theorem 3.1) and the perturbation formulation for fast transfer (Section 4.2) do not intrinsically depend on μ\muP.

Q2. The condition T=o(n)T=o(\sqrt{n}) is sharp in general under our assumptions. We have added Proposition H.2, which constructs an example satisfying Assumption 3.2 where fast transfer fails at T≍nT\asymp\sqrt{n}.

The critical step in the proof of Theorem 3.4 is ∣ηn,T−η∞,T∣=Op(n−1/2)=op(T−1)|\eta_{n,T}-\eta_{\infty,T}|=O_p(n^{-1/2})=o_p(T^{-1}), which requires T=o(n)T=o(\sqrt{n}). At T≍nT\asymp\sqrt{n}, the hyperparameter gap ∣ηn,T−η∞,T∣|\eta_{n,T}-\eta_{\infty,T}| becomes comparable to the T−1T^{-1} scale, and the analysis of Theorem 3.4 no longer applies.

Q3. Our current results can be extended to two independently initialized finite-width models. Let n≤Nn\le N denote the proxy and target widths, respectively, with N/n→ζ∈[1,∞]N/n\to\zeta\in[1,\infty]. Theorem 3.4 yields ∣ηn,T−ηN,T∣=Op(n−1/2+N−1/2)=Op(n−1/2)|\eta_{n,T}-\eta_{N,T}|=O_p(n^{-1/2}+N^{-1/2})=O_p(n^{-1/2}). Combining this with the local curvature estimates gives the transfer suboptimality gap cn,N,T:=ϕN,T(ηn,T)−ϕN,T(ηN,T)=Op(T2q∗2T−2/n)c_{n,N,T}:=\phi_{N,T}(\eta_{n,T})-\phi_{N,T}(\eta_{N,T})=O_p(T^2q_*^{2T-2}/n). By Theorem 3.4 and independence, the normalized difference between their optimized losses converges to the difference of two independent Gaussian variables, which is nondegenerate. Consequently, the loss gap satisfies an,N,T:=∣ϕn,T(ηn,T)−ϕN,T(ηN,T)∣=Θp(Tq∗2T−1/n)a_{n,N,T}:=|\phi_{n,T}(\eta_{n,T})-\phi_{N,T}(\eta_{N,T})|=\Theta_p(Tq_*^{2T-1}/\sqrt{n}). Thus cn,N,T/an,N,T=Op(T/n)=op(1)c_{n,N,T}/a_{n,N,T}=O_p(T/\sqrt{n})=o_p(1), establishing fast transfer between independently initialized finite-width models.

Q4. For a fixed condition number τ=λ1/λr>1\tau=\lambda_1/\lambda_r>1, we have q∗=(τ−1)/(τ+1)q_*=(\tau-1)/(\tau+1). Theorem 3.4 shows that varying τ\tau changes the horizon-dependent factors and constants, but not the asymptotic width exponents.

Regarding weaker label projections, the same asymptotic rates hold for arbitrarily small but fixed nonzero endpoint label projections. Their imbalance affects the O(T−1)O(T^{-1}) correction to the limiting optimizer through log⁡(c1λ1/(crλr))\log(c_1\lambda_1/(c_r\lambda_r)) (Proposition E.1). Even when both endpoint projections are weakened proportionally, leaving this ratio unchanged, intermediate spectral modes may remain influential over longer finite horizons.

We have added numerical experiments in Appendix B.1. We observe that changing the condition number has little effect on the finite-width agreement, while weakening endpoint projections produces larger finite-range deviations.

Q5. Our results can be extended to approximate proxy tuning by using the local curvature bounds in Section 4.2. Let η^n,T\hat\eta_{n,T} denote the learning rate selected by grid search on the width-nn proxy model, with grid spacing hn,Th_{n,T}. By the convexity of the finite-width objective and the local curvature bounds, we can show hat the corresponding transfer suboptimality gap satisfies $\phi_{\infty,T}(\hat\eta_{n,T})-\phi_{\infty,T}(\eta_{\infty,T}) =O_p(\kappa_T(n^{-1}+h_{n,T}^2))$, where κT=T2q∗2T−2\kappa_T=T^2q_*^{2T-2} is the local curvature scale. Since Theorem 3.4 establishes an,T=Θp(Tq∗2T−1/n)a_{n,T}=\Theta_p(Tq_*^{2T-1}/\sqrt{n}), fast transfer is preserved whenever hn,T=o(n−1/4T−1/2)h_{n,T}=o(n^{-1/4}T^{-1/2}) under T=o(n)T=o(\sqrt{n}).

Thank you for your response and clarifications; I will keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.