- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies learning rate transfer across width in a linear MLP with a single trainable matrix and frozen input/output weights. Under some spectral assumptions, it establishes fast transfer and derives limiting distributions for the optimal learning rate and optimised loss. The analysis is sound and I have not found any major technical flaws. However, the contribution is very close to related prior work.
Critical strengths
The paper is well-written, and easy to follow. The analysis appears technically sound under the stated assumptions. The growing horizon is a strength since most muP papers consider a fixed horizon. Analysing the spectral view is also interesting.
Critical weaknesses
The main weaknesses is related to a significant overlap with prior work, and limited technical novelty.
-
Substantial overlap with Wen et al. (2026). That work already establishes fixed-horizon and growing-horizon analysis of learning rate transfer, and sufficient conditions for fast transfer. Both works reduce training to a quadratic objective involving the power of the covariance operator, etc. The main component that this paper adds is a random input layer. The paper should distinguish more sharply between the existing framework and its additional results.
-
Spectral view: Wen et al. analyse an infinite spectrum where the slow frequencies basically produce polynomial loss decay and (therefore) the optimal learning rate approaches the stability boundary. This paper assumes a fixed finite spectrum, which allows the exponential loss decay. The explicit growing-horizon CLT is an additional result, but the overall contribution is just a refinement of the prior techniques. Also, I’m not sure what additional insights it brings to neural-network training in this case.
Questions for authors
See weaknesses.
3 comments
Response
Thank you for your response.
I carefully went over the proofs and technical results in the paper, and I still believe that there is a substantial overlap with Wen et al. (2026) and the technical contributions are marginal. The proof techniques are mostly refinements of that previous work.
Thank you for taking another careful look at our work. We acknowledge that the overall proof strategy is closely related to Wen et al. (2026). However, our finite-spectrum setting leads to a different long-horizon behavior, where the optimal learning rate is determined by both spectral endpoints and remains strictly inside the stability region. We believe that this distinction, together with the counterexamples and explicit limiting distributions highlighted above, provides meaningful insights beyond the existing literature.
We appreciate your consideration.
Thank you for your valuable comments. As you pointed out, our work studies a setting closely related to Wen et al. (2026). However, our finite-spectrum setting allows us to obtain several additional insights.
First, we establish two counterexamples to fast transfer: (i) Proposition 3.8 demonstrates that fast transfer can fail even at a fixed training horizon due to the spectral structure of the training data, and (ii) the newly added Proposition H.2 shows that fast transfer can fail when T≍n, even under our spectral assumptions. These counterexamples highlight that successful fast transfer depends not only on network architecture but also on the properties of the training data and the balance between the training horizon T and width n.
Second, by deriving explicit limiting distributions (Theorem 3.4), we characterize the directional sensitivities of the optimized loss and optimal learning rate to finite-width perturbations, revealing that these two quantities can respond differently to the same perturbation (Section 5).