acceptodds
Under review as a conference paper at ICLR 2027

On fast hyperparameter transfer under fixed and diverging time horizons

Abstract

Hyperparameter (HP) transfer, which avoid expensive tuning on large neural networks by reusing values tuned on small models, can substantially reduce large-scale pretraining costs. Yet existing evidence is largely qualitative and does not precisely characterize the computational gain. We study this methodology through the framework of fast transfer ghosh2025understanding, which relates computational benefits to the convergence rates of the optimal loss and HPs with model scale . We first prove a second-order Tensor Program master theorem for general neural computations at a fixed training horizon . For fully connected polynomial networks trained by gradient descent under the Maximal Update Parameterization (P), this yields a sharp fluctuation of the loss and optimal HPs. While this result quantifies loss and HP convergence, we argue that fixed- asymptotics do not explain the computational benefit of -transfer, as the leading centered loss fluctuations conflict with empirically observed scaling laws and loss monotonicity. This motivates us to investigate jointly diverging model width and training horizon in a minimal tractable setting: tuning the gradient descent learning rate in a power-law random features model. Using a resolvent expansion, we derive sharp loss asymptotics and HP convergence, and establish necessary and sufficient spectral conditions for fast transfer at width-dependent training horizons. We further discuss the scaling behaviors at the compute-optimal frontier and the effect of stochastic gradients.

Primary area:
learning theory
Keywords:
Hyperparameter transfer, scaling laws, scaling limit, tensor program

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_7R5DOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper asks when transferring a learning rate from a small model yields a computational benefit, rather than merely a convergent sequence of optimal learning rates. At a fixed training horizon, it develops a second-order tensor-program analysis and obtains loss-curve and optimizer fluctuations for polynomial muP MLPs. It then explains why centered finite-width fluctuations can satisfy a fast-transfer rate criterion without a systematic benefit from increasing width. For jointly growing width and horizon, a power-law sketched linear-regression model separates centered fluctuations from positive capacity bias and yields conditions for transfer to outperform direct uniform-grid tuning under a specified compute budget. The paper also discusses compute-optimal schedules and illustrates a different mechanism for stochastic gradients.

Critical strengths
  1. The paper identifies a meaningful limitation of using optimal-HP convergence or the ratio of transfer loss to an absolute finite-width loss gap as evidence of computational benefit. The centered-fluctuation interpretation makes this distinction precise and provides a well-motivated reason to study the joint width-horizon limit. The proposed second-order tensor-program result could also be a valuable technical contribution beyond this application.
  2. The growing-horizon analysis connects the stability edge, changing curvature, capacity bias and tuning cost within one tractable model. The manuscript carefully distinguishes proportional from common horizons and acknowledges that the SGD mechanism differs from population GD. The synthetic figures include substantial seed/sketch variation and validation-loss diagnostics, which help make the asymptotic mechanisms concrete.
Critical weaknesses
  1. The compute-optimal equality case needs a more explicit connection to the admissible tuning protocols. Definition 1 (p.4) requires widths and grid sizes to be fixed powers of the budget, whereas Theorem 4 (pp.8-9) includes beta = 3 alpha, an asymptotically smaller proxy n = o(m), and a common training horizon. At this boundary the target horizon is of order sqrt(m). For a strictly smaller-power proxy, its stability-edge fluctuations of order n^(-1/2) can exceed the infinite-width optimum's distance from the edge, of order (log T)/T in Proposition 1. It is therefore not clear from the main statement how the selected proxy rate remains safe for an independently initialized target with probability tending to one. A logarithmically separated proxy may address this comparison, but would require clarification of the fixed-power restriction. Please give the precise allocation and boundary argument, or delimit the statement accordingly. This concern is about the endpoint and definitions; it does not by itself challenge the strict-interior regimes.

  2. The scope of the fixed-horizon nonlinear result should be stated consistently. Section 3.1 (p.4) describes coverage of GD-trained MLPs with pseudo-Lipschitz activations, while Theorem 2 (p.5) obtains the entire learning-rate curve through a finite polynomial expansion and explicitly assumes polynomial activation and loss. Figure 2 uses tanh. It would help to state whether an additional result extends the split-program reduction to this non-polynomial setting, or whether the tanh experiment is empirical evidence beyond the theorem. Similarly, the broad conclusion about fixed horizons should retain the positive fluctuation-variance and nondegenerate-optimum conditions of Corollary 1. Clear separation of these cases would make the theoretical contribution easier to assess without diminishing the value of the experiment.

Questions for authors
  1. At beta = 3 alpha under a common horizon, what are the exact proxy width and grid-size scalings in Theorem 4? Are slowly varying factors allowed in Definition 1, and how is the probability of selecting a rate above the target stability edge controlled?
  2. Does the supplementary tensor-program argument establish the loss-curve CLT for tanh or other non-polynomial activations? If so, which assumptions justify moving scalar-dependent training operations into the split-program representation?
  3. What is the boundary behavior at rho = 1 in Theorem 3? Equation (6) lists rho < 1 and rho > 1, while Assumption 1 allows equality; a concise statement of any logarithmic correction or exclusion would be useful.
5 comments

Thank you for the careful reading and constructive feedback. We address the technical concerns below. Line numbers refer to the revised version.

Compute-optimal allocation

  • The exact allocation is n=⌊m/log⁡m⌋n=\lfloor m/\log m\rfloor and Gn=⌈(log⁡m)1/2⌉G_n=\lceil(\log m)^{1/2}\rceil, with the horizon schedule Tj=⌊h⋆jγ⋆⌋T_j=\lfloor h_\star j^{\gamma^\star}\rfloor (γ⋆=1/2\gamma^\star=1/2 at β=3α\beta=3\alpha, and h⋆h_\star the constant such that the single target run attains L(F)\mathcal{L}(\mathcal{F}), including its leading constant), and with mm the largest width within the budget. The target is trained for TmT_m steps, and the proxy for TnT_n steps at proportional horizons and for TmT_m steps at a common horizon. In the submission, these formulas appeared only in the appendix due to the page limit: the allocation in Appendix N (Equations (356)–(357)) and T=h⋆nγ⋆T=h_\star n^{\gamma^\star} in Appendix M (Equation (351)). We have now moved them to the main text, right after Theorem 4 (starting at line 457). You are right that this allocation is not clearly explained in Definition 1 as written. We intended "fixed powers" to fix only the polynomial exponents, allowing slowly varying factors; the revised Definition 1 states this explicitly (lines 182–183): all widths and grid sizes are now of the form Fc+o(1)\mathcal{F}^{c+o(1)} for a fixed c≥0c\ge0.
  • Why does the common horizon not break this? This is discussed in the last sentences of the paragraph "Sharpness of the window T≪nT\ll\sqrt{n}" in Appendix A.4: at the boundary T≍nT\asymp\sqrt{n}, the distance of the optimum from the edge still exceeds the edge fluctuations by a logarithmic factor, and the boundary case of Theorem 4 is handled by a separate stability argument in Appendix N. Intuitively, a proxy whose own stability edge lies above 2 would like to use η=2\eta=2, but only if the horizon is long enough for its leading direction to converge there, which takes of order nlog⁡n\sqrt{n}\log n steps (see also Figure 6(d)). The common horizon Tm≍mT_m\asymp\sqrt{m} is longer than n\sqrt{n}, but for n=m/log⁡mn=m/\log m it is still shorter than nlog⁡n\sqrt{n}\log n, so the proxy selects the last grid point below 2, namely 2−1/Gn2-1/G_n, with probability tending to one. Since 1/Gn≫m−1/21/G_n\gg m^{-1/2}, this learning rate lies below the target's stability edge 2/λmax⁡(Am)=2+OP(m−1/2)2/\lambda_{\max}(A_m)=2+O_{\mathbb{P}}(m^{-1/2}) with probability tending to one; this is how the probability of selecting a rate above the target's edge is controlled. We have moved this discussion to the main text, after Theorem 4 (starting at line 480).
  • The case β=3α\beta=3\alpha sits exactly on the boundary, where polynomial exponents alone cannot decide the outcome, so there we resolve the logarithm (logarithmic factors are needed only at the boundary). A proxy smaller by a power of mm selects η=2\eta=2 with positive probability (Lemma 33), and at η=2\eta=2 the target never learns its leading direction, so transfer fails, exactly as you anticipated. This is now stated explicitly after Theorem 4.

(Continued in the next comment.)

Scope of the constant-TT result

Thank you for pointing this out. Our intention in using the tanh activation in Figure 2 was to illustrate that the same rate characterization is expected to hold beyond the polynomial restriction; we now remark on this in lines 289–292 of the revision. We make the following clarifications.

  • Our CLT (Theorem 1) holds for nonlinearities satisfying Assumption 2, which include tanh; polynomial activations are needed only for recasting GD training of an MLP as a split program (Proposition 9). This is because, after the first update, the inputs to the matrix multiplications depend on scalars such as the loss derivative, and only for polynomials can these scalar factors be pulled out exactly. We have revised the corresponding sentence in Section 3.1 to avoid confusion (lines 212–214).
  • For tanh, the recasting is still possible, but it requires extra steps: one linearizes each scalar-dependent input around the deterministic limit of its scalars. Since the scalars fluctuate at scale n−1/2n^{-1/2} and the activation has a bounded second derivative, the remainder changes every average only by O(n−1)O(n^{-1}), below the scale of the CLT, while the linear terms can be expressed by a split program. This gives the CLT of the loss at any fixed learning rate. It does not by itself give Corollary 1 (tuned loss and optimal learning rate), which also requires uniform control of the curve and its derivatives in η\eta; we do not carry this out for tanh, since it would only lengthen the proof, and the polynomial case already yields the conclusions of Section 3. This is why the submitted version stated Theorem 2 for polynomial activations; for tanh, Figure 2(b,c) provides empirical evidence beyond our theorems. We have added Remark 6 to the appendix (lines 2448–2479) explaining the extension, and the caption of Figure 2 (line 281) now refers to it.
  • Following your suggestion, the revision points to the conditions of Corollary 1 more explicitly when stating the positive-variance result (lines 088 and 268–269). These conditions are mild: by Theorem 2, s2(η)>0s^2(\eta) > 0 for all but at most 2D2D learning rates, under conditions at initialization that hold almost surely for nondegenerate data (together with σV2>0\sigma_V^2 > 0). A nondegenerate optimum is not even needed for the conclusion: degenerate optima are covered by Theorem 5. We state this assumption in the main text only to connect with the criterion of [GWB25], which is expressed in terms of the curvature μn\mu_n; the extension to degenerate optima requires more notation and is orthogonal to our message, so we defer it to the appendix.

Boundary case ρ=1\rho=1 in Theorem 3

  • We omitted ρ=1\rho=1 from Equation (6) for brevity. It is covered in the appendix: Lemma 29 (Appendix I.3) gives σn,T≍n−1/2T−1log⁡T\sigma_{n,T}\asymp n^{-1/2}T^{-1}\sqrt{\log T}. We have now added it to Equation (6) in the main text.

We would be happy to answer any follow-up questions in the discussion period.

References

[GWB25] Understanding the Mechanisms of Fast Hyperparameter Transfer

Thank you for the detailed response and revisions. These address my concerns, and I will maintain my original rating.

Reviewer_25X2OpenReview ↗
Rating
4
Confidence
3(+1)
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper develop the mathematical foundation for HP transfer across widths. It first consider fixed-horizon. In this case, it first proves a second-order Tensor Program theorem, capturing the asymptotic normality of the n−1/2n^{-1/2} fluctuation with covariance term that can be tracked recursively in tensor program. Then using this, it proves fast HP transfer for MLP with polynomial activations. The author then argue that this fixed-horizon fast convergence result is misleading, and studies a joint width–horizon limit in sketched linear regression. This gives regimes in which loss becomes monotone in width and leads to compute-based comparisons between proxy transfer and direct grid search.

Critical strengths
  • The paper is an impressive mix of many solid theoretical contributions that explores HP transfer in-depth. The second-order Tensor Program theorem is technically substantial and potentially useful beyond HP transfer. The study of joint width-horizon limit treats various theoretical aspects simultaneously, including optimal learning rate, stability edge, grid resolution, and compute allocation, giving a complete picture of what's going on.
  • Beyond rigorous mathematical proofs, the paper also supplies novel and interesting insights. The distinction between centered fluctuations and one-sided capacity bias is interesting, explaining when we expect wider-is-better and when we do not. It explains clearly why the first result might be misleading. The explicit comparison of compute is also interesting, explicit account for the compute saving of HP transfer.
  • Different segments of the theory are put together to give a coherent story that enhance the existing understanding of HP transfer significantly.
  • However, I do not have the expertise to carefully read and verify the second half, growing-horizon regime, so I have to leave the judgement to other reviewers. The first half of fixed-horizon seems to be sound.
Critical weaknesses

I do not find any significant weakness of the paper. It does make some strong, somewhat unrealistic assumptions. For example, the fixed-horizon HP transfer is proved only for polynomial MLP. Moreover, the growing-horizon part focuses on a sketched linear regression model. However, I understand that the theory is already very complicated on this simplified cases, and also these theory gives interesting insights that's transferable to more complicated case.

Questions for authors
  1. how does fig 1(a) reconcile with the evidence from other paper that shows wider is better in fixed-horizon regime? e.g. Fig 8 in Tensor Program V paper.
  2. Is the second-order TP theorem necessary for the polynomial-MLP fast-transfer result? The original TP theorem already claims O(n^{-1/2}) second order fluctuation, even though they do not prove asymptotic normality or explicitly tracks the covariance. Is the original TP result enough to show fast convergence, or does the second-order TP result gives some essential ingredient?
2 comments

Thank you for your positive feedback and thoughtful comments. We address the technical concerns below.

Fixed-horizon regime

Thank you for bringing this up. We would like to clarify that having the same horizon across scales does not imply the fixed-TT asymptotics, which describe only the T≪nT\ll n regime. In particular:

  • Tensor Program theory, including our Section 3, takes the horizon TT to be fixed as n→∞n\to\infty, and thus T≪nT\ll n. For example, in Figure 1(a), T=4T=4 and the width is at least 256. In this regime, the leading finite-width correction is a centered, initialization-dependent fluctuation, and wider is not reliably better. In contrast, TP V [YHB+22] reports a monotone gain from width, which suggests that the leading term in the loss is not the CLT fluctuation, and thus that the training time TT is not small compared with the model width.
  • In other words, Figure 8 of [YHB+22] does not correspond to the fixed-TT regime of the TP theory; instead, it is more accurately captured by the growing-horizon regime, more precisely by the common-horizon scaling of our Section 4 (which existing theory works on TP do not cover): the figure compares widths at a common but long horizon (Fig. 8: widths 256 to 32,768 trained for over 10910^9 tokens; similarly, Fig. 7: widths 128 to 4096 trained for 10410^4 steps). In fact, [YHB+22] remarks (footnote 27) that its intuition is derived assuming that the width is much larger than the training time, while its experiments, e.g., training to convergence, are often outside this regime. In Section 4, we show that at such long horizons (and for ρ>1\rho > 1), the one-sided capacity bias indeed dominates the centered fluctuation, so that the wider model has lower loss with probability tending to one (Theorem 3; Proposition 2(ii) at a common horizon).

We have added this clarification as remark at the end of page 6 of the revised manuscript.

Necessity of the Second-Order TP Theorem

  • The fast-transfer criterion of [GWB25], μbn2=o(an)\mu b_n^2=o(a_n), requires a lower bound on the loss gap ana_n. The O(n−1/2)O(n^{-1/2}) rate available from the original TP theory [YL23] is an upper bound: it gives bn=O(n−1/2)b_n=O(n^{-1/2}), hence μbn2=O(n−1)\mu b_n^2=O(n^{-1}), but it does not exclude, e.g., an=O(n−1)a_n=O(n^{-1}), in which case μbn2=o(an)\mu b_n^2=o(a_n) could fail. The need for a lower bound is what motivated us to prove asymptotic normality: by Corollary 1, an≍Pn−1/2a_n\asymp_{\mathbb{P}}n^{-1/2} whenever s(η∞⋆)>0s(\eta^\star_\infty) > 0.
  • Moreover, our CLT is a distributional result rather than a bound on the size of the fluctuations; this is also necessary to identify the gap in the wider-is-better explanation based on Tensor Programs, since a rate cannot tell whether the n−1/2n^{-1/2} correction is centered or one-sided. Our CLT shows that it is centered, so that PLm⋆<Ln⋆→1/2\mathbb{P}\\{\mathcal{L}^\star_m < \mathcal{L}^\star_n\\}\to1/2 (Corollary 1), and this is what leads us to the joint width–horizon scaling of Section 4.
  • To sum up, the original TP theorem is not enough to prove or disprove fast transfer. We also remark that the second-order theorem requires a new ingredient in the proof: after conditioning on earlier matrix products, the finite-rank projections that do not affect the law of large numbers contribute at order n−1/2n^{-1/2}, so we analyze the Gaussian conditioning in a finer, hierarchical way (Section 3.1). This extends the classical Gaussian conditioning technique from a law of large numbers, or an O(n−1/2)O(n^{-1/2}) upper bound, to a CLT, which may be of independent interest.

We would be happy to answer any follow-up questions in the discussion period.

References

[GWB25] Understanding the Mechanisms of Fast Hyperparameter Transfer

[YHB+22] Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer

[YL23] Tensor Programs IVb: Adaptive Optimization in the Infinite-Width Limit

Thank you for the clear explanation. I learned a lot from this paper and the authors' response. I maintain my positive score and raise my confidence score, as I have spent more time reading the paper in more detail (on the second half, which I mentioned that I do not have expertise in).

Reviewer_ggMiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper considers the problem of hyperparameter transfer (HT) under the Maximal Update Parameterization (μP\mu P), focusing on the theoretical aspects of this phenomenon. Compared to prior work, the authors provide many meaningful extensions in their analysis, including CLT characterization, more general model architectures and trainable weights, and polynomial training/loss validation, to name a few. The paper thereby advances the theoretical understanding of fast HT under μP\mu P.

Critical strengths

First things first, I am not really an expert in this direction, so my assessment might be a bit off.

I think that the paper presentation is very clear and easy to follow, even for a general audience. The motivation is sound, and I think this paper can be generally considered a theoretical follow-up of the paper [1], which is (as far as I know; correct me if I am wrong) the first real theoretical guarantee for HP under μP\mu P. However, this work provides many meaningful extensions, including: (1) MLP with poly activations instead of a linear MLP, (2) all weight matrices are trainable, (3) considering both fixed-horizon TT and growing-TT models, (4) polynomial training/validation loss instead of just squared training loss, and (5) CLT for the loss curve and the optimal instead of merely convergence.

Another strength of this paper is that the authors also identify an important limitation of fixed-horizon TT analysis, as the convergence of the optimal learning rate does not necessarily translate to the computational benefits from HT. To address this issue, the authors investigate a simple growing-horizon setting and establish conditions under which HF can be computationally advantageous.

Critical weaknesses

There is not much of a weakness of this paper that I know of. I have two questions, though:

  1. Can the authors explicitly specialize their CLT to the deep linear network of [1] and demonstrate what additional information it gives beyond [1]’s OP(n−1/2)O_P(n^{-1/2}) rate? I would like to see a clearer relationship between the two works.

  2. While the paper establishes a general CLT for polynomial activation MLPs under a fixed training horizon, the growing horizon analysis is currently restricted to the sketched linear regression. Can the authors elaborate on the challenge here, and how can we potentially extend this to polynomial activation MLPs?

Final comment

I think the contribution of this paper is strong and therefore recommend acceptance. However, since I am not an expert in this direction, I can only provide a weak accept, though with high confidence. Good luck with the submissions.

References

[1] Hayou, A proof of learning rate transfer under, AISTATS’26

Questions for authors

See Weaknesses

2 comments

Thank you for your review and positive assessment. We address the technical concerns below.

Comparison with [Hay25]

  • The deep linear network setting of [Hay25] is arguably the simplest special case of our Theorem 2, which allows any polynomial activation, polynomial loss, and training of all layers. More specifically, the training loss of linear networks after TT GD steps is a polynomial in η\eta whose coefficients are scalar observables of a split NETSOR program. Our Theorem 1 therefore applies to these coefficients, and the arguments of Theorem 2 and Corollary 1 carry over with only notational changes for the frozen layers. Under the conditions of Corollary 1, this gives n,(ηn⋆−η∞⋆)⇒−G′(η∞⋆)L∞′′(η∞⋆),n,(Ln⋆−L∞⋆)⇒N(0,s2(η∞⋆)),\sqrt{n}\\,(\eta^\star_n-\eta^\star_\infty)\Rightarrow-\frac{\mathbb{G}'(\eta^\star_\infty)}{\mathcal{L}''_\infty(\eta^\star_\infty)}, \qquad \sqrt{n}\\,(\mathcal{L}^\star_n-\mathcal{L}^\star_\infty)\Rightarrow\mathcal{N}\big(0,s^2(\eta^\star_\infty)\big), where G\mathbb{G} is the centered Gaussian limit of n(Ln−L∞)\sqrt{n}(\mathcal{L}_n-\mathcal{L}_\infty), with covariance given by the recursion of Theorem 1.

  • Compared with [Hay25], which proves only the upper bound ηn⋆−η∞⋆=OP(n−1/2)\eta^\star_n-\eta^\star_\infty=O_{\mathbb{P}}(n^{-1/2}) after one GD step (for T>1T>1 steps, only almost sure convergence is stated, without a rate), our paper gives a much stronger characterization: (i) the rate bn=OP(n−1/2)b_n=O_{\mathbb{P}}(n^{-1/2}) at every fixed horizon TT, with the exact limiting law; (ii) a matching lower bound for the loss gap, an≍Pn−1/2a_n\asymp_{\mathbb{P}}n^{-1/2} whenever s(η∞⋆)>0s(\eta^\star_\infty) > 0, so that μbn2=OP(n−1)=oP(an)\mu b_n^2=O_{\mathbb{P}}(n^{-1})=o_{\mathbb{P}}(a_n), which an upper bound alone does not provide; (iii) both nan\sqrt{n} a_n and nbn\sqrt{n} b_n converge to absolute values of centered Gaussians.

  • The proof strategies also differ: [Hay25] obtains the convergence of these coefficients as an immediate consequence of the existing Tensor Program master theorem (a law of large numbers), and this proof works only for linear networks. Extending it to a CLT for nonlinear models beyond two layers is highly nontrivial — we introduce the split program to prove a joint CLT for these coefficients, which requires careful treatment of the Gaussian conditioning steps of Tensor Programs at the next order (Section 3.1).

  • More importantly, as argued in Section 2 of [GWB25], the convergence of the optimal learning rate in [Hay25] does not establish the computational benefit of μ\mu-transfer. Our sharper characterization in point (ii) is what allows us to address this question: by Corollary 1, of two independently initialized networks at a fixed horizon, the smaller one has lower loss with probability tending to 1/21/2. This is at odds with the monotone improvement with scale observed in practice, and defeats the purpose of μ\mu-transfer, since larger models do not give better performance. Hence we argue that the fixed-TT asymptotics of the original Tensor Program series [YH20] (and consequently of [Hay25]) are not the right regime for understanding the benefits of μ\muP transfer — a point often overlooked in the literature — and this motivates our study of diverging-TT scaling (Section 3.3).

Growing-horizon analysis

  • The challenge is quite fundamental: we lack the mathematical tools to analyze the limit of general neural computations at diverging training horizons. Our fixed-horizon analysis treats TT as a constant, and existing non-asymptotic analyses based on Gaussian conditioning (the technique behind both Tensor Programs and AMP state evolution) have error bounds that grow at least exponentially in the number of steps, e.g., CT/nC^T/\sqrt{n} with C>1C > 1 [CCP+25, RV16]. Such bounds allow TT only up to order log⁡n/log⁡log⁡n\log n/\log\log n, far from the polynomial horizons T≍nγT\asymp n^\gamma that we consider in Section 4 (and that are used in practice).
  • Despite this difficulty, our sketched linear regression analysis demonstrates why a growing horizon is a crucial feature in understanding hyperparameter transfer: it captures the monotone decrease of the tuned loss across scale, and it exhibits interesting phenomena of transfer that the fixed-TT regime does not reflect (e.g., that the benefit of transfer depends on the spectral properties of the data; see Section 4). We see the large-TT analysis of nonlinear MLPs as an important and difficult open problem.

We would be happy to answer any follow-up questions in the discussion period.

References

[CCP+25] State evolution beyond first-order methods

[GWB25] Understanding the Mechanisms of Fast Hyperparameter Transfer

[Hay25] A Proof of Learning Rate Transfer under μ\muP

[RV16] Finite Sample Analysis of Approximate Message Passing Algorithms

[YH20] Feature Learning in Infinite-Width Neural Networks

Reviewer_ggMiOpenReview ↗

Official response by Reviewer ggMi

I thank the reviewer for their detailed response. I feel like I learned something here! My rating is already positive, and I won't change it. Good luck with the submission!

Comments on the paper

1 comment

Dear Reviewers,

Thank you to those who have already engaged with the authors' rebuttal. As we approach the end of the discussion period, if you have not yet replied, please do so soon.

Thank you for your time and effort in contributing to the ICLR community.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.