acceptodds
Under review as a conference paper at ICLR 2027

Quantifying Hyperparameter Transfer and the Importance of Embedding Layer Learning Rate

Abstract

Hyperparameter transfer allows extrapolating optimal optimization hyperparameters from small to large scales, making it critical for training large language models (LLMs). This is done either by fitting a scaling law to the hyperparameters or by a judicious choice of parameterization, such as Maximal Update (P), that renders optimal hyperparameters approximately scale-invariant. In this paper, we first develop a framework to quantify hyperparameter transfer through three metrics: (1) the quality of the scaling law fit, (2) the robustness to extrapolation errors, and (3) the asymptotic loss penalty due to choice of parameterization. Next, we investigate through a comprehensive series of ablations why P appears to offer high-quality learning rate transfer relative to standard parameterization (SP), as existing theory is inadequate. We find that the overwhelming benefit of P relative to SP when training with AdamW arises simply from maximizing the learning rate of the embedding layer. In SP, the embedding layer learning rate acts as a bottleneck that induces training instabilities; increasing it by a factor of width to match P dramatically smooths out training while improving hyperparameter transfer. We also find that weight decay improves the scaling law fits, while, in the fixed token-per-parameter setting, it hurts the robustness of the extrapolation.

Primary area:
optimization
Keywords:
Hyperparameter Transfer, Training Dynamics, Training Instabilities

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_CRvyOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies hyperparameter transfer across model widths, focusing on why Maximal Update Parameterization (μ\muP) provides better learning rate transfer than Standard Parameterization (SP). The authors first propose three metrics to quantify transfer quality, including loss predictability, robustness to extrapolation errors, and asymptotic loss degradation. They then systematically ablate four differences between μ\muP and SP, and find that increasing the embedding layer learning rate in SP is sufficient to achieve similar transfer quality as μ\muP. Additional experiments investigate the effect of embedding learning rate during training, weight decay, and compute-optimal scaling.

Critical strengths
  • The paper provides a systematic empirical analysis of the differences between μ\muP and SP. The 16 parameterization ablations help isolate the effect of different components, and the experiments on changing embedding learning rate during training provide additional evidence for its importance.
  • The proposed three metrics provide a useful perspective on hyperparameter transfer beyond whether the optimal learning rate remains constant across widths. I think considering predictability, robustness, and asymptotic performance together is reasonable, although the reliability of these metrics still needs stronger validation.
Critical weaknesses
  • My main concern is the limited novelty compared with Everett et al. (2024). Their work has already proposed the same learning rate scaling for SP under Adam, where the embedding learning rate scales as O(1)O(1) and the hidden and readout learning rates scale as O(1n)O(\frac 1n). They also showed that this configuration achieves hyperparameter transfer and can outperform µP, with experiments on much larger models. This is essentially the same as the main finding of SP+Embd in this paper. Although Everett et al. (2024) is cited, the paper does not sufficiently discuss this overlap or explain what new understanding is provided beyond the prior work. The additional ablations are useful, but I am not convinced they constitute a sufficiently novel contribution given that the main result has already been demonstrated.
  • I am also not fully convinced by the validation of the proposed transfer metrics. The predictability error EE is evaluated on the same model widths used for fitting, so a good fit does not necessarily demonstrate accurate extrapolation to larger models. Similarly, the robustness exponent and asymptotic loss degradation rely on fitted scaling laws, but the paper does not directly validate these predictions on held-out larger widths. The fitting also involves several heuristic choices, including exponent clipping and handling degenerate solutions, whose effect on the conclusions is unclear. Since the quantitative framework is another main contribution of this paper, I think stronger evidence is needed to show that these metrics reliably predict actual hyperparameter transfer.
Questions for authors

Here are my main questions and concerns. I would be happy to reconsider my opinion if the major concerns are resolved.

  • How does the main finding differ from Everett et al. (2024), particularly Section 4.2.1? That work already proposed the same embedding and hidden layer learning rate scaling for SP, and demonstrated that it achieves transfer and outperforms μ\muP. I would like the authors to clarify what is fundamentally new beyond the additional ablation experiments.
  • Can the authors validate the proposed metrics using held-out model widths? For example, fitting the scaling laws on smaller models and testing whether EE and κ\kappa actually predict the loss penalty from transferred learning rates on larger models would be helpful. I would also like to understand how sensitive the conclusions are to the fitting choices, especially the exponent clipping and degeneracy handling.
4 comments
AuthorsOpenReview ↗

Rebuttal (Part 1/3)

Thank you for the detailed review and for recognizing the value of the ablations and transfer metrics. We clarify what our metrics and experiments add relative to Everett et al. and evaluate the fitted loss model on held-out widths. We also clarify exponent clipping and degeneracy handling below.

W1/Q1: Novelty relative to Everett et al. (2024)

How does the main finding differ from Everett et al. (2024), particularly Section 4.2.1?

Thank you for raising this. Everett et al. relaxed μ\muP's alignment assumption for the last layer and showed that SP transfers with Θ(1)\Theta(1) embedding and Θ(1/n)\Theta(1/n) hidden and readout learning rates (their Section 4.2.1). We will cite Everett et al. for this learning rate prescription. This relaxation concerns one of the four differences between SP and μ\muP, the last-layer initialization, and leaves three: the embedding learning rate, the LayerNorm learning rate, and the attention scale.

Our paper makes the following contributions beyond the additional ablations.

  • Quantifying transfer.

    As you note, our three metrics provide a perspective on transfer beyond whether the optimal learning rate remains constant across widths. The loss predictability error E\mathcal{E} measures how accurately the loss landscape is captured by Equation 6 of the paper. The transfer robustness exponent κ\kappa measures how errors in the extrapolated learning rate affect the loss at scale. The asymptotic loss degradation R(∞)\mathcal{R}(\infty) measures the performance gap relative to the best asymptotic loss. Together, these quantify loss predictability, transfer robustness, and asymptotic performance. Separating these three metrics lets us identify loss predictability as a limiting factor in SP's transfer. The observed loss spikes and ablations that change only the embedding learning rate connect this limitation to training instability caused by SP's small embedding learning rate. Our framework therefore provides a quantitative diagnosis of the aspect of transfer improved by the embedding learning rate prescription.

  • The embedding layer learning rate controls training stability.

    Everett et al. scale the embedding and LayerNorm learning rates together: their Appendix C.2 groups token embeddings, positional embeddings, and LayerNorm scale parameters. Our ablations separate these effects across all 16 combinations of the four differences between SP and μ\muP. These ablations identify the embedding layer learning rate as the key factor: increasing the LayerNorm learning rate in SP worsens instability, while decreasing it in μ\muP has a negligible effect.

    Furthermore, Figure 4 of the paper examines when the embedding learning rate matters by switching it during training at width 768768, with λ=10−3\lambda=10^{-3} for SP and λ=6×10−3\lambda=6\times10^{-3} for μ\muP. Raising SP's embedding learning rate early eliminates its instabilities and improves its loss, while lowering μ\muP's causes a lasting loss gap that is larger the earlier the switch. These experiments identify early training as the critical period for the embedding learning rate.

  • Broader scope: weight decay and other architectures.

    Everett et al. use no weight decay in their main experiments and examine one constant weight decay in their Appendix G. We examine transfer across eight weight decays, quantifying its effect on loss predictability, transfer robustness, and asymptotic performance. Appendix F and Appendix G of the paper extend the investigation to LLaMA-style Transformers and CNNs, respectively.

AuthorsOpenReview ↗

Rebuttal (Part 2/3)

W2/Q2: Held-out validation

Can the authors validate the proposed metrics using held-out model widths?

Thank you for raising this point.

Loss predictability on held-out widths.

We refit the loss model in Equation 6 of the paper using only widths 128128, 256256, 512512, 768768, and 10241024, and compute E\mathcal{E} on the held-out widths 15361536 and 20482048. The fitted parameters are held fixed when predicting the held-out losses. We use the paper's 1.35×1.35\times per-width loss filter and compute the mean squared error on the retained observations. Figure 24 of Appendix L of the updated text compares the held-out E\mathcal{E} with E\mathcal{E} reported in Figure 5b of the paper for seven parameterizations and four weight decays. One held-out fit is unavailable, leaving 27 combinations with both errors available. Across these 27 combinations, the Spearman rank correlation between the two errors is 0.870.87, and the median ratio of held-out E\mathcal{E} to the paper's E\mathcal{E} is 1.461.46.

Loss penalty from a transferred learning rate.

We freeze the separately fitted scaling laws from the same five widths and select the learning rate with minimum raw final validation loss at width 128128. This gives η=1\eta=1 for SP, 0.90.9 for SP+Embd, and 0.10.1 for μ\muP at the weight decays in the table. Let νtransfer=log⁡2η\nu_{\rm transfer}=\log_{2}\eta be its fixed log learning rate and L^∗(n)\hat{L}^{\ast}(n) the minimum observed loss over the scanned learning rates at width nn. We compare the observed loss penalty L^(νtransfer;n)−L^∗(n)\hat{L}(\nu_{\rm transfer};n)-\hat{L}^{\ast}(n) with the prediction

12Hfit(n)(νtransfer−νfit∗(n))2,\tfrac12 H_{\rm fit}(n)\big(\nu_{\rm transfer}-\nu^{\ast}_{\rm fit}(n)\big)^2,

where the parameters of the fitted curvature and optimal-learning-rate laws remain fixed when evaluating the held-out widths. The transferred learning rates are exact scanned points, and all six losses pass the paper's 1.35×1.35\times filter.

Held-out loss penalty from reusing the width-128128 optimal scanned learning rate.

Parameterizationλ\lambdannObservedPredicted
SP10−310^{-3}153615360.09390.09390.00160.0016
SP10−310^{-3}204820480.00000.00000.00150.0015
SP+Embd10−310^{-3}153615360.00720.00720.00280.0028
SP+Embd10−310^{-3}204820480.00430.00430.00290.0029
μ\muP6×10−36\times10^{-3}153615360.06130.06130.08630.0863
μ\muP6×10−36\times10^{-3}204820480.06190.06190.08870.0887

SP's prediction is poor at width 15361536. The maximum absolute prediction error across the two widths is 0.09240.0924 for SP, 0.00450.0045 for SP+Embd, and 0.02690.0269 for μ\muP. These are predictions from the fitted curvature and optimal-learning-rate laws together; the sign of κ\kappa alone does not predict the penalty from reusing a fixed raw learning rate. We report these checks in Appendix L of the updated text.

Optimal-loss predictions.

We also freeze the optimal-loss scaling law fitted to the five training widths and compare its prediction L∗(n)L^{\ast}(n) with the held-out scanned minimum L^∗(n)\hat{L}^{\ast}(n). The prediction error is ∣L∗(n)−L^∗(n)∣|L^{\ast}(n)-\hat{L}^{\ast}(n)|. As a baseline, we carry the width-10241024 minimum forward unchanged, giving error ∣L^∗(1024)−L^∗(n)∣|\hat{L}^{\ast}(1024)-\hat{L}^{\ast}(n)|.

Held-out optimal-loss prediction error and the width-10241024 baseline.

Parameterizationλ\lambdannL^∗(n)\hat{L}^{\ast}(n)ErrorBaseline error
SP10−310^{-3}153615363.03773.03770.09310.09310.03270.0327
SP10−310^{-3}204820482.91052.91050.01500.01500.15990.1599
SP+Embd10−310^{-3}153615362.87632.87630.02060.02060.07990.0799
SP+Embd10−310^{-3}204820482.83482.83480.04150.04150.12140.1214
μ\muP6×10−36\times10^{-3}153615362.87912.87910.01710.01710.09060.0906
μ\muP6×10−36\times10^{-3}204820482.83082.83080.03640.03640.13900.1390

The fitted prediction has smaller error than this baseline in five of six comparisons; SP at width 15361536 is the exception. These checks assess finite-width predictions from one run per configuration. We report them in Appendix L of the updated text.

AuthorsOpenReview ↗

Rebuttal (Part 3/3)

W2/Q2 continued: Transfer robustness

I would also like to understand how sensitive the conclusions are to the fitting choices, especially the exponent clipping and degeneracy handling.

Transfer robustness exponent.

We also compare κ=α−2β+γ\kappa=\alpha-2\beta+\gamma estimated from the separate scaling laws using five widths (128128–10241024) and all seven widths (128128–20482048), with the paper's 1.35×1.35\times loss filter and the fitting procedure in Appendix D. All three signs remain negative, while the magnitudes and the ordering of SP+Embd and μ\muP change. This checks the stability of the exponent estimate as larger widths are added; it is not an independent prediction of the loss penalty from reusing a smaller model's learning rate.

Transfer robustness exponent from five and seven widths.

Parameterizationλ\lambdaFive widthsSeven widths
SP10−310^{-3}−3.64-3.64−3.56-3.56
SP+Embd10−310^{-3}−2.87-2.87−2.33-2.33
μ\muP6×10−36\times10^{-3}−2.35-2.35−2.71-2.71

We report this comparison in Appendix L.4 of the updated text.

Exponent clipping and degeneracy handling

Appendix D of the paper caps the scaling exponents at 22 as a practical proxy for exponents becoming arbitrarily large. For β\beta, the bound represents rapid convergence. At β=2\beta=2, each doubling of width reduces the remaining correction to the optimal log-learning-rate by a factor of four. We assess whether the remaining change is resolved by the scanned learning rates by comparing the fitted change across widths with the grid spacing.

Our formalism expresses ν∗(n)\nu^{\ast}(n) as an infinite-width value plus a correction Bn−βBn^{-\beta}. When β\beta approaches zero, the correction becomes nearly constant, and the fit cannot determine ν∗(∞)\nu^{\ast}(\infty) and BB separately. When BB is zero, the optimal learning rate is constant and the data do not determine β\beta. Appendix D addresses this ambiguity by increasing the lower bound on β\beta and comparing the resulting fitted-exponent trend with a step function and a linear function. When the step function fits better, we select the best solution on the rapid-convergence branch. Otherwise, we retain the small-β\beta solution.

For the new perturbation analysis of κ\kappa, Appendix L.3 of the updated text uses an additional fitting convention, distinct from the submitted degeneracy rule. Let Δν\Delta\nu be the smallest adjacent spacing among distinct log learning rates with finite losses in the unperturbed sweep, and let nmin⁡n_{\min} and nmax⁡n_{\max} be the smallest and largest fitted widths. If the procedure in Appendix D retains the fit with the lowest mean squared error over the lower bounds, and its fitted change ∣B(nmin⁡−β−nmax⁡−β)∣|B(n_{\min}^{-\beta}-n_{\max}^{-\beta})| is below Δν\Delta\nu, we select the fit with lower bound βmin⁡=1.5\beta_{\min}=1.5 and upper bound 22. The curvature and optimal-loss fits are retained. This additional choice does not change the loss filtering or the E\mathcal{E} and R(∞)\mathcal{R}(\infty) calculations. The new κ\kappa perturbation results are conditional on this convention; the five-versus-seven-width comparison uses Appendix D.

Exponent-cap sensitivity.

We compare upper caps of 11 and 44 on β\beta with the paper's cap of 22, keeping α\alpha and γ\gamma fixed and refitting the optimal-learning-rate law. For this comparison, the lower bounds range from 00 to 0.75c0.75c for upper cap cc. When the additional grid-spacing condition above holds, we select the fit at the largest lower bound.

Parameterizationλ\lambdaβ≤1\beta\leq1β≤2\beta\leq2β≤4\beta\leq4
SP10−310^{-3}−1.56-1.56−3.56-3.56−7.56-7.56
SP+Embd10−310^{-3}−1.32-1.32−2.33-2.33−5.32-5.32
μ\muP6×10−36\times10^{-3}−1.41-1.41−2.71-2.71−5.41-5.41

All three parameterizations have negative κ\kappa for each cap in the table. Changing the cap affects the separate optimal-learning-rate fit used for κ\kappa; this comparison keeps the E\mathcal{E} and R(∞)\mathcal{R}(\infty) calculations fixed. The magnitudes depend on the cap and the rapid-convergence choice, while the robust-transfer classification is unchanged in this comparison.

Thank you to the authors for the detailed rebuttal and additional experiments! I appreciate that the authors have revised their novelty claims relative to Everett et al. and addressed some of my concerns with the new held-out validation results. While I still have reservations about whether the remaining contributions provide sufficient novelty, I acknowledge these improvements and have raised my rating to 2.

Reviewer_FWcCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors argue that under AdamW the primary training benefits of μ\muP can be achieved under SP by simply adjusting the learning rate of the embedding layer. To support their claims they introduce quantitative metrics to measure the predictability, robustness, and loss degradation for learning rate transfer. Using these metrics they conduct a thorough 16-way ablative analysis to support their claims. They also conduct experiments on LLaMA-style and CNN networks and explore how weight decay affects their conclusions.

Critical strengths
  • Quantification of transfer: Most practical LR transfer analysis is vibes based. Coordinate checks are weak quantitative tools and practitioners often fly by the seat of their pants. This paper offers quantitative tests which can be used to define transfer. Their tests are well-reasoned based on what μ\muP actually is, and I think these tests are strong candidates for valuable and durable tools in the literature.
Critical weaknesses
  • The central finding of their paper, that SP + the correct embedding layer learning rate recovers transferability, is already known (Everett et al. 2024), Section 4.2.1. Therefore, their frequent "surprise" is misplaced. While a dedicated and thorough investigation of this topic is lacking in the literature (e.g. separating the contributions of embedding layer and LayerNorm), the central framing of their work misses the fact that these results have been known for a while (see also Appendix E of Everett). Even the mechanism described here is also known and has been for some time (see Chickering 2025, section on "Implied Dynamics of μ\muP" or Everett Table 1).
  • Building off the previous point: the framing of their work is inaccurate as written, and would be misleading to readers who are not well-versed in the literature and practice of μ\muP.
  • The main track experiments and results seem to agree with results found in the μ\muP community, however some of their appendix results are odd. For example I suspect that their results in Appendix H are a consequence of having practically zero weight decay which, if true, invalidates the conclusions they draw from this experiment (see below).
Questions for authors

To move a weak reject to a weak accept, the authors need to convince me that they will overhaul the framing of their paper, which as it currently sits is inaccurate. It is inaccurate primarily because they frame these results as novel and "surprising", but in reality their central empirical claims have already been published and the "surprising" mechanism is well-known amongst practitioners of μ\muP.

Concerns Affecting the Review

  1. In lines 56-59 you claim that μ\muP "assumes a finite number of training steps in the infinite width limit, whereas practical training runs are in the opposite limit, of a large number of training steps compared to width". But this isn't really true, is it? In practice n≪Tn \ll T, but that's nowhere near either limit: we never take n→∞n\rightarrow \infty, and a large number of training steps doesn't break transfer, as your own fixed-step runs show. In your case scaling only matters in your compute-optimal runs, but in these runs you're fixing batch size, which is already quite an artificial setting. In light of this can you re-examine this statement and make a more accurate claim?
  2. In lines 60-61 you claim that learning rate warm-up is "crucial to observe transfer" for μ\muP. This statement is false and contradicts the literature. Yang's original TPV paper uses a constant learning rate for Figure 4. Can you share the experimental setup for the ablation you run in Appendix I? In particular your Adam ε\varepsilon, the specific μ\muP config, your β1/β2\beta_1/\beta_2, etc.
  3. In lines 63-64 you say "despite these assumptions not holding, μ\muP typically exhibits high-quality learning rate transfer in fixed step settings". Which assumptions do you mean? Going through your list in lines 56-63: in the fixed-step setting TT is held fixed as nn grows (see Q1), every width sees the same dataset, and warmup isn't an assumption of the derivation (and isn't needed for transfer, see Q2). The only one that clearly fails is full alignment, which is well documented (Everett et al. 2024; Kosson et al. 2026). Can you rewrite this passage to say which assumptions you think actually fail in the fixed-step setting?
  4. Can you give me an intuition as to why the functional form (4)(4) captures the "reliability" of transfer? Like why a Taylor expansion is the correct tool for measuring transfer "reliability". Then add this to the paper since I think it will aid readers.
  5. Lines 356-360: You say that both adding 1/d1/d to SP and removing it from μ\muP make "transfer brittle". But with d=64d = 64 fixed, 1/d1/d versus 1/d1/\sqrt{d} is just a constant factor of 8 on the attention logits, not a width scaling (as you note, and as Everett et al. note in their Appendix C.2), so I'd expect re-tuning that constant to absorb it. Did you re-tune it? Appendix C tunes the other layerwise constants but not this one. For what it's worth, in my experience the choice between 1/d1/d and 1/d1/\sqrt{d} for μ\muP is immaterial for learning rate transfer, and even for the loss. Can you either re-tune the constant and re-run, or soften this claim and point to Figures 12-13?
  6. Lines 395-396 and Appendix J: You say "This is surprising, as one might expect later layers to learn useful representations even with a randomly initialized embedding". I don't think the surprise is warranted. With V=50,304≫nV = 50{,}304 \gg n, a large fraction of the network parameters would be frozen at small widths, and a gap that shrinks with width is what I'd expect. Figure 23 (top right) is also exactly what you should expect: SP's embedding learning rate is η/n\eta/n, so SP's embeddings are already nearly frozen, and SP and frozen SP should be very similar, which they are at large width. So I think that you're measuring capacity, not necessarily something about embeddings. Can you comment on this?
  7. I'm also confused by the parameter accounting in Figure 23 (bottom). Your unfrozen counts match an untied model (about 7.1M at n=64n = 64 and 163M at n=768n = 768), but the frozen counts (about 0.66M and 86M) exclude 2Vn2Vn, i.e. both the embedding and the unembedding. Did you freeze both? If only the input embedding is frozen, the frozen points should sit at about 3.9M and 124M, and by my reading of your figure frozen SP is then no better than unfrozen SP at any matched trainable-parameter count, which reverses your claim in lines 1451-1452. Can you clarify which parameters were frozen and correct the plot if needed? Can you clarify your claims on lines 1451-1452?
  8. Lines 466-469 and Appendix H: I don't think these conclusions follow from your experiment. First, in the compute-optimal setting the training horizon grows with width, so the leftward drift of ν∗(n)\nu^*(n) under ηλ=Θ(1)\eta\lambda = \Theta(1) in Figure 21(a) is what you'd expect from the horizon changing, not necessarily a failure of width transfer: when you scale steps but don't scale batch size, the optimal learning rate (or weight decay) has to change. Correcting for this by holding τ=B/(ηλD)\tau = B/(\eta\lambda D) fixed is exactly the prescription of (Wang & Aitchison 2024) and Bergsma et al. (2025), which you cite but describe as "not well motivated" (lines 1331-1333). Framing it as a width exponent, ηλ=Θ(1/n2)\eta\lambda = \Theta(1/n^2), is misleading: it only holds because TT is tied to nn at fixed TPP, and at your widths TT grows like n1.5n^{1.5}, not n2n^2, because the embedding parameters are linear in nn. Second, I don't think the results in panel (b) functionally have any weight decay, and theory says it shouldn't. Under ηλ=Θ(1/n2)\eta\lambda = \Theta(1/n^2) the decay term in each optimizer step vanishes with width, so in the large nn limit AdamW reduces to vanilla Adam. At your widths even the accumulation over steps is negligible: if λn2=0.001\lambda n^2 = 0.001 uses the raw width, then at ν=0\nu = 0 the cumulative decay is ηλT≈4×10−5\eta\lambda T \approx 4\times10^{-5} at n=256n = 256 and ≈1.6×10−5\approx 1.6\times10^{-5} at n=1536n = 1536, versus 2.7 and 37.8 in panel (a). Panel (b) behaves like your λ=0\lambda = 0 runs where you already report negative κ\kappa (line 465), which is exactly what theory predicts. Isn't panel (b) just training without weight decay? If so, it defeats the purpose of an experiment meant to find the right weight decay scaling, and the conclusions in lines 466-469 and 483-485 don't follow from it. Please remove them, or re-run with a weight decay that is actually active at your base width.
  9. Lines 481-483: You say "This suggests that the full set of μ\muP conditions is excessive, and practitioners using SP can recover comparable transfer performance by simply correcting the embedding layer learning rate". This may be true, but it should be qualified with what practitioners leave on the table by not using μ\muP. Can you give a more refined statement about what exactly (if anything) is left on the table when you use SP + embedding scaling but not the other aspects of μ\muP?
  10. How do your metrics compare to the learning rate sensitivity metric of (Wortsman et al. 2023)?
  11. Your metrics implicitly depend on how the learning rate sweep is run and processed. The learning rate grid, filtering threshold, smoothing, and cap all affect your metrics. You variously use f=1.3f=1.3 and f=1.35f=1.35 in the paper. How sensitive are E\mathcal{E}, κ\kappa and R(∞)\mathcal{R}(\infty) to these choices? Do you have guidance for practitioners on how to choose and calibrate them to tell whether they are getting good transfer?

Suggested Additional Citations

  • Lines 48-50: You're missing (Chickering et al. 2026) and (Peng et al. 2026), which are as recent as extensions you already cite (Jiang et al. 2026; Shigida et al. 2026).
  • More broadly, the following papers should be included in your references as they are directly relevant to your work: (Wang & Aitchison 2024), which precedes Bergsma et al. and introduces 1/(ηλ)1/(\eta\lambda) as an EMA timescale for AdamW, and (Wortsman et al. 2023), which defines a quantitative learning rate sensitivity metric and tracks it across scale.

Minor Comments

  • Lines 151 - 154: You describe the Θ(1)\Theta(1) learning rate for the embedding and LayerNorm layers as a "natural choice", which does not communicate that this scaling is arrived at via mathematical derivation. Consider instead "By comparison, the embedding and LayerNorm use a Θ(1)\Theta(1) learning rate since they do not involve a sum over the width dimension". This wording avoids the reader thinking that the learning rate is simply guessed.
  • Equation (6): Change LL to L^\hat{L} to avoid ambiguity with the equation in Definition 3.2.
  • Line 371: Section 5 says it will "dig deeper into why training the embedding layer fast enough is critical", but the switch experiments show when it matters rather than why. Consider rewording.
  • Figure 1: κ\kappa caption disagrees with panel.
  • Appendix E.1: κ\kappa values disagree with Figures 9, 12 and 13.
  • The compute arithmetic gives ~82k GPU-hours, not ~160k.
6 comments
AuthorsOpenReview ↗

Rebuttal (Part 1)

General response

Thank you for the detailed review and for sharing Chickering's blog post. We have revised the framing to distinguish the previously known learning rate prescription from our quantitative analysis and ablations. The responses below clarify the theoretical assumptions and address the warmup, attention, freezing, weight decay, and fitting choices.

Framing relative to Everett et al. (2024)

The central finding of their paper, that SP + the correct embedding layer learning rate recovers transferability, is already known…

Thank you for the suggestions and for sharing the blog post by Chickering (2025). Everett et al. group the embedding and LayerNorm learning rates (their Appendix C.2), so their prescription corresponds to SP+Embd+LN in the width scalings of our 1616 ablations. Our ablations separate these effects and identify the embedding layer learning rate as the key factor: increasing the LayerNorm learning rate in SP worsens instability, while decreasing it in μ\muP has a negligible effect. Chickering's blog post also discusses the role of the embedding learning rate, noting that slow embedding learning in SP can leave the embeddings near their random initialization, add noise to the gradient, and reduce model capacity. We will cite Everett et al. for this learning rate prescription and Chickering's blog post for this argument in the updated version of the paper. Below, we clarify our contributions relative to them.

  • Transfer metrics quantify transfer. Our three transfer metrics separately quantify loss predictability, robustness to learning rate extrapolation errors, and asymptotic performance. Using these metrics we showed that SP, like μ\muP, can exhibit robust learning rate transfer, as quantified by a negative robustness exponent κ\kappa. What limits it is the training instability, as reflected in its larger loss predictability error E\mathcal{E}. By separating transfer robustness from loss predictability, our metrics identify what limits SP's transfer, which is difficult to infer otherwise.

  • Embedding layer learning rate. Our switching experiments additionally show when the embedding learning rate matters. Raising SP's embedding learning rate earlier removes instabilities sooner and improves final loss; lowering μ\muP's produces a larger final loss gap for earlier switches. Switching at late times still affects final loss in both parameterizations.

  • Broader scope: weight decay and other architectures. Everett et al. use no weight decay in their main experiments and test one small constant value in their Appendix G, while Chickering's blog post shows transfer of appropriately scaled weight decay and the EMA timescale at fixed tokens per parameter. Our three metrics show that weight decay can reduce the loss predictability error while worsening fitted asymptotic performance in fixed-step training. At fixed TPP, the asymptotic performance penalty largely disappears for stable parameterizations, but transfer robustness degrades as weight decay increases (Figure 5 of the paper). Furthermore, Appendix F and Appendix G of the paper extend the investigation of the embedding layer learning rate to LLaMA-style Transformers and CNNs.

Q1: Finite TT vs. the infinite-width limit

… a large number of training steps doesn't break transfer, as your own fixed-step runs show.

We would like to clarify that our statement concerns the theoretical derivation of μ\muP, which takes width n→∞n\to\infty with the number of training steps TT fixed (Section 3 of Tensor Programs IV and Section 3.1 of Everett et al.). We did not mean that a large number of training steps breaks learning rate transfer empirically. We have revised the framing to avoid this confusion.

Furthermore, even at fixed TT, order-one update scaling alone does not guarantee optimal learning rate transfer across practical finite widths. For example, [1] shows that the test-optimal learning rate does not transfer across the interpolation threshold in fixed-step experiments.

For further illustration, suppose the updates Δt\Delta_t to an activation coordinate have mean zero, unit variance, and covariance 1/n1/n between distinct steps. Then

Var⁡(∑t=1TΔt)=T+T(T−1)n.\operatorname{Var}\left(\sum_{t=1}^{T}\Delta_t\right)=T+\frac{T(T-1)}{n}.

At fixed TT, this converges to TT as n→∞n\to\infty, but the correlation term can dominate at finite widths when T≫nT\gg n. Thus, order-one updates at individual steps can accumulate with a dependence on width through their correlations, even when TT is held fixed across widths, and as a result, learning rates may not transfer.

References

[1] D. S. Kalra and M. Barkeshli. Learning Rates Do Not Transfer Across Double Descent. High-dimensional Learning Dynamics, 2026.

AuthorsOpenReview ↗

Rebuttal (Part 2)

Q2: Warmup

Can you share the experimental setup for the ablation you run in Appendix I?

We did not mean warmup as a requirement for μ\muP transfer, and we have rewritten this passage to avoid confusion. Warmup is known to reduce training instabilities and enables training at large learning rates [1,2,3]. When training is stable, μ\muP transfers without warmup, and we speculate that Figure 4 of Tensor Programs V [4] is one such case. When training is inherently unstable, warmup improves transfer. The experiment in Appendix I of the paper is an example of the second case. The shared architecture, training horizon, and optimizer settings are described in Appendix B. Here, we used AdamW with β1=0.9\beta_1=0.9, β2=0.95\beta_2=0.95 and ϵ=10−8\epsilon=10^{-8}, which are standard choices for language model pretraining, with no weight decay and global gradient clipping at 11. With these standard AdamW hyperparameters and 1% warmup, training is unstable and transfer is poor. In this experiment, warmup gives consistent transfer without width-dependent AdamW hyperparameters. We have revised the text to avoid confusion and described the warmup schedule in Appendix I.

References

[1] J. Gilmer, B. Ghorbani, A. Garg, S. Kudugunta, B. Neyshabur, D. Cardoze, G. E. Dahl, Z. Nado, and O. Firat. A Loss Curvature Perspective on Training Instabilities of Deep Learning Models. ICLR, 2022.

[2] D. S. Kalra and M. Barkeshli. Why Warmup the Learning Rate? Underlying Mechanisms and Improvements. NeurIPS, 2024.

[3] L. Liu, H. Jiang, P. He, W. Chen, X. Liu, J. Gao, and J. Han. On the Variance of the Adaptive Learning Rate and Beyond. ICLR, 2020.

[4] G. Yang, E. J. Hu, I. Babuschkin, S. Sidor, X. Liu, D. Farhi, N. Ryder, J. Pachocki, W. Chen, and J. Gao. Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer. arXiv:2203.03466, 2022.

Q3: Which assumptions fail

Can you rewrite this passage to say which assumptions you think actually fail in the fixed-step setting?

We would like to clarify that our statement concerns the assumptions used in the theoretical derivation. We did not mean warmup as an assumption of the derivation; it is an empirical consideration for training stability. We have updated the text to make these distinctions explicit and avoid confusion.

Q4: Why a Taylor expansion measures transfer reliability

Can you give me an intuition as to why the functional form (4) captures the "reliability" of transfer?

Near a smooth minimum of the loss versus log learning rate ν\nu, the leading variation in loss is quadratic (Equation 4 of the paper). We examine two aspects of transfer reliability:

  • Loss predictability. If the local quadratic approximation combined with scaling laws accurately describes the loss curves across widths, it provides a basis for predicting the loss at larger widths. The loss predictability error E\mathcal{E} measures how well this form describes the observed losses. Training instabilities, for example, can make the curves irregular and poorly described by this form, increasing E\mathcal{E} and making loss prediction unreliable.

  • Robustness to learning rate errors. For reliable transfer, small errors in the optimal learning rate estimated at small widths should not be amplified into large normalized loss penalties at larger widths. For an error δν\delta\nu in the transferred log learning rate relative to the optimum at width nn, the leading loss penalty is:

    ΔL≈12H(n) δν2,\Delta L\approx\frac{1}{2}H(n)\,\delta\nu^2,

    where H(n)H(n) is the curvature of the loss versus ν\nu at the optimum. For the same learning rate error, a larger curvature produces a larger loss penalty. Our transfer robustness exponent κ\kappa combines the width scaling of curvature, optimal learning rate, and minimum loss to quantify whether the normalized penalty grows or shrinks with width.

The quadratic approximation therefore lets us assess both the predictability of the loss and its sensitivity to learning rate errors around the minimum.

Q5: Attention 1/d1/d vs. 1/d1/\sqrt{d}

Can you either re-tune the constant and re-run, or soften this claim and point to Figures 12-13?

We agree that, with head dimension d=64d=64 fixed, 1/d1/d and 1/d1/\sqrt{d} differ by a constant factor of eight that can be absorbed into the attention constant. We have now retuned this constant for SP and μ\muP. In this sweep, with weight decay λ=10−3\lambda=10^{-3}, the best scanned constants vary by at most a factor of two across widths 128128–10241024 for both parameterizations (Appendix L of the updated text). We will revise the statement to describe the results for the fixed attention constants used in the original ablation.

AuthorsOpenReview ↗

Rebuttal (Part 3)

Q6: Freezing and capacity

So I think that you're measuring capacity, not necessarily something about embeddings. Can you comment on this?

We would like to clarify that our statement concerns the comparison by trainable parameter count in Figure 23 in Appendix J of the updated text. When the frozen and unfrozen μ\muP cases are compared by trainable parameter count, the frozen case still has a higher loss. Thus, the additional trainable parameters in the hidden layers do not compensate for freezing the embedding in our experiments. The similarity between SP and frozen SP at large widths is expected because SP's embedding learning rate is already small. We have revised the text to make this comparison explicit and avoid confusion.

Q7: Parameter accounting in Figure 23

Can you clarify which parameters were frozen and correct the plot if needed?

Thank you for the careful reading. We found a bug in the embedding parameter counting function: it also included the readout parameters, which were therefore incorrectly subtracted from the trainable parameter count. The corrected count excludes the token and positional embeddings in the frozen case and includes the readout parameters. We have corrected Figure 23 in Appendix J of the updated text and the associated claims and text. Our response to Q6 remains unchanged: when compared by trainable parameter count, μ\muP with frozen embeddings still has a higher loss than μ\muP with trained embeddings. This correction does not affect any results presented in the main paper.

Q8: Appendix H, weight decay at fixed TPP

… when you scale steps but don't scale batch size, the optimal learning rate (or weight decay) has to change. … Isn't panel (b) just training without weight decay?

We would like to clarify that our statement concerns the fixed-step intuition behind the standard weight decay convention ηλ=Θ(1)\eta\lambda=\Theta(1). This intuition does not determine the appropriate weight decay scaling when the number of training steps grows with width, motivating our empirical investigation in the fixed TPP setting. Our TPP budget excludes token embedding and readout parameters, giving approximately quadratic growth in the number of training steps with width. We will modify the statement to make this motivation explicit and avoid confusion.

Thank you for noticing that the weight decay coefficient in the additional experiment may be too small. We have removed this experiment in Appendix H and Figure 21 of the paper. The observed degradation of transfer robustness with increasing weight decay in the fixed TPP setting remains unchanged (Figure 5 of the paper).

Q9: What is left on the table without full μ\muP

Can you give a more refined statement about what exactly (if anything) is left on the table when you use SP + embedding scaling but not the other aspects of μP?

As quantified by our three transfer metrics, we do not observe a consistent disadvantage of SP+Embd relative to μ\muP in our experiments, which extend to approximately 11B parameters. It would be interesting to investigate whether a gap between μ\muP and SP+Embd emerges at larger scales. We leave this for future work.

Q10: Comparison with Wortsman et al. (2023)

How do your metrics compare to the learning rate sensitivity metric of (Wortsman et al. 2023)…

Wortsman et al. (2023) define the learning rate sensitivity as

S(n)=Eν[min⁡(L^(ν;n),L0)−L^∗(n)],S(n)=\mathbb{E}_{\nu}\Big[\min\big(\hat{L}(\nu;n), L_{0}\big)-\hat{L}^{\ast}(n)\Big],

where L0L_{0} is the loss at initialization, L^∗(n)\hat{L}^{\ast}(n) is the optimal loss at size nn, and the expectation averages over learning rates. They track SS as model size grows. Their metric is model-free, as it directly measures the capped deviation L^(ν;n)−L^∗(n)\hat{L}(\nu;n)-\hat{L}^{\ast}(n) from the optimal loss. This metric is in spirit related to our loss predictability error E\mathcal{E} (Definition 3.2 of the paper): both average, over learning rates, the deviation of the observed loss from a reference. SS is model-free and uses the optimal loss L^∗(n)\hat{L}^{\ast}(n) as the reference. Dropping the initialization-loss cap, replacing L^∗(n)\hat{L}^{\ast}(n) by the prediction L(ν;n)L(\nu;n) of our quadratic model with the scaling-law ansatz (Equation 6 of the paper), squaring the deviation, and averaging over widths as well on the retained observations gives

E=1NνNn∑i,j[L^(νi;nj)−L(νi;nj)]2.\mathcal{E}=\frac{1}{N_{\nu} N_{n}}\sum_{i,j}\Big[\hat{L}(\nu_{i};n_{j})-L(\nu_{i};n_{j})\Big]^2.

Therefore, SS measures how far the loss deviates from its optimum, while E\mathcal{E} measures how much of this deviation our quadratic model with the scaling-law ansatz fails to capture across widths.

Reviewer_wLGzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes three metrics for learning-rate transfer across width, all derived from fits of the final loss as a function of log learning rate ν and width n (Eq. 6): the in-sample fit error E\mathcal{E}, an exponent κ=α−2β+γ\kappa = \alpha - 2\beta + \gamma meant to capture how errors in the transferred learning rate affect the loss at scale, and an extrapolated asymptotic loss gap R(∞)\mathcal{R}(\infty). With these metrics, the authors compare SP (with a global 1/n1/n learning rate) and μP for GPT-2-style Transformers trained with AdamW on FineWeb-Edu (widths 128–2048, 10k steps, learning-rate and weight-decay sweeps), and ablate all 16 combinations of the four differences between them. They conclude that the embedding learning rate accounts for μP's advantage: SP with a Θ(1) embedding learning rate matches μP, and a slow embedding layer produces loss spikes, especially early in training (switching and freezing experiments, plus small LLaMA-style and CNN experiments in the appendix).

Critical strengths

The grid of 16 differences between MuP and SP is interesting. The raw loss curves show that SP's loss spikes near the optimal learning rate disappear when only the embedding learning rate is raised.

Critical weaknesses

The main finding overlaps substantially with prior work that the paper cites but does not give credit Everett et al. (2024, arXiv:2407.05872), write in their introduction: "we scale the learning rate for the embedding layer as O(1) and the learning rate for hidden and readout layers as O(1/n) for standard parameterization, and this outperforms all other combinations of parameterizations and learning rate prescriptions for Adam". However, this is not properly credited in this paper. In the abstract, the authors write "We find that the overwhelming benefit of muP relative to SP when training with AdamW arises simply from maximizing the learning rate of the embedding layer", which is not novel and not a finding. However, this is the main practical take of the paper.

The proposed metrics are not properly validated The metrics rely on fitting several scaling laws and subsequently computing quantities from their estimated parameters. Although the authors acknowledge parameter degeneracies in Appendix D and introduce heuristics to address them, they do not demonstrate that the resulting estimates are robust to measurement noise or fitting choices.

This is particularly concerning for the robustness exponent (κ=α−2β+γ\kappa=\alpha-2\beta+\gamma), whose sign determines whether transfer is deemed robust. Small errors in estimating the individual exponents could potentially change this conclusion. The authors should quantify the uncertainty of these estimates, for example through bootstrap experiments or perturbations of the measured losses, and demonstrate that their conclusions remain stable.

Furthermore, the loss predictability error (E) is evaluated on the same underlying measurements used for fitting, rather than on held-out model widths. A low fitting error therefore does not establish reliable extrapolation.

Since these metrics constitute a central contribution of the paper, their statistical reliability and predictive power should be demonstrated experimentally. Without such validation, it is hard to assess whether the proposed framework provides meaningful information beyond the original measurements.

Questions for authors

See above.

3 comments
AuthorsOpenReview ↗

Rebuttal (Part 1)

Thank you for the careful review. We have revised the framing to clarify the distinction from Everett et al. and discuss their parameterization in further detail. We also provide held-out loss predictions and loss-perturbation analyses to assess the transfer metrics.

W1: Overlap with Everett et al. (2024)

The main finding overlaps substantially with prior work that the paper cites but does not give credit…

Thank you for raising this. Everett et al. relaxed μ\muP's alignment assumption for the last layer and showed that SP transfers with Θ(1)\Theta(1) embedding and Θ(1/n)\Theta(1/n) hidden and readout learning rates. This relaxation accounts for one of the four differences between SP and μ\muP, the last-layer initialization, and leaves three: the embedding learning rate, the LayerNorm learning rate and the attention scale.

They scale the embedding and LayerNorm learning rates together (their Appendix C.2 places token embeddings, positional embeddings and LayerNorm parameters in one group). We instead take an empirical approach and ablate all 16 combinations of the four differences, which Reviewer FWcC identifies as a gap in the literature. Among these 16 ablations, their SP corresponds to SP+Embd+LN in its width scalings. Relative to Everett et al., our paper makes the following contributions.

The embedding layer learning rate controls training stability.

As you note, SP's loss spikes near the optimal learning rate disappear when only the embedding learning rate is raised. Our ablations identify the embedding layer learning rate as the key factor: increasing the LayerNorm learning rate in SP worsens instability, while decreasing it in μ\muP has a negligible effect.

We further examine how the embedding layer affects training by switching its learning rate during training. Raising SP's embedding learning rate early in training eliminates its instabilities and improves its loss, while lowering μ\muP's causes a lasting loss gap that is larger the earlier the switch. A small embedding learning rate therefore not only slows down training but also causes instabilities, with early training being the most critical period.

AuthorsOpenReview ↗

Rebuttal (Part 2)

W1 (continued)

The main finding overlaps substantially with prior work…

Quantifying transfer.

Everett et al. assess transfer mainly through how the optimal learning rate scales with width. We introduce three metrics that measure transfer beyond the location of the optimal learning rate: the loss predictability error E\mathcal{E}, the robustness exponent κ\kappa, which measures how errors in the transferred learning rate affect the loss at scale, and the asymptotic loss gap R(∞)\mathcal{R}(\infty).

Separating these three metrics lets us identify loss predictability as a limiting factor in SP's transfer. The observed loss spikes and ablations that change only the embedding learning rate connect this limitation to training instability caused by SP's small embedding learning rate. Our framework therefore provides a quantitative diagnosis of the aspect of transfer improved by the embedding learning rate prescription. Reviewers FWcC and CRvy find these metrics useful, and we further validate them on held-out widths in W2–W3.

Broader scope: weight decay and other architectures.

Everett et al. use no weight decay in their main experiments and check a single small constant weight decay of 10−410^{-4} in their Appendix G. By comparison, we sweep weight decay from 0 to 10−210^{-2} and find that it mainly improves the loss predictability error E\mathcal{E}, at the cost of a larger asymptotic loss gap R(∞)\mathcal{R}(\infty) in the fixed-step setting.

Furthermore, Appendix F of the paper extends the embedding result to LLaMA-style Transformers, where SP shows a large training-loss spike at its optimal learning rate that both μ\muP and SP+Embd avoid. Appendix G extends it to CNNs, where SP with a Θ(1)\Theta(1) input-layer learning rate matches μ\muP's transfer quality.

In response to your concern about the abstract, we have revised the sentence to: “Our ablations identify the embedding layer learning rate as the primary factor in μ\muP's improved training stability and learning rate transfer relative to SP under AdamW.” We have further enhanced the discussion of Everett et al. in the introduction and moved the related work section from the Appendix to the main text.

W2: Robustness of the fitted metrics

Small errors in estimating the individual exponents could potentially change this conclusion.

Thank you for your suggestion. In response, we performed the analysis. To estimate the noise level, we repeated 10,000-step training with 3 random seeds for SP, SP+Embd and μ\muP at p=7p=7 learning rates, log⁡2η∈{−2,−1.5,−1,−0.5,0,0.5,1}\log_2\eta\in\{-2,-1.5,-1,-0.5,0,0.5,1\}, and q=2q=2 widths (256 and 1024), with independent weight decay λ=10−3\lambda=10^{-3}.

For each width nn and learning rate η\eta, let L^r(n,η)\hat L_r(n,\eta) be the final loss for seed rr and L^‾(n,η)\overline{\hat L}(n,\eta) its three-seed mean. The sample variance is s(n,η)2=13−1∑r=13[L^r(n,η)−L^‾(n,η)]2s(n,\eta)^2=\frac{1}{3-1}\sum_{r=1}^{3}[\hat L_r(n,\eta)-\overline{\hat L}(n,\eta)]^2. We summarize the noise by the pooled standard deviation sˉ=(1pq∑n,ηs(n,η)2)1/2\bar{s}=\big(\frac{1}{pq}\sum_{n,\eta}s(n,\eta)^2\big)^{1/2}. The pooled standard deviation is 0.008 for μ\muP and SP+Embd and 0.044 for SP.

Each bootstrap sample adds independent Gaussian noise with standard deviation sˉ\bar{s} of the corresponding parameterization to every final loss in the original sweep before repeating the loss filtering and fitting procedure. The original-sweep fits use λ=10−3\lambda=10^{-3} for SP and SP+Embd and λ=6×10−3\lambda=6\times10^{-3} for μ\muP; the μ\muP noise estimate at λ=10−3\lambda=10^{-3} is applied at λ=6×10−3\lambda=6\times10^{-3}. For κ\kappa, we additionally use the fitting convention based on the learning rate grid resolution described in Appendix L of the updated text. These results are summarized in Appendix L of the updated text.

AuthorsOpenReview ↗

Rebuttal (Part 3)

W2 (continued)

The authors should quantify the uncertainty of these estimates…

Under the measured noise, the loss predictability ordering remains the same. The tables report 100 samples per parameterization; the 90% intervals are the 5th and 95th percentiles. SP's E\mathcal{E} is 2.5×2.5\times that of μ\muP (medians 0.0085 and 0.0035), consistent with the roughly 3×3\times reported in the paper, and the ordering SP+Embd << μ\muP << SP holds in every pair of bootstrap samples.

Bootstrap results for E\mathcal{E} (100 samples per parameterization). Original: fit to the original losses without added noise.

ParameterizationOriginalMedian90% interval
μ\muP0.00340.0035[0.0033,0.0036][0.0033,0.0036]
SP+Embd0.00050.0006[0.0005,0.0006][0.0005,0.0006]
SP0.0110.0085[0.0067,0.014][0.0067,0.014]

For R(∞)\mathcal{R}(\infty), we compare SP and SP+Embd at λ=10−3\lambda=10^{-3} with μ\muP at λ=6×10−3\lambda=6\times10^{-3}. The best asymptotic loss reference is held fixed at its fit to the unperturbed data. For each comparison, we compute Δij=RA(i)(∞)−RμP(j)(∞)\Delta_{ij}=\mathcal{R}_{A}^{(i)}(\infty)-\mathcal{R}_{\mu\mathrm{P}}^{(j)}(\infty), where AA is SP or SP+Embd, over all pairs of the two independently generated sets of 100 bootstrap samples. We report the median difference and its 90% perturbation interval, defined by the 5th and 95th percentiles of these differences. SP+Embd has a median difference of 0.070.07, with an interval of [−0.02,0.16][-0.02,0.16]. SP has a median difference of 0.280.28, with an interval of [−0.02,0.45][-0.02,0.45]. Both intervals include zero.

Bootstrap results for R(∞)\mathcal{R}(\infty) (100 samples per parameterization). Original: fit without added noise.

ParameterizationOriginalMedian90% interval
μ\muP0.240.22[0.15,0.28][0.15,0.28]
SP+Embd0.270.28[0.22,0.35][0.22,0.35]
SP0.440.49[0.21,0.65][0.21,0.65]

For κ\kappa, μ\muP and SP+Embd are robust under the additional fitting convention in Appendix L of the updated text: κ\kappa is negative in every bootstrap sample, with medians −2.77-2.77 and −2.32-2.32 and upper ends of the 90% interval at −2.64-2.64 and −2.27-2.27. SP also has a negative median of −2.52-2.52, but κ\kappa is positive in 11% of samples, with the upper end reaching 0.520.52.

Bootstrap results for κ\kappa under this fitting convention (100 samples per parameterization). Original: fit without added noise.

ParameterizationOriginalMedian90% interval
μ\muP−2.71-2.71−2.77-2.77[−3.44,−2.64][-3.44,-2.64]
SP+Embd−2.33-2.33−2.32-2.32[−3.32,−2.27][-3.32,-2.27]
SP−3.56-3.56−2.52-2.52[−3.65,0.52][-3.65,0.52]

W3: E\mathcal{E} is evaluated in-sample

A low fitting error therefore does not establish reliable extrapolation.

Thank you for raising this point. In response, we tested E\mathcal{E} on held-out widths. We repeated the paper's fitting procedure using only widths 128–1024 and computed E\mathcal{E} on the held-out widths 1536 and 2048, keeping the fitted parameters fixed. At each held-out width we retain raw losses within 1.351.35 times its observed minimum. The results are shown in Figure 24 of Appendix L of the updated text.

Held-out E\mathcal{E} follows the same trends as E\mathcal{E} reported in Figure 5b of the paper: μ\muP and SP+Embd exhibit low held-out E\mathcal{E}, whereas SP and other unstable variants exhibit a higher value. Across the 27 combinations of parameterization and weight decay in the figure, the Spearman rank correlation between held-out E\mathcal{E} and E\mathcal{E} reported in the paper is 0.870.87, and held-out E\mathcal{E} is a median 1.46×1.46\times higher than E\mathcal{E} reported in the paper. These results support the same loss predictability trends on the held-out widths.

The held-out point for μ\muP-Embd at λ=10−4\lambda=10^{-4} is omitted because fewer than four valid training widths remained after interpolation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.