- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies hyperparameter transfer across model widths, focusing on why Maximal Update Parameterization (P) provides better learning rate transfer than Standard Parameterization (SP). The authors first propose three metrics to quantify transfer quality, including loss predictability, robustness to extrapolation errors, and asymptotic loss degradation. They then systematically ablate four differences between P and SP, and find that increasing the embedding layer learning rate in SP is sufficient to achieve similar transfer quality as P. Additional experiments investigate the effect of embedding learning rate during training, weight decay, and compute-optimal scaling.
Critical strengths
- The paper provides a systematic empirical analysis of the differences between P and SP. The 16 parameterization ablations help isolate the effect of different components, and the experiments on changing embedding learning rate during training provide additional evidence for its importance.
- The proposed three metrics provide a useful perspective on hyperparameter transfer beyond whether the optimal learning rate remains constant across widths. I think considering predictability, robustness, and asymptotic performance together is reasonable, although the reliability of these metrics still needs stronger validation.
Critical weaknesses
- My main concern is the limited novelty compared with Everett et al. (2024). Their work has already proposed the same learning rate scaling for SP under Adam, where the embedding learning rate scales as and the hidden and readout learning rates scale as . They also showed that this configuration achieves hyperparameter transfer and can outperform µP, with experiments on much larger models. This is essentially the same as the main finding of SP+Embd in this paper. Although Everett et al. (2024) is cited, the paper does not sufficiently discuss this overlap or explain what new understanding is provided beyond the prior work. The additional ablations are useful, but I am not convinced they constitute a sufficiently novel contribution given that the main result has already been demonstrated.
- I am also not fully convinced by the validation of the proposed transfer metrics. The predictability error is evaluated on the same model widths used for fitting, so a good fit does not necessarily demonstrate accurate extrapolation to larger models. Similarly, the robustness exponent and asymptotic loss degradation rely on fitted scaling laws, but the paper does not directly validate these predictions on held-out larger widths. The fitting also involves several heuristic choices, including exponent clipping and handling degenerate solutions, whose effect on the conclusions is unclear. Since the quantitative framework is another main contribution of this paper, I think stronger evidence is needed to show that these metrics reliably predict actual hyperparameter transfer.
Questions for authors
Here are my main questions and concerns. I would be happy to reconsider my opinion if the major concerns are resolved.
- How does the main finding differ from Everett et al. (2024), particularly Section 4.2.1? That work already proposed the same embedding and hidden layer learning rate scaling for SP, and demonstrated that it achieves transfer and outperforms P. I would like the authors to clarify what is fundamentally new beyond the additional ablation experiments.
- Can the authors validate the proposed metrics using held-out model widths? For example, fitting the scaling laws on smaller models and testing whether and actually predict the loss penalty from transferred learning rates on larger models would be helpful. I would also like to understand how sensitive the conclusions are to the fitting choices, especially the exponent clipping and degeneracy handling.
4 comments
Rebuttal (Part 2/3)
W2/Q2: Held-out validation
Can the authors validate the proposed metrics using held-out model widths?
Thank you for raising this point.
Loss predictability on held-out widths.
We refit the loss model in Equation 6 of the paper using only widths , , , , and , and compute on the held-out widths and . The fitted parameters are held fixed when predicting the held-out losses. We use the paper's per-width loss filter and compute the mean squared error on the retained observations. Figure 24 of Appendix L of the updated text compares the held-out with reported in Figure 5b of the paper for seven parameterizations and four weight decays. One held-out fit is unavailable, leaving 27 combinations with both errors available. Across these 27 combinations, the Spearman rank correlation between the two errors is , and the median ratio of held-out to the paper's is .
Loss penalty from a transferred learning rate.
We freeze the separately fitted scaling laws from the same five widths and select the learning rate with minimum raw final validation loss at width . This gives for SP, for SP+Embd, and for P at the weight decays in the table. Let be its fixed log learning rate and the minimum observed loss over the scanned learning rates at width . We compare the observed loss penalty with the prediction
where the parameters of the fitted curvature and optimal-learning-rate laws remain fixed when evaluating the held-out widths. The transferred learning rates are exact scanned points, and all six losses pass the paper's filter.
Held-out loss penalty from reusing the width- optimal scanned learning rate.
| Parameterization | Observed | Predicted | ||
|---|---|---|---|---|
| SP | ||||
| SP | ||||
| SP+Embd | ||||
| SP+Embd | ||||
| P | ||||
| P |
SP's prediction is poor at width . The maximum absolute prediction error across the two widths is for SP, for SP+Embd, and for P. These are predictions from the fitted curvature and optimal-learning-rate laws together; the sign of alone does not predict the penalty from reusing a fixed raw learning rate. We report these checks in Appendix L of the updated text.
Optimal-loss predictions.
We also freeze the optimal-loss scaling law fitted to the five training widths and compare its prediction with the held-out scanned minimum . The prediction error is . As a baseline, we carry the width- minimum forward unchanged, giving error .
Held-out optimal-loss prediction error and the width- baseline.
| Parameterization | Error | Baseline error | |||
|---|---|---|---|---|---|
| SP | |||||
| SP | |||||
| SP+Embd | |||||
| SP+Embd | |||||
| P | |||||
| P |
The fitted prediction has smaller error than this baseline in five of six comparisons; SP at width is the exception. These checks assess finite-width predictions from one run per configuration. We report them in Appendix L of the updated text.
Rebuttal (Part 3/3)
W2/Q2 continued: Transfer robustness
I would also like to understand how sensitive the conclusions are to the fitting choices, especially the exponent clipping and degeneracy handling.
Transfer robustness exponent.
We also compare estimated from the separate scaling laws using five widths (–) and all seven widths (–), with the paper's loss filter and the fitting procedure in Appendix D. All three signs remain negative, while the magnitudes and the ordering of SP+Embd and P change. This checks the stability of the exponent estimate as larger widths are added; it is not an independent prediction of the loss penalty from reusing a smaller model's learning rate.
Transfer robustness exponent from five and seven widths.
| Parameterization | Five widths | Seven widths | |
|---|---|---|---|
| SP | |||
| SP+Embd | |||
| P |
We report this comparison in Appendix L.4 of the updated text.
Exponent clipping and degeneracy handling
Appendix D of the paper caps the scaling exponents at as a practical proxy for exponents becoming arbitrarily large. For , the bound represents rapid convergence. At , each doubling of width reduces the remaining correction to the optimal log-learning-rate by a factor of four. We assess whether the remaining change is resolved by the scanned learning rates by comparing the fitted change across widths with the grid spacing.
Our formalism expresses as an infinite-width value plus a correction . When approaches zero, the correction becomes nearly constant, and the fit cannot determine and separately. When is zero, the optimal learning rate is constant and the data do not determine . Appendix D addresses this ambiguity by increasing the lower bound on and comparing the resulting fitted-exponent trend with a step function and a linear function. When the step function fits better, we select the best solution on the rapid-convergence branch. Otherwise, we retain the small- solution.
For the new perturbation analysis of , Appendix L.3 of the updated text uses an additional fitting convention, distinct from the submitted degeneracy rule. Let be the smallest adjacent spacing among distinct log learning rates with finite losses in the unperturbed sweep, and let and be the smallest and largest fitted widths. If the procedure in Appendix D retains the fit with the lowest mean squared error over the lower bounds, and its fitted change is below , we select the fit with lower bound and upper bound . The curvature and optimal-loss fits are retained. This additional choice does not change the loss filtering or the and calculations. The new perturbation results are conditional on this convention; the five-versus-seven-width comparison uses Appendix D.
Exponent-cap sensitivity.
We compare upper caps of and on with the paper's cap of , keeping and fixed and refitting the optimal-learning-rate law. For this comparison, the lower bounds range from to for upper cap . When the additional grid-spacing condition above holds, we select the fit at the largest lower bound.
| Parameterization | ||||
|---|---|---|---|---|
| SP | ||||
| SP+Embd | ||||
| P |
All three parameterizations have negative for each cap in the table. Changing the cap affects the separate optimal-learning-rate fit used for ; this comparison keeps the and calculations fixed. The magnitudes depend on the cap and the rapid-convergence choice, while the robust-transfer classification is unchanged in this comparison.
Thank you to the authors for the detailed rebuttal and additional experiments! I appreciate that the authors have revised their novelty claims relative to Everett et al. and addressed some of my concerns with the new held-out validation results. While I still have reservations about whether the remaining contributions provide sufficient novelty, I acknowledge these improvements and have raised my rating to 2.
Rebuttal (Part 1/3)
Thank you for the detailed review and for recognizing the value of the ablations and transfer metrics. We clarify what our metrics and experiments add relative to Everett et al. and evaluate the fitted loss model on held-out widths. We also clarify exponent clipping and degeneracy handling below.
W1/Q1: Novelty relative to Everett et al. (2024)
Thank you for raising this. Everett et al. relaxed μP's alignment assumption for the last layer and showed that SP transfers with Θ(1) embedding and Θ(1/n) hidden and readout learning rates (their Section 4.2.1). We will cite Everett et al. for this learning rate prescription. This relaxation concerns one of the four differences between SP and μP, the last-layer initialization, and leaves three: the embedding learning rate, the LayerNorm learning rate, and the attention scale.
Our paper makes the following contributions beyond the additional ablations.
Quantifying transfer.
As you note, our three metrics provide a perspective on transfer beyond whether the optimal learning rate remains constant across widths. The loss predictability error E measures how accurately the loss landscape is captured by Equation 6 of the paper. The transfer robustness exponent κ measures how errors in the extrapolated learning rate affect the loss at scale. The asymptotic loss degradation R(∞) measures the performance gap relative to the best asymptotic loss. Together, these quantify loss predictability, transfer robustness, and asymptotic performance. Separating these three metrics lets us identify loss predictability as a limiting factor in SP's transfer. The observed loss spikes and ablations that change only the embedding learning rate connect this limitation to training instability caused by SP's small embedding learning rate. Our framework therefore provides a quantitative diagnosis of the aspect of transfer improved by the embedding learning rate prescription.
The embedding layer learning rate controls training stability.
Everett et al. scale the embedding and LayerNorm learning rates together: their Appendix C.2 groups token embeddings, positional embeddings, and LayerNorm scale parameters. Our ablations separate these effects across all 16 combinations of the four differences between SP and μP. These ablations identify the embedding layer learning rate as the key factor: increasing the LayerNorm learning rate in SP worsens instability, while decreasing it in μP has a negligible effect.
Furthermore, Figure 4 of the paper examines when the embedding learning rate matters by switching it during training at width 768, with λ=10−3 for SP and λ=6×10−3 for μP. Raising SP's embedding learning rate early eliminates its instabilities and improves its loss, while lowering μP's causes a lasting loss gap that is larger the earlier the switch. These experiments identify early training as the critical period for the embedding learning rate.
Broader scope: weight decay and other architectures.
Everett et al. use no weight decay in their main experiments and examine one constant weight decay in their Appendix G. We examine transfer across eight weight decays, quantifying its effect on loss predictability, transfer robustness, and asymptotic performance. Appendix F and Appendix G of the paper extend the investigation to LLaMA-style Transformers and CNNs, respectively.