- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper derives muP parameterization for Gated Delta Networks for AdamW and SGD, and then confirms empirically that the usual learning rate transfer across scale holds.
Critical strengths
Clarity: The paper is clearly written and straightforward with its contribution to the community.
Impact/Informativeness: muP is extremely practical in practice, generally because it defines how to parameterize and initialize models to enable learning rate transfer and other desirable feature learning properties. Computing the muP for Gated Delta Networks is therefore impactful.
Soundness: The paper seems sound, and the empirical experiments are a good sanity check that the derived parameterization is in fact correct.
Critical weaknesses
The main weakness of this paper is just its limited novelty and therefore potential lack of impact. While muP for Gated Delta Networks may be novel, muP has already been computed for a wide range of architectures. It's not clear to me whether this paper adds any substantially new technical argument, or whether the technical argument mainly follows the pattern of prior work.
Questions for authors
Why doesn't Figure 1 compare to the original muP configuration?
3 comments
I would keep my score.
Thank you again for your recognition of our work!
We thank the reviewer for recognizing the empirical value of the study. Here are the responses to your questions
A1: Original µP gives rules for conventional input, hidden, and output transformations. GDN introduces a matrix-valued recurrent state and distinct scalar decay/write pathways. Their gradients depend on the recurrent adjoint and innovation contractions, and cannot be classified correctly using fan-in/fan-out alone. In particular, our SGD analysis yields$\eta_{W_\alpha},\eta_{W_\beta}=\Theta(d^{-1/2}),\quad \eta_{a_{\log}},\eta_b=\Theta(d^{1/2}),$ in contrast to Θ(1) for ordinary hidden projections. This difference is absent in the usual hidden-weight rule and is not supplied by analyses of diagonal SSMs, whose state transitions lack GDN's matrix-valued rank-one delta update. Appendix D further demonstrates how changing the gating structure changes the backward-scaling analysis, even when forward coordinate orders appear similar. There is also direct practical evidence: Figure 2 compares SP, original µP, and our parameterization under momentum SGD; the proposed rule shows the most consistent optimal learning rate across widths. Appendix E extends this check to width 1536: validation loss is 2.9737 / 2.9511 / 2.9540 at learning rates 0.3 / 0.4 / 0.5, and additional seeds at smaller widths show the same preferred SGD learning rate 0.4. These are results already contained in the submitted manuscript. They reinforce that the new gate-specific exponents are consequential, not merely cosmetic.
A2: The two optimizer cases should be distinguished. Under AdamW, our proposed learning-rate exponents coincide with the conventional µP fan-in rules for the parameter categories in Table 2. In particular, Wα,Wβ fall under the hidden-weight 1/d rule, while a{log},b retain width-independent learning rates. A separately labeled “original µP learning-rate rule” would therefore not constitute a distinct third prediction of optimal LR-versus-width scaling in Figure 1. The comparison there is deliberately SP versus the resulting AdamW µP configuration. Under SGD, the original and gate-specific prescriptions genuinely differ, which is why Figure 2 contains all three. For clarity, we will state the shared AdamW learning-rate exponents in the Figure 1 caption and refer to the existing Table 2. This is a comparison of scaling prescriptions, not an unsupported assertion that any two implementations with different RMSNorm epsilon conventions are numerically identical. The architectural normalization choice is analyzed separately below; a repeated full AdamW sweep is not needed to establish that the tabulated LR exponents agree.