acceptodds
Under review as a conference paper at ICLR 2027

Unlocking Feature Learning in Gated Delta Networks at Scale

Abstract

Training and scaling Large Language Models demand enormous computational resources, motivating both efficient sub-quadratic architectures and principled hyperparameter tuning methods. While the Maximal Update Parametrization (P) has enabled zero-shot hyperparameter transfer for standard Transformers, its extension to linear models, particularly those with structured state transitions and complicated architectures, remains largely unexplored. By rigorously propagating coordinate-size estimates through the forward pass, gating mechanisms, and recurrent state dynamics, we derive the scaling rules for Gated Delta Network. Experiments on language-model pre-training confirm that our configurations enables stable learning-rate transfer across model widths under both AdamW and SGD, whereas standard parametrization fails to transfer, validating the correctness and practical utility of our analysis.

Primary area:
optimization
Keywords:
Linear attention, hyperparameter, optimizer

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_TWjKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper derives muP parameterization for Gated Delta Networks for AdamW and SGD, and then confirms empirically that the usual learning rate transfer across scale holds.

Critical strengths

Clarity: The paper is clearly written and straightforward with its contribution to the community.

Impact/Informativeness: muP is extremely practical in practice, generally because it defines how to parameterize and initialize models to enable learning rate transfer and other desirable feature learning properties. Computing the muP for Gated Delta Networks is therefore impactful.

Soundness: The paper seems sound, and the empirical experiments are a good sanity check that the derived parameterization is in fact correct.

Critical weaknesses

The main weakness of this paper is just its limited novelty and therefore potential lack of impact. While muP for Gated Delta Networks may be novel, muP has already been computed for a wide range of architectures. It's not clear to me whether this paper adds any substantially new technical argument, or whether the technical argument mainly follows the pattern of prior work.

Questions for authors

Why doesn't Figure 1 compare to the original muP configuration?

3 comments

We thank the reviewer for recognizing the empirical value of the study. Here are the responses to your questions

Q1: What is technically new relative to prior µP analyses?

A1: Original µP gives rules for conventional input, hidden, and output transformations. GDN introduces a matrix-valued recurrent state and distinct scalar decay/write pathways. Their gradients depend on the recurrent adjoint and innovation contractions, and cannot be classified correctly using fan-in/fan-out alone. In particular, our SGD analysis yields$\eta_{W_\alpha},\eta_{W_\beta}=\Theta(d^{-1/2}),\quad \eta_{a_{\log}},\eta_b=\Theta(d^{1/2}),$ in contrast to Θ(1)\Theta(1) for ordinary hidden projections. This difference is absent in the usual hidden-weight rule and is not supplied by analyses of diagonal SSMs, whose state transitions lack GDN's matrix-valued rank-one delta update. Appendix D further demonstrates how changing the gating structure changes the backward-scaling analysis, even when forward coordinate orders appear similar. There is also direct practical evidence: Figure 2 compares SP, original µP, and our parameterization under momentum SGD; the proposed rule shows the most consistent optimal learning rate across widths. Appendix E extends this check to width 1536: validation loss is 2.9737 / 2.9511 / 2.9540 at learning rates 0.3 / 0.4 / 0.5, and additional seeds at smaller widths show the same preferred SGD learning rate 0.4. These are results already contained in the submitted manuscript. They reinforce that the new gate-specific exponents are consequential, not merely cosmetic.

Q2: Why is there no original µP baseline in AdamW Figure 1?

A2: The two optimizer cases should be distinguished. Under AdamW, our proposed learning-rate exponents coincide with the conventional µP fan-in rules for the parameter categories in Table 2. In particular, Wα,WβW_\alpha,W_\beta fall under the hidden-weight 1/d1/d rule, while a{log⁡},ba_\{\log\},b retain width-independent learning rates. A separately labeled “original µP learning-rate rule” would therefore not constitute a distinct third prediction of optimal LR-versus-width scaling in Figure 1. The comparison there is deliberately SP versus the resulting AdamW µP configuration. Under SGD, the original and gate-specific prescriptions genuinely differ, which is why Figure 2 contains all three. For clarity, we will state the shared AdamW learning-rate exponents in the Figure 1 caption and refer to the existing Table 2. This is a comparison of scaling prescriptions, not an unsupported assertion that any two implementations with different RMSNorm epsilon conventions are numerically identical. The architectural normalization choice is analyzed separately below; a repeated full AdamW sweep is not needed to establish that the tabulated LR exponents agree.

I would keep my score.

Thank you again for your recognition of our work!

Reviewer_MpXDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper aims to extend μ\muP to GDN, deriving width-dependent rules for initialization, forward multipliers, and learning rates to enable stable feature learning and hyperparameter transfer.

Theoretically, the authors analyze the forward and backward scaling of GDN under several simplifying assumptions. Compared with standard μP, the main new recipe is the learning rates of gating parameters for SGD as in Table 1.

Experimentally, the paper compares SP, original μ\muP, and the proposed μ\muP configuration on GDN across several widths. The results demonstrate improved learning-rate transfer over SP and original μ\muP under SGD.

Critical strengths

The problem addressed is important: enabling hyperparameter transfer for a practically relevant architecture, GDN. The paper provides convincing empirical evidence that the proposed parameterization improves learning-rate transfer across model widths under the tested settings, especially for SGD.

Critical weaknesses
  1. The most concerning problem is that the theoretical presentation lacks clarity and rigor. To establish reliable theoretical results, it is essential to explicitly state the assumptions, characterize the scope, and provide sufficiently rigorous and reproducible derivations. However, the current presentation does not achieve these requirements.

    Several important simplifications are not adequately justified: (1) Assumption 5.1: the paper uses this assumption to disregard the BPTT tail and retain only the direct-gradient contribution. However, bounded recurrent accumulation does not, by itself, guarantee that the full gradient has the same nonvanishing asymptotic order as the direct term; (2) omitting SiLU in Remark 4.1: the paper argues this simplification is an approximation merely changing constant factors, while actually SiLU can introduce nonzero means, which may affect the width exponents such as $ q^\top k $.

    More generally, many critical scaling relations are asserted through direct manipulations of Θ\Theta terms without sufficient derivations and illustrations of the required independence and conditions. This substantially weakens the reusability of its theoretical derivations.

  2. The analysis seems not to sufficiently capture the core property of GDN. The main architectural difference of GDN is its delta-rule state update. However, much of its influence is removed or made subleading in the analysis. In the gradient derivation for the write gate (around line 353), the erase contribution is treated as lower order, while the leading exponent is determined by the rank-one write term. Similarly, for the decay gate (around line 319), the erase correction is subleading. These remove the effect of the delta-rule erase mechanism. While such a result could still be useful, the theoretical contribution appears substantially narrower than claimed.

Questions for authors
  1. Could the authors provide a more rigorous justification for the above SiLU simplification in Remark 4.1 and the BPTT tail simplification in Assumption 5.1?
4 comments

We thank the reviewer for the close reading. We respectfully disagree that the scaling analysis overlooks the essential recurrent mechanism or that its conclusions follow from manipulating Θ\Theta-symbols alone. The key results are obtained by tracking the matrix-state recurrence, the normalized query/key contractions, and the different feature-update geometries of vector and scalar gates. We agree that some intermediate assumptions deserve a more explicit statement, and we can sharpen these arguments without changing the central gate-specific learning-rate prescriptions or the experimental findings.

Q1: Why are the SiLU simplification and BPTT approximation justified?

A1 We thank the reviewer for this question. The role of suppressing SiLU in Remark 4.1 is to isolate width exponents of the coordinate magnitudes, not to assume that SiLU preserves Gaussianity, exact 0 means, or independence. These are different. Let m=Θ(d)m=\Theta(d) be the q/k head dimension, φ=SiLU, and u=φ(z) be a post-ShortConv feature. Under the width-stable, nondegenerate pre-activation moments used in our initialization analysis, E[ϕ(Z)2]=Θ(1)\mathbb E[\phi(Z)^2]=\Theta(1), so ∣u∣2=Θ(m)|u|_2=\Theta(\sqrt m) in the coordinate-concentration regime. More importantly, after the actual L2 normalization q=u/∣u∣2q=u/|u|_2, we have the exact identity 1m∑i=1mqi2=1m,\frac{1}{m}\sum_{i=1}^{m}q_i^2=\frac{1}{m},and likewise for k. Thus, their coordinate RMS is exactly m−1/2=Θ(d−1/2)m^{-1/2}=\Theta(d^{-1/2}), regardless of whether SiLU introduces a nonzero mean. SiLU is also a width-independent pointwise operation with a bounded derivative, so it introduces no additional polynomial width factor in these local magnitude estimates. This justifies using the SiLU-suppressed model to expose the coordinate-size bookkeeping. The separate issue is the scaling of cross-feature contractions such as qt⊤ktq_t^\top k_t, which depends on alignment as well as individual coordinate magnitudes. We do not infer their scaling from L2 normalization alone. Importantly, the paper already examines this quantity in the implemented network: Figure 11 (Appendix C.4) plots the averaged query–key cosine at several sequence positions, four widths, and training stages (including the initial stage), using the actual ShortConv–SiLU–L2Norm path. Since qtq_t and ktk_t have unit norm, the plotted cosine is exactly qt⊤ktq_t^\top k_t. Its reported averages are close to zero in the evaluated setting, supporting the weak-alignment approximation used for the cross-feature contractions.

For BPTT, write $A_t=\alpha_t(I-\beta_t k_tk_t^\top), D_t=g_tq_t^\top, G_t=\partial L/\partial S_t.$ The exact adjoint identity is$G_t=D_t+G_{t+1}A_{t+1}^\top =D_t+\sum_{\tau>t}D_\tau A_\tau^\top\cdots A_{t+1}^\top.$ For unit ktk_t and 0<βt<10<\beta_t<1, the eigenvalues of I−βtktkt⊤I-\beta_tk_tk_t^\top are 1−βt1-\beta_t along ktk_t and 11 in orthogonal directions. Hence, for head dimension >1>1, ∥At∥2=αt\|A_t\|_2=\alpha_t. It follows without discarding any BPTT term that $|G_t|_F\leq \sum_{\tau\geq t}|D_\tau|_F\prod_{s=t+1}^{\tau}\alpha_s.$ If the measured effective-memory sum remains O(1) in width, full BPTT cannot generate an additional positive width exponent. To obtain the corresponding lower order, a noncancellation condition is needed. For instance, writing Gt=Dt+RtG_t=D_t+R_t, if the relevant mean-field cross term satisfies E⟨Dt,Rt⟩F=0\mathbb E\langle D_t,R_t\rangle_F=0, then $\mathbb E|G_t|_F^2 =\mathbb E|D_t|_F^2+\mathbb E|R_t|_F^2 \geq\mathbb E|D_t|_F^2.$ The same variance argument can be stated for the specific gate-gradient contractions, rather than incorrectly deriving entrywise two-sided estimates from a norm upper bound. This is a standard nondegeneracy requirement in a mean-field gradient-scaling argument, not a new architecture or learning-rate rule. Our paper already reports relevant evidence in Appendix F: median effective-memory estimates of 62.29 and 79.65 tokens for widths 256 and 1024 (versus context length 1024), and average relative differences between full and detached gradients of 0.272 for WkW_k and 0.231 for WvW_v in the fourth layer. These results show that the BPTT correction is not zero, but that its observed magnitude is consistent with a finite correction rather than a width-driven blow-up. We will present the exact adjoint, bounded-gain inequality, and noncancellation criterion together in the revised explanation. No change to the gate-specific SGD prescription follows merely from retaining the BPTT tail.

Q2: Does treating the erase term as subleading remove GDN's defining delta rule?

A2: No. The defining delta-rule mechanism is present in every state transition and every recurrent derivative. A useful exact rewriting is $S_t=\alpha_tS_{t-1} +\beta_t\underbrace{(b_t-\alpha_tS_{t-1}k_t)}_{\text{innovation / prediction error}}k_t^\top.$ Thus the write gate multiplies the innovation, not a write-only term. With full BPTT its exact derivative is $\frac{\partial L}{\partial\beta_t} =\left\langle G_t,(b_t-\alpha_tS_{t-1}k_t)k_t^\top\right\rangle_F.$ Further, the erase operator changes state evolution and all historical gradient paths through AtA_t. Its action is anisotropic: it contracts the key direction by 1−βt1-\beta_t, while leaving the orthogonal subspace unchanged apart from the scalar decay αt\alpha_t. Therefore, identifying a subleading local contraction in one weak-coherence scaling regime is not equivalent to omitting the delta rule. In the coherent regime discussed above the erase contraction can even have the same width order as the write contraction. In both cases, the relevant question for our purpose is whether the total innovation contraction acquires a different leading width power; this is not established by the size of one algebraic term alone. This identity makes explicit the innovation structure already present in Eq. (3.1). Our technical contribution is precisely to track how a matrix-state delta update and scalar gates interact with optimizer-specific feature updates; the resulting SGD exponents are not obtained by treating GDN as an ordinary feedforward layer.

Thank you for the response. The revisions have added the weak-coherence condition for the SiLU analysis, the noncancellation condition for BPTT, and the BPTT recursion and norm upper bound. It is also explained that a subleading local erase contribution does not remove the delta-rule mechanism.

These clarifications help improve the clarity of the work, but my main concern about the rigor of the scaling derivations remains. For example, coordinate-size estimates and adjoint norm bounds do not, by themselves, lead to the width orders of specific gate-gradient contractions. How the stated statistical conditions yield the claimed orders still requires more explicit derivations. Similarly, a near-zero average query–key cosine does not directly show that the RMS scales as Θ(d−1/2)\Theta(d^{-1/2}). I therefore maintain the score.

Thank you for your response. And we wish that the following can address your concern.

Q: Coordinate-size and adjoint-norm bounds do not establish the width orders of gate-gradient contractions; Figure 11's near-zero mean cosine does not establish its RMS scaling.

A: Thank you for specifying the remaining gap. We agree with the distinction between a norm upper bound and a two-sided estimate of a specific contraction. We therefore analyze the exact output sensitivities and their second moments, not ∣Gt∣F\\|G_t\\|_F alone. Let n=Θ(d)n=\Theta(d) be the value-head width, As=αs(I−βsksks⊤)A_s=\alpha_s(I-\beta_s k_sk_s^\top), and Pt,τ=At+1⋯AτP_{t,\tau}=A_{t+1}\cdots A_\tau (Pt,t=IP_{t,t}=I). Write vtv_t for the value vector and oτ=Sτqτo_\tau=S_\tau q_\tau. The exact sensitivities of a future readout to the gates at t are Ut,τα=St−1(I−βtktkt⊤)Pt,τqτ,Ut,τβ=(vt−αtSt−1kt)(kt⊤Pt,τqτ)U^\alpha_{t,\tau}=S_{t-1}(I-\beta_tk_tk_t^\top)P_{t,\tau}q_\tau,U^\beta_{t,\tau}=(v_t-\alpha_tS_{t-1}k_t)(k_t^\top P_{t,\tau}q_\tau). If hτh_\tau is the loss gradient after readout RMSNorm, and JτJ_\tau is its Jacobian with respect to oτo_\tau, then full BPTT, without truncation, gives ∂L/∂γt=∑τ≥thτ⊤JτUt,τγ\partial L/\partial\gamma_t=\sum_{\tau\ge t}h_\tau^\top J_\tau U^\gamma_{t,\tau} for γ∈α,β\gamma\in\\{\alpha,\beta\\}. Here is the explicit second-moment step. Set aτ2=∣oτ∣22/na_\tau^2=\\|o_\tau\\|_2^2/n. For the implemented d\sqrt d multiplier followed by RMSNorm, Jτ=1aτ2+ϵ/d(I−oτoτ⊤n(aτ2+ϵ/d)).J_\tau=\frac{1}{\sqrt{a_\tau^2+\epsilon/d}}\left(I-\frac{o_\tau o_\tau^\top}{n(a_\tau^2+\epsilon/d)}\right). Let Πτ=I−oτoτ⊤/∣oτ∣22\Pi_\tau=I-o_\tau o_\tau^\top/\\|o_\tau\\|_2^2. For active positions beyond the empty-state boundary, sufficient statistical conditions are (i) ϵ/d=O(aτ2)\epsilon/d=O(a_\tau^2); (ii) E[hτhτ⊤∣F]⪯(C/d2)I\mathbb E[h_\tau h_\tau^\top\mid\mathcal F]\preceq (C/d^2)I and its restriction to the readout-tangent subspace is ⪰(c/d2)Πτ\succeq(c/d^2)\Pi_\tau ; and (iii) the projected sensitivity has ∣ΠτUt,τγ∣22=Θ(daτ2wt,τ2)\\|\Pi_\tau U^\gamma_{t,\tau}\\|_2^2=\Theta(d a_\tau^2 w_{t,\tau}^2), with ∣Ut,τγ∣22=O(daτ2wt,τ2)\\|U^\gamma_{t,\tau}\\|_2^2=O(d a_\tau^2 w_{t,\tau}^2) and width-independent summable recurrent weights wt,τw_{t,\tau}. Conditioning on the forward activations yields E[(hτ⊤JτUt,τγ)2∣F]=Θ(∣JτUt,τγ∣22d2)=Θ(d−1wt,τ2)\mathbb E[(h_\tau^\top J_\tau U^\gamma_{t,\tau})^2\mid\mathcal F]=\Theta\left(\frac{\\|J_\tau U^\gamma_{t,\tau}\\|_2^2}{d^2}\right)=\Theta(d^{-1}w_{t,\tau}^2).

These conditions have concrete meanings in GDN. At τ=t\tau=t in the weak-overlap regime, ∣kt⊤qt∣2=Θ(d−1)|k_t^\top q_t|^2=\Theta(d^{-1}), ∣vt−αtSt−1kt∣2=Θ(d)|v_t-\alpha_t S_{t-1}k_t|^2=\Theta(d), ∣St−1qt∣2=Θ(1)|S_{t-1}q_t|^2=\Theta(1) and ∣St−1kt∣=O(1)|S_{t-1}k_t|=O(1) imply ∣Ut,tα∣2,∣Ut,tβ∣2=Θ(1)|U^\alpha_{t,t}|^2,|U^\beta_{t,t}|^2=\Theta(1). For β\beta, the identity ∣Ut,τβ∣2=∣vt−αtSt−1kt∣2∣kt⊤Pt,τqτ∣2|U^\beta_{t,\tau}|^2=|v_t-\alpha_t S_{t-1}k_t|^2|k_t^\top P_{t,\tau}q_\tau|^2 gives the required energy when the innovation has Θ(d)\Theta(d) squared norm and the transported overlap has second moment Θ(aτ2wt,τ2)\Theta(a_\tau^2w_{t,\tau}^2). For α\alpha, expanding St−1S_{t-1} into past rank-one writes gives the same order when retained value directions have nondegenerate conditional variance, transported past-key/query overlaps have second moment Θ(aτ2wt,τ2)\Theta(a_\tau^2w_{t,\tau}^2), and effective memory is bounded. The tangent-direction condition excludes an exact alignment with the readout, which RMSNorm would annihilate; it is not implied by an adjoint norm bound. More explicitly, with Zτ=hτ⊤JτUt,τγZ_\tau=h_\tau^\top J_\tau U^\gamma_{t,\tau}, a sufficient temporal noncancellation condition is ∑τ≠sE[ZτZs]≥−(1−δ)∑τE[Zτ2]\sum_{\tau\ne s}\mathbb E[Z_\tau Z_s]\ge-(1-\delta)\sum_\tau\mathbb E[Z_\tau^2] for a width-independent δ>0\delta>0. Together with bounded recurrent weights, summing BPTT terms preserves RMS⁡(∂L/∂αt),RMS⁡(∂L/∂βt)=Θ(d−1/2)\operatorname{RMS}(\partial L/\partial\alpha_t),\operatorname{RMS}(\partial L/\partial\beta_t)=\Theta(d^{-1/2}). Width-stable, nonsaturated gate derivatives and nondegenerate input/temporal statistics then yield the reported per-coordinate orders for Wα,Wβ,alog⁡,bW_\alpha,W_\beta,a_{\log},b.

On SiLU and Figure 11: We do not infer RMS from a signed mean. Indeed, E[ct2]=Var⁡(ct)+(Ect)2\mathbb E[c_t^2]=\operatorname{Var}(c_t)+(\mathbb E c_t)^2 for ct=qt⊤ktc_t=q_t^\top k_t; Figure 11 reports signed averages, not the variance or RMS. More generally, after SiLU, ct=ρd+Op(d−1/2)c_t=\rho_d+O_p(d^{-1/2}) under coordinate concentration, so an uncentered d−1/2d^{-1/2} RMS needs ρd=O(d−1/2)\rho_d=O(d^{-1/2}). The point relevant to the gate-gradient result is that the derivation above allows either weak or coherent overlap: changing the readout scale aτa_\tau also changes the RMSNorm Jacobian by its reciprocal, and the aτa_\tau factors cancel in the contraction second moment provided the stated sensitivity and tangent nondegeneracy conditions hold. This does not claim that Figure 11 proves an RMS exponent, nor that these conditions follow from L2 normalization alone.

We have further incorporated the contraction-level second-moment derivation into Appendix G.2 and clarified the statistical interpretation of Figure 11 in Appendix C.4. These revisions explicitly distinguish the assumptions controlling the sensitivity norms, the gradient contractions, and the full-BPTT contributions.

Comments on the paper

5 comments

We thank the reviewer for focusing on the link between the mathematical analysis and the implemented computation. The following are our answers to your questions.

Q1: With SiLU, does ⟨qt,kt⟩\langle q_t,k_t\rangle scale as d−1/2d^{-1/2}?

A1: The important distinction is between the coordinate scaling of L2-normalized features and the scaling of their inner product. For a query/key head of dimension m=Θ(d)m=\Theta(d), normalization guarantees ∣qt∣2=∣kt∣2=1|q_t|_2=|k_t|_2=1 and hence exact coordinate RMS m−1/2m^{-1/2}. This fact holds with SiLU and ShortConv retained. The inner product additionally depends on possible coherent alignment. To make that dependence precise, let ui=SiLU(ziq)u_i=\mathrm{SiLU}(z^q_i) and vi=SiLU(zik)v_i=\mathrm{SiLU}(z^k_i). Under suitable weak-dependence and finite-moment conditions for coordinate concentration,$q_t^\top k_t =\frac{m^{-1}\sum_i u_iv_i}{\sqrt{m^{-1}\sum_i u_i^2}\sqrt{m^{-1}\sum_i v_i^2}} =\rho+O_p(m^{-1/2}),$where ρ=E[uivi]E[ui2]E[vi2].\rho=\frac{\mathbb E[u_iv_i]}{\sqrt{\mathbb E[u_i^2]\mathbb E[v_i^2]}}. Thus, the d−1/2d^{-1/2} term is the fluctuation scale; an O(d−1/2)O(d^{-1/2}) scale for the uncentered inner product additionally requires its coherent component ρ\rho to be comparably small. We are not assuming that SiLU makes ρ\rho identically zero. This formulation directly addresses the reviewer's observation about its nonzero mean, without conflating that observation with a change in the normalized-coordinate exponent.

Moreover, the relevant measurement already appears in our paper. Figure 11 (Appendix C.4) reports cos⁡(qt,kt)=qt⊤kt\cos(q_t,k_t)=q_t^\top k_t from the actual SiLU/ShortConv implementation at widths d∈{256,512,768,1024}d\in\{256,512,768,1024\}, across five sequence positions and multiple training stages, including the start of training. For the evaluated input and fourth layer, the signed averages remain numerically small rather than exhibiting a large, persistent O(1)O(1) alignment. This provides direct empirical support for the weak-overlap regime invoked in our width analysis; it is not an inference from the SiLU-omitted surrogate. Figure 11 reports averaged cosine, not E[(qt⊤kt)2]\sqrt{\mathbb E[(q_t^\top k_t)^2]}, so we do not present it as a standalone measurement of the RMS exponent. Rather, it supplies the already-available check of the potential coherent contribution, while the m−1/2m^{-1/2} fluctuation term follows from the concentration argument above.

Q2: Doesn't RMSNorm cancel d\sqrt d scaling, including upstream gradients?

A2: Yes, exactly, when ϵ=0\epsilon=0. This point is an algebraic identity and does not require an empirical ablation. For$R_\epsilon(x)=\frac{x}{\sqrt{|x|_2^2/n+\epsilon}},$we have, for every c>0c>0,Rϵ(cx)=Rϵ/c2(x).R_\epsilon(cx)=R_{\epsilon/c^2}(x). Consequently, R0(cx)=R0(x)R_0(cx)=R_0(x), and differentiating these identical functions with respect to the original input gives the same upstream Jacobian. This is fully compatible with our implementation: the positive pre-RMSNorm multiplier is not an independent gradient amplifier in exact RMSNorm. Its role is to set the effective normalization floor. With c=dc=\sqrt d, its effect is precisely ϵ{{eff}}=ϵ/d\epsilon_\{\mathrm\{eff\}\}=\epsilon/d. In the small-readout regime where the unscaled coordinate mean square is Θ(d{−1})\Theta(d^\{-1\}), a fixed ϵ\epsilon would otherwise become relatively more important with width; scaling keeps the floor proportional to the predicted signal scale. We will state the identity and this numerical rationale explicitly in Section 4.3. This refinement changes neither the recurrent state equation nor the algebraic derivation of the scalar gate's learning-rate exponent in the ideal normalization regime. We make no claim that the multiplier accomplishes something beyond its exact effective-epsilon interpretation.

Q3: Does the AdamW argument survive actual moving moments and batch averaging?

A3: Yes, the scale cancellation in the actual AdamW moment recursion is more general than the sign-gradient illustration in Appendix B. Let gˉj(s)\bar g_j^{(s)} denote the batch-reduced gradient, and suppose the width-dependent factor is gˉj(s)=d−γhj(s)\bar g_j^{(s)}=d^{-\gamma}h_j^{(s)} with a width-stable nondegenerate process hh. With fixed β1,β2\beta_1,\beta_2, linearity and quadratic homogeneity give exactly $\hat m_j^{(s)}=d^{-\gamma}\hat m_{h,j}^{(s)}, \hat v_j^{(s)}=d^{-2\gamma}\hat v_{h,j}^{(s)}.$ The adaptive part of the AdamW update is therefore $-\eta\frac{\hat m_j^{(s)}}{\sqrt{\hat v_j^{(s)}}+\epsilon_A} =-\eta\frac{\hat m_{h,j}^{(s)}}{\sqrt{\hat v_{h,j}^{(s)}}+\epsilon_A d^\gamma}.$

When the adaptive epsilon is negligible relative to the moment RMS, the normalized update is invariant to a common gradient width factor for arbitrary moment histories, not only a single-step sign update. The same argument applies after batch aggregation, because the common width factor can be taken outside a fixed batch reduction. The separately decoupled weight-decay term remains controlled by the chosen parameter scale and is not part of this cancellation. To turn normalized coordinate updates into Θ(1) feature updates, as in standard µP reasoning, one additionally needs nonvanishing update–feature correlation. A sufficient form for zi=∑jWijxjz_i=\sum_jW_{ij}x_j is $\frac1d\sum_j \frac{\hat m_{ij}}{\sqrt{\hat v_{ij}}+\epsilon_A}x_j \xrightarrow{p}c_i\neq0.$ Then Δzi=−ηd(ci+op(1))\Delta z_i=-\eta d(c_i+o_p(1)), yielding the prescribed η=Θ(1/d). This explicitly identifies the same feature-alignment premise tacitly used by the original sign illustration; it does not introduce a different AdamW LR rule. We will present the exact moment argument in Appendix B so the conclusion does not rely on identifying AdamW with signSGD.

Q4: Does full BPTT preserve exponents for Wα,Wβ,alog⁡,bW_\alpha,W_\beta,a_{\log},b? Are their gradients measured?

A4: We thank the reviewer for this question. The direct-gradient approximation is not required: the full BPTT adjoint retains future-state contributions and lets us examine their width dependence. Let Gt=∂L/∂StG_t=\partial L/\partial S_t, Dt=gtqt⊤D_t=g_tq_t^\top, and At=αt(I−βtktkt⊤)A_t=\alpha_t(I-\beta_tk_tk_t^\top). The exact backward recurrence is Gt=Dt+Gt+1At+1⊤.G_t=D_t+G_{t+1}A_{t+1}^\top. Since ∥kt∥2=1\|k_t\|_2=1 and 0<αt,βt<10<\alpha_t,\beta_t<1, we have ∥At∥2≤αt\|A_t\|_2\le\alpha_t. Consequently, ∥Gt∥F≤∑τ=tT∥Dτ∥F∏s=t+1ταs\|G_t\|_F\le\sum_{\tau=t}^{T}\|D_\tau\|_F\prod_{s=t+1}^{\tau}\alpha_s. Under the bounded-effective-memory condition in Assumption 5.1, recurrent propagation does not introduce additional width-dependent amplification.

In Eq. (3.1), write the value vector as vtv_t (called btb_t in the manuscript) to distinguish it from the scalar gate bias bb. Define At=αt(I−βtktkt⊤),St=St−1At+βtvtkt⊤,ot=Stqt,γt=∂L∂ot.A_t=\alpha_t(I-\beta_t k_tk_t^\top), S_t=S_{t-1}A_t+\beta_t v_tk_t^\top, o_t=S_tq_t,\gamma_t=\frac{\partial L}{\partial o_t}. For τ≥t\tau\ge t, let Pt→τ=At+1At+2⋯AτP_{t\to\tau}=A_{t+1}A_{t+2}\cdots A_\tau and Pt→t=IP_{t\to t}=I, and define rτ∣t=Pt→τqτr_{\tau\mid t}=P_{t\to\tau}q_\tau. The exact state adjoint, including every future readout, is Gt=∂L∂St=∑τ=tTγτrτ∣t⊤=γtqt⊤+Gt+1At+1⊤.G_t=\frac{\partial L}{\partial S_t}=\sum_{\tau=t}^{T}\gamma_\tau r_{\tau\mid t}^{\top}=\gamma_tq_t^\top+G_{t+1}A_{t+1}^\top. So the exact full-BPTT derivatives with respect to the scalar gate outputs are δα,t:=∂L∂αt=⟨Gt,St−1(I−βtktkt⊤)⟩F=∑τ=tTγτ⊤St−1(I−βtktkt⊤)rτ∣t\delta_{\alpha,t}:=\frac{\partial L}{\partial\alpha_t}=\langle G_t,S_{t-1}(I-\beta_tk_tk_t^\top)\rangle_F=\sum_{\tau=t}^{T}\gamma_\tau^\top S_{t-1}(I-\beta_tk_tk_t^\top)r_{\tau\mid t},δβ,t:=∂L∂βt=⟨Gt,(vt−αtSt−1kt)kt⊤⟩F=∑τ=tT[γτ⊤(vt−αtSt−1kt)][kt⊤rτ∣t].\delta_{\beta,t}:=\frac{\partial L}{\partial\beta_t}=\langle G_t,(v_t-\alpha_tS_{t-1}k_t)k_t^\top\rangle_F=\sum_{\tau=t}^{T}[\gamma_\tau^\top(v_t-\alpha_tS_{t-1}k_t)][k_t^\top r_{\tau\mid t}]. For the implemented gate parametrization from Section 4.4,zα,t=Wαxt+b,αt=exp⁡[−ealog⁡softplus⁡(zα,t)],zβ,t=Wβxt,βt=σ(zβ,t),z_{\alpha,t}=W_\alpha x_t+b,\alpha_t=\exp[-e^{a_{\log}}\operatorname{softplus}(z_{\alpha,t})], z_{\beta,t}=W_\beta x_t,\quad \beta_t=\sigma(z_{\beta,t}), we obtain the exact gradients of the four parameter classes:∂L∂Wα=−∑t=1Tδα,t αtealog⁡σ(zα,t) xt⊤\frac{\partial L}{\partial W_\alpha}=-\sum_{t=1}^T\delta_{\alpha,t}\,\alpha_t e^{a_{\log}}\sigma(z_{\alpha,t})\,x_t^\top,∂L∂Wβ=∑t=1Tδβ,t βt(1−βt) xt⊤\frac{\partial L}{\partial W_\beta}=\sum_{t=1}^T\delta_{\beta,t}\,\beta_t(1-\beta_t)\,x_t^\top,∂L∂alog⁡=−∑t=1Tδα,t αtealog⁡softplus⁡(zα,t)\frac{\partial L}{\partial a_{\log}}=-\sum_{t=1}^T\delta_{\alpha,t}\,\alpha_t e^{a_{\log}}\operatorname{softplus}(z_{\alpha,t}),∂L∂b=−∑t=1Tδα,t αtealog⁡σ(zα,t).\frac{\partial L}{\partial b}=-\sum_{t=1}^T\delta_{\alpha,t}\,\alpha_t e^{a_{\log}}\sigma(z_{\alpha,t}). The time ordering of Pt→τP_{t\to\tau} is essential. Here γτ\gamma_\tau denotes the total gradient reaching the output oτo_\tau; GtG_t accounts for the recurrent path in this GDN layer. For a mini-batch, gradients are additionally accumulated/averaged over examples, according to the loss reduction. The identities are exact; predicting a Θ(d−1/2)\Theta(d^{-1/2}) width exponent for their aggregate magnitudes additionally requires the statistical non-cancellation and width-stability conditions in the scaling analysis.

AuthorsOpenReview ↗

Summary of changes in the revised PDF

We have uploaded a revised PDF. New and changed text is in red. The title, the section structure and the theorem numbering are unchanged, and the main text stays within 9 pages.

Main text

  • Sec. 4.2: add analysis on ∣At∣2<1\\|A_t\\|_2<1 to validate the convergence
  • Sec. 4.3: add Remark 4.1 as the prerequisite; add analysis on ϵ\epsilon in RMSNorm to validate scale-invariance property
  • Remark 4.1: add analysis on the SiLU function
  • Sec. 5.1: rectify the Assumption 5.1 for BPTT
  • Sec. 5.3: add reference to justification of the gate-gradient orders

Appendix

  • Sec. A.1: add Remark 4.1 as the eark-coherence mean-field precondition
  • Sec. A.2: add the centered mean-field reference assumption of the math derivation
  • Sec. B.1: add analysis on the batched gradient in AdamW optimization
  • Sec. B.2&B.3: rephrase the wording
  • Sec. C.4: modify the analysis on Figure 11.
  • Sec. G.1: add explanation on Figure 1.
  • Sec. G.2: add justification of the gate-gradient orders

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.