acceptodds
Under review as a conference paper at ICLR 2027

Beyond Stability: Attention-to-MLP Interaction in Infinite-Depth Transformers

Abstract

Standard Transformer blocks pass the attention output to the MLP. However, we find that the depth scaling used to stabilize residual accumulation can make this interaction vanish as depth increases, even though the architecture remains computationally sequential. We study this effect under Depth-P and CompleteP residual scalings at initialization and during training (one gradient-descent update). By jointly controlling representation moments and weight-reuse responses, we identify sufficient stability conditions under which the interaction coefficient can remain order one independently of residual scaling. As a result, the accumulated interaction can remain order one as depth increases. By contrast, under the standard tied choice, the accumulated interaction decays as for Depth-P and for CompleteP, where is the model depth. Consistent with this analysis, experiments support the predicted depth trends and show that suitable independent coefficients can retain interaction and improve learning over the tied baseline. However, we also observe that the performance gains are sensitive to the interaction coefficient. This leads us to explore input-dependent gates that adapt the interaction strength during training. In our comparisons, these adaptive gates indeed further improve performance, with signed SiLU and linear gates outperforming positive-only sigmoid gates. These findings motivate separating the interaction coefficient from residual scaling, enabling a model design choice between retaining interaction when beneficial for learning and using explicitly parallel blocks for concurrent computation.

Primary area:
learning theory
Keywords:
Transformers, infinite-depth neural networks, depth scaling, $\mu$P, feature learning, attention-to-MLP interaction

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.
Reviewer_RjRJOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The authors study the interaction between the attention output and the MLP in deep Transformers. They consider the residual stream Xℓ+1=Xℓ+aLAℓ(Xℓ)+bLMℓ(Hℓ)X_{\ell+1}=X_{\ell}+a_{L}\mathcal{A}_{\ell}(X_{\ell})+b_{L}\mathcal{M}_{\ell}(H_{\ell}) with Hℓ=Xℓ+cLAℓ(Xℓ)H_{\ell}=X_{\ell}+c_{L}\mathcal{A}_{\ell}(X_{\ell}), where Aℓ\mathcal{A}_{\ell} is an attention, Mℓ\mathcal{M}_{\ell} is an MLP, and LL is the depth. In a standard sequential Transformer, aL=cLa_{L}=c_{L}.

The authors find that setting aL=bL=L−αa_{L}=b_{L}=L^{-\alpha} with α≥1/2\alpha\ge1/2 and cL=Θ(1)c_{L}=\Theta(1) preserves stability (i.e. no blowups) of the interaction between the attention output and the MLP, both at initialization and after one gradient update. Specifically, choosing α=1\alpha=1 as in CompleteP results in the interaction magnitude of O(L−1/2)O(L^{-1/2}) at initialization and O(1)O(1) after one update, while choosing α=1/2\alpha=1/2 as in Depth-μ\muP results in a magnitude of O(1)O(1) in both cases. Furthermore, if we set aL=cLa_{L}=c_{L} as in the standard practice, these interaction magnitudes vanish as depth LL increases, provided that α≥1/2\alpha\ge1/2.

Numerical experiments suggest that performance may be improved by setting gates based on each XℓX_{\ell}, rather than using the fixed constant cLc_{L}.

Critical strengths

I believe it is an important finding that, in deep Transformers, while the interaction vanishes under the standard setting of aL=bL=cL=L−αa_{L}=b_{L}=c_{L}=L^{-\alpha}, setting aL=bL=L−αa_{L}=b_{L}=L^{-\alpha} and cL=Θ(1)c_{L}=\Theta(1) may preserve the interaction after one gradient update. Although the main theoretical results address only the case of one update, experimental results support the same conclusion for multiple updates.

Critical weaknesses

The most of the theoretical results only derive the upper bounds. Therefore, as the authors clarify in the paper, making cL=Θ(1)c_{L}=\Theta(1) does not automatically imply that the interaction is also Θ(1)\Theta(1).

However, I note that the experimental result presented in Figure 1(c) addresses this concern, demonstrating that the interaction does not decay under the experimental setting in this paper.

Questions for authors

None

1 comment

We sincerely thank the reviewer for taking the time to read our paper carefully and for their positive assessment. We particularly appreciate the reviewer's attention to the distinction between our theoretical one-update upper bound and how it is complemented by the empirical evidence in Figure 1(c).

Reviewer_X2VnOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors consider how the residual multiplier cc of an attention layer should be scaled before being fed into an MLP. Prior work has tied this residual multiplier to the residual multiplier aa of the attention outside the MLP; since the latter must scale smaller as depth increases for stability, so too must the former under this tying scheme. Via a theoretical analysis, the authors find that tying cc to aa in this way results in vanishing contribution of attention to MLP output as depth increases. Moreover, they find that this tying of cc to aa is actually unnecessary, and that in fact cc can scale as Θ(1)\Theta(1) in depth while maintaining stability of the forward pass both at initialization and after a single gradient descent step, which preserves the contribution of attention to MLP outputs. The authors then demonstrate empirically that suitable Θ(1)\Theta(1) choices for cc result in better learning, which can be further enhanced by different kinds of gating.

Critical strengths
  1. The theoretical results rigorously demonstrate the limitations of prior depth scaling practices and the proposed fix. The proposed fixes may be useful in architecture design in the future.
  2. The empirical support is substantial and convincing.
Critical weaknesses

None

Questions for authors

These points do not affect my rating. However, I believe their being addressed will improve the paper.

  1. The authors use the word "stability" without ever defining it. From the way it is used in the paper, I believe it can be inferred that the authors are looking for the network outputs scale as O(1)O(1) as depth goes to infinity, but this is never specified. I think it would be helpful to those who are not intimately familiar with this literature that stability be defined precisely.

  2. The authors also say at various points that their results demonstrate that scaling cc correctly improves learning (by which many would understand validation performance), yet the results included in their main paper only demonstrate improved optimization. Improved validation performance is also demonstrated in the appendix, but appears not to be mentioned in the main body of the paper. I think the improved validation performance is worth mentioning somewhere in the main body of the paper.

1 comment

Thank you for the positive assessment and for these excellent suggestions!

Q1. What do we mean by stability? Your interpretation is correct. The precise definition of stability is that the per-token, per-coordinate network output scales as O(1)O(1) as depth goes to infinity. We have added a short definition at its first use in the introduction (highlighted in red in the revision).

Q2. Validation performance. We agree that the validation results should appear alongside the training ones. We are moving them from the appendix to the main paper and splitting the original Figure 2 into three distinct comparisons. Each figure now shows both training and validation performance.

We thank you again for your careful review and constructive feedback to continuously improve our paper's quality.

Reviewer_tu6FOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates that how depth scaling affect the interaction of MLP and Attention mechanism independent of residual connections in vanilla transformer architecture. It used proposed methods to control this interaction. The paper provides theoretical justification at initialization and after one gradient update, alongside experiments showing improvements.

Critical strengths
  1. Clear and meaningful research question. This paper provides a new perspective on information flow in the transformer blocks and it distinguishes stable residual accumulation from preserving interaction of MLP and Attention mechanism.

  2. Clear Theory support. The initialization analysis helps explain how depth scaling can weaken attention-to-MLP interaction. It also provides conditions under which the interaction coefficient can be chosen independently while maintaining stable representations.

Critical weaknesses

Limited experimental scale. The experiments use relatively narrow models with widths of 128 and 256, trained on 1B tokens from C4-English. While these settings support the initial findings, it remains unclear whether the observed gains extend to larger models and longer training runs.

Questions for authors
  1. Are the validation improvements consistent across multiple random seeds?

  2. The MLP output also contributes to the input of the next attention layer through the depth-scaled residual connection. Would independently controlling MLP-to-attention interaction provide similar or additional benefits?

1 comment

Thank you for the positive assessment and the constructive questions.

Q1. Are the validation improvements consistent across seeds? Yes. We repeated the main comparisons with a second random seed and observed consistent validation improvements across both runs. The updated plots report the mean of these two seeds. Additional runs are currently underway; we aim to report at least three seeds, with five as our target for the camera-ready version. We will share any completed results during the rebuttal period and include the final set in the revision.

Q2. Can we independently control MLP-to-attention interaction? Yes. By reindexing the sublayers as an MLP followed by the next attention sublayer, we can express the architecture as: \begin{aligned}
X_{\ell+1}
= H_\ell+d_L\mathcal M_\ell(H_\ell),\qquad 
H_{\ell+1}
= H_\ell+L^{-\alpha}\mathcal M_\ell(H_\ell)
+L^{-\alpha}\mathcal A_{\ell+1}(X_{\ell+1}).
\end{aligned} Here, HℓH_\ell represents the primary residual stream, while Xℓ+1X_{\ell+1} becomes the local input to the next attention sublayer. Then, the coefficient dLd_L controls how much of the current MLP output enters the subsequent attention input, allowing for dL=Θ(1)d_L = \Theta(1) (via either a fixed coefficient or a learned gate) without destabilizing the network. It mirrors our analysis and findings for attention-to-MLP interaction. We plan to empirically investigate this MLP-to-attention interaction in the camera-ready version to see whether preserving one pathway over the other yields superior benefits.

On the other hand, controlling both interactions jointly introduces significant challenges: because both XℓX_\ell and HℓH_\ell may act as residual streams without clean local variables, the scaling and analysis frameworks developed in this work do not directly apply. Thus, we leave a rigorous stability analysis for future exploration.

Experimental scale. We agree that our experiments would benefit from a broader scale, and we appreciate the suggestion. During the rebuttal, we expanded our evaluations to width 384 (averaged over multiple seeds) and plan to include width 512 and 5B tokens in the final version. While resources restrict us from scaling further at this stage, we have updated the limitations section to discuss this clearly. Our primary contribution is theoretical analysis and characterization, with experiments serving to verify the mechanism and its practical relevance rather than claim state of the art training performance.

We hope our response can address the reviewer's concerns and we are happy to address any further concern.

Reviewer_Qpm4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how much attention changes what the MLP writes to the residual stream as Transformer depth LL grows. At each block, the authors measure the difference Mℓ(Xℓ+cLAℓ(Xℓ))−Mℓ(Xℓ)M_\ell(X_\ell+c_L A_\ell(X_\ell))-M_\ell(X_\ell) at the same incoming state XℓX_\ell, then sum these differences across blocks with the MLP residual multiplier bLb_L. Under Depth-µP and CompleteP, they show that this accumulated interaction vanishes at initialization and after the first gradient update when cLc_L is tied to the attention residual multiplier aLa_L. They also show that a depth-independent cLc_L is compatible with stability in their setting, leaving open whether the accumulated interaction stays nonzero.

Critical strengths

(1) Quantitative interaction decay after one step. The authors control the responses arising from weight reuse and separate centered contributions, which are orthogonal across layers, from learning and response corrections that can accumulate more strongly. For aL=bL=L−αa_L=b_L=L^{-\alpha}, the resulting root-mean-square bound in the width limit is O(∣cL∣[L1/2−α+ηLL1−2α])O(|c_L|[L^{1/2-\alpha}+\eta_L L^{1-2\alpha}]), where ηL\eta_L is the effective learning-rate coefficient. Under tying, this gives bounds of O(L−1/2)O(L^{-1/2}) for Depth-µP and O(L−1)O(L^{-1}) for CompleteP at their respective learning-rate scalings. This is concrete result and says something useful about the attention-MLP interaction in two commonly studied scalings.

(2) Separating attention-to-MLP coupling from residual scaling. Separating the attention signal sent to the MLP from the direct attention write identifies a coupling that residual stability alone does not determine. The finite-width initialization bounds give a baseline for seeing what changes after the first update.

Critical weaknesses

(1) The novelty is concentrated in the interaction measure and the first-update bound. Tensor Programs VI (§9) already shows that depth scaling can suppress interactions between parameter updates within a block. This paper studies a different effect, namely how depth scaling suppresses the change in the MLP's residual write caused by its attention input. Once this quantity is defined, the initialization decay follows from a local Lipschitz bound and conditional centering across layers. The first-update calculation goes further by accounting for dependence from weight reuse. Its conclusion extends the suppression result to the first gradient update in the width limit, and I am less convinced that this extension constitutes a substantial theoretical advance on its own. [Tensor Programs VI, §9]

(2) The bounds leave open whether decoupling preserves the interaction. The authors show that the accumulated attention-to-MLP interaction vanishes under tying and that a depth-independent interaction coefficient is compatible with stability. For the decoupled choice, the first-update result gives an order-one upper bound, which also allows the interaction to vanish (Appendix B, Remark 4). The theory therefore explains why tying suppresses the interaction and leaves open whether decoupling keeps it nonzero even in the setting analyzed. The scope also matters for interpreting the result. The interaction measure sums local differences along the original trajectory, leaving open how much the output would change if a parallel network were run separately.

(3) The empirical study is difficult to reproduce and assess. Appendix D does not specify the gate predictor’s architecture, initialization, or learning-rate scaling. The manuscript also leaves unclear how fixed coefficients were selected and how the gains vary across seeds. Figure 4(b), labeled as validation, repeats Figure 2(b)’s training panel.

Questions for authors

(1) What does the first-update bound establish beyond the within-block suppression results in Tensor Programs VI (§9)?

(2) Does the decoupled interaction provably stay nonzero in any case covered by the analysis? If so, under what assumptions? Otherwise, please qualify the preservation claim in the main text.

(3) Please specify the gate predictor’s architecture, initialization, and learning-rate scaling. How were the fixed coefficients selected?

(4) Please replace Figure 4(b) with the intended fixed-coefficient validation results and report any available variation across seeds for the main comparisons.

4 comments

Thank you for your reading and feedback. Below, we provide our responses and clarifications to your questions.

Q1. Relation to Tensor Programs VI. The within-block suppression in Tensor Programs VI arises because, in a residual branch with two (or more) trainable layers, L−1/2L^{-1/2} scaling makes products of parameter updates (such as ΔW2ΔW1\Delta W_2 \Delta W_1) subleading—a phenomenon also noted in Transformers by Bordelon et al. (2024) under the same depth scaling using Dynamic Mean-Field Theory. CompleteP terms this "lazy training with a subset of parameters" and suggests L−1L^{-1} residual scaling to retain higher-order parameter-update terms.

Our focus is fundamentally different: we analyze the attention output flowing into the MLP input rather than the trainable parameters themselves. Our analysis shows that under the standard tied setup, both Depth-μ\muP and CompleteP cause the attention-to-MLP interaction to vanish as depth increases. Therefore, we propose separating the interaction strength coefficient from the residual stream to preserve this interaction, and we demonstrate that retaining it outperforms the baseline.

Q2. Does the decoupled interaction provably stay nonzero in any case covered by the analysis? Our theory proves that cL=Θ(1)c_L = \Theta(1) is compatible with training stability and yields an O(1)O(1) upper bound on the accumulated interaction. Establishing a matching lower bound to guarantee it never vanishes is an open question outside our current scope. Nonetheless, our empirical results in Figure 1(c) demonstrate that the interaction remains substantial across all tested depths. We will explicitly clarify this scope between our theoretical upper bounds and empirical evidence in the main text.

Q3. Gate architecture and coefficient selection. The gate considered in this paper is standard, consisting of a single linear projection followed by an elementwise activation, gℓ(Xℓ)=σg(XℓWg,ℓ⊤)g_\ell(X_\ell) = \sigma_g(X_\ell W_{g,\ell}^{\top}), where σg\sigma_g is sigmoid, SiLU, or the identity. The gate projection follows the same initialization and learning-rate scaling as the query projections. We have specified these details in Appendix D of the revised paper.

Regarding the fixed-coefficient comparison sweeps, we fix the depth to L=128L = 128 and select coefficient strengths of the form cL=x/Lc_L = x/L for x∈{16,32,64,128}x \in \{16, 32, 64, 128\}. We have added these details to the revised paper accordingly.

Q4. Corrected validation figure and seed variation. We have replaced the duplicated training panel with the corresponding validation results. The updated performance plots average two random seeds, with both individual runs showing consistent improvements. We are currently running additional seeds and width settings—aiming for at least three seeds, with five as our target for the final version. We will share any further results that finish during the rebuttal period and incorporate the completed runs into the final revision.

We hope these responses address the reviewers' concerns, and we are happy to discuss and address any additional concerns or provide additional clarity.

Dear Reviewer Qpm4,

As the discussion period nears its close, we wanted to share that our third random seed is complete: each individual run shows consistent improvements, and the figures in the current revision now average across all three random seeds.

We also wanted to kindly check whether our responses and revised manuscript (including the corrected validation panel in Figure 4, the gate and coefficient details in Appendix D, and the clarified scope of our upper bounds) have resolved your original concerns. We are happy to address any remaining points or questions. We understand the discussion status is relatively quiet this time, but we still truly appreciate the reviewer taking the time to carefully read our paper and provide constructive feedback to improve our paper's quality.

Thank you for the response and following up with an additional seed. I view my reproducibility concerns (e.g. the validation panel in the appendix) as resolved.

Onto theory. While I do not disagree with your claim to technical distinction from tensor programs, my reservations about significance persist. Once the interaction is defined, what does the first-update bound teach us beyond the decay already established at initialization? I credit the authors for calculating that suppression survives the first update. My reservation is about how much this explains training, especially given the paper's claim that suppression accounts for similar performance of trained tied and parallel models.

It is on this point I align, partially, with the comments by the AC. Pushing back on the AC's remarks, I acknowledge that a long-horizon theorem is outside the paper's scope. However, while Figure 1(b) supports the claim that suppression persists at later checkpoints beyond one step, it also shows the interaction grows with training time at fixed depth. Eq. (3) measures the change in each MLP's residual output at the same incoming state. Eliminating the pathway throughout a trained model would also change states entering later blocks. Thus a small accumulated interaction does not necessarily imply a small change in predictions. Figure 2(a) shows that separately trained tied and parallel models reach similar perplexities, but is this owed to a suppression mechanism? Figure 1(b) gives evidence that the measured interaction is suppressed with depth, and Figure 2(a) shows that the performance gap narrows with depth. I am not convinced the former explains the latter, in particular because Eq. (3) says nothing about how much trained model predictions depend on this interaction. If I were convinced of this, I might view the results differently. I therefore continue to view the first-update calculation as a novel contribution, while remaining unconvinced of its broader significance.

My recommendation remains unchanged and I lean towards weak reject.

Dear Reviewer Qpm4,

Thank you very much for taking the extra time to examine our revised manuscript and to engage so thoughtfully with both our responses and the discussion with the AC. We genuinely appreciate your confirmation that your reproducibility concerns are resolved and your recognition of the technical distinction from TP VI and the novelty of our first-update analysis.

Regarding your remaining hesitation on whether suppressed interaction with depth explains the narrowing performance gap between tied and parallel models, we completely understand and respect you remain unconvinced. Our goal here is not to push back, but simply to share how we arrived at our conclusion and to warmly invite your perspective on other possibilities:

  • Algebraic decomposition: In Eq. (4), decomposing the final representation XLX_L gives the exact identity: XL=X0+aL∑l=0L−1Al(Xl)+bL∑l=0L−1Ml(Xl)+bL∑l=0L−1Il(Xl).X_{L} = X_{0} + a_{L}\sum_{l=0}^{L-1}\mathcal{A}_{l}(X_{l}) + b_{L}\sum_{l=0}^{L-1}\mathcal{M}_{l}(X_{l}) + b_{L}\sum_{l=0}^{L-1}\mathcal{I}_{l}(X_{l}). Since Theorems 2 and 3 show that the fourth term (bL∑l=0L−1Il(Xl)b_L\sum_{l=0}^{L-1}\mathcal{I}_l(X_l)) vanishes in the infinite-depth limit under tied scaling, XLX_L reduces algebraically to a parallel summation. This suggests that suppressing accumulated interaction at large depth effectively collapses a tied network toward a parallel form.

  • Experimental isolation: Across all our experiments (Figures 2–4 and 6–9), the interaction coefficient cLc_L (or gate glg_l) is the sole variable changed, while all backbones, initializations, hyperparameters, and data budgets are held identical. Because changing cLc_L alone both closes the tied-parallel gap (Figures 2 & 7) and drives the gains from decoupled or signed gates (Figures 3–4 & 8–9), we viewed the attention-to-MLP interaction pathway as the mechanism accounting for these differences.

In the final version, we will gladly soften our causal claims, particularly in the Limitations section (as well as Sections 3 and 6), framing our bounds as a concrete structural mechanism that aligns with the empirical trends rather than a complete proof of equivalence across training.

From your perspective, if forward interaction suppression in Eq. (4) is not the primary driver of the convergence in Figure 2(a-b), what other mechanisms or explanations do you think might be? We would be grateful for any thoughts or pointers, and we would be glad to include them in our Limitations discussion.

Thank you again for your time, care, and constructive feedback throughout the review process.

Comments on the paper

5 comments
Area_Chair_4RiQOpenReview ↗

Discussion

Dear authors and reviewers,

Given this discrepancy between the scores, and the fact this paper belongs to my domain of expertise, I wanted to chime in and direct the discussions from this point.

Let me start by saying, I don't think the scores are representative of how borderline this paper currently sits with me right now. I have a very strong opinion on this topic, and I'm willing to overrule the reviewers, but I would much rather a discussion so we all reach the same understanding. So please, I invite everyone to engage and work towards a consensus here.

My understanding is the main results are two fold:

  • Theorem 2 proves a result that essentially bounds the contribution of the interaction between the MLP block and the attention block.
  • Theorem 3 proves a similar bound but after one step of training.

Now my question is - why is this quantity of interaction between MLP and attention block interesting? The impression I get here is that if the interaction vanishes, then we can run these blocks in parallel rather than in series. And story seems to be that this is a phenomenon we want to avoid.

My first reaction to this is well, this true by default for any depth scaling of any blocks. The reason is a proper depth scaling is an Euler discretization.

X1=X0+hb0(X0),X2=X1+hb1(X1)=X0+hb0(X0)+hb1(X0+b(X0))≈X0+h(b0(X0)+b1(X0))+O(h2).X_1 = X_0 + h b_0(X_0) , \quad X_2 = X_1 + h b_1(X_1) = X_0 + h b_0(X_0) + h b_1(X_0 + b(X_0)) \approx X_0 + h( b_0(X_0) + b_1(X_0) ) + O(h^2).

In the limit, you can always argue the interaction vanishes because it's a "Markov update" in "depth time".

My understanding of what the authors are proposing instead is choosing the coefficients in a way that more or less combines the MLP and attention blocks into a larger residual block, which is equivalent to something like

X1=X0+hb1(b0(X0)).X_1 = X_0 + h b_1( b_0(X_0) ) .

Sure, this is a different limit, but then should continue to investigate whether or not the second attention+MLP block interacts with the first block? I'm not so sure if this is a productive investigation. And yes I invite the authors and reviewers to comment and tell me if they believe otherwise.

My second reaction is that can this continue to hold after a large number of training steps? Generally speaking the answer is no, as we have seen with the NTK regime's finite width correction term vanishing near initialization, but accumulates after T=widthT = \sqrt{\text{width}} number of steps.

In fact, I did a quick simulation just to check, and indeed I observed the accumulation of the RMS to be proportional to T\sqrt{T}. This is simply because the accumulation happens at the squared norm level instead of the root level, but more importantly, the interaction does not vanish after training. In fact in practice, the number of training steps is much much larger than depth of a network, so we are very deep in the non-vanishing interaction regime.

If you want to replicate the simulation, here's the prompt I used with 6.1 Sol high during a discussion with this paper

"Can we run a short experiment to estimate the interaction quantity as a function of depth and number of training steps in transformers? We can use the tied 1/L1/L scaling with next token prediction setup. I just want two log log plots, fixed depth vary steps, fixed step (at 10 at least) and vary depth, and tell me what the slope is on both."

So at this point, I'm once again asking the question, why is this interaction quantity interesting? And why does it matter whether or not it vanishes near initialization, and why should we fix it? This is in context of both of my thoughts above.

Once again, I invite both reviewers and authors to engage with me, as I want us to reach a consensus here instead of me making a unilateral decision.

Regards,

AC

We thank the AC for the close reading and the opportunity to clarify these points. We organize our response around your key questions.

1. Why is the Attention-to-MLP Interaction Quantity Interesting? The main reason is that it exposes a key limitation in standard Transformers: under standard tied scaling, the network pays the full computational and memory cost of sequential execution, yet yields only parallel-like performance while also losing the representational richness of sublayer interaction.

The loss of interactions and the resulting parallel-like performance are demonstrated in our Figures 1 and 2, respectively. Furthermore, previous large-scale studies such as PaLM demonstrated that explicitly parallel pathways achieve performance comparable to standard sequential models as models scale up. These parallel architectures deliver roughly a 15% training speedup due to concurrent computation. However, they reported this phenomenon solely as an empirical observation, leaving the underlying mechanism unexplained.

Therefore, standard tied transformers suffer the worst of both worlds: they lack the hardware efficiency of parallel blocks, yet they silently lose sublayer interaction due to depth scaling. This motivates us to consider two fundamental questions: Can we preserve this interaction as depth increases? And does preserving it improve learning performance? As Reviewers RjRJ and tu6F explicitly noted, these are meaningful and important research questions.

2. Addressing the Math: Euler Discretization and Decoupled Formulations We agree with the AC that the standard tied setup leads to vanishing interactions is due to Euler discretization: Using the AC's notation, X2=X0+hb0(X0)+hb1(X0+hb0(X0))≈X0+h(b0(X0)+b1(X0))+O(h2)X_2 = X_0 + hb_0(X_0) + hb_1(X_0 + h b_0(X_0)) \approx X_0 + h(b_0(X_0) + b_1(X_0)) + O(h^2)

However, our proposal is not X1=X0+hb1(b0(X))X_1 = X_0 + h b_1(b_0(X)) as a larger residual block. Instead, we introduce an independent, decoupled interaction coefficient cc: X2=X0+hb0(X0)+hb1(X0+cb0(X0))≈X0+hb0(X0)+hb1(X0)+O(h⋅c)X_2 = X_0 + hb_0(X_0) + hb_1(X_0 + c b_0(X_0)) \approx X_0 + hb_0(X_0) + hb_1(X_0) + O(h \cdot c)

Using c=Θ(1)c = \Theta(1) explicitly preserves the interaction as order-one under both Depth-μ\muP and CompleteP, while tied setups with c=h=L−αc=h=L^{-\alpha} cause it to decay (O(L−1/2)O(L^{-1/2}) and O(L−1)O(L^{-1})), as shown in Figure 1.

3. Long-Horizon Training Dynamics and Empirical Validation The AC questions whether our interaction results continue to hold after a large number of training steps. We used the prompt provided by the AC to replicate the simulation.

We observe that under Depth-μ\muP, the interaction RMS decays further, while CompleteP increases slightly, which perfectly matches our Figure 1. Beyond just the depth trends, we must examine the magnitudes as well: the magnitude remains exceptionally small (10−310^{-3} to 10−210^{-2} in our Figure 1, and 10−310^{-3} to 10−410^{-4} in the simulation), whereas an order-one decoupled coefficient keeps it robustly at magnitude 1.

Thus, long-horizon training cannot reverse the fact that standard tied Transformers yield parallel-like performance. As shown in our Figure 2, tied models and parallel models become nearly identical at higher depths (e.g., depth 16 and beyond), where parallel models can even slightly outperform tied ones in our tested setups. This also echoes and explains the empirical findings in PaLM: parallel models catch up to sequential ones primarily because vanishing interactions render them functionally parallel, even over extended training.

Ultimately, this proves that standard tied scaling traps the network in a parallel-like performance regime without providing the hardware speedups of true parallel execution. As Reviewers RjRJ and X2Vn noted, our empirical results successfully bridge the theoretical one-step analysis to multi-step training dynamics, confirming these insights are robust.

4. Feature Learning vs. Lazy Regime (Near-Initialization Analysis) The AC also questions whether our analysis is limited by focusing near initialization or on a single gradient update.

First, we want to clarify that our theoretical analysis operates strictly in the feature-learning regime (μ\muP framework) rather than the lazy/kernel regime. Thus, the one-step update captures genuine feature evolution rather than remaining trivially close to initialization. Furthermore, previous foundational studies (such as μ\muP and DMFT) routinely rely on order-one training horizons to yield insightful predictions.

Consequently, a one-step analysis in the feature-learning regime does not imply that the insights are restricted solely to near-initialization behavior; rather, it primarily stems from the natural mathematical challenges associated with proving deep non-linear dynamics. In fact, as demonstrated by our empirical findings and highlighted by Reviewers RjRJ and X2Vn, the insights derived from this analysis translate robustly and convincingly to multi-step training dynamics.

5. Why Should We Fix It? Model Design Implications for Practitioners Given that parallelism is critical in modern neural network training, we believe our findings provide practitioners with a clear structural trade-off:

  • If you prefer efficient training: Explicitly parallel blocks can be an excellent choice because their performance is competitive with standard tied models at scale while offering significant hardware efficiency and speedups.
  • If you prefer higher performance: Using a decoupled interaction coefficient—via fixed scaling or adaptive gates (Figures 3 and 4)—retains sublayer interaction and improves representation learning.

On the other hand, the current standard tied setup is suboptimal in either scenario: it lacks the computational efficiency of parallel structures yet yields parallel-like performance as models depth scales up. Our study gives researchers and practitioners the architectural flexibility to choose the right balance based on their preferences.

I hope that addresses the AC's main concerns, and we are happy to discuss more.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.