acceptodds
Under review as a conference paper at ICLR 2027

It's better to be second: Second-Order Residuals prevent vanishing gradients

Abstract

The residual connection has long been viewed as crucial for the stable training of deep architectures, from multi-layered perceptrons and ResNets to the transformer that modern large language models (LLMs) are based upon. The common understanding is that it is necessary to preserve gradient flow and speed up training convergence. However, is that really true? In this work, we prove that residual connections do not prevent vanishing (or exploding) gradients. Furthermore, they can lead to inefficient parameter usage by blowing up the output norm over depth. Instead, we show that using second-order residuals, where x_t+1 is a function of two previous layers x_t and x_t-1, can provably prevent vanishing gradients regardless of model depth. Second-order residuals can be substituted into modern deep architecture for little overhead cost, leading to comparable or better downstream performance on several tasks (synthetic multihop retrieval and multitask parity, as well as large-scale language modeling).

Primary area:
foundation or frontier models, including LLMs
Keywords:
foundation model, llm, language model, transformer, vanishing gradients

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_rvycOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper revisits worst-case gradient failures of first-order residual connections and proposes a second-order recurrence using two previous layer states. Its theoretical argument uses the determinant of an augmented-state Jacobian to establish a non-vanishing spectral-norm property, and then claims a non-vanishing parameter-gradient guarantee. The practical update replaces the ordinary skip with a negative two-layer skip and a scaled tanh-transformed module output. Experiments report ten-run comparisons on synthetic multihop retrieval and multitask parity, plus 12/24/36-layer language models trained on 10B FineWeb-Edu tokens and evaluated on perplexity, standard zero-shot tasks, and BABILong.

Critical strengths
  1. The proposed connectivity change is simple enough to be experimentally useful, and the synthetic results cover multiple initializations and a much deeper regime. In particular, testing 128-layer models on concrete retrieval/parity tasks is more informative than showing that a modified residual can merely run. The language-model results also indicate that the alternative can produce usable models with broadly comparable conventional benchmark performance. This supports continued investigation of the architecture even though it does not establish the claimed guarantee.
Critical weaknesses
  1. Corollary 4.4 is false as stated. Consider the scalar recurrence of Eq. (7) with alpha=0, beta=-1, T=2, x_{-1}=1, x_0=-1, W_0=W_1=0, and the differentiable loss L(x_2)=x_2. Then x_1=-1 and x_2=1, so the intermediate states and dL/dx_2 are nonzero. However, dL/dW_0=W_1 x_0=0, whereas Eq. (10) claims a positive parameter-gradient lower envelope with right-hand side |dL/dx_2| |x_0|=1. This even uses the practical initial-condition relation x_0=-x_{-1}. The augmented state still has a norm-preserving Jacobian in this example; the problem is that a lower bound on the largest singular value does not preserve every loss-gradient direction or every parameter-gradient projection. Consequently, Appendix A.3's Eq. (23) cannot justify the claimed bound. This directly breaks the main inference that the architecture provably prevents the gradients relevant to learning from vanishing. In addition, Proposition 4.2's equality to 1 needs qualification for |beta|>1: the 2d-dimensional determinant argument implies sigma_max >= |beta|^(T/2)>1.

  2. The empirical evidence does not independently establish the mechanism claimed by the theory. Eq. (12) changes both the skip connectivity and the module output through scaled tanh. Appendix C.1 ablates initialization, but it does not isolate the two-layer skip from tanh or show actual parameter-gradient propagation across depth. Table 1's standard language-model results are essentially comparable, and the language-model experiments do not report the repeated-run uncertainty supplied for the synthetic tasks. After correcting the theorem, an attribution ablation and gradient measurements would be necessary to establish whether the observed deep-task improvements arise from the proposed anti-vanishing mechanism.

Questions for authors
  1. How do you reconcile Corollary 4.4 with the scalar counterexample above? What corrected theorem, and what additional assumptions on loss direction, readout, or Jacobian singular values, would guarantee non-vanishing parameter gradients rather than only a non-vanishing augmented-state operator norm?

  2. Can you correct Proposition 4.2's equality for |beta|>1 and distinguish the valid determinant/spectral-norm property from the stronger optimization conclusion throughout the paper?

  3. Can you isolate second-order connectivity from scaled tanh and measure per-layer parameter-gradient norms and singular-value behavior on the deep synthetic models? I recognize the initialization ablation already provided; the missing control concerns the remaining architectural changes. A valid revised theory and a mechanism-specific empirical result would be needed to change my recommendation.

1 comment
AuthorsOpenReview ↗

Response to Reviewer rvyc

Thanks for your feedback. We apologize for the delay in response as we needed to rerun our full set of experiments (which we will explain below). In response to your specific questions:

  1. We thank you for pointing this out: our original proof did not account for the norm-preserving direction of the residual Jacobian possibly lying inside the loss gradient's nullspace. We have updated the previous claim (Corollary 4.4) to a stronger one (Proposition 4.4). The key differences are that: (1) we need to additionally assume an upper bound on the spectral norms of the weight matrices Wt\mathbf{W}_t. The trade-off here is that this bound is exponentially tighter w.r.t. depth (can be improved and we are doing so in future iterations), but it still ensures strictly positive norm on dL/dWdL/d\mathbf{W} on all depths, preserving the central claim of our paper. Importantly, note that under the same conditions, the standard first-order residual can still vanish, so there is still a material difference between the two systems. Also, (2) we need to connect our model to the last two layers instead of the last layer alone. We have rerun our language experiments to reflect this new change. We note that it has made our results stronger, resulting in an improvement in validation PPL and downstream BABILong reasoning on all depths.

  2. The actual sign in Proposition 4.2 should be ≥\geq instead of ==. That was a typo and we've fixed the statement. The actual proof remains unchanged. Regarding your comment about "distinguishing the valid determinant/spectral-norm property from the stronger optimization conclusion", please see Point 1 above.

  3. Please see Appendix C for additional ablations. In all our experiments, the tanh⁡\tanh scaling made very little difference and it was more of an arbitrary choice for optimization safety rather than any sort of performance gain. We also attached some plots for gradient norms in Appendix C.3, showing that the first-order residual has smaller gradient norms on average (learning rate and all other hyperparameters are held constant) and gradient norms tends to decay more rapidly mid-training. Finally, we note that despite the language model performance being only slightly better, our model achieves better (synthetic) or slightly better (language) performance for essentially negligible cost and further comes with stronger gradient guarantees.

Reviewer_FoMvOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes second-order residual connections that use two preceding layer states to improve deep-network training. It presents theoretical gradient analyses and a practical implementation. Experiments show gains on deep synthetic tasks and some BABILong settings, while language modeling performance is broadly comparable to the baseline.

Critical strengths

1.The paper addresses an important foundational question about the limitations of standard residual connections.

2.The proposed connection is simple to integrate into transformers, with a reported FLOP overhead of only 0.24% in the 12-layer experiment.

Critical weaknesses
  1. The paper lacks direct measurements of gradient problems during training. Such evidence would establish practical relevance more convincingly than constructed counterexamples alone.

  2. Independent ablations of the negative skip coefficient, tanh⁡\tanh, and both d\sqrt{d} factors are missing. The initialization ablation is also incomplete, leaving the source of improvements unclear.

  3. Table 1 shows performance largely comparable to the baseline, with gains mainly on synthetic tasks and some BABILong settings. The 36-layer model remains small; a scaling study extending to at least 1B parameters is needed.

  4. Corollary 4.4 is false: take T=2T=2, W1=−αIW_1=-\alpha I, and L(x)=c⊤xL(x)=c^\top x, with c,x0≠0c,x_0\neq 0. Then ∇W0L=0\nabla_{W_0}L=0, while the claimed right-hand side is strictly positive.

  5. Equation (23) misapplies the norm inequalities: the row-vector factor lacks full column rank, and the tall factor lacks full row rank, violating the conditions required for positive lower bounds.

  6. Both the writing and formatting need substantial improvement.

Questions for authors

1.How can Corollary 4.4 be reconciled with the two-layer counterexample described above, and how is Equation (23) justified given the missing rank conditions?

2.Could you provide controlled ablations of the negative skip coefficient, tanh⁡\tanh, and both d\sqrt{d} scaling factors under matched initialization and training settings?

1 comment
AuthorsOpenReview ↗

Response to Reviewer FoMv

Thanks for your feedback. We apologize for the delay in response as we needed to rerun our full set of experiments (which we will explain below). In response to your specific questions:

  1. We thank you for pointing this out: our original proof did not account for the norm-preserving direction of the residual Jacobian possibly lying inside the loss gradient's nullspace. We have updated the previous claim (Corollary 4.4) to a stronger one (Proposition 4.4). The key differences are that: (1) we need to additionally assume an upper bound on the spectral norms of the weight matrices Wt\mathbf{W}_t. The trade-off here is that this bound is exponentially tighter w.r.t. depth (can be improved and we are doing so in future iterations), but it still ensures strictly positive norm on dL/dWdL/d\mathbf{W} on all depths, preserving the central claim of our paper. Importantly, note that under the same conditions, the standard first-order residual can still vanish, so there is still a material difference between the two systems. Also, (2) we need to connect our model to the last two layers instead of the last layer alone. We have rerun our language experiments to reflect this new change. We note that it has made our results stronger, resulting in an improvement in validation PPL and downstream BABILong reasoning on all depths.

  2. Please see Appendix C for additional ablations. In all our experiments, the tanh⁡\tanh scaling made very little difference and it was more of an arbitrary choice for optimization safety rather than any sort of performance gain. The choice of negative over positive coefficient in Equation (12) is also an artifact of our proof in Proposition 4.4, addition is equally valid and we ablate for it too.

We also want to address some points mentioned in the Weaknesses section:

  1. Please see Appendix C.3 for plots of gradients norms, displaying empirical evidence that the first-order residual tend to have smaller gradient norms on average (learning rate and all other hyperparameters are held constant) and its gradient norms tends to decay more rapidly mid-training.

  2. Addressed in Questions.

  3. As mentioned above, our language results have improved after making a change to our model. Our updated paper shows these improved results.

4/5. Addressed in Questions.

Reviewer_7VMdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes second-order residual connections, where each layer depends on the two preceding states, and argues that this design prevents vanishing gradients. Experiments show improvements on deep synthetic tasks and broadly comparable language-modeling performance.

Critical strengths

The change is simple and adds little computation. For the 12-layer model, the authors report only a 0.24% increase in FLOPs.

The experiments cover both synthetic tasks and language modeling. The improvements on the synthetic tasks at 128 layers are encouraging and are reported across multiple runs.

Critical weaknesses

It is unclear why the earlier state should be subtracted. Section 4.1 says that choosing β=−1\beta=-1 helps control output norms. However, Propositions 4.2 and 4.5 also allow β=+1\beta=+1, so these results do not explain why subtraction is preferable. A direct comparison between the two signs, using the same nonlinear function and initialization, is needed.

The paper does not show that its Pre-LN baselines suffer from serious gradient vanishing. Proposition 3.2 gives a constructed example: setting W0=−αIW_0=-\alpha I makes the first layer cancel exactly. This establishes a worst-case possibility, but does not show that the problem occurs during normal training with suitable initialization and residual scaling. Measurements of gradients across layers are needed to connect this example to the reported training difficulties. Activation growth and small contributions from deeper layers should also be distinguished from vanishing gradients.

The guarantee for parameter gradients appears incorrect. A lower bound on the largest singular value of the full state Jacobian does not guarantee that the actual loss gradient is preserved. Consider xt+1=wtxt−xt−1x_{t+1}=w_tx_t-x_{t-1}, with x−1=−1x_{-1}=-1, x0=1x_0=1, w0=1w_0=1, and w1=0w_1=0. Then x1=2x_1=2 and x2=−1x_2=-1. For L(x2)=x22/2L(x_2)=x_2^2/2, we have ∂L/∂w0=0\partial L/\partial w_0=0, although both x0x_0 and L′(x2)L'(x_2) are nonzero. This appears to contradict Corollary 4.4.

The effects of the different changes are not sufficiently separated. The method changes the connection, adds a scaled tanh function, and uses a different initialization. The initialization ablation is only performed at 12 layers, whereas the strongest gains occur at 128 layers. Further ablations are needed to establish which changes produce these gains.

The evaluation needs deeper language models and stronger baselines. The 128-layer experiments are limited to synthetic tasks, while the language models have at most 36 layers and perform broadly similarly to standard residuals. The authors should test deeper language models and compare against DeepNorm, carefully tuned residual scaling, and Progressive Residual Warmup under comparable training budgets. These comparisons would help establish whether changing the connection offers benefits beyond existing ways of improving deep-network training.

Questions for authors

Why is subtracting xt−1x_{t-1} preferable to adding it, when the nonlinear function and initialization are held fixed?

Do appropriately initialized and scaled Pre-LN baselines show substantial gradient decay, and does this explain their training difficulties?

How do the authors reconcile Corollary 4.4 with the scalar example above?

Do the improvements persist in deeper language models when compared with stronger stabilization methods?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 7VMd

Thanks for your feedback. We apologize for the delay in response as we needed to rerun our full set of experiments (which we will explain below). In response to your specific questions:

  1. There is no material difference, our choice was motivated early experiments and Proposition 4.4, but addition will work too. See Appendix C.1 for an ablation on the additive version.

  2. Please see Appendix C.3 for plots of gradients norms, displaying empirical evidence that the first-order residual tend to have smaller gradient norms on average (learning rate and all other hyperparameters are held constant), decaying more significantly after some number of iterations. Note that the baseline first-order residual system here is initialized and scaled properly and we make no other changes.

  3. Our original proof did not account for the norm-preserving direction of the residual Jacobian possibly lying inside the loss gradient's nullspace. We have updated the previous claim (Corollary 4.4) to a stronger one (Proposition 4.4). The key differences are that: (1) we need to additionally assume an upper bound on the spectral norms of the weight matrices Wt\mathbf{W}_t. The trade-off here is that this bound is exponentially tighter w.r.t. depth (can be improved and we are doing so in future iterations), but it still ensures strictly positive norm on dL/dWdL/d\mathbf{W} on all depths, preserving the central claim of our paper. Importantly, note that under the same conditions, the standard first-order residual can still vanish, so there is still a material difference between the two systems. Also, (2) we need to connect our model to the last two layers instead of the last layer alone. We have rerun our language experiments to reflect this new change. We note that it has made our results stronger, resulting in an improvement in validation PPL and downstream BABILong reasoning on all depths.

  4. Due to compute limitations, we were unable to train significantly deeper/larger models on the language dataset. The synthetic experiments, which are built off similar reasoning tasks to language models, help to establish some evidence that there is a difference at high depth.

Finally, please also see Appendix C.1 for additional ablations of the factors that you mentioned in the Weaknesses section, showing that they are not accountable for the difference in performance.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for the time spent to review our work. We have updated our manuscript and ask all reviewers to refer to this new version. Since all reviewers shared similar critique, we will reiterate these points in a general response here below.

  1. The biggest issue was a flaw in Corollary 4.4 that did not account for the fact that the norm-preserving direction of the Jacobian can be orthogonal to the loss gradient. We have fixed Corollary 4.4 (now renamed Proposition 4.4) by making two changes: (a) adding an extra assumption in the form of a bound on parameter norms and (b) changing the model design such that the output/unembedding layer connects to the final two layers instead of just the final layer.

    While (a) is a more restrictive assumption than our original version, it still preserves our central claim that gradients cannot vanish regardless of depth. In contrast, the standard first-order system can still vanish even under these new assumptions, showing that there is still a material difference between the two systems after fixing the math.

    Most importantly, our fix here inspired us to make a change to our model design and we reran all the language experiments post-fix. We note that our new results are stronger than the initial submission, showing better validation perplexity across all depths.

  2. We have added multiple ablations in Appendix C.1 and provided empirical evidence of decaying gradient norms (for the standard system) in Appendix C.3. We note that these ablations do not account for significant differences from the main results we report.

We note that our approach leads to better performance on both synthetic (much stronger) and language experiments (slightly stronger) for negligible computational overhead (no change to model design beyond the residual, and compatible with other interventions on architecture).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.