acceptodds
Under review as a conference paper at ICLR 2027

Certifying Residual Architectures from Their Primitives: A Sharp Stability Threshold

Abstract

Whether a deep residual architecture trains stably is usually determined by training it, which is expensive and answers the question only for the architecture, depth, and floating-point format that were tried. In practice stability is secured by heuristics on where in the residual block to place a normalization and which one to use; each is supported by experiments, and no common principle explains why and when they work. We show that stability can instead be certified before training, from the architectural primitives of the residual block, as an explicit function of depth and floating-point format. The certificate rests on two elements: (a) on a power-law growth bound for the residual block, , whose exponent is computed from the block's primitives by an arithmetic of exponents (forward); (b) on bounds on the gradients through a Lipschitz condition on the block over the reachable states (backward). The certificate determines the depth at which the state can reach the largest finite floating-point value : for , overflow requires at least order layers, whereas for it can occur within order layers; the threshold is sharp. Every normalization and every bounded activation sets , and the arithmetic identifies minimal modifications that bring a block from to without normalization. Empirically, in forecasting only blocks blow up, as the forward certificate predicts; in GPT-2 on OpenWebText, blocks without normalization also diverge, and there the forward certificate holds while the backward one fails, in the attention block with the largest query-key coefficient. Relaxing normalization from to improves out-of-distribution generalization in operator learning, and gives accuracy gains in time-series forecasting. As recent deep models, including foundation models, grow more complex and more costly to train, our results give a way to settle the trade-off between stability and representational flexibility of a residual architecture directly from the primitives of its blocks, before any training.

Primary area:
learning theory
Keywords:
Residual networks, Layer normalization, Stability, Deep learning theory, Neural architecture design

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_73oQOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates under what conditions residual networks maintain finite forward and backward costs within a given depth, without requiring global Lipschitz continuity. The author decomposes the growth behavior of each residual block into two exponents for discussion.

Critical strengths

1.The paper explains that q=1 is a sharp threshold. 2.The paper establishes a set of primitive level exponential arithmetic, which enables researchers to symbolically derive the upper bounds (q, σ) of each residual block from the computational graph of the model before training, and locate potential unstable primitives based on this.

Critical weaknesses

1.The core theoretical prediction of the paper is the authentication depth, but the experiment has never directly calculated or reported specific values, nor has it demonstrated the correspondence between the authentication depth predicted by the theory and the actual observed explosion depth. 2.The boundary that the theory relies on in the most relevant scenario (attention) is acknowledged by the author as potentially loose, weakening the credibility of quantitative predictions.

Questions for authors

1.Why does the paper choose "layer-by-layer uniform boundary" instead of "trajectory adaptive boundary", and what impact does this choice have on the universality of the conclusion? 2.Compared with spectral normalization, ReZero/LayerScale, and stability analysis of neural ODE, in what sense does the "authentication" in this article provide something they do not have?

Reviewer_VjcMOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

For the residual update Xk+1=Xk+Δt v(Xk)X_{k+1} = X_k + \Delta t\, v(X_k), the paper proposes a certificate that computes, before training, the depth Dcert=min⁡{DF,D∇}D_{\mathrm{cert}} = \min\{D_F, D_\nabla\} up to which states and costates stay below the largest finite floating-point value MM, from a growth exponent qq of the block (∥v(x)∥≤c∥x∥q+b\|v(x)\| \le c\|x\|^q + b) and a Jacobian growth exponent σ\sigma (L(R)≤ℓ1Rσ+ℓ0L(R) \le \ell_1 R^\sigma + \ell_0). The reachable radius grows polynomially for q<1q<1, exponentially for q=1q=1, and doubly exponentially for q>1q>1 (Thm 2.2), making q=1q=1 a sharp threshold (Table 1). An arithmetic of exponents for composition, sums, products, residual wrapping, and normalization (Thm 3.1, B.2), together with a table of exponents of standard primitives (Table 2), allows the pair (q,σ)(q,\sigma) of a block to be derived mechanically from its computation graph (attention is (1,2)(1,2), a gated MLP is (2,1)(2,1), un-normalized Mamba has q=5q=5, and so on). Empirically, in time-series forecasting only the no-LN Mamba block with q=5q=5 blows up while a linear-growth block repaired to q=1q=1 is stable; in operator learning the no-LN FNO (q=1q=1) extrapolates better; in GPT-2 the no-LN model diverges despite q=1q=1, on the backward side through the σ=2\sigma=2 attention block; and in Toto 2.0 the q=2q=2 gated MLP amplifies first.

Critical strengths
  1. With extremely simple rules (composition multiplies exponents, sums take the max, Hadamard products add, normalization sets 0), the paper treats Transformers, Mamba, FNOs, and gated MLPs in one framework and derives concrete repairs to q=1q=1 without normalization (the linear-growth Mamba block, QK-norm). The distance from theory to design is short, and the procedure is reproducible (App. B.5).

  2. Experiments across several architectures and tasks exhibit both the forward side (only q>1q>1 blocks blow up) and the backward side (divergence at q=1q=1 through σ=2\sigma=2), providing empirical support that the two exponents are independently necessary.

Critical weaknesses
  1. There is a gap in wording between the one-directional nature of the certificate (a sufficient condition) and the empirical claims ("only q>1q>1 blows up", "the backward certificate localizes the failure"). The GPT-2 and Toto 2.0 results demonstrate consistency with the theory, not its predictive power.

  2. The certificate says nothing about learning performance. What it provides is only the depth DcertD_{\mathrm{cert}} reachable without overflow; no result relates the value of DcertD_{\mathrm{cert}} or of (q,σ)(q,\sigma) to any accuracy or generalization metric. The claims about accuracy and OOD performance (§4.1, 4.2, App. D.6) are empirical comparisons, among stable models, of normalization present (q=0q=0) versus absent (q=1q=1), and since removing normalization also changes parameter count, optimization, and affine parameters, the mechanism is not isolated (as the authors acknowledge). There is also no theoretical explanation of why q=1q=1 should extrapolate better. App. D.3 mentions a Welch t-test, but no significance results appear in the main text. Of the abstract's claim that the trade-off between stability and representational flexibility can be settled from the primitives, the paper therefore supports only the stability side; the flexibility side should be withdrawn or stated as a limited empirical observation.

Questions for authors
  1. Did the divergence of no-LN GPT-2 actually occur as an overflow (non-finite value), or did the loss degrade first? Since the forward norm stays small in Figure 3c, please state what became non-finite first on the backward side (the softmax of the attention logits, or the gradients).

  2. L(RD)L(R_D) in Thm 2.3 is a supremum over all of Θ\Theta and the ball of radius RDR_D. What are the actual certificate values (D∇D_\nabla) for the trained Pre-LN and no-LN GPT-2 weights? The certificate itself, rather than the surrogate in Table 13, would show how conservative the theory is.

Reviewer_FNUvOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a theoretical framework for investigating numerical instability in deep residual neural networks. Specifically, it analyzes activation growth in forward propagation and gradient amplification in backward propagation independently, deriving mathematical conditions that enable the assessment of network stability before training. The authors also validate their theoretical findings through extensive experiments involving diverse neural network architectures.

Critical strengths

(1) The paper offers a theoretical contribution by separately examining forward and backward stability in deep residual neural networks. It establishes a mathematical framework for evaluating the stability of practical neural network architectures, providing valuable insights into the mechanisms underlying activation growth and gradient amplification, as well as their potential roles in causing numerical instability during training.

(2) While rigorously verifying the proposed theoretical results in practical neural networks is inherently challenging, the authors provide convincing empirical evidence through carefully designed experiments across a variety of architectures. In particular, the experimental investigations of forward and backward instability complement the theoretical analysis, highlighting the relevance of the proposed framework and its potential applicability to practical neural networks.

Critical weaknesses

Although the authors present empirical results supporting the proposed theoretical framework, the extent to which its predictions reliably characterize training stability in practical neural networks remains insufficiently explored.

In particular, while the proposed growth exponents (q) and (\sigma) serve as useful indicators of potential architectural instability, actual training stability is also influenced by other factors, including model weights and growth coefficients. Consequently, these exponents alone may not provide a sufficiently comprehensive characterization of stability or guarantee stable training in practice.

Questions for authors

The experiments show that some (q=1) architectures outperform their (q=0) counterparts in predictive accuracy and OOD generalization. However, modifying normalization or gating operations also changes other architectural and optimization properties. Could the authors provide additional ablation studies to clarify whether the observed improvements are primarily associated with the growth exponent (q) or with these accompanying changes?

Comments on the paper

1 comment
Area_Chair_RF2fOpenReview ↗

Reminder to reviwers to read author's responses

Dear Reviewers,

Thank you for your efforts in providing thorough reviews.

As the discussion period is underway, I would like to kindly remind you to read the authors' responses to your reviews and provide follow-up comments where appropriate or leave a comment. In particular, please consider whether the responses address your questions and concerns, and update your assessment if the clarifications or additional information affect your initial evaluation.

Your engagement in this discussion is important for ensuring a fair and thorough review process.

Thank you again for your time and contributions!

Best regards, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.