acceptodds
Under review as a conference paper at ICLR 2027

Criticality and Saturation in Orthogonal Neural Networks

Abstract

It has been known for a long time that initializing weight matrices to be orthogonal instead of having i.i.d. Gaussian components can improve training performance. This phenomenon can be analyzed using finite-width corrections, where the infinite-width statistics are supplemented by a power series in . In particular, recent empirical results by Day et al. show that the tensors appearing in this treatment stabilize for large depth, as opposed to the tensors of i.i.d.-initialized networks. In this article, we derive explicit layer-wise recursion relations for the tensors appearing in the finite-width expansion of the network statistics in the case of orthogonal initializations. We also provide an extension of recently-introduced Feynman diagrams for the corresponding recursions in the i.i.d.-case which are valid to all orders in . Finally, we show explicitly that the recursions we derive reproduce the stability of the finite-width tensors which was observed for activation functions with vanishing fixed point. This work therefore provides a theoretical explanation for the stability of nonlinear networks of finite width initialized with orthogonal weights, closing a long-standing gap in the literature. We validate our theoretical results experimentally by showing that numerical solutions of our recursion relations and their analytical large-depth expansions agree excellently with Monte-Carlo estimates from network ensembles.

Primary area:
learning theory
Keywords:
Neural Tangent Kernel, Finite-Width Neural Networks, Orthogonal Initialization, Neural Network Theory, Feynman Diagrams

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5BdMOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents deep MLPs whose weight matrices are random orthogonal matrices (Haar-distributed) instead of i.i.d. Gaussian. It uses the finite-width “effective theory”. Author validates the effectiveness of the approach proposed through Monte Carlo ensembles of tanh MLPs (n = 50, 600 networks). The paper claim is that normalized tensors such as V/K² and F/(KΘ) “saturate” with depth for orthogonal weights, instead of growing like ℓ/n as in the Gaussian case.

Novelty/Quality: The finding that orthogonality changes only the source terms of V, F and B is a useful clarification.

Critical strengths

A clean empirical and mathematical-based result about where orthogonality matters at finite width.

Critical weaknesses

Weakness: No training experiments. The introduction says no theory had shown that orthogonal init “leads to improved performance… In this article, we fill this gap”, and the abstract says it “closes a long-standing gap”. But the paper only analyzes statistics at initialization. Nothing links these tensors to training speed, loss or accuracy.

Questions for authors

1.Training experiments. Train tanh MLPs with orthogonal versus Gaussian init across several depth-to-width ratios ℓ/n. Check whether the predicted size of the NTK change during training, and the training speed, follow the theory. 2.Other activations. Extend to ReLU (the scale-invariant, K* ≠ 0 class), where orthogonal init is known to lose dynamical isometry, and to at least one other architecture.

1 comment

We thank the reviewer for highlighting that orthogonality changes only the source terms of VV, FF and BB.

Overclaiming. Indeed, we only analyze the network statistics via the tensors at initialization. However, Roberts & Yaida (2022, Ch. ∞\infty) showed that precisely these tensors evaluated at initialization are sufficient to capture the training dynamics at order 1/n1/n. In fact, the ten tensors we compute are exactly those that are needed to consistently truncate the training dynamics to order 1/n1/n. Therefore, our stability results also apply to trained networks. The gap in the literature that we fill with our contribution is that it was previously unknown why the tensors at initialization are stable in depth. The numerical results of Day et al. empirically confirmed the theoretical link between the stable tensors at initialization and the stability of the trained networks. However, their work could not establish theoretically why the tensors at initialization are stable.

Q1 (training experiments). The experiments which the reviewer proposes were carried out by Day et al. (Sec. 6), who found that orthogonal tanh networks train faster and exhibit much smaller NTK changes during training for deep networks (their Figs. 4–9), consistent with these saturating tensors.

Q2 (other activations). For ReLU, the finite-width corrections still grow linearly with depth, though with a reduced slope. Our new Sec. 5.4 explains this difference. The experimental verification of this that the reviewer asks for can be found in Day et al. App. B.

Soundness. We note that the concerns raised concern the scope of our claims rather than the correctness of the derivations. Our recursions are cross-checked algebraically, diagrammatically, and against Monte Carlo estimates up to ℓ=100>n\ell=100>n. We would be grateful for pointers to any specific step the reviewer considers incorrect.

Presentation. To improve readability, we have added precise definitions in Sec. 5.1 and a conclusion section.

Reviewer_mgnWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies why orthogonal weight initialisation leads to more stable behaviour in deep, finite-width neural networks. It extends existing theory by deriving equations describing how activation and gradient-related statistics evolve with depth. The theoretical development appears sound, and experiments show reasonable agreement with predictions. However, the results mainly explain the stability of network statistics at initialisation, rather than directly explaining improved training or generalisation.

Critical strengths
  1. The paper provides analytical results for several quantities whose saturation was previously demonstrated mainly through experiments.
  2. The proposed calculation method extends existing techniques to orthogonal networks and supports higher-order analysis.
  3. Numerical predictions closely match simulations, including experiments where network depth exceeds width.
Critical weaknesses
  1. The contribution of this paper over existing methods (Day et al. 2025, Guillen et al. 2026) is unclear. Additionally, the relationship to the works on self-normalising neural networks [1,2] needs to be discussed.
  2. Validation focuses on tanh networks without biases, mostly at small widths. The generality across widths, activations, and architectures remains unclear.
  3. The paper gives a general framework, but explicit analytical saturation results are largely restricted to the single-input tanh setting. The proof of the broader criticality result is also quite brief.
  4. The paper does not investigate whether the predicted stability leads to faster convergence, better feature learning, or improved generalisation. Its claim of explaining the practical benefits of orthogonal initialisation is therefore stronger than the evidence supports.
  • [1] Klambauer, Günter, et al. "Self-normalizing neural networks." Advances in neural information processing systems 30 (2017).
  • [2] Lu, Yao, et al. "Bidirectionally self-normalising neural networks." Neural Networks 167 (2023): 283-291.
Questions for authors
  1. Could this analysis be extended to training dynamics or generalisation?
1 comment

We thank the reviewer for noting that the theoretical development appears sound and that our results provide analytical control over quantities previously studied only experimentally. This is precisely how we view our contribution.

W1 (relation to prior work). Day et al. derived the recursion for VV, measured the NTK-related correlators numerically, and left both their analytic derivation and an all-orders treatment open. Our paper derives all ten recursions, shows that orthogonality modifies only those of VV, FF and BB, computes their large-depth asymptotics, and, crucially, identifies the mechanism behind saturation. Guillen et al. (2026) developed Feynman rules for Gaussian weights only. Extending them to orthogonal weights requires new ingredients, namely Weingarten weights, the orthogonality charge and the Möbius structure.

Thank you for pointing out the references about self-normalizing networks. These control the first two moments of the signal at the mean-field level, whereas we study the 1/n1/n fluctuations beyond it. We added a reference and short discussion to Sec. 2.

W2 (scope of validation). Bias-free networks are not a restriction, since Cb=0C_b=0 is precisely the critical point for tanh. At the critical point, the biases vanish at initialization and we can hence drop them, since our tensors are all evaluated at initialization.

Regarding width, small nn is the more demanding test of a 1/n1/n expansion. In fact, [Guillen et al. 2026] showed empirically that 1/n1/n expansions lead to accurate predictions down to n≈20n\approx 20. For large width, the higher-order contributions vanish at order 1/n21/n^2 and our results become more and more precise.

Regarding other activations, our recursions hold for any smooth activation, while saturation is specific to activation functions with vanishing fixed point, as explained in the general response.

W3 (single input and Theorem 5.1). The single-input setting is the standard one for analytic large-depth analysis, and our multi-input Monte Carlo results (Figs. 13–22) show the same behaviour. We have also made Theorem 5.1 precise and expanded its proof; please see our reply to Reviewer FKvN.

W4 and Q1 (training and generalization). Our analysis connects to training dynamics: by Roberts & Yaida (2022, Ch. ∞\infty), the network output after gradient-descent training is determined at O(1/n)O(1/n) by exactly the initialization tensors we compute, and this holds for orthogonal weights as well. Their saturation therefore controls the depth dependence of the finite-width corrections to the trained predictor. The empirical connection to faster convergence and better generalization was established by Day et al. (Sec. 6), and extending our analysis to a quantitative treatment of generalization is a natural next step.

Reviewer_FKvNOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates finite-width statistics of orthogonally initialized neural networks. It derives recursion relations for preactivation and NTK tensors, extends a Feynman-diagrammatic framework to orthogonal initialization, and provides a theoretical explanation for the saturation behavior observed in previous empirical work.

Critical strengths

Substantial theoretical contribution. The derivation of finite-width recursion relations and the extension of the diagrammatic framework to arbitrary orders in inverse width represent interesting technical contributions.

Relevant research question. Understanding the finite-width behavior of orthogonal neural networks is important, particularly given their advantages over Gaussian initialization.

Critical weaknesses

Unclear terminology. The paper repeatedly refers to "stability", "saturation", and "criticality" without providing sufficiently precise mathematical definitions. In particular, it is unclear whether stability refers to boundedness, polynomial growth, or merely the absence of exponential growth. Please clarify throughout the paper.

Overstated practical implications. While the paper provides a theoretical explanation for the behavior of finite-width statistics, unless I'm missing something, it does not directly establish that these properties explain improved optimization or generalization under orthogonal initialization. Please elaborate on that.

Questions for authors

Could the authors provide precise mathematical definitions of stability, saturation and criticality for the finite-width tensors? In particular, what exactly does Theorem 5.1 guarantee about their asymptotic behavior with depth?

1 comment

We thank the reviewer for the positive assessment and for the precise comments on terminology.

Definitions. We agree that these terms were used too loosely, and we now define them in Sec. 5.1. Writing O~(ℓ)\tilde{\mathcal O}^{(\ell)} for the normalized version of a finite-width tensor (Eq. 124), we call O\mathcal O critical if ∣O~(ℓ)∣≤Cℓq|\tilde{\mathcal O}^{(\ell)}|\le C\ell^q, meaning it neither grows nor decays exponentially with depth, and saturating if sup⁡ℓ∣O~(ℓ)∣<∞\sup_\ell|\tilde{\mathcal O}^{(\ell)}|<\infty. Saturation is strictly stronger: for Gaussian initialization with variance set to the critical value of the tanh nonlinearity, the normalized tensors are critical but grow linearly with depth, e.g. V~≃23ℓ\tilde V\simeq\tfrac23\ell, hence they are not saturating. What we previously called "stability" corresponds to criticality, and we now use only these two terms.

Theorem 5.1. The theorem guarantees criticality, not saturation. At any fixed order in 1/n1/n, each recursion takes the form

O(ℓ+1)=Ξ(ℓ)O(ℓ)+S(ℓ),\mathcal O^{(\ell+1)}=\Xi^{(\ell)}\mathcal O^{(\ell)}+\mathcal S^{(\ell)},

where the homogeneous coefficient Ξ(ℓ)\Xi^{(\ell)} is identical to the Gaussian case and orthogonality modifies only the source, through lower-rank tensors. Since the Gaussian tensors are critical, the criticality of all orthogonal tensors follows by induction on the rank. Saturation is a stronger statement that follows from the large-depth analysis in Secs. 5.3 and 5.4. We have restated the theorem in this form and expanded its proof.

Practical implications. As discussed in the general response, Roberts & Yaida (2022, Ch. ∞\infty) express the O(1/n)O(1/n) correction to the trained network output in terms of the tensors we compute at initialization, and their derivation carries over to orthogonal weights. For Gaussian initialization the normalized tensors grow linearly with depth, so this correction grows as ℓ/n\ell/n. For orthogonal initialization they saturate, so within the 1/n1/n expansion the correction does not grow with depth, and our Monte Carlo results indicate that this persists even for ℓ>n\ell>n. That this saturation results in improvements in training and generalization was established previously with empirical experiments by Day et al.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General response

We thank all reviewers for their careful reading and constructive feedback. Before answering individually, we address two points raised across the reviews.

Relation to training and performance. Our results connect to training through two complementary results. On the theoretical side, Roberts & Yaida (2022, Ch. ∞\infty) showed for Gaussian initialization that, at O(1/n)O(1/n), the output of a network trained by gradient descent is the infinite-width kernel prediction plus a correction determined by the initialization tensors V4,A,B,D,F,P,Q,R,S,T,UV_4,A,B,D,F,P,Q,R,S,T,U. This connection carries over to orthogonal initialization, since the gradient-descent solution is derived for a single network and the initialization distribution enters only through these ensemble averages, which have the same tensor decomposition and 1/n1/n scaling for orthogonal weights (Eqs. 6, 11, 26, 30–32). On the empirical side, Day et al. (2025) showed that these tensors saturate with depth for orthogonal but not for Gaussian initialization (their Figs. 2–3), and that orthogonal tanh networks correspondingly train faster and generalize better, with smaller NTK fluctuations during training (their Sec. 6, App. B). The analytic derivation of this saturation was left open in their work, and this is what our paper provides.

Mechanism and scope. In the new Sec. 5.4 we explain why saturation occurs. Orthogonal weights preserve norms, which fixes the quartic vertex to V=−2K2V=-2K^2. For activation functions with vanishing fixed point, the network becomes effectively linear at large depth, so this constraint persists and cancels the leading source terms of the DD and AA recursions, while those of FF and BB are reduced from O(K)O(K) and O(1)O(1) to O(K2)O(K^2). This also explains why the "noise" correlators AA and DD are strongly suppressed, as observed by Day et al. For ReLU, by contrast, the source term survives and V~(ℓ)=3ℓ−5\tilde V^{(\ell)}=3\ell-5 keeps growing (Day et al., Eq. 4.19), so saturation is specific to activation functions with vanishing fixed point.

Summary of revisions (marked in blue in the updated PDF):

  • Introduction: explained why the training link of Roberts & Yaida (2022) carries over to orthogonal weights.
  • Sec. 2: clarified the relation to Day et al. (2025) and added a discussion of self-normalizing networks.
  • Sec. 5.1: precise definitions of criticality and saturation, and a restated Theorem 5.1 (with an expanded proof in App. G.4).
  • New Sec. 5.4: mechanism of saturation and its scope.
  • New Sec. 6: conclusion.
  • Minor corrections: typos and references for the Gaussian comparison.
Area_Chair_sSLqOpenReview ↗

Discussion

Hi everyone,

First of all, I would like to encourage the reviewers to respond to the authors at this stage, given that we are past the half way point of the discussion period.

Second of all, I would like to comment on my views of this paper, given this is a topic that I have thought about myself. Hopefully this will also contribute towards the discussion, and I welcome reviewers to comment on my perspective as well.

Broadly speaking, I'm not sure if I believe in the program of studying the depth limits of networks at "criticality". This was an older perspective that I believe has outlived its usefulness, as the depth limit of the network is still numerically unstable. In fact, one can try to train such as network in this regime, and very quickly we will see that the features of different inputs at the same hidden layers are extremely correlated at initialization. This significantly weakens the gradient signal and leads to very poor training loss even.

See for example section 12 of the deep kernel shaping paper for a very detailed discussion: https://arxiv.org/abs/2110.01765

As a result, I believe the field should (and has started to) move on to depth dependent architecture scalings, such as the activation shaping approach proposed by the same deep kernel shaping paper, or using a residual architecture with depth dependent residual weight. See for example https://arxiv.org/abs/2405.15712

While I believe the perspective does not favour the paper for me personally, this is not sufficient for me to disregard the paper. So I will also consider the reviewers' perspectives as well, and I have also dug a bit deeper into the calculations with the help of codex.

I will say I applaud the authors for doing these calculations by hand, as codex found some calculation mistakes and other issues. I don't think they really hurt the paper's goals, but here are the main ones:

  • In equation 117, the term λW,+CWF+\frac{\lambda_{W,+}}{C_W}F_+ should be λW,+CWF++λW,+CWΘ⟨σ2⟩K⟨(σ′)2⟩K\frac{\lambda_{W,+}}{C_W}F_+ +\lambda_{W,+}C_W\Theta \langle\sigma^2\rangle_K\langle(\sigma')^2\rangle_K to include Hessian contribution.
  • After equation 167, you said "The scale ℓ0\ell_0 is fixed by the initial condition (161) after replacing ℓ=1\ell = 1 in (167)." which eventualy gets evaluated to log⁡ℓ0≈−2.74\log \ell_0 \approx -2.74. This is not as accurate because the higher order terms are not suppressed with small ℓ\ell, but codex suggests to use a known value at layer one, and use equation 5 to compute the sequence layer by layer, which gives a better estimate.

Again these are not serious issues, but I think it will help improve the paper regardless.

Finally, I encourage the reviewers to also engage with the discussion on these points as well. I hope we can reach a concensus with the discussion period.

AC

We thank the AC for the detailed engagement and for checking our calculations; we address the two computational points in a separate comment. Here we respond to the broader concern that, at criticality, features of different inputs become strongly correlated with depth (DKS, Sec. 12). All numbers below will be shown in a new appendix together with Figure (https://ibb.co/wFfGtVtw).

Which criticality. We follow Roberts & Yaida and call a network critical when χ∥=χ⊥=1\chi_\parallel=\chi_\perp=1. For tanh this requires CW=1C_W=1, Cb=0C_b=0. The edge-of-chaos initializations discussed in DKS tune only χ⊥=1\chi_\perp=1, with Cb>0C_b>0, which leaves χ∥<1\chi_\parallel<1 (e.g. χ∥≈0.52\chi_\parallel\approx0.52 for Cb=0.05C_b=0.05). The two regimes behave differently, and we will make this distinction explicit in the revision.

1. At criticality, correlations are preserved. For Cb=0C_b=0, K(ℓ)≃1/(2ℓ)K^{(\ell)}\simeq1/(2\ell), and the odd nonlinearity changes the input correlation only at O(1/ℓ2)O(1/\ell^2) per layer, which is summable. Iterating the two-input version of Eq. (5), ρ(1)=0.5\rho^{(1)}=0.5 converges to 0.4710.471 (ℓ=100\ell=100 and ℓ=1000\ell=1000 alike). Orthogonal networks of width 64–256 match this within errors (0.4710.471–0.4740.474 at ℓ=100\ell=100), with deviations shrinking as 1/n1/n. Edge-of-chaos tanh with Cb=0.05C_b=0.05 and critical ReLU, by contrast, collapse (ρ=0.94\rho=0.94 and 0.9970.997 at ℓ=100\ell=100). We confirmed by Monte Carlo that this collapse happens for orthogonal and Gaussian weights alike, faster for Gaussian weights. The AC's concern therefore applies to the edge-of-chaos regime, but not to the setting we study.

2. The orthogonal–Gaussian difference lives at finite width. At infinite width both initializations share the NNGP kernel and the NTK (Eq. 5; Huang et al., 2021). An infinite-depth degeneracy therefore cannot explain the advantage of orthogonal initialization; that advantage must come from finite-width effects, which is what our recursions capture. At width 64 and ℓ=100\ell=100, Gaussian networks drift away from the infinite-width prediction. Their mean correlation falls to 0.370.37–0.420.42, with a network-to-network spread about 15× larger than for orthogonal weights. Their mean kernel is about 30% too low, and n Var(K)/K2≈90n\,\mathrm{Var}(K)/K^2\approx90, versus 0.020.02 for orthogonal weights. The 1/n1/n truncation is also far more accurate for orthogonal weights: at L/n≈0.5L/n\approx0.5, our V4V_4 recursion is within 0.3% of Monte Carlo, while the Gaussian one is off by 15%.

3. The framework extends to depth-dependent architectures. Each derivation step goes from layer ℓ\ell to ℓ+1\ell+1, with coefficients that are Gaussian expectations at layer ℓ\ell. Layer- or depth-dependent activations therefore enter only through σ→σ(ℓ)\sigma\to\sigma^{(\ell)}. We verified this numerically for V4V_4, the NTK mean, FF and DD, with a shaped activation atanh⁡(z/a)a\tanh(z/a), a∝La\propto\sqrt L, and with alternating tanh/sin layers. In all cases the Monte-Carlo deviation from the recursion halves when nn doubles, as expected for an exact O(1/n)O(1/n) result. The criticality bound of Thm. 5.1 uses a fixed-σ\sigma property, so its hypothesis would need to be restated for shaped activations. Residual networks with depth-scaled branches require modified recursions and are future work. Existing finite-width results for shaped networks (Li, Nica & Roy, 2022) concern Gaussian weights and the covariance at initialization; our framework adds orthogonal weights and the NTK/dNTK/ddNTK statistics.

Planned revisions. A new appendix with the figure and contrast cases. A paragraph that defines criticality precisely, states that our saturation results concern Cb=0C_b=0, and discusses depth-dependent scalings (DKS; Li et al.; Bordelon et al., 2024).

Computational points

Eq. (117). The Hessian term was missing from the recursion for QQ in the manuscript. Our exact numerical solutions already included it, but our large-ℓ\ell expansions were derived from the recursion without it. We have corrected Eq. (117) and rederived the expansions of QQ and TT.

Scale ℓ0\ell_0. Following the AC’s suggestion, we now fix the constants of all large-ℓ\ell expansions by matching to the exact layer-by-layer solution at ℓ∗=200\ell_*=200, instead of at ℓ=1\ell=1. This gives log⁡ℓ0=3.68077\log\ell_0=3.68077, instead of −2.74-2.74. The NNGP expansion is now within 1% of the exact solution from ℓ=3\ell=3, and within 10−610^{-6} at ℓ=100\ell=100. All coefficients, expansions, figures and captions in App. J and Fig. 2(b) have been updated, and a new table lists the exact tensor values at ℓ=100\ell=100 and 200200 used for the matching.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.