- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper presents deep MLPs whose weight matrices are random orthogonal matrices (Haar-distributed) instead of i.i.d. Gaussian. It uses the finite-width “effective theory”. Author validates the effectiveness of the approach proposed through Monte Carlo ensembles of tanh MLPs (n = 50, 600 networks). The paper claim is that normalized tensors such as V/K² and F/(KΘ) “saturate” with depth for orthogonal weights, instead of growing like ℓ/n as in the Gaussian case.
Novelty/Quality: The finding that orthogonality changes only the source terms of V, F and B is a useful clarification.
Critical strengths
A clean empirical and mathematical-based result about where orthogonality matters at finite width.
Critical weaknesses
Weakness: No training experiments. The introduction says no theory had shown that orthogonal init “leads to improved performance… In this article, we fill this gap”, and the abstract says it “closes a long-standing gap”. But the paper only analyzes statistics at initialization. Nothing links these tensors to training speed, loss or accuracy.
Questions for authors
1.Training experiments. Train tanh MLPs with orthogonal versus Gaussian init across several depth-to-width ratios ℓ/n. Check whether the predicted size of the NTK change during training, and the training speed, follow the theory. 2.Other activations. Extend to ReLU (the scale-invariant, K* ≠ 0 class), where orthogonal init is known to lose dynamical isometry, and to at least one other architecture.
We thank the reviewer for highlighting that orthogonality changes only the source terms of V, F and B.
Overclaiming. Indeed, we only analyze the network statistics via the tensors at initialization. However, Roberts & Yaida (2022, Ch. ∞) showed that precisely these tensors evaluated at initialization are sufficient to capture the training dynamics at order 1/n. In fact, the ten tensors we compute are exactly those that are needed to consistently truncate the training dynamics to order 1/n. Therefore, our stability results also apply to trained networks. The gap in the literature that we fill with our contribution is that it was previously unknown why the tensors at initialization are stable in depth. The numerical results of Day et al. empirically confirmed the theoretical link between the stable tensors at initialization and the stability of the trained networks. However, their work could not establish theoretically why the tensors at initialization are stable.
Q1 (training experiments). The experiments which the reviewer proposes were carried out by Day et al. (Sec. 6), who found that orthogonal tanh networks train faster and exhibit much smaller NTK changes during training for deep networks (their Figs. 4–9), consistent with these saturating tensors.
Q2 (other activations). For ReLU, the finite-width corrections still grow linearly with depth, though with a reduced slope. Our new Sec. 5.4 explains this difference. The experimental verification of this that the reviewer asks for can be found in Day et al. App. B.
Soundness. We note that the concerns raised concern the scope of our claims rather than the correctness of the derivations. Our recursions are cross-checked algebraically, diagrammatically, and against Monte Carlo estimates up to ℓ=100>n. We would be grateful for pointers to any specific step the reviewer considers incorrect.
Presentation. To improve readability, we have added precise definitions in Sec. 5.1 and a conclusion section.