acceptodds
Under review as a conference paper at ICLR 2027

Curvature, Residual Scale, and Signal Propagation in Hyperbolic Neural Networks

Abstract

How do curvature and residual scale jointly determine signal propagation in hyperbolic neural networks? We study intrinsic input sensitivity for residual layers on hyperbolic space of sectional curvature . Along an aligned reference trajectory, we derive an exact Jacobi-field expression for the layer differential and identify a unique transition between contraction and expansion. The transition depends on the dimensionless residual displacement and the covariant derivative of the residual field, rather than on curvature alone. We establish sufficient conditions for these regimes to persist under bounded derivative perturbations and turning of the reference direction. A constrained neural parameterization realizes these conditions, and explicit smoothness estimates extend the reference analysis to contracting neighborhoods and finite-time escape from expanding cones. We further construct a nonlinear correction that preserves the reference trajectory and its entire input-Jacobian product while changing the network away from that trajectory. Numerical experiments verify the differential formulas and sufficient bounds. Controlled learning experiments reveal limited nonlinear-task performance for the original parameterization and substantial improvements after adding the correction, including when the reference Jacobian remains fixed throughout training. These results separate certified reference sensitivity from nonlinear learning behavior and provide a setting in which their relationship can be studied explicitly.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Riemannian Geometry, Hyperbolic Neural Networks, Dynamical Systems

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JZLAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies intrinsic input sensitivity in hyperbolic residual networks of the form Fℓ(x)=exp⁡x(hVℓ(x))F_\ell(x)=\exp_x(hV_\ell(x)). For an aligned reference trajectory with structured covariant damping, it derives an exact differential and a unique contraction–expansion threshold governed jointly by curvature, residual displacement, and damping. It then provides sufficient bounds that allow derivative defects and turning, a constrained neural realization, depth-uniform neighborhood contraction, and finite-time cone escape. Numerical experiments check the formulas and certificates. With the original backbone parameters frozen, a correction that vanishes in value and first derivative at the reference points improves a synthetic nonlinear classification task while preserving the augmented backbone's reference trajectory and complete reference Jacobian product throughout correction training.

Critical strengths
  1. The paper provides a precise, intrinsic account of the interaction between curvature, residual displacement, and covariant damping. The exact threshold is complemented by bounds allowing turning and derivative defects, a realizable constrained parameterization, and finite-neighborhood results.

  2. The fixed-reference learning experiment is carefully controlled. Freezing the original backbone parameters and training a correction with zero value and derivative at each reference point preserves the augmented backbone's reference trajectory and full reference Jacobian product, while substantially improving XOR-type accuracy.

Critical weaknesses

My main concern is the significance and transferability of the insights beyond the controlled construction. The parameterization in Section 4.1 is expressly designed to satisfy the derivative and trajectory conditions, and it remains unclear how the resulting quantities would diagnose or guide a residual network outside this realization. The learning study does not substantially close this gap: a correction vanishing to first order can change nonlinear behavior in a general smooth coordinate chart, and the tested multiplicative correction supplies the type of second-order interaction required by the four-dimensional XOR task. Moreover, the input radius of 0.02 exceeds the reported sufficient contracting radii (Appendix C.4), so these experiments do not demonstrate how the finite-neighborhood certificates inform learning in the tested input regime. I recognize that the authors explicitly disclose these limitations, and the controlled results remain valid. Nevertheless, I am not yet persuaded that the combination yields a sufficiently substantial additional insight for acceptance.

Questions for authors
  1. Could you provide a concrete vector-field or residual-network example beyond the realization in Section 4.1 for which zℓz_\ell, the derivative defect, and turning can be computed or estimated? What nontrivial propagation conclusion or design decision would the conditions reveal in that example? An analytic example would help establish the contribution's transferability without requiring a large-scale benchmark.

  2. What additional learning insight is obtained from the fixed-reference experiment beyond the general freedom to change higher-order terms while preserving first-order information? In particular, is there a specific conclusion attributable to the curvature-dependent propagation analysis? A clear analytic comparison with the flat case, or a focused matched control, could help isolate the knowledge gained from this part of the study.

Reviewer_qS52OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
1
Soundness
2
Read review
Paper summary

The paper focuses on the components affecting signal propagation in hyperbolic residual networks. They define a residual layer as F(x) = exp_x(h V(x)), the hyperbolic analogue of x + h V(x). Here h scales the residual branch. The authors ask how the manifold's curvature and h together decide whether the embeddings of two nearby inputs move apart or closer together from layer to layer. Negative curvature pushes them apart, while the covariant derivative of V can pull them back together. The authors argue that curvature alone doesn't decide which effect is greater, and instead focus on the residual displacement in curvature units, z = kappa h ||V||, compared to the amount of damping. To make this exact they follow one fixed reference path through the network and compute the layer Jacobian along it in closed form using Jacobi fields. Perturbations orthogonal to the path get scaled by cosh z - a sinh(z)/z, which gives a single threshold between contraction and expansion (Thm 4.1). They then show this survives small deviations and give a constrained layer parameterization that meets the conditions by construction.Lastly they add a trainable correction term to each layer whose value and first derivative are zero at the reference points. So on the reference path the network and its Jacobian are exactly the same as before and the guarantees on the path still hold. But for inputs off the path the correction is nonzero and it changes the function that the network does. They use this to show that the sensitivity analysis on the path doesn't determine what the network can learn. They then verify their method on small synthetic datasets.

Critical strengths
  • The proofs and theorems in the paper seem to hold, and I don't see any problems with the mathematical reasoning.
  • Using z = kappa h v instead of curvature alone makes comparisons across curvatures more meaningful. *The correction experiment is able to properly show that first-order sensitivity at the reference doesn't decide how the model actually learns.
Critical weaknesses
  • My main concern is that the core of theorem 4.1 seems to already exist in the numerical analysis literature. [1] studies the same step y -> exp_y(h X(y)) and writes its variation as a Jacobi field with the same initial data. It also solves it on constant curvature with the same cosh / sinh(k)/k split with k = h||X|| sqrt|K|, and gives step-size conditions for non-expansiveness on hyperbolic space. [2] and the work on non-expansive ResNets [3, 4] are also not cited. I think the authors should better address these and explain where the work stands in comparison
  • Some other related work is also missing as well. Mainly, [5] builds exp-map gradient-step layers exp_x(-tau grad V(x)) in hyperbolic space and the authors show state that they are non-expansive globally, without needing a reference trajectory. [6] already has depth-compounding sinh(z)/z factors for hyperbolic GNNs, and [7] would fit in the geometric sensitivity paragraph.
  • The guarantees seem to hold at a single point and a small volume around it and not for the entire network since the reference trajectory only depends on (v_l, t_l) and not on W_l, B_l or the data. Cor. 4.4 then only directly relates to the input p_0. The certified radii (2.3e-3 at kappa = 0.6 and 3.7e-5 at kappa = 1.5) are much smaller than the input scale used in the experiments (0.02). The limitations mention this but the title and abstract read like general claims about signal propagation, which is slightly misleading.
  • The experiments are on a synthetic dataset with non-parameter matched baselines. They show that the math holds but we can't derive meaningful statements on the actual impact of this on performance.

[1] Ghirardelli, Owren, Celledoni. "Conditional Stability of the Euler Method on Riemannian Manifolds."

[2] Arnold, Celledoni, Cokaj, Owren, Tumiotto. "B-stability of numerical integrators on Riemannian manifolds."

[3] Celledoni, Murari, Owren, Schonlieb, Sherry. "Dynamical systems-based neural networks."

[4] Sherry, Celledoni, Ehrhardt, Murari, Owren, Schonlieb. "Designing stable neural networks using convex analysis and ODEs."

[5] Murari et al. "1-Lipschitz Neural Networks on Hadamard Manifolds."

[6] Balla. "Over-squashing in Riemannian Graph Neural Networks."

[7] Guo, Wang, Chen, Yu. "Clipped Hyperbolic Classifiers Are Super-Hyperbolic Classifiers."

Questions for authors
  1. Which parts of theorems 4.1 and 4.2 are new compared to [1] and [2]?
Reviewer_fjK8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how curvature and residual scale affect feature propagation in hyperbolic neural networks. For a specific architecture of hyperbolic neural networks, the authors establish the Jabobian field for a layer and then a phase transition between contraction and expansion. The authors perform theoretical analysis under perturbations and numerical analysis to verify the formulae.

Critical strengths
  1. It is interesting to study neural networks using language and techniques from dynamical systems. I think it will provide a unique perspective for understanding the behavior of neural networks.
  2. The proposed framework seems useful in analyzing perturbation and sensitivity in hyperbolic neural networks. There seems to be some important insight around the quantity κhv\kappa h v.
  3. The theorems are carefully stated with necessary assumptions, and validated numerically.
Critical weaknesses
  1. It appears to me that the same analysis can be carried out in the Euclidean space. Why is hyperbolic space so special? Which property of hyperbolic space is so important in the proofs that are not doable in Euclidean space? Can there be a head-to-head comparison between Euclidean and hyperbolic spaces when applied to similar architectures of residual networks and using a similar analysis?
  2. Although the authors carefully study a specific type of hyperbolic neural networks, I have never seen a similar architecture used in practice. In particular, use of residual layers in hyperbolic space is not as common as in Euclidean space. The scope might be too limited. It seems that the residual structure is in place because it has to be, in order to connect to some tools in dynamical systems.
  3. The paper is hard to read, not just because the heavy use of mathematical formulas, but also because the use of language is not careful. For instance, in the title the authors say “signal propagation”, but it is unclear what is a “signal” and the authors never have defined it. Only after reading the paper, I understand that a signal is simply a feature or an input of the neural network. Similarly, one can expect that curvature is important, but why is “residual scale” important? I actually have not heard the term “residual scale” before. Also, from “residual scaling and stable architectures” in Related Works, it is unclear what is the prior belief about the effect of “residual scaling”, and what makes it an important problem to study.
Questions for authors

See "Weaknesses".

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.