- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper assembles Klein-coordinate versions of MLR/BMLR classifiers, FC/BFC layers, and GyroBN into a hyperbolic neural-network toolkit. It derives closed-form expressions and evaluates predictive performance, numerical behavior, and runtime against other hyperbolic implementations. The central claim is that the Klein realization offers computational and numerical advantages while maintaining competitive accuracy.
Critical strengths
- The manuscript is carefully organized, with clear figures, comparison tables, and extensive supplementary explanations.
- The evaluation reports multiple seeds, significance tests, and runtime measurements, providing useful detail beyond headline accuracy.
Critical weaknesses
- Limited substantive contribution and insufficient theoretical support
W1 — Methodological novelty. Sections 3.1–3.3 largely instantiate existing signed-distance, Busemann, and FC constructions in Klein coordinates. The authors explicitly position the contribution at the framework level, but assembling these counterparts does not establish a substantive new learning mechanism or a compelling methodological advance.
W2 — Theory does not establish the claimed advantages. The derivations establish closed forms and geometric correspondences, rather than comparative stability or optimization guarantees. Proposition E.2 provides only special-case equivalences and a local midpoint approximation; it does not explain the claimed training advantages, and its applicability to the actual GyroBN experiments is not established.
2. The experiments do not convincingly isolate the promised advantages
W3 — Stability diagnostics do not compare the same geometric objective. Table 4 differentiates the squared Euclidean norm of each model's coordinates, not a common intrinsic loss; gradient differences therefore need not indicate numerical instability. Moreover, Table 15 reports only 0.99% Klein accuracy without clipping, versus 0.5% chance: avoiding NaNs is not equivalent to effective learning.
W4 — Predictive gains are limited and confounded. CIFAR improvements over the strongest hyperbolic baseline are not significant (Table 27). Some graph gains are significant, but Section D.3.2 additionally gives Klein/PV GyroBN and varies activation placement/type, preventing clean attribution to the geometry and contradicting the boundary-type-only interpretation of Table 25.
W5 — Efficiency claims exceed the evidence. Table 26 shows only approximately 1.1–1.2x speedup over BMLR-L, the fastest competing head, without establishing an end-to-end training benefit. Section E.2.1 gives the full Klein BFC minus BFC-L cost as , so the broadly stated lowest-FLOP claim is conditional, not general.
Additional technical errors and corrections:
- Equation (27), p. 20: Under the stated forward map, the inverse must be , not ; the printed pair is consistent only when .
- Equation (30), p. 20: For , , and , the formula returns , whose squared Klein norm at is rather than . It therefore fails to preserve the tangent norm.
- Running-mean update above Algorithm 1, p. 38: For and identical one-dimensional means , the printed update gives , not . It cannot be the stated binary Einstein midpoint.
- Section E.1, p. 33: The zero-score horosphere should satisfy , not , given the score definition .
Questions for authors
- Regarding W1: What is the principal methodological result beyond transporting existing constructions into Klein coordinates? Why is the resulting framework more than an incremental combination of known operators?
- Regarding W2: Which result directly supports a comparative stability or optimization advantage? Were Equations (27), (30), and the running-mean update implemented as printed, and would their correction affect the reported results?
- Regarding W3: Do the stability differences persist with matched intrinsic radii, a common geometric loss, and controlled projection policies? What evidence separates effective learning from merely returning finite values?
- Regarding W4: Do the graph gains persist with matched normalization, activation type/placement, and tuning budgets? How is Table 25 a boundary-type-only comparison given the differences documented in Section D.3.2?
- Regarding W5: What efficiency advantage remains under consistent full-cost accounting and matched, optimized baseline implementations? Is there a corresponding end-to-end training-time improvement?
14 comments
(Continuation of 1/10.)
4. Why the framework is more than a combination. The layers share these primitives rather than sitting side by side. The convolution reuses the FC closed form, the batch normalization takes its centre as the closed-form Einstein midpoint of Gulcehre et al. [e] in the same coordinates, and the classifier is again a Klein closed form, so the network never leaves Klein coordinates. Written as derived, the classifier also avoids the two FP32 steps at which the reference Poincaré and Lorentz heads fail (answer 2).
In short, the methodological result of KNN++ is the Klein closed forms of these constructions (Thms. 3.1–3.4), the convolution they allow as two standard convolutions (Prop. F.3 of the revision), and a network built only from Klein layers, which reaches the accuracy of HCNN with less memory than both its reference and its hand-optimized implementation. The revision states the contribution in these terms in §1.
2. Theoretical support and the result behind the comparison (W2, Q2)
The reviewer is right that our theorems give closed forms rather than comparative guarantees, and that Prop. E.2 is not a stability result. What we can add is a short floating-point account that predicts where each classifier fails, which matches the failures measured in the stability sweeps, and a precise statement of what Prop. E.2 covers. The revision adds the first to §4.1 and the second to App. E.
1. Predicting where each classifier fails. Reviewers and asked for the same mechanism. Every non-finite run fails in one step of the baseline head, and the limits of FP32 predict where (Table B).
Table B- Where each classifier fails in FP32 (stability sweeps of Tabs. 2–3)
| Head | Failing step | Predicted | Observed |
|---|---|---|---|
| Lorentz MLR [a] | subtracts two terms of size $ e ^{2a}/4$ | $ a \approx 8.3$– | $ a = 8.4$– |
| Poincaré MLR [d] | squares a term of size $ e ^{x}/2$ | $ x \approx 45$ | $ x = 40$– (5 of 6) |
| Klein MLR | evaluates $ e ^{\rho}$ once | $ \rho \approx 87$ ($ 72$ on the boundary) | no failure in 30 runs |
FP32 stores numbers up to about $ e ^{88.7}$ with 24 bits of precision. Every head contains a term that grows exponentially with the offset of its hyperplane from the origin, written $ a$, $ x$ and $ \rho$ in Table B and measured as geodesic distances in units of $ 1/\sqrt{-K}$. The Klein logit evaluates this term once, without squaring it or dividing by a difference of such terms. The Poincaré head squares it, which halves the usable range. The Lorentz head subtracts two squared terms whose difference is small and then divides by that difference, which rounds to zero once the terms exceed $ 2 ^{24}$ times its size. We located the failing steps by recomputing every non-finite run term by term.
The predicted and observed failure points agree, and the sixth Poincaré failure overflowed in the unsquared term at $ x = 99$, also past the FP32 limit. For Lorentz, rewriting the failing step or evaluating the head in FP64 removes the failures (0 of 3 runs at $ K = -10$), so its row of Table B is a statement about how one formula is evaluated. The Poincaré head still fails in FP64, because training drives its offset $ x$ from $ 0$ to $ 41$ within ten steps until the logits overflow. Swapping the feature radius and initialization between the two heads at $ K = -0.3$ (3 seeds, 400 steps) shows that the Poincaré head is the more sensitive one. With the Klein settings it trains, and the Klein head with the Poincaré settings stays finite with logits below $ 30$. These are measurements rather than guarantees, and §4.1 states them at that scope. When FP32 rounds a point onto the boundary sphere, $ \gamma _{x}$ is clamped at $ 3.2 \times 10 ^{7}$ and the Klein limit drops to $ \rho \approx 72$, which is where the one Klein failure outside these sweeps occurred, at $ K = -1$ and $ r = 10$ with $ \rho = 73$–.
References
[a] Ahmad Bdeir, Kristian Schwethelm, and Niels Landwehr. Fully Hyperbolic Convolutional Neural Networks for Computer Vision. ICLR, 2024.
[d] Ryohei Shimizu, Yusuke Mukuta, and Tatsuya Harada. Hyperbolic Neural Networks++. ICLR, 2021.
[e] Caglar Gulcehre, Misha Denil, Mateusz Malinowski, Ali Razavi, Razvan Pascanu, Karl Moritz Hermann, Peter Battaglia, Victor Bapst, David Raposo, Adam Santoro, and Nando de Freitas. Hyperbolic Attention Networks. ICLR, 2019.
(Continued in 3/10.)
(Continuation of 2/10.)
2. What Prop. E.2 covers. Prop. E.2 justifies using the Einstein midpoint as the batch centre in place of the Fréchet mean, which needs an iterative solver. It shows that the two coincide for two points with equal weights and agree to second order when all points lie close to the origin. The running-mean update weights its two points by $ 1-\eta$ and $ \eta$, so part (iii) does not cover it exactly. The weighted midpoint still lies on the geodesic between the two points, which is what a running average needs, but not at the same position as the weighted Fréchet mean. For the batch centre the applicability is tested rather than proved. Swapping the midpoint for the converged Fréchet mean inside the same layer gives $ 80.57$ against $ 80.77$ on Disease and $ 72.50$ against $ 73.08$ on PubMed (Tab. 7), and on Airport the Fréchet mean reaches only $ 85.67 \pm 7.99$ while the midpoint stays above $ 97$.
3. The result that directly supports the comparison. It is the localization of point 1, which places every baseline failure in one step of the head, together with the swap of point 1, which shows that the Klein head trains under the radius and initialization at which the Poincaré head fails. The Klein head had no non-finite value in any of its 30 runs. In accuracy we claim no advantage where all heads train, as our response to Reviewer explains.
4. The printed equations. None of the three equations is implemented as printed, and correcting them changes no reported number. The revision adds the localization to §4.1 and corrects the three equations in App. C and App. E.
Table I- The three printed equations and the code
| Printed equation | In the code |
|---|---|
| Eq. (27), Lorentz to Klein | used only in the centre swap of our response to Reviewer ZRE9, as the corrected inverse $ x _{s}/(\sqrt{-K},x _{t})$ |
| Eq. (30), parallel transport | not used, since no layer calls a parallel transport and Thm. 3.2 uses only that it is a linear isometry |
| Running-mean update above Alg. 1 | used in its correct form, the weighted two-point Einstein midpoint below |
The code updates the running mean as $ \mu _{r} \leftarrow \frac{(1-\eta)\gamma _{\mu _{r}}\mu _{r} + \eta\gamma _{\mu _{b}}\mu _{b}}{(1-\eta)\gamma _{\mu _{r}} + \eta\gamma _{\mu _{b}}}$.
In short, the theorems give closed forms and no comparative guarantee. The comparative evidence is the floating-point account of Table B and, against Poincaré, its sensitivity to the feature radius and initialization, and Prop. E.2 covers the batch centre only approximately, with its use in GyroBN supported by Tab. 7. None of the three printed equations is implemented as printed, so their correction changes no reported number.
3. Stability diagnostics under matched conditions and effective learning (W3, Q3)
The reviewer is right on both observations. Tab. 4 differentiates the squared norm of each model's coordinates, which is a different function on each chart, so its gradients do not measure instability. Tab. 4 was a supplementary probe, and the sweeps of Tabs. 2–3 and their numbers do not depend on it. The revision replaces it in §4.1 with the localization of the failures on the training loss. On Tiny-ImageNet, the near-chance Klein entries of Tab. 15 come from runs that used only part of the training set, and a five-seed rerun on the full set learns at every radius.
1. A common objective in place of Tab. 4. Following the reviewer's suggestion, the replacement works on the training loss, which all three models share. Table B in answer 2 locates every failure of the sweeps in one step of the baseline head, a cancellation in the Lorentz MLR and an overflow in the Poincaré MLR, and predicts from the limits of FP32 where each occurs. This also corrects the conclusion that the submission drew from Tab. 4. The Lorentz failures occur in the classifier rather than in the coordinate map, and the revision removes that sentence from §4.1. The Klein head had no non-finite value in any of its 30 runs in the sweeps of Tabs. 2–3.
2. Tiny-ImageNet at large clipping radii. Reviewer raised the same point on Tab. 15, and Tables C1 and C2 are the same tables as in our response to them. In checking Tab. 15 we found that its Klein entries at $ r = 5$, $ 10$ and $ 10 ^{9}$ come from runs that iterated over only 31, 47 and 116 of the 782 training batches per epoch, and reached $ 99.9%$ training accuracy on that subset. Rerun on the full training set with five seeds under the protocol of Tab. 15, Klein learns at every one of these radii. Without clipping one of the five seeds becomes non-finite, and projecting the features back into the ball removes it, with $ 63.74 \pm 0.23$ over five finite seeds.
(Continued in 4/10.)
We thank Reviewer fKfm for the valuable comments and for checking the appendix equations line by line. We respond to the concerns as follows.
Table, equation and section numbers refer to the submitted PDF, items that exist only in the revised PDF are marked "of the revision", and "answer N" refers to the numbered answers of this response. Answers 1–5 each address one weakness together with the question that refers to it, and answers 6–9 the four corrections.
1. Methodological contribution and the principal result beyond the transport (W1, Q1)
We thank the reviewer for this question. The signed-distance, Busemann and FC constructions of Sections 3.1–3.3 come from prior work, which the paper cites as templates. The methodological result of KNN++ is that these constructions become layers of the Klein model, where none of them had a closed form, and that all of them together form one network. Reviewers w1vu and ZRE9 raised the same question from two sides. Our response to Reviewer w1vu pairs each prior construction with its Klein closed form, and our response to Reviewer ZRE9 compares KNN++ with the Klein network of Mao et al. [f]. Below we state the result in four parts, the closed forms, the network they allow, the efficiency of that network and why it is more than a combination.
1. Closed forms in the Klein model. A construction in hyperbolic space is defined through distances, and it becomes a layer only once its closed form in some coordinate system is derived. For the Klein model the closed forms of the point-to-hyperplane distance, the MLR, the Busemann function and the FC layers did not exist, and Thms. 3.1–3.4 derive them. Klein geodesic hyperplanes are Euclidean hyperplanes cut by the ball, so the decision boundaries of the Klein MLR and FC layers are linear equations in the coordinates, $ \langle x, \hat z _{k}\rangle = \tanh(\sqrt{-K}, r _{k})/\sqrt{-K}$ (Thm. 3.2), and their direct implementation is numerically stable. The network of Mao et al. [f] builds each layer in the tangent space at the origin instead, so its decision boundaries are not geodesic hyperplanes.
2. A network built only from Klein layers. After submission we built Klein-ResNet, a ResNet-18 in which convolution, batch normalization, residual addition and classifier are all Klein layers, and trained it with the training setup of HCNN [a]. On CIFAR-100 it reaches the accuracy of HCNN with about a third of the GPU memory and a quarter of the time per epoch of the reference HCNN implementation. Against an HCNN that we optimized by hand the speed is the same and the memory saving shrinks to about a fifth (answer 5).
Table A- Fully hyperbolic ResNet-18 on CIFAR-100, 5 seeds
Note: Baseline accuracies as reported in their papers. Memory and time measured by us on one A100, using the released code of HCNN and Poincaré ResNet and our implementation of ILNN.
3. Why this network needs less memory and time. Most of the saving is in the convolution. A hyperbolic convolution applies an FC layer to the $ 3 \times 3$ neighbourhood of every pixel. The reference HCNN first copies each neighbourhood into a separate vector, so every input value is stored nine times. The Klein FC needs from a neighbourhood only its inner products with the weights and its squared norm, and both are ordinary convolutions of the input, one with the weights and one with an all-ones kernel (Prop. F.3 of the revision). Klein-ResNet therefore never builds the copies. Its convolution goes from slower than that of HCNN at 16 channels to 4.5 times faster at 512 channels, and its batch normalization is 1.7 times faster at 512 channels (Table G in answer 5).
References
(Continued in 2/10.)