acceptodds
Under review as a conference paper at ICLR 2027

A Homomorphism and Busemann Approach to Building Hyperbolic Transformers

11 views

Abstract

Hyperbolic space provides compact representations for hierarchical and tree-like data, yet a principled construction of attention that faithfully respects this geometry remains challenging. To remedy this gap, we decompose Euclidean attention into three modules: linear map, similarity, and aggregation. We redesign the linear map and similarity intrinsically in the Lorentz model by leveraging its rich geometric and algebraic structure, which leads to Homomorphism and Busemann transformers (HBformer). We use gyro-homomorphisms for query and value updates, generalizing Euclidean linear layers in a way that respects the Lorentz gyrovector structure. As the Busemann function naturally extends the Euclidean inner product, we derive curvature-aware attention scores via the Lorentz Busemann function. Since Busemann similarity requires keys in the Euclidean tangent space, we introduce a Lorentz-to-Euclidean fully connected layer (L2EFC). Experiments on graph benchmarks of different scales, large-scale node property prediction, and semi-supervised image and text classification demonstrate consistent improvements over strong Euclidean and hyperbolic baselines. The code will be open-sourced once accepted.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Hyperbolic neural networks, Lorentz neural networks, Transformers, Attention, Busemann functions, Gyrovector spaces

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_yNLGOpenReview ↗
Rating
3(+2)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors introduce HBFormer, a hyperbolic transformer-based model designed to operate across multiple domains, including graph learning, computer vision, and natural language processing. They reformulate the fundamental components of the Transformer, including linear transformations, similarity computation, and feature aggregation, such that HBFormer operates intrinsically within hyperbolic space under the Lorentz model. In particular, the authors employ the Busemann function to formulate a hyperbolic counterpart of the dot product, enabling attention scores to be computed directly in hyperbolic space. Nevertheless, the paper raises several significant concerns regarding the architectural design choices, and insufficient justification to explain the effectiveness of the proposed components.

Critical strengths
  • Applying Busemann function as similarity function for hyperbolic graph transformer, instead of geodesic distance, is both well motivated and novel, with a clear interpretation provided in the Introduction (lines 45-53)
Critical weaknesses
  • The authors do not provide any supplementary materials, such as source code or a link to a GitHub repository, making it difficult to assess the reproducibility of the proposed method.

  • Some existing hyperbolic Transformer methods, such as Hypformer, still employ the standard Euclidean dot product rather than geodesic distance or other intrinsic hyperbolic similarities. Therefore, the paper would benefit from a clearer explanation of why the intrinsic properties of the Busemann function are advantageous over the Euclidean dot product in the Lorentz model. Reporting higher accuracy alone is insufficient to explain the source of this advantage. Additional theoretical analysis or experiments are needed to identify which geometric properties are preserved by the Busemann function and under what conditions it is expected to outperform the Euclidean dot product.

  • In Eq. (5), the Lorentz inner-product term should be written as ⟨x,−K0ˉ+(0,v)⟩L\langle x, \sqrt{-K}\bar{\mathbf{0}} + (0,v)\rangle_{\mathcal{L}} to ensure dimensional and notational consistency. This issue may also affect to Eq. (13), which defines the similarity function in the Lorentz model and serves as a core component of the proposed Busemann attention mechanism.

  • Eq. (7) lacks a clear definition of SO(n)\mathrm{SO}(n). To the best of my knowledge, SO(n)\mathrm{SO}(n) denotes the special orthogonal group, i.e., the set of orthogonal matrices with determinant one. If this is indeed the intended definition, then the proposed isometry map ΦR\Phi_R appears closely related to the formulation introduced in H2H-GCN [1]. This may weaken the novelty claim regarding the proposed orthogonal-weight construction, and the authors should clarify the distinction between their approach and prior work.

  • The authors construct an intrinsic hyperbolic fully connected layer by decomposing the standard FC layer into multiple regression hyperplanes and then applying Lorentz multinomial logistic regression. However, compared with a conventional FC layer followed by logarithmic mapping, the Lorentz multinomial logistic regression formulation is considerably more complex and may introduce substantial computational overhead. Moreover, the empirical benefit appears marginal: Appendix G.2 reports an accuracy improvement of less than 0.30.3\\%. The authors should therefore provide a direct comparison of the computational efficiency of L2EFC against the simpler FC + logarithmic mapping alternative. If L2EFC is not more efficient, the paper should further justify whether such a complex replacement is needed, given its limited performance gain.

[1] Jindou Dai, Yuwei Wu, Zhi Gao, and Yunde Jia. A hyperbolic-to-hyperbolic graph convolutional network. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 154–163, 2021.

Questions for authors
  • How are the weight matrices in the Lorentz homomorphism transformation and point-to-hyperplane transformation optimized in practice? In particular, the weight matrices in the Lorentz homomorphism transformation must satisfy the special orthogonal group constraints, which likely require a specialized optimization procedure to preserve these constraints during training. Such a procedure may introduce additional computational overhead compared with simpler alternatives such as HypFormer. A more detailed discussion of the optimization strategy, together with an empirical comparison of computational cost, would therefore strengthen the paper.

  • Which version of HBFormer is reported in the main experimental results: the full-attention variant or the linear-attention variant? It would be helpful if the authors could report the performance of both versions across all main experiments for a more complete comparison.

Reviewer_pRsnOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HBformer, an attention layer for the Lorentz model of hyperbolic space. It argues that existing hyperbolic transformers either apply linear maps in the ambient space and project the result back, or score keys by geodesic distance, which ignores direction. HBformer redesigns the three parts of attention. Queries and values are transformed by spatial rotations, which the paper shows are isometries and gyrovector homomorphisms. The attention score is the Lorentz Busemann function, which recovers the dot product as the curvature goes to zero. Keys come from a new layer, L2EFC, and values are combined with the Lorentzian midpoint. A linear-time variant (BLA) is used on the large graphs. The method is evaluated on node classification, kNN graphs built from image and text data, and image classification with LViT, together with ablations and runtime comparisons.

Critical strengths

S1. The overall design is clear and principled. Existing Lorentz transformers apply linear maps outside the manifold and then project the result back. HBformer instead uses rotations for queries and values, which keep points on the manifold without any projection, and combines values with the Lorentzian midpoint. Each part of attention has a clear hyperbolic counterpart, and the block becomes close to standard attention in the flat limit (Sec. 3.5), which makes the method easy to understand.

S2. The Busemann attention score is well motivated. Scores based on geodesic distance ignore direction, so two keys at the same distance from the query get the same score (Fig. 1). Scores computed in the tangent space are only accurate near the origin. The Busemann score avoids both problems. It measures how far the query lies in the direction of the key, and it has a closed form on the Lorentz model. In its exact form (Eq. 13), it reduces to the usual dot product as the curvature goes to zero.

S3. The paper is well written and easy to follow.

Critical weaknesses

W1. The contribution appears incremental relative to work that the paper cites but does not discuss in detail. GyroAtt (Wang et al., NeurIPS 2025) already builds attention on matrix manifolds with the same split into transformation, similarity and aggregation, using gyro homomorphisms for the transformation. Its Theorem 3.2 states that any isometry that fixes the identity is a gyro homomorphism, which already implies Theorem 1, since diag(1,R)\mathrm{diag}(1,R) is such an isometry. The Busemann score has the same form as the Busemann MLR logit of HBNN (Chen et al., CVPR 2026), with the key in place of the class direction, and HBNN already proves its flat limit. L2EFC keeps only the first step of the point-to-hyperplane layer of ILNN (Shi et al., ICLR 2026), and Theorem 2 restates its definition. The main new step is therefore applying these existing tools to Lorentz attention, in particular using the Busemann logit as the query-key score. Discussing these overlaps would make the contribution clearer.

W2. In Sec. 3.2, the key norm ∥kj∥\|k_j\| in the score is replaced by a learnable scalar τ\tau, but the paper does not clearly explain why. It describes τ\tau as a learnable temperature. However, τ\tau does more than rescale the score: it removes the key length, so the evaluated score no longer reduces to the dot product as claimed in Sec. 3.5. The paper says the change improves performance but gives no experimental results comparing τ\tau with ∥kj∥\|k_j\|.

W3. It is unclear how the curvature is chosen for results in Table 2 and 3. The paper does not report it or say whether it was selected on validation data. On Cora, PubMed, Mini-ImageNet and 20News-Groups, the main results equal the best of the six curvatures in Table 5. The gains over Hypformer may therefore be optimistic.

W4. Some important baselines are missing. For the large graphs, tuned classic GNNs (Luo et al., NeurIPS 2024) are not included, although they are known to be strong on these benchmarks. Hypformer+PLFC (Shi et al., ICLR 2026), the closest Lorentz model, is cited but not compared. Including these would make the comparison more strong.

Questions for authors

Q1. App. H states that all baselines were trained under the same splits and tuned within the same ranges as HBformer. However, the baseline entries in Tables 2 and 3 are identical to those published in Hypformer. Could the authors clarify which baselines were re-run and which were taken from prior work?

And for others check weakness

Reviewer_9QjzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors proposed a hyperbolic Transformer model based on the homomorphism and Busemann approach. This paper investigates the foundational formulation of self-attention within hyperbolic space, specifically on the Lorentz model. The authors identify two primary limitations in existing hyperbolic Transformers: (1) query/key/value transformations typically rely on extrinsic Euclidean linear layers in ambient Minkowski space followed by manifold reprojections, failing to preserve intrinsic gyrovector structures; and (2) similarity scoring commonly uses geodesic distance, which conflates radial proximity with directional alignment, or relies on tangent-space logarithmic projections that suffer from metric distortion away from the base point.

The paper splits attention into three parts, i.e., linear map, similarity, aggregation, and rebuilds the first two. Aggregation is the standard Lorentz midpoint. A linear-attention variant, BLA, is used on the three large graphs. With the Hypformer backbone and training held fixed, the model is best on all seven graph benchmarks and on Mini-ImageNet.

Critical strengths
  1. The paper provides a mathematically elegant formulation of hyperbolic attention. Rather than treating the Lorentz manifold as a constrained sub-manifold‬, the operations respect the gyrovector space axioms.

  2. The critique of distance-based hyperbolic attention is clear and well-illustrated (Figure 1): geodesic distance measures spatial closeness but collapses directional coherence. Framing attention similarity via the Busemann potential restores directional discrimination while preserving intrinsic manifold evaluation.

  3. Comprehensive Empirical Benchmark: The evaluation spans graph node classification, large-scale OGB benchmarks, semi-supervised text/image classification, and conventional ViT drop-in evaluations, outperforming competing hyperbolic architectures such as Hypformer and HGCN across all tasks.

Critical weaknesses
  1. BLA carries the large-scale results but is never derived or checked. Seems there is no approximation argument and no BLA-vs-full-attention comparison anywhere.

  2. The comparison set stops in 2024. App. H.3's own baseline list: newest full model is Hypformer (2024), newest Euclidean Transformer SGFormer (2023); the 2025 Lorentz work is only an ablation row, never run as a model.

Questions for authors
  1. Is there any derivation, or even a numerical check, that BLA (Eq. 19) approximates the full Busemann attention? BLA vs full attention on PubMed would do. And what keeps the denominator s(q_i, K_sᵀ1_N) positive?

  2. Which split was curvature (and the other hyperparameters) selected on? Validation-selected, or fixed-k results close to Table 2, closes this. Test-selected lowers the rating to 3.

  3. Why does rotation-only beat KAK Lorentz in Table 7? A parameter-matched run or training curves would help. A geometric explanation raises Contribution; an optimization explanation means the framing should change.

  4. §5 says the pipeline "remains entirely on the Lorentz model," but §3.2–3.3 build the keys as tangent directions at the origin from classification logits, which means this is not fully on Lorentz model?

Comments on the paper

2 comments

Hi Reviewers, We are now near the end of the rebuttal period. If you haven't done so yet, please review the authors' responses and post a reply addressing whether their clarifications or new results resolve your concerns. Even if your score or recommendation remains unchanged, a brief note confirming what did or didn't shift your assessment—and why—is essential for the discussion and final decision. Thank you for your time and engagement. Best regards, Area Chair

AuthorsOpenReview ↗

Summary of Revisions in the Updated Manuscript

We thank all reviewers for their constructive comments, which helped us improve the clarity and presentation of our work. In the revised manuscript, we added several new experiments and analyses (highlighted in blue\color{blue}{\text{blue}}) that directly address the raised concerns. Below we summarize the main revisions.

  • Recent baselines and both attention variants in Table 2 (Sec. 4.1). We add tuned GCN (Luo et al., 2024), QGT (Le & Ta, 2025), and Hypformer+PLFC (Shi et al., 2026), and report both full Busemann attention and BLA on all seven benchmarks. HBformer remains the best on every benchmark, and BLA stays within 0.15 points of full attention on the four small and medium graphs.
  • Temperature versus key norm in Sec. 4.2 (Table 9). We compare the learnable temperature τ\tau with the raw key norm ∥kj∥\|k_j\| as the score multiplier. τ\tau gives higher accuracy on Cora (86.00 vs. 84.96) and PubMed (83.30 vs. 82.12). The flat-limit statement in Sec. 3.5 is corrected accordingly.
  • Relation to GyroAtt, HBNN, and ILNN in App. B.3. We discuss the three works HBformer builds on and state what is new: realizing the framework on the Lorentz model with different similarity, using the Busemann function as the query-key similarity, and L2EFC as a Lorentz-to-Euclidean map that provides the key directions.
  • Busemann score versus Euclidean dot product in App. E.7. Proposition 5 shows that the Busemann score grows at most linearly with the hyperbolic distance, whereas the dot product can grow exponentially. A numerical check (Table 11) confirms this: at distance 5 from the origin, the softmax saturates for 96% of queries with the dot product and for none with the Busemann score.
  • Efficiency of L2EFC in App. G.3 (Table 14). We compare L2EFC with a Lorentz FC layer followed by the logarithmic map. L2EFC is 37.6% to 62.6% faster and uses 11.1% to 15.1% less extra memory at all tested scales.
  • Clarifications in Secs. 2, 3.1, 3.2 and App. H. We correct the Lorentz inner-product term in Eqs. (5) and (13), define SO(n)\mathrm{SO}(n) at Eq. (7) with a citation to H2H-GCN (Dai et al., 2021), and state that the baseline results are taken from Hypformer (Yang et al., 2024) under the same split protocol.

We hope these revisions address the reviewers' concerns and further clarify the advantages of HBformer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.