A Homomorphism and Busemann Approach to Building Hyperbolic Transformers

Hyperbolic space provides compact representations for hierarchical and tree-like data, yet a principled construction of attention that faithfully respects this geometry remains challenging. To remedy this gap, we decompose Euclidean attention into three modules: linear map, similarity, and aggregation. We redesign the linear map and similarity intrinsically in the Lorentz model by leveraging its rich geometric and algebraic structure, which leads to Homomorphism and Busemann transformers (HBformer). We use gyro-homomorphisms for query and value updates, generalizing Euclidean linear layers in a way that respects the Lorentz gyrovector structure. As the Busemann function naturally extends the Euclidean inner product, we derive curvature-aware attention scores via the Lorentz Busemann function. Since Busemann similarity requires keys in the Euclidean tangent space, we introduce a Lorentz-to-Euclidean fully connected layer (L2EFC). Experiments on graph benchmarks of different scales, large-scale node property prediction, and semi-supervised image and text classification demonstrate consistent improvements over strong Euclidean and hyperbolic baselines. The code will be open-sourced once accepted.