HypTBench: Rethinking Hyperbolic Geometry for Transformers

Hyperbolic geometry has been widely incorporated into Transformers to model hierarchical and relational structures. However, existing Hyperbolic Transformer methods often modify the geometric representation, Transformer operators, and training configurations simultaneously, making it difficult to determine whether observed performance differences arise from the hyperbolic models themselves or from the geometric operators. To address this problem, we introduce HypTBench, a benchmark for Hyperbolic Transformers. Across 556 experimental runs, our evaluation shows that existing Lorentz manifold methods generally achieve stronger performance across tasks while offering higher computational efficiency, whereas current Poincaré manifold methods tend to underperform. Further experiments demonstrate that compatibility among individual modules can substantially affect overall model performance. These findings suggest that the effectiveness of Hyperbolic Transformers is determined not merely by the choice of hyperbolic representation, but by how hyperbolic geometry and fundamental Transformer operators are integrated into a stable, compatible, and computationally efficient system.