acceptodds
Under review as a conference paper at ICLR 2027

HypTBench: Rethinking Hyperbolic Geometry for Transformers

12 views

Abstract

Hyperbolic geometry has been widely incorporated into Transformers to model hierarchical and relational structures. However, existing Hyperbolic Transformer methods often modify the geometric representation, Transformer operators, and training configurations simultaneously, making it difficult to determine whether observed performance differences arise from the hyperbolic models themselves or from the geometric operators. To address this problem, we introduce HypTBench, a benchmark for Hyperbolic Transformers. Across 556 experimental runs, our evaluation shows that existing Lorentz manifold methods generally achieve stronger performance across tasks while offering higher computational efficiency, whereas current Poincaré manifold methods tend to underperform. Further experiments demonstrate that compatibility among individual modules can substantially affect overall model performance. These findings suggest that the effectiveness of Hyperbolic Transformers is determined not merely by the choice of hyperbolic representation, but by how hyperbolic geometry and fundamental Transformer operators are integrated into a stable, compatible, and computationally efficient system.

Primary area:
datasets and benchmarks
Keywords:
Hyperbolic Transformers, Hyperbolic Geometry, Geometric Deep Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_w8QkOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents HypTBench, a benchmark that decomposes hyperbolic Transformers into interchangeable components and evaluates them under a shared training protocol. It compares a Euclidean Transformer with ten hyperbolic methods from the Euclidean-backbone (HiT, ConeHeads, Hyp-ViT), Poincaré-manifold (HVT, GGBall, HNN++), and Lorentz-manifold (HyboNet, HyTE-FH block, Hypformer, HELM) families on 20 datasets spanning NLP, computer vision, pretraining reasoning, and genomic sequence modeling, with three seeds and 556 runs in total. The authors report that Lorentz-family methods, led by Hypformer, achieve the best results and are more efficient than Poincaré-family methods. They further find that re-expressing hidden states in Poincaré coordinates does not consistently reduce accuracy and that transplanting Hypformer modules into other Lorentz models yields base-model- and task-dependent, non-additive effects, and conclude that performance depends on how geometric operators are integrated rather than only on the choice of hyperbolic model.

Critical strengths
  1. A benchmark for hyperbolic models is genuinely valuable for research on hyperbolic learning.
  2. The experimental work is extensive, with 556 experimental runs.
Critical weaknesses
  1. Section 5.1 does not demonstrate that methods based on the Lorentz model generally outperform those based on the Poincaré ball, since the compared methods differ in their designs, architectures, and parameter counts.

  2. The Lorentz model is numerically more stable than the Poincaré ball, which is why research has already shifted from Poincaré-based HNN [1] and HNN++ [2] to Lorentz-based HCNN [3] and Hypformer [4]. This has already been pointed out in papers such as HCNN [3], so the advantage of Lorentz-based methods over Poincaré-based ones cannot be counted as a contribution of this paper.

  3. Section 5.2.1 only changes the representation of the intermediate Transformer hidden states to Poincaré coordinates. A fair comparison should switch both the hidden states and the operators to the Poincaré model. If only the hidden states are changed while the operators are kept, the hyperbolic computation is still essentially carried out in the Lorentz model.

References

[1] Hyperbolic Neural Networks

[2] Hyperbolic Neural Networks++

[3] Fully Hyperbolic Convolutional Neural Networks for Computer Vision

[4] Hypformer: Exploring Efficient Transformer Fully in Hyperbolic Space

Questions for authors
  1. I am curious about the Pretraining Track. Could the authors explain (i) how the near-chance Pretraining accuracies in Table 10 can distinguish methods, (ii) why each model in Figure 13 mostly outputs the same option letter, suggesting position bias rather than reasoning, and (iii) why HVT and ConeHeads have identical means and standard deviations in four of the five Table 10 columns?

  2. Could the authors provide comparisons in which some methods are switched to the operators of the other hyperbolic model? Such comparisons would be fairer and more intuitive.

  3. Why are these datasets suitable for evaluating hyperbolic methods? Although they have been adopted in previous hyperbolic papers, a benchmark should describe in its dataset section the properties that make them suitable for evaluating hyperbolic methods.

6 comments
AuthorsOpenReview ↗

Response to the critical weaknesses and the questions for authors 2

We thank the reviewer for the constructive and professional comments, which are very helpful for improving the quality of our paper. Our point-by-point responses follow:

W1. Differences in Parameter Counts

We thank the reviewer for raising this important issue. We agree that hyperbolic Transformer methods differ in their internal operator designs, overall architectures, and parameter counts, and that relying solely on the overall performance comparison in Section 5.1 is therefore insufficiently rigorous.

First, Section 5.1 primarily aims to compare the overall performance of existing hyperbolic Transformer methods under unified experimental conditions; it does not directly demonstrate that Lorentz geometry itself is superior to Poincaré geometry. Second, to address architectural and operator differences, we designed dedicated ablations in Sections 5.2.1 and 5.2.2 to further analyze their effects. For example, in Table 4, we keep the original Transformer operator designs of HyboNet and Hypformer unchanged, altering only the intermediate hidden-state representation to Poincaré coordinates and converting coordinates when invoking Lorentz operators. HyboNet’s accuracy improves by 2.25 and 3.67 percentage points on CIFAR-100 and Food-101, respectively, while Hypformer’s accuracy improves by 1.63 percentage points on CIFAR-100.

In Table 6, we further decompose the Transformer into core modules, including Projection, Attention Score, Value Aggregation, Residual, Normalization, and FFN, and perform controlled replacements across Lorentz Transformer architectures. For example, replacing HELM’s Attention Score with the corresponding Hypformer module improves accuracy by 8.79 and 10.34 percentage points on CIFAR-100 and Food-101, respectively. Moreover, replacing HyboNet’s Projection or FFN individually may reduce performance, whereas replacing them jointly can yield improvements.

Finally, these experiments constitute one of HypTBench’s core contributions. Beyond comparing the overall performance of existing Lorentz and Poincaré Transformers, we investigate the sources of performance differences through coordinate-representation and module-level ablations. Our conclusion is that the evaluated Lorentz-family Transformer implementations perform better overall, and that this advantage is closely associated with their specific operator designs and module compatibility.

W2. Definition of the Contribution

We thank the reviewer for raising this point. Previous work has indeed reported that the Lorentz model is numerically more stable than the Poincaré ball model, and recent research has increasingly shifted toward the Lorentz model. However, those findings were obtained using different training configurations and data. We are the first to reach this conclusion under a unified protocol and training configuration. Our contribution primarily lies in providing the hyperbolic Transformer community with its first unified benchmark.

W3 & Q2. Fair Ablation Experiments

This comment is valuable for improving the quality of our paper. We have therefore added the corresponding ablations; please see Table 4 of the revised manuscript. For ease of review, we reproduce them here:

We clarify the three variants as follows. The original Lorentz variant uses both Lorentz hidden representations and the original Lorentz operators. The Wrapper variant stores intermediate hidden states in Poincaré coordinates but converts them back to Lorentz coordinates when executing the original operators. The Direct variant rewrites the original operators as direct formulas in ball coordinates, without reconstructing full hidden states on the Lorentz manifold during the forward pass.

Table A. Geometric ablations

MethodVariantCIFAR-100 F1CIFAR-100 AccFood F1Food Acc
HyboNetWrapper51.16±0.1551.45±0.0453.85±0.5154.20±0.68
HypformerWrapper51.44±0.3151.63±0.3052.18±0.0552.37±0.00
HyboNetDirect46.02±0.1646.16±0.1346.52±0.1646.67±0.12
HypformerDirect46.98±0.2347.12±0.2548.17±0.3748.48±0.41
AuthorsOpenReview ↗

Response to the questions for authors 1

Q1. Questions Regarding the Pretraining Track

We appreciate the reviewer’s valuable questions. Our point-by-point responses follow:

(i) Near-chance results

First, the primary purpose of the Pretraining Track is not to establish that 100M parameter models possess strong general reasoning capabilities. This experimental design follows HELM (He et al., NeurIPS 2025) exactly. HELM likewise evaluates approximately 100M parameter models on the same five reasoning benchmarks. As shown in Table 1 of the original HELM paper, its results are also close to chance: DeepSeekV3 achieves an average accuracy of 22.2%, while HELM-MiCE achieves 24.1%, an improvement of only 1.9 percentage points. Nevertheless, the original paper uses relative performance differences between architectures to analyze the effectiveness of hyperbolic designs. Despite the low absolute accuracy, HELM treats this relative advantage as important evidence for hyperbolic architectures, explicitly emphasizing consistent improvements over Euclidean baselines in Section 5.1 and citing performance gains of up to 4% as evidence of enhanced reasoning in its abstract.

In our Table 12, HyTE-FH outperforms the Euclidean baseline on all five tasks, increasing average accuracy from 23.28% to 25.03%, a gain of 1.75 percentage points. All results report means and standard deviations over three random seeds, which is sufficient to demonstrate the stability of HyTE-FH’s relative improvements.

References:

[1] He, N., Anand, R., Madhu, H., Maatouk, A., Krishnaswamy, S., Tassiulas, L., Yang, M., & Ying, R. (2025). HELM: Hyperbolic Large Language Models via Mixture-of-Curvature Experts. Advances in Neural Information Processing Systems, 38, 142604-142635.

(ii) Position bias

This is a valuable question. We acknowledge an oversight in the presented case and thank the reviewer for pointing out potential option-position bias in the Pretraining Track. We conducted additional option-order perturbation experiments to examine the models’ prediction behavior.

The additional experiments confirm the reviewer’s concern: different models exhibit clear option-position preferences. After further changing the option order, we observed that models continued to show high consistency in their selected positions. However, this neither indicates a failure of our benchmark nor is unique to it. Prior work, such as Zheng et al., has already identified position bias in large language models including the LLaMA family. Those researchers attribute this phenomenon to prior probability preferences for different option-label tokens. For our approximately 100M parameter models, the limited pretraining scale may further amplify this bias, although its specific cause remains uncertain. Meanwhile, reordering did not substantially change HypTBench’s overall Accuracy or the relative rankings of the methods. As noted in (i), our overall performance also differs little from the results reported in HELM when we follow its framework exactly. In the revision, we will explicitly report the observed position bias and revise the discussion to position the Pretraining Track as an exploratory evaluation under limited pretraining conditions. Future versions will further expand the training scale to investigate this phenomenon.

We wish to emphasize that HypTBench’s core contribution is to provide the hyperbolic community with its first unified benchmark and systematically compare existing hyperbolic Transformer architectures and their computational operators, revealing performance differences and potential failure modes across tasks and training conditions.

(iii) Identical numerical results

We thank the reviewer for pointing this out. This was a mistake in recording experimental data. We have included the correct results in the revised manuscript and highlighted them, and updated all figures that use Avg. values. For ease of review, we provide the correct data in this response:

Table A. Pretraining Track

DatasetConeHeads (Acc)HVT (Acc)
CommonsenseQA19.62±0.0619.59±0.02
HellaSwag25.91±0.0625.19±0.12
OpenBookQA25.29±0.1025.31±0.10
MMLU23.05±0.0922.99±0.07
ARC-Challenge23.68±0.0423.30±0.52
Avg.23.51±0.0623.28±0.15
AuthorsOpenReview ↗

Response to the questions for authors 3

Q3. Dataset Suitability

We thank the reviewer for pointing out that the rationale for dataset selection requires further clarification. We agree that prior use of a dataset in a hyperbolic-model paper does not, by itself, establish that its structure is suitable for hyperbolic modeling. We have therefore added feature-space relative hyperbolicity diagnostics covering all datasets.

We compute the Euclidean distance matrix between frozen features, use the first point in each sample as the base point to obtain the fixed-base-point deviation δp\delta_p of the Gromov-product four-point condition, and calculate:

δrel=2δpdiam⁡(X),\delta_{\mathrm{rel}} = \frac{2\delta_p}{\operatorname{diam}(X)},

where diam⁡(X)\operatorname{diam}(X) is the maximum pairwise distance within the sampled feature set.

For image tasks, we use frozen features from an ImageNet-pretrained ResNet34. For NLP tasks and downstream multiple-choice tasks in the Pretraining Track, we use aggregated features from the final layer of BERT-base-uncased. For DNA sequences, we use fixed 5-mer frequency features. The feature pools reuse the union of the evaluation subsamples obtained with the three previous sampling seeds.

For each dataset, we fix the random seed to 42 and perform 10 sampling rounds with replacement, typically drawing 1,000 instances per round. The frozen feature pools for OpenBookQA and Enhancers contain only 498 and 400 instances, respectively, so we use 498 and 400 draws per round for these datasets. The table reports the mean and population standard deviation of the 10 relative δ estimates.

Table A. Feature-Space Relative Hyperbolicity Diagnostics for All Datasets

TrackDatasetFeature Pool NDraws per Round nRelative δ (Mean ± Std)
NLPAG News260610000.2589±0.0175
NLPMNLI269510000.2115±0.0121
NLPFew-NERD291310000.2343±0.0087
NLPSemEval-2010201110000.2598±0.0107
NLPSQuAD 2.0277510000.2545±0.0183
CVCIFAR-10272210000.2642±0.0118
CVCIFAR-100272410000.2595±0.0253
CVFood-101288210000.2513±0.0134
CVEuroSAT202910000.2711±0.0115
PretrainCommonsenseQA121510000.2711±0.0162
PretrainHellaSwag271010000.3159±0.0165
PretrainOpenBookQA4984980.2608±0.0102
PretrainMMLU279510000.2591±0.0134
PretrainARC-Challenge117010000.2911±0.0122
BioLTR Copia162110000.1242±0.0128
BioCMC-EnSpm164010000.2651±0.0409
BioProcessed Pseudogene160010000.1744±0.0179
BioH3K4me3226010000.1736±0.0269
BioPromoter All248410000.1632±0.0493
BioEnhancers4004000.1204±0.0267

We hope that our responses have addressed your concerns.

Reviewer_aYczOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes HypTBench. This benchmark puts ten hyperbolic Transformer variants (grouped as Euclidean-backbone, Poincare-manifold, Lorentz-manifold) plus a Euclidean baseline into one training pipeline and evaluates them on four tracks: five NLP datasets, four image classification datasets, five multiple-choice reasoning sets after a small pretraining run, and six genomic sequence tasks. Three seeds per configuration, 24h budget, out-of-time runs counted as zero in the macro average. Main claims: i) Lorentz methods (Hypformer in particular) come out on top, Poincare methods are weaker and slower. ii) swapping the coordinate representation of HyboNet/Hypformer to Poincare while keeping Lorentz operators does not hurt, so the representation itself is not the cause of the gap. iii) a module transplant ablation on CIFAR-100/Food-101 shows that single operator swaps have inconsistent effects and that Hypformer's advantage comes from the combination of its components.

Critical strengths
  1. The question is the right one! Usually hyperbolic Transformer papers change geometry, operators and training recipe at the same time, and a controlled comparison under one protocol is overdue. The decomposition into the operator slots projection, attention score, value aggregation, residual, norm, FFN is a sensible way to think about this. The Poincare-coordinate ablation is a nice small experiment that makes a clean point
  2. Three seeds, four fairly different domains, timing and memory was reported, OOT handled explicitly rather than silently dropped, and the whole paper is critical and honest to the results.
Critical weaknesses
  1. The Pretraining track does not measure anything. All numbers in Tab. 10 are at chance level (CommonsenseQA is 5-way, so 20%, everything else 4-way, so 25%; the reported values are 19.6-21.3 and 22.8-26.7). A 100M model pretrained on WikiText-103 (this is only mentioned in the caption of Fig. 19) will not do zero-shot MMLU or ARC-C. Differences of 1.5 points between methods at chance are noise, yet "HyTE-FH leads the Pretraining track" is a headline result.
  2. The benchmark cannot support its own motivation, and several things needed to judge fairness are missing. The introduction motivates hyperbolic geometry with hierarchies (coarse/fine entity types, CIFAR-100 superclasses, phylogeny), but nothing in the evaluation looks at hierarchical structure: no superclass accuracy, no coarse-vs-fine error breakdown for Few-NERD (the 88-89% accuracy there is token accuracy dominated by O-tags, F1 is 51-55), no distortion or tree-likeness measure of the learned representations. So the paper cannot say whether any of these models uses the geometry for what it was introduced for --> it only says which implementation trains better under one fixed recipe.
  3. Tab. 4 and Tab. 6 are single-seed as far as I can tell (no std), and several of the "gains" discussed in the text are just the result of a collapse to one class in all seeds.
Questions for authors
  1. What is the pretraining corpus, how many tokens, and how are the downstream multiple-choice tasks scored? Please give the chance level for each dataset next to the results. If all methods are at chance, how do you justify keeping the track in the macro average?
  2. Are the HVT and ConeHeads rows in Tab. 10 really from separate runs? Four of five cells are identical to two decimals including std.
  3. Model configuration: width, depth, heads, patch size, embedding dimension of the hyperbolic space, curvature (fixed or learned), optimizer, data augmentation for CV. Were any hyperparameters tuned per method, or was one setting used for all?
  4. Will code, configs and raw logs be released? The paper calls the benchmark "reproducible" but there is nothing to reproduce from.
5 comments
AuthorsOpenReview ↗

Response to the critical weaknesses 1 and the questions for authors 1

We thank the reviewer for the constructive and professional comments, which are very helpful for improving the quality of our paper. Our point-by-point responses follow:

W1 & Q1. Near-Chance Results in the Pretraining Track

We thank the reviewer for raising this issue. To ensure fairness and facilitate direct interpretation of current model performance, we adopted exactly the same architecture as HELM, replacing only its Transformer modules with different hyperbolic Transformer modules for analysis. HELM likewise uses models with approximately 100M parameters and evaluates them on the same five reasoning benchmarks. As shown in Table 1 of the original HELM paper, its results are also close to chance: DeepSeekV3 achieves an average accuracy of 22.2%, while HELM-MiCE achieves 24.1%, an improvement of only 1.9 percentage points. Despite the low absolute accuracy, HELM treats this relative advantage as important evidence for the effectiveness of hyperbolic architectures, explicitly emphasizing consistent improvements over Euclidean baselines in Section 5.1 and citing performance gains of up to 4% as evidence of enhanced reasoning in its abstract. For pretraining, we use WikiText-103 with the GPT-2 tokenizer. After tokenization, the actual training corpus contains 50M tokens and the validation corpus contains 244,949 tokens. Models are trained from random initialization with a context length of 512 tokens. Completed training runs perform 2400 parameter updates.

Candidates are scored by their conditional log-likelihood given the prompt. For a prompt xx and candidate answer yjy_j, the score is defined as:

S(yj∣x)=1∣yj∣∑t=1∣yj∣log⁡pθ(yj,t∣x,yj,<t),S(y_j \mid x) = \frac{1}{|y_j|} \sum_{t=1}^{|y_j|} \log p_\theta(y_{j,t} \mid x, y_{j,<t}),

The final prediction is:

j^=arg⁡max⁡jS(yj∣x).\hat{j} = \arg\max_j S(y_j \mid x).

Furthermore, absolute accuracy close to chance does not necessarily imply that relative differences between architectures are entirely due to random noise. In Table 12, HyTE-FH outperforms the Euclidean baseline on all five reasoning tasks, increasing average accuracy from 23.28% to 25.03%, a gain of 1.75 percentage points. Our results also come from three different random seeds rather than a single run. Although these observations do not establish strong reasoning ability, they provide descriptive evidence of relative performance differences between architectures under unified training and evaluation conditions.

Finally, the central goal of HypTBench is to systematically compare different hyperbolic Transformer architectures within a unified experimental framework, rather than to achieve state-of-the-art language reasoning performance. The current Pretraining results are therefore sufficient for this purpose.

References:

[1] He, N., Anand, R., Madhu, H., Maatouk, A., Krishnaswamy, S., Tassiulas, L., Yang, M., & Ying, R. (2025). HELM: Hyperbolic Large Language Models via Mixture-of-Curvature Experts. Advances in Neural Information Processing Systems, 38, 142604-142635.

AuthorsOpenReview ↗

Response to the critical weaknesses 2

W2. Hierarchical Evaluation in the Benchmark

We thank the reviewer for pointing out the insufficient analysis of hierarchical structure in the current evaluation. We agree that conventional classification accuracy and F1-score alone cannot fully characterize how well models capture hierarchical label relationships. We have therefore added further analyses on CIFAR-100 and Few-NERD.

For CIFAR-100, we add two superclass accuracy measures: Superclass Acc (Mapped), obtained by mapping fine-grained predictions to their corresponding superclasses, and Superclass Acc (Aggregated), obtained by aggregating fine-grained class scores. We further divide classification errors into Within-superclass Error and Cross-superclass Error to examine whether models preserve higher-level category information. The results are as follows:

Table A. CIFAR-100 Hierarchical Evaluation

MethodFine F1Fine AccSuperclass Acc (Mapped)Superclass Acc (Aggregated)Within-superclass ErrorCross-superclass Error
Euclidean49.6049.2461.1161.6511.8738.89
HiT48.7148.7961.7661.8612.9738.24
ConeHeads49.1849.4362.0962.3812.6637.91
Hyp-ViT48.9149.1661.5061.7312.3438.50
HVT33.5334.7147.7348.6613.0252.27
GGBall43.6443.5157.2856.9413.7742.72
HNN++33.3234.1847.3647.1013.1852.64
HyboNet49.2249.0261.8362.1512.8138.17
HyTE-FH42.8043.1456.5656.9613.4243.44
Hypformer49.9150.0762.4864.5412.4137.52
HELM41.7441.9154.8455.2712.9345.16

The results show that Hypformer achieves the highest fine-grained F1, fine-grained accuracy, and both superclass accuracy measures. Compared with the Euclidean baseline, its fine-grained accuracy increases from 49.24% to 50.07%, a gain of 0.83 percentage points; mapped superclass accuracy increases from 61.11% to 62.48%, a gain of 1.37 percentage points; and aggregated superclass accuracy increases from 61.65% to 64.54%, a gain of 2.89 percentage points. Its cross-superclass error rate also decreases from the Euclidean baseline’s 38.89% to 37.52%, a reduction of 1.37 percentage points.

For Few-NERD, we additionally report fine-grained entity F1, coarse-grained entity F1, and entity boundary F1 to evaluate the ability to identify specific entity types, higher-level entity categories, and entity boundaries, respectively. The results are as follows:

Table B. Few-NERD Hierarchical Evaluation

MethodFine-grained Entity F1Coarse-grained Entity F1Boundary F1
Euclidean51.8361.8170.84
HiT51.9562.3171.52
ConeHeads52.5462.7672.00
Hyp-ViT52.0462.2971.56
HVT42.0350.2058.43
GGBall52.5862.2470.52
HNN++30.2435.7243.60
HyboNet53.2863.1671.60
HyTE-FH54.3963.7572.34
Hypformer55.1764.9973.15
HELM51.6661.4770.11

The results show that Hypformer performs best across all three evaluation dimensions. Compared with the Euclidean baseline, its fine-grained entity F1 increases from 51.83% to 55.17%, a gain of 3.34 percentage points; coarse-grained entity F1 increases from 61.81% to 64.99%, a gain of 3.18 percentage points; and entity boundary F1 increases from 70.84% to 73.15%, a gain of 2.31 percentage points. HyTE-FH also outperforms the Euclidean baseline, achieving 54.39% fine-grained entity F1 and 63.75% coarse-grained entity F1.

AuthorsOpenReview ↗

Response to the critical weaknesses 3

W3. Multi-Seed Ablations and Potentially Collapsed Results

The reviewer is correct that multi-seed ablations can effectively demonstrate the stability of our performance improvements. We have added further multi-seed ablations to the revision; please see Tables 4 and 6 of the revised manuscript. For ease of review, we reproduce the results below:

Table A. Multi-seed geometric ablations

MethodVariantCIFAR-100 F1CIFAR-100 AccFood F1Food Acc
HyboNetWrapper51.16±0.1551.45±0.0453.85±0.5154.20±0.68
HypformerWrapper51.44±0.3151.63±0.3052.18±0.0552.37±0.00
HyboNetDirect46.02±0.1646.16±0.1346.52±0.1646.67±0.12
HypformerDirect46.98±0.2347.12±0.2548.17±0.3748.48±0.41

Table B. Multi-seed module ablations

Method / VariantCIFAR-100 F1CIFAR-100 AccFood F1Food AccGain Sum
HyboNet original49.42±0.2849.21±0.2650.52±0.1250.54±0.11--
HyTE-FH original43.14±0.4743.52±0.5444.24±0.4744.61±0.40--
HELM original42.01±0.3742.16±0.3536.75±0.2136.74±0.21--
Hypformer original49.95±0.0550.01±0.0954.40±0.1654.71±0.17--
HyboNet + Proj.51.51±0.1651.59±0.0944.87±0.2045.14±0.184.47±0.30
HyboNet + FFN47.41±0.0747.78±0.0249.35±0.4149.48±0.480.00±0.00
HyboNet + Proj.+FFN50.00±0.2550.11±0.4754.10±0.2154.19±0.468.72±0.62
HyTE-FH + Proj.48.24±0.2548.27±0.3352.93±0.7153.02±0.7626.95±0.99
HyTE-FH + Attn. score47.50±0.5748.07±0.3751.86±0.6551.81±0.5323.73±1.65
HyTE-FH + Residual43.15±0.4543.58±0.4548.34±0.4848.26±0.478.46±0.91
HyTE-FH + FFN48.96±0.3048.69±0.2348.13±0.7448.61±0.9418.89±0.73
HELM + Proj.49.64±0.4149.68±0.3146.97±0.7046.99±0.5935.62±0.86
HELM + Attn. score50.70±0.1350.95±0.0247.02±0.3547.07±0.3038.08±0.62
HELM + Residual42.58±0.0542.83±0.02CollapseCollapse1.24±0.80
HELM + Norm50.83±0.1850.93±0.0146.86±0.7846.83±0.7437.78±0.57
HELM + FFN50.18±0.3650.25±0.2547.12±0.2047.18±0.0637.08±1.51

Because HELM + Residual collapsed in its first Food-101 run, we did not proceed with a second run for this task for computational convenience, and retain “Collapse” in the table.

The reviewer also states that “some of the ‘performance gains’ discussed in the main text are actually only the result of models collapsing to a single class across all random seeds.” Although no specific track is identified, we believe this may refer to the case study in the Pretraining Track. Our explanation below therefore focuses on that track.

We shuffled the answer options across all test sets and found that different models exhibit some preference for option positions. This phenomenon is not unique to our benchmark and has been systematically studied previously. For example, Zheng et al. [1] observed it in models with 7B-70B parameters and attributed it to prior probability preferences for different option-label tokens. For our approximately 100M parameter models, the limited pretraining scale may further amplify this bias, although its specific cause remains uncertain. Meanwhile, reordering did not substantially change HypTBench’s overall Accuracy or the relative rankings of the methods: the HyTE-FH Block remained the best-performing method, and Lorentz methods generally outperformed Poincaré methods. As discussed in W1, our overall performance also differs little from that reported in HELM when we follow its framework exactly. We will revise the discussion to position the Pretraining Track as an exploratory evaluation under limited pretraining conditions, and expand computational resources and training scale in future versions to investigate the causes of this phenomenon.

We wish to emphasize that HypTBench’s core contribution is to provide the hyperbolic community with its first unified benchmark and systematically compare existing hyperbolic Transformer architectures and their computational operators, revealing performance differences and potential failure modes across tasks and training conditions.

References:

[1] Zheng, C., Zhou, H., Meng, F., Zhou, J., & Huang, M. (2024). Large Language Models Are Not Robust Multiple Choice Selectors. In The Twelfth International Conference on Learning Representations (ICLR 2024).

Reviewer_7647OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper is about the comparison of tools utilizing hyperbolic geometry for transformers. Some tools utilizing Euclidean geometry are also included. Authors collect 10 hyperbolic transformers and check their performance under various tasks (e.g., NLP, computer vision, problems related to biological studies,...). They find that tools based on Minkowski hyperboloid (Lorentz model) perform better than the tools based on Poincare model.

Critical strengths
  • Benchmarks of tools based on hyperbolic geometry in machine learnings become a vivid research area, should be of interest of at least small community among ICLR participants
Critical weaknesses
  • Paper misses some related literature and discussion:

--> benchmarks of tools using hyperbolic geometry in ML have been on the rise recently. Your paper misses comparison of the results with the results from other benchmarks (if possible I link to arxivs, but those papers are also published):

--> closely related HypBench (https://arxiv.org/html/2406.02772v1) --> comparison of hyperbolic embedders from ICLR (https://proceedings.iclr.cc/paper_files/paper/2026/hash/120c9ab5c58ba0fa9dd3a22ace1de245-Abstract-Conference.html) --> comparison of hyperbolic embedders in link prediction from LoG (https://arxiv.org/html/2608.07029v1) --> results on numerical stability among hyperbolic models/projections (https://arxiv.org/abs/2404.09039). Should be cited if you recall the numerical stability of Minkowski hyperboloid model. Also, including this name beside the Lorentz would be also appreciated.

--> the "history" of usage of hyperbolic geometry in ML/computer science does not start with Nickel& Kiela, 2017. Contrary to popular belief, they were not the first to include hyperbolic geometry in modelling graphs/networks/hierarchies. See aforementioned ICLR paper for the history. You are missing at least Tamara Munzner (AFAIK the first in computer science to model/visualize hierarchies with this geometry), Krioukov, Papadopoulos and Bogunya (2010) to introduce hyperbolic geometry to networks (more general case than just hierarchies) or Muscoloni and Cannistraci, 2017 (coalescent embeddings -- the first work on hyperbolic embeddings in ML community, predates Nickel& Kiela, 2017).

  • There are some common misconceptions in the paper, popular in machine learning community

--> "The central motivation of hyperbolic representation learning is to model hierarchical structures that are difficult to represent effectively in Euclidean space using negatively curved geometry."

This is untrue. Hierarchies are just the special case of networks, actually an easier one.

--> Poincare embeddings demonstrated...

Yes, the paper contained such a trivial result. There are earlier papers that should be cited instead, that actually have been published much earlier.

  • Some strange choices for the wording

In Table 1, we have Poincare and Lorentz under "Space". I am not sure about it -- those are either projections of models (it is common to call them models), but the underlying space is the same. I would ask to change it to "geometry (model)" -- you would, e.g., put "Hyperbolic (Poincare)". I really do not like incorrectly used terms in mathematical papers.

Tasks and evaluation metrics -- you use accuracy, this is a "measure" not a "metric". Please, improve the wording. Not every measure is a metric, despite the incorrect usage of this word in ML papers.

Questions for authors
  • Could you test other hyperbolic embedders in such a study? As far as I know, at least Mercator was adapted to be used in GNNs (it is in HypBench paper), and other benchmarks papers do not refrain from modifying/extending available codes of the embedders to enlarge the base in the experiments.

  • Do you actually need the c constant in the preliminaries? Most of the papers on usage of hyperbolic geometry in ML assume some c, which makes the formulations much easier (apart that a popular way to present those models is in polar coordinates). I have not seen any tool that actually benefited/used the curvature.

  • Could you provide bootstrapped confidence intervals for means instead of the standard deviations? As far as I know, the distributions of results resulting from such studies are not the best ones for such "trivial" reporting. From what I see, after introducing conf intervals, some of the results may become statistically insignificant (which may weaken your final conclusion about the dominance of Lorentz over Poincare).

6 comments
AuthorsOpenReview ↗

Response to the critical weaknesses

We thank the reviewer for the positive and highly professional assessment. Our point-by-point responses are provided below:

W1. Missing Related Literature

We thank the reviewer for pointing this out. This highly informed suggestion can substantially improve the quality of our paper. HypBench is an important, high-quality benchmark in this area, and we were aware of it before writing this paper. However, it focuses on graph neural networks whereas our study focuses on hyperbolic Transformers, we did not include it in our comparison. We have now cited and compared it in the revised manuscript, with the changes highlighted for your review. Please see lines 079-088.

In addition, we have added a section distinguishing different benchmark studies. Please see lines 097-128. We have also explicitly identified the Lorentz model as the Minkowski hyperboloid in the revision. Please see line 50 of the revised manuscript. Thank you again for your suggestion.

W2. Misunderstanding of the Machine Learning Literature

We thank the reviewer for this highly professional guidance. We agree that hierarchical structure is only one particular type of structure in complex networks, and that the motivation for hyperbolic representation learning should not be restricted to hierarchies. Our primary motivation, however, is to provide the hyperbolic Transformer community with its first unified benchmark for research. We will revise the discussion of the motivation for using hyperbolic geometry in the main text.

We also agree that research on the suitability of hyperbolic space for representing hierarchies and complex networks predates Nickel & Kiela (2017).

Indeed, Sarkar (2011) theoretically established that trees can be embedded in the hyperbolic plane with distance distortion arbitrarily close to 1. We have revised the manuscript accordingly. For ease of review, we quote the revised text below:

Low-distortion Delaunay embeddings of trees show that, with appropriate global scaling, tree metrics can be represented in the hyperbolic plane with arbitrarily low distortion (Sarkar, 2011).

W3. Terminological Precision Regarding Geometric Models and Evaluation Measures

We thank the reviewer for the suggestions concerning mathematical terminology. We agree that some expressions in the original manuscript should be made more precise. The Poincaré ball and the Lorentz hyperboloid are not two different underlying hyperbolic spaces, but two mathematical models of hyperbolic geometry. Accordingly, we have replaced “Space” in Table 1 with “Geometry (Model)” and adopted labels such as “Hyperbolic (Poincaré)” and “Hyperbolic (Lorentz)” to distinguish the underlying geometry from its specific mathematical model.

Moreover, a single label for an entire model may obscure differences between modules. We therefore further divide “Geometry (Model)” into “Backbone” and “Output Adapter,” specifying the geometric models used by the Transformer backbone and the subsequent representation mapping, respectively. Here, Output Adapter denotes the representation-mapping module following the backbone. Regarding evaluation terminology, we have changed the title of Section 4.2 from “Tasks and Evaluation Metrics” to “Tasks and Evaluation Measures,” and replaced the corresponding wording for statistical evaluation quantities such as Accuracy and F1 with “measure” or “measures” throughout the main text, table captions, and appendices. We thank the reviewer again for attending to these details.

AuthorsOpenReview ↗

Response to the questions for authors

Q1. Inclusion of Related Methods

We thank the reviewer for suggesting broader coverage of hyperbolic methods. We would like to clarify that HypTBench does not target all hyperbolic embedding algorithms, but rather hyperbolic architectural designs that can be organized into Transformer computational modules.

In fact, we have not restricted ourselves to directly using existing implementations. As described in Appendix C, we have extracted and adapted several methods originally developed for visual representation learning, graph learning, and retrieval-augmented generation into unified Transformer modules, including HVT, HNN++, GGBall, and the HyTE-FH Block.

Regarding Mercator, we acknowledge its importance in hyperbolic network embedding. However, Mercator primarily infers nodes’ hyperbolic coordinates from graph topology and geometric network models, rather than providing computational architectures that can directly replace Transformer components such as Attention and FFN. Incorporating it into the current benchmark would require additional graph-construction and sequence-modeling mechanisms, introducing new architectural variables that would make direct comparisons with the existing methods difficult.

Nevertheless, we recognize Mercator’s value and have included it in the discussion in the revised manuscript. We will also consider formally incorporating it into experimental comparisons in future versions.

Q2. Necessity of the Curvature Parameter in the Preliminaries

We thank the reviewer for the suggestion regarding mathematical presentation. We agree that, for hyperbolic models with fixed curvature, setting the curvature directly to −1-1 can simplify the formulas and improve readability.

In the revised manuscript, we will explicitly state that all experiments use fixed curvature −1-1 and simplify the main equations in Section 3 consistently to the form with c=1c=1.

Q3. Bootstrap Confidence Intervals and Statistical Significance

We thank the reviewer for the suggestions on statistical reporting. We agree that bootstrap confidence intervals can provide valuable supplementary information about model performance estimates and differences between methods, and that means and standard deviations alone do not establish statistically significant differences.

First, we would like to clarify our purpose in reporting Mean ± Std. HypTBench repeats experiments with three independent random seeds primarily to describe the training stability of different architectures under a unified experimental setup. We do not treat standard deviations as significance tests, nor do we regard higher mean performance as necessarily indicating a statistically significant advantage.

More importantly, we would like to clarify the scope of our conclusions concerning the Lorentz and Poincaré models. The central goal of HypTBench is not to establish that Lorentz geometry universally outperforms Poincaré geometry in a statistical sense, but to systematically compare the architectural implementations, geometric models, and computational operators of existing hyperbolic Transformers within a unified benchmarking framework. We will narrow this claim in the revision and formally incorporate bootstrap comparisons in a future extended version.

AuthorsOpenReview ↗

Follow up on Our Response

Thank you for your time and constructive feedback. We have carefully addressed your comments and concerns in our response. We would greatly appreciate it if you could let us know whether our response has adequately addressed your concerns or if any further clarification is needed.

We look forward to your feedback and further discussion.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the HypTBench Rebuttal

Dear Area Chair and Reviewers,

First, we sincerely thank all three reviewers for their time and effort throughout the review process, as well as for their professional questions and constructive suggestions regarding HypTBench.

We would like to particularly emphasize the following core contributions of HypTBench:

(i) Across four tracks we systematically evaluate different architectural implementations and computational behaviors of existing hyperbolic Transformers.

(ii) Beyond overall performance comparisons, we investigate the relationships among geometric representations and Transformer components, including Attention, Projection, Residual, and FFN, through coordinate substitution and module replacement experiments.

In response to the reviewers' concerns, we conducted the following additional experiments, analyses, and clarifications during the rebuttal period:

ReviewerMain ConcernOur Response and Additions
7647Insufficient coverage of related work and early researchWe added discussions of related benchmarks such as HypBench, revisited the historical contributions of early hyperbolic embedding research, and further improved citations to the early work of Tamara Munzner and other researchers.
7647Mathematical terminology and statistical rigorWe refined the classification of the Poincaré and Lorentz geometric models and the terminology used for evaluation measures, clarified the fixed-curvature setting.
aYczLack of evaluation directly addressing hierarchical structuresWe added superclass accuracy, within-superclass and cross-superclass error analyses on CIFAR-100, as well as coarse-grained entity, fine-grained entity, and boundary F1 evaluations on Few-NERD.
aYcz, w8QkInsufficient repeated runs in ablation experiments to assess result stabilityWe added ablation results using random seeds 42 and 1337 for Tables 4 and 6, reported Mean ± Std, and introduced comparisons between the Wrapper and Direct coordinate implementations.
aYcz, w8QkNear-chance performance and option-position bias in the Pretraining TrackWe provided additional details on the pretraining corpus, model size, training steps, and candidate-answer scoring protocol. We conducted option-shuffling experiments, confirmed that some models exhibit option preferences, and further discussed the evaluation limitations of this track in light of HELM and prior research on option-selection bias.
w8QkWhether the datasets exhibit structures suitable for hyperbolic modelingWe added feature-space relative hyperbolicity diagnostics covering all 20 datasets, providing supplementary analyses of their geometric structures.

Once again, we sincerely thank all three reviewers and the Area Chair for their valuable time and constructive feedback. We hope that these additional experiments and clarifications will help the Area Chair more comprehensively assess the contributions and current limitations of HypTBench.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.