- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents HypTBench, a benchmark that decomposes hyperbolic Transformers into interchangeable components and evaluates them under a shared training protocol. It compares a Euclidean Transformer with ten hyperbolic methods from the Euclidean-backbone (HiT, ConeHeads, Hyp-ViT), Poincaré-manifold (HVT, GGBall, HNN++), and Lorentz-manifold (HyboNet, HyTE-FH block, Hypformer, HELM) families on 20 datasets spanning NLP, computer vision, pretraining reasoning, and genomic sequence modeling, with three seeds and 556 runs in total. The authors report that Lorentz-family methods, led by Hypformer, achieve the best results and are more efficient than Poincaré-family methods. They further find that re-expressing hidden states in Poincaré coordinates does not consistently reduce accuracy and that transplanting Hypformer modules into other Lorentz models yields base-model- and task-dependent, non-additive effects, and conclude that performance depends on how geometric operators are integrated rather than only on the choice of hyperbolic model.
Critical strengths
- A benchmark for hyperbolic models is genuinely valuable for research on hyperbolic learning.
- The experimental work is extensive, with 556 experimental runs.
Critical weaknesses
-
Section 5.1 does not demonstrate that methods based on the Lorentz model generally outperform those based on the Poincaré ball, since the compared methods differ in their designs, architectures, and parameter counts.
-
The Lorentz model is numerically more stable than the Poincaré ball, which is why research has already shifted from Poincaré-based HNN [1] and HNN++ [2] to Lorentz-based HCNN [3] and Hypformer [4]. This has already been pointed out in papers such as HCNN [3], so the advantage of Lorentz-based methods over Poincaré-based ones cannot be counted as a contribution of this paper.
-
Section 5.2.1 only changes the representation of the intermediate Transformer hidden states to Poincaré coordinates. A fair comparison should switch both the hidden states and the operators to the Poincaré model. If only the hidden states are changed while the operators are kept, the hyperbolic computation is still essentially carried out in the Lorentz model.
References
[1] Hyperbolic Neural Networks
[2] Hyperbolic Neural Networks++
[3] Fully Hyperbolic Convolutional Neural Networks for Computer Vision
[4] Hypformer: Exploring Efficient Transformer Fully in Hyperbolic Space
Questions for authors
-
I am curious about the Pretraining Track. Could the authors explain (i) how the near-chance Pretraining accuracies in Table 10 can distinguish methods, (ii) why each model in Figure 13 mostly outputs the same option letter, suggesting position bias rather than reasoning, and (iii) why HVT and ConeHeads have identical means and standard deviations in four of the five Table 10 columns?
-
Could the authors provide comparisons in which some methods are switched to the operators of the other hyperbolic model? Such comparisons would be fairer and more intuitive.
-
Why are these datasets suitable for evaluating hyperbolic methods? Although they have been adopted in previous hyperbolic papers, a benchmark should describe in its dataset section the properties that make them suitable for evaluating hyperbolic methods.
6 comments
Response to the questions for authors 1
Q1. Questions Regarding the Pretraining Track
We appreciate the reviewer’s valuable questions. Our point-by-point responses follow:
(i) Near-chance results
First, the primary purpose of the Pretraining Track is not to establish that 100M parameter models possess strong general reasoning capabilities. This experimental design follows HELM (He et al., NeurIPS 2025) exactly. HELM likewise evaluates approximately 100M parameter models on the same five reasoning benchmarks. As shown in Table 1 of the original HELM paper, its results are also close to chance: DeepSeekV3 achieves an average accuracy of 22.2%, while HELM-MiCE achieves 24.1%, an improvement of only 1.9 percentage points. Nevertheless, the original paper uses relative performance differences between architectures to analyze the effectiveness of hyperbolic designs. Despite the low absolute accuracy, HELM treats this relative advantage as important evidence for hyperbolic architectures, explicitly emphasizing consistent improvements over Euclidean baselines in Section 5.1 and citing performance gains of up to 4% as evidence of enhanced reasoning in its abstract.
In our Table 12, HyTE-FH outperforms the Euclidean baseline on all five tasks, increasing average accuracy from 23.28% to 25.03%, a gain of 1.75 percentage points. All results report means and standard deviations over three random seeds, which is sufficient to demonstrate the stability of HyTE-FH’s relative improvements.
References:
[1] He, N., Anand, R., Madhu, H., Maatouk, A., Krishnaswamy, S., Tassiulas, L., Yang, M., & Ying, R. (2025). HELM: Hyperbolic Large Language Models via Mixture-of-Curvature Experts. Advances in Neural Information Processing Systems, 38, 142604-142635.
(ii) Position bias
This is a valuable question. We acknowledge an oversight in the presented case and thank the reviewer for pointing out potential option-position bias in the Pretraining Track. We conducted additional option-order perturbation experiments to examine the models’ prediction behavior.
The additional experiments confirm the reviewer’s concern: different models exhibit clear option-position preferences. After further changing the option order, we observed that models continued to show high consistency in their selected positions. However, this neither indicates a failure of our benchmark nor is unique to it. Prior work, such as Zheng et al., has already identified position bias in large language models including the LLaMA family. Those researchers attribute this phenomenon to prior probability preferences for different option-label tokens. For our approximately 100M parameter models, the limited pretraining scale may further amplify this bias, although its specific cause remains uncertain. Meanwhile, reordering did not substantially change HypTBench’s overall Accuracy or the relative rankings of the methods. As noted in (i), our overall performance also differs little from the results reported in HELM when we follow its framework exactly. In the revision, we will explicitly report the observed position bias and revise the discussion to position the Pretraining Track as an exploratory evaluation under limited pretraining conditions. Future versions will further expand the training scale to investigate this phenomenon.
We wish to emphasize that HypTBench’s core contribution is to provide the hyperbolic community with its first unified benchmark and systematically compare existing hyperbolic Transformer architectures and their computational operators, revealing performance differences and potential failure modes across tasks and training conditions.
(iii) Identical numerical results
We thank the reviewer for pointing this out. This was a mistake in recording experimental data. We have included the correct results in the revised manuscript and highlighted them, and updated all figures that use Avg. values. For ease of review, we provide the correct data in this response:
Table A. Pretraining Track
| Dataset | ConeHeads (Acc) | HVT (Acc) |
|---|---|---|
| CommonsenseQA | 19.62±0.06 | 19.59±0.02 |
| HellaSwag | 25.91±0.06 | 25.19±0.12 |
| OpenBookQA | 25.29±0.10 | 25.31±0.10 |
| MMLU | 23.05±0.09 | 22.99±0.07 |
| ARC-Challenge | 23.68±0.04 | 23.30±0.52 |
| Avg. | 23.51±0.06 | 23.28±0.15 |
Response to the questions for authors 3
Q3. Dataset Suitability
We thank the reviewer for pointing out that the rationale for dataset selection requires further clarification. We agree that prior use of a dataset in a hyperbolic-model paper does not, by itself, establish that its structure is suitable for hyperbolic modeling. We have therefore added feature-space relative hyperbolicity diagnostics covering all datasets.
We compute the Euclidean distance matrix between frozen features, use the first point in each sample as the base point to obtain the fixed-base-point deviation of the Gromov-product four-point condition, and calculate:
where is the maximum pairwise distance within the sampled feature set.
For image tasks, we use frozen features from an ImageNet-pretrained ResNet34. For NLP tasks and downstream multiple-choice tasks in the Pretraining Track, we use aggregated features from the final layer of BERT-base-uncased. For DNA sequences, we use fixed 5-mer frequency features. The feature pools reuse the union of the evaluation subsamples obtained with the three previous sampling seeds.
For each dataset, we fix the random seed to 42 and perform 10 sampling rounds with replacement, typically drawing 1,000 instances per round. The frozen feature pools for OpenBookQA and Enhancers contain only 498 and 400 instances, respectively, so we use 498 and 400 draws per round for these datasets. The table reports the mean and population standard deviation of the 10 relative δ estimates.
Table A. Feature-Space Relative Hyperbolicity Diagnostics for All Datasets
| Track | Dataset | Feature Pool N | Draws per Round n | Relative δ (Mean ± Std) |
|---|---|---|---|---|
| NLP | AG News | 2606 | 1000 | 0.2589±0.0175 |
| NLP | MNLI | 2695 | 1000 | 0.2115±0.0121 |
| NLP | Few-NERD | 2913 | 1000 | 0.2343±0.0087 |
| NLP | SemEval-2010 | 2011 | 1000 | 0.2598±0.0107 |
| NLP | SQuAD 2.0 | 2775 | 1000 | 0.2545±0.0183 |
| CV | CIFAR-10 | 2722 | 1000 | 0.2642±0.0118 |
| CV | CIFAR-100 | 2724 | 1000 | 0.2595±0.0253 |
| CV | Food-101 | 2882 | 1000 | 0.2513±0.0134 |
| CV | EuroSAT | 2029 | 1000 | 0.2711±0.0115 |
| Pretrain | CommonsenseQA | 1215 | 1000 | 0.2711±0.0162 |
| Pretrain | HellaSwag | 2710 | 1000 | 0.3159±0.0165 |
| Pretrain | OpenBookQA | 498 | 498 | 0.2608±0.0102 |
| Pretrain | MMLU | 2795 | 1000 | 0.2591±0.0134 |
| Pretrain | ARC-Challenge | 1170 | 1000 | 0.2911±0.0122 |
| Bio | LTR Copia | 1621 | 1000 | 0.1242±0.0128 |
| Bio | CMC-EnSpm | 1640 | 1000 | 0.2651±0.0409 |
| Bio | Processed Pseudogene | 1600 | 1000 | 0.1744±0.0179 |
| Bio | H3K4me3 | 2260 | 1000 | 0.1736±0.0269 |
| Bio | Promoter All | 2484 | 1000 | 0.1632±0.0493 |
| Bio | Enhancers | 400 | 400 | 0.1204±0.0267 |
We hope that our responses have addressed your concerns.
Response to the critical weaknesses and the questions for authors 2
We thank the reviewer for the constructive and professional comments, which are very helpful for improving the quality of our paper. Our point-by-point responses follow:
W1. Differences in Parameter Counts
We thank the reviewer for raising this important issue. We agree that hyperbolic Transformer methods differ in their internal operator designs, overall architectures, and parameter counts, and that relying solely on the overall performance comparison in Section 5.1 is therefore insufficiently rigorous.
First, Section 5.1 primarily aims to compare the overall performance of existing hyperbolic Transformer methods under unified experimental conditions; it does not directly demonstrate that Lorentz geometry itself is superior to Poincaré geometry. Second, to address architectural and operator differences, we designed dedicated ablations in Sections 5.2.1 and 5.2.2 to further analyze their effects. For example, in Table 4, we keep the original Transformer operator designs of HyboNet and Hypformer unchanged, altering only the intermediate hidden-state representation to Poincaré coordinates and converting coordinates when invoking Lorentz operators. HyboNet’s accuracy improves by 2.25 and 3.67 percentage points on CIFAR-100 and Food-101, respectively, while Hypformer’s accuracy improves by 1.63 percentage points on CIFAR-100.
In Table 6, we further decompose the Transformer into core modules, including Projection, Attention Score, Value Aggregation, Residual, Normalization, and FFN, and perform controlled replacements across Lorentz Transformer architectures. For example, replacing HELM’s Attention Score with the corresponding Hypformer module improves accuracy by 8.79 and 10.34 percentage points on CIFAR-100 and Food-101, respectively. Moreover, replacing HyboNet’s Projection or FFN individually may reduce performance, whereas replacing them jointly can yield improvements.
Finally, these experiments constitute one of HypTBench’s core contributions. Beyond comparing the overall performance of existing Lorentz and Poincaré Transformers, we investigate the sources of performance differences through coordinate-representation and module-level ablations. Our conclusion is that the evaluated Lorentz-family Transformer implementations perform better overall, and that this advantage is closely associated with their specific operator designs and module compatibility.
W2. Definition of the Contribution
We thank the reviewer for raising this point. Previous work has indeed reported that the Lorentz model is numerically more stable than the Poincaré ball model, and recent research has increasingly shifted toward the Lorentz model. However, those findings were obtained using different training configurations and data. We are the first to reach this conclusion under a unified protocol and training configuration. Our contribution primarily lies in providing the hyperbolic Transformer community with its first unified benchmark.
W3 & Q2. Fair Ablation Experiments
This comment is valuable for improving the quality of our paper. We have therefore added the corresponding ablations; please see Table 4 of the revised manuscript. For ease of review, we reproduce them here:
We clarify the three variants as follows. The original Lorentz variant uses both Lorentz hidden representations and the original Lorentz operators. The Wrapper variant stores intermediate hidden states in Poincaré coordinates but converts them back to Lorentz coordinates when executing the original operators. The Direct variant rewrites the original operators as direct formulas in ball coordinates, without reconstructing full hidden states on the Lorentz manifold during the forward pass.
Table A. Geometric ablations