- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes HexFormer, a vision Transformer built in the Lorentz model of hyperbolic space. It computes attention scores from the squared Lorentzian distance and aggregates value vectors in the tangent space via the exponential map (ExpAgg), instead of the commonly used centroid aggregation. Two variants are presented: a fully hyperbolic HexFormer and HexFormer-Hybrid, which pairs a hyperbolic encoder with a Euclidean classification head. The experiments cover CIFAR-10, CIFAR-100, and Tiny-ImageNet and examine performance across different model sizes, warmup settings, optimizers, aggregation methods, and image corruptions. The hybrid model achieves the best classification accuracy, but at a substantially higher computational cost than the Euclidean ViT, more than doubling both training and inference time.
Critical strengths
- The exponential-map aggregation is simple, but it fixes the NaN problem that centroid aggregation runs into.
- The experiments are thorough, covering different scales, optimizers, gradient behavior and corruption robustness.
- The authors are honest about the limitations, such as the higher cost and no gain on non-hierarchical data.
Critical weaknesses
-
The novelty is limited. Most components are borrowed from prior work, such as HyboNet [1], HCNN [2] and LResNet [3], and the authors admit that ExpAgg itself barely improves accuracy, so the main contribution comes down to a stability fix. [1] Fully Hyperbolic Neural Networks. ACL, 2022. [2] Fully Hyperbolic Convolutional Neural Networks for Computer Vision. ICLR, 2024. [3] Lorentzian Residual Neural Networks. KDD, 2025.
-
The accuracy gains are small. HexFormer improves over the Euclidean ViT by only 0.04 points on CIFAR-10 (93.42 vs. 93.38) and 0.13 points on Tiny-ImageNet (60.87 vs. 60.74), both well within the standard deviation (±0.18 and ±0.64). There are also no results on larger benchmarks like ImageNet-1k. While gains are clearer on CIFAR-100 (+1.01) and for the hybrid variant on Tiny-ImageNet (+2.19), they are not consistent across datasets and variants.
-
The Euclidean baseline is weak. The authors deliberately avoid a competitive training recipe (Section 5.1), and the baseline even gets worse as the model grows, dropping from 74.64 to 73.60 to 70.54 from Tiny to Base on CIFAR-100 with warmup (Table 8). This raises the question of whether it was properly tuned at larger scales (Table 7).
-
The efficiency cost is insufficiently emphasized in the main paper. On CIFAR-100 with ViT-Tiny, HexFormer and HexFormer-Hybrid take 117.42 and 121.26 ms per training iteration versus 43.77 ms for the Euclidean ViT, roughly 2.7–2.8× slower (Table 19). Inference takes about 0.31 ms per image versus 0.13 ms, roughly 2.3× slower (Table 20), and reserved GPU memory rises from 1.90 GB to 3.33 GB (Table 21). These costs are only reported in the appendix and briefly mentioned in Section 5.5. Given the small accuracy gains, the accuracy–efficiency trade-off should be presented alongside the main results.
-
It’s not clear the gains really come from hyperbolic geometry. The paper says the two models “differ only in geometry” (Section 5.1), but HexFormer also uses Lorentz batch normalization instead of standard pre-layer normalization, plus different projections and scale constraints, and any of these could affect accuracy and warmup sensitivity on their own. In fact, with AdamW and warmup, the two models perform almost identically on CIFAR-100 (74.63 vs. 74.64, Table 10). The gap only shows up without warmup, which is exactly where normalization matters most. Ablations like a Euclidean ViT with batch norm would help separate the effect of curvature from everything else.
Questions for authors
- The paper says the converse hybrid (Euclidean encoder with a hyperbolic head) is not competitive, but no numbers are given. Could you report these results?
- How does HexFormer do on ImageNet-1k?
- Could the authors report FLOPs including the hyperbolic operations omitted by THOP?
2 comments
I thank the authors for the detailed response and the additional experiments, in particular the ImageNet-1k results, the converse hybrid numbers, and the corrected FLOP count. However, some of my concerns remain unresolved:
- The gains on CIFAR-10 and Tiny-ImageNet are still within noise, and the gain on ImageNet-1k is only +0.79.
- The authors do not explain why the Hybrid variant helps on small datasets but hurts on ImageNet-1k, so it remains unclear which variant should be used when.
- The accuracy–efficiency trade-off is still not shown with the main results. That higher cost is a known challenge for hyperbolic networks does not tell us whether these small gains are worth a ~2.7× training slowdown (Appendix A.5).
We thank the reviewer for the detailed review. Table, section and figure numbers refer to the revised PDF.
W1: Novelty
Sec. 4.1 credits the building blocks from HyboNet, HCNN and LResNet. Our contributions are:
W2 / Q2: Gains and ImageNet-1k
New Table 2 (224×224, ViT-Tiny; mean ± std over 3 seeds):
HexFormer outperforms the Euclidean ViT by 0.79 points, a statistically significant difference (Welch's t-test, p = 0.002), while LViT is on par with the Euclidean ViT. One of our two models is the most accurate on every dataset: the hybrid on CIFAR-10, CIFAR-100 and Tiny-ImageNet, and HexFormer on ImageNet-1k. Over LViT, HexFormer gains 1.45, 4.42, 6.51 and 0.72 points on the four datasets (statistically significant on ImageNet-1k, p = 0.015). Removing warmup costs the Euclidean ViT 2.78 points on CIFAR-100, but HexFormer only 0.59 (Table 9).
W3: Euclidean baseline
To compare the geometries, we used the same simple ViT and training recipe for both models, so the benefit of the geometry is clear before further improvements are added (Sec. 5.1). The Euclidean ViT was tuned at every scale (Table 8), and all models use drop path 0.1 and the same augmentation. Its decline with scale comes from overfitting: the training–validation gap of Euclidean ViT-Base ends near +8 points, while HexFormer-Base stays below zero (A.3.1, Fig. 4). On ImageNet-1k, it is still outperformed.
W4: Efficiency
The cost is reported in A.5, mentioned in Sec. 5.5, and stated as a limitation. Inference remains fast (about 0.31 ms per image). Training time and memory are higher, which is a known challenge for hyperbolic networks. The corrected FLOP count (Q3) shows only about 4% arithmetic overhead; the time overhead comes mainly from memory-bound element-wise operations.
W5: Do the gains come from the geometry?
Lorentz linear layers and normalization on the manifold are the hyperbolic counterparts of the Euclidean layers, needed to keep representations on the hyperboloid and used in the same places; they are not additional design choices. HexFormer's CLS token and normalization offsets are points on the manifold, which Riemannian AdamW keeps there; for all other parameters it performs the AdamW update. With AdamW for both models, the warmup-10 comparison is a tie, but without warmup HexFormer leads by 2.8 points (Table 11). Matched-optimizer ablation (Table 12; CIFAR-100, ViT-Tiny, warmup 10, same seed):
We also ran the suggested Euclidean ViT with batch normalization (CIFAR-100, ViT-Tiny, AdamW; Table 13):
Batch normalization lowers accuracy in both settings and still loses 2.4 points without warmup, so normalization alone does not explain HexFormer's behavior.
Q1: Converse hybrid
A Euclidean encoder with a Lorentz MLR head reaches 73.63 ± 0.28 (CIFAR-100, ViT-Tiny, warmup 10, 3 seeds), below the Euclidean ViT (74.64). The benefit therefore comes from hyperbolic feature extraction. Added to A.3.
Q3: FLOPs
We recomputed the FLOPs with an operator-level counter (at the PyTorch dispatcher). It includes all hyperbolic operations, as well as the attention products, which THOP misses for every model. Forward FLOPs per image (ViT-Tiny, CIFAR-100):
The hyperbolic overhead is about 4% (not 0.4%), and ExpAgg costs about the same as the centroid. Element-wise operations are memory-bound, so the measured runtime (Tables 22–24) remains the practical measure. Table 25 is updated.