acceptodds
Under review as a conference paper at ICLR 2027

Rethinking the Fully Hyperbolic Vision Transformer in Polar Coordinates

Abstract

Hyperbolic space can embed intrinsic hierarchies in data with low distortion due to the exponential growth of volume with distance from the origin. However, current Lorentz transformer blocks are formulated in ambient coordinates, where numerical errors increase at large radii due to instability in the Lorentzian inner product, leading many models to limit the radius to avoid this issue. This prevents us from utilizing the regions of hyperbolic space that motivate the geometry. To address this, we revisit the components of the transformer block in polar coordinates, where hyperbolic operations such as distance calculation and attention centroids can be computed without the numerical cancellation in their ambient-coordinate formulations. Specifically, we propose a polar fully-connected layer that separately maps an embedding's direction and radius, allowing the radius of a feature to be learned rather than determined by the norm of a linear map. We additionally introduce horospherical shifts as relative positional encodings whose query-key distances grow logarithmically with the token gap. Finally, we reformulate the residual connection as a Lorentz boost, which is an isometry at any radius. Combining these components, we develop a fully hyperbolic transformer that substantially improves performance over Euclidean and hyperbolic baselines on standard vision tasks. We further evaluate our model on ImageNet and demonstrate its ability to generalize to other datasets using pretrained weights, similar to Euclidean counterparts.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
hyperbolic transformers, hyperbolic deep learning, vision transformers

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_aYy7OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes to use polar coordinates instead of what the authors call “ambient coordinates”. The authors analyze the precision loss if the “ambient coordinates” are used and give formulae of polar operations. Based on these building blocks, the paper proposes a hyperbolic vision transformer called the polar-ViT, with fully connected layers, attention and positional encoding in polar coordinates. The authors use some image classification datasets and compare polar-ViT with some other vision transformer models.

Critical strengths
  1. This paper has a good and comprehensive survey of previous related works.
  2. This paper is well-organized and generally easy to follow.
  3. The paper focuses on vision transformer models and in the experiments compares polar-ViT with many other ViT models.
Critical weaknesses
  1. The contribution of the paper is overly incremental. Although using polar coordinates is not well-studied for hyperbolic neural networks before, the transform between “ambient coordinates” and polar coordinates is straightforward. The precision loss is an extension of (Mishne et al., 2023); the Lorentz fully connected layer and attention layer follow (Chen et al., 2022); the horosphere idea has been used in (Chen et al., 2026); the layer normalization is similar to batch normalization of (Chen et al., 2025).
  2. The experiments only consider “for 100 epochs with the HexFormer training recipe”. It is not a natural setting because in image classification one can select number of epochs, etc., using validation.
  3. The paper is not comparing polar-ViT with the state-of-the-art Euclidean ViT. It is unclear why a hyperbolic version of ViT would be expected to work better than a Euclidean counterpart for Euclidean image datasets.
Questions for authors

See "weaknesses".

4 comments

We thank the reviewer for their time. Below we highlight the differences between us and established literature and hopefully answer your other points. All of these improve clarity and will be added to the paper as well. In the rebuttals above we also try our polar forms and ablate in multiple other settings which we would appreciate you taking into consideration as well.

1. Incremental over Mishne et al. [1] and Chen et al. [2, 3, 4]

The change of coordinates is straightforward but causes gradient issues in backpropagation. The main contribution is the derivation of cancellation-free forms of the operations (Eqs. 5 to 7) and of their gradients (Appendix D.1), the handling of the origin, and the precision bounds (Propositions 1 and 2). At the origin the direction of a point is not defined and its gradient grows as 1/‖x‖, so Eq. 8 takes the Klein vector, which stays smooth there, and Appendix B gives the origin cases of the tangent inner product and the centroid. As for the other conributions:

  • Mishne et al. [1] bound the storage of a single Lorentz point in double precision (their Prop. 3.2) and leave the distance between two points as future work (their Section 6). Proposition 1 gives the failure radius of the ambient distance at any precision, Section 3.1 extends it to the weighted centroid, and the factorization of the centroid in Eqs. 6 and 7 is new. We also derive the gradient forms mentioned above, which stay smooth at the origin.
  • Chen et al. [2] train a bounded form of their linear layer, where the time coordinate passes through a sigmoid. van der Klis et al. [13] prove that when using this layer the radius grows only logarithmically with the number of gradient steps, and in their simple training test it reaches the iteration limit at a maximum radius of 5 to 6 and cannot reasonably learn further radii. Our PolarFC fixes that since it equals their general form at initialization only. It then learns the radius without the bound, and reaches every target up to radius 18 in the same test (Appendix E.3). Performance wise, in Table 6 the sigmoid-bounded radius is the lowest rule that trains (74.04), a hard radius bound diverges, and our PolarFC reaches 78.45.
  • Chen et al. [3] use the Busemann function for a classifier and a linear layer. They define no isometry and no positional encoding. We use the parabolic one-parameter subgroup of the Lorentz group as a relative positional encoding (Eq. 10) and derive the distance law it relies on (Eq. 11).
  • Our derivation of the layer norm is more straightforward but it is not the core part of our work.

We will make these four distinctions explicit in the related work.

2. Training for 100 epochs with the HexFormer recipe is not a natural setting

The HexFormer [5] recipe matches the small-dataset ViT protocol of [6], with depth 9, width 192, 100 epochs, batch 128 and 10 warm-up epochs. Gani et al. [7] call it the default training setting for small-scale datasets. We use it so that our rows share the protocol of the Euclidean ViT and HexFormer rows of Table 1. If you mean selecting on a validation split, we do this for ImageNet-1k, for the segmentation experiments of Appendix F, and for the graph and ResNet experiments in the rebuttals to other reviewers above.When we do report the last epoch it is to match the published baselines and compare fairly against them.

Additionally, the ImageNet experiments follow two further published recipes, [8] for ImageNet-1k and [9] for fine-tuning. These are standard published recipes for Euclidean ViT training.

3. Comparison with state-of-the-art Euclidean ViTs, and why a hyperbolic model should help on image data

Polar-ViT modifies the plain ViT, so the matched comparison is the plain ViT at equal depth, width, heads, patch size and recipe. HexFormer makes the same comparison. We also follow LViT [10] and compare with a stronger fine-tuned Euclidean ViT (Flat-ViT, Table 3). For additional context, published from-scratch Euclidean ViTs of about 3M parameters at 100 epochs reach 73.81 (ViT in [6]), 74.64 [5], and 57.07 and 60.74 on Tiny-ImageNet. Published numbers above ours come from models three times larger (SL-CaiT [6]), from convolutional tokenizers trained for 300 to 5000 epochs [11], or from an added 200-epoch self-supervised stage [7]. These additions are independent of the geometry and can be applied to Polar-ViT in the same way. On ImageNet-1k, where we trained both models with one known published recipe, the margin is 3.5 points at the Tiny size and 2.2 at the Small size (Table 2), and it carries over to fine-tuning (Table 3).

As for why a hyperbolic model should help on image data, this is a well studied and discussed setting in the literature. The main consensus is that the geometry applies to the learned representation not to the pixels. A Euclidean ViT also maps patches into a feature space unrelated to pixel geometry, and the question is which geometry fits that space. Image tasks can carry hierarchical structure, for example, ImageNet classes are hierarchies of WordNet, and CIFAR-100 groups its classes into 20 superclasses, and CUB and Aircraft, where Polar-ViT gains 3.9 and 5.3 points over the Euclidean ViT after fine-tuning at the Tiny size (Table 3), come with label hierarchies used by H-CAST [9]. CNN features of image datasets are also moderately tree-like, and Khrulkov et al. study this using gromov hyperbolicity distortion measurements. They found a relative δ-hyperbolicity of 0.17 to 0.26 on CIFAR-10, CIFAR-100, CUB and MiniImageNet [12], where 0 is a tree and 1 is not hyperbolic at all. We do not claim that negative curvature helps on every image task but the hyperbolicity values hint at possible benefits from using that space. The gain of Polar-ViT comes mainly from components that depend on the curvature. The horospherical shift adds 2.1 points (Table 5), and under it the query-key distance grows logarithmically with the token gap (Eq. 11). In flat space the same construction is a translation, and the distance grows linearly.

References

  1. Mishne et al. The numerical stability of hyperbolic representation learning. ICML 2023.
  2. Chen et al. Fully hyperbolic neural networks. ACL 2022.
  3. Chen et al. Hyperbolic Busemann neural networks. CVPR 2026.
  4. Chen et al. Gyrogroup batch normalization. ICLR 2025.
  5. Alyoussef et al. HexFormer: Hyperbolic vision transformer with exponential map aggregation. arXiv:2601.19849.
  6. Lee et al. Vision transformer for small-size datasets. arXiv:2112.13492.
  7. Gani et al. How to train vision transformer on small-scale datasets? BMVC 2022.
  8. Beyer et al. Better plain ViT baselines for ImageNet-1k. arXiv:2205.01580.
  9. Park et al. Visually consistent hierarchical image classification. ICLR 2025.
  10. He et al. HyperCore: The core framework for building hyperbolic foundation models with comprehensive modules. arXiv:2504.08912.
  11. Hassani et al. Escaping the big data paradigm with compact transformers. arXiv:2104.05704.
  12. Khrulkov et al. Hyperbolic image embeddings. CVPR 2020.
  13. van der Klis et al. Fast and geometrically grounded Lorentz neural networks. arXiv:2601.21529.
Reviewer_aYy7OpenReview ↗

Thanks for the rebuttal

Thanks for the rebuttal.

  1. I think your rebuttal exactly shows the incremental nature of the work.
  2. Thanks for the clarification. I now agree that the setting is okay.
  3. It is unclear how "the geometry applies to the learned representation not to the pixels" is reflected in your model. It seems to me that your neural layers directly apply to the pixel space.

Thank you for the reply. We are glad the training setting is resolved.

3. Where the geometry acts

There are two main approaches to applying hyperbolic models to images. Hybrid models keep a Euclidean encoder and map its features to hyperbolic space before the head. Khrulkov et al. [1] do this for example, and Guo et al. [2] describe the same design on MNIST, CIFAR-10/100 and ImageNet. We also followed this example for other settings in the rebuttal experiments we conducted.

On the other hand, fully hyperbolic models place the image on the manifold before the first learned layer. This has been done in Poincaré ResNet [3] which learns "directly from the pixel-level", HCNN [4] treats an image as "an ordered set of color vectors", and HCNN+ [5], FGG-LNN [6] and ILNN [7] also train fully hyperbolic image classifiers. The Hyperbolic U-Net [8] does the same for segmentation, "operating entirely in the Poincaré ball". For transformers, HexFormer [9] and LViT [10] project images "into the Lorentz space" before a Lorentz convolutional patch embedding.

Our approach follows the second group and uses the same lift as HCNN where we keep the flattened patch as the space coordinates and calculate the time coordinate with no exponential map. We believe this preserves the features without assuming direct hyperbolicity and transforming them accordingly.

However, based on you suggestion we attempted to test the effect of the geometry in early stages of the model. We replaced the patch embedding and the first blocks with their Euclidean counterparts and kept the rest of the model hyperbolic (CIFAR-100, recipe of Table 1, same seed).

Euclidean parttop-1
none (Polar-ViT, same seed)80.04
patch embedding80.05
patch embedding + block 179.14
patch embedding + blocks 1 to 379.15
patch embedding + blocks 1 to 678.34
patch embedding + blocks 1 to 9 (Euclidean encoder, hyperbolic pooling and classifier)76.18

Making the direct pixel embedding layer Euclidean changes nothing almost (80.05 against 80.04), so the gain does not come from treating pixels as hyperbolic points. It comes from the blocks that act on learned features. When all the nine encoder blocks are Euclidean the model loses 3.9 points, and the last three blocks account for 2.2 of them. Early blocks contribute less, but block 1 still contributes 0.9.

To explain this trend we measured the relative δ-hyperbolicity of the token features at every depth, following Khrulkov et al. [1]. Lower means more tree-like.

siteEuclidean ViTPolar-ViT
patch embedding0.3010.186
block 10.2790.149
block 20.2720.124
block 30.2670.105
block 40.2720.101
block 50.2690.092
block 60.2440.085
block 70.2390.072
block 80.2250.066
block 90.2020.077

The Euclidean ViT has no hyperbolic component, but its features become more tree-like with depth (0.30 to 0.20), and its last block falls in the range Khrulkov et al. report for CNN features (0.17 to 0.26). This agrees with the ablation and could explain why removing the hyperbolic blocks deeper in hurt performance more. The early features are less tree-like but not flat, which fits the smaller gain of the early blocks.

This deserves a wider study across models and datasets. The depth/hyperbolicity trend also suggests using a different curvature per block. This normally could lead to instability since the curvature greatly affects the maximum radius we can represent. Learning the curvature without warmup in Table B of our reply to Reviewer HKB5 turned non-finite in epoch 1.

A second, smaller source of gain reported in the literature is that hyperbolic space represents data better at small embedding dimensions [11, 12]. HCNN shows this for images, where the hyperbolic models outperform the Euclidean ones at low dimensions of the final ResNet block. This could explain why it worked better for other methods. We will add both tables to the appendix.

References

  1. Khrulkov et al. Hyperbolic image embeddings. CVPR 2020.
  2. Guo et al. Clipped hyperbolic classifiers are super-hyperbolic classifiers. CVPR 2022.
  3. van Spengler et al. Poincaré ResNet. ICCV 2023.
  4. Bdeir et al. Fully hyperbolic convolutional neural networks for computer vision. ICLR 2024.
  5. Bdeir et al. Robust hyperbolic learning with curvature-aware optimization. NeurIPS 2026.
  6. van der Klis et al. Fast and geometrically grounded Lorentz neural networks. arXiv:2601.21529.
  7. Shi et al. Intrinsic Lorentz neural network. ICLR 2026.
  8. Mishra et al. Hyperbolic U-Net for robust medical image segmentation. MIDL 2026.
  9. Alyoussef et al. arXiv:2601.19849.
  10. He et al. arXiv:2504.08912.
  11. Nickel and Kiela. Poincaré embeddings for learning hierarchical representations. NeurIPS 2017.
  12. Sala et al. Representation tradeoffs for hyperbolic embeddings. ICML 2018.
Reviewer_KQFUOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper propose a fully hyperbolic Vision Transformers that remain numerically stable for embeddings that have large radius. Existing Lorentz operations (such as centroid) causes numerical errors in distance calculations and centroid aggregation. To address this, the paper proposes Polar-ViT, which represents hyperbolic embeddings using their radius and direction and reformulates hyperbolic operations in polar coordinates to improve numerical stability. The model also introduces a polar fully connected layer that learns radius and direction separately, horospherical shifts for relative positional encoding, and a boost-based residual connection. The authors theoretically analyze the precision limits of ambient and polar operations, showing that polar operations can accurately resolve distances at much larger radii. They validate these results through numerical precision and single-layer optimization experiments, and evaluate Polar-ViT on CIFAR-100, Tiny-ImageNet, ImageNet-1k, and fine-grained classification tasks. Polar-ViT outperforms the reported Euclidean and hyperbolic baselines on image classification, and its pretrained weights transfer effectively to downstream tasks. The authors also extend their polar operations to a hyperbolic U-Net, which improves over the ambient Lorentz counterpart on medical image segmentation.

Critical strengths
  1. The paper proposes a fully hyperbolic Vision Transformer that addresses numerical instability in Lorentz operations by representing embeddings in polar coordinates. The authors derive more stable formulations for hyperbolic distance and centroid aggregation, and introduce several new architectural components, including a polar fully connected layer, horospherical positional encoding, and boost-based residual connections. This is a meaningful architectural contribution that allows hyperbolic models to operate at larger radii.

  2. The authors evaluate Polar-ViT on CIFAR-100, Tiny-ImageNet, ImageNet-1k, and fine-grained classification tasks, where it consistently outperforms the matched Euclidean baselines and improves over existing hyperbolic baselines on the smaller datasets.

Critical weaknesses
  1. While the paper focuses on fully hyperbolic ViTs, the components proposed in the paper are not limited to the ViT architecture or vision tasks. How does the method compare with other hyperbolic Transformers and CNNs, or generalize to other modalities (e.g. Hypformer, Yang et al.; HCNN, Bdeir et al.; HELM, He et al.; Fully HNN, Chen et al.; etc.)? As the comparisons with other hyperbolic baselines are currently quite limited, and several experiments only compare against Euclidean ViTs, at least selectively comparing with additional existing hyperbolic models and applying the method to additional settings would better demonstrate the benefits and generalizability of the proposed model components.
  2. One of the main motivations for the method is that it enables better stability for embeddings with large radius. However, while there are ablations on operation accuracy, there doesn't seem to be a clear connection of the stability to model performance. I wonder if the authors could provide some qualitative or quantitate analysis that demonstrates how the numerical stability of polar methods correlates to model performance (e.g. does polar-vit allow the model to use embeddings with larger radius and therefore leads to better separation and expressiveness?)
Questions for authors

See weakness

4 comments

We thank the reviewer for their time and expertise. Reviewer HKB5 shared the same point and we believe expanding the experimental settings and comparing to other established baselines also enriches the paper, even if the focus is on Polar-ViT.

1. "How does the method compare with ... or generalize to other modalities"

We agree that the components are not limited to ViT. We found that they apply as well, and often better, in settings where the radius of the embeddings is naturally larger. The polar U-Net of the appendix is one of those settings and we repost its table below. We also add genome sequences, ResNets and graphs. In each we take a published hyperbolic network, keep the authors' code and protocol, and put it in polar form. We also ran our model in the setting of Hypformer [1] (last table below). HELM [2] is a language model and has no results in these settings.

For Fully HNN [8], the first row of Table 5 of the submission is a split-head Lorentz ViT that follows their Lorentz Transformer (Section 4.2) and reaches 73.99 against 80.15 for Polar-ViT under the same recipe, and Table 6 compares their sigmoid-bounded linear layer (74.04) with the polar layer (78.45) (our run).

Medical segmentation, test Dice (Table 9 of the submission, with a new row that keeps the ambient layers, evaluates them in polar form and removes the caps).

modelISIC 2016ISIC 2018Kvasir-SEGSANETBUSI
Euclidean U-Net (published, [3])0.920.890.860.780.82
Poincaré U-Net (published, [3])0.910.870.830.760.80
Lorentz U-Net, ambient0.920.810.790.730.71
Lorentz U-Net, polar arithmetic, caps removed0.920.850.840.780.74
Lorentz U-Net, polar with polar fc, caps removed0.930.890.860.810.78

Genome sequences, TEB benchmark, MCC × 100. The Euclidean CNN, HCNN-S and HCNN-M columns and the hybrids E2H-CNN (Euclidean CNN with a Lorentz MLR head) and H2E-CNN (HCNN with a Euclidean MLR head) are from [4], five seeds. The ambient column is our network with the same layers in ambient arithmetic.

datasetEuclidean CNNHCNN-SHCNN-ME2H-CNNH2E-CNNambient (ours)polar (ours)
LINEs70.63±1.2476.12±2.1677.10±2.9275.65±0.8379.15±2.3674.76±5.4983.93±0.29
SINEs85.15±1.6485.45±1.1681.85±2.9589.65±2.1379.49±3.4089.22±3.8094.73±0.66
hAT-Ac87.45±0.9089.61±1.3491.04±1.5889.76±0.8585.63±1.4490.21±0.5293.59±0.63
processed pseudogenes60.66±0.8268.30±0.9365.41±5.5466.68±1.3166.12±0.4365.40±2.5571.53±1.97
unprocessed pseudogenes51.94±2.6956.13±0.5658.36±1.8058.09±0.9658.16±1.4059.51±0.5362.93±1.26

ResNet-18 on CIFAR-100. Each block uses the code and protocol of the cited paper

networkreferencepolar form (ours)
Fully hyperbolic FGG-LNN, protocol of [5], fp32their Table 1: FGG-LNN 76.14±0.11, HCNN 76.09±0.23, Euclidean ResNet-18 76.65±0.2476.89±0.35
the same, fp32 checkpoint evaluated in bfloat1629.8776.43
the same, trained in bfloat16non-finite in epoch 173.87
Euclidean ResNet-18 with a hyperbolic MLR head, recipe of [6]their Table 1: Euclidean 77.72±0.15, Poincaré MLR head 77.19±0.50, Lorentz MLR head 78.03±0.2178.38±0.43

Graph node classification from node features, without message passing. Neither [6] nor [7] reports these datasets, so we build the Lorentz HNN from the layers of [6] and HNN++ from the layers of [7]. Our row replaces the fully connected layers of the Lorentz HNN by the polar layer of Eq. 8 and removes the radius caps. All rows are our runs, each tuned over 16 settings on a separate seed.

modelDiseaseAirportPubMedCora
HNN++ (layers of [7])80.15±0.4988.80±0.4070.52±2.3949.64±1.77
Lorentz HNN (layers of [6])79.62±0.8371.44±2.2169.40±1.1954.42±2.35
Polar HNN (with polar fc)82.34±0.1195.42±0.5076.26±0.3757.58±1.58

Hypformer's node classification benchmarks, accuracy. We run our model in the authors' released code [1]. Their fully connected layers are replaced by the polar layer of Eq. 8 and their attention by a linear-cost form of ours in polar form. All values are over 5 runs.

modelDiseaseAirportCoraCiteseerPubMed
Hypformer (published, [1])93.0±0.795.0±0.585.0±0.373.3±0.481.3±0.3
Polar model in their setting (ours)95.28±0.6996.76±0.2786.33±0.4874.00±0.2281.83±0.31

We see here that the polar arithmetic improves accuracy by removing the clips and improving stability aside from the new polar fc layer. We will add these results as an appendix and keep the scope of the main paper at the vision transformer.

2. "demonstrates how the numerical stability of polar methods correlates to model performance"

Table A in our reply to Reviewer HKB5 gives the radius by depth for every model. The models push the embeddings outward as depth increases, and most of all at the output layers. We believe the model uses the radius there to separate the classes.

The link between stability and performance is in Table B of the same reply. When the radii stay within the accurate range for single precision, the ambient and polar forms give the same accuracy (when no other layer or component is added). However, when the radius drifts, the ambient form fails or needs a cap but the polar form keeps its accuracy. This happened in the case of learnable curvature without warm-up, the doubled learning rate search, the bfloat16 rows and the ViT on ImageNet-1k, where two of three ambient attempts crashed and the third reached 65.23 against 71.68 in polar form. In the U-Net, where most output radii pass 8.3, moving the same layers to polar form and removing the caps gains 0.03 to 0.05 Dice on four of the five datasets and ties on the fifth. In the genome CNN, which has the largest classifier radii, the polar form is 3.4 to 9.2% above the same network in ambient form on all five datasets.

We do believe that the effect of the embeddings' radial distribution does need to be studied further to see how we could separate the decision boundaries further through specific loss functions.

References

  1. Yang et al. Hypformer: Exploring efficient transformer fully in hyperbolic space. KDD 2024.
  2. He et al. HELM: Hyperbolic large language models via mixture-of-curvature experts. NeurIPS 2025.
  3. Mishra et al. Hyperbolic U-Net for robust medical image segmentation. MIDL 2026.
  4. Khan et al. Hyperbolic genome embeddings. ICLR 2025.
  5. van der Klis et al. Fast and geometrically grounded Lorentz neural networks. arXiv:2601.21529.
  6. Bdeir et al. Fully hyperbolic convolutional neural networks for computer vision. ICLR 2024.
  7. Shimizu et al. Hyperbolic neural networks++. ICLR 2021.
  8. Chen et al. Fully hyperbolic neural networks. ACL 2022.

I appreciate the authors' detailed replies, which address my concerns. My assessment of the paper remain positive and have updated my score accordingly

Thank you very much, we really appreciate the time you put into the review and the suggestions, the work is much better for it.

Reviewer_HKB5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Polar-ViT, a fully hyperbolic vision Transformer based on a polar-coordinate representation of the Lorentz model. The authors point out that existing Lorentz operations in ambient coordinates suffer from severe floating-point cancellation, which particularly affects hyperbolic distance computation and weighted centroid aggregation. To address this, the paper derives more stable polar-coordinate formulations and further designs a polar fully connected layer, a relative positional encoding based on horospheres, and a residual connection based on Lorentz boosts.

Experiments show that the method achieves good performance on CIFAR-100, Tiny-ImageNet, and ImageNet-1k, and demonstrates some generalization ability on fine-grained image classification and medical image segmentation.

Critical strengths
  1. The numerical analysis is clear and practical. The paper analyzes the numerical issues very explicitly, and insufficient numerical stability is an inherent and well-known weakness of hyperbolic space, so the paper is well motivated.
  2. The experimental performance is competitive. Polar-ViT clearly outperforms the reported Euclidean and hyperbolic baselines on both CIFAR-100 and Tiny-ImageNet. The method design is fairly complete, and the ablation studies are relatively thorough. Rather than modifying only a single hyperbolic operation, the paper proposes corresponding designs for the fully connected layer, attention aggregation, positional encoding, and residual connection.
Critical weaknesses

There is insufficient causal evidence that the improved numerical stability is what drives the gains in downstream performance. The paper does not report the radius distribution of the features at each layer during actual training and inference.

The experimental scale is too limited to adequately demonstrate the importance of numerical stability.

Questions for authors

What are the radius distributions of the residual stream, q/k/v, and the centroid outputs of Polar-ViT on CIFAR-100 and ImageNet-1k, both during training and after convergence? What fraction of them exceeds the single-precision failure radius of 8.3 for the ambient operations?

With the final architecture unchanged, could the authors compare the accuracy obtained with ambient operations (in single and double precision) against that obtained with polar operations? Could the authors also split the second row of Table 5 into two steps, one that switches only to the polar operations and one that switches only the gyro layer normalization, and state at which step the readout changes from the class token to the centroid?

Could the authors report the training throughput of Polar-ViT-S and a Euclidean ViT-S in a speed test on the large-scale ImageNet-1k dataset, and provide a compute-matched comparison?

How does the radial residual update in Eq. (12) ensure that the output radius remains non-negative? Can the authors provide an explicit mathematical guarantee, or describe how this boundary case is handled in the implementation?

4 comments

Thank you for the review. We agree that documenting the radius of the embeddings improves the paper. We measured it in the existing experiments and ran additional models, settings and datasets to show where the stability of the polar forms matters.

1. Radii and effect

We merged the reply to Weakness 1 and Question 1 since they are closely related. Table A reports the mean and the maximum radius of the layer outputs by depth on trained checkpoints. It covers the ViT classifiers, the hyperbolic U-Net of Appendix F, a hyperbolic ResNet-18 that follows [1], a hybrid model (Euclidean ResNet-18 feeding a hyperbolic head), and the hyperbolic genome CNN of our reply to Reviewer KQFU, which we found to reach large radii most consistently.

Table A. Mean / maximum radius of the layer outputs by depth. The radius is the hyperbolic distance from the origin, measured before any clip or rescale.

modelstemearlymiddlelast block or stagelargest radius in the model (site)
ViT-Tiny, CIFAR-100 (residual stream)2.75 / 3.83block 3: 3.01 / 6.46block 6: 3.61 / 7.13block 9: 4.05 / 7.1112.28 (output of the last MLP)
ViT-Small, ImageNet-1k (residual stream)4.06 / 5.08block 4: 4.83 / 7.23block 8: 6.39 / 9.88block 12: 6.40 / 9.1810.60 (stream after block 11)
Hyperbolic ResNet-18, CIFAR-100 (stage outputs)3.26 / 5.92stage 1: 4.81 / 6.92stage 2: 6.51 / 8.34; stage 3: 8.44 / 9.77stage 4: 8.89 / 10.8711.54 (stage 4)
Hyperbolic U-Net, ISIC 20180.91 / 3.18encoder levels: 1.14 to 1.93 / 4.13decoder levels: 2.14 to 2.67 / 8.11last decoder level: 5.63 / 11.4928.99 (output layer, mean 18.42)

Radius at the classifier head (5 % / 50 % / 95 % quantiles).

modellast layer before the headhead input
Euclidean ResNet-18 with a hyperbolic MLR head, CIFAR-100feature norm 7.9 / 10.6 / 14.91.84 / 2.12 / 2.45
the same with the exponential map before the headfeature norm 8.4 / 9.4 / 10.83.27 / 3.66 / 4.18
Hyperbolic CNN, LINEsconv blocks, median 0.73.6 / 7.5 / 10.7
Hyperbolic CNN, hAT-Acconv blocks, median 1.62.6 / 10.8 / 14.1
Hyperbolic CNN, SINEsconv blocks, median 1.84.6 / 11.8 / 15.0

We see consistent patterns across the models:

  • The radius increases with depth. On ImageNet-1k the mean of the residual stream goes from 4.1 after the embedding to 6.4 in the last block, and in the ResNet-18 from 3.3 after the stem to 8.9 after the last stage.

  • The largest radii sit at the output. In the hyperbolic U-Nets 91% to 98% of the output-layer radii exceed 8.3 and up to 71% exceed 17.3.

  • The radius grows during training. In the CIFAR-100 ViT the largest radius rises from 8.0 at epoch 9 to 12.3. In the ImageNet ViT-Small the stream maximum rises from 7.1 at epoch 9 to 10.6 at epoch 89.

  • The attention centroid stays below radius 7 and the query and key slices below 8 in the CIFAR-100 and ImageNet ViT-Small models. However, some of the residual-stream radii of the ImageNet ViT-Small surpass that limit (upto 10.6). This is similar for CIFAR-100 ViT where around 5% of the MLP output radii exceed it as well (up to 12.28).

The goal of the polar coordinates is to prevent instability since a small fraction above the limit can lead to model failure. This is why the ambient models need radius clipping, gradient clipping, or lower learning rates. Clips are even set by default in the Geoopt library at the output of the projection maps and FC layers (of [2]), they are also often applied at different parts in the model or optimizer. Using polar representations allows us to remove these clips or greatly reduce them. This gives us stable training without crashes and it improves accuracy when the clip restricts the embeddings like on ImageNet-1k (Table B). In the actual ViT experiments, the amount of radii exceeding the limit is small so removing the clipping affects training stability more than accuracy. It also allows us to learn curvature more freely and learn in bfloat16.

For example, in ImageNet-1k the ViT could not be trained reliably without the polar form. Two of our three attempts to train the same ViT-Tiny in ambient arithmetic crashed with non-finite values at different epochs despite standard radius clips. The non-finite values came from the layer norms and the optimizer step (failure sites listed in Appendix A). The one ambient run that finished reached 65.23, below both Polar-ViT (71.68) and the Euclidean ViT (68.19). These failures are why we developed the polar form which is able to train to the end.

We will add these tables, the full distributions at convergence and the training trajectories to the paper, and state explicitly in the introduction that the polar form is what allows us to train without radius clips, in bfloat16 and with a learnable curvature (rather than being the main accuracy driver in ViT).

2. "could the authors compare the accuracy obtained with ambient operations"

When the radius of the embeddings does not pass the largest radius the precision can represent, the polar and the ambient arithmetic give the same result within seed noise. This is expected since they are two forms of one function. The polar form helps training stability and it also helps accuracy in the cases where clips restrict the representation too severely. Table B shows both, we also run U-Net in double precision to test the difference (in double precision the ambient radius limit moves from 8.3 to 18.4 (Proposition 1) and it agrees with the polar form operations as stated in the appendix). Fp64 however required around 2x the memory and 4x the time.

Table B. Ambient against polar arithmetic, same model and seed

modelsettingambientpolar
ViT-Tiny, CIFAR-100recipe of Table 179.8379.73
learnable curvature, with warm-up79.0680.17
learnable curvature, no warm-upnon-finite in epoch 179.87
ViT-Tiny, ImageNet-1krecipe of Table 2, same model in ambient arithmetic with the standard clipsnon-finite in 2 of 3 attempts, 65.23 in the third71.68±0.11 (Table 2)
ResNet-18 of [1], CIFAR-100authors' recipe, fp3276.2676.80
double lr tried during hp search in fp32, seeds 0 / 175.99 / non-finite76.68 / 75.93
fp32 checkpoint evaluated in bfloat1629.8776.43
trained in bfloat16, 200 epochsnon-finite in epoch 173.87
Hyperbolic CNN, genome (MCC × 100)LINEs74.76±5.4983.93±0.29
SINEs89.22±3.8094.73±0.66
hAT-Ac90.21±0.5293.59±0.63
processed pseudogenes65.40±2.5571.53±1.97
unprocessed pseudogenes59.51±0.5362.93±1.26

As an extra note, cancellation in the operations in ambient form returns points at inflated radii which we no longer saw in the polar model. We keep the clips in ambient models to keep outputs inside the range where their operations are safe, and on ImageNet-1k even that was not enough (new failure points kept appearing). The UNet segmentation model also has relatively large radii and we ablate it in table B below which also shows clear polar benefits

Table B, U-Net of Appendix F. Test Dice, same seed

modelprecisionISIC 2016ISIC 2018Kvasir-SEGSANETBUSI
ambient, with capsfp320.920.810.790.730.71
ambient, caps removedfp640.910.840.840.760.73
polar arithmetic, caps removedfp320.920.850.840.780.74
+ polar fcfp320.930.890.860.810.78

In the rebuttal to Reviewer KQFU we run additional experiments to test the performance of polar in other settings and see similar results.

We add these to the appendix and retest and fix Table 5 with the separate modes. Changing the model to polar only boosts the performance from 73.99 to 74.51, the readout from the class token to the centroid takes it to 75.31, and the gyro layer norm takes it to 76.27.

3. Training throughput and compute

You are right and this is a weakness of hyperbolic methods in general as evident in the literature. A large part of this gap is the state of the implementation since Euclidean operations benefit from highly optimized CUDA implementations where hyperbolic operations still rely on simple operations (like the convolution unfold approach). We show that optimization already helps, as shown by the fused kernel in Appendix E.

Table C. Throughput of ViT-Small, fp32, batch 128 (100 for ImageNet-1k), one RTX PRO 6000.

inputtraining, Euclidean ViTtraining, Polar-ViT
CIFAR-100, 64 tokens46 s/epoch (2.9 GB)84 s/epoch (7.5 GB)
Tiny-ImageNet, 64 tokens92 s/epoch (2.9 GB)169 s/epoch (7.6 GB)
ImageNet-1k, 196 tokens1316 s/epoch (6.7 GB)2534 s/epoch (20.6 GB)

4. Non-negativity of the radius in Eq. 12 (Question 4)

Eq. 12 is evaluated as x' = (√k cosh a', √k sinh a' u'). Since cosh is even and sinh is odd, a negative a' is the point at distance |a'| from the origin in the direction −u'. The output is on the hyperboloid for every real a', and no clamp is applied. When the pair (r, u) is passed on, it is stored with r ≥ 0 and the direction flipped. We will state this below Eq. 12.

References

  1. van der Klis et al. Fast and geometrically grounded Lorentz neural networks. arXiv:2601.21529.
  2. Chen et al. Fully hyperbolic neural networks. ACL 2022.

Thanks for the additional experiments and the ablation. However, the response reinforces my primary concerns. The original paper motivates Polar-ViT primarily through floating-point cancellation errors at large radii (> 8.3), yet the authors' empirical measurements show that in the ViT models on both CIFAR-100 and ImageNet-1k, the radii of the query and key slices consistently remain below 8 and the attention centroid stays below 7. Consequently, when fixing the final architecture and only switching the operations, ambient coordinates (79.83%) and polar coordinates (79.73%) ; the decomposed ablation further confirms that switching solely to polar operations only marginally improves the baseline from 73.99% to 74.51%, whereas the primary gains stem from architectural designs such as centroid pooling, gyro layer normalization.

The authors also acknowledge in their response that the polar form is not the main accuracy driver in ViT, shifting the emphasis instead to clip-free and low-precision training stability while relying primarily on non-Transformer models such as U-Net and the genome CNN to demonstrate accuracy gains at large radii. This shows a misalignment with the paper's core narrative centered on the large-radius representational capacity of a hyperbolic Vision Transformer, which calls for a substantial restructuring of the paper's motivation and main storyline rather than minor revision. Therefore, I retain my rating.

Thank you for the follow-up. We agree with the numbers you quote, and that on CIFAR-100, with the default recipe, the accuracy gain of the ViT comes from the components. However, we do not think this makes the cancellation analysis irrelevant to the ViT.

1. The fixed-architecture comparison has more than one ViT row

ViT-Tiny, same architecture and seedambient, standard clipspolar, no clips
CIFAR-100, recipe of Table 179.8379.73
CIFAR-100, learnable curvature, with warm-up79.0680.17
CIFAR-100, learnable curvature, no warm-upnon-finite in epoch 179.87
ImageNet-1k, recipe of Table 2non-finite in 2 of 3 attempts, 65.23 in the third71.68±0.11, all 3 seeds

On ImageNet-1k only the arithmetic changes, and the ambient ViT either crashes or ends more than 6% lower, 3% below the Euclidean ViT. Your Weakness 2 was that the scale was too limited to show the importance of numerical stability. Imagenet is the larger scale dataset, and the effect appears there in a ViT despite the clips being in place. In terms of more experimental settings the same is shown in the experiments suggested by KQFU, where genome increased performance 4 to 10% and graphs as well.

2. Additionally, the 79.83 against 79.73 row is a control experiment

The ambient row keeps the radius clips and since a clip holds the model inside the range where the two forms are the same function, equal accuracy is the expected. It does not show that the polar form is unnecessary, because the clip is itself the workaround for the ambient errors. The polar form is what lets us remove the clip. On CIFAR-100 with the default recipe the clip is enough, however when we add a learnable curvature or when we run it on ImageNet-1k, it is not.

3. The components and the other models

Two of the components exist to remove a radius bound. The Polar-FC learns the radius as its own unbounded channel and we show this in in Table 6 where the sigmoid-bounded radius of Chen et al. reaches 74.04, a hard bound diverges, and Eq. 8 reaches 78.45. The boost residual of Eq. 12 replaces residuals that require a radius cap.

The U-Net, genome, graph and ResNet experiments were added at the request of Reviewer KQFU. The U-Net is also the controlled test of your Weakness 1 where when the layers are fixed, removing the caps needs either double precision or the polar form, and both give the same gain (ISIC 2018: 0.81 with caps, 0.84 in fp64, 0.85 in polar fp32). The remaining experiments show large gains for polar over ambient.

4. Radii below 8.3 are not free of cancellation

  • 8.3 is where the error reaches the size of the distance itself (Proposition 1). Below it the error on cosh⁡d−1\cosh d - 1 is εe2a/4\varepsilon e^{2a}/4, which at radius 7 is 0.018, the entire value for two points at distance 0.19. The weight-decay example of Section 3.1 is at radius 8, the optimization step should return radius 7.985 but instead it returns 8.14.
  • The ImageNet-1k failures came from the layer norms and the optimizer step, which act on the residual stream and the parameters, not on the query and key slices. In the ImageNet-1k ViT-Small the stream reaches radius 10.6 and 1.5% of its radii are above 8.3. One invalid value is enough to cause an error in the run, so the tail is also important to target not just the mean.

5. The storyline

The introduction already justifies the motivation in these terms: existing models "confine their embeddings to a small radius through strict normalization ... norm clipping ... and frequent rescaling", the failure is "silent" and leads "to training degradation or divergence", and the polar form "lessens the need for normalization, clipping and rescaling components". Table B verifies this claim. The abstract also attributes the accuracy to the combination of the components and not to the polar arithmetic alone "Combining these components... substantially improves performance over Euclidean and hyperbolic baselines".

However, we do agree that the abstract and the first paragraph of the introduction put more weight on reaching the far regions of the space which is more effective in other settings. We will reword both to state that the polar form is what allows training without radius clips, at ImageNet-1k scale, with a learnable curvature and in bfloat16, and that on the small datasets the gain of the ViT comes from the components. We will also add Tables A, B, and the new experimental settings to Section 5 and the split row to Table 5. The method, the propositions and the existing experiments stay as they are, so we see this as a small change of framing in the abstract and introduction plus one added subsection.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.