acceptodds
Under review as a conference paper at ICLR 2027

Hyperbolic Residual Connections for Stability at Scale

Abstract

Large language models (LLMs) have become highly capable at modeling text, largely due to systematic gains from increasing model size and training scale. More recently, LLMs with hyperbolic embedding spaces and operations have shown strong performance by modeling natural language on manifolds that better reflect its latent hierarchical structure. However, hyperbolic models have not been scaled to the same extent as their large pre-trained Euclidean counterparts, in part due to difficulties maintaining training stability as model size increases. In this work, we identify two instabilities of existing hyperbolic residual connections: requiring exceedingly large layer embedding norms to change embedding directions and not being robust to larger layer outputs. We propose a new hyperbolic residual connection, TangentAttnRes, that addresses this instability through two features: (1) aggregation in tangent space, which addresses the first instability by reducing the norm required to make directional changes to the residual vector; and (2) attention-based aggregation, which addresses the second instability by selectively weighting past layer outputs to dampen the impact of unstable layers. We apply our method to multiple hyperbolic LLMs and language Transformers across diverse benchmarks, demonstrating that it improves performance over the existing residual connections. This is especially the case as we scale model parameters, where TangentAttnRes shows continually improved performance while the same scaling causes models with prior methods to show negligible gains or even collapse to near-random performance. Our findings demonstrate that TangentAttnRes is crucial for achieving further performance gains as hyperbolic LLMs scale, presenting a path toward models that can compete at the frontier.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
hyperbolic geometry, residual connections, large language models, training stability

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_DTFKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines training instability in hyperbolic Transformers as model size increases. The authors identify two issues with existing hyperbolic residual connections: (1) large layer norms are required to induce directional changes in the residual stream, and (2) residual connections can be overly sensitive to unusually large layer outputs. They propose TangentAttnRes, which performs attention-weighted aggregation in the tangent space, and provide theoretical results supporting its stability. The method is evaluated on HELM-D, hyperbolic BERT, and HyTE-FH across multiple model scales. Experiments on HELM-D, hyperbolic BERT, and HyTE-FH show that TangentAttnRes generally improves perplexity or GLUE performance, especially at larger scales where the original residual connections become unstable.

Critical strengths
  1. The paper gives a clear explanation of where the instability comes from. Rather than proposing another residual variant without much motivation, the authors identify two specific issues and connect them to the growth of representation norms and embedding dimensions. This makes TangentAttnRes's design fairly easy to understand.

  2. The results are consistent across different models and scales. TangentAttnRes works well for both decoder- and encoder-style hyperbolic models, with larger gains at the larger tested scales. The 1.483B-parameter hyperbolic BERT experiment is particularly helpful because it shows that the results are not specific to HELM-D. The attention ablation also provides some evidence that learned weighting is useful.

Critical weaknesses
  1. The scaling claim is somewhat broader than the experiments support. The largest decoder model is 259M parameters and the largest encoder model is 1.483B parameters. These experiments show that TangentAttnRes improves stability at the tested scales, but they do not yet establish that it will continue to work for frontier-scale hyperbolic LLMs.

  2. The role of the residual connection could be isolated more carefully. The experiments show a clear relationship between norm growth, the residual formulation, and training collapse, but it is not completely clear how much of the improvement comes specifically from the residual design versus differences in optimization.

Questions for authors
  1. How sensitive are the scaling results to the learning rate and other optimization choices?
  2. It would enhance readability if the citations were colored in different colors.
1 comment
AuthorsOpenReview ↗

Response to Reviewer

Thank you for your valuable comments. We have run some additional experiments based upon your recommendations. We have also made the citations more readable per your suggestion.

Model Scaling

We have scaled our HELM-D model further to an 8x configuration of 3,120 embedding dimensionality, 12,480 feedforward width, and 48 attention heads with TangentAttnRes as our residual connection to build a 1.74B model and find that it improves perplexity over the 2x variant by more than 40% across all held-out corpora shown in Table 1. We scale the depth of the model further from 6 layers to 34 in order to build a 6.09B model and find that the model continues to train stably with a 48.16% improvement over the 2x variant. This model size is, to our knowledge, more than six times larger than any previous fully hyperbolic Transformer in the literature and approaches the limit of what our resources allow us to train; however, we see no evidence that the model should not be able to scale further and expect that it would continue to improve in performance.

ModelLayersParamsWikiText (PPL)LAMBADA (PPL)C4 (PPL)OpenWebText (PPL)
TangentAttnRes 1x6115M105.7252.5163.5128.9
TangentAttnRes 2x6259M59.6164.3104.579.6
TangentAttnRes 4x6640M41.9118.776.257.1
TangentAttnRes 8x61.74B31.895.260.944.5
TangentAttnRes 8x346.09B29.487.955.940.6

HELM-D Sensitivity to Optimization Choices

We have tested whether our results are robust to our optimization choices, such as a learning rate of 4e-4 and warmup of 3%. We perform a sweep of learning rates and warmups and find that LResNet 2x still collapses with learning rates of 1e-4 and 2e-4 and warmup ratios of 5% and 10%. We find that TangentAttnRes 2x remains stable with warmup ratios of 1% or 0% and learning rates of 8e-4, but that it collapses when we scale the learning rate to 1.6e-3. This collapse is not wholly unexpected, as a fourfold increase in learning rate should create some instability.

ModelParams1e-4 (final loss)2e-4 (final loss)4e-4 (final loss)8e-4 (final loss)1.6e-3 (final loss)
TangentAttnRes 1x115M4.5703.8383.4253.1943.053
LResNet 1x115M5.0894.8754.0527.736†8.485†
TangentAttnRes 2x259M3.6503.2122.9713.0287.707†
LResNet 2x259M7.702†7.694†7.934†7.903†7.674†

† Training failed to converge.

MethodWarmupMin loss (update)Loss at 1000 stepsMax grad norm
LResNet 2x05.11 (515)8.556.3×10116.3\times10^{11}
LResNet 2x0.014.99 (869)7.806.8×1056.8\times10^{5}
LResNet 2x0.035.36 (637)8.071.0×1081.0\times10^{8}
LResNet 2x0.056.95 (132)7.814.8×1094.8\times10^{9}
LResNet 2x0.17.19 (220)7.804.5×10104.5\times10^{10}
TangentAttnRes 2x03.50 (887)3.633.7
TangentAttnRes 2x0.013.51 (887)3.647.2
TangentAttnRes 2x0.033.49 (887)3.613.7
TangentAttnRes 2x0.053.53 (887)3.654.9
TangentAttnRes 2x0.13.65 (887)3.744.1
Reviewer_K6dwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies two potential sources of instability in existing hyperbolic residual connections: excessive norm growth and sensitivity to large inputs. The authors propose TangentAttnRes, combining tangent-space aggregation with attention-weighted residuals. They provide theoretical results and demonstrate improved training stability and performance across several hyperbolic Transformer architectures.

Critical strengths
  • Provides theoretical results motivating the proposal. Mix of theory and empirics makes for a great read.
  • The proposal achieves improvements across multiple architectures and model scales, with little training-time overhead.
  • Eq. 4 is very intuitive. Basically applying log map, add residual, and then exp map.
  • Theorems are all rather clean.
Critical weaknesses
  • I am not sure I understand the logic in 4.1.1. Hyperbolic models have a residual that scales like r + log sin theta, whereas standard models have a residual that scales like r*sin(theta). For standard models, the sin theta makes the residual smaller, which is why we make the embeddings scale like sqrt(d)~r (not clear why that fixes the scaling?). Hyperbolic models, on the other hand, don't have the sin theta, which is why the scaling ought to be different. I don't quite see the exact details of this appearing in the subsection and so the whole claim remains rather unclear and vague. For example, how should the norm of the residual grow for hyperbolic and regular models and why?
  • Evidence for instability 2 is unclear: Proposition 3 establishes a worst-case upper bound on the norm, but there is no empirical measurements showing that this actually happens in practice.
  • The results would be stronger if they form in the form of a scaling law experiment. Because then the trend with scale would become clearer.
  • Table 1 has rather small model scales but perhaps those are the biggest the authors can run?
Questions for authors
  • In what sense does proposition 4 solve the high norm inputs issue?
  • Why train Bert when it’s so outdated? Is there some specific reason to believe it would shine when using hyperbolic geometry?
  • How do normalization layers affect the propositions in the paper? Namely, if the concern is activation magnitudes growing too large, don't normalization layers address it?
4 comments
AuthorsOpenReview ↗

Response to Reviewer [1/2]

Thank you for your insightful comments. We have taken them into consideration and made some additions to the work.

Examples and Explanation of Instability 1

To address the reviewer’s concern about instability 1, we would like to clarify that we are not making the point that the growth of norm with dimension fixes the scaling in any way in the standard (Euclidean) case. Instead, our aim was to illustrate how traditional hyperbolic residual connections require layer outputs to grow more strongly with the growth of their input layers (which naturally become larger as embedding dimension increases). For example, the table below shows that for a layer input with a distance to origin of 100, the Euclidean lower bound for layer output size to create a given angular change grows vastly smaller as that angle shrinks, while the equivalent hyperbolic lower bound decreases much more slowly. If the necessary norm of the layer output grew for hyperbolic models as r*sin(theta), just as their Euclidean counterparts do and TangentRes allows for, then this would resolve the issue.

Angular change θ\theta (radians)Euclidean boundHyperbolic boundHyperbolic bound as % of input radius
0.011.000095.394895.39%
0.0010.100093.092293.09%
0.00010.010090.789790.79%

Empirical Validation of Proposition 3 and Motivation for Proposition 4

To demonstrate the importance of Proposition 3, we have performed further experiments by running inference on the test set of HellaSwag for the TangentRes 2x HELM-D model. We found that it suffers from residual connection outputs having a higher L2 norm than each of its inputs and even the sum of both inputs’ norms in 75% and 64% of token-level residual applications, respectively. Proposition 4 prevents this from happening in all cases with LAttnRes and TangentAttnRes. This does not directly solve the high-norm inputs issue, but the attention weights help with this in practice since they downweight high-norm inputs as seen in Figure 6.

Output L2 norm exceeds…TangentRes 2xTangentAttnRes 2x
each of its inputs74.8%0%
the sum of its inputs63.8%0%

Evidence for Instability 2

To address the reviewer’s concern, we point to the results found in Figure 4. Particularly, they showcase empirical evidence for instability 2 in that LResNet is susceptible to large outputs since increasingly large outputs dominate an increasingly larger amount of the residual stream, which is not the case for LAttnRes.

AuthorsOpenReview ↗

Response to Reviewer [2/2]

Model Scaling

We have scaled our HELM-D model further to an 8x configuration of 3,120 embedding dimensionality, 12,480 feedforward width, and 48 attention heads with TangentAttnRes as our residual connection to build a 1.74B model and find that it improves perplexity over the 2x variant by more than 40% across all held-out corpora shown in Table 1. We scale the depth of the model further from 6 layers to 34 in order to build a 6.09B model and find that the model continues to train stably with a 48.16% improvement over the 2x variant. This model size is, to our knowledge, more than six times larger than any previous fully hyperbolic Transformer in the literature and approaches the limit of what our resources allow us to train. However, we see no evidence that the model should not be able to scale further and expect that it would continue to improve in performance.

We also formulate our results as a scaling law experiment and plot the improvement of HELM-D TangentAttnRes with respect to our scaling in Figure A3. This shows a clean relationship between train loss and embedding dimension as we scale our model.

ModelLayersParamsWikiText (PPL)LAMBADA (PPL)C4 (PPL)OpenWebText (PPL)
TangentAttnRes 1x6115M105.7252.5163.5128.9
TangentAttnRes 2x6259M59.6164.3104.579.6
TangentAttnRes 4x6640M41.9118.776.257.1
TangentAttnRes 8x61.74B31.895.260.944.5
TangentAttnRes 8x346.09B29.487.955.940.6

Motivation for Training BERT Models

As for the reviewer’s concern about training hyperbolic BERT models, the premise of our inclusion of encoder models is to maintain coverage across different hyperbolic LLM architectures and show that our methods generalize well.

Relevance of Normalization to Our Work

To address the reviewer’s concern, we run additional experiments on normalization. First, we note that normalization modules enforce a unit expected magnitude on each embedding dimension. Hence, as the embedding dimension grows, these modules lead to larger embedding norms as shown in Figure 2. If, instead, we enforce a unit norm on the HELM-D embeddings during normalization so that they train more stably, it comes at a high cost to performance as seen in the Table below. This degradation can be attributed to forcing embeddings to lie exceedingly close to the origin on each dimension. It is also important to note that our propositions and motivation for instability deal with layer inputs and outputs following an attention or feedforward operation, neither of which are normalized before being fed into the residual connection.

ModelNormalizationParamsWikiText (PPL)LAMBADA (PPL)C4 (PPL)OpenWebText (PPL)
LResNet 1xStandard115M350.3365.4311.0242.9
LResNet 1xUnit normalization115M1,902.11,770.11,609.41,368.0
LResNet 2x†Standard259M4,482.15,558.25,089.84,559.8
LResNet 2xUnit normalization259M1,295.81,153.81,057.5900.8

The reviewers have fully addressed my concerns. I retain my acceptance recommendation.

AuthorsOpenReview ↗

Response to Reviewer

We thank the reviewer for reading our rebuttal and for recommending the paper’s acceptance. If any concerns remain that prevent a stronger recommendation, we would be grateful if the reviewer could let us know, and we would be happy to address them before the discussion period closes.

Reviewer_PwN6OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TangentAttnRes, a residual connection designed to improve the training stability of hyperbolic Transformers at scale. The authors first analyze two problems of existing hyperbolic residual connections from a theoretical perspective. (1) To change the direction of the residual vector, a layer may need to produce outputs with large norms. (2) Conventional residual aggregation is not sufficiently robust to abnormally large layer outputs. To address these problems, the authors combine tangent-space aggregation with cross-layer attention residuals. They perform a weighted aggregation of the previous layer outputs in the tangent space at the origin and map the result back to the hyperbolic manifold, which limits the norm growth of the residual stream. The authors conduct experiments on three models, HELM-D, a hyperbolic BERT, and HyTE-FH, showing that the method improves training stability and downstream performance as the model width increases. They also implement fused Triton kernels to reduce the additional computational overhead.

Critical strengths

The paper addresses the stability of residual connections when scaling hyperbolic language models, which is a problem of clear practical significance. Numerical stability has long been a challenge in hyperbolic space. The paper conducts experiments on the decoder model HELM-D as well as on the hyperbolic BERT and HyTE-FH encoders. In particular, in settings where some conventional residual connections collapse during training, TangentAttnRes still trains successfully and maintains reasonable performance, which demonstrates the practical value of the method.

Critical weaknesses

Proposition 5 shows that the TangentAttnRes output is no farther from the origin, in geodesic distance, than the farthest previous layer output. However, this result mainly follows from the norm bound of a convex combination in the tangent space, and it does not guarantee that an individual layer will not produce abnormally large outputs.

In essence, TangentAttnRes maps hyperbolic features to the tangent space at the origin, aggregates them in this Euclidean tangent space with the existing Attention Residuals mechanism, and maps the result back to the hyperbolic manifold through the exponential map. Both tangent-space aggregation (Chami et al., 2019) and cross-layer attention residuals (Team et al., 2026) have already been studied in prior work.

Although the paper is framed around stability at scale, the largest HELM-D decoder in the experiments has only 259M parameters. The largest model, with 1.483B parameters, is a hyperbolic BERT encoder rather than a modern generative large language model.

The necessity of aggregating the outputs of all previous layers is not validated. In Table A1, TangentAttnRes with uniform weights (averaging all log_o h_i) gives nearly the same results as TangentRes (summing them), which indicates that the radius bound from the convex combination brings no benefit. The improvement mainly comes from the learned attention weights, and AttnRes is also effective in Euclidean models. The advantage of TangentAttnRes over TangentRes therefore looks more like a generic effect of Attention Residuals than a resolution of the hyperbolic-specific Instability 2. In addition, the score w_l^T log_o(h_i) grows linearly with the norm, so downweighting large-norm outputs is not guaranteed by the structure. The paper also lacks an ablation that aggregates only the most recent few layers, and it does not report the memory overhead of storing all layer outputs.

Questions for authors

What learning rate, batch size, number of training tokens, sequence length, warmup, gradient clipping, curvature, and numerical precision are used for pretraining? Was the learning rate retuned for the 2x models? Does LResNet 2x still collapse with a smaller learning rate or a longer warmup? How does the training setup differ from the billion-parameter training in the original HELM paper?

The norm bound in Proposition 5 neither limits the norm of an individual layer output nor guarantees that the attention weights suppress abnormal outputs. Could the authors further clarify the connection between this theoretical result and the actual training stability, and provide gradient-stability analyses or tests with abnormal inputs? Could the authors also report the geodesic radii (or space-like norms) of the layer outputs and the residual stream during training?

As the network depth increases, the cross-layer attention must access an increasing number of previous layer outputs. Could the authors provide results on memory usage, inference latency, and mixed-precision training?

5 comments
AuthorsOpenReview ↗

Response to Reviewer [1/3]

Thank you for the careful review and for recognizing the importance of our research questions and contributions. We address your concerns below.

Empirical Validation of Propositions 3 and 5

We believe that the norm bound in Proposition 5 is practically meaningful. We have run additional experiments and found that the TangentRes 2x HELM-D model suffers from residual connection outputs having a higher L2 norm than each of its inputs and even the sum of both inputs’ norms in 75% and 64% of token-level residual applications, respectively, when running inference on the test set of HellaSwag. Reducing these rates to zero, as Proposition 5 guarantees, curtails large outputs that may be numerically unstable from reaching the residual stream.

Output L2 norm exceeds…TangentRes 2xTangentAttnRes 2x
each of its inputs74.8%0%
the sum of its inputs63.8%0%

Relation of Our Methods to Prior Works

Our methods take inspiration from (Team et al., 2026) and (Chami et al., 2019), but we depart from them in a few crucial ways. The original Attention Residuals work normalizes layer outputs prior to computing attention scores, which we refrain from in order to leverage the hierarchical structure uniquely offered by the hyperbolic embedding space that arises when embeddings are allowed to vary in their distance to the origin. To our knowledge, we are also the first to implement attention weights into a hyperbolic Transformer residual connection, and no prior work has combined this with a tangent space aggregation in the residual connection.

Decoder Model Scaling

We have scaled our HELM-D model further to an 8x configuration of 3,120 embedding dimensionality, 12,480 feedforward width, and 48 attention heads with TangentAttnRes as our residual connection to build a 1.74B model and find that it improves perplexity over the 2x variant by more than 40% across all held-out corpora shown in Table 1. We scale the depth of the model further from 6 layers to 34 in order to build a 6.09B model and find that the model continues to train stably with a 48.16% improvement over the 2x variant. This model size is, to our knowledge, more than six times larger than any previous fully hyperbolic Transformer in the literature and approaches the limit of what our resources allow us to train. However, we see no evidence that the model should not be able to scale further and expect that it would continue to improve in performance.

ModelLayersParamsWikiText (PPL)LAMBADA (PPL)C4 (PPL)OpenWebText (PPL)
TangentAttnRes 1x6115M105.7252.5163.5128.9
TangentAttnRes 2x6259M59.6164.3104.579.6
TangentAttnRes 4x6640M41.9118.776.257.1
TangentAttnRes 8x61.74B31.895.260.944.5
TangentAttnRes 8x346.09B29.487.955.940.6
AuthorsOpenReview ↗

Response to Reviewer [2/3]

Stability of TangentAttnRes over TangentRes

To address the reviewer’s concern about TangentAttnRes improving over TangentRes generically rather than having heightened stability, we have run experiments comparing the two methods on HyTE-FH and show loss curves over the first 2959 steps of training in Appendix H. We find that, even on its 1x scale, a TangentRes HyTE-FH model fails to train stably and plateaus at a loss of 7.42 at step 2959 while TangentAttnRes improves smoothly at both 1x and 2x scales to 2.38 and 2.08, respectively, over the same number of steps. This indicates that combining tangent space and an attention-weighted aggregation over all layers provides not only broadly stronger performance than just leveraging tangent space, but also improved stability.

ModelLoss at step 2959
TangentRes 1x7.42 (plateau)
TangentAttnRes 1x2.38
TangentAttnRes 2x2.08

As for the reviewer’s concern that downweighting large-norm outputs is not guaranteed by the structure of LAttnRes and TangentAttnRes, we find that in practice this is an emergent capability learned by the residual connections, as shown empirically in Figure 6 by the negative relationship between the norm of an output and its attention weight under all four of our main LAttnRes and TangentAttnRes configurations.

TangentAttnRes Recent Layers Ablation

We perform an ablation on TangentAttnRes 2x HELM-D models of aggregating only the most recent two or four layers per your recommendation and show the results below. We find that held-out perplexity degrades by 7.7% on average across our four evaluation corpora with only the four most recent layers available and by 16.3% with only two layers accessible. Thus, the ablation causes drops in performance, yet training remains stable. This is an expected result and does not indicate that aggregating previous layers does not provide stability benefits, since leveraging tangent space alone is sufficient for training stable HELM-D 2x models, as seen by the performance of TangentRes.

WikiText (PPL)LAMBADA (PPL)C4 (PPL)OpenWebText (PPL)Average
TangentAttnRes, full access59.6164.3104.579.6
TangentAttnRes, 4 most recent70.1 (+17.6%)167.3 (+1.8%)110.2 (+5.5%)84.3 (+5.9%)+7.7%
TangentAttnRes, 2 most recent72.9 (+22.3%)187.9 (+14.4%)118.8 (+13.7%)91.4 (+14.8%)+16.3%

HELM-D Training Configuration and Sensitivity to Optimization Choices

Our training configurations match those of the original works at all scales except where noted for parameter scaling. For HELM-D models, this was a 4e-4 learning rate, batch size of 4, 4.6B training tokens, 2048 sequence length, 3% warmup ratio, gradient norm clipped to 1, negative curvature of 1, and autocast bf16 (to fp32) with fp32 model weights, gradients, and optimizer state. The billion-parameter model in the original HELM paper is a different model architecture featuring MoE, MLA, and an alternative configuration of shared components, so it is not directly comparable here. We perform a sweep of learning rates and warmups and find that LResNet 2x still collapses with learning rates at 1e-4 and 2e-4 and warmup ratios at 5% and 10%.

ModelParams1e-4 (final loss)2e-4 (final loss)4e-4 (final loss)8e-4 (final loss)1.6e-3 (final loss)
TangentAttnRes 1x115M4.5703.8383.4253.1943.053
LResNet 1x115M5.0894.8754.0527.736†8.485†
TangentAttnRes 2x259M3.6503.2122.9713.0287.707†
LResNet 2x259M7.702†7.694†7.934†7.903†7.674†

† Training failed to converge.

MethodWarmupMin loss (update)Loss at 1000 stepsMax grad norm
LResNet 2x05.11 (515)8.556.3×10116.3\times10^{11}
LResNet 2x0.014.99 (869)7.806.8×1056.8\times10^{5}
LResNet 2x0.035.36 (637)8.071.0×1081.0\times10^{8}
LResNet 2x0.056.95 (132)7.814.8×1094.8\times10^{9}
LResNet 2x0.17.19 (220)7.804.5×10104.5\times10^{10}
TangentAttnRes 2x03.50 (887)3.633.7
TangentAttnRes 2x0.013.51 (887)3.647.2
TangentAttnRes 2x0.033.49 (887)3.613.7
TangentAttnRes 2x0.053.53 (887)3.654.9
TangentAttnRes 2x0.13.65 (887)3.744.1
AuthorsOpenReview ↗

Response to Reviewer [3/3]

Gradient and Embedding Norm Analyses

We additionally perform analyses of the gradient norm throughout training at the 1x and 2x scales for the LResNet and TangentAttnRes HELM-D variants. Plots in Appendix G illustrate the strong instability of LResNet 2x and that LResNet 1x has more variance in its gradient norms throughout training than either of the TangentAttnRes models. Analyses of embedding norms throughout training are already shown in Figures 1, 3, and 5 and demonstrate that at both scales, TangentRes and TangentAttnRes models’ layer outputs mostly decrease smoothly over training while the LResNet 2x model’s increase dramatically over the first quarter of model training.

Memory and Runtime Analysis

At our standard HELM-D 2x configuration in bf16, TangentAttnRes adds 4.9% peak training memory and 6.3% step time over LResNet. We have additionally implemented a memory-saving mode that recomputes layer history in the backward pass, which shrinks peak memory overhead to within 1% of LResNet and with 8.3% step time over LResNet. Appendix F reports a more detailed methodology for the memory-saving mode.

At inference, TangentAttnRes adds 6.6% and 6% latency for the original method and the memory-saving mode, respectively. BF16 is 37–39% faster than fp32 for every method.

Training step time and peak memory at standard 2x HELM-D experimental configuration. Δ is relative to LResNet. Negative memory Δ partly reflects recomputation that LResNet does not use.

MethodBF16 ms/stepBF16 ΔBF16 GBBF16 ΔFP32 ms/stepFP32 ΔFP32 GBFP32 Δ
LResNet198.5+0.0%35.73+0.0%321.1+0.0%37.04+0.0%
LAttnRes, original202.1+1.8%36.80+3.0%325.8+1.5%38.11+2.9%
LAttnRes, memory-saving206.3+3.9%34.80−2.6%330.1+2.8%36.10−2.5%
TangentAttnRes, original211.1+6.3%37.47+4.9%333.2+3.8%38.78+4.7%
TangentAttnRes, memory-saving214.9+8.3%35.49−0.7%339.2+5.6%36.80−0.6%

Inference latency and peak memory. Δ is relative to LResNet.

dtypeBatch size, Sequence lengthMethodGBΔmsΔ
BF164, 2048LResNet7.76+0.0%62.5+0.0%
BF164, 2048LAttnRes, original8.07+4.0%64.1+2.6%
BF164, 2048LAttnRes, memory-saving8.07+4.0%64.0+2.4%
BF164, 2048TangentAttnRes, original8.09+4.3%66.6+6.6%
BF164, 2048TangentAttnRes, memory-saving8.09+4.3%66.3+6.0%
BF161, 512LResNet1.80+0.0%8.0+0.0%
BF161, 512LAttnRes, original1.82+1.1%9.2+14.0%
BF161, 512LAttnRes, memory-saving1.82+1.1%9.3+15.2%
BF161, 512TangentAttnRes, original1.82+1.2%12.6+57.0%
BF161, 512TangentAttnRes, memory-saving1.82+1.2%12.5+56.2%
FP324, 2048LResNet5.33+0.0%94.4+0.0%
FP324, 2048LAttnRes, original5.64+5.8%95.7+1.5%
FP324, 2048LAttnRes, memory-saving5.64+5.8%95.7+1.4%
FP324, 2048TangentAttnRes, original5.66+6.2%98.1+4.0%
FP324, 2048TangentAttnRes, memory-saving5.66+6.2%98.0+3.9%
FP321, 512LResNet1.34+0.0%8.2+0.0%
FP321, 512LAttnRes, original1.36+1.4%10.1+22.5%
FP321, 512LAttnRes, memory-saving1.36+1.4%10.1+23.2%
FP321, 512TangentAttnRes, original1.36+1.6%12.8+55.3%
FP321, 512TangentAttnRes, memory-saving1.36+1.6%12.7+54.7%

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Our Rebuttal and Manuscript Revisions

We thank the reviewers for their careful reviews and the AC for facilitating the discussion. Our work identifies two sources of training instability in hyperbolic residual connections as models grow wider: layers must produce large-norm outputs to change the direction of the residual stream, and the residual stream is sensitive to abnormally large layer outputs. We introduce TangentAttnRes, which aggregates all previous layer outputs with learned attention weights in the tangent space at the origin. We support the design with theoretical results and evaluate it on three hyperbolic architectures (HELM-D, hyperbolic BERT, and HyTE-FH), where it trains stably in settings where existing residual connections collapse. All three reviewers recommend acceptance, noting the clear account of where the instability comes from (DTFK), the clean theoretical results (K6dw), and the consistent gains across models and scales (PwN6, K6dw, DTFK).

The reviewers’ main concerns, and our responses, were:

  • Model scale (PwN6, K6dw, DTFK). We scaled HELM-D with TangentAttnRes from 259M to 1.74B and 6.09B parameters; both train stably, and the 6.09B model reduces perplexity by 48% over the 259M model (Appendix D, Figure A3).
  • Whether gains are hyperbolic-specific or due to optimization (PwN6, DTFK). TangentRes fails to train on HyTE-FH even at 1x while TangentAttnRes trains stably (Appendix H), and LResNet 2x collapses across all tested learning rates and warmups (Appendix F).
  • Link between theory and training behavior (PwN6, K6dw). TangentRes 2x residual outputs exceed their input norms in 74.8% of cases versus 0% for TangentAttnRes, as Proposition 5 guarantees, and learned attention weights decrease as output norm grows (Figure 6).
  • Overhead (PwN6). TangentAttnRes adds 6.3% training step time and 4.9% peak memory, and a memory-saving mode reduces the memory overhead to under 1% (Appendix C.1).

All new results are in the revised PDF, which we will add to the camera-ready version. We thank the reviewers again for their constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.