- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper examines training instability in hyperbolic Transformers as model size increases. The authors identify two issues with existing hyperbolic residual connections: (1) large layer norms are required to induce directional changes in the residual stream, and (2) residual connections can be overly sensitive to unusually large layer outputs. They propose TangentAttnRes, which performs attention-weighted aggregation in the tangent space, and provide theoretical results supporting its stability. The method is evaluated on HELM-D, hyperbolic BERT, and HyTE-FH across multiple model scales. Experiments on HELM-D, hyperbolic BERT, and HyTE-FH show that TangentAttnRes generally improves perplexity or GLUE performance, especially at larger scales where the original residual connections become unstable.
Critical strengths
-
The paper gives a clear explanation of where the instability comes from. Rather than proposing another residual variant without much motivation, the authors identify two specific issues and connect them to the growth of representation norms and embedding dimensions. This makes TangentAttnRes's design fairly easy to understand.
-
The results are consistent across different models and scales. TangentAttnRes works well for both decoder- and encoder-style hyperbolic models, with larger gains at the larger tested scales. The 1.483B-parameter hyperbolic BERT experiment is particularly helpful because it shows that the results are not specific to HELM-D. The attention ablation also provides some evidence that learned weighting is useful.
Critical weaknesses
-
The scaling claim is somewhat broader than the experiments support. The largest decoder model is 259M parameters and the largest encoder model is 1.483B parameters. These experiments show that TangentAttnRes improves stability at the tested scales, but they do not yet establish that it will continue to work for frontier-scale hyperbolic LLMs.
-
The role of the residual connection could be isolated more carefully. The experiments show a clear relationship between norm growth, the residual formulation, and training collapse, but it is not completely clear how much of the improvement comes specifically from the residual design versus differences in optimization.
Questions for authors
- How sensitive are the scaling results to the learning rate and other optimization choices?
- It would enhance readability if the citations were colored in different colors.
Response to Reviewer
Thank you for your valuable comments. We have run some additional experiments based upon your recommendations. We have also made the citations more readable per your suggestion.
Model Scaling
We have scaled our HELM-D model further to an 8x configuration of 3,120 embedding dimensionality, 12,480 feedforward width, and 48 attention heads with TangentAttnRes as our residual connection to build a 1.74B model and find that it improves perplexity over the 2x variant by more than 40% across all held-out corpora shown in Table 1. We scale the depth of the model further from 6 layers to 34 in order to build a 6.09B model and find that the model continues to train stably with a 48.16% improvement over the 2x variant. This model size is, to our knowledge, more than six times larger than any previous fully hyperbolic Transformer in the literature and approaches the limit of what our resources allow us to train; however, we see no evidence that the model should not be able to scale further and expect that it would continue to improve in performance.
HELM-D Sensitivity to Optimization Choices
We have tested whether our results are robust to our optimization choices, such as a learning rate of 4e-4 and warmup of 3%. We perform a sweep of learning rates and warmups and find that LResNet 2x still collapses with learning rates of 1e-4 and 2e-4 and warmup ratios of 5% and 10%. We find that TangentAttnRes 2x remains stable with warmup ratios of 1% or 0% and learning rates of 8e-4, but that it collapses when we scale the learning rate to 1.6e-3. This collapse is not wholly unexpected, as a fourfold increase in learning rate should create some instability.
† Training failed to converge.