- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
- Hyperbolic LLMs leverage non-Euclidean, hierarchical structure of tokens to improve performance.
- Fully hyperbolic LMs like HELM add hyperbolic operations throughout the model, which results in significant computational overhead
- Authors propose Lorentz Busemann Language Models (LBLM), which retains standard Euclidean computations up until the output layer, which incorporates learnable hyperbolic token embeddings.
- The authors also introduce Curvature Learning Riemannian Adam (CRAdam) to keep the embeddings in hyperbolic space and preserve the moment norm across curvature changes (in a computationally efficient manner).
- Across rather small model sizes (0.3-1.4B), LBLM improves mean accuracy by 7.8% relative to Euclidean baseline trained with same recipe, with at most 5.0% training overhead.
- Compared to HELM, LBLM attains higher accuracy alongside 5.3x higher training throguhput and 2.8x higher inference throughput.
Critical strengths
- The core idea seems to be mathematically well-motivated. Experiments are comprehensive.
- The CRAdam ablation justifies the moment transition effectively; comparing CRAdam with CRAdam without MT helps connect the final optimization technique with the preceding geometric justifications.
Critical weaknesses
- The results overall feel disconnected from the mathematics. The hierarchical structure of tokens is supposed to be a primary motivation for doing operations in hyperbolic space rather than Euclidean, and indeed it seems like the LBLM outperforms both baselines, but it is not clear why. As the authors themselves say, the difference in performance ought to be attributed to method and recipe jointly, rather than architecture alone. In part this is unavoidable (due to the necessity of training each model with its own optimal recipe), but has the secondary effect of obfuscating the actual impact the non-Euclidean geometry had. For example, if there were some intervention or figure or something that showed definitively the hierarchical structure being leveraged, that would be interesting.
- Overall the paper is somewhat well written but not sufficiently clear. The presentation is too dense. For exmple, Figure 1 and 2 have an incredible amount of information packed onto it, and overall the prose is dense and hard-to-follow. It might be useful to add a notation and acronym table because it’s quite hard to keep track of all of them. Generally, the paper lacks explanations that would make it easier to understand why certain decision are being made and how do they relate to standard transformers.
Questions for authors
- What's the computational and memory complexity of the standard euclidean vs whats discussed in this work? maybe the question doesnt matter because the change is only to the last layer?
- Is the proposal better than standard transformers when it comes to test perplexity vs train-flop efficiency? (See scaling laws literature)
- The promise of hyperbolic spaces is that they are suitable for embedding data with hierarchical structure. Does one observe hierarchical structure emerge from the trained models?
- The experiments investigate the effect of depth. But what about the effect of width? Wouldn’t hyperbolic space be less efficient when width is higher (since there’s less interference between words in the vocabulary)? Or is the depth and width simultaneously changing in these experiments?
- Any guarantees for the optimization? How did you derive these steps?