acceptodds
Under review as a conference paper at ICLR 2027

Lorentz Busemann Language Models for Efficient Hyperbolic Language Modeling

30 views

Abstract

Prior studies suggest that token level structure is hierarchical and non-Euclidean, motivating hyperbolic large language models (LLMs). Fully hyperbolic models such as HELM add hyperbolic operations throughout the Transformer, increasing training and inference costs. Besides, evidence for hyperbolic structure mainly concerns token embeddings and last hidden states, which together determine the logits at the output layer. We therefore propose the Lorentz Busemann Language Model (LBLM), a hyperbolic LLM that retains standard Transformer backbone and introduces learnable hyperbolic token embeddings at the output layer. LBLM uses the Lorentz Busemann Logit (LBL) to score tokens with the Busemann function, treating hidden states as boundary directions and avoiding their exponential mapping into hyperbolic space. To optimize the token embeddings under learnable curvature, we develop Curvature Learning Riemannian Adam (CRAdam), which keeps the embeddings in the hyperbolic space and preserves the moment norm across curvature changes, without repeated logarithmic and exponential maps through the tangent space. Across scales from 0.3B to 1.4B parameters, LBLM improves mean accuracy by up to 7.8% relative to Euclidean baseline trained with the same recipe, with at most 5.0% training overhead. Compared with the fully hyperbolic HELM, which cannot be trained stably under this recipe, LBLM attains higher accuracy with each model trained under its own recipe. Under identical workloads, LBLM achieves at least 5.3 the training throughput and 2.8 the inference throughput of HELM. LBLM offers a simple and efficient hyperbolic LLM, showing that hyperbolic space need not permeate the Transformer backbone to benefit language modeling.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Hyperbolic Language Modeling, Busemann Function, Hyperbolic Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_RWQ5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary
  • Hyperbolic LLMs leverage non-Euclidean, hierarchical structure of tokens to improve performance.
  • Fully hyperbolic LMs like HELM add hyperbolic operations throughout the model, which results in significant computational overhead
  • Authors propose Lorentz Busemann Language Models (LBLM), which retains standard Euclidean computations up until the output layer, which incorporates learnable hyperbolic token embeddings.
  • The authors also introduce Curvature Learning Riemannian Adam (CRAdam) to keep the embeddings in hyperbolic space and preserve the moment norm across curvature changes (in a computationally efficient manner).
  • Across rather small model sizes (0.3-1.4B), LBLM improves mean accuracy by 7.8% relative to Euclidean baseline trained with same recipe, with at most 5.0% training overhead.
  • Compared to HELM, LBLM attains higher accuracy alongside 5.3x higher training throguhput and 2.8x higher inference throughput.
Critical strengths
  • The core idea seems to be mathematically well-motivated. Experiments are comprehensive.
  • The CRAdam ablation justifies the moment transition effectively; comparing CRAdam with CRAdam without MT helps connect the final optimization technique with the preceding geometric justifications.
Critical weaknesses
  • The results overall feel disconnected from the mathematics. The hierarchical structure of tokens is supposed to be a primary motivation for doing operations in hyperbolic space rather than Euclidean, and indeed it seems like the LBLM outperforms both baselines, but it is not clear why. As the authors themselves say, the difference in performance ought to be attributed to method and recipe jointly, rather than architecture alone. In part this is unavoidable (due to the necessity of training each model with its own optimal recipe), but has the secondary effect of obfuscating the actual impact the non-Euclidean geometry had. For example, if there were some intervention or figure or something that showed definitively the hierarchical structure being leveraged, that would be interesting.
  • Overall the paper is somewhat well written but not sufficiently clear. The presentation is too dense. For exmple, Figure 1 and 2 have an incredible amount of information packed onto it, and overall the prose is dense and hard-to-follow. It might be useful to add a notation and acronym table because it’s quite hard to keep track of all of them. Generally, the paper lacks explanations that would make it easier to understand why certain decision are being made and how do they relate to standard transformers.
Questions for authors
  • What's the computational and memory complexity of the standard euclidean vs whats discussed in this work? maybe the question doesnt matter because the change is only to the last layer?
  • Is the proposal better than standard transformers when it comes to test perplexity vs train-flop efficiency? (See scaling laws literature)
  • The promise of hyperbolic spaces is that they are suitable for embedding data with hierarchical structure. Does one observe hierarchical structure emerge from the trained models?
  • The experiments investigate the effect of depth. But what about the effect of width? Wouldn’t hyperbolic space be less efficient when width is higher (since there’s less interference between words in the vocabulary)? Or is the depth and width simultaneously changing in these experiments?
  • Any guarantees for the optimization? How did you derive these steps?
Reviewer_ARYyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the Lorentz Busemann Language Model (LBLM). Instead of changing the entire Transformer, LBLM keeps a standard Euclidean backbone and introduces hyperbolic geometry only at the output layer. It introduces two core ideas: Lorentz Busemann Logits (LBL) and Curvature Learning Riemannian Adam (CRAdam). Experiments on models from 0.3B to 1.4B parameters show the effectiveness of the proposed method.

Critical strengths
  1. The paper is well organized and written.

  2. The proposed method is efficient.

  3. The Riemannian optimization is clear.

Critical weaknesses

I have the following concerns.

  1. Pretraining is limited to small models (e.g., up to 1.4B parameters) trained with relatively small token amount (1.3B to 8.7B tokens). It is not clear that whether the superior performance will be maintained for larger models, such as 7B+ models. This needs to be well justified as it is key to the performance claim of this work.

  2. The authors trained HELM with their own settings. So, the comparison in Table 3 is not that convincing to me.

Questions for authors
  1. Since LBL only operates on the output layer and requires no changes to the backbone, can it be easily adapted to fine-tune existing, open-weight pretrained Euclidean LLMs (e.g., Llama-3 or Qwen-2.5) without retraining from scratch?

  2. In Table 1, the relative gain over the Euclidean baseline drops from +7.8% at 0.3B parameters to +4.0% at 0.5B, and then stays around +5.3% at 0.9B and 1.4B. Do you expect these gains to saturate further on larger models?

  3. How sensitive is the final performance to the choice of when to start updating the curvature (e.g., the 50% vs. 60% training step threshold in Table 8)?

Reviewer_AQL3OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LBLM, a language model that builds on a standard Euclidean Transformer, keeping the output layer hyperbolic. Its key idea is pretty neat. Instead of mapping each hidden state into hyperbolic space with the exponential map, as most hybrid models do, it keeps only the direction of the hidden state and treats it as a point on the boundary of hyperbolic space. Each token is then scored by its Busemann function toward that boundary point. This removes the exp/log maps from the output layer entirely, and the logits come down to a single matrix multiplication plus cheap elementwise operations. It is a clever way to avoid the additional cost associated with additional exponential mapping operations. The second contribution is CRAdam, a Riemannian Adam variant that trains the token embeddings together with a learnable curvature. When the curvature changes, CRAdam keeps the spatial coordinates fixed and rescales the first moment, which avoids transporting every embedding through a log/exp round trip. This is a well-motivated approach.

Experiments use NanoChat models from 0.3B to 1.4B parameters pretrained on ClimbMix. LBLM improves average zero/few-shot accuracy over the Euclidean baseline at all four model sizes and lowers validation BPB, with little extra training cost. It is also several times faster than HELM, a fully hyperbolic LLM. Ablations compare the proposed head against BMLR (another Busemann-based head) and its own Euclidean baseline, and compare CRAdam against Adam and an existing Riemannian optimizer for learnable curvature.

Critical strengths

S1. Avoiding the exp map by using the hidden-state direction as a boundary point. Taking only the direction of the hidden state and treating it as a boundary point means the output layer needs no exp/log maps and does not depend on how the hidden state is scaled. The logits reduce to one matrix multiplication, so training overhead is at most 5% over the Euclidean baseline (Table 10). Avoiding the additional overhead of exp/log maps makes this type of Hyperbolic LLM practical at LLM scale.

S2. A curvature-aware optimizer (CRAdam). When the curvature changes, CRAdam keeps the spatial coordinates fixed, recomputes the time coordinate, and rescales the first moment so its norm is preserved. CRAdam gives the best average accuracy and uses less memory and update time than Bdeir-RAdam (Table 6, Fig. 6c-d).

Critical weaknesses

W1. It is unclear how much of the gain comes from the geometry rather than the optimizer. With plain Adam, the LBL head falls below the Euclidean baseline (36.19 and 37.69 in Table 6 vs 38.75 in Table 1). It only wins with CRAdam. The authors attribute this to Adam being ill-suited to the Lorentz parameterization, which is reasonable. But the Euclidean head uses plain Adam, while LBLM adds per-token adaptive scaling, step clipping, and per-depth settings. A Euclidean head, trained with matched optimizer features and tuning, would show how much of the improvement is due to the hyperbolic geometry, which is the paper's central claim.

W2. The accuracy results may be noisy at this scale. With this token budget, several benchmarks are at or near chance, and the ablations use a single seed. The speed comparison with HELM is informative, but the accuracy comparison is not, since HELM does not appear to train properly in this setup (validation BPB 1.405 vs 0.890; Fig. 3a). Validation BPB is the most reliable signal here, and a longer training run at one scale would make the accuracy gains more convincing.

Questions for authors

Clarification:

  1. Which HELM variant was reimplemented (dense or MiCE)?
  2. What are ρmax⁡\rho_{\max} and ϵclip\epsilon_{\rm clip}?
  3. What output-layer LR and initialization does the Euclidean baseline use? Was it tuned?
  4. What is the training vocabulary size?

Minor Comments:

Tables/Figures:

  1. Throughout the results, the paper keeps referring to gains. But the metric is not named in the main-text tables; only App. E.1 says it is accuracy. Please also state the number of shots and whether accuracy is raw or length-normalized.
  2. Table 14: the best CSQA (36.0, Variant III) and BB-Lang (25.7, Ours) values are not bold.
  3. Figs. 4 and 5 have the same caption, although they show different comparisons. The captions should say which.
  4. Table 6: BB-Lang has a tie at 25.7 (CRAdam w/o MT and CRAdam), but only one entry is bold.

Notations:

  1. Some conflicting notations like: ρ\rho means c t\sqrt c\,t in Eq. 24 but the step length ρt\rho_t and the bound ρmax⁡\rho_{\max} on p.25.
  2. The superscript (o)(o) for "orthogonal" (Eqs. 3, 91) clashes with the pole o\boldsymbol o.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.