acceptodds
Withdrawn from ICLR 2027

Inside the Scaling of Riemannian Flow Language Models: Geometry, Generation, and Conditioning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Riemannian flow language models have not been studied at practical scale. We train SwaFlow, a suite of hyperspherical flow language models with up to 3B parameters on 2T tokens. As training progresses, token embeddings become less isotropic, shrinking the range of noise levels from which local geometry alone recovers a token. Yet the models recover tokens from increasingly higher noise levels using context, and the gap between these regimes grows with scale. Consistent with this trend, held-out loss and unconditional generation improve with model size and training budget, but conditioning on clean context does not: multiple-choice accuracy stays at chance at every size and budget. We trace this failure to the training distribution rather than the geometry: standard training assigns the same noise level to all positions and never exposes the model to mixed clean–corrupted contexts. Prior work under a different geometry has shown that clean-prefix corruption improves conditional generation; for Riemannian flow language models, scaling alone does not remove this dependence on partially clean training examples. Introducing them only during the final 100B training tokens lifts conditioning off chance at every size, with gains that grow with model size and training budget and carry over to task finetuning on GSM8K. We will release all intermediate checkpoints, optimizer states, and exact data order upon acceptance, enabling these scaling trajectories to be reproduced, branched, and re-evaluated.

Primary area:
generative models
Keywords:
flow language models, Riemannian flow matching, non-autoregressive language generation, scaling, training dynamics, embedding geometry

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JUk4OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how Riemannian (hyperspherical) flow language models scale, a family that has so far only been studied near 0.1B. The authors train SwaFlow, four sizes from 150M to 3B on 2T tokens, beside autoregressive twins that share the corpus, data order, and schedule and differ only in objective. Because the corruption level α\alpha is continuous, the held-out loss is a curve over α\alpha rather than a single number. The authors read that curve through two points: αsol\alpha_{sol}, the noise level where the loss falls below 0.1 nats, and αgeo\alpha_{geo}, a closed-form threshold read off the trained embedding table, above which context-free nearest-neighbour decoding works from per-position evidence alone. The interval between the two is where the model must be using context, and it widens with scale while the embeddings grow more anisotropic. The other main result is that unconditional loss and generation improve with size and budget, but conditioning on clean context does not: multiple-choice accuracy stays close to random at every size and budget, and given a clean prefix the model predicts the continuation worse than the corpus unigram does. The authors trace this to the corruption schedule, since one α\alpha is drawn per sequence and pretraining never sees a sequence that is clean in one part and noised in another. Keeping a random prefix of half the sequences clean for the final 100B pretraining tokens, with all else fixed, removes the deficit at every size, with gains that grow with scale and transfer to GSM8K finetuning.

Critical strengths

S1: The causal test of the schedule diagnosis is well designed. Each model branches from its own uniform run: the mid-training arm keeps a random prefix of half the sequences clean for the last 100B tokens, and the uniform arm continues over the same tokens in the same order at the same learning rate, so the two arms differ only in which positions are clean. The autoregressive twins on identical data and order isolate the objective from everything else, and the (promised) released artifacts will be an important resource for the community. This makes the main claim an intervention result rather than a correlation.

S2: Reading the loss along α\alpha, and the corrected geometric threshold, are useful reframings for continuous language models. Reading cmeanc_{mean} off the trained embedding table instead of assuming isotropic embeddings lets αgeo\alpha_{geo} be tracked while scaled training makes the embeddings anisotropic, and the band between αsol\alpha_{sol} and αgeo\alpha_{geo} gives a simple, interpretable way to separate what per-position evidence can explain from what the model adds through context.

Critical weaknesses

W1: The clarity is lacking, and this is the main reason my score stays at weak accept. The paper stacks one mechanism on top of another (solvable frontier, geometric threshold, information band etc.) without enough explanation for a reader to follow where each one comes from, and the writing reads as if it were heavily postprocessed with an LLM. The results themselves are strong, but as written they are hard to evaluate and hard to reuse, and several key claims are only understandable after going back and forth through the appendices.

W2: The conditioning gains are task-selective, and the remaining gap to autoregression is large. The multiple-choice improvement is carried mostly by BoolQ (binary task), while OBQA sits at or below random (0.226–0.256 vs. 0.25) and HellaSwag stays near random. On the same suite the autoregressive twins lead on five of six tasks. The recipe is also fixed in advance and never swept, so the authors shows the schedule helps, but not how much it could help.

Questions for authors

Q1: BoolQ drives most of the multiple-choice gain, and OBQA stays at or below random after mid-training. Is there evidence that the model uses the clean prefix in a task-generic way, for instance on tasks where the correct answer actually depends on the prefix content, rather than having learned task-specific scoring behaviour on binary questions?

Q2: Would a model trained with the clean-prefix schedule from the start, instead of as a 100B branch, show the same or better conditioning and come closer to the autoregressive twins?

Q3: The twins lead on five of six multiple-choice tasks after mid-training. How much of that remaining gap do the authors attribute to the flow objective versus the training regime? Is there anything in the analysis that could separate these two?

Reviewer_SKHCOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies hyperspherical flow language models across four nominal sizes, from 150M to 3B parameters, generally trained on 2T tokens, with autoregressive twins at three sizes. It relates denoising loss to an embedding-derived geometric threshold and finds that improving unconditional modeling does not reliably improve clean-prefix conditioning. Replacing uniform corruption with a clean-prefix mixture during the final 100B tokens improves several conditioning diagnostics. The proposed release includes intermediate checkpoints, optimizer states, and data-order records.

Critical strengths
  1. The matched intervention provides useful evidence about a training-distribution mismatch. Branches share the checkpoint, subsequent data, and learning-rate schedule; several also share noise draws. Prefix-probe accuracy improves from 0.451–0.593 to 0.704–0.954, and clean versus corrupted instruction-prompt loss reverses direction at every size. These results independently support improved clean-prefix handling, beyond the multiple-choice results. [Sections 4–5; Table 10, p. 21; Appendix F.1.]

  2. The trajectory study is unusually informative. Multiple scales, a common corruption grid, and autoregressive controls support comparisons across training trajectories. Reporting repetitive collapse alongside generation perplexity also exposes an important metric failure: lower conditional Gen-PPL can accompany more repetitive output. [Appendices A–B; E.1, p. 20.]

Critical weaknesses
  1. The geometric band does not establish the claimed contextual mechanism. Equation (2) is a sufficient, conservative condition for local nearest-neighbor recovery. Appendix C explicitly notes that local decoding can succeed below it because the separately maximized competitor terms need not occur together. Consequently, a model frontier below this threshold does not imply that context was necessary. Nevertheless, the Abstract, Introduction, and Section 3.2 interpret the widening band as improved contextual recovery. Direct local-decoder and context-ablation measurements are needed to establish this central interpretation. [pp. 1–4, 17–18.]

  2. The headline downstream improvement needs stronger separation from answer-prior calibration. Matched mid-trained models score 0.611–0.621 on BoolQ, near its approximately 62.2% majority baseline. The exact split frequencies and prediction distributions are needed to distinguish useful contextual prediction from answer-prior recalibration. At 150M, the BoolQ change contributes 0.0407 of the 0.041 average MCQ gain; Appendix F.3 attributes the budget-axis gain to BoolQ. Independent prefix benefits remain credible, but broader conditional capability is less established. The practical trade-off also remains unquantified: generation uses 64 function evaluations, while the paper does not report a matched quality–runtime comparison with the autoregressive twins. [Table 2, p. 8; Appendices E.1–E.2 and F, pp. 20–22.]

Questions for authors
  1. On representative existing checkpoints, do measured local-decoding curves and matched full-versus-shuffled-context losses reveal a substantial contextual advantage within the proposed band? Positive results would directly address the main mechanistic concern.

  2. What are the exact BoolQ split frequencies, predicted-answer proportions, confusion matrices, and balanced accuracies? Do the improvements survive majority-baseline and counterfactual-passage controls, supporting capability beyond answer-prior recalibration?

  3. For a representative existing model, what quality–runtime trade-off is obtained with fewer sampling steps and with 64 evaluations, relative to its autoregressive twin? This would clarify the practical value of the demonstrated conditioning improvements.

Reviewer_AKmyOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents SwaFlow, a suite of hyperspherical Riemannian flow language models scaled up to 3B parameters. The authors analyze how scaling affects embedding geometry, denoising performance, and conditional generation. They find that while unconditional generation improves with scale, conditioning on clean contexts remains poor due to a mismatch between training and inference. To address this, they adopt clean-prefix mid-training, which improves conditional generation and downstream performance.

Critical strengths

1, Extended experiments are done on the scaling behaviors of Riemannian flow language models.

Critical weaknesses

1, This paper is clearly written by AI or human imitating AI writing style. For example, there are lots of unnecessary references to previous sections in the context. Also, the layout of the figures, tables, and text makes the paper hard to follow. However, I noticed that the author only describes the AI usage in the paper as polishing in the AI usage statement.

2, The motivation and the contribution of this paper are both very limited and problematic. For example, the loss of the Riemannian flow language models does not indicate the perplexity of the language model, thus not able to serve as a metric for scaling experiments. While the paper builds the analyzing experiments upon this metric, the result could be biased. Moreover, the clean prefix mid-training, the mismatch between full sequence training and conditional generation, the training recipe, are all well-studied topics. The paper seems to only reproduce them in the context of flow language models, which does not constitutes real contributions.

Questions for authors
  1. Since the model is trained with a single noise level per sequence, clean-prefix conditioning introduces OOD inputs with different noise levels across positions. Why is the resulting failure considered a meaningful training–inference mismatch rather than an expected consequence of an unsupported inference protocol?

  2. The geometric threshold is only a sufficient, not necessary, condition for context-free token recovery. How can the authors conclude that a wider information band indicates greater contextual utilization rather than simply reflecting a loose geometric bound?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.