- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how Riemannian (hyperspherical) flow language models scale, a family that has so far only been studied near 0.1B. The authors train SwaFlow, four sizes from 150M to 3B on 2T tokens, beside autoregressive twins that share the corpus, data order, and schedule and differ only in objective. Because the corruption level is continuous, the held-out loss is a curve over rather than a single number. The authors read that curve through two points: , the noise level where the loss falls below 0.1 nats, and , a closed-form threshold read off the trained embedding table, above which context-free nearest-neighbour decoding works from per-position evidence alone. The interval between the two is where the model must be using context, and it widens with scale while the embeddings grow more anisotropic. The other main result is that unconditional loss and generation improve with size and budget, but conditioning on clean context does not: multiple-choice accuracy stays close to random at every size and budget, and given a clean prefix the model predicts the continuation worse than the corpus unigram does. The authors trace this to the corruption schedule, since one is drawn per sequence and pretraining never sees a sequence that is clean in one part and noised in another. Keeping a random prefix of half the sequences clean for the final 100B pretraining tokens, with all else fixed, removes the deficit at every size, with gains that grow with scale and transfer to GSM8K finetuning.
Critical strengths
S1: The causal test of the schedule diagnosis is well designed. Each model branches from its own uniform run: the mid-training arm keeps a random prefix of half the sequences clean for the last 100B tokens, and the uniform arm continues over the same tokens in the same order at the same learning rate, so the two arms differ only in which positions are clean. The autoregressive twins on identical data and order isolate the objective from everything else, and the (promised) released artifacts will be an important resource for the community. This makes the main claim an intervention result rather than a correlation.
S2: Reading the loss along , and the corrected geometric threshold, are useful reframings for continuous language models. Reading off the trained embedding table instead of assuming isotropic embeddings lets be tracked while scaled training makes the embeddings anisotropic, and the band between and gives a simple, interpretable way to separate what per-position evidence can explain from what the model adds through context.
Critical weaknesses
W1: The clarity is lacking, and this is the main reason my score stays at weak accept. The paper stacks one mechanism on top of another (solvable frontier, geometric threshold, information band etc.) without enough explanation for a reader to follow where each one comes from, and the writing reads as if it were heavily postprocessed with an LLM. The results themselves are strong, but as written they are hard to evaluate and hard to reuse, and several key claims are only understandable after going back and forth through the appendices.
W2: The conditioning gains are task-selective, and the remaining gap to autoregression is large. The multiple-choice improvement is carried mostly by BoolQ (binary task), while OBQA sits at or below random (0.226–0.256 vs. 0.25) and HellaSwag stays near random. On the same suite the autoregressive twins lead on five of six tasks. The recipe is also fixed in advance and never swept, so the authors shows the schedule helps, but not how much it could help.
Questions for authors
Q1: BoolQ drives most of the multiple-choice gain, and OBQA stays at or below random after mid-training. Is there evidence that the model uses the clean prefix in a task-generic way, for instance on tasks where the correct answer actually depends on the prefix content, rather than having learned task-specific scoring behaviour on binary questions?
Q2: Would a model trained with the clean-prefix schedule from the start, instead of as a 100B branch, show the same or better conditioning and come closer to the autoregressive twins?
Q3: The twins lead on five of six multiple-choice tasks after mid-training. How much of that remaining gap do the authors attribute to the flow objective versus the training regime? Is there anything in the analysis that could separate these two?