acceptodds
Under review as a conference paper at ICLR 2027

Spherical Brownian Diffusion for Language Modeling

Abstract

Continuous diffusion language models involve two coupled design choices: how to corrupt continuous token representations and how to invert that corruption at generation time. Recent work has explored language generation on the unit sphere through geodesic and von Mises–Fisher (vMF) paths, raising the question of which forward process to use on this geometry and how much that choice matters relative to inference. We introduce Spherical Brownian Diffusion, using Brownian motion as the canonical isotropic diffusion on the sphere. Its heat semigroup provides a principled forward process, an additive dimension-free noise clock, and an exact ancestral bridge, while a clean-token posterior learned with cross-entropy supports multiple samplers. To separate corruption from inference, we derive high-dimensional approximation rates between Brownian and moment-matched vMF corruption and use this relationship for controlled substitutions. On Sudoku, Brownian and moment-matched vMF give nearly identical results across four samplers and five budgets under otherwise identical settings. In contrast, with the corruption process fixed, sampler choice produces a 25.0-point spread at a sampler budget of 180 calls. Using the ancestral bridge, we split this gain into a memory effect from re-noising the state, present on Sudoku and GSM8K, and a token-selection effect on GSM8K and largest at small budgets. Across reasoning tasks, categorical refinement improves performance, whereas unconditional text generation benefits from stochastic updates that preserve diversity.

Primary area:
generative models
Keywords:
dllm, diffusion, spherical noising, language modeling, samplers, generative models, continuous diffusion language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_u8j6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Spherical Brownian Diffusion (SBD) which uses the "spherical heat kernel" for forward corruption in language diffusion models. One of the main contributions is the "Heat bridge", that allows reverse sampling, with a tunable amount of the previous state preserved at each step. The method is evaluated on Sudoku, GSM8K (reasoning), and OpenWebText (open-ended text generation). The paper demonstrates that, in controlled settings they have, inference choice can matter more than kernel choice.

Critical strengths

The main clear strength is methodological novelty. To be more specific, adding noise to token embeddings using spherical heat kernels appears novel in diffusion language modeling. This enables the "Heat bridge" that allows control over how much of the current noisy state is kept during sampling, which is separate from token selection.

Critical weaknesses

1- The empirical results do not show consistent benefit. To be more specific, on figure 3, their method "SBD q-argmax" clearly does not beat the baselines on most of the budgets (on GSM8K). On figure 4, discrete baselines (Duo and MDLM) have better quality vs diversity trade-off (on OpenWebText). The same problem is for the kernel comparison in Tables 24 and 28, which makes the kernel's benefit inconsistent with the claim. On the other hand, on figure 2a, the small advantage that "Heat" has over matched mixture is not statistically significant (overlapping the intervals) while we have one seed of training. 2- The order in which the material is presented overcomplicates the paper, while some key results are kept in the appendix, making it hard to read. For instance, Proposition 1 only shows that Heat and vMF are similar under a specific noise scaling. It does not show improved sampling throughput or generation quality. Although the proposition is useful, it and the method section make the theoretical part of the paper too hard to understand. Meanwhile, important results such as Table 24 are in the appendix, forcing readers to go back and forth to understand these results.

Questions for authors

1- The 3 datasets (GSM8K, Sudoku, and OpenWebText) may not be diverse enough to support the conclusion the authors made, especially since most results focus on only one of them and as above mentioned, results are not significantly better. Could you test your claim that changing the inference procedure affects generation quality more than changing the kernel on one open-ended generation dataset (e.g., LM1B)? 2- Can you repeat the figure 2a with several training seeds to see whether small gain is reliable?

I also suggest including more results in the main paper.

I was more leaning towards borderline, and willing to change my rating during the discussion period.

5 comments

We thank the reviewer for highlighting the novelty of spherical heat-kernel corruption and of the Heat bridge, which separates state memory from token selection. We address the questions below; changes in the revised PDF are marked in red.

W1 (consistency with our claim). We realize our method-first presentation invited reading Heat as the better kernel and SBD as aiming for SOTA. We claim neither.

Our claim is that, with matched training and transport, nearby kernels perform similarly (Heat vs. vMF within 0.80.8 points under every sampler on Sudoku; Table 1a, Tables 25, 29), while the inference rule changes accuracy by 20–2520\text{–}25 points. Fig. 2a shows that matched fine-tuning recovers most of the mismatch loss (0.03% → 25.3%, vs. 27.9% for Heat). The revision states our claims explicitly (Sec. 1; Limitations, App. A) and acknowledges where discrete baselines are stronger (Sec. 5).

W2 (presentation). New Table 1 opens Sec. 4 with kernel ×\times sampler results for all three tasks (including former Table 24), so the main results no longer require the appendix or the theory. Proposition 1 serves only to justify vMF as a principled nearby control; readers can skip it without losing the empirical argument.

Table 1 (revised PDF): kernel vs. inference under matched training and sampling. Each row is a kernel used for both training and sampling, each column an inference rule. Range is the spread across inference rules for a fixed kernel.

(a) Hard Sudoku, accuracy (%) ↑\uparrow, B=180B=180, mean over 3 training seeds

KernelODESDEq-argmaxGatedRange
Heat58.462.873.983.425.0
vMF58.863.173.982.623.8

(b) GSM8K, accuracy (%) ↑\uparrow, B=256B=256 with posterior decoding

KernelODEq-argmaxRange
Heat7.627.119.6
vMF7.427.219.8
Heat, fine-tuned8.027.920.0
Mixture, fine-tuned7.325.318.1

(c) LM1B, log generative perplexity ↓\downarrow at low / high matched entropy, 1,0241{,}024 NFEs, 3 training seeds, 1M1\mathrm{M} updates

KernelHybrid k=1Hybrid k=4SDEQuantile ODERange
Heat4.13 / 4.864.29 / 4.904.50 / 5.034.62 / 5.120.50 / 0.26
vMF4.10 / 4.844.29 / 4.904.47 / 5.034.61 / 5.100.51 / 0.26
Mixture (λ=.5)4.33 / 5.114.26 / 5.084.36 / 4.944.45 / 5.010.18 / 0.17

Continued in Part 2 (Q1, Q2).

Q1 (LM1B). We trained Heat, moment-matched vMF and the λ=.5\lambda=.5 mixture from scratch on LM1B (1M1\mathrm{M} updates, 3 seeds each) and sampled each with four valid samplers at 256256 and 1,0241{,}024 NFEs (Table 1c, Sec. 4.6, App. E.12).

  • Heat and vMF are indistinguishable (∣Δlog⁡gen-PPL∣≤0.03|\Delta \log \text{gen-PPL}| \le 0.03).
  • The sampler effect exceeds the kernel effect in every setting (0.06–0.370.06\text{–}0.37 vs. 0.03–0.100.03\text{–}0.10).

LM1B at 256256 NFEs, log generative perplexity ↓\downarrow (low / high matched entropy, 3 training seeds, 1M1\mathrm{M} updates; Table 47)

KernelHybrid k=1Hybrid k=4SDEQuantile ODERange
Heat4.30 / 5.094.43 / 5.134.49 / 5.164.63 / 5.240.33 / 0.14
vMF4.29 / 5.114.44 / 5.144.49 / 5.154.61 / 5.210.32 / 0.10
Mixture (λ=.5)4.31 / 5.344.34 / 5.304.39 / 5.084.40 / 5.080.09 / 0.26

LM1B main effects (log gen-PPL; range of means averaged over the other factor)

NFEEntropyKernel effectSampler effect
256low0.100.25
256high0.040.06
1,024low0.030.37
1,024high0.070.14

LM1B over training (256 NFEs, low / high entropy; Table 48; 250k and 500k use one training seed, 1M the mean over three; negative favors the mixture)

UpdatesKernel effectSampler effectHeat–vMF max ∣Δ∣\lvert\Delta\rvertMixture best − Heat best
250k0.19 / 0.080.23 / 0.060.03−0.06 / −0.17
500k0.14 / 0.010.24 / 0.090.02−0.02 / −0.12
1M0.10 / 0.040.25 / 0.060.03+0.02 / −0.02
  • With each kernel’s best sampler, the mixture ties Heat at 256256 NFEs and is worse at 1,0241{,}024. It mainly helps continuous samplers, and its early advantage disappears with training (Table 48).
  • As on Sudoku and GSM8K, refinement improves with budget, while continuous transport improves much less.

The claim therefore extends to open-ended text: the inference rule matters more than the kernel.

Q2 (training seeds). With three training seeds per configuration, Heat reaches 27.93 ± 0.87% and the matched mixture 25.33 ± 0.32% (revised Fig. 2a, Table 43). Heat is ahead on every seed (paired difference −2.60±1.15-2.60 \pm 1.15 points), but the 95% interval [−5.46,+0.26][-5.46, +0.26] includes zero. Either way, the gap is small relative to the 28-point cost of mismatch. Since both arms start from a Heat checkpoint, part of it may reflect the mixture’s shorter adaptation (App. E.9).

GSM8K matched fine-tunes (posterior decoding, 257 NFEs; each value is the mean over 3 inference seeds; Table 43)

Training seedHeat → HeatMixture → mixtureMixture − Heat
027.1425.70−1.44
128.8625.12−3.74
227.8025.17−2.63
Mean ± sd27.93 ± 0.8725.33 ± 0.32−2.60 ± 1.15

Mismatched (Heat-trained, sampled with the mixture): 0.03%. Paired 95% tt-interval: [−5.46,+0.26][-5.46, +0.26].

Where to find the changes: Sec. 1 and Limitations (claims); Table 1 (W2, Q1); Sec. 4.6 and App. E.12 (Q1); Fig. 2a and Table 43 (Q2).

We hope these changes address the reviewer’s concerns and would be grateful if they would consider updating their score.

AuthorsOpenReview ↗

Gentle reminder

Dear Reviewer,

As the discussion period is coming to an end, we wanted to follow up and ask whether our response has addressed your concerns. We would be happy to provide any further clarifications if needed.

Thank you in advance for your time and consideration.

The authors

Reviewer_k4PXOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Spherical Brownian Diffusion (SBD), a continuous language modeling framework that uses Brownian motion on the unit sphere as the forward corruption process. The authors develop a clean-token posterior supporting multiple sampling strategies and provide theoretical analysis connecting Brownian and vMF corruption kernels. Through experiments on Sudoku, GSM8K, and OpenWebText, the paper investigates the effects of corruption kernels, sampling strategies, and state memory on generation performance, showing that inference strategies often have a greater impact than the choice of corruption kernel.

Critical strengths
  1. The paper provides a principled mathematical formulation of spherical Brownian diffusion, including an ancestral bridge and theoretical approximation bounds between Brownian and vMF corruption kernels. These analyses provide a theoretical foundation for investigating different corruption processes.

  2. The paper conducts experiments across multiple benchmarks, corruption kernels, and sampling strategies. The controlled comparisons and ablation studies help disentangle the effects of kernel choice, sampling strategies, and state memory on generation performance.

Critical weaknesses
  1. The proposed method does not consistently outperform existing baselines. On GSM8K, SBD underperforms several existing diffusion methods, particularly at larger inference budgets. On OpenWebText, MDLM and Duo achieve better quality-diversity trade-offs across much of the evaluated range.

  2. The controlled kernel comparisons are primarily conducted on Sudoku, while the analysis of state memory focuses on Sudoku and GSM8K. Moreover, the theoretical approximation between Brownian and vMF kernels applies only to a specific high-dimensional noise regime. It remains unclear whether these findings generalize to larger language models, more diverse generation tasks, or other diffusion architectures.

Questions for authors

Given that SBD does not consistently outperform existing baselines on GSM8K and OpenWebText, what are its main advantages in terms of generation quality or efficiency, particularly at larger inference budgets?

2 comments

We thank the reviewer for highlighting our principled formulation, including the exact bridge and heat–vMF bounds, and our controlled ablations. We address the concerns below; changes in the revised PDF are marked in red.

W1 / Q (advantages). We agree that SBD does not consistently outperform MDLM [1] / Duo [2], and the revision states this explicitly (Sec. 1; Limitations, App. A). We also added low-temperature MDLM/Duo at 512512 and 1,0241{,}024 NFEs: at 1,0241{,}024 NFEs, Duo reaches 36.6% vs. 30.1% for SBD (Table 36). Our claim is not SOTA but that, on the sphere, the inference rule matters more than the kernel, which SBD’s exact bridge makes measurable. We therefore respectfully ask that the paper be assessed on this claim and the evidence for it, rather than on benchmark leadership, which we do not claim. SBD is what makes this analysis possible, and it remains competitive in several settings:

  • best continuous model on hard Sudoku (83.4% vs. 81.7% for FMLM+ [4]).
  • on GSM8K at T=1T=1, above MDLM at every budget and above Duo from 128128 NFEs.
  • q-argmax roughly doubles S-FLM’s [3] top-1 accuracy (Table 38).
  • efficiency: latency comparable to MDLM/Duo, with q-argmax using the lowest peak memory at matched NFE (Table 18).

At larger budgets, we do not claim an advantage over low-temperature discrete models, though at matched T=1T=1 SBD’s lead grows with budget (28.8% vs. 19.0% for Duo at 1,0241{,}024 NFEs; Table 15). On the sphere, large-budget gains come from categorical refinement and are limited by retained state memory (Sec. 4.5).

GSM8K accuracy (%) at larger budgets (Table 15; SBD at T=1T=1 is the memoryless sampler, at low temperature q-argmax; bold is best per regime)

NFEMDLM (T=1T=1)Duo (T=1T=1)SBD (T=1T=1)MDLM (T=0.1T=0.1)Duo (T=0.1T=0.1)SBD (q-argmax)
25615.118.522.433.135.327.1
51214.918.326.233.334.330.0
1,02416.919.028.835.836.630.1

W2 (scope and generalization). The kernel study is not Sudoku-only: the submission already included GSM8K kernel crosses, trained-kernel comparisons, matched fine-tunes and a length analysis (Tables 39, 40, 42, 43). New Table 1 brings these into the main text, alongside a new from-scratch LM1B study (Sec. 4.6, 1M1\mathrm{M} updates, 3 seeds). On LM1B, Heat and vMF differ by at most 0.030.03 log gen-PPL, and the sampler effect exceeds the kernel effect in every setting.

Task (metric)Heat vs. vMF, max gap in any cellRange across inference rules (Heat)
Hard Sudoku (accuracy %)0.825.0
GSM8K (accuracy %)0.219.6
LM1B (log gen-PPL, low / high entropy)0.030.50 / 0.26

The state-memory analysis remains on Sudoku and GSM8K, where categorical refinement has a verifiable metric; on LM1B, refinement again benefits more from budget than continuous transport (Sec. 4.6).

Proposition 1 is intentionally local (Sec. 3.5, App. C.5): its role is to justify moment-matched vMF as a principled nearby control, not to predict generation quality.

Regarding generalization, our models follow the standard scale and protocol of the compared baselines (≈130M\approx 130\mathrm{M} DiT; the TinyGSM/OWT setups of MDLM [1], Duo [2], S-FLM [3] and FMLM+ [4], evaluated from their released checkpoints). LM1B adds a more diverse task, and the S-FLM transfer covers a different forward process and architecture.

We hope these clarifications and the new results address the reviewer’s concerns, and we would be grateful if they would reconsider their assessment. If concerns about soundness or presentation remain, we would appreciate the reviewer pointing us to them so that we can address them during the discussion.

[1] Sahoo, S. S., Arriola, M., Schiff, Y., Gokaslan, A., Marroquin, E., Chiu, J. T., ... & Kuleshov, V. (2024). Simple and effective masked diffusion language models. Advances in Neural Information Processing Systems, 37, 130136-130184.

[2] Sahoo, S. S., Deschenaux, J., Gokaslan, A., Wang, G., Chiu, J., & Kuleshov, V. (2025). The diffusion duality. Proceedings of machine learning research, 267, 52584.

[3] Deschenaux, J., & Gulcehre, C. (2026). Language modeling with hyperspherical flows. arXiv preprint arXiv:2605.11125.

[4] Agarwal, M., Shah, S., Lee, C., Yoo, J., Huang, J., Hong, S., ... & Boffi, N. M. (2026). Posterior Refinement: Fast Language Generation via Any-Order Flow Maps. arXiv preprint arXiv:2606.24773.

AuthorsOpenReview ↗

Gentle reminder

Dear Reviewer,

As the discussion period is coming to an end, we wanted to follow up and ask whether our response has addressed your concerns. We would be happy to provide any further clarifications if needed.

Thank you in advance for your time and consideration.

The authors

Reviewer_K2jbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Spherical Brownian Diffusion as a novel discrete diffusion model. It encodes the tokens to be vectors on a sphere, and uses brownian motion on sphere as the forward process. The paper derives its corresponding backward process and loss function, as well as the compatible decoding strategies. It finally conduct extensive pretraining experiments on general language datasets and logical reasoning tasks, to compare the performance to prior continuous diffusion works and masked diffusion models.

Critical strengths

The paper is clearly written and is easy to follow. It uses an appropriate amount of math to show necessary derivations, and gives a comprehensive comparison to existing works.

The experiments are extensive and convincing, even though it is only done on a toy scale. The improvement over existing methods is mild, but I think they suffice for demonstrating a new generative model.

Critical weaknesses

I am not fully convinced why we should tokenize the vocabulary on a sphere. The paper makes the point that the dimensionality would not grow as vocabulary grow, and may capture useful semantic distance. However, it is not clear to me whether this is the bottleneck that people are facing in scaling up (diffusion) language models. One naive pushback could be that modern AR models have large vocabulary, are trained using one-hot tokenizers and already achieved great intelligence, while it is not clear whether changing tokenizers would substantially speed up training.

Questions for authors
  1. Could the authors provide more motivation on why a spherical tokenization is important to (diffusion) language models? See Weakness.
  2. Perplexity could be misleading, especially for diffusion models that are not trained to generate autoregressively. Could the authors give some other evidence about the trained model is performing better on downstream tasks that we care about, or some discussion about the capability difference?
3 comments

We thank the reviewer for highlighting the clarity of the paper, the comprehensive comparison with prior work, and the extensiveness of our experiments. We address the questions below; changes in the revised PDF are marked in red.

Q1 (why the sphere). We do not claim spherical embeddings are a bottleneck for scaling. We adopt the sphere as used by recent continuous LMs (S-FLM [1], vMF flows [2]) and ask which forward process to use there and how much it matters relative to inference. Measured benefits include a state dimension independent of vocabulary size (d=768d=768 vs. 50k50\mathrm{k}) and a learned codebook that acquires semantic structure (Table 52). We do not propose replacing conventional tokenizers, nor do we claim faster training.

Q2 (perplexity). We agree with the reviewer that reporting only perplexity can be misleading. This is why our primary metrics are verifiable downstream tasks: Sudoku exact match and GSM8K executed-answer accuracy, where generated programs are run and their answers checked. For open-ended text we report generative perplexity only jointly with entropy [3], since perplexity can be lowered by reducing diversity.

[1] Deschenaux, J., & Gulcehre, C. (2026). Language modeling with hyperspherical flows. arXiv:2605.11125.

[2] Chemseddine, J., Kornhardt, G., & Steidl, G. (2026). Spherical Flows for Sampling Categorical Data. arXiv:2605.05629.

[3] Pynadath, P., Shi, J., & Zhang, R. (2026). Generative frontiers: Why evaluation matters for diffusion language models. arXiv:2604.02718.

Thank you for the response. I am happy to keep my score. I would love to see similar methods being scaled up in future work, and perhaps provide a better understanding of different tokenizers.

We thank the reviewer for their response and positive assessment, and for their interest in future extensions of this work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.