- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Spherical Brownian Diffusion (SBD) which uses the "spherical heat kernel" for forward corruption in language diffusion models. One of the main contributions is the "Heat bridge", that allows reverse sampling, with a tunable amount of the previous state preserved at each step. The method is evaluated on Sudoku, GSM8K (reasoning), and OpenWebText (open-ended text generation). The paper demonstrates that, in controlled settings they have, inference choice can matter more than kernel choice.
Critical strengths
The main clear strength is methodological novelty. To be more specific, adding noise to token embeddings using spherical heat kernels appears novel in diffusion language modeling. This enables the "Heat bridge" that allows control over how much of the current noisy state is kept during sampling, which is separate from token selection.
Critical weaknesses
1- The empirical results do not show consistent benefit. To be more specific, on figure 3, their method "SBD q-argmax" clearly does not beat the baselines on most of the budgets (on GSM8K). On figure 4, discrete baselines (Duo and MDLM) have better quality vs diversity trade-off (on OpenWebText). The same problem is for the kernel comparison in Tables 24 and 28, which makes the kernel's benefit inconsistent with the claim. On the other hand, on figure 2a, the small advantage that "Heat" has over matched mixture is not statistically significant (overlapping the intervals) while we have one seed of training. 2- The order in which the material is presented overcomplicates the paper, while some key results are kept in the appendix, making it hard to read. For instance, Proposition 1 only shows that Heat and vMF are similar under a specific noise scaling. It does not show improved sampling throughput or generation quality. Although the proposition is useful, it and the method section make the theoretical part of the paper too hard to understand. Meanwhile, important results such as Table 24 are in the appendix, forcing readers to go back and forth to understand these results.
Questions for authors
1- The 3 datasets (GSM8K, Sudoku, and OpenWebText) may not be diverse enough to support the conclusion the authors made, especially since most results focus on only one of them and as above mentioned, results are not significantly better. Could you test your claim that changing the inference procedure affects generation quality more than changing the kernel on one open-ended generation dataset (e.g., LM1B)? 2- Can you repeat the figure 2a with several training seeds to see whether small gain is reliable?
I also suggest including more results in the main paper.
I was more leaning towards borderline, and willing to change my rating during the discussion period.
5 comments
Part 2/2
Q1 (LM1B). We trained Heat, moment-matched vMF and the mixture from scratch on LM1B ( updates, 3 seeds each) and sampled each with four valid samplers at and NFEs (Table 1c, Sec. 4.6, App. E.12).
- Heat and vMF are indistinguishable ().
- The sampler effect exceeds the kernel effect in every setting ( vs. ).
LM1B at NFEs, log generative perplexity (low / high matched entropy, 3 training seeds, updates; Table 47)
| Kernel | Hybrid k=1 | Hybrid k=4 | SDE | Quantile ODE | Range |
|---|---|---|---|---|---|
| Heat | 4.30 / 5.09 | 4.43 / 5.13 | 4.49 / 5.16 | 4.63 / 5.24 | 0.33 / 0.14 |
| vMF | 4.29 / 5.11 | 4.44 / 5.14 | 4.49 / 5.15 | 4.61 / 5.21 | 0.32 / 0.10 |
| Mixture (λ=.5) | 4.31 / 5.34 | 4.34 / 5.30 | 4.39 / 5.08 | 4.40 / 5.08 | 0.09 / 0.26 |
LM1B main effects (log gen-PPL; range of means averaged over the other factor)
| NFE | Entropy | Kernel effect | Sampler effect |
|---|---|---|---|
| 256 | low | 0.10 | 0.25 |
| 256 | high | 0.04 | 0.06 |
| 1,024 | low | 0.03 | 0.37 |
| 1,024 | high | 0.07 | 0.14 |
LM1B over training (256 NFEs, low / high entropy; Table 48; 250k and 500k use one training seed, 1M the mean over three; negative favors the mixture)
| Updates | Kernel effect | Sampler effect | Heat–vMF max | Mixture best − Heat best |
|---|---|---|---|---|
| 250k | 0.19 / 0.08 | 0.23 / 0.06 | 0.03 | −0.06 / −0.17 |
| 500k | 0.14 / 0.01 | 0.24 / 0.09 | 0.02 | −0.02 / −0.12 |
| 1M | 0.10 / 0.04 | 0.25 / 0.06 | 0.03 | +0.02 / −0.02 |
- With each kernel’s best sampler, the mixture ties Heat at NFEs and is worse at . It mainly helps continuous samplers, and its early advantage disappears with training (Table 48).
- As on Sudoku and GSM8K, refinement improves with budget, while continuous transport improves much less.
The claim therefore extends to open-ended text: the inference rule matters more than the kernel.
Q2 (training seeds). With three training seeds per configuration, Heat reaches 27.93 ± 0.87% and the matched mixture 25.33 ± 0.32% (revised Fig. 2a, Table 43). Heat is ahead on every seed (paired difference points), but the 95% interval includes zero. Either way, the gap is small relative to the 28-point cost of mismatch. Since both arms start from a Heat checkpoint, part of it may reflect the mixture’s shorter adaptation (App. E.9).
GSM8K matched fine-tunes (posterior decoding, 257 NFEs; each value is the mean over 3 inference seeds; Table 43)
| Training seed | Heat → Heat | Mixture → mixture | Mixture − Heat |
|---|---|---|---|
| 0 | 27.14 | 25.70 | −1.44 |
| 1 | 28.86 | 25.12 | −3.74 |
| 2 | 27.80 | 25.17 | −2.63 |
| Mean ± sd | 27.93 ± 0.87 | 25.33 ± 0.32 | −2.60 ± 1.15 |
Mismatched (Heat-trained, sampled with the mixture): 0.03%. Paired 95% -interval: .
Where to find the changes: Sec. 1 and Limitations (claims); Table 1 (W2, Q1); Sec. 4.6 and App. E.12 (Q1); Fig. 2a and Table 43 (Q2).
We hope these changes address the reviewer’s concerns and would be grateful if they would consider updating their score.
Gentle reminder
Dear Reviewer,
As the discussion period is coming to an end, we wanted to follow up and ask whether our response has addressed your concerns. We would be happy to provide any further clarifications if needed.
Thank you in advance for your time and consideration.
The authors
Part 1/2
We thank the reviewer for highlighting the novelty of spherical heat-kernel corruption and of the Heat bridge, which separates state memory from token selection. We address the questions below; changes in the revised PDF are marked in red.
W1 (consistency with our claim). We realize our method-first presentation invited reading Heat as the better kernel and SBD as aiming for SOTA. We claim neither.
Our claim is that, with matched training and transport, nearby kernels perform similarly (Heat vs. vMF within 0.8 points under every sampler on Sudoku; Table 1a, Tables 25, 29), while the inference rule changes accuracy by 20–25 points. Fig. 2a shows that matched fine-tuning recovers most of the mismatch loss (0.03% → 25.3%, vs. 27.9% for Heat). The revision states our claims explicitly (Sec. 1; Limitations, App. A) and acknowledges where discrete baselines are stronger (Sec. 5).
W2 (presentation). New Table 1 opens Sec. 4 with kernel × sampler results for all three tasks (including former Table 24), so the main results no longer require the appendix or the theory. Proposition 1 serves only to justify vMF as a principled nearby control; readers can skip it without losing the empirical argument.
Table 1 (revised PDF): kernel vs. inference under matched training and sampling. Each row is a kernel used for both training and sampling, each column an inference rule. Range is the spread across inference rules for a fixed kernel.
(a) Hard Sudoku, accuracy (%) ↑, B=180, mean over 3 training seeds
(b) GSM8K, accuracy (%) ↑, B=256 with posterior decoding
(c) LM1B, log generative perplexity ↓ at low / high matched entropy, 1,024 NFEs, 3 training seeds, 1M updates
Continued in Part 2 (Q1, Q2).