- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents Sigma, which is a continuous diffusion language model at 3B and 8B scales. Instead of denoising discrete tokens, Sigma performs Gaussian diffusion over learned 16 D token embeddings by using an autoregressive structure across blocks and continuous denoising within each block. The model is initialised from pretrained AR model weights and trained with a combination of diffusion and autoregressive objectives. The authors also explore CFG and ST as a inference trick to improve the performance to improve generation quality and reduce sampling steps. Experiments on math reasoning and coding benchmarks show performance comparable to discrete dLLMs.
Critical strengths
Good scalability. This paper scales continuous dLMs to 3B and 8B, which is not an easy task. The results are competitive with discrete dLLMs on math and coding benchmarks, showing that continuous diffusion can also work well at larger scales.
Fair baseline comparisons. I appreciate that the authors try to keep the model architecture, training data, and training setup consistent with the discrete dLLM baselines. This makes the comparisons more meaningful and helps us better understand how continuous diffusion performs against discrete diffusion.
Detailed inference experiments and useful insights. The paper provides quite a few experiments on CFG and different sampling methods such as DDPM and DDIM. I also find the analysis of embedding regularisation and few-step distillation useful, as it shows both the potential and the challenges of reducing diffusion steps.
Good presentation I particularly like how well the paper is presented. The writing is clear and easy to follow, and the figures are nicely designed. They make the overall method and experimental results much easier to understand.
Critical weaknesses
Too many extra components. The method combines quite a few existing techniques, such as AR regularisation, self conditioning, CFG, ST, SUSReg, and PDD, but I am not fully convinced why all of them are necessary or what the main technical novelty is beyond combining them. For example, the motivation for combining AR and diffusion training still feels somewhat ad hoc, despite the empirical improvements in PPL in Figure 3. (please see my Q1 below).
Lack of controlled ablations for key design choices. The paper mainly uses 16D embeddings and block size 32, without showing how sensitive the results are to these choices. Although Figure 3 includes block size 64, other settings also change, making it difficult to draw a clear conclusion (see my Q3).
Insufficient analysis of embedding collapse. The authors observe that the 16D embeddings collapse to around four effective dimensions and introduce SUSReg to address this. This observation is good and insightful. Yet, it is unclear what causes this issue, whether SUSReg actually fixes the collapse, or whether simply increasing the embedding dimension would work just as well.
Heavy dependence on hyperparameter tuning. The results seem quite sensitive to CFG weight and score temperature. Although the paper provides parameter sweeps, some settings seem to be selected based on the evaluation benchmarks themselves. I am not sure whether a separate validation set was used, as I could not find this clearly explained in the paper. This also makes me concerned about how well the reported performance would generalise to unseen tasks without further tuning (see my Q5).
Unclear practical inference efficiency. Sigma uses 256 diffusion steps per block for its strongest results in Table 1, while discrete dLLMs like NLD can decode multiple tokens per step and generally adopts much less diffusion steps within a block (about less than 16 for block size 32). Also, performance drops with fewer steps in sigma, and I could not find a direct comparison of actual generation latency and throughput with existing dLLMs. This makes me confused about its practical efficiency compared with existing discrete dLLMs (see Q2 below). I will consider raising the score if authors address my main concern.
Questions for authors
Why use two training streams? My concern is that I could not find a convincing technical motivation for combining AR and diffusion training within the same backbone. Although AR regularisation may improve performance empirically, the underlying reason remains unclear and the design choice makes me feel somewhat ad hoc, as AR and continuous diffusion are two quite different training objectives and may cause training conflict. Could the authors provide more insight or technical motivation into why this combination is necessary and why AR component leads to a better PPL in Figure 3? Also, could the AR objective introduce conflicting training signals or a mismatch with block based diffusion style inference?
Is Sigma actually faster? The strongest results require 256 diffusion steps per block to achieve its best performance as in Table 1 with a block size of 32. But many existing discrete dLLMs with block size 32 can finish a block in around 16 steps or fewer. This is because they can decode at least two tokens at a time. Even your NLD baseline can finish a block in fewer than 32 steps. This raises my another main concern since Sigma seems to require significantly more diffusion steps during inference compared to existing dLLMs with the same block size budget. Could the authors provide actual inference latency and throughput comparisons with discrete diffusion (like Fast-dLLM v1 or v2) and AR models under similar compute budgets?
Why fixed to 16D and why only train and test on block size 32? Figure 3 includes experiments with block sizes 32 and 64, but other training settings also change. Could the authors provide controlled ablations or any hyperparameter sensitivity analysis with different embedding dimensions and block sizes? How sensitive is the method to these choices?
What actually causes embedding collapse? How exactly are the four effective dimensions measured? Does SUSReg increase the effective dimensionality? Could using 32D or 64D embeddings reduce the collapse without additional regularisation?
How sensitive are the results to CFG weight w and score temperature ? How did you select them? The tested parameter combinations seem somewhat cherry picked and I could not find a clear justification for these hyperparameter combination choices in the draft. Since much of the performance improvement in your main table comes from selecting the best combinations for the evaluation benchmarks, I am concerned about how well the results would generalise to unseen tasks without further hyperparameter tuning by given validation sets. How would Sigma compare with models like LLaDA and Dream under the same tuning budget, especially when the optimal w and are not known in advance? Are those hyperparameters chosen from the validation set? I did not find any validation set details in the paper.
Will the code and models be released? Since one of the main contributions in this work is scaling continuous dLLMs to 3B and 8B, do the authors plan to release the training and inference code as well as the pretrained checkpoints? I think this is particularly important given the number of additional components and hyperparameters involved in this pipeline. I will raise the score if authors address my main concern above.
11 comments
Clarification 2: Lack of ablations for design choices
The paper mainly uses 16D embeddings and block size 32, without showing how sensitive the results are to these choices. Although Figure 3 includes block size 64, other settings also change, making it difficult to draw a clear conclusion (see my Q3).
Block size: We chose 32 to match NLD’s inference block size for a controlled comparison.
[New experiment] As requested, we additionally trained 3B SUSReg models with block sizes 64 and 128 using the same pipeline. Each model generates at its training block size using DDIM, self-conditioning, and uniform- time discretization. CFG/ST are selected by average accuracy on GSM8K and HumanEval, then shared across all six tasks. We compare at matched nominal NFE/token, , where is denoising steps per block (excluding the final decoding step at ) and is block size.
| Nominal NFE/token | Block size 32 | Block size 64 | Block size 128 |
|---|---|---|---|
| 2 | 63.60 | 58.59 | 46.20 |
| 1 | 61.85 | 56.47 | 44.54 |
| 1/2 | 58.50 | 52.01 | 42.51 |
| 1/4 | 42.76 | 42.68 | 40.63 |
| 1/8 | 11.80 | 25.05 | 33.49 |
Values are six-task average accuracies (%). Block size 32 outperforms both larger sizes on every task at nominal NFE/token ≥ 1/2, supporting our choice for reasoning quality [Appendix G.5, Table 15]. Sizes 32 and 64 perform similarly at 1/4. Larger blocks win at 1/8, where they receive 8 or 16 steps/block versus only 4 for size 32, suggesting that too few denoising steps becomes limiting.
We agree that Fig. 3 does not isolate block size; the new ablations directly address this question.
Embedding dimension:
[New experiment] Preliminary 2k-step training runs with 64D embeddings across several SUSReg weights all yielded worse validation PPL than the 16D SUSReg baseline (≥ 5.153 versus 4.792). Stronger regularization worsened PPL, while weaker regularization allowed the PCA effective dimension to fall to approximately 25. These preliminary results support our choice of 16D, although they do not establish performance after full training.
RePlaid’s [1] dimension sweep [1, Appendix X.1] also finds near-optimal likelihood bounds at 16–32D, with somewhat (but not significantly) worse bounds at smaller and larger dimensions. A systematic study at 3B/8B scale remains future work.
References:
[1] Yang et al., “Continuous Diffusion Scales Competitively with Discrete Diffusion for Language,” NeurIPS 2026.
Clarification 3: Insufficient analysis of embedding collapse
What actually causes embedding collapse? How exactly are the four effective dimensions measured? Does SUSReg increase the effective dimensionality?
We hypothesize that embedding collapse is driven primarily by the MSE diffusion loss on jointly learned embeddings. Making all embeddings identical yields zero diffusion loss regardless of the predicted token probabilities due to categorical reparameterization. Reconstruction cross-entropy counteracts the loss of token identity, yet does not explicitly enforce a well-spread geometry. Embedding collapse due to MSE loss is a relatively well-known phenomenon [1-2].
We measure effective dimension using the participation ratio of the PCA spectrum. Let be the diffusion embedding matrix whose each row has unit-norm. We subtract the mean embedding across the tokens to obtain (whose each column sums to zero), then compute the covariance . If are its eigenvalues, the effective dimension is . Larger values indicate that variance is spread across more embedding dimensions.
SUSReg indeed increases the effective dimensions of : the one of our 3B model train without SUSReg has 3.4008 effective dimensions, while the one with SUSReg has 15.9593 effective dimensions, which agrees with the qualitative measurement in [Fig 8].
Could using 32D or 64D embeddings reduce the collapse without additional regularisation?
In our experiments, we observed that without regularization, the 16D embeddings have an effective dimension of approximately 3.4, indicating strong low-dimensional concentration. This supports our view that a larger embedding space alone may not resolve the issue.
Consistent with this observation, RePlaid [3] reports that even with 768D embeddings, only 9 principal components explain 90% of the embedding variance [3, Appendix U, Fig. 15(a)]. These results suggest that increasing embedding dimension alone does not necessarily prevent low-dimensional concentration.
References:
[1] Gulrajani and Hashimoto, “Likelihood-Based Diffusion Language Models,” NeurIPS 2023.
[2] Meshchaninov et al., “How to Train Your Latent Diffusion Language Model Jointly With the Latent Space,” NeurIPS 2026.
[3] Yang et al., “Continuous Diffusion Scales Competitively with Discrete Diffusion for Language,” NeurIPS 2026.
Clarification 1: Too many extra components
We thank the review for the detailed review. We hope our responses could address the reviewer’s concerns and we are happy to provide further clarifications and results.
As discussed in
Section 3.2.2, the auxiliary AR loss provides direct next-token supervision on clean context, complementing the high-variance diffusion objective. Sharing the backbone allows this additional supervision to train the clean-context representations used by the diffusion denoiser. Our ablations empirically support this design: adding the AR loss improves diffusion likelihood[Fig. 3c]and downstream performance[Figs. 12–13].Importantly, such joint AR–diffusion training has multiple precedents in the discrete diffusion LM literature (e.g., [1–2]), supporting its use as a well-established design choice. Our work shows that it’s also effective for continuous dLMs.
Our contribution goes beyond combining existing techniques: we establish a principled and simple recipe for large-scale continuous diffusion LMs and new findings about their roles and effectiveness. Specifically, our work is the first to scale continuous diffusion LMs to 3B/8B and achieve performance comparable to NLD under controlled settings. Although the individual techniques originate in prior work, their roles and effectiveness in the context of large language continuous diffusion models were previously unclear; we establish these through detailed ablations in our original submission.
Note that Parallel Decoding Distillation (PDD) [9] is for few-step distillation, which is a separate stage beyond the training/inference techniques described above.
References:
[1] Fu et al., “Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding,” EMNLP 2026.
[2] Zhu et al., “FLARE: Diffusion for Hybrid Language Model,” NeurIPS 2026.
[3] Chen et al., “LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling,” NeurIPS 2026.
[4] Hu et al., “ELF: Embedded Language Flows,” NeurIPS 2026.
[5] Meshchaninov et al., “How to Train Your Latent Diffusion Language Model Jointly With the Latent Space,” NeurIPS 2026.
[6] Nicollier et al., “SPHERE-JEPA: Spherical Prediction with Homogeneous Embeddings,” arXiv 2026.
[7] Ho and Salimans, “Classifier-Free Diffusion Guidance,” arXiv 2022.
[8] Dieleman et al., “Continuous Diffusion for Categorical Data,” arXiv 2022.
[9] Shaul et al., “Parallel Decoding Distillation for Fast Image and Video Generation,” arXiv 2026.