acceptodds
Under review as a conference paper at ICLR 2027

Large Language Continuous Diffusion Models

30 views

Abstract

Despite success of discrete diffusion language models (dLMs) for fast parallel decoding, their non-smooth, high-dimensional space hinders trajectory steering for reasoning and inference acceleration. To overcome this, we present Sigma, the first large-scale (3B/8B) continuous dLM built on steerable, low-dimensional ODE/SDE latent trajectories. Trained blockwise via likelihood optimization, Sigma jointly denoises Gaussian-corrupted tokens while learning an optimal embedding geometry. To scale training, Sigma leverages pre-trained weights from existing autoregressive (AR) models for warm-starting. During inference, we identify classifier-free guidance and score temperature as essential for high-fidelity reasoning and coding. Across comprehensive math reasoning and coding evaluations against state-of-the-art discrete counterparts (masked dLMs and AR baselines), Sigma achieves competitive performance with discrete models on standard benchmarks (e.g., GSM8K, Minerva, HumanEval, MBPP) after pre-training and on challenging reasoning tasks (e.g., MATH-500, AIME) after supervised fine-tuning. Beyond performance parity, we uncover key structural properties unique to continuous dLMs: (i) embedding-space steering effectively governs the quality-diversity trade-off, yielding strong pass@k performance and (ii) continuous trajectories enable graceful degradation for low NFEs and efficient distillation. These establish continuous dLMs as a promising paradigm for efficient language generation.

Primary area:
generative models
Keywords:
Diffusion Language Models, Continuous Diffusion

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_yMjnOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Sigma, which is a continuous diffusion language model at 3B and 8B scales. Instead of denoising discrete tokens, Sigma performs Gaussian diffusion over learned 16 D token embeddings by using an autoregressive structure across blocks and continuous denoising within each block. The model is initialised from pretrained AR model weights and trained with a combination of diffusion and autoregressive objectives. The authors also explore CFG and ST as a inference trick to improve the performance to improve generation quality and reduce sampling steps. Experiments on math reasoning and coding benchmarks show performance comparable to discrete dLLMs.

Critical strengths

Good scalability. This paper scales continuous dLMs to 3B and 8B, which is not an easy task. The results are competitive with discrete dLLMs on math and coding benchmarks, showing that continuous diffusion can also work well at larger scales.

Fair baseline comparisons. I appreciate that the authors try to keep the model architecture, training data, and training setup consistent with the discrete dLLM baselines. This makes the comparisons more meaningful and helps us better understand how continuous diffusion performs against discrete diffusion.

Detailed inference experiments and useful insights. The paper provides quite a few experiments on CFG and different sampling methods such as DDPM and DDIM. I also find the analysis of embedding regularisation and few-step distillation useful, as it shows both the potential and the challenges of reducing diffusion steps.

Good presentation I particularly like how well the paper is presented. The writing is clear and easy to follow, and the figures are nicely designed. They make the overall method and experimental results much easier to understand.

Critical weaknesses

Too many extra components. The method combines quite a few existing techniques, such as AR regularisation, self conditioning, CFG, ST, SUSReg, and PDD, but I am not fully convinced why all of them are necessary or what the main technical novelty is beyond combining them. For example, the motivation for combining AR and diffusion training still feels somewhat ad hoc, despite the empirical improvements in PPL in Figure 3. (please see my Q1 below).

Lack of controlled ablations for key design choices. The paper mainly uses 16D embeddings and block size 32, without showing how sensitive the results are to these choices. Although Figure 3 includes block size 64, other settings also change, making it difficult to draw a clear conclusion (see my Q3).

Insufficient analysis of embedding collapse. The authors observe that the 16D embeddings collapse to around four effective dimensions and introduce SUSReg to address this. This observation is good and insightful. Yet, it is unclear what causes this issue, whether SUSReg actually fixes the collapse, or whether simply increasing the embedding dimension would work just as well.

Heavy dependence on hyperparameter tuning. The results seem quite sensitive to CFG weight and score temperature. Although the paper provides parameter sweeps, some settings seem to be selected based on the evaluation benchmarks themselves. I am not sure whether a separate validation set was used, as I could not find this clearly explained in the paper. This also makes me concerned about how well the reported performance would generalise to unseen tasks without further tuning (see my Q5).

Unclear practical inference efficiency. Sigma uses 256 diffusion steps per block for its strongest results in Table 1, while discrete dLLMs like NLD can decode multiple tokens per step and generally adopts much less diffusion steps within a block (about less than 16 for block size 32). Also, performance drops with fewer steps in sigma, and I could not find a direct comparison of actual generation latency and throughput with existing dLLMs. This makes me confused about its practical efficiency compared with existing discrete dLLMs (see Q2 below). I will consider raising the score if authors address my main concern.

Questions for authors

Why use two training streams? My concern is that I could not find a convincing technical motivation for combining AR and diffusion training within the same backbone. Although AR regularisation may improve performance empirically, the underlying reason remains unclear and the design choice makes me feel somewhat ad hoc, as AR and continuous diffusion are two quite different training objectives and may cause training conflict. Could the authors provide more insight or technical motivation into why this combination is necessary and why AR component leads to a better PPL in Figure 3? Also, could the AR objective introduce conflicting training signals or a mismatch with block based diffusion style inference?

Is Sigma actually faster? The strongest results require 256 diffusion steps per block to achieve its best performance as in Table 1 with a block size of 32. But many existing discrete dLLMs with block size 32 can finish a block in around 16 steps or fewer. This is because they can decode at least two tokens at a time. Even your NLD baseline can finish a block in fewer than 32 steps. This raises my another main concern since Sigma seems to require significantly more diffusion steps during inference compared to existing dLLMs with the same block size budget. Could the authors provide actual inference latency and throughput comparisons with discrete diffusion (like Fast-dLLM v1 or v2) and AR models under similar compute budgets?

Why fixed to 16D and why only train and test on block size 32? Figure 3 includes experiments with block sizes 32 and 64, but other training settings also change. Could the authors provide controlled ablations or any hyperparameter sensitivity analysis with different embedding dimensions and block sizes? How sensitive is the method to these choices?

What actually causes embedding collapse? How exactly are the four effective dimensions measured? Does SUSReg increase the effective dimensionality? Could using 32D or 64D embeddings reduce the collapse without additional regularisation?

How sensitive are the results to CFG weight w and score temperature τ\tau? How did you select them? The tested parameter combinations seem somewhat cherry picked and I could not find a clear justification for these hyperparameter combination choices in the draft. Since much of the performance improvement in your main table comes from selecting the best combinations for the evaluation benchmarks, I am concerned about how well the results would generalise to unseen tasks without further hyperparameter tuning by given validation sets. How would Sigma compare with models like LLaDA and Dream under the same tuning budget, especially when the optimal w and τ\tau are not known in advance? Are those hyperparameters chosen from the validation set? I did not find any validation set details in the paper.

Will the code and models be released? Since one of the main contributions in this work is scaling continuous dLLMs to 3B and 8B, do the authors plan to release the training and inference code as well as the pretrained checkpoints? I think this is particularly important given the number of additional components and hyperparameters involved in this pipeline. I will raise the score if authors address my main concern above.

11 comments
AuthorsOpenReview ↗

Clarification 1: Too many extra components

We thank the review for the detailed review. We hope our responses could address the reviewer’s concerns and we are happy to provide further clarifications and results.


My concern is that I could not find a convincing technical motivation for combining AR and diffusion training within the same backbone.

As discussed in Section 3.2.2, the auxiliary AR loss provides direct next-token supervision on clean context, complementing the high-variance diffusion objective. Sharing the backbone allows this additional supervision to train the clean-context representations used by the diffusion denoiser. Our ablations empirically support this design: adding the AR loss improves diffusion likelihood [Fig. 3c] and downstream performance [Figs. 12–13].

Importantly, such joint AR–diffusion training has multiple precedents in the discrete diffusion LM literature (e.g., [1–2]), supporting its use as a well-established design choice. Our work shows that it’s also effective for continuous dLMs.


The method combines quite a few existing techniques, such as AR regularisation, self conditioning, CFG, ST, SUSReg, and PDD, but I am not fully convinced why all of them are necessary or what the main technical novelty is beyond combining them.

Our contribution goes beyond combining existing techniques: we establish a principled and simple recipe for large-scale continuous diffusion LMs and new findings about their roles and effectiveness. Specifically, our work is the first to scale continuous diffusion LMs to 3B/8B and achieve performance comparable to NLD under controlled settings. Although the individual techniques originate in prior work, their roles and effectiveness in the context of large language continuous diffusion models were previously unclear; we establish these through detailed ablations in our original submission.

Relationship to prior workOur contribution at 3B/8B scaleAblation
AR regularizationA well-established technique in the discrete DLM literature (e.g., [1-2]).We demonstrate that it is helpful for continuous DLM as well.Fig. 3c; Figs. 12–13
Self-conditioningWidely used in continuous DLMs (e.g., [3-5]), with prior evaluations limited to smaller models, unconditional generation, or simple conditional tasks.We demonstrate that it is helpful for improving likelihood and downstream performance on standard math/coding tasks.Figs. 3a–b (likelihood); Fig. 7 (downstream performance)
SUSRegIntroduced for self-supervised visual representation learning in SPHERE-JEPA [6], where it encourages uniform spherical embeddings and prevents collapse.Identify embedding collapse in continuous DLMs and apply SUSReg to improve few-step generation and distillation.Fig. 7; Table 8; Fig. 8 (embedding geometry)
CFG/STStandard techniques for improving generation quality for image diffusion models [7] or small-scale continuous DLM [8].We demonstrate that combining them can help continuous DLM match downstream math/coding performance of discrete DLMs.Fig. 5; Fig. 12

Note that Parallel Decoding Distillation (PDD) [9] is for few-step distillation, which is a separate stage beyond the training/inference techniques described above.


References:

[1] Fu et al., “Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding,” EMNLP 2026.

[2] Zhu et al., “FLARE: Diffusion for Hybrid Language Model,” NeurIPS 2026.

[3] Chen et al., “LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling,” NeurIPS 2026.

[4] Hu et al., “ELF: Embedded Language Flows,” NeurIPS 2026.

[5] Meshchaninov et al., “How to Train Your Latent Diffusion Language Model Jointly With the Latent Space,” NeurIPS 2026.

[6] Nicollier et al., “SPHERE-JEPA: Spherical Prediction with Homogeneous Embeddings,” arXiv 2026.

[7] Ho and Salimans, “Classifier-Free Diffusion Guidance,” arXiv 2022.

[8] Dieleman et al., “Continuous Diffusion for Categorical Data,” arXiv 2022.

[9] Shaul et al., “Parallel Decoding Distillation for Fast Image and Video Generation,” arXiv 2026.

AuthorsOpenReview ↗

Clarification 2: Lack of ablations for design choices

The paper mainly uses 16D embeddings and block size 32, without showing how sensitive the results are to these choices. Although Figure 3 includes block size 64, other settings also change, making it difficult to draw a clear conclusion (see my Q3).

Block size: We chose 32 to match NLD’s inference block size for a controlled comparison.

[New experiment] As requested, we additionally trained 3B SUSReg models with block sizes 64 and 128 using the same pipeline. Each model generates at its training block size using DDIM, self-conditioning, and uniform-γ\gamma time discretization. CFG/ST are selected by average accuracy on GSM8K and HumanEval, then shared across all six tasks. We compare at matched nominal NFE/token, S/BS/B, where SS is denoising steps per block (excluding the final decoding step at t=0t=0) and BB is block size.

Nominal NFE/tokenBlock size 32Block size 64Block size 128
263.6058.5946.20
161.8556.4744.54
1/258.5052.0142.51
1/442.7642.6840.63
1/811.8025.0533.49

Values are six-task average accuracies (%). Block size 32 outperforms both larger sizes on every task at nominal NFE/token ≥ 1/2, supporting our choice for reasoning quality [Appendix G.5, Table 15]. Sizes 32 and 64 perform similarly at 1/4. Larger blocks win at 1/8, where they receive 8 or 16 steps/block versus only 4 for size 32, suggesting that too few denoising steps becomes limiting.

We agree that Fig. 3 does not isolate block size; the new ablations directly address this question.

Embedding dimension:

[New experiment] Preliminary 2k-step training runs with 64D embeddings across several SUSReg weights all yielded worse validation PPL than the 16D SUSReg baseline (≥ 5.153 versus 4.792). Stronger regularization worsened PPL, while weaker regularization allowed the PCA effective dimension to fall to approximately 25. These preliminary results support our choice of 16D, although they do not establish performance after full training.

RePlaid’s [1] dimension sweep [1, Appendix X.1] also finds near-optimal likelihood bounds at 16–32D, with somewhat (but not significantly) worse bounds at smaller and larger dimensions. A systematic study at 3B/8B scale remains future work.


References:

[1] Yang et al., “Continuous Diffusion Scales Competitively with Discrete Diffusion for Language,” NeurIPS 2026.

AuthorsOpenReview ↗

Clarification 3: Insufficient analysis of embedding collapse

What actually causes embedding collapse? How exactly are the four effective dimensions measured? Does SUSReg increase the effective dimensionality?

We hypothesize that embedding collapse is driven primarily by the MSE diffusion loss Ldiff\mathcal{L}_{\text{diff}} on jointly learned embeddings. Making all embeddings identical yields zero diffusion loss regardless of the predicted token probabilities xθ\mathbf{x}_{\theta} due to categorical reparameterization. Reconstruction cross-entropy Lrec\mathcal{L}_{\text{rec}} counteracts the loss of token identity, yet does not explicitly enforce a well-spread geometry. Embedding collapse due to MSE loss is a relatively well-known phenomenon [1-2].

We measure effective dimension using the participation ratio of the PCA spectrum. Let E∈RV×de\mathbf{E}\in\mathbb{R}^{V\times d_{e}} be the diffusion embedding matrix whose each row has unit-norm. We subtract the mean embedding across the VV tokens to obtain Ec\mathbf{E}_{c} (whose each column sums to zero), then compute the covariance 1V−1Ec⊤Ec\frac{1}{V-1}\mathbf{E}_{c}^{\top}\mathbf{E}_{c}. If λ1,…,λde\lambda_{1},\ldots,\lambda_{d_{e}} are its eigenvalues, the effective dimension is (∑iλi)2∑iλi2∈[1,de]\frac{(\sum_{i}\lambda_{i})^2}{\sum_{i}\lambda_{i}^2}\in[1,d_{e}]. Larger values indicate that variance is spread across more embedding dimensions.

SUSReg indeed increases the effective dimensions of E\mathbf{E}: the one of our 3B model train without SUSReg has 3.4008 effective dimensions, while the one with SUSReg has 15.9593 effective dimensions, which agrees with the qualitative measurement in [Fig 8].


Could using 32D or 64D embeddings reduce the collapse without additional regularisation?

In our experiments, we observed that without regularization, the 16D embeddings have an effective dimension of approximately 3.4, indicating strong low-dimensional concentration. This supports our view that a larger embedding space alone may not resolve the issue.

Consistent with this observation, RePlaid [3] reports that even with 768D embeddings, only 9 principal components explain 90% of the embedding variance [3, Appendix U, Fig. 15(a)]. These results suggest that increasing embedding dimension alone does not necessarily prevent low-dimensional concentration.


References:

[1] Gulrajani and Hashimoto, “Likelihood-Based Diffusion Language Models,” NeurIPS 2023.

[2] Meshchaninov et al., “How to Train Your Latent Diffusion Language Model Jointly With the Latent Space,” NeurIPS 2026.

[3] Yang et al., “Continuous Diffusion Scales Competitively with Discrete Diffusion for Language,” NeurIPS 2026.

Reviewer_ULCXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces Sigma, a large-scale continuous diffusion language model operating in a learned 16-dimensional embedding space. It combines blockwise generation, AR initialization and regularization, and inference-time steering techniques. Experiments show competitive performance with discrete diffusion models on reasoning and coding tasks, while also exploring efficient few-step generation.

Critical strengths

S1. The paper demonstrates that continuous diffusion language models can scale to 3B and 8B parameters while achieving competitive performance with strong discrete diffusion baselines. The unified training and evaluation framework provides valuable empirical evidence for the feasibility of continuous diffusion in large scale language modeling.

S2. The paper systematically studies CFG, score temperature, sampling algorithms, and few-step generation. The observations regarding quality-diversity trade-offs, embedding collapse, and the benefits of embedding regularization provide useful insights into the distinctive properties of continuous diffusion language models.

Critical weaknesses

W1. Sigma underperforms discrete diffusion models on several mathematical reasoning and coding tasks. However, the paper does not provide a detailed analysis of the underlying reasons. It remains unclear whether this is related to the use of 16-dimensional token embeddings or to inherent limitations of continuous diffusion models.

W2. Although the paper successfully develops a continuous diffusion language model and achieves performance comparable to discrete diffusion models, most of its key components are adopted from existing methods, and multiple modules are required to achieve strong performance.

Questions for authors

Could the authors provide a more detailed analysis of why Sigma exhibits a noticeable performance degradation on long reasoning tasks compared to discrete diffusion language models?

5 comments
AuthorsOpenReview ↗

Clarification 1: Gaps to discrete diffusion on specific tasks

We thank the review for the detailed review. We hope our responses could address the reviewer’s concerns and we are happy to provide further clarifications and results.


Sigma underperforms discrete diffusion models on several mathematical reasoning and coding tasks.

Our results demonstrate highly comparable overall performance under controlled settings; closing task-specific gaps remains future work. Below, we provide more details separately for base models and instruct models.

Base models

For both 3B and 8B base models, Sigma is indeed weaker‾\underline{\textrm{weaker}} on MBPP (coding) and MBPP+ (coding), but comparable or stronger on GSM8K (math), Minerva Math (Math), HumanEval (coding), and HumanEval+ (coding). Therefore, the average six-task performance is still comparable.

GSM8KMinerva MathHumanEvalHumanEval+MBPPMBPP+Six-task avg
NLD-3B83.4063.9656.1053.0557.6074.3464.74
Sigma-3B84.1864.0060.2454.0251.0269.7963.88
Sigma-3B-SUSReg84.1263.6857.3252.2056.08‾\underline{56.08}72.65‾\underline{72.65}64.34

Setting: DDIM-256 sampler. All Sigma numbers are averaged over 10 seeds. Numbers copied from [Table 7] .

[New experiment]After initial submission, we performed additional tuning and debugging for our training pipeline, while remaining faithful to the NLD setup. We found that Sigma-3B-SUSReg performance further improves (we label the method as Sigma-3B-SUSReg*), and become comparable with NLD on every math and coding task for base model evaluation.

GSM8KMinerva MathHumanEvalHumanEval+MBPPMBPP+Six-task avg
NLD-3B83.4063.9656.1053.0557.6074.3464.74
Sigma-3B-SUSReg*85.7364.7455.4352.8757.4072.0464.70

Setting: DDIM-256 sampler. All Sigma numbers are averaged over 10 seeds.

The result is scalable 8B as well: our base model is comparable with NLD on every math and coding task while even outperforming NLD on average:

GSM8KMinerva MathHumanEvalHumanEval+MBPPMBPP+Six-task avg
NLD-8B87.2665.1662.8057.3267.0081.7570.22
Sigma-8B-SUSReg*89.1267.0667.0162.5665.4880.2671.92

Setting: DDIM-256 sampler. All Sigma numbers are averaged over 10 seeds.

Instruct models

  • For 3B instruct model, Sigma-3B is indeed weaker‾\underline{\textrm{weaker}} on LiveCodeBench, but stronger on GPQA and IFEval. Overall, the ten-task performance is slightly better than NLD [Table 2] .
  • For 8B instruct model, Sigma-SUSReg-8B is indeed weaker‾\underline{\textrm{weaker}} on Math500, AIME24, and AIME25, but again is noticeably stronger on GPQA and IFEval, with an improvement of over 10 points on GPQA. Overall, the ten-task performance is comparable with NLD [Table 3] .

We acknowledge the remaining gaps (<10%) on MATH-500 and AIME 2024/2025 at 8B. Our results demonstrate comparable overall performance under controlled settings; closing these task-specific gaps remains future work. We also note that AIME has only 30 problems per task, so missing a single problem would lead to ~3% difference in accuracy.

AuthorsOpenReview ↗

Clarification 2: Relationship between performance and embedding dimension/geometry

It remains unclear whether this [gap in math/coding performance] is related to the use of 16-dimensional token embeddings or to inherent limitations of continuous diffusion models.

Our results show that improving embedding geometry substantially improves low-NFE sampling and distillation, even while retaining 16D embeddings. We conducted two complementary experiments:

1. Increasing embedding dimension. In preliminary 2k-step runs, all tested 64D configurations across several SUSReg weights had worse validation PPL than the 16D SUSReg baseline (≥ 5.153 vs. 4.792). These early results support retaining 16D, consistent with RePlaid’s near-optimal likelihood bounds at 16–32D [1, Appendix X.1]. Full-training ablations of embedding dimension exceed our current compute budget and remain future work.

2. Improving geometry at fixed dimension. At 16D, SUSReg increases the effective dimension from 3.40 to 15.96 [Fig. 8] and improves six-task average accuracy:

ModelSampling settingWithout SUSRegWith SUSReg
3BDDIM, 16 steps55.1758.50
3BDDIM, 8 steps37.0642.76
3BPDD, 8 steps42.1246.79
8BDDIM, 8 steps46.0552.01

The DDIM gains hold on every task at these budgets. The PDD comparison uses the same inference CFG/ST settings, demonstrating that the benefit also transfers to distillation [Tables 7–8].

These gains require no increase in embedding dimension. They are strongest at low NFEs; 256-step DDPM averages remain essentially unchanged. Embedding geometry therefore provides a concrete avenue for improvement for Sigma, while fully explaining the remaining performance gap requires further study.


References:

[1] Yang et al., “Continuous Diffusion Scales Competitively with Discrete Diffusion for Language,” NeurIPS 2026.

AuthorsOpenReview ↗

Clarification 3: Multiple modules are required to achieve good performance

most of its key components are adopted from existing methods, and multiple modules are required to achieve strong performance

Our contribution goes beyond combining existing techniques: we establish a principled and simple recipe for large-scale continuous diffusion LMs and new findings about their roles and effectiveness. Specifically, our work is the first to scale continuous diffusion LMs to 3B/8B and achieve performance comparable to NLD under controlled settings. Although the individual techniques originate in prior work, their roles and effectiveness in the context of large language continuous diffusion models were previously unclear; we establish these through detailed ablations in our original submission.

Relationship to prior workOur contribution at 3B/8B scaleAblation
AR regularizationA well-established technique in the discrete DLM literature (e.g., [1-2]).We demonstrate that it is helpful for continuous DLM as well.Fig. 3c; Figs. 12–13
Self-conditioningWidely used in continuous DLMs (e.g., [3-5]), with prior evaluations limited to smaller models, unconditional generation, or simple conditional tasks.We demonstrate that it is helpful for improving likelihood and downstream performance on standard math/coding tasks.Figs. 3a–b (likelihood); Fig. 7 (downstream performance)
SUSRegIntroduced for self-supervised visual representation learning in SPHERE-JEPA [6], where it encourages uniform spherical embeddings and prevents collapse.Identify embedding collapse in continuous DLMs and apply SUSReg to improve few-step generation and distillation.Fig. 7; Table 8; Fig. 8 (embedding geometry)
CFG/STStandard techniques for improving generation quality for image diffusion models [7] or small-scale continuous DLM [8].We demonstrate that combining them can help continuous DLM match downstream math/coding performance of discrete DLMs.Fig. 5; Fig. 12

Note that Parallel Decoding Distillation (PDD) [9] is for few-step distillation, which is a separate stage beyond the training/inference techniques described above.


References:

[1] Fu et al., “Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding,” EMNLP 2026.

[2] Zhu et al., “FLARE: Diffusion for Hybrid Language Model,” NeurIPS 2026.

[3] Chen et al., “LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling,” NeurIPS 2026.

[4] Hu et al., “ELF: Embedded Language Flows,” NeurIPS 2026.

[5] Meshchaninov et al., “How to Train Your Latent Diffusion Language Model Jointly With the Latent Space,” NeurIPS 2026.

[6] Nicollier et al., “SPHERE-JEPA: Spherical Prediction with Homogeneous Embeddings,” arXiv 2026.

[7] Ho and Salimans, “Classifier-Free Diffusion Guidance,” arXiv 2022.

[8] Dieleman et al., “Continuous Diffusion for Categorical Data,” arXiv 2022.

[9] Shaul et al., “Parallel Decoding Distillation for Fast Image and Video Generation,” arXiv 2026.

Reviewer_8yB4OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work proposed a type of continuous diffusion model, and tested its performance in 3B/8B params. The results demonstrates that, in the experimental setting of this paper, the proposed model is stronger than the discrete diffusion baseline, although the experimental setting may not be fair and comprehensive. Also, the results show that, the proposed continuous diffusion model has a better steering property than discrete diffusion baseline.

Critical strengths

The proposed continuous diffusion model, named Sigma:

  1. has better downstream accuracy in several tasks compared with discrete baselines;
  2. has better controllability in the steering setting;
Critical weaknesses
  1. The comparison in inference & training efficiency is lacking (such as, token throughput during inference time and tokens-per-gpu-per-second during training time);

  2. It is not sure whether the comparison is Iso-FLOPs or not (most likely that, continuous mode using larger FLOPs, thus the advantage in downstream accuracy may be not solid);

  3. The baselines of the other continuous diffusion baselines are missing.

Questions for authors

See Weakness.

8 comments
AuthorsOpenReview ↗

Concern 1: unfair comparisons

We thank the reviewer for the comments.

However, some of the reviewer’s concerns reflect misinterpretations of our work, which we clarify below. We hope our responses could address the reviewer’s concerns and we are happy to provide further clarifications and results.


Concern 1: unfair comparisons

most likely that, continuous mode using larger FLOPs, thus the advantage in downstream accuracy may be not solid.

We believe this characterization of our work is not quite accurate. Also, we only claim that Sigma achieves comparable, rather than better, downstream performance with Nemotron-Labs-Diffusion (NLD).

Training FLOPs compared to NLD

We kindly note that: the training comparison between NLD (masked DLM) and Sigma (our continuous DLM) is already approximately Iso-FLOPs and exactly Iso-Data, as we mentioned in [Sec 3.2.1] of our initial submission. In particular,

  1. Dataset, batch size and sequence length for training both methods are the same. Matching training data is a deliberate control: both methods learn from exactly the same examples.
  2. Sigma adopts the same transformer backbone of NLD.
  3. Similar to NLD which trains on a single noise level per block per sequence, Sigma uses a Monte Carlo estimator [Eq 5] for its training objective to avoid spending two forward passes to evaluate both diffusion loss and reconstruction loss per block per sequence.

Over 150k steps, Sigma incurs only 8% more training FLOPs compared to NLD due to its self-conditioning (SC) mechanism [Sec 2, Sec 3.2.1]. Despite this, we retain SC because it is widely used in continuous [1-3] and uniform-state DLMs [4-5]. Developing alternatives that remove SC while preserving performance is an active research direction [6]. To the best of our knowledge, comparable benefits of SC have not been demonstrated for large-scale masked diffusion LMs.

References:

[1] Chen et al., “LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling,” NeurIPS 2026.

[2] Hu et al., “ELF: Embedded Language Flows,” NeurIPS 2026.

[3] Meshchaninov et al., “How to Train Your Latent Diffusion Language Model Jointly With the Latent Space,” NeurIPS 2026.

[4] Jo et al., “Loopholing Discrete Diffusion: Deterministic Bypass of the Sampling Wall,” ICLR 2026.

[5] DiffusionGemma Team, “DiffusionGemma Technical Report,” arXiv 2026.

[6] Davis et al., “On the Design Space of Continuous Diffusion Language Models,” arXiv 2026.

[New ablation] For an IsoFLOPs ablation, we shortened Sigma-SUSReg-3B training from 150k to 138k steps, retaining SC and completing learning-rate decay at the new endpoint. This reduces our training compute by around 8%, giving a closely FLOP-matched comparison. Importantly, in this case, Sigma-SUSReg-3B uses less training data than NLD.

We evaluate the model performance with the same generation configuration as 150k full training model, and report the mean accuracy over 10 generations per question. As shown below, at this FLOP-matched training budget, DDPM-256 remains competitive on math: GSM8K is 84.16% versus NLD's 83.40%, and Minerva Math is 63.78% versus 63.96%. The six-task average decreases slightly from 64.22% to 62.79% (~1.4% gap), compared with NLD's 64.74%. Such performance remains significantly stronger than other continuous baselines (see Concern 3).

MethodGSM8KMinerva MathHumanEvalHumanEval+MBPPMBPP+Avg.
NLD-3B, AR83.7865.0653.0547.5660.8078.8464.85
Sigma-SUSReg-3B, 150k, AR82.1865.6453.6649.3960.0077.7864.77
Sigma-SUSReg-3B, 138k, AR81.8864.3450.0046.3460.4076.9863.32
NLD-3B, diffusion83.4063.9656.1053.0557.6074.3464.74
Sigma-SUSReg-3B, 150k, DDPM-25684.6163.8256.5950.4356.3473.5264.22
Sigma-SUSReg-3B, 138k, DDPM-25684.1663.7853.9047.9354.5272.4662.79
Sigma-SUSReg-3B, 150k, DDIM-3283.1763.2855.2450.9151.6466.8861.85
Sigma-SUSReg-3B, 138k, DDIM-3282.4863.1251.8948.2950.2265.9860.33
AuthorsOpenReview ↗

Concern 1: unfair comparisons (continued)

Inference FLOPs compared to NLD

[New experiment] Since confidence-based sampling for NLD does not fix a pre-defined NFE budget per block, we now sweep its confidence threshold (across 0.1 to 0.9) for parallel decoding to obtain accuracy-NFE tradeoffs on 6 based model tasks at 3B scale. We then compare NLD’s tradeoff against the tradeoff obtained by Sigma [Fig 10].

From the plot, we observe that

  • NLD takes on average around 16 steps per block to reach its best performance.
  • Before distillation, Sigma achieves comparable tradeoff with NLD (except on MBPP/MBPP+).
  • After distillation, Sigma achieves significantly better tradeoff than NLD at low NFEs. On HumanEval and HumanEval+, distillation even improves performance over pre-distillation and NLD.

For Sigma, during inference, having CFG increases FLOPs/forward compared to NLD due to an additional unconditional path [Fig 4], but that runs in parallel with the conditional path, adding only 4.2% latency in our measured setting [App. G.11]. Also, CFG can be distilled into the student model, so sigma-3B-SUSREG*-Distill* is CFG free and matches FLOPs/forward with NLD. Further, using score temperature (ST) alone, which requires no additional FLOPs/forward, can already obtain strong performance [Fig 5].

Finally, we’d like to emphasize that this is the first study of large-scale continuous dLMs in a controlled comparison with a state-of-the-art discrete dLM. Our primary goal is to demonstrate comparable reasoning performance:

  1. Although Sigma achieves meaningful inference efficiency (see above), these results are not a central claim of our paper; a comprehensive LLM serving study remains future work.
  2. Sigma with high NFE matches NLD, suggesting room for further gains through distillation, as our current approach uses an off-the-shelf method (PDD).
  3. We will open-source our models to support further research on continuous DLMs, including improved distillation and efficient serving.
AuthorsOpenReview ↗

Concern 2: benchmarking efficiency

The comparison in inference & training efficiency is lacking (such as, token throughput during inference time and tokens-per-gpu-per-second during training time);

[New experiment] We benchmark tokens-per-gpu-per-second during training. As expected (see Concern 1), Sigma has a slightly smaller but comparable throughput compared to NLD.

Tokens per gpu per second (↑\uparrow)△\triangle
NLD-3B5816.1
Sigma-3B (with self-conditioning)5477.25.8% fewer

Setting: 8 nodes of H100s, with a per-GPU batch size of 512/8/8=8 (sequence length 4096) and a per-GPU micro batch size of 2 (4 gradient accumulation steps per batch). We run 100 warmup training steps and then benchmark the following 100 training steps with our training pipeline.

[New experiment] We benchmark inference throughput in Fig 11. Observations there agree with the accuracy-NFE comparison (see Concern 1) since, on average, throughput directly correlate with NFE.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.