acceptodds
Under review as a conference paper at ICLR 2027

Joint Discrete Diffusion

Abstract

Discrete diffusion models generate tokens in any order, making conditionally independent predictions within each denoising step. Sampling multiple tokens independently in parallel can however severely degrade generation quality, motivating tractable models of joint token distributions. Here we introduce joint discrete diffusion, which uses a small learned conditional random field (CRF) head on a frozen diffusion backbone. The head defines a joint distribution over the sequence at each denoising step, allowing exact normalization, marginalization, and joint sampling. Two novel contributions are introduced to make this idea work in practice: A low-rank parameterization of the CRF that allows efficient use of the conditioning sequence and a one-step reveal objective. This reveal the remaining tokens objective is motivated by the fact that the CRF ELBO objective in the low reveal rate regime assigns a very low probability to revealing adjacent tokens simultaneously. Supervision of token dependence is therefore by construction substantially weakened. We tested our approach across large diffusion language models iLLaDA, Dream, SDLLM-MDLM, and SDLLM-Duo in prompt continuation tasks. Comparing the joint against the native backbone model, we observe over a range of conditions that the native is outperformed by a joint with half the number of generation steps. Our joint token modeling achieves a measured 1.8–1.9x speedup at equal or better scores under both text evaluators.

Primary area:
foundation or frontier models, including LLMs
Keywords:
diffusion language models, discrete diffusion, parallel decoding, joint token modeling, few-step generation, conditional random fields, large language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8YLYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces Joint Discrete Diffusion, which adds a small low-rank CRF head to a frozen diffusion language model to model dependencies among tokens generated in the same step. A one-step reveal objective trains the joint distribution over the remaining tokens. Experiments on iLLaDA, Dream, SDLLM-MDLM, and SDLLM-Duo evaluate generation with fewer steps and measure inference cost.

Critical strengths
  1. The low-rank CRF supports exact normalization, marginalization, and joint sampling with a small head. Across the four comparisons in Table 1A, it halves the number of generation steps while lowering NLL under both evaluators, with a measured 1.8–1.9× throughput gain.

  2. The analysis of weakened dependence supervision at low reveal rates motivates the one-step reveal objective. Paired training runs from identical starting weights support its benefit. Comparisons with CRF independent marginals and a factorized adapter matched in parameter count and training budget also support the additional value of joint sampling.

Critical weaknesses
  1. Evidence for text quality still relies mainly on external-model NLL. The entropy, repetition, and temperature controls are useful, but do not directly assess prefix relevance or coherence. The digit-heavy outputs reported for Duo in Appendix D.8 also warrant a closer check of generation quality.

  2. The contribution of the contextual gate is less clear. In Eqs. (12)–(15), the vocabulary features are shared across contexts, and the gate adjusts the strength of pairwise compatibility. The current controls support joint modeling but do not isolate the benefit of this gate over a shared constant.

Questions for authors
  1. Could you add a small blind human evaluation for one backbone from Table 1A, comparing Native at its listed budget with joint sampling and CRF independent marginals at half that budget? Please assess prefix relevance, coherence, and obvious generation errors.

  2. How much does the contextual gate contribute? Could you replace ηi(z)\eta_i(z) with a shared learnable positive scalar, keep the other training and sampling settings unchanged, and compare joint reconstruction NLL, evaluator NLL, and runtime?

4 comments

tl;dr: We added evidence for the quality of our method. In both math, code and text (even in a blind human study), our method performs significantly better at fixed compute. We enable speedups even in hard tasks (code and math reasoning) which typically allow only a few tokens per call.

Thank you for the suggestions! We believe the paper stands more strongly now.


Question 1. Could you add a small blind human evaluation for one backbone from Table 1A, comparing Native at its listed budget with joint sampling and CRF independent marginals at half that budget?

Excellent idea. We have added a blinded chatbot-arena comparison of the different approaches and budgets. The human study gathers 320 comparisons from 16 participants. We included a few more model configurations than the ones suggested -- and compared every model against each other.

See our human study here: https://anonymous.4open.science/r/joint-diffusion-C978/rebuttal-figures/human_preference.png

This figure has been added to the paper (page 10). Our CRF-based joint improves the speed--quality frontier, far ahead of other joint modelling approaches:

ModelCallsTime (s)Preference (%)WinsTiesLosses
Finetuned Native1285.0391.59485
JoDD (ours)321.5667.3612124
Finetuned Native642.6456.7501936
Finetuned Native381.5829.0133658
Tensor train271.5728.9163162
CoDD251.5726.7142963

Scores average equally over model opponents, counting ties as half. Our JoDD row is bolded.

Weakness 1. Evidence for text quality still relies mainly on external-model NLL

We have now added code and math (MBPP/GSM8K).

Our method outperforms both a matched LoRA finetuning as well as the other joint approaches, thus achieving speedups across all settings. Notably, the other joint approaches do not reliably beat the finetuned backbone.

See our math/code results here (and added to the revised paper, page 8):

https://anonymous.4open.science/r/joint-diffusion-C978/rebuttal-figures/math_code_text.png

Weakness 2 & Question 2. How much does the contextual gate contribute? Could you replace with a shared learnable positive scalar, keep the other training and sampling settings unchanged, and compare joint reconstruction NLL, evaluator NLL, and runtime?

We performed the requested ablation on all four backbones: replacing the contextual gate with one shared learnable positive scalar, while keeping the remaining training and decoding settings fixed.

The contextual gate improves held-out reconstruction NLL in all 12 comparisons across three training seeds and four backbones:

GateiLLaDADreamSDLLM-MDLMSDLLM-Duo
Shared scalar2.2177 ±0.00082.2177\thinspace{\scriptstyle\pm\thinspace 0.0008}2.2878 ±0.00382.2878\thinspace{\scriptstyle\pm\thinspace 0.0038}2.5254 ±0.00082.5254\thinspace{\scriptstyle\pm\thinspace 0.0008}1.7886 ±0.00071.7886\thinspace{\scriptstyle\pm\thinspace 0.0007}
Contextual gate2.1891 ±0.0008\mathbf{2.1891}\thinspace{\scriptstyle\pm\thinspace 0.0008}2.2420 ±0.0028\mathbf{2.2420}\thinspace{\scriptstyle\pm\thinspace 0.0028}2.4943 ±0.0001\mathbf{2.4943}\thinspace{\scriptstyle\pm\thinspace 0.0001}1.7788 ±0.0018\mathbf{1.7788}\thinspace{\scriptstyle\pm\thinspace 0.0018}

Held-out reconstruction NLL (lower is better); mean ± standard deviation across three training seeds. Unit: nats per missing token

Albeit adding only 4097 parameters, the gate gives a reproducible performance improvement at a negligible cost! We will include this new ablation as an Appendix.

Thanks for suggesting ablating the gate; the results have now been added to our response above. The anonymous repository is also accessible again after an outage earlier today.

I thank the authors for the detailed response and additional experiments, which address most of my concerns. Based on my overall assessment of the paper's contributions, I will maintain my original score.

Thank you for your engagement and the positive score!

Reviewer_KsyLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Joint Discrete Diffusion, which adds a conditional random field (CRF) head to pretrained diffusion language models to model dependencies between tokens generated in the same denoising step. The authors use a low-rank parameterization to make joint sampling computationally efficient. They also introduce a joint reconstruction objective to better capture token dependencies during training. The method is evaluated on four diffusion language models, including iLLaDA, Dream, SDLLM-MDLM, and SDLLM-Duo. The experiments report lower NLL scores and faster generation compared with the original models. The authors also compare joint sampling with independent sampling to examine how much of the improvement comes from modeling token dependencies.

Critical strengths

Strengths

  1. The paper proposes a relatively lightweight CRF head to model dependencies between neighboring tokens. Its low-rank design allows exact inference and joint sampling without constructing large vocabulary-pair matrices.

  2. The authors analyze why the standard diffusion ELBO provides weak supervision for token dependencies when the reveal rate is low. This motivates their joint reconstruction objective, supported by theoretical bounds and examples.

  3. The paper provides an efficient implementation of CRF inference using transfer matrices. The proposed method can also be applied to both masked diffusion models and uniform-state diffusion models.

Critical weaknesses

Weakness

  1. Limited Methodological Novelty: The idea of modeling joint token distributions in diffusion language models has already been explored in prior works, such as CoDD [1] and Tensor-Train Diffusion [2]. The authors also acknowledge that their CRF can be represented as a constrained tensor train. While the proposed low-rank parameterization and analysis of the training objective provide some contributions, the methodological novelty over these existing approaches remains somewhat limited.

  2. Limited Evaluation of Generation Quality: The main experiments rely primarily on NLL scores from GPT-2 and Qwen2.5, without human evaluation or downstream benchmarks such as mathematical reasoning and code generation. As the authors also observe in Appendix D.6, lower NLL does not necessarily indicate better generation quality and can sometimes accompany severe repetition. It is therefore unclear whether the reported improvements translate into better text generation in practical settings.

  3. Insufficient Comparisons with Existing Joint Diffusion Methods: Although the paper includes comparisons with Tensor-Train Diffusion, the methods differ substantially in model capacity, parameter count, and training configurations. The comparison with CoDD [1] is also limited to a scheduler analysis on a small GSM8K development set, rather than a direct comparison under the main evaluation setting. More controlled comparisons would help establish the advantages of the proposed method. Additionally, most main experiments use a single trained head per configuration, leaving the robustness of the improvements across training seeds unclear.

References

[1] Li et al. Breaking the Factorization Barrier in Diffusion Language Models. 2026.

[2] Kim and Sung. Tensor-Train Joint Modeling for Few-Step Masked Diffusion. 2026.

Questions for authors

Please check the Critical Weaknesses part.

2 comments

tl;dr: We now show improvements across the board in math, coding, and a human blind study. Regarding positioning: our approach, Coupled Discrete Diffusion (CoDD) and Tensor-Train Diffusion (TTD) are closely related but differ fundamentally in their parameterisation.

Thank you for making these points. This has helped us to make the paper much stronger; we hope you may now consider clearly accepting.


Weakness 1. The idea of modeling joint token distributions in diffusion language models has already been explored in prior works, such as CoDD [1] and Tensor-Train Diffusion [2]. The authors also acknowledge that their CRF can be represented as a constrained tensor train. While the proposed low-rank parameterization and analysis of the training objective provide some contributions, the methodological novelty over these existing approaches remains somewhat limited.

Thanks for pointing this out. We agree that our original exposition was not detailed enough. In our answer to Reviewer Gn9m, we recapped two paper changes:

  • a more detailed related works section, covering both new joint models and older CRF works,
  • a new Appendix C.2 comparing our method (now JoDD) against CoDD and TTD.

Feel free to check such changes. Our novelty lies in unlocking the practical parameterisation and application to modern LLMs for the CRF kernel.

Weakness 2 & 3. The main experiments rely primarily on NLL scores from GPT-2 and Qwen2.5, without human evaluation or downstream benchmarks such as mathematical reasoning and code generation. [...] More controlled comparisons would help establish the advantages of the proposed method.

We have now added mathematical reasoning (GSM8K; Cobbe et al. (2021)), Python coding (MBPP; Austin et al. (2021)) and a human blind eval in a chatbot arena setup.

Our method outperforms both a compute-matched LoRA finetuning as well as the other joint approaches, thus achieving speedups across all settings. Notably, the other joint approaches do not reliably beat the finetuned backbone.

See our math/code here:
https://anonymous.4open.science/r/joint-diffusion-C978/rebuttal-figures/math_code_text.png

See our human study here: https://anonymous.4open.science/r/joint-diffusion-C978/rebuttal-figures/human_preference.png

Those two figures have been added to the main text (see page 8 and 10); we also added an ethics statement. A brief snippet of the human study clearly shows our joint discrete diffusion performance:

BaselineTime (s)Preference for JoDD
CoDD1.5788%
Tensor train1.5786%
Finetuned Native (38 calls)1.5879%
Finetuned Native (64 calls)2.6469%
Finetuned Native (128 calls)5.0314%

JoDD takes 1.56 seconds per continuation. Ties count as half.

Across math/code/human preference, our method unlocks new speedups!

We have put substantial work into the concerns raised in your review, with highly encouraging results across benchmarks and human evaluation.

We hope you may assess these results, and we would welcome the chance to address any concerns.

Reviewer_Gn9mOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors consider a factorization problem of discrete diffusion, which leads to quality degradation when using a smaller number of steps. They propose a CRF-based extension of the factorized denoiser parametrization from factorized conditional marginal prediction by using additional multipliers that depend on pairs of tokens. The authors parametrize these additional multipliers for pair dependencies with a product of low-rank contextual nonnegative features. The authors test the proposed method by training CRF extended parametrization with 4 frozen backbones and evaluating generated text NLL and unigram entropy.

Critical strengths

None

Critical weaknesses

Limited novelty and lack of positioning with related work There are very close studies on using CRF parametrization for diffusion and non-autoregressive sequence generation [1, 2] that were omitted in the current submissions. This narrows the novelty of the paper to the specific CRF parametrization, which should be supported with enough evaluations and ablations on this design.

Insufficient evaluation The authors use FineWeb continuations with NLL of GPT-2 and Qwen in Table 1. However, it is known that the exponent of NLL (generative perplexity) is very vulnerable for texts consisting of repeated words, making generative perplexity alone not a reliable metric of text quality. Furthermore, even comparison of generative perplexity together with token diversity metrics such as standard unigram entropy is also vulnerable for texts consisting of repeated tokens, as was shown in [3]. To justify that the proposed method provides better generation quality/generation speed, additional setups are required, e.g., standard code and math benchmarks, that have a clear metric of the generation quality.

In Figure 3, the sampling from CRF marginals already shows most of the improvement over the frozen backbone, while joint sampling does not improve the frontier given the confidence intervals. A clearer analysis is needed to show that the proposed compact CRF-based extension of the denoiser benefits from the CRF joint sampling and not from better fitting of marginals due to longer training and more parameters. There is such analysis in Table 1B, but only for SDLLM-MDLM, and only for NLL alone without at least unigram entropy, which is not sufficient to conclude better text generation.

[1] Sun Z. et al. Fast structured decoding for sequence models //Advances in Neural Information Processing Systems. – 2019. – Т. 32.

[2] Floquet N., Roux J. L., Tomeh N. Approximate Structured Diffusion for Sequence Labelling //arXiv preprint arXiv:2606.18856. – 2026.

[3] Franca A., Tong A. Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs Distributional Metrics //arXiv preprint arXiv:2606.08417. – 2026.

Questions for authors

Please see weaknesses

6 comments

tl;dr: We agree that the first version relied overly on the NLL metric and did not do enough to connect with existing work. Across math, code and a human study, our CRF-based joint discrete diffusion outperforms other joint models and its compute-matched backbone. We have extended the Related work and added an Appendix C.2 to specifically discuss our relation to CoDD and tensor-train diffusion.

We therefore kindly ask the reviewer to consider clearly accepting. Thanks for pointing us in this direction!


Weakness 1. There are very close studies on using CRF parametrization for diffusion and non-autoregressive sequence generation [1, 2] that were omitted in the current submissions. This narrows the novelty of the paper to the specific CRF parametrization, which should be supported with enough evaluations and ablations on this design.

Thank you for these comments. We agree that our Related work paragraph as well as benchmark were too limited. We have now substantially expanded both. (We also found two highly relevant references about structured hidden Markov models that we have included.)

We reworked the related work in the introduction:

Related works. Chiu et al. (2021) and Yang et al. (2022) propose efficient structured hidden Markov models for large size vocabularies. Sun et al. (2019) augment non-autoregressive transformer-based models with a CRF head. Floquet et al. (2026), like our method, introduce a CRF head for discrete diffusion, but use mean field instead of exact inference and consider only small vocabulary sized problems. The Discrete Copula Diffusion sampler uses an auxiliary auto-regressive model whose computation can dominate runtime at small denoising budgets (Liu et al., 2025). Coupled Discrete Diffusion (CoDD) combines a probabilistic circuit with a window-based decoding scheduler (Li et al., 2026). Tensor-train diffusion (TTD) couples different positions with tensor-train along the sequence. Notably, CoDD, TTD and our approach may all be viewed as different tensor-train parameterisations, described in detail in our unified view (Appendix C.2). We show that our approach, JoDD, outperforms CoDD and TTD.

Additionally, a unified view with method comparisons is in a new Appendix C.2. The changes are now reflected in the submission.

Weakness 2. To justify that the proposed method provides better generation quality/generation speed, additional setups are required, e.g., standard code and math benchmarks, that have a clear metric of the generation quality.

We agree with the reviewer that NLL evaluations can be unreliable, so we have now done:

  • Mathematical reasoning with GSM8K (Cobbe et al. (2021))
  • Python coding with MBPP (Austin et al. (2021))
  • Human blind eval in a chatbot arena setup (requested by other reviewers)

Our method outperforms both a compute-matched LoRA finetuning as well as the other joint approaches, thus achieving speedups across all settings. Notably, the other joint approaches do not reliably beat the finetuned backbone.

See our math and code results here (and added to the revised paper, page 8):

https://anonymous.4open.science/r/joint-diffusion-C978/rebuttal-figures/math_code_text.png

Critical Strengths: None

We find this a very harsh judgement. We hope that the above can change the reviewer’s mind!

Regarding entropy: We have added an additional appendix with temperature sweeps across all methods for SDLLM-MDLM. For each method, we sweep temperatures at both 8; 16; and 32 steps. We then pick the temperature that precisely shares unigram entropy with JoDD at temperature 1.0 (Panel A).

At every decoding budget, JoDD significantly outperforms the other models (Panel B).

This appendix can be seen in the paper or on:

https://anonymous.4open.science/r/joint-diffusion-C978/rebuttal-figures/entropy_matched.png

Following your feedback, we now have much stronger evidence that our method is useful in practice, including math and code benchmarks and a blinded human study.

We believe the results are highly encouraging, so we would appreciate if you have the time to assess them. We welcome the chance to address any concerns you currently have.

Comments on the paper

2 comments

Dear reviewers,

The author-reviewer discussion period is ending soon. If you have not done so yet, please read the authors’ rebuttal and reply to it. Tell the authors whether it addresses your concerns, and update your score if your assessment has changed.

The ICLR 2027 Reviewer Guidelines list « discussing with the authors to clarify and improve the paper » as one of the core duties you accepted as a reviewer. With the high number of submissions this year, your input matters more than ever. Recall that reviewers that do not fulfill their duties may also face penalties on their own submissions.

Thank you for your time and help.

Best regards,


Your Area Chair

AuthorsOpenReview ↗

Summary for the AC and reviewers

We thank the reviewers for their feedback, which helped us strengthen the paper.

Our paper additions during discussion are:

  • Human evaluation: With 320 comparisons from 16 participants, JoDD achieves a 69% preference score against the finetuned backbone while generating 1.7× faster.

  • Math and code: a single shared head achieves a 1.39× speedup on MBPP at 51.4% accuracy, and a 1.17× speedup on GSM8K at 83% accuracy. These settings are hard, requiring very few tokens per step, hence do not gain as much from joint sampling.

  • Direct comparisons and clearer positioning: Outperforming CoDD and tensor-train baselines, expanded related work, and a unified perspective over the methods’ differences.

  • Additional ablations: Temperature-adjusted comparisons at matched unigram entropy, and contextual versus scalar gates across three seeds and four backbones.

We appreciate the reviewers’ suggestions, which helped us establish the practical benefits more clearly.

Kind regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.