- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces Joint Discrete Diffusion, which adds a small low-rank CRF head to a frozen diffusion language model to model dependencies among tokens generated in the same step. A one-step reveal objective trains the joint distribution over the remaining tokens. Experiments on iLLaDA, Dream, SDLLM-MDLM, and SDLLM-Duo evaluate generation with fewer steps and measure inference cost.
Critical strengths
-
The low-rank CRF supports exact normalization, marginalization, and joint sampling with a small head. Across the four comparisons in Table 1A, it halves the number of generation steps while lowering NLL under both evaluators, with a measured 1.8–1.9× throughput gain.
-
The analysis of weakened dependence supervision at low reveal rates motivates the one-step reveal objective. Paired training runs from identical starting weights support its benefit. Comparisons with CRF independent marginals and a factorized adapter matched in parameter count and training budget also support the additional value of joint sampling.
Critical weaknesses
-
Evidence for text quality still relies mainly on external-model NLL. The entropy, repetition, and temperature controls are useful, but do not directly assess prefix relevance or coherence. The digit-heavy outputs reported for Duo in Appendix D.8 also warrant a closer check of generation quality.
-
The contribution of the contextual gate is less clear. In Eqs. (12)–(15), the vocabulary features are shared across contexts, and the gate adjusts the strength of pairwise compatibility. The current controls support joint modeling but do not isolate the benefit of this gate over a shared constant.
Questions for authors
-
Could you add a small blind human evaluation for one backbone from Table 1A, comparing Native at its listed budget with joint sampling and CRF independent marginals at half that budget? Please assess prefix relevance, coherence, and obvious generation errors.
-
How much does the contextual gate contribute? Could you replace with a shared learnable positive scalar, keep the other training and sampling settings unchanged, and compare joint reconstruction NLL, evaluator NLL, and runtime?
4 comments
Thanks for suggesting ablating the gate; the results have now been added to our response above. The anonymous repository is also accessible again after an outage earlier today.
I thank the authors for the detailed response and additional experiments, which address most of my concerns. Based on my overall assessment of the paper's contributions, I will maintain my original score.
Thank you for your engagement and the positive score!
tl;dr: We added evidence for the quality of our method. In both math, code and text (even in a blind human study), our method performs significantly better at fixed compute. We enable speedups even in hard tasks (code and math reasoning) which typically allow only a few tokens per call.
Thank you for the suggestions! We believe the paper stands more strongly now.
Excellent idea. We have added a blinded chatbot-arena comparison of the different approaches and budgets. The human study gathers 320 comparisons from 16 participants. We included a few more model configurations than the ones suggested -- and compared every model against each other.
See our human study here: https://anonymous.4open.science/r/joint-diffusion-C978/rebuttal-figures/human_preference.png
This figure has been added to the paper (page 10). Our CRF-based joint improves the speed--quality frontier, far ahead of other joint modelling approaches:
Scores average equally over model opponents, counting ties as half. Our JoDD row is bolded.
We have now added code and math (MBPP/GSM8K).
Our method outperforms both a matched LoRA finetuning as well as the other joint approaches, thus achieving speedups across all settings. Notably, the other joint approaches do not reliably beat the finetuned backbone.
See our math/code results here (and added to the revised paper, page 8):
https://anonymous.4open.science/r/joint-diffusion-C978/rebuttal-figures/math_code_text.png
We performed the requested ablation on all four backbones: replacing the contextual gate with one shared learnable positive scalar, while keeping the remaining training and decoding settings fixed.
The contextual gate improves held-out reconstruction NLL in all 12 comparisons across three training seeds and four backbones:
Held-out reconstruction NLL (lower is better); mean ± standard deviation across three training seeds. Unit: nats per missing token
Albeit adding only 4097 parameters, the gate gives a reproducible performance improvement at a negligible cost! We will include this new ablation as an Appendix.