- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Score Entropy Discrete Diffusion (SEDD) predicts score ratios that may not be consistent with a clean-token posterior mass function. Empirically, the authors study how often trained SEDD models output inconsistent scores and show that projecting them onto the set of consistent scores improves generative performance. Motivated by this defect, the paper proposes mean-to-score (M2S): a model that directly predicts the clean-token posterior mean, which can be easily converted to the score. In image and text experiments, M2S shows competitive empirical performance.
Critical strengths
- Clear presentation and well-communicated results.
- The paper is generally easy to follow and the author's main findings and insights are clearly presented.
- Interesting theoretical insight and score-repair arguments.
- The authors study the image of the bridge , which is a polytope that lies strictly within the mentioned coordinate box and give theoretical arguments when the true score uniquely recovers the true posterior via a rank condition (Theorem 4.4).
- These findings suggest a straightforward repair for models that predict arbitrary scores via a projection on the polytope, which is then inverted to obtain the posterior.
- Competitive empirical results and extensive evaluation.
- The score repair experiments isolate the effect of lacking Bayes realizability and conclude that Bayes-inconsistent score predictions hurt generative performance of the considered SEDD checkpoint.
- Compared to other pure-uniform discrete diffusion methods, M2S outperforms them in both image and text modeling.
Critical weaknesses
- Posterior parameterizations that enforce Bayes realizability lack novelty.
- As mentioned by the authors, the fact that methods like SEDD are not Bayes realizable has been discussed before, as well as the proposed posterior parameterization: Shi et al. [1] write "In Lou et al. [32], the score is parameterized as a neural network without enforcing the constraint in (9). This means the learned reverse model can be incompatible with the forward process. We find that our parameterization, which enforces the constraint, leads to more stable training and better results."
- Campbell et al. [2] also use uniform corruptions together with a posterior parameterization: Their Proposition 1 gives essentially the posterior-to-score conversion in the author's Eq. 6. The main methodological difference seems to be the training loss.
- Unclear advantage of M2S compared to MD4.
- It is unclear what benefit M2S has when compared to MD4, which also predicts Bayes realizable scores by construction (as it recovers the M2S loss under absorbing diffusion). Further, MD4 outperforms M2S w.r.t. CIFAR-10 test BPD.
- Text experiments do not compare against MD4.
- The OpenWebText experiments in Figure 3/Table 2 do not show MD4. MD4 would be an especially relevant comparison, as it also enforces Bayes realizability, and outperforms M2S on CIFAR-10 (Table 1).
[1] Shi, J., Han, K., Wang, Z., Doucet, A. and Titsias, M., 2024. Simplified and generalized masked diffusion for discrete data. Advances in neural information processing systems, 37, pp.103131-103167.
[2] Campbell, A., Benton, J., De Bortoli, V., Rainforth, T., Deligiannidis, G. and Doucet, A., 2022. A continuous time framework for discrete denoising models. Advances in Neural Information Processing Systems, 35, pp.28266-28279.
Questions for authors
- Methods like MD4 or Campbell et al. output Bayes realizable scores by construction and show strong generative performance: What are the main advantages of M2S compared to them?
- Why is MD4 omitted from the experiment shown in Table 2/Figure 3?
Thank you for the detailed suggestions. We agree that the original main-table comparison was incomplete and that GenPPL alone cannot establish generative quality. We have added the two requested training controls, DUO with its ELBO objective and a full-posterior cross-entropy baseline, together with held-out OpenWebText variational estimates and explicit entropy measurements. These additions appear in Table 2 and the revised appendix. We address the remaining evaluation limitations and clarify the scope of our contribution below.
The two new baselines were trained from scratch with the M2S setting: the same approximately 170M-parameter backbone, OpenWebText split, GPT-2 tokenizer, length-512 context, global batch, optimizer, learning-rate schedule, EMA, and approximately 131B padded-token-position budget. Both completed 161,395 updates. The CE baseline retains M2S's softmax full-posterior head and uses unweighted denoising cross entropy over all training positions. This is the plain-CE control requested here; we distinguish it from a time-weighted posterior-ELBO objective.
We also account for the prediction-target distinction identified by Gourevitch et al.: the DUO plug-in ELBO targets a leave-one-out predictor, while ordinary denoising CE targets the full clean-token posterior. Accordingly, our DUO baseline uses its ELBO loss, and we convert its LOO output to a full posterior using the known forward kernel before applying the shared sampler. LOO describes the target of this DUO baseline; it is not a separately trained third baseline. This distinction follows Uniform Diffusion Models Revisited.
For a controlled comparison, all three models use the same Euler-linear implementation at 16, 32, 64, and 128 actual model calls, without an extra final denoising call. Gemma 2 9B GenPPL is:
Values are means ± sample standard deviations over five sampling seeds, each generating 1,024 sequences. Each newly trained baseline uses one training seed, so these deviations describe sampling variability, not training variability. M2S has lower GenPPL at all four budgets under this protocol. In particular, the CE comparison controls the posterior head as well as the backbone and sampler. It supports a benefit of the M2S training objective for this measured outcome, but does not by itself establish better distributional quality.
The matched DUO experiment evaluates its ELBO under the M2S training recipe. It does not reproduce the additional Gaussian curriculum or distillation in The Diffusion Duality, or the predictor-corrector samplers in Chapter II. The earlier appendix checkpoint, trained for 524B tokens at length 1024, remains a separate comparison. We therefore do not interpret the matched Euler result as a comparison against every DUO configuration. We have not completed a separate matched UDLM run, and do not relabel DUO or CE as that baseline.
We agree that likelihood-related results should also be reported for OWT, given the image BPD evaluation. The revised appendix now includes a common held-out finite-grid variational evaluation of all four final EMA models. It uses all 21,990 held-out length-512 sequences, includes EOS padding in the denominator, and includes reconstruction, bridge KL terms over 128 reverse intervals, and the terminal-prior KL. We use 16 Monte Carlo replicates per document.
These are variational estimates for the specified finite-grid models, not exact continuous-time likelihoods or external-model GenPPL. They are also not directly comparable to packed-OWT literature perplexities. The appendix reports Monte Carlo uncertainty and the terminal-prior conventions; specifically, the MD4 likelihood calculation uses a full-support prior, while its native generation starts from all MASK. MD4 and DUO obtain lower estimated variational PPL than M2S. We report this ordering explicitly: the GenPPL advantage does not imply an advantage on every metric, and variational estimates themselves can differ in tightness.