acceptodds
Under review as a conference paper at ICLR 2027

Mean-to-Score discrete diffusion: posterior-mean denoisers for score entropy

Abstract

Score Entropy Discrete Diffusion (SEDD) parameterizes discrete reverse processes with unconstrained positive score ratios. Although positivity ensures nonnegative reverse jump rates, it does not ensure Bayes realizability: at a fixed noisy state, all candidate score ratios must arise jointly from a single clean-token posterior under the forward kernel. We show that the score-entropy loss has the correct population optimum but does not enforce Bayes realizability away from it. In a trained pure-uniform SEDD checkpoint, roughly one quarter of complete score vectors violate the coordinate box, while more than half satisfy every coordinate bound but remain materially incompatible with any valid clean-token posterior. Although the corresponding continuous-time reverse jump rates remain nonnegative, these violations can induce negative pre-normalization weights in the finite-step sampler update. Projecting the checkpoint's raw scores onto the bridge polytope removes all observed negative weights and lowers external generative PPL from to without changing the sampler. To enforce Bayes realizability by construction rather than through post-hoc projection, we introduce mean-to-score (M2S): the network predicts a clean-token posterior mean and converts it to the score through an exact kernel-dependent linear map. The map applies to any known coordinate-wise continuous-time Markov chain (CTMC) satisfying a mild support condition. For uniform corruption, it maps the probability simplex onto the bridge polytope; for absorbing-mask corruption, the resulting objective recovers MD4 exactly. In a controlled 28.4M-parameter CIFAR-10 comparison, M2S lowers test BPD from to and FID-50k from to . A 170M-parameter M2S model trained on approximately 131B OpenWebText token slots outperforms the evaluated pure-uniform SEDD, GIDD, and Neural CTMC checkpoints at every tested sampling budget, reaching mean generative PPL at 128 steps compared with for the strongest pure-uniform baseline.

Primary area:
generative models
Keywords:
discrete diffusion models, large language model, posterior-mean denoising, continuous-time Markov chains

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_21bvOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Score Entropy Discrete Diffusion (SEDD) predicts score ratios that may not be consistent with a clean-token posterior mass function. Empirically, the authors study how often trained SEDD models output inconsistent scores and show that projecting them onto the set of consistent scores improves generative performance. Motivated by this defect, the paper proposes mean-to-score (M2S): a model that directly predicts the clean-token posterior mean, which can be easily converted to the score. In image and text experiments, M2S shows competitive empirical performance.

Critical strengths
  • Clear presentation and well-communicated results.
    • The paper is generally easy to follow and the author's main findings and insights are clearly presented.
  • Interesting theoretical insight and score-repair arguments.
    • The authors study the image of the bridge BB, which is a polytope that lies strictly within the mentioned coordinate box and give theoretical arguments when the true score uniquely recovers the true posterior via a rank condition (Theorem 4.4).
    • These findings suggest a straightforward repair for models that predict arbitrary scores via a projection on the polytope, which is then inverted to obtain the posterior.
  • Competitive empirical results and extensive evaluation.
    • The score repair experiments isolate the effect of lacking Bayes realizability and conclude that Bayes-inconsistent score predictions hurt generative performance of the considered SEDD checkpoint.
    • Compared to other pure-uniform discrete diffusion methods, M2S outperforms them in both image and text modeling.
Critical weaknesses
  • Posterior parameterizations that enforce Bayes realizability lack novelty.
    • As mentioned by the authors, the fact that methods like SEDD are not Bayes realizable has been discussed before, as well as the proposed posterior parameterization: Shi et al. [1] write "In Lou et al. [32], the score is parameterized as a neural network without enforcing the constraint in (9). This means the learned reverse model can be incompatible with the forward process. We find that our parameterization, which enforces the constraint, leads to more stable training and better results."
    • Campbell et al. [2] also use uniform corruptions together with a posterior parameterization: Their Proposition 1 gives essentially the posterior-to-score conversion in the author's Eq. 6. The main methodological difference seems to be the training loss.
  • Unclear advantage of M2S compared to MD4.
    • It is unclear what benefit M2S has when compared to MD4, which also predicts Bayes realizable scores by construction (as it recovers the M2S loss under absorbing diffusion). Further, MD4 outperforms M2S w.r.t. CIFAR-10 test BPD.
  • Text experiments do not compare against MD4.
    • The OpenWebText experiments in Figure 3/Table 2 do not show MD4. MD4 would be an especially relevant comparison, as it also enforces Bayes realizability, and outperforms M2S on CIFAR-10 (Table 1).

[1] Shi, J., Han, K., Wang, Z., Doucet, A. and Titsias, M., 2024. Simplified and generalized masked diffusion for discrete data. Advances in neural information processing systems, 37, pp.103131-103167.

[2] Campbell, A., Benton, J., De Bortoli, V., Rainforth, T., Deligiannidis, G. and Doucet, A., 2022. A continuous time framework for discrete denoising models. Advances in Neural Information Processing Systems, 35, pp.28266-28279.

Questions for authors
  • Methods like MD4 or Campbell et al. output Bayes realizable scores by construction and show strong generative performance: What are the main advantages of M2S compared to them?
  • Why is MD4 omitted from the experiment shown in Table 2/Figure 3?
Reviewer_vAqiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Studies the score parameterization for discrete diffusion models, focusing on uniform variants. They introduce the notion of Bayes Realizability -- whether an outputted score from the network can be used to obtain a valid posterior over the vocabulary. Using this definition, they find that the network often outputs scores that do not lead to a valid posterior / distribution. To fix this, they propose decoupling the parameterization from the objective -- the network outputs a posterior, but then this is converted into a score that is used with the score based objective (SEDD). They demonstrate that this improves uniform diffusion in terms of generative quality (Cifar 10 experiment, OWT gen ppl results) and likelihood evals.

Critical strengths

The biggest strength of this paper is the explanation of why SEDD underperforms. After reading their definition of Bayes Identifiability, and looking at the diagnostic results of SEDD in terms of this, it becomes a lot more clear why it has suffered relative to newer approaches. I also liked how it showed a way to get around this without defaulting to the plug in parameterization or a plain cross entropy loss.

Critical weaknesses
  1. Baselines -- duo is notably absent in the papers experiments and comparisons. This is confusing, as duo is a very well known uniform baseline, which has been examined both for images and for text [1, 2]. Furthermore, it can be seen as an alternative to the mean / posterior parameterization, as the optimum for the Duo ELBO is a leave one out denoiser (LOO) [3]. Furthermore, the simplest baseline is not tested -- if the model is parameterized as a posterior, why not just train with cross entropy, as investigated in [3]? Such a comparison would make the paper much stronger. It would also be good to compare against UDLM from [4], although it is not as common anymore.

[1] The Diffusion Duality. Sahoo et al. 2025. [2] The Diffusion Duality, Chapter II: Ψ-Samplers. Deschenaux et al. 2026. [3] Uniform Diffusion Models Revisited: Leave-One-Out Denoiser and Absorbing State Reformulation. Gourevitch et al. 2026 [4] Simple Guidance Mechanisms for Discrete Diffusion Models. Schiff et al. 2024

  1. Experimental Methodology. The experiment section feels weak -- first, there are no likelihood results reported for OWT. On its own, this is fine as likelihood metrics can be misleading / not adequately represent generative quality. But if this is the case, then why report BPD in table 1? I would advise the authors to include likelihood for OWT for completeness, since likelihood metrics are already used for images. Furthermore, the treatment of generative quality for text is not up to the standards the field has been moving in. As shown in [1, 2, 3], generative perplexity is generally a poor measure of generative quality -- overly repetitive samples can lead to very low perplexity. For this reason, the field has started to report generative frontiers as in [1, 4, 5, 6] -- these allow us to compare methods at matched entropy, which is at least more fair. Even more recent methods like [2, 3, 7] introduce metrics that are better than entropy / generative perplexity frontiers. Finally, it would be good to include the tiny gsm experiment from [7] as it is good benchmark for capturing whether the method can tackle difficult reasoning / coding tasks (of course, matched towards the 170 million parameter scale).

[1] Generative Frontiers: Why Evaluation Matters for Diffusion Language Models. Pynadath et al. 2026. [2] Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs New Metrics. Franca et al. 2026 [3] SOL: Measuring Gaps between Text Distributions by Double Sliced Wasserstein Metrics. Kornhardt et al. 2026 [4] Spherical Flows for Sampling Categorical Data. Chemseddine et al. 2026. [5] Language Modeling with Hyperspherical Flows. Deschenaux et al. 2026. [6] Uniform Diffusion Models Revisited: Leave-One-Out Denoiser and Absorbing State Reformulation. Gourevitch et al. 2026 [7] Beyond Single Tokens: Distilling Discrete Diffusion Models via Discrete MMD. Hoogeboom et al. 2026

  1. Scope. The scope of the paper feels a bit narrow for an ICLR paper -- the core point is that the score parameterization may lead to invalid distributions, and that the parameterization should be such that the implied posterior is always a valid distribution. While this is an interesting insight, it only applies to the score parameterization -- it does not apply towards the plug in parameterization of Duo or a posterior parameterization.
Questions for authors

Clarifications:

  1. What is the benefit for training with the score objective instead of the objectives in [1, 2, 3]? The motivation for fixing the SEDD objective is clear, but I do not see the point in staying with the score objective.

Suggestions.

  1. (Experiments) I would highly recommend including duo (ELBO) AND a cross entropy baseline. I would also recommend including some likelihood metrics for OWT, as well as better generative quality metrics.

  2. (Scope) I would read [3] carefully as it is a good example of a paper that studies existing approaches. While it is entirely focused on studying the existing ELBO objective and cross entropy objective of uniform diffusion, it has a very concrete theoretical insight that is non trivial, it has extensive experiments understanding the impacts of this performance in terms of performance, and shows that the result has larger implications than just fixing a method (showing that uniform is not very different from mask, given proper parameterization).

[1] The Diffusion Duality. Sahoo et al. 2025. [2] Simple Guidance Mechanisms for Discrete Diffusion Models. Schiff et al. 2024 [3] [Uniform Diffusion Models Revisited: Leave-One-Out Denoiser and Absorbing State Reformulation. Gourevitch et al. 2026

1 comment

Thank you for the detailed suggestions. We agree that the original main-table comparison was incomplete and that GenPPL alone cannot establish generative quality. We have added the two requested training controls, DUO with its ELBO objective and a full-posterior cross-entropy baseline, together with held-out OpenWebText variational estimates and explicit entropy measurements. These additions appear in Table 2 and the revised appendix. We address the remaining evaluation limitations and clarify the scope of our contribution below.

The two new baselines were trained from scratch with the M2S setting: the same approximately 170M-parameter backbone, OpenWebText split, GPT-2 tokenizer, length-512 context, global batch, optimizer, learning-rate schedule, EMA, and approximately 131B padded-token-position budget. Both completed 161,395 updates. The CE baseline retains M2S's softmax full-posterior head and uses unweighted denoising cross entropy over all training positions. This is the plain-CE control requested here; we distinguish it from a time-weighted posterior-ELBO objective.

We also account for the prediction-target distinction identified by Gourevitch et al.: the DUO plug-in ELBO targets a leave-one-out predictor, while ordinary denoising CE targets the full clean-token posterior. Accordingly, our DUO baseline uses its ELBO loss, and we convert its LOO output to a full posterior using the known forward kernel before applying the shared sampler. LOO describes the target of this DUO baseline; it is not a separately trained third baseline. This distinction follows Uniform Diffusion Models Revisited.

For a controlled comparison, all three models use the same Euler-linear implementation at 16, 32, 64, and 128 actual model calls, without an extra final denoising call. Gemma 2 9B GenPPL is:

Method163264128
DUO (Euler)1326.529 ± 49.056345.236 ± 6.199230.276 ± 6.312194.217 ± 4.547
Posterior CE (Euler)1178.519 ± 28.505342.220 ± 7.643235.947 ± 5.252197.514 ± 3.960
M2S (Euler)811.714 ± 35.608257.325 ± 2.888181.785 ± 3.697158.820 ± 2.885

Values are means ± sample standard deviations over five sampling seeds, each generating 1,024 sequences. Each newly trained baseline uses one training seed, so these deviations describe sampling variability, not training variability. M2S has lower GenPPL at all four budgets under this protocol. In particular, the CE comparison controls the posterior head as well as the backbone and sampler. It supports a benefit of the M2S training objective for this measured outcome, but does not by itself establish better distributional quality.

The matched DUO experiment evaluates its ELBO under the M2S training recipe. It does not reproduce the additional Gaussian curriculum or distillation in The Diffusion Duality, or the predictor-corrector samplers in Chapter II. The earlier appendix checkpoint, trained for 524B tokens at length 1024, remains a separate comparison. We therefore do not interpret the matched Euler result as a comparison against every DUO configuration. We have not completed a separate matched UDLM run, and do not relabel DUO or CE as that baseline.

We agree that likelihood-related results should also be reported for OWT, given the image BPD evaluation. The revised appendix now includes a common held-out finite-grid variational evaluation of all four final EMA models. It uses all 21,990 held-out length-512 sequences, includes EOS padding in the denominator, and includes reconstruction, bridge KL terms over 128 reverse intervals, and the terminal-prior KL. We use 16 Monte Carlo replicates per document.

ModelEstimated NELBO, nats per padded positionEstimated variational PPL
M2S (uniform)2.85805817.428
DUO2.84241517.157
Posterior CE2.87323917.694
MD4 (mask)2.77282416.004

These are variational estimates for the specified finite-grid models, not exact continuous-time likelihoods or external-model GenPPL. They are also not directly comparable to packed-OWT literature perplexities. The appendix reports Monte Carlo uncertainty and the terminal-prior conventions; specifically, the MD4 likelihood calculation uses a full-support prior, while its native generation starts from all MASK. MD4 and DUO obtain lower estimated variational PPL than M2S. We report this ordering explicitly: the GenPPL advantage does not imply an advantage on every metric, and variational estimates themselves can differ in tightness.

Reviewer_XhXrOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the parameterization of SEDD, which is based on unconstrained positive score ratios. This choice does not enforce what in this paper is called Bayes realizability, which requires that, at a fixed noisy state, all candidate score ratios come jointly from a single clean-token posterior under the forward process. In order to solve this problem, they propose M2S, in which the network predictsaclean-tokenposteriormeanandthenconvertsitintothescoresthroughakernel-dependent linear map (depending on the type of corruption kernel used in the forward process, e.g., uniform or absorbing). Finally, they test their method in a 28.4M-parameter CIFAR-10 comparison and on OpenWebText with a 170M-parameter model trained on 131B tokens, showing lower test BPD and FID-50k on the former, and lower generative PPL than other baselines, such as SEDD, GIDD and Neural CTMC, on the latter.

Critical strengths

The paper points out the interesting problem of Bayes realizability. In fact, as shown, SEDD’s score vectors along its own trajectories are almost never induced by a valid posterior, and a quarter leave even the coordinate box. This is a concrete, falsifiable property of a widely used model that nobody had measured.

Moreover, the recovery of MD4 is also very nice. This positions M2S as the uniform-kernel counterpart of the x0-parameterization for the masked case, connecting it to the RADD/MD4 line of work.

Critical weaknesses

M2S changes the admissible score set, the output head (softmax vs. exponentiated) and, on Open- WebText, the sampler (Bayes-cosine vs. Euler) all at once, so the gains cannot be credited to Bayes realizability alone; Figure 2(c)–(d), where M2S reaches a lower value of the same objective despite a smaller hypothesis class, points to an optimization effect of the head.

Moreover, GIDD-uniform is Bayes-realizable by construction yet is the worst uniform model in Table 2, which contradicts the proposed mechanism and is not discussed. Since the paper’s central claim is that Bayes realizability drives the improvements, this unresolved attribution is what determines my recommendation.

Questions for authors

Could the authors train the same softmax posterior head, backbone, data and sampler with the standard cross-entropy ELBO under the uniform kernel (as in D3PM/GIDD) instead of the bridged score-entropy loss, and report GenPPL at all four budgets? This would separate the effect of the objective from that of the posterior parameterization and would explain the GIDD-uniform result. Could Table 2 be reported with a matched sampler, i.e. M2S with Euler-linear and/or projected SEDD with the Bayes-cosine sampler, which applies directly since µproj is a valid posterior?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.