acceptodds
Under review as a conference paper at ICLR 2027

What Does a Discrete Diffusion Model Learn? One Reverse Process in Three Coordinates

Abstract

What does a discrete diffusion model actually learn? The literature trains denoisers, bridge plug-ins, and concrete scores, each deriving its own ELBO objective under different conventions. This fragmentation makes the object being learned unclear and ELBO values difficult to interpret and compare. All three methods learn the same object in different coordinates: the exact time-reversal of the noising process. We first prove the Oracle Distance identity: for any noising process, the negative ELBO is not just a bound on the likelihood; it equals the data entropy plus the path KL divergence from the reversed noising process to the learned one. Next, for token-factorizing noise, we show that denoiser, bridge plug-in, and concrete score are three coordinates of this reverse process, derive exact conversions between them, and identify the probability distribution learned by each. In particular, contrary to its standard interpretation, the bridge plug-in learns not a denoiser but the cavity distribution: the clean token given the noisy sequence excluding its own position. We confirm the theory by training all three coordinates with masked, uniform, and GIDD kernels on OpenWebText under a shared recipe, and by training uniform-diffusion solvers on four logic tasks. The learned coordinates agree after conversion, while confusing the cavity with the denoiser can severely degrade sampling quality. Together with exact boundary-term formulas, closed-form NELBO calibrations at initialization, and a clock interpretation of time schedules, these results provide a conceptual framework and a mechanical recipe for deriving, implementing, and comparing discrete diffusion processes.

Primary area:
generative models
Keywords:
discrete diffusion, ELBO, continuous-time Markov chain (CTMC), denoiser, concrete score, path KL divergence, masked diffusion, uniform diffusion

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_U7RZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates what object is identified by the training objectives of discrete diffusion models. The authors decompose the expected negative ELBO into the data entropy and a path-space KL divergence from the oracle reverse process to the learned reverse process. Under token-factorizing corruption, they characterize the denoiser, bridge plug-in, and concrete score as three representations of the same reverse process. An important conclusion is that the optimal prediction target for the bridge plug-in is the conditional distribution of a clean token given the noisy sequence excluding that token’s own noisy observation, termed the cavity distribution.

Critical strengths
  1. The paper brings several formulations into a common mathematical framework. Expressing different training objectives through reverse jump rates helps clarify how network outputs define a generative process.

  2. The experiments cover multiple noise kernels and include both language modeling and structured reasoning tasks, providing empirical support for the framework across different settings.

Critical weaknesses
  1. The new contributions need to be distinguished more clearly from existing results. The closely related work already discusses the cavity distribution and conversions between parameterizations. The authors should discuss more explicitly which results are new and how they extend previous findings.

  2. A shared oracle optimum does not isolate trainability as the only source of empirical differences. Section 5.3 argues that differences between coordinates must reflect trainability because they share the same NELBO floor. This conclusion is too strong and need more discussions.

Questions for authors

How much does the score-to-probability correction affect generation quality?

2 comments

We thank the reviewer for their feedback.

W1. New contributions against existing results. For a discussion on novelty, we kindly refer to the general response.

W2. Trainability as the only source of differences. §5.3 now says what trainability covers: optimization, finite capacity and, for the score, the normalization that the head must learn. We also state the equivalence only "at the optimum" wherever we invoke it. The differences between heads also hold at a tuned learning rate and in two seeds.

Q1. Effect of the score-to-probability correction on generation quality. The correction helps rather than hurts the score models: it lowers the NELBO for every kernel and in both seeds, by a quarter to a half of the gap to the cavity (new Table 14). We measured its effect on the likelihood because our samplers take a distribution, so a score head can only be sampled after the correction. The reported sample quality therefore already includes it. Under masked diffusion, where the correction is only a normalization, it accounts for half of the NELBO gap to the cavity, and the sample quality of the score is close to the cavity's. On the other kernels the score remains worse.

Presentation. We also revised the text for clarity, most visibly in §5.3 and in the related work.

Dear Reviewer, the author-reviewer discussion ends on 18 November (AoE). We are happy to answer any further questions before then. Thank you in advance for your time.

Reviewer_px9tOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a unified theoretical framework for understanding discrete diffusion models through three parameterizations of the same reverse process: denoiser, bridge plug-in (cavity), and concrete score. The key idea is to connect these parameterizations through their corresponding reverse transition rates and to interpret the negative evidence lower bound (NELBO) as the data entropy plus a KL divergence between the oracle and learned reverse trajectory distributions, under appropriate conditions.

The paper further derives an optimal reverse-rate projection and explains why the bridge plug-in parameterization learns a cavity distribution rather than the conventional denoiser target. In particular, the denoiser and cavity coincide in the relevant masked-diffusion states but generally differ under uniform diffusion. Experiments on OpenWebText and four structured reasoning tasks investigate the effects of parameterization mismatch, initialization calibration, and sampling choices. The results show that incorrectly treating a cavity prediction as a denoiser can substantially degrade generation quality under uniform and GIDD diffusion.

Critical strengths
  1. The paper provides a coherent interpretation of NELBO by relating it to the KL divergence between oracle and learned reverse trajectories. This helps distinguish irreducible data uncertainty from errors introduced by the learned reverse process.

  2. The distinction between denoiser and cavity targets is useful for understanding discrete diffusion objectives. In particular, the paper explains why the two targets coincide in the relevant masked-diffusion states but generally differ under uniform diffusion.

  3. The coordinate-mismatch experiments provide clear evidence that incorrectly interpreting a cavity prediction as a denoiser can severely degrade generation quality under uniform and GIDD diffusion.

  4. The initialization calibration results and implementation-oriented analysis may be useful for diagnosing errors in discrete diffusion training objectives.

Critical weaknesses
  1. Limited theoretical novelty. The main theoretical components rely substantially on established concepts, including continuous-time Markov chain reversal, path-space KL divergence, conditional-expectation projections under Bregman divergences, and Bayesian transformations between conditional distributions. The paper combines these concepts into a coherent framework, but the incremental theoretical contribution beyond closely related work is not sufficiently clear. The authors need to identify precisely which results are genuinely new and explain why the combination provides a substantial advance rather than primarily a reformulation of existing theory.

  2. Insufficient differentiation from related work. The discussion of cavity or leave-one-out targets in uniform diffusion and existing path-KL interpretations of diffusion objectives raises questions about the originality of the central claims. The paper should provide a more precise theorem-by-theorem comparison with the closest prior work, especially concerning the reverse-rate projection, the Oracle Distance identity, and the correspondence among the three parameterizations.

  3. Limited robustness of the empirical comparisons. Each text-model configuration is trained with only one random seed. Consequently, the observed performance differences among denoiser, cavity, and score parameterizations may be affected by optimization variability. The experiments do not establish that the reported ranking is robust to different random seeds or reasonable parameterization-specific hyperparameter tuning.

  4. The experimental scope is narrower than the theoretical scope. The text experiments cover masked, uniform, and two GIDD settings, while the structured reasoning experiments focus on uniform diffusion and coordinate mismatch. These experiments support the importance of correct parameterization conversion, but they do not comprehensively validate the practical equivalence of the three parameterizations across different tasks and noise processes.

  5. Limited evidence for broader practical advantages. The paper demonstrates that incorrect coordinate conversion can be harmful, but this result does not establish that the proposed unified framework improves generation quality, training efficiency, or sampling efficiency over existing correctly implemented approaches. The practical benefits of the framework therefore remain somewhat limited relative to the breadth of its theoretical presentation.

Questions for authors
  1. The paper relates NELBO to path-space KL divergence and connects denoiser, cavity, and concrete-score parameterizations. Could the authors provide a precise comparison with the closest prior work, identifying which theorems or technical results are genuinely new? In particular, what substantive theoretical advance is provided beyond existing path-KL interpretations and cavity or leave-one-out formulations?

  2. The paper derives the optimal reverse rate using a conditional-expectation projection. Since conditional expectations are standard optimal predictors under suitable Bregman divergences, what is the novel theoretical insight in this derivation beyond applying the existing projection principle to discrete diffusion reverse rates?

  3. The text experiments use one training seed per configuration. Could the authors provide multi-seed results or additional evidence showing that the observed differences among parameterizations are robust to optimization variability and reasonable hyperparameter tuning?

  4. The text experiments optimize the path-integral term while evaluating the complete finite-window NELBO. Could the authors clarify whether this distinction affects the comparisons among parameterizations and provide evidence that the main conclusions remain unchanged when the complete objective is optimized?

  5. The experiments demonstrate severe degradation when cavity predictions are incorrectly treated as denoiser predictions. Beyond this implementation mismatch, what evidence shows that the proposed unified framework provides practical advantages over existing methods that already use the correct parameterization and reverse-rate conversion?

3 comments

We thank the reviewer for their feedback. The additional seeds, the tuned learning rate and the result-by-result comparison with prior work are now part of the experiments (§5, App. G.1) and the related work (§6, App. H) of the submission.

W1–W2/Q1. Which results are new, theorem by theorem. We kindly refer the reviewer to the general response for our main reply on novelty. Here we add the requested comparison for the three results the reviewer names, which we have added to the related work of the revised submission:

  • Oracle Distance (Thm. 1): prior work has the path-KL bound up to a constant (Campbell et al., 2022; Generator Matching, Holderrieth et al., 2025). New: we give the identity, NELBO = data entropy + path KL to the oracle reverse process, for any model rates and on clipped windows with both boundary terms (Thm. C.1).
  • Reverse-rate projection (Cor. 1): we agree it is well known as a general principle (Campbell et al. Prop. 1, Generator Matching Prop. 1–2). New: applied to the path KL of Thm. 1, it gives the ELBO optimum of every head in one argument and shows that the excess NELBO over the optimum is exactly the path KL to the oracle.
  • Three parameterizations (Thm. 2): each prior result covers one head, one objective or one kernel family, for example the score in SEDD (Lou et al., 2024), the cavity under ratio matching in Sun et al. (2023), and the cavity in discrete time for a static-mixture distribution in Gourevitch et al. (2026). New: we show that the three optima are the same object in three coordinates for every token-factorizing kernel, including time-varying mixtures such as GIDD (von Rütte et al., 2025).

Q2. What does the projection derivation add beyond the standard principle? We agree that the projection principle is well known. What is new is that applying this one principle to token-factorizing processes subsumes many of the known results and derivations: starting from the path KL, a single projection now gives the ELBO optimum of every head and kernel, where earlier work derived each optimum separately, with ad hoc hypotheses and derivations. It then easily follows that the optimum of the plug-in ELBO is the cavity, not the denoiser as it is usually described. The works that train the plug-in were missing the distinction, and it also explains published findings that were until now purely empirical.

W3/Q3. Single seeds and untuned hyperparameters. We tuned the learning rate of each of the 11 configurations over {1e-4, 3e-4, 1e-3, 3e-3}, and trained a second seed of each at the selected rate (3e-4 for all). All text results now come from these runs, with the standard error over the two seeds. The original conclusions remain unchanged and are now strengthened: the score trails the cavity and the denoiser at every learning rate, on every kernel and in both seeds, by several times the difference between seeds. The denoiser and the cavity stay close, and neither is better overall.

W4. Experiments narrower than the theory. The logic tasks were not designed to test the equivalence of the three heads. Instead, with a fixed kernel, they measure how much the choice between cavity and denoiser affects sampling across different tasks. We test the equivalence of heads on text only, across the four kernels (Table 3). Additionally, the revised submission now says "at the optimum" wherever it states the equivalence.

Q4. Does training on the path term alone affect the comparisons? No. We did not train on the complete objective, but it is not needed for two reasons:

  1. The reconstruction term has the same minimizer as the path term. It is the cross-entropy of the denoiser at the single time t1t_1, minimized by the true posterior q(z0i∣zt1)q(z_0^i \mid z_{t_1}), which the minimizer of the path term already equals at every time tt, so the reconstruction term would only add a second loss with the same minimizer at t1t_1. The prior term does not depend on the head.
  2. After training, both terms are negligible for our choice of kernels and window [t1,t2][t_1, t_2] (under 0.1% of the NELBO).

W5/Q5. Practical advantages beyond the mismatch experiment. Since this question is of interest to all reviewers, we kindly refer to the general response.

References

  • Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. A Continuous Time Framework for Discrete Denoising Models. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, 2022.
  • Samson Gourevitch, Yazid Janati, Dario Shariatian, Umut Simsekli, Eric Moulines, Eric P. Xing, and Alain Durmus. Uniform Diffusion Models Revisited: Leave-One-Out Denoiser and Absorbing State Reformulation. arXiv:2605.22765, 2026.
  • Peter Holderrieth, Marton Havasi, Jason Yim, Neta Shaul, Itai Gat, Tommi Jaakkola, Brian Karrer, Ricky T. Q. Chen, and Yaron Lipman. Generator Matching: Generative Modeling with Arbitrary Markov Processes. In International Conference on Learning Representations (ICLR), 2025.
  • Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution. In Proceedings of the 41st International Conference on Machine Learning (ICML), volume 235 of Proceedings of Machine Learning Research, pp. 32819–32848. PMLR, 2024.
  • Haoran Sun, Lijun Yu, Bo Dai, Dale Schuurmans, and Hanjun Dai. Score-based Continuous-time Discrete Diffusion Models. In International Conference on Learning Representations (ICLR), 2023.
  • Dimitri von Rütte, Janis Fluri, Yuhui Ding, Antonio Orvieto, Bernhard Schölkopf, and Thomas Hofmann. Generalized Interpolating Discrete Diffusion. In Proceedings of the 42nd International Conference on Machine Learning (ICML), volume 267 of Proceedings of Machine Learning Research, pp. 61810–61843. PMLR, 2025.

Dear Reviewer, the author-reviewer discussion ends on 18 November (AoE). Following your review, we added the theorem-by-theorem comparison with prior work to the discussion in the related work section of the paper, and a learning-rate sweep plus a second seed for each of the 11 OWT training runs. We would be grateful to know whether our answers and changes to the paper address your concerns, and we are happy to answer any further questions. Thank you in advance for your time.

Reviewer_dxjkOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies what discrete diffusion models actually learn under different parameterizations. It shows that the denoiser, bridge plug-in, and concrete score can be understood as different representations of the same reverse process. The main theoretical results include an identity relating the NELBO to data entropy and a reverse process path KL, a characterization of the optimal reverse rates, and conversions between the three parameterizations. In particular, the authors show that the bridge plug-in learns a leave-one-out "cavity" distribution rather than usual denoiser. The theory is further validated through synthetic experiments, language modeling (on OpenWebText), as well as four logic tasks.

Critical strengths

[S1] Useful unifying perspective and a nice clarification of what different parameterizations learn. I liked the distinction between denoiser and cavity, and that the authors derive their relationship instead of just treating the different objectives separately. The observation that a bridge plug-in actually learns the cavity distribution is useful, particularly because these heads are often interpreted as clean-token predictors. The boundary-term analysis and initialization identities can have meaningful practical implications.

[S2] Fairly thorough empirical validation of the theory. The paper evaluates different corruption processes and parameterizations under a shared training setup, and includes both language modeling and logic tasks. I particularly liked the experiments showing what happens when a cavity head is incorrectly interpreted as a denoiser, where the performance degradation can be very large. The head-agreement and own-token sensitivity experiments also help validate the theoretical story.

Critical weaknesses

[W1] The theoretical novelty seems somewhat limited relative to existing literature. While the unified presentation is useful, several central ingredients are already quite well understood. The path-KL interpretation follows from standard variational and CTMC arguments, and the reverse-rate projection is closely related to Generator Matching. Sun et al. already develop the cavity representation (even if they don't term it that way), while Gourevitch et al. (2026) obtain closely related results for uniform diffusion. A recent paper "Simple Diffusion Language Models Are More Effective Few-Step Generators Than Reported" (Amin et al., 2026) also gives a more detailed theoretical treatment of the finite-step factorization error discussed around Eq. (14), although this is contemporaneous work and should not count against the submission for missing the comparison. Overall, I see the contribution more as a useful synthesis and clarification than a fundamentally new theoretical understanding of discrete diffusion. The cavity ELBO characterization is probably the most interesting part, but the paper should be more precise about what is new beyond the existing results.

[W2] Some of the empirical conclusions about the equivalence of parameterizations are stronger than what is actually demonstrated. The three heads agree at the population optimum, but this does not necessarily mean that their learned versions or finite-step samplers are equivalent. In particular, App G.1 normalizes and sometimes clips the score head when converting it for sampling, meaning the sampled reverse process can differ from the one used to compute the score NELBO. Even for masked diffusion, the conversion is only rate-preserving if the score has the correct normalization (see Eq. E.10), so the statement in footnote 4 that the inverse is exact doesnt necessarily seem true. This makes it harder to attribute the worse score-model performance purely to trainability. Also, the text experiments are run just once without confidence intervals, which limits conclusions about the relatively small differences between the correctly interpreted heads.

Questions for authors
  1. Can the authors clarify the novelty relative to Sun et al., Gourevitch et al., and Holderrieth et al.? In particular, what would they consider the main new theoretical insight beyond putting these existing connections into a common framework?

  2. Could the authors clarify the score-to-cavity conversion used for sampling? Since the learned scores need not correspond to a normalized distribution, does the conversion change the reverse rates? For masked diffusion, how much of the score NELBO gap comes from the normalization slack in Eq. (E.10), and does the sampling gap remain when this is controlled for?

  3. Is there a concrete example where the cavity interpretation identifies an actual mistake or improves an existing published implementation, rather than primarily changing how we interpret its learned head? This would help establish the practical significance of the result.

3 comments

We thank the reviewer for their feedback.

W1/Q1. Novelty and the main new insight. For the novelty of the framework and its main new insight, we kindly refer to the general response. The related work now states what we add to each of the closest papers:

  • Sun et al. (2023): the authors train on a ratio-matching loss, whose minimizer is the cavity. We show that the cavity is also the ELBO optimum of the bridge plug-in.
  • Gourevitch et al. (2026, concurrent): the authors prove their results for the discrete-time ELBO with a static-mixture distribution. Ours follow from a single projection and hold in continuous time for every token-factorizing kernel, including time-varying mixtures such as GIDD, to which their derivations do not transfer.
  • Generator Matching (Holderrieth et al., 2025): the authors give the projection for general Markov processes, which is our Cor. 1 and is now credited. They leave the constant of the path-KL bound unidentified and do not derive the ELBO optimum of any head.
  • Campbell et al. (2022): the authors relate the ELBO to a path KL up to a constant and without boundary terms. We give the identity with the data entropy and both boundary terms, on the clipped time windows used in practice.

We now also cite Amin et al. (2026) for the finite-step factorization error (Eq. 14).

W2. Equivalence claims stronger than the evidence for trained heads. The equivalence is a statement about the optimum, and the revised submission now says "at the optimum" wherever it states it. For trained heads we measure the agreement: after conversion, two heads agree as closely as two seeds of the same head (Table 3). Regarding sampling, our sampler first converts each head to the denoiser and then runs it through the same ancestral step, so the samples differ only through the learned head.

W2/Q2. Score-to-cavity conversion and footnote 4. We agree about footnote 4: the conversion clips the distribution only under the non-masked kernels, but it normalizes it under every kernel, and it is thus also not exact for masked diffusion. We have corrected the footnote and define the conversion precisely in the new App. G.5. On the specific questions:

  • Does the conversion change the reverse rates? Yes, but it helps rather than hurts performance: we have evaluated the NELBO of the head after the clipping+renormalization conversion (new Table 14). There we show that the conversion lowers the NELBO for every kernel, by a quarter to a half of the gap to the cavity, thus benefiting a score model.
  • How much of the masked NELBO gap is the slack of Eq. (E.10)? About half. Here the conversion only normalizes, so the raw and the converted NELBO differ by exactly this slack.
  • Does the sampling gap remain when the normalization is controlled for? Yes. All samples were already drawn from the converted head, so the reported gap is already the one that remains. Under masked diffusion the quality–entropy curves of the score and the cavity nearly coincide, and on the other kernels the score remains worse.
  • Attributing the score's worse performance to trainability alone. §5.3 now states in more detail what trainability covers: optimization, finite capacity and, for the score, the normalization.

W2. Single runs without confidence intervals. We tuned the learning rate of each of the 11 configurations over {1e-4, 3e-4, 1e-3, 3e-3} and trained a second seed at the selected rate. All text results now come from these runs, with the standard error over the two seeds. The score's lag behind the cavity is several times the difference between seeds and consistent across kernels, learning rates, and seeds. The denoiser and the cavity are much closer, and neither is clearly better overall.

Q3. A concrete mistake or improvement in a published implementation. Since this question is of interest to all reviewers, we kindly refer to the general response.

References

  • Hasan Amin, Ming Yin, and Rajiv Khanna. Simple Diffusion Language Models Are More Effective Few-Step Generators Than Reported. arXiv:2609.33947, 2026.
  • Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. A Continuous Time Framework for Discrete Denoising Models. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, 2022.
  • Samson Gourevitch, Yazid Janati, Dario Shariatian, Umut Simsekli, Eric Moulines, Eric P. Xing, and Alain Durmus. Uniform Diffusion Models Revisited: Leave-One-Out Denoiser and Absorbing State Reformulation. arXiv:2605.22765, 2026.
  • Peter Holderrieth, Marton Havasi, Jason Yim, Neta Shaul, Itai Gat, Tommi Jaakkola, Brian Karrer, Ricky T. Q. Chen, and Yaron Lipman. Generator Matching: Generative Modeling with Arbitrary Markov Processes. In International Conference on Learning Representations (ICLR), 2025.
  • Haoran Sun, Lijun Yu, Bo Dai, Dale Schuurmans, and Hanjun Dai. Score-based Continuous-time Discrete Diffusion Models. In International Conference on Learning Representations (ICLR), 2023.

Dear Reviewer, the author-reviewer discussion ends on 18 November (AoE). We are happy to answer any further questions before then. Thank you in advance for your time.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response

We thank the reviewers for their reviews and feedback. Here we answer two shared questions: on novelty, and on examples of improvements and mistakes in published work.

Novelty. We see our contribution not only in each single theorem and the way in which we extend it, but also in the unified and simplified theoretical framework we provide and in the consequences that follow from it. We thus ask the reviewers to recognize that novelty and value lie not just in individual results and ideas, but additionally in how these are connected and shown to be related to each other. Concretely:

  • One derivation for all methods that strengthens individual results: one simple and mechanical recipe now gives the ELBO, the reverse rates, and the ELBO optimum of every parameterization and noising process, where before they were obtained separately for each method with ad hoc hypotheses and derivations. Additionally, this streamlined framework strengthens the previous results: the path-KL bound, known only up to a constant (Campbell et al., 2022; Holderrieth et al., 2025), is now an identity with exact constants on the clipped time windows used in practice, and the ELBO optimum now follows from a single projection for any token-factorizing process, including time-varying mixtures such as GIDD, to which the derivations of Gourevitch et al. (2026) do not transfer.
  • What a discrete diffusion model learns: at the ELBO optimum, denoisers, bridge plug-ins and scores all learn the same time-reversal of the noising process, in three coordinates. Before, each result at most identified what one of them learns, for one objective and one process, such as the score in SEDD (Lou et al., 2024) or the cavity for a ratio-matching loss but not the ELBO (Sun et al., 2023). In particular, we show that the bridge plug-in ELBO is optimized by the cavity and not the denoiser, contrary to its usual description. This distinction was missed in the works that train the bridge plug-in except for Gourevitch et al. (2026), who concurrently reach the same conclusion but only for a discrete-time ELBO and static-mixture distributions (excluding e.g. GIDD).
  • Tools for implementations: we give the NELBO of an untrained head in closed form, so that a single test simultaneously checks at initialization the implementation of the ELBO, the time-sampling schedule, the map from head to reverse rates, and the boundary terms. We also give exact formulas for the boundary terms, which published work often omits although they can be as large as the reported gaps between methods (see below).
  • Empirical validation: we evaluate the framework across the three heads and on four kernels under a single training recipe, to our knowledge the first such cross-head comparison.

Lastly, we would like to point out that, under the ICLR 2027 reviewer guidelines, Gourevitch et al. (2026) is contemporaneous work: it was only on arXiv until its NeurIPS 2026 acceptance on 24 September, one day before the ICLR deadline, and the submission already cites it as concurrent.

Improvements and mistakes in published work. The following examples have been chosen from both classic and recent published works.

  • Improvements:

    • D3PM (Austin et al., 2021): they found through an ablation that an auxiliary cross-entropy term in the loss improves the masked model on text8 but worsens the uniform one. We predict exactly this: the cross-entropy is minimized by the denoiser but their plug-in ELBO term by the cavity, and the two coincide under masked but not uniform diffusion. The compute of the ablation could thus have been saved.
    • GIDD (von Rütte et al., 2025): their self-correction step resamples tokens directly from the head's output. Since they use the bridge plug-in, they train a cavity, which ignores the current token and can therefore replace it, whereas near the clean endpoint a denoiser almost always keeps it. We thus predict that the step owes most of its gain to the cavity. To verify this prediction, we performed a new experiment using the authors' released checkpoint: the step improves GenPPL from 111 to 60 with the cavity (the raw network, as in their code), but only to 95 with the denoiser.
    • Duo (Sahoo et al., 2025): they report that replacing the last ancestral step with a greedy decoding of the network's output lowers GenPPL (their Table 7). Our framework predicts this, as in GIDD: they use a bridge plug-in, so the network learns the cavity and the final decoding acts as a self-correction step, while using the denoiser would keep most of the tokens.
    • SEDD (Lou et al., 2024) and MDLM (Sahoo et al., 2024): in published comparisons, the score model of SEDD trails MDLM in likelihood. We now explain part of the reason: under masked diffusion a score head is a cavity head whose normalization is learned instead of imposed. This adds an extra normalization term to the loss that we give exactly (Eq. E.10, building on Ou et al., 2025). Accordingly, converting a trained score head to the cavity lowers its NELBO without retraining, for every kernel and seed (new Table 14).
  • Mistakes:

    • SLM (Song et al., 2025): the released text8 code leaves out a prior term of at least 0.038 bits per character, more than every gap in their Table 1. Such an omission can now be detected at initialization, where the NELBO has a known value.
    • RADD (Ou et al., 2025): the authors evaluate two of the four losses using tmin⁡t_{\min} = 1e-3 and with no boundary terms. The missing terms change perplexity by up to 1%, the size of some of the gaps in their main Tables 1–2.
    • DiGress (Vignac et al., 2023): they sample with the head as a denoiser and compute the NLL with the head as the bridge plug-in. Since DiGress uses uniform and marginal noise, where the denoiser and the cavity differ, the reported likelihood is not that of the sampled model.

References

  • Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured Denoising Diffusion Models in Discrete State-Spaces. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pp. 17981–17993, 2021.
  • Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. A Continuous Time Framework for Discrete Denoising Models. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, 2022.
  • Samson Gourevitch, Yazid Janati, Dario Shariatian, Umut Simsekli, Eric Moulines, Eric P. Xing, and Alain Durmus. Uniform Diffusion Models Revisited: Leave-One-Out Denoiser and Absorbing State Reformulation. arXiv:2605.22765, 2026.
  • Peter Holderrieth, Marton Havasi, Jason Yim, Neta Shaul, Itai Gat, Tommi Jaakkola, Brian Karrer, Ricky T. Q. Chen, and Yaron Lipman. Generator Matching: Generative Modeling with Arbitrary Markov Processes. In International Conference on Learning Representations (ICLR), 2025.
  • Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution. In Proceedings of the 41st International Conference on Machine Learning (ICML), volume 235 of Proceedings of Machine Learning Research, pp. 32819–32848. PMLR, 2024.
  • Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data. In International Conference on Learning Representations (ICLR), 2025.
  • Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T. Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and Effective Masked Diffusion Language Models. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, 2024.
  • Subham Sekhar Sahoo, Justin Deschenaux, Aaron Gokaslan, Guanghan Wang, Justin T. Chiu, and Volodymyr Kuleshov. The Diffusion Duality. In Proceedings of the 42nd International Conference on Machine Learning (ICML), volume 267 of Proceedings of Machine Learning Research, pp. 52584–52619. PMLR, 2025.
  • Yuxuan Song, Zhe Zhang, Yu Pei, Jingjing Gong, Qiying Yu, Zheng Zhang, Mingxuan Wang, Hao Zhou, Jingjing Liu, and Wei-Ying Ma. ShortListing Model: A Streamlined Simplex Diffusion for Discrete Variable Generation. In Advances in Neural Information Processing Systems (NeurIPS), volume 38, 2025.
  • Haoran Sun, Lijun Yu, Bo Dai, Dale Schuurmans, and Hanjun Dai. Score-based Continuous-time Discrete Diffusion Models. In International Conference on Learning Representations (ICLR), 2023.
  • Clement Vignac, Igor Krawczuk, Antoine Siraudin, Bohan Wang, Volkan Cevher, and Pascal Frossard. DiGress: Discrete Denoising Diffusion for Graph Generation. In International Conference on Learning Representations (ICLR), 2023.
  • Dimitri von Rütte, Janis Fluri, Yuhui Ding, Antonio Orvieto, Bernhard Schölkopf, and Thomas Hofmann. Generalized Interpolating Discrete Diffusion. In Proceedings of the 42nd International Conference on Machine Learning (ICML), volume 267 of Proceedings of Machine Learning Research, pp. 61810–61843. PMLR, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.