- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates what object is identified by the training objectives of discrete diffusion models. The authors decompose the expected negative ELBO into the data entropy and a path-space KL divergence from the oracle reverse process to the learned reverse process. Under token-factorizing corruption, they characterize the denoiser, bridge plug-in, and concrete score as three representations of the same reverse process. An important conclusion is that the optimal prediction target for the bridge plug-in is the conditional distribution of a clean token given the noisy sequence excluding that token’s own noisy observation, termed the cavity distribution.
Critical strengths
-
The paper brings several formulations into a common mathematical framework. Expressing different training objectives through reverse jump rates helps clarify how network outputs define a generative process.
-
The experiments cover multiple noise kernels and include both language modeling and structured reasoning tasks, providing empirical support for the framework across different settings.
Critical weaknesses
-
The new contributions need to be distinguished more clearly from existing results. The closely related work already discusses the cavity distribution and conversions between parameterizations. The authors should discuss more explicitly which results are new and how they extend previous findings.
-
A shared oracle optimum does not isolate trainability as the only source of empirical differences. Section 5.3 argues that differences between coordinates must reflect trainability because they share the same NELBO floor. This conclusion is too strong and need more discussions.
Questions for authors
How much does the score-to-probability correction affect generation quality?
2 comments
Dear Reviewer, the author-reviewer discussion ends on 18 November (AoE). We are happy to answer any further questions before then. Thank you in advance for your time.
We thank the reviewer for their feedback.
W1. New contributions against existing results. For a discussion on novelty, we kindly refer to the general response.
W2. Trainability as the only source of differences. §5.3 now says what trainability covers: optimization, finite capacity and, for the score, the normalization that the head must learn. We also state the equivalence only "at the optimum" wherever we invoke it. The differences between heads also hold at a tuned learning rate and in two seeds.
Q1. Effect of the score-to-probability correction on generation quality. The correction helps rather than hurts the score models: it lowers the NELBO for every kernel and in both seeds, by a quarter to a half of the gap to the cavity (new Table 14). We measured its effect on the likelihood because our samplers take a distribution, so a score head can only be sampled after the correction. The reported sample quality therefore already includes it. Under masked diffusion, where the correction is only a normalization, it accounts for half of the NELBO gap to the cavity, and the sample quality of the score is close to the cavity's. On the other kernels the score remains worse.
Presentation. We also revised the text for clarity, most visibly in §5.3 and in the related work.