- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Semantic Diffusion Language Models (SDLM), which replace tokens with semantically related alternatives during corruption. A low-rank semantic kernel enables exact and efficient training, while the resulting corruption allows revision rates to depend on the full posterior rather than only the confidence in the observed token. The paper further proposes SHARP to concentrate revisions on tokens with high revision probability. Experiments on a 425M model show improved generation quality over several masked and uniform diffusion baselines.
Critical strengths
The paper provides a clear analysis showing that uniform corruption makes revision timing depend only on confidence in the observed token, while semantic corruption can use the identities of alternative tokens in the posterior.
The empirical results are generally strong.
Critical weaknesses
I do not find significant weaknesses of this paper.
Questions for authors
Could the authors provide a clearer explanation of how the fixed semantic kernel is constructed from the pretrained token embeddings in practice?
Could the same benefit be achieved by constructing semantically structured corrupted training examples, rather than encoding semantic relationships into the diffusion transition kernel? What is the advantage of modeling semantic corruption as part of the diffusion process compared with directly training on such constructed corruptions?
3 comments
-
Generation still requires the reverse dynamics. The computation that cross-entropy avoids during training returns at every generation step. For semantic corruption, GDDS approximates it, propagating each new prediction through the corruption with a truncated series. In their semantic-kernel sampling ablation, Gen-PPL worsens beyond 64 steps, which the authors attribute to accumulated approximation error [2, Appendix D.2, Table 12].
-
The sampling decisions are supervised indirectly. Cross-entropy learns which clean token produced the observation, while our path objective directly penalises errors in the resulting reverse CTMC dynamics. For example, uncertainty between similar words can imply different revision timing from uncertainty involving unrelated words, even at the same confidence in the current token (Section 3). Both objectives share the same optimum, since an exact posterior supplies the information needed in either case. The distinction is which errors the loss penalises when the model is imperfect.
To our knowledge, we provide the first semantic DLM trained directly against the true reverse dynamics, so the loss supervises the sampling decisions discussed above. Our low-rank construction makes this tractable without approximating the semantic operators, with only about 6% more training FLOPs than MDLM.
We hope these clarifications make the kernel construction and the advantage of training the reverse dynamics clear, and provide grounds for the reviewer to consider raising their recommendation. We would be glad to address any further questions during the discussion.
References
[1] Campbell et al. A Continuous Time Framework for Discrete Denoising Models. NeurIPS 2022.
[2] Zekri et al. Generalized Discrete Diffusion from Snapshots. 2026. Appendix D.2, Table 12.
Thanks for the response. I have no further question.
We thank the reviewer for their time and helpful questions. We answer here and have made clarifications in the paper where stated.
Q1 — How is the semantic kernel constructed from embeddings?
We agree that the practical construction should be clearer and have reorganised it into a dedicated subsection, Appendix E.2, “Constructing the fixed semantic kernel.” We want successive replacements to retain some semantic relationship to the original token, so that corruption removes semantic information gradually. We therefore favour replacements between related tokens while requiring a low-rank kernel for computational tractability.
We build the kernel once, before training, using the input and output token embeddings of GIDD-small (pu=0), a pretrained masked-diffusion model with the same tokenizer.
(a) Group tokens using their input embeddings. We normalise these embeddings and apply spherical k-means to form R=256 groups. Each token is assigned to its nearest group centre. These assignments form A.
(b) Give each group a replacement distribution (B). We average and normalise the output embeddings within each group, then compare this average with every token’s output embedding using cosine similarity. We use these similarities to construct replacement probabilities, favouring similar tokens while enforcing normalisation and a uniform stationary distribution for K=AB⊤ (Appendix E.2). Replacements can therefore leave the original input-embedding group.
We initially explored fitting a low-rank approximation to an embedding-based similarity kernel using KL-NMF. In the configurations we tried, the replacement distributions were too diffuse, spreading probability over too many tokens. This motivated the more concentrated replacement distributions in the construction above. The separate absorbing-mask process ensures terminal forgetting, so the semantic kernel need not rapidly mix to its stationary distribution.
We chose this construction before the main training run, using forward-kernel diagnostics and corrupted samples to assess its behaviour.
We have now expanded Appendix E.2 into a four-step construction, with explicit embedding calculations, the transport objective and balancing procedure, numerical settings, and a diagnostic of cross-cluster replacement mass.
Q2 — Why put semantic corruption inside the diffusion process?
We thank the reviewer for this insightful question. Summary: As suggested, one may train a model simply to recover clean tokens from constructed corruptions using cross-entropy. This avoids evaluating reverse rates during training, but generation by reversing the forward CTMC still requires the modelled reverse dynamics. Our contribution is to make these computations tractable during both training and generation, allowing us to directly train the reverse CTMC rates.
The standard approach since Campbell et al. [1] is a path-space bound, whose latent variable is the entire noising trajectory. Its path term is the time integral of our local loss, which compares the true and modelled reverse rates (Eq. 9a; Appendix A.3). Evaluating these rates requires the network’s prediction propagated through the corruption, Ptxθ, which can be expensive with a general semantic kernel.
Cross-entropy instead needs only samples of corrupted text. GDDS [2] also uses CTMC-based semantic corruption but follows this route, deriving its ELBO with the noisy snapshot (Zt,t) rather than the trajectory as the latent variable. The distinction becomes clearer if we write dθ for the model's clean-token posterior and Ft for the calculation that converts this prediction into reverse rates. Schematically, for one training example,
\mathcal L_{\rm rates}\left(F_t(d_\theta),\widehat Q_t(\cdot\mid x_0)\right)$$ $$\text{Cross-entropy:}\qquad \mathcal L_{\rm CE}\left(d_\theta,x_0\right)$$ where $\widehat Q_t(\cdot\mid x_0)$ is the reverse-rate target available from the known clean token. This raises two issues. [continued in next comment]