acceptodds
Under review as a conference paper at ICLR 2027

Semantic Diffusion Language Models via Low-Rank Corruption

Abstract

Diffusion language models are commonly trained to denoise text in which tokens have been replaced by a [MASK] or by tokens sampled uniformly from the vocabulary. Such replacements carry no information about the original token. To train on corruptions that more closely resemble plausible mistakes, we instead consider semantically related alternatives, which can be harder to identify as corrupted while preserving clues that aid recovery. We introduce Semantic Diffusion Language Models (SDLMs), a family of discrete diffusion models that draw these replacements from a nonnegative low-rank kernel, which permits exact training-loss evaluation at linear rather than quadratic cost in vocabulary size. Like uniform replacements, semantic ones are not marked as corrupted, so the model must learn not only what each token should become but also when to revise it. We show that with uniform noise, models time revisions by their confidence in the observed token alone: belief in a plausible alternative is indistinguishable from belief in an unrelated word. SDLMs instead use the full posterior, keeping a token settled when the model's belief falls on words similar to the observed token. SDLM exceeds the MAUVE of matched MDLM and GIDD baselines with 4× fewer steps. We also introduce SHARP, a predictor–corrector sampler that reallocates revisions toward tokens the model is most likely to change. With SHARP, SDLM outperforms ReMDM at every sampling budget and Duo with Ψ-samplers at all but the lowest.

Primary area:
generative models
Keywords:
discrete diffusion, diffusion language models, semantic corruption, language modeling, generative modeling, discrete generative models, text generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_mLf1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Semantic Diffusion Language Models (SDLM), which replace tokens with semantically related alternatives during corruption. A low-rank semantic kernel enables exact and efficient training, while the resulting corruption allows revision rates to depend on the full posterior rather than only the confidence in the observed token. The paper further proposes SHARP to concentrate revisions on tokens with high revision probability. Experiments on a 425M model show improved generation quality over several masked and uniform diffusion baselines.

Critical strengths

The paper provides a clear analysis showing that uniform corruption makes revision timing depend only on confidence in the observed token, while semantic corruption can use the identities of alternative tokens in the posterior.

The empirical results are generally strong.

Critical weaknesses

I do not find significant weaknesses of this paper.

Questions for authors

Could the authors provide a clearer explanation of how the fixed semantic kernel is constructed from the pretrained token embeddings in practice?

Could the same benefit be achieved by constructing semantically structured corrupted training examples, rather than encoding semantic relationships into the diffusion transition kernel? What is the advantage of modeling semantic corruption as part of the diffusion process compared with directly training on such constructed corruptions?

3 comments

We thank the reviewer for their time and helpful questions. We answer here and have made clarifications in the paper where stated.

Q1 — How is the semantic kernel constructed from embeddings?

Could the authors provide a clearer explanation of how the fixed semantic kernel is constructed from the pretrained token embeddings in practice?

We agree that the practical construction should be clearer and have reorganised it into a dedicated subsection, Appendix E.2, “Constructing the fixed semantic kernel.” We want successive replacements to retain some semantic relationship to the original token, so that corruption removes semantic information gradually. We therefore favour replacements between related tokens while requiring a low-rank kernel for computational tractability.

We build the kernel once, before training, using the input and output token embeddings of GIDD-small (pu=0p_u=0), a pretrained masked-diffusion model with the same tokenizer.

  • (a) Group tokens using their input embeddings. We normalise these embeddings and apply spherical kk-means to form R=256R=256 groups. Each token is assigned to its nearest group centre. These assignments form AA.

  • (b) Give each group a replacement distribution (BB). We average and normalise the output embeddings within each group, then compare this average with every token’s output embedding using cosine similarity. We use these similarities to construct replacement probabilities, favouring similar tokens while enforcing normalisation and a uniform stationary distribution for K=AB⊤K=AB^\top (Appendix E.2). Replacements can therefore leave the original input-embedding group.

We initially explored fitting a low-rank approximation to an embedding-based similarity kernel using KL-NMF. In the configurations we tried, the replacement distributions were too diffuse, spreading probability over too many tokens. This motivated the more concentrated replacement distributions in the construction above. The separate absorbing-mask process ensures terminal forgetting, so the semantic kernel need not rapidly mix to its stationary distribution.

We chose this construction before the main training run, using forward-kernel diagnostics and corrupted samples to assess its behaviour.

We have now expanded Appendix E.2 into a four-step construction, with explicit embedding calculations, the transport objective and balancing procedure, numerical settings, and a diagnostic of cross-cluster replacement mass.

Q2 — Why put semantic corruption inside the diffusion process?

Could the same benefit be achieved by constructing semantically structured corrupted training examples, rather than encoding semantic relationships into the diffusion transition kernel? What is the advantage of modeling semantic corruption as part of the diffusion process compared with directly training on such constructed corruptions?

We thank the reviewer for this insightful question. Summary: As suggested, one may train a model simply to recover clean tokens from constructed corruptions using cross-entropy. This avoids evaluating reverse rates during training, but generation by reversing the forward CTMC still requires the modelled reverse dynamics. Our contribution is to make these computations tractable during both training and generation, allowing us to directly train the reverse CTMC rates.

The standard approach since Campbell et al. [1] is a path-space bound, whose latent variable is the entire noising trajectory. Its path term is the time integral of our local loss, which compares the true and modelled reverse rates (Eq. 9a; Appendix A.3). Evaluating these rates requires the network’s prediction propagated through the corruption, PtxθP_t x_\theta, which can be expensive with a general semantic kernel.

Cross-entropy instead needs only samples of corrupted text. GDDS [2] also uses CTMC-based semantic corruption but follows this route, deriving its ELBO with the noisy snapshot (Zt,t)(Z_t,t) rather than the trajectory as the latent variable. The distinction becomes clearer if we write dθd_\theta for the model's clean-token posterior and FtF_t for the calculation that converts this prediction into reverse rates. Schematically, for one training example,

\mathcal L_{\rm rates}\left(F_t(d_\theta),\widehat Q_t(\cdot\mid x_0)\right)$$ $$\text{Cross-entropy:}\qquad \mathcal L_{\rm CE}\left(d_\theta,x_0\right)$$ where $\widehat Q_t(\cdot\mid x_0)$ is the reverse-rate target available from the known clean token. This raises two issues. [continued in next comment]
  • Generation still requires the reverse dynamics. The computation FtF_t that cross-entropy avoids during training returns at every generation step. For semantic corruption, GDDS approximates it, propagating each new prediction through the corruption with a truncated series. In their semantic-kernel sampling ablation, Gen-PPL worsens beyond 64 steps, which the authors attribute to accumulated approximation error [2, Appendix D.2, Table 12].

  • The sampling decisions are supervised indirectly. Cross-entropy learns which clean token produced the observation, while our path objective directly penalises errors in the resulting reverse CTMC dynamics. For example, uncertainty between similar words can imply different revision timing from uncertainty involving unrelated words, even at the same confidence in the current token (Section 3). Both objectives share the same optimum, since an exact posterior supplies the information needed in either case. The distinction is which errors the loss penalises when the model is imperfect.

To our knowledge, we provide the first semantic DLM trained directly against the true reverse dynamics, so the loss supervises the sampling decisions discussed above. Our low-rank construction makes this tractable without approximating the semantic operators, with only about 6% more training FLOPs than MDLM.

We hope these clarifications make the kernel construction and the advantage of training the reverse dynamics clear, and provide grounds for the reviewer to consider raising their recommendation. We would be glad to address any further questions during the discussion.

References

[1] Campbell et al. A Continuous Time Framework for Discrete Denoising Models. NeurIPS 2022.

[2] Zekri et al. Generalized Discrete Diffusion from Snapshots. 2026. Appendix D.2, Table 12.

Thanks for the response. I have no further question.

Reviewer_qf4eOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces Semantic Diffusion Language Models (SDLMs), that corrupt text with semantically related alternatives instead of uniform or mask tokens. Since calculating the exact transition probability between semantically related words is too computationally expensive, the authors propose to draw these replacements from a nonnegative low-rank kernel, going from a quadratic to a linear cost. Finally, the authors propose SHARP, a predictor-collector sampler. SLDMs achieve and exceed the MAUVE score of other baselines with 4 times fewer steps.

Critical strengths
  • The first main strength is the low-rank semantic kernel that makes evaluating PtP_t and PtxθP_t x_\theta practical at vocabulary scale.
  • The second main strength is the empirical efficiency
Critical weaknesses
  • SDLMs requires slightly more FLOPs than baselines, and has a higher latency
Questions for authors
  • The optimal rank appears to be 64 from the experiments shown in the paper. Does this imply that it is recommended to run an ablation study for new deployments?
  • Can the authors provide their hypothesis for why a lower-rank bottleneck improves the generation quality?
3 comments

We thank the reviewer for their time and helpful questions. We address each point below and note where we have added material to the paper.

W1 — Training cost and the quality–time trade-off

SDLMs requires slightly more FLOPs than baselines, and has a higher latency

We agree that semantic corruption adds computation, training requires about 6% more FLOPs than MDLM and 2% more than Duo.

Prompted by this comment, we re-measured inference for all methods after optimising each implementation, and added a quality-time comparison to the paper (Appendix E.4, Figure E.1). At 32 NFE, SDLM takes 31% longer than MDLM and 16% longer than Duo. However, fixed-NFE latency does not show how quickly each method reaches a given quality, so the table below pairs the manuscript’s native-sampler MAUVE scores with the updated timings (seconds per 512-token sequence, batch size one, RTX 4090).

NFEMDLM time (s)MDLM MAUVEDuo time (s)Duo MAUVESDLM time (s)SDLM MAUVE
160.1550.05190.1770.18820.2050.1688
320.3120.19410.3530.42300.4100.4186
640.6280.38260.7070.52650.8200.5554
1281.2580.50731.4140.58161.6420.6007
2562.5210.53642.8300.61613.2840.6617
5125.0410.58575.6620.63606.5500.6741

Beyond the lowest NFE budgets, the additional cost per step is offset by reaching higher quality with fewer steps:

  • At 128 NFE, SDLM exceeds MDLM's best tested MAUVE and is 3.07x faster.
  • At 128 NFE, SDLM almost matches Duo on MAUVE at 256 NFE and is 1.72x faster.
  • At 256 NFE, SDLM exceeds Duo's best tested MAUVE and is 1.72x faster.

Both Duo and SDLM contribute points to the measured Pareto frontier, but the curves suggest a more favourable quality-time trade-off for SDLM from around 64 NFE onwards, even after accounting for its additional inference cost.

Q1 — Choosing the rank for a new deployment

The optimal rank appears to be 64 from the experiments shown in the paper. Does this imply that it is recommended to run an ablation study for new deployments?

For any new dataset, tokenizer or model scale, yes, we agree and would encourage a short sweep (over rank and kernels) before committing to long training runs. Pynadath et al.'s Generative Frontiers [1] also reports that the quality-diversity frontiers of MDLM, Duo and CANDI improve relatively little between 50k and 1M training steps, so comparisons of different kernels using small runs may similarly carry over to full training.

In particular, our short runs showed that rank 64 gives the highest MAUVE at all five NFE budgets (Table F.1). We will test whether rank 64's advantage persists by training it for the full token budget, using the same recipe as our rank 256 model, and report the comparison before camera-ready.

Q2 — Why might an intermediate rank improve generation?

Can the authors provide their hypothesis for why a lower-rank bottleneck improves the generation quality?

This is a great question that warrants future research. Our hypothesis is that an intermediate rank produces replacements that are informative enough to support recovery, while remaining distinguishable enough from unchanged tokens to support useful revision. This relates to the two aspects of denoising captured by the exit rate and the replacement law: how quickly a token is revised and what replaces it, as shown in Section 3.

View the low-rank kernel as a soft clustering of the vocabulary through RR semantic prototypes. At R=1R=1, replacements are uniform thus often easy to recognise as misplaced, but uninformative about their source (Section 5.1, Appendix F.1). Finer groups preserve more clues about the original, but produce replacements that look increasingly correct in context (“tea” becoming “coffee” may leave a perfectly plausible sentence). An intermediate rank may therefore provide useful practice in correcting plausible but misplaced tokens. This is our proposed explanation for R=64R=64 achieving higher generation MAUVE than R∈{16,256}R\in\{16,256\} in the small runs.

Results in related settings point the same way. HDLM [2] obtains its best generative perplexity at 64 clusters and best validation perplexity at 128, with both worse at 256, although it uses explicit cluster states. ELECTRA’s authors [3] speculate that an overly strong replacement generator makes detection too hard, although they study representation learning rather than generation.

[continued in next comment]

A complementary possibility is that a lower rank exposes the model to a wider range of corrupted sequences during training, which could help it recover from its own errors during generation where errors can accumulate across steps.

We hope the new quality–time comparison and the clarifications on rank selection address the reviewer’s concerns and provide grounds to consider raising their recommendation. We remain happy to discuss any further questions.

References

[1] Pynadath, Shi and Zhang. Generative Frontiers: Why Evaluation Matters for Diffusion Language Models. 2026. Section 5 and Figure 6.

[2] Zhou et al. Next Semantic Scale Prediction via Hierarchical Diffusion Language Models. NeurIPS 2025.

[3] Clark et al. ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators. ICLR 2020.

I am satisfied with the authors response.

Reviewer_6vL8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Semantic Diffusion Language Models (SDLMs) within the existing continuous-time discrete diffusion framework. Its main contributions are:

  1. An efficiently computable semantic corruption process. A nonnegative low-rank replacement kernel, K=AB⊤K=AB^\top, is combined with an independent reset to a known prior. This permits exact algebraic expressions for forward transitions and efficient evaluation of the existing local diffusion loss.
  2. An analysis of token revision timing. Unlike shared-noise corruption, source-dependent corruption allows revision rates to depend on which alternative clean tokens the model considers plausible, beyond confidence in the observed token.
  3. SHARP sampling and empirical validation. SHARP concentrates revisions on selected positions while preserving the current step's expected edit count and conditional replacement distributions. Experiments with approximately 425M-parameter models on OpenWebText report improved generation quality.

The contribution is the tractable specialization, timing analysis, and sampling rule. Semantic corruption, Poisson uniformization, and the general CTMC training framework precede this work.

Critical strengths
  1. The structured process is mathematically sound and computationally useful. The factorization supports repeated semantic replacements through a smaller intermediate space, with stated per-token cost O(VR+R3/L)O(VR+R^3/L) for vocabulary size VV, factorization rank RR, and sequence length LL. The independent prior reset separates terminal forgetting from the semantic kernel's mixing behavior; the stationarity condition yields the commuting generators and explicit marginal in Lemma 4.2. These are useful properties for making semantic corruption practical at vocabulary scale.

  2. The timing analysis identifies a meaningful structural distinction. Section 3 explains why shared-noise models reduce revision timing to observed-token confidence, whereas source-dependent corruption can use information about alternative clean tokens. This offers a useful way to reason about how the forward corruption design affects the learned reverse process.

  3. SHARP has a precise, limited guarantee and a measurable benefit. Its capacity constraint makes the redistribution of departure probabilities feasible, preserving conditional replacement laws and expected edits at the current state without additional network evaluations. Table E.2 compares SHARP with an otherwise matched SDLM+Ψ\Psi sampler: at 64 NFE, MAUVE improves from 0.6618 to 0.7595. This is a substantial controlled gain, even though the mechanism behind it warrants further investigation.

  4. The evaluation contains useful controls and complementary diagnostics. The main comparisons approximately match model size and training-token budget. The paper includes rank, alignment, and sampler ablations, and evaluates GenPPL–entropy trade-offs, repetition, and external-scorer sensitivity in addition to MAUVE. These results support the practical promise of the approach.

Critical weaknesses
  1. The added value of the particular kernel construction remains insufficiently isolated.

    The proofs justify the general low-rank family as a valid and tractable corruption process. They do not distinguish the implemented combination of hard input-embedding clusters, output-embedding affinities, transport balancing, and uniform vocabulary stationarity from other valid choices. I do not regard the absence of an optimality theorem as a weakness; the concern is that the existing ablations do not establish the value of these additional design choices.

    A concrete comparison would be uniform resampling within the same semantic clusters. For RR nonempty clusters of ordinary tokens, define:

    Kcluster(x,y)=1c(x)=c(y)∣Cc(x)∣.K_{\mathrm{cluster}}(x,y) = \frac{\mathbf{1}\\{c(x)=c(y)\\}}{|C_{c(x)}|}.

    This kernel is stochastic, uniform-stationary, rank RR, and idempotent. Its semantic transition law therefore has the closed form

    eΛ(Kcluster−I)=e−ΛI+(1−e−Λ)Kcluster,Λ=∫0tβssem,ds.e^{\Lambda(K_{\mathrm{cluster}}-I)} = e^{-\Lambda}I + (1-e^{-\Lambda})K_{\mathrm{cluster}}, \qquad \Lambda = \int_0^t \beta_s^{\mathrm{sem}}\\,ds.

    Combined with the same absorbing reset, this provides a simpler tractable semantic process; its lack of cross-cluster mixing does not prevent terminal forgetting. A comparison using the same clusters, training budget, and sampler, with comparable corruption-level calibration, would test whether the richer construction provides an advantage. This is a proposed diagnostic baseline, not a claim that it will perform equally well.

    Uniform vocabulary stationarity also requires a clearer motivation in the deployed setting. Any row-stochastic vocabulary kernel extended with an absorbing mask preserves the mask prior. Uniform stationarity over ordinary tokens is therefore an additional design choice, rather than a requirement for the absorbing-prior guarantee.

  2. The advance over the closest structured and semantic diffusion methods needs clearer positioning and comparison.

    D3PM already studies embedding-based corruption and discusses low-rank transition matrices in its original supplementary material [1]. GDDS uses general semantic kernels and Poisson uniformization [2]. These are not identical constructions: SDLM's particular factorized process and efficient marginal/loss evaluation remain meaningful distinctions. Nevertheless, the principal baselines are masking, uniform, and mixed-noise models, which do not establish the advantage over close semantic approaches such as HDLM [3] or GDDS. Semantic DLM+ [4] is also relevant to semantic-plus-global corruption and should be discussed. One convincing comparison with a close or simpler semantic method would be more informative than additional comparisons with shared-noise baselines.

  3. The connection between semantic timing and the generation gains is not yet established.

    Proposition B.5 relates revision rates to conditional expected replacement counts under the chosen kernel, rather than to an independently defined semantic distance. As Appendix B.3 recognizes, the semantic interpretation depends on the kernel. Moreover, with the implemented hard assignments in AA, alternative clean tokens x≠zx\ne z in the same source cluster have identical corruption likelihoods and clean-conditioned exit rates for a fixed observation zz. The timing signal therefore depends on cluster-level posterior masses and observed-token confidence. This remains richer than confidence alone, but limits its granularity.

    The diagnostic with uniformly injected substitutions supports sensitivity to unrelated errors; it does not establish appropriate behavior for semantically close but contextually incorrect alternatives. More directly, SHARP is not compared against confidence-only or randomized position allocation while holding the model, conditional replacement laws, and expected edit budget fixed. Prior work already explores confidence- and alternative-margin-based update selection [5], although it does not supply the same SHARP rule. A matched allocation experiment would test whether the proposed timing signal offers a distinct advantage over generic concentration of revisions.

  4. The alignment control does not fully separate semantics from other changes in corruption difficulty.

    The shuffled-kernel permutation is not frequency-stratified, and the information-based schedule fitted for the original kernel is reused without recalibration. A token permutation preserves the abstract kernel but can alter the corruption experienced by frequent tokens and the information retained at a given time. The result therefore supports the value of token–kernel alignment as a whole, as the appendix acknowledges, rather than isolating semantics alone. The controls also use one training seed and approximately one tenth of the main training budget. Frequency-aware or information-matched controls would make the attribution more convincing.

  5. The empirical evidence is encouraging but narrower than the proposed interpretation.

    The main generation results concern OpenWebText at one model scale and sequence length. The additional metrics are valuable, but they do not directly establish better contextual correction. The main generation comparisons report point estimates; multiple generation and clustering seeds do not quantify variation across independently trained models. The disagreement between MAUVE and external-scorer rankings in Appendix E.4 also motivates a complementary quality or controlled correction evaluation, without invalidating MAUVE itself. Finally, the reported fixed-NFE latency measurements do not establish the complete quality-versus-time trade-off: SDLM has additional cost per evaluation, so fewer evaluations should be distinguished from an equivalent wall-clock improvement.

References

[1] Austin et al. Structured Denoising Diffusion Models in Discrete State-Spaces. NeurIPS 2021. See supplementary Appendices A.2.4 and A.4.1. Supplementary material.

[2] Zekri et al. Generalized Discrete Diffusion from Snapshots. 2026. Paper.

[3] Zhou et al. Next Semantic Scale Prediction via Hierarchical Diffusion Language Models. NeurIPS 2025. Paper.

[4] Jiang et al. Semantic DLM+: Improving Diffusion Language Models through Bias-variance Trade-off in Transition Kernel Design. 2026. Paper.

[5] Zhao et al. Informed Correctors for Discrete Diffusion Models. Paper.

Questions for authors

The first three questions would most affect my assessment; the remaining questions concern the strength and reproducibility of the evidence.

  1. What does the specific kernel design add? Can you compare the proposed construction with the within-cluster kernel above, or provide an ablation that isolates a consequential factor-construction choice? Please explain the role of uniform vocabulary stationarity when the generation prior is an absorbing mask, and how corruption levels would be made comparable across kernels.

  2. Does the timing signal improve allocation beyond simpler scores? On the same SDLM checkpoint and base sampler, can SHARP's position ranking be compared with confidence-only and randomized rankings while retaining the same number of selected positions, conditional replacement laws, and expected edits at each current state? This requires no retraining. A diagnostic involving semantically close but contextually wrong substitutions would further clarify what the timing signal captures.

  3. How should the contribution be distinguished from the closest prior methods? Please clarify the advance over D3PM's discussion of low-rank transitions in Appendix A.4.1 and the semantic constructions in GDDS, HDLM, and Semantic DLM+. Can one close semantic comparison be supplied, or can the obstacles to a meaningful matched comparison be explained? I am not requesting implementations of every related method.

  4. How robust is the semantic-alignment result? Does the advantage over shuffled alignment survive a frequency-aware permutation or recalibration to comparable information loss? Please provide uncertainty for the principal comparisons and distinguish variation from generation, MAUVE clustering, and model training.

  5. Can the evaluation and numerical protocol be made fully reproducible? Appendix E.2 draws reference documents from train[-10000:]: were these withheld from the training data of every evaluated model? Were sampler-selection and final-evaluation pools separate, and did baseline samplers receive comparable tuning? Please provide the precise affinity/transport objective, the numerical schedule file, and the evaluated finite-step sampler/grid and matrix-function tolerance. A quality-versus-generation-time comparison would also help assess practical efficiency. These are clarification requests; I am not asserting data overlap or an implementation error.

6 comments

We thank the reviewer for the thorough review and the time taken to engage carefully with both the theory and experiments. Our new comparison supports the richer kernel over uniform within-cluster replacements (Figure 5), we clarify SHARP’s role and evaluate the reviewer’s suggested confidence-based ranking, and we have added Appendix H and Table H.1 to clarify our contribution relative to prior semantic methods.

Q1 — What does the kernel add beyond uniform within-cluster replacements?

We thank the reviewer for suggesting this comparison. We have trained the proposed within-cluster baseline and added the results to Figure 5 (left). Our rank-256 semantic kernel achieves higher MAUVE at every evaluated sampling budget than uniform resampling within the same 256 input clusters, with rank 64 improving further.

Kernel16 NFE32 NFE64 NFE128 NFE256 NFE
Uniform (within 256 clusters).1062.2838.3970.4808.4793
Semantic, rank 256.1465.3153.4273.5162.5197
Semantic, rank 64.1761.3599.4709.5474.5495

We calibrated the within-cluster kernel to match the semantic kernel’s approximately linear information-loss trajectory (Appendices E.1 and F.2), measured using normalised token conditional entropy under the same empirical unigram marginal following the information-based scheduling approach of Austin et al. [1, Section 3.2 and Appendix A.7].

The suggested within-cluster kernel is a natural special case of our family Qt=βt(AB⊤−I)Q_t=\beta_t(AB^\top-I), with simple marginals owing to idempotence. We chose a more general construction to demonstrate that semantic mixing remains computationally practical without this additional structure (Section 4.2), while retaining the freedom to revise tokens across input clusters. Within-cluster resampling instead confines later revisions to a predefined group, even where the model favours a replacement outside it. Our construction allows these revisions while embedding affinities favour related tokens (Appendix E.2).

Under the absorbing mask, uniform stationarity over ordinary tokens is indeed not needed for terminal forgetting, since the mask extension already satisfies the stationarity condition of Appendix C.2. We chose it to avoid a potentially low-entropy stationary distribution, which we expected to add unnecessary reconstruction work in reverse, and so that the same kernel can be paired with a uniform reset prior, enabling a direct comparison of masking versus a uniform prior.

Q2 — Does SHARP need exit-rate ranking?

We thank the reviewer for this insightful question. Across 16–256 NFE, confidence ranking performs comparably to SHARP, while random ranking consistently performs worse. This supports confidence as an alternative ranking within the same revision-concentration procedure.

With step size hh, the base sampler revises each visible token with probability hrtθ(z)h r_t^\theta(z) to first order (Eq. 10), where the exit rate is trained directly by the ELBO’s timing term (Eq. 9b). Like nucleus sampling [2] over tokens, SHARP keeps positions holding 90% of the total finite-step revision probability and rescales their probabilities so the expected number of revisions is unchanged (Appendix D.1). As requested by the reviewer, we test confidence-based and random selection in place of this ranking, keeping the number of selected positions, replacement laws and expected revisions fixed.

In this matched comparison, rtθ(z)r_t^\theta(z) and dθ(z)d_\theta(z) come from the same network evaluation, and the selected positions still retain SDLM’s departure probabilities as the quantities being rescaled. Replacing the ranking therefore does not simplify training or reduce model evaluations. Both rankings also use the same model trained under source-dependent corruption, where the exit rate depends on which alternatives the posterior favours, and the timing term supervises that rate directly. The comparison consequently tests whether explicitly using the richer exit rate for selection improves sampling, rather than whether training with that richer timing structure is beneficial.

On the same checkpoint and Ψ-Cap sampler, we evaluated confidence and random ranking across 16–256 NFE, with 5,000 samples per configuration pooled across five generation seeds. We compare against SHARP, reporting mean MAUVE and SD across clustering seeds.

[continued in next comment]

NFESHARPConfidenceRandom
160.3504 ± 0.01200.3514 ± 0.01850.0638 ± 0.0045
320.6423 ± 0.01270.6372 ± 0.02240.3725 ± 0.0151
640.7403 ± 0.00790.7382 ± 0.01630.5952 ± 0.0144
1280.8076 ± 0.01050.7871 ± 0.00800.6798 ± 0.0096
2560.8156 ± 0.00880.8099 ± 0.00540.7055 ± 0.0091

Confidence ranking and SHARP select largely the same positions (89% shared, compared with 76% for random selection), consistent with their similar MAUVE scores, suggesting that exit-rate ranking is not uniquely benefitial when concentrating revisions on SDLM. The corruption process may nevertheless matter for whether this concentration remains useful at higher NFE. For Duo, SHARP’s ranking is equivalent to confidence ranking (Eq. 15), and we showed that its benefit disappears by 128 NFE and reverses at 256 as repetition increases (Section 5.4), whereas it continues to help SDLM. This suggests that semantic corruption may support a broader range of effective sampling interventions, including the confidence-based variant tested here.

Informed Correctors [3], suggested by the reviewer, provides related work on confidence-based correction, using additional Gibbs-style steps and a hollow transformer for masked diffusion. SHARP instead concentrates revisions within the existing sampling step, without additional network evaluations or architectural changes. We will clarify this relationship in Section 4.3 and add the reference.

We will add the comparison to Appendix D and, before camera ready, rewrite the SHARP section to improve readability and clearly distinguish the revision-concentration procedure from the choice of ranking within it.

Q3 — How does SDLM differ from prior semantic diffusion methods?

We agree that these distinctions should be clearer and have added Appendix H and Table H.1. Our novel contribution is to make semantic mixing on the vocabulary computationally tractable under the usual pathwise objective, directly training the reverse CTMC rates and efficiently computing the transitions needed during generation.

D3PM [1] discusses low-rank transitions in Appendix A.4.1 but does not implement a low-rank semantic construction. Its semantic mixing retains quadratic transition costs. At 64 generation steps, its semantic model takes approximately 71×71\times longer to sample than its uniform model, even on the small 8,192-token vocabulary ([1], Table 7). Our own compute analysis illustrates the training bottleneck. Even with transition matrices precomputed, generic dense semantic mixing requires roughly five times MDLM's training FLOPs, while SDLM adds approximately 6%.

GDDS [4] and Semantic DLM+ [5] instead train clean-token denoisers, and GDDS's semantic ancestral sampler requires costly operator approximations ([4], Appendix D.2). HDLM [6] achieves tractability through auxiliary hierarchical states, but cannot revise already decoded words, limiting its self-correction ability.

We have also added the requested comparison with uniform resampling within semantic clusters, reported in Q1 and the updated Figure 5.

We considered GDDS as a close baseline and attempted reproduction, but could not obtain a reliable comparison. Its published results, and those of Semantic DLM+, also lack MAUVE and matched cross-model GenPPL–entropy frontiers. Generative Frontiers [7] and Acquaviva's Fixing Flow Model Evals [8] explain why GenPPL comparisons must account for the entropy trade-off, while Hacking Generative Perplexity [9] demonstrates failures even with entropy safeguards and motivates complementary distributional metrics such as MAUVE. This is why we report both MAUVE and temperature-swept GenPPL–entropy frontiers. GDDS's NFE sweep ([4], Figure 5) and Semantic DLM+'s minimum-entropy threshold ([5], Section 5.2) do not provide the same comparison, so we cannot infer a reliable generation-quality ranking from the published numbers alone.

Q4 — Does semantic alignment help beyond token frequency and information loss?

We thank the reviewer for this useful suggestion. The semantic kernel still performs better under both suggested controls, now included in Figure 5 and Appendix F.2.

We ran two additional short training runs with the same model. For the random shuffle, we adjusted the corruption schedule to match how quickly the semantic kernel removes information. For the frequency-aware shuffle, we shuffled only among tokens with similar training frequencies. Its information-loss curve already closely matches the semantic kernel’s.

Rank-256 kernel16 NFE32 NFE64 NFE128 NFE256 NFE
Semantic.147.315.427.516.520
Frequency-aware shuffle.105.216.279.321.360
Random shuffle, adjusted schedule.073.160.231.282.280

[continued in next comment]

The advantage holds at every sampling budget, supporting the value of the embedding-derived geometry beyond token frequency or differences in information loss.

All MAUVE results in the paper already use 5,000 generations pooled across five generation seeds, evaluated against the same 5,000 reference documents and averaged over five MAUVE clustering seeds, following the authors’ recommended practice (Appendix E.3). For readability, the kernel-comparison tables show means without standard deviations. We report the clustering standard deviations at 128 NFE for these controls in Appendix F.2. At every sampling budget, even the lowest semantic-kernel score across clustering seeds exceeds the highest score of either new shuffle control.

Q5 — Can the evaluation and numerical protocol be made fully reproducible?

We thank the reviewer for these useful clarification requests.

Data and evaluation. We retrained MDLM, Duo and SDLM under a common setup, using the same data split, backbone, training token budget and evaluation protocol, with the reference documents withheld from training for all three models. The confusing train[...] notation referred to Hugging Face’s original dataset split rather than our training partition, we have replaced it with an explicit description of the holdout.

Sampler selection. Selection and final evaluation used separate generation pools, and we selected SDLM’s Ψ-Cap strength on the selection pools while using the recommended Ψ configuration for Duo. SHARP uses a fixed threshold of .90, analogous to the cumulative-mass threshold in nucleus sampling, and the comparison with .95 checks sensitivity rather than tuning this threshold.

Reproducibility and efficiency. We have expanded Appendix E.2 with the precise affinity and transport construction and clarified the evaluated sampling grid and matrix-function tolerance in the appendices. We have also added a quality-versus-generation-time comparison in Appendix E.4 to make the practical trade-off easier to assess.

We hope the new controlled comparisons and clarifications address the reviewer’s main concerns and provide grounds to consider raising their recommendation. If any substantive questions remain, we would welcome the opportunity to address them during the discussion.

References

[1] Austin et al. Structured Denoising Diffusion Models in Discrete State-Spaces. NeurIPS 2021. Section 3.2, Appendices A.4.1–A.4.2 and A.7, and Table 7.

[2] Holtzman et al. The Curious Case of Neural Text Degeneration. ICLR 2020.

[3] Zhao et al. Informed Correctors for Discrete Diffusion Models. NeurIPS 2025.

[4] Zekri et al. Generalized Discrete Diffusion from Snapshots. 2026. Section 4.3, Figure 5 and Appendix D.2.

[5] Jiang et al. Semantic DLM+: Improving Diffusion Language Models through Bias-variance Trade-off in Transition Kernel Design. 2026. Sections 5.2–5.3 and Appendix E.1.

[6] Zhou et al. Next Semantic Scale Prediction via Hierarchical Diffusion Language Models. NeurIPS 2025. Sections 3.1–3.2.

[7] Pynadath, Shi and Zhang. Generative Frontiers: Why Evaluation Matters for Diffusion Language Models. 2026. Sections 3–4.

[8] Acquaviva. Fixing Flow Model Evals.

[9] Franca and Tong. Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs New Metrics. 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.