acceptodds
Under review as a conference paper at ICLR 2027

Know When to Hold 'em: Correct-Token Retention in Uniform-State Diffusion Language Models

Abstract

Uniform-state diffusion models (USDMs) can revise any token at any denoising step, which lets them correct their own mistakes, a key advantage over masked diffusion. Self-correction, however, requires both revising incorrect tokens and retaining correct ones, and we show that current USDMs lack the latter. Even under greedy-tail decoding, state-of-the-art USDMs (DUO, UDLM, and uniform-noise SEDD) keep revising 173–270 of 512 positions at every step, and these large, uncoordinated edits collapse sample diversity. A random-token corruption experiment traces this deficit to the models themselves: they reconstruct clean and corrupted tokens with nearly identical accuracy, even though clean tokens are easier targets. A decomposition of the validation NELBO shows that training barely rewards retention: incorrect predictions are heavily penalized at corrupted positions but almost free at clean ones. We propose Correct-Token Retention Regularization (CTR-Reg), a simple but effective auxiliary loss that trains the model to retain tokens left unperturbed by the forward process and requires no change to the sampler. CTR-Reg improves clean-token accuracy by 26.5 percentage points on average across six benchmarks, while leaving corrupted-token accuracy virtually unchanged, and its per-step revisions converge to only 3–11 positions. With just five greedy-tail steps, generative perplexity more than halves under CTR-Reg for all three models while diversity is preserved, and these gains hold across sampling budgets. Our results identify correct-token retention as a key missing ingredient for self-correcting diffusion language models, and demonstrate an effective fix.

Primary area:
generative models
Keywords:
uniform-state diffusion language models, discrete diffusion language models, discrete diffusion models, diffusion language models, masked diffusion language models, non-autoregressive text generation, generative language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_NQGVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper designs a simple auxiliary loss for UDM training to improve the retention of correct tokens during generation. Experiments demonstrate improvements in correct token accuracy and PPL in the regime of greedy-tail decoding for UDMs.

Critical strengths
  1. The method to add a simple auxiliary loss to improve the correct token retention in UDM generation is clear and simple to follow.
Critical weaknesses
  1. The main concern of the experiments is that the training scale is too small: only about 20B tokens for 0.1B models, substantially less than all of the baselines in the field of DLMs, e.g. DUO (524B tokens) and SEDD (209B tokens).
  2. The small-scale training raises concerns about whether the 'retention issue' actually exists in the normal setting, because, theoretically, if UDMs are well-trained, the concrete score of 'clean to noisy conversion' in generation should be well suppressed. By the way, it is unclear why a lack of retention results in the collapse of diversity, the logic is not direct.
  3. The setting is limited to greedy-tail decoding of UDMs, and the experiments are merely about PPL, which raises concerns about whether the method could be beneficial for UDMs with standard training and inference in real downstream task evaluations.
Questions for authors

Please see the weaknesses section.

3 comments

We thank the reviewer for the review and for finding CTR-Reg clear and simple to follow. Your concerns prompted several new experiments, most importantly on the officially released, fully trained DUO checkpoints (169M on 524B tokens and 1.7B on 2.1T tokens); we address each concern below, and all revisions are marked in blue in the updated manuscript.

W1 & W2 (training scale; does the deficit exist in the normal setting; why it collapses diversity)

Thank you for these questions. To address them, we have fully rewritten Appendix F ("Scaling Data and Model Size Does Not Close the Retention Deficit") and added Appendix L (CTR-Reg fine-tuning of a pretrained checkpoint), which Reviewer UQDX requested and which also bears on your concern. All changes are marked in blue; the manuscript gives the full details, and we summarize them below. In short: the deficit is present in the official, fully trained DUO checkpoint you cite and at 1.7B/2.1T scale, and our experiments indicate that it comes from the training objective, not from limited data.

(1) Training scale, and whether the deficit exists in well-trained models. Our models use the same 169M backbone as the DUO and SEDD baselines, but a smaller token budget (20B) for compute reasons. To test whether the deficit is a small-scale artifact, Appendix F applies the corruption probe and greedy-tail sampling to the official DUO checkpoint trained on 524B tokens and to DUO-1.7B (1.7B parameters, 2.1T tokens). The 524B corruption probe was already in the original submission; the other results are new:

ModelTrain tokensCorrupted acc.Clean acc.GapEntropy, g = 0 → 20Real-text entropyChanged at g = 20
DUO 169M20B45.247.22.05.18 → 3.054.83173/512 (33.8%)
DUO 169M, official524B51.652.71.15.54 → 2.145.40357/1024 (34.9%)
DUO-1.7B, official2.1T70.572.52.05.75 → 3.075.92†354/2048 (17.3%)
DUO 169M + CTR-Reg20B46.377.431.15.19 → 5.044.8311/512 (2.2%)

Accuracy in %, mean over six corpora (Table 7); entropy and changed positions from Tables 8 and 18 and Figure 10. †Not trained on OpenWebText; indicative only.

Scale makes DUO a far better denoiser but raises clean- and corrupted-token accuracy together: the official checkpoints keep only 52.7% and 72.5% of the clean tokens, barely more than the corrupted tokens they recover (51.6% and 70.5%), so clean-to-noisy conversion is not suppressed even after 524B–2.1T training tokens. Both checkpoints also collapse under the greedy tail, and our 20B DUO rewrites almost the same share of positions as the 524B one (47.7% vs. 43.9% at g = 1; 33.8% vs. 34.9% at g = 20), so our small-scale models are representative. Fine-tuning the 524B checkpoint with CTR-Reg for 12.6B tokens (2.4% of its pretraining compute) widens its gap from 0.4–1.4 to 27.8–30.5 points on OWT, AG News, and arXiv (Appendix L). Since 26× more NELBO training leaves the deficit intact while a short CTR-Reg fine-tune removes it, the cause is the objective, not the data: the NELBO barely asks for retention. For 1 − α_t < 0.4, a wrong prediction at a clean position costs only 0.09–1.32 nats more than a correct one, vs. 13–100 nats at a corrupted position, and mispredicted clean positions make up 23–31% of positions but only 3.5–5.2% of the NELBO (Figure 5, Appendix E.2).

(2) Why missing retention collapses diversity. We explain this in Section 3.1 ("Where the collapse comes from") and Section 4.3. In brief, a greedy step sets every position to its argmax at once (Eq. 7). Without retention, the argmax does not favor the token already there, so about half the sequence is rewritten at the first step and 173–270 of 512 positions are still rewritten at the 20th (Figure 2). These edits are made independently per position, so they can be mutually inconsistent (Gu et al., 2018), and repeated mode-seeking drifts into self-reinforcing repetition, as with greedy decoding in AR models (Holtzman et al., 2020). Entropy thus falls at every step, and samples degenerate (NELBO-only DUO, g = 20):

… the, the, the, the, the, … Spain, Spain, Spain, Spain Spain, Spain, Spain Spain, … the banks the banks the banks, banks banks, banks banks …

Retention is the cause: fine-tuning the same 524B checkpoint with CTR-Reg cuts the positions changed at the first step from 449 to 173 and keeps entropy at 5.33 instead of 2.14 at g = 20 (Appendix L).

W3 (greedy-tail setting, PPL-only evaluation, downstream tasks)

Standard inference. The standard sampler of USDMs is ancestral sampling, and DUO's default sampler (Sahoo et al., 2025) already ends with one greedy step (g = 1). There, CTR-Reg lowers Gen PPL from 110.88 to 64.67 while raising entropy (4.96 → 5.09); under pure ancestral sampling (g = 0), it matches the baseline. As discussed in Section 3.1 and Appendix D, ancestral sampling hides the retention deficit rather than resolving it: the noise schedule lets only a small fraction of positions adopt the model's prediction at each step (at most about 1 in 5), so which positions get revised is decided by sampling stochasticity rather than by the model. The greedy tail removes this bound and exposes the deficit directly. Our paper therefore uses it as a tool to reveal the deficit, traces the deficit to the NELBO objective, and proposes a simple fix.

Beyond PPL. Our evaluation covers three aspects:

  • The deficit itself, independent of any sampler: reconstruction accuracy on six corpora, measured in a single forward pass (Figure 1, Table 9), and the NELBO decomposed by clean/corrupted position and correct/incorrect prediction, which localizes its cause (Figure 5).
  • Its effect in generation: the number of positions changed per greedy step (Figure 2).
  • Sample quality: the Gen PPL–entropy frontier, since PPL alone can be lowered by repetition (Figure 4); MAUVE, which compares generated and human text distributions (Figure 3); an LLM judge on five criteria (Appendix I, new); and Gen PPL and entropy across budgets of 16–1024 steps, under each model's default sampler and with a three-step greedy tail (Figure 8).

Standard training. CTR-Reg adds a single loss term to each model's own training recipe and leaves the sampler unchanged; we apply it to DUO, UDLM, and SEDD. It can also be added to an existing, conventionally trained checkpoint by a short fine-tune (2.4% of its pretraining compute; Appendix L).

Downstream tasks. To address your concern about real downstream evaluations, and following Reviewer SwXk's suggestion, we evaluate on two tasks increasingly used in recent evaluations of diffusion language models: training on TinyGSM (grade-school math problems with Python solutions) and testing on GSM8K, and Sudoku solving. We kindly ask you to refer to our response to Reviewer SwXk (W1) for the setup and results.

Dear Reviewer NQGV,

Thank you for your review and feedback on our submission.

We have posted a detailed response addressing the comments and concerns raised in your review. As the discussion period approaches its conclusion, we would greatly appreciate it if you could take a moment to review our response and let us know whether any questions or concerns remain. We would be more than happy to provide further clarification.

Thank you again for your time and consideration.

Best regards, The Authors

Reviewer_SwXkOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This work focuses on identifying, analyzing, and mitigating a phenomenon of uniform-state diffusion models (USDMs) which offer the benefit of self-correcting all predicted positions but tend to conflate correct token positions with incorrect ones. The authors find that USDMs achieve similar accuracy when predicting noised and un-noised positions, which they attribute to a failure for incentivizing the retention of correct tokens in the standard variational bound used to train these models. They therefore propose a regularization scheme that mitigates this issue and achieve substantial improvements when producing open-ended generation from these models.

Critical strengths
  • This paper tackles an important issue for USDMs (which are increasingly being adopted by frontier discrete diffusion models such as GemmaDiffusion).
  • The paper methodically identifies a key issue for these models, the source of this issue, and an easy-to-implement solution.
  • This paper is very clearly written and easy to follow.
  • The empirical gains for open-ended generation are convincing,
Critical weaknesses
  • Experiments on conditional generation would significantly improve the impact of the findings. For example, the tinyGSM dataset has proven to be a useful and popular small model setting where structured and conditional generation can be studied. Sudoku puzzles would also be an interesting domain, although I personally believe that tinyGSM would be more impactful.
  • The authors focus on diversity collapse coming from greedy-tail sampling. This seems a bit narrow to me in terms of a “key” weakness of USDMs. Can the same analysis not be also useful for identifying why overall performance of USDMs tends to lag behind that of MDLMs in many language modeling settings?
  • It would be useful to add other baseline methods (e.g. AR & MDLMs) as reference points to results, e.g., Figure 4
  • Introduction can be shortened; it goes into substantial detail about experimental setup and findings, which can be deferred to later in the text. For example, much of the Figure 1 experimental setup from Lines 375-383 is repeated in the introduction.

Minor:

  • In Figure 5, it might be simpler to just show the dashed-solid gap as a single line vs. having the reader need to infer this gap by visualizing the difference between the two line types.
  • Consider adding a comment that the choice of the “arXiv” dataset in Figure 1 is not cherry picked and is representative of the overall trend observed in Table 9.
Questions for authors
  • In Figure 5, for all noise regions (1−αt)>0.8(1 - \alpha_t) > 0.8, it seems that the clean position dominates the NELBO. Have the authors explored mitigating the USDM clean token retention issue by using a noising schedule that more heavily samples from this region or from regions where the NELBO contribution is more balanced for corrupted and clean positions?
  • Additionally, given that the NELBO contribution gap varies across noise levels, would a time varying λ\lambda parameter perhaps be more useful?
  • Another interesting ablation is in the form of the regularization. Here the authors elect to upweight the CE loss of clean positions. Have the authors also explored a simple binary cross entropy loss on each position that teaches the model to explicitly identify noised vs. un-noised tokens?
6 comments

We thank the reviewer for the positive assessment of the importance of the problem as USDMs gain adoption, the methodical diagnosis of the deficit and its source, the simplicity of the fix, the clarity of the writing, and the convincing gains in open-ended generation. We address the remaining comments below; all revisions are marked in blue in the updated manuscript.

W1 (conditional generation: TinyGSM and Sudoku)

Thank you for this suggestion; we ran both. Following your preference, TinyGSM is in the main text (new Section 4.4 and Figure 6; full setup and results in Appendix O), and Sudoku is in Appendix P.

TinyGSM → GSM8K. We follow the setup of Deschenaux & Gulcehre (2026): each example is a question followed by a Python program, and only the program is noised. We compare their released NELBO-only DUO with DUO + CTR-Reg trained from scratch with the same recipe (250k steps, λ = 0.02), both at their final checkpoint. On the 1,319 GSM8K test problems, the generated program is executed and checked against the reference answer, and we test significance with an exact McNemar test. We use three samplers with 32–1024 steps: ancestral sampling, predictor–corrector (the strongest uniform-state sampler on GSM8K in Deschenaux et al., 2026a), and confidence-based decoding (Nie et al., 2025).

Samplerτ32 steps512 steps1024 stepsSignificantly better
Ancestral113.3 → 15.517.7 → 18.417.5 → 16.20 / 6
Ancestral0.128.6 → 31.134.5 → 37.833.7 → 34.62 / 6
Predictor–corrector111.1 → 18.626.2 → 39.627.4 → 44.16 / 6
Predictor–corrector0.129.0 → 33.743.8 → 49.246.8 → 49.35 / 6
Confidence-based13.1 → 4.917.7 → 30.317.1 → 30.36 / 6
Confidence-based0.15.3 → 8.322.7 → 30.218.2 → 33.46 / 6
Confidence-based04.6 → 7.019.5 → 32.420.1 → 31.56 / 6

GSM8K accuracy (%), NELBO-only DUO → DUO + CTR-Reg. Bold: significantly better (p < 0.05, exact McNemar test); no setting is significantly worse. All step counts: Table 16 and Figure 6.

Under ancestral sampling, the two models are close. Under the two samplers that act on the model's prediction at every step, CTR-Reg is ahead in all 30 settings, significantly in 29, and its gain grows with the sampler's reliance on retention: under predictor–corrector at τ = 1, from 7.5 points at 32 steps to 16.7 at 1024. This matches our diagnosis. Predictor–corrector rewrites every position and re-noises at random, so a correct token survives a step with probability α_s⟨x_θ^τ, x⟩ + (1 − α_s)/K, which at τ = 1 is linear in the probability CTR-Reg raises; confidence-based decoding keeps a correct token only if it is re-predicted and ranked among the most confident. Ancestral sampling instead keeps a displayed token almost surely unless the model deems it noise, which hides the deficit (Section 3.1). Since the two models are close only where retention is hidden, the gains come from retention, not from generally better predictions.

Sudoku. We follow the setup of Deschenaux et al. (2026a): 9×9 puzzles with 30 of 81 cells given, 200k for training and 5k for evaluation, and their 8-layer DiT trained for 50k steps. With 180 sampling steps, CTR-Reg improves exact-match accuracy by about 3 points under both samplers (mean ± std over five seeds; Table 17): from 77.3 ± 1.8% to 80.1 ± 1.4% with ancestral sampling and from 93.0 ± 1.4% to 95.9 ± 1.2% with confidence-based decoding. Here ancestral sampling benefits as well: with a vocabulary of only 14 tokens, whether its posterior keeps a correct token depends on the probability the model gives it, which CTR-Reg raises. Both DUO models outperform the AR (3.1%) and masked diffusion (69.4%) baselines reported for this setup by Deschenaux et al. (2026a), whose simplex models, trained with a different objective, reach higher accuracy.

We believe these results substantially strengthen the impact of our findings, as you anticipated.

W2 (greedy-tail collapse as a "key" weakness; relation to the USDM–MDLM gap)

We agree that the collapse is a symptom. The underlying weakness is the retention deficit: measured without any sampler (Figure 1), present in the official 524B-token and 1.7B DUO checkpoints (Appendix F), and visible in generation as indiscriminate revision (Figure 2). It matters beyond long greedy tails: CTR-Reg improves DUO's default sampler (one greedy step) at every budget from 16 to 1024 steps (Figure 8), e.g., Gen PPL 110.88 → 64.67 with entropy 4.96 → 5.09, and it raises GSM8K accuracy under the standard samplers that leave retention to the model (see W1).

It also explains part of the gap to MDLMs. Self-correction requires revising wrong tokens and keeping correct ones; MDLMs get the latter for free (carry-over unmasking never revises a decoded token), whereas USDMs must learn it and the NELBO barely teaches it. So prediction-driven steps also revise correct tokens, ancestral sampling leaves revisions to chance, and self-correction, the main advantage of USDMs, cannot pay off. Appendix K (new) confirms this: at matched entropy, one greedy step leaves every NELBO-only USDM behind MDLM (Gen PPL 110.9/81.1/98.5 for DUO/UDLM/SEDD vs. 70.7/70.7/51.4), whereas with CTR-Reg, DUO and UDLM at g = 10 reach 47.4 and 41.9 at entropy ≈ 5.05, where MDLM, trained on 262B tokens (over 10× ours), needs 113.9 and 108.3 even with tuned nucleus sampling.

Retention does not explain the likelihood gap (CTR-Reg changes validation perplexity by at most 2.4%; Table 3); our NELBO decomposition suggests looking instead at mispredicted corrupted positions, which carry 90–92% of the USDM NELBO and which CTR-Reg does not target (Figure 5, Appendix E.2).

We will reframe the collapse as a symptom in the introduction and add this discussion. If you have a specific setting in mind where USDMs lag behind MDLMs, we would be happy to discuss it.

W3 (AR and MDLM reference points)

Thank you for the suggestion; we added both baselines. Figure 4 now shows AR and MDLM as hollow reference points under their standard samplers, and since Gen PPL can always be lowered at the cost of entropy, Appendix K (new) also sweeps both over nucleus p from 0.8 to 1 to compare at matched entropy. AR is a 181M Llama-style transformer we trained on OpenWebText for 20B tokens, our budget; MDLM is the released 169M checkpoint of von Rütte et al. (2025), trained on 262B tokens. All models generate 512-token sequences in 512 sampling steps.

With CTR-Reg, the greedy tail moves DUO and UDLM well below MDLM's curve: at g = 10 they reach Gen PPL 47.4 and 41.9 at entropy ≈ 5.05, where MDLM needs 113.9 and 108.3 even with tuned nucleus sampling, and SEDD + CTR-Reg is on par with MDLM (84.3 vs. 88.7); without CTR-Reg, one greedy step already leaves every USDM behind MDLM (see W2). AR remains the strongest at matched entropy (Gen PPL 27–28 at entropy ≈ 5.05), consistent with the gap between diffusion and AR models reported at this scale (Sahoo et al., 2024); closing that gap is not the goal of CTR-Reg.

W4 (shortening the introduction)

Thank you for the suggestion. In the final version, we will shorten the introduction by moving the setup details of the corruption experiment and the NELBO decomposition to Sections 3 and 4, keeping only the main findings and contributions. We will use the space saved to bring key results from the rebuttal into the main text.

Minor comments

Figure 5. We agree that plotting the gap as a single line would be easier to read, and we will consider this for the final version.

arXiv in Figure 1. Thank you for the heads-up; we have updated the Figure 1 caption to note that both corpora shown are representative of the six in Table 9.

Q1 (sampling high-noise regions more heavily)

Thank you for this suggestion. We tested it on UDLM with the same 20B-token training as our main runs: instead of sampling all training times t uniformly, we draw 25% of them from the high-noise region 1 − α_t ∈ [0.8, 1], keeping the log-linear schedule, the loss, and all other settings unchanged.

UDLMNoise samplingCorrupted acc. (OWT / AG News / arXiv)Clean acc. (OWT / AG News / arXiv)Entropy, g = 0 → 20
NELBOuniform55.3 / 42.8 / 47.456.7 / 46.0 / 51.55.18 → 3.49
NELBO25% from [0.8, 1]53.4 / 40.8 / 45.063.1 / 51.1 / 56.75.20 → 4.16
NELBO + CTR-Reguniform55.2 / 42.6 / 47.680.1 / 68.2 / 73.05.17 → 5.04

Accuracy in %, averaged over p ∈ {10, 20, 30, 40}% (Table 9); entropy from Table 18 (real text: 4.83).

Emphasizing high noise helps retention only partially, and at a cost: clean-token accuracy rises by 5.1–6.4 points, about a quarter of CTR-Reg's gain (21.5–23.4), while corrupted-token accuracy, which CTR-Reg leaves unchanged, drops by 1.9–2.4 points. The greedy-tail collapse is milder, but entropy still falls below that of real text (4.16 at g = 20 vs. 4.83).

This is what our NELBO decomposition predicts. Biasing the sampling toward high noise gives more weight to 1 − α_t ≥ 0.8, where the NELBO already penalizes clean errors. But since it only reweights noise levels, it leaves the imbalance at lower noise in place: there, a clean error costs only 0.09–1.32 nats, against 13–100 for a corrupted one (1 − α_t < 0.4; Figure 5).

This extra weight on high noise explains the partial gain. However, at 1 − α_t ≥ 0.8 most of the context is itself corrupted, so the model cannot use the context to recognize clean tokens; it can only learn to keep displayed tokens regardless of context. This is not the retention needed at low noise, where the mostly correct context should decide which tokens to keep. It may even lead the model to keep some corrupted tokens, consistent with the drop in corrupted-token accuracy. CTR-Reg instead adds a loss on clean positions at every noise level, including the low-noise levels where the context supports retention. We have added this ablation to the paper as Appendix M (Table 14).

Q2 (time-varying CTR-Reg weight)

Thank you for this great suggestion. We had already explored this direction experimentally and have now added the results to the paper as Appendix J (new). As you note, the gap is largest at low noise and vanishes at the highest noise, where the NELBO already penalizes clean-position errors (Figure 5). On UDLM, we therefore compared our constant λ = 0.02 with two variants that concentrate the weight at low noise (same 169M model and 20B OpenWebText tokens; noise level t ≈ 1 − α_t under the log-linear schedule):

  • Cut-off: λ(t) = 0.02 for t ≤ 0.8, else 0, dropping the term where the NELBO already supplies the signal;
  • Cut-off + decay: λ(t) = 0.02 (1 − t) for t ≤ 0.8, else 0.
NELBO onlyConstant (ours)Cut-offCut-off + decay
Clean-token acc. (OWT / AG News / arXiv)56.7 / 46.0 / 51.580.1 / 68.2 / 73.079.1 / 68.0 / 72.278.8 / 67.3 / 71.5
Corrupted-token acc.55.3 / 42.8 / 47.455.2 / 42.6 / 47.654.9 / 43.1 / 48.155.0 / 42.7 / 47.6
Gen PPL (16 / 128 / 1,024 steps)105.4 / 92.2 / 92.056.7 / 48.3 / 47.657.3 / 49.8 / 48.357.8 / 49.9 / 47.9
Entropy (all budgets)4.68–4.825.00–5.065.00–5.065.00–5.05

Table 11; NELBO only from Tables 1 and 19. Accuracy (%) averaged over p ∈ {10, 20, 30, 40}%; three greedy-tail steps; real-text entropy: 4.83.

Neither variant beats the constant weight: it has the highest clean-token accuracy on every corpus (by 0.2–1.5 points) and the lowest Gen PPL at every budget from 16 to 1,024 steps (by 0.3–1.6), while corrupted-token accuracy and entropy differ by at most 0.5 and 0.01. Since the variants keep 93–99% of CTR-Reg's clean-token gain, nearly all of it comes from the term at t ≤ 0.8, where the NELBO lacks the clean-position signal, consistent with your observation; but neither dropping the term above 0.8 nor decaying it helps. We therefore keep the constant λ = 0.02 (its magnitude is ablated in Section 4.5), so CTR-Reg adds only one hyperparameter.

Reviewer_UQDXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates the correct-token retention problem in uniform-state diffusion language models (USDMs). Unlike masked diffusion models, USDMs allow all token positions to be revised during denoising, potentially enabling self-correction. However, the authors observe that existing USDMs frequently modify already-correct tokens, particularly under greedy-tail decoding, leading to a substantial collapse in sample diversity.

To address this problem, the paper proposes Correct-Token Retention Regularization (CTR-Reg), an auxiliary training objective that explicitly encourages the model to retain tokens that remain unchanged during the forward corruption process. The authors also extend the regularization to score-parameterized models while leaving the sampling procedure unchanged.

The experimental results demonstrate that CTR-Reg improves clean-token reconstruction accuracy by an average of 26.5 percentage points across six evaluation corpora, while leaving corrupted-token accuracy nearly unchanged. The method also substantially reduces generative perplexity and mitigates diversity collapse under greedy-tail decoding across three evaluated models: DUO, UDLM, and SEDD.

The paper aims to demonstrate that correct-token retention is an important capability for USDMs and that explicitly incorporating this objective into training can improve their generation behavior.

Critical strengths
  1. The paper identifies a concrete and relevant failure mode in uniform-state diffusion language models and distinguishes correct-token retention from the ability to correct corrupted tokens.

  2. The controlled random-token corruption experiments provide a clear diagnostic for evaluating these two capabilities separately.

  3. CTR-Reg is simple to implement, introduces little additional computational overhead during training, and does not require modifying the sampling procedure.

  4. The evaluation covers three USDM formulations, six reconstruction-evaluation corpora, multiple sampling-step budgets, and different regularization strengths.

  5. The experimental results consistently demonstrate improved clean-token accuracy and reduced diversity collapse in the evaluated settings.

Critical weaknesses
  1. Limited methodological novelty. CTR-Reg is a relatively straightforward auxiliary loss that encourages the model to preserve tokens known to be correct during training. Although effective, the paper does not sufficiently distinguish this formulation from more general auxiliary reconstruction, copy-preservation, or consistency-based objectives.

  2. Insufficient comparison with alternative objectives. The experiments demonstrate that CTR-Reg works, but do not establish that its specific formulation is necessary or superior to simpler alternatives. This makes it difficult to assess the incremental algorithmic contribution.

  3. Limited generation-quality evaluation. The principal generation experiments focus on unconditional OpenWebText generation and automated metrics. Human evaluation and broader assessments of semantic quality are missing, limiting the evidence that improvements in the reported metrics translate into better overall text generation.

  4. Unclear applicability to pretrained models. The main experiments train models from scratch using the modified objective. The effectiveness and computational cost of applying CTR-Reg to existing pretrained checkpoints through fine-tuning remain unclear.

  5. Limited evidence of scalability. The main experiments use 169M-parameter models trained on approximately 20B tokens. The generality of the results at substantially larger model scales has not been established.

Overall, the paper presents a useful empirical diagnosis and an effective intervention. However, the limited methodological novelty and the absence of stronger comparisons with alternative training objectives weaken the case for acceptance.

Questions for authors
  1. Comparison with alternative training objectives. CTR-Reg explicitly encourages the model to preserve tokens that are known to be correct during training. How does the proposed formulation differ conceptually and empirically from simpler auxiliary reconstruction, copy-preservation, or consistency-based losses? Could the authors compare CTR-Reg against these alternatives under identical training and sampling configurations? Consistent improvements over these alternatives would strengthen the claim that the proposed formulation represents a meaningful methodological contribution.

  2. Applicability to pretrained models. The primary experiments train models from scratch using the modified objective. Can CTR-Reg be applied to existing pretrained USDM checkpoints through fine-tuning, and what additional training cost is required to obtain the reported improvements? A successful fine-tuning experiment would strengthen the practical case for the method, whereas a requirement for full retraining would limit its applicability.

  3. Generalization of generation improvements. Can the authors evaluate generation quality on additional domains or conditional generation tasks, preferably using human judgments or semantic-quality metrics? Such results would help determine whether the improvements in generative perplexity, entropy, and MAUVE translate into consistently better text generation beyond the current evaluation setting.

  4. Scalability of correct-token retention. Do the authors expect the retention deficit and the effectiveness of CTR-Reg to persist at substantially larger model scales? Evidence from larger models would help establish whether the proposed intervention addresses a general limitation of USDMs or primarily benefits the configurations evaluated in this paper.

8 comments
AuthorsOpenReview ↗

Clarification request on alternative objectives

Dear Reviewer UQDX,

Thank you for your review. We are preparing a full response with additional experiments and want to make sure we compare against the objectives you have in mind.

Could you kindly point us to the works you are referring to for the auxiliary reconstruction, copy-preservation, and consistency-based objectives? Given the short discussion period, this would help us prioritize the most relevant comparisons.

Best regards, The Authors

Thank you for asking for specific references. My earlier wording was too broad, and I should distinguish related ideas from directly comparable training objectives.

  • Auxiliary reconstruction. A concrete reference is Austin et al., Structured Denoising Diffusion Models in Discrete State-Spaces (NeurIPS 2021), Section 3.4, Eq. (5). Their objective adds an auxiliary cross-entropy term for predicting the original data from a noisy input. In your denoiser-parameterized formulation, CTR-Reg instead restricts the additional supervision to positions satisfying zt=xz_t=x. I recognize that you already cite Austin et al.; the point to clarify is the relationship between these objectives, rather than a missing citation.

  • Copy-preservation. A relevant conceptual contrast is the carry-over unmasking parameterization in Sahoo et al., Simple and Effective Masked Diffusion Language Models (NeurIPS 2024). However, this preserves unmasked tokens by construction under masked diffusion. It is not a directly comparable auxiliary objective for learning selective retention under uniform corruption, and I should not have grouped the two together without that qualification.

  • Consistency-based objectives. One concrete example is Amin et al., Consistent Diffusion Language Models (ICML 2026), which introduces denoiser consistency across stochastic posterior bridges. This concerns a different constraint from your clean-position retention objective. I would treat it as conceptual context, not as evidence that CTR-Reg has already been proposed or as a required experimental baseline.

The specific question I would like to retain is whether, in the DUO/UDLM setting, restricting the additional reconstruction supervision to clean positions provides benefits beyond adding a generic auxiliary reconstruction term. A concise discussion of this distinction, together with any available supporting evidence, would be helpful. I am not requesting a broad set of new comparisons during the discussion period.

I appreciate the opportunity to clarify this point and will reassess the novelty concern in light of your explanation and full response.

We thank the reviewer for the careful review and for recognizing the concrete failure mode we identify, the corruption probe that separates retention from correction, the simplicity and low cost of CTR-Reg, the breadth of the evaluation, and the consistency of the gains. We address the remaining concerns below, including the requested comparison with alternative objectives; all revisions are marked in blue in the updated manuscript.

W1, W2 & Q1 (relation to alternative objectives)

Thank you for the quick clarification. CTR-Reg is simple by design: our main contribution is diagnosing the retention deficit and its source in the NELBO, and CTR-Reg is the minimal fix this implies.

Clean-only vs. generic auxiliary reconstruction. The auxiliary cross-entropy of Austin et al. (Eq. 5) covers all positions, i.e., CTR-Reg's clean-position term plus a corrupted-position term. The NELBO's signal already sits on corrupted positions (mispredicted ones carry 90–92% of it, vs. 3.5–5.2% for mispredicted clean ones; Figure 5, Appendix E.2), so CTR-Reg adds only the missing clean-position signal. To test whether this restriction matters, we trained UDLM with two all-position alternatives under identical training (20B tokens, λ = 0.02) and sampling: (1) Austin et al.'s cross-entropy and (2) a clean-vs-corrupted binary cross-entropy on the model's probability of the displayed token, suggested by Reviewer SwXk. Both equal CTR-Reg plus a corrupted-position term, so the comparison isolates exactly this restriction (Appendix N, new):

Corrupted acc. ↑Clean acc. ↑Overall acc. ↑Gen PPL ↓, g = 0 / 3 / 10 / 20Entropy, g = 0 → 20
NELBO only55.3 / 42.8 / 47.456.7 / 46.0 / 51.556.1 / 45.0 / 50.395.6 / 91.3 / 59.0 / 25.75.18 → 3.49
CTR-Reg (clean positions)55.2 / 42.6 / 47.680.1 / 68.2 / 73.073.6 / 61.9 / 66.793.3 / 47.7 / 41.9 / 42.65.17 → 5.04
CE, all positions (Austin et al.)53.4 / 39.9 / 44.180.0 / 67.1 / 71.073.0 / 60.0 / 64.498.1 / 52.0 / 46.2 / 45.75.16 → 5.02
BCE, all positions53.7 / 40.1 / 44.980.0 / 67.9 / 71.773.1 / 60.6 / 65.397.2 / 51.8 / 46.8 / 44.65.15 → 5.01

Table 15. Accuracy (%) on OWT / AG News / arXiv, averaged over p ∈ {10, 20, 30, 40}%; Gen PPL with g greedy-tail steps in a 512-step budget (real-text entropy: 4.83; NELBO-only's low PPL at large g reflects repetition). Bold: best of the three auxiliary losses.

Both alternatives contain CTR-Reg's clean-position term and therefore also improve retention and avoid the greedy-tail collapse, which confirms our diagnosis. Restricting the term to clean positions provides benefits beyond this: the extra corrupted-position term lowers corrupted-token accuracy by 1.5–3.5 points, whereas CTR-Reg keeps it within 0.2 points of NELBO-only, so CTR-Reg has the highest overall accuracy on every corpus (by 0.5–2.3 points) and the lowest Gen PPL at every g from 0 to 20 (by 2.0–5.8), at equal or higher entropy.

Austin et al. introduced their term for a different problem, to stabilize training, and evaluated it by likelihood without considering retention; it helped their absorbing-state model but hurt their uniform-state ones (their Sections 5.1–5.2 and Appendix B.2), and USDMs have since been trained with the NELBO alone. Our results suggest why its value for USDMs went unnoticed: the likelihood barely reflects retention (CTR-Reg changes validation perplexity by at most 2.4%; Section 3.2). Identifying the retention deficit is what reveals that USDMs need a clean-position term beside the NELBO.

Sahoo et al. (carry-over). MDLM keeps unmasked tokens by construction: because of carry-over, unmasked positions receive no loss during training and are never revised during inference. This works only because the mask reveals which tokens are clean. In USDMs, clean and corrupted tokens look alike, so retention must be learned; CTR-Reg learns it selectively, copying corrupted tokens at under 0.25% of corrupted positions (Table 1).

Amin et al. Their consistency loss aligns the model's predictions across noise levels for few-step generation, whereas CTR-Reg supervises clean positions with the ground truth. The two are complementary, and we will discuss this in the related work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Additional Experiments

General response to all reviewers

We thank all reviewers for their thoughtful feedback. Reviewers found the paper clear and easy to follow (all three), the identified failure mode important and relevant for USDMs (SwXk, UQDX), the diagnosis methodical and the corruption probe a clear diagnostic (SwXk, UQDX), CTR-Reg simple to implement and cheap (all three), and the gains convincing and consistent (SwXk, UQDX). Our main contribution is the diagnosis: current USDMs fail to retain correct tokens because their NELBO barely rewards retention; CTR-Reg is the minimal fix this implies.

We apologize for the timing of our responses: several of the requested experiments required training new models. All changes in the revised manuscript are marked in blue; the main additions are:

  • Scale. The retention deficit persists in the officially released, fully trained DUO checkpoints (169M on 524B tokens; 1.7B on 2.1T tokens), and a short CTR-Reg fine-tune of the 524B checkpoint (2.4% of its pretraining compute) removes it (Appendices F and L).
  • Alternative objectives. An all-position cross-entropy and a clean-vs-corrupted binary cross-entropy (Appendix N), a time-varying weight (Appendix J), and sampling high noise levels more often (Appendix M); restricting the term to clean positions with a constant weight performs best.
  • Generation quality. An LLM judge on five criteria (Appendix I) and AR and MDLM reference points (Figure 4, Appendix K).
  • Conditional generation. On TinyGSM → GSM8K, CTR-Reg is ahead in all 30 settings of the two samplers that act on the model's prediction at every step (significantly in 29), e.g., 43.8% → 49.2% with predictor–corrector at 512 steps, and on Sudoku it adds about 3 points under both samplers (Section 4.4, Appendices O and P; details in our response to Reviewer SwXk, W1).

We respond to each reviewer's comments in the individual threads.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.