- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper designs a simple auxiliary loss for UDM training to improve the retention of correct tokens during generation. Experiments demonstrate improvements in correct token accuracy and PPL in the regime of greedy-tail decoding for UDMs.
Critical strengths
- The method to add a simple auxiliary loss to improve the correct token retention in UDM generation is clear and simple to follow.
Critical weaknesses
- The main concern of the experiments is that the training scale is too small: only about 20B tokens for 0.1B models, substantially less than all of the baselines in the field of DLMs, e.g. DUO (524B tokens) and SEDD (209B tokens).
- The small-scale training raises concerns about whether the 'retention issue' actually exists in the normal setting, because, theoretically, if UDMs are well-trained, the concrete score of 'clean to noisy conversion' in generation should be well suppressed. By the way, it is unclear why a lack of retention results in the collapse of diversity, the logic is not direct.
- The setting is limited to greedy-tail decoding of UDMs, and the experiments are merely about PPL, which raises concerns about whether the method could be beneficial for UDMs with standard training and inference in real downstream task evaluations.
Questions for authors
Please see the weaknesses section.
3 comments
Part 2/2
W3 (greedy-tail setting, PPL-only evaluation, downstream tasks)
Standard inference. The standard sampler of USDMs is ancestral sampling, and DUO's default sampler (Sahoo et al., 2025) already ends with one greedy step (g = 1). There, CTR-Reg lowers Gen PPL from 110.88 to 64.67 while raising entropy (4.96 → 5.09); under pure ancestral sampling (g = 0), it matches the baseline. As discussed in Section 3.1 and Appendix D, ancestral sampling hides the retention deficit rather than resolving it: the noise schedule lets only a small fraction of positions adopt the model's prediction at each step (at most about 1 in 5), so which positions get revised is decided by sampling stochasticity rather than by the model. The greedy tail removes this bound and exposes the deficit directly. Our paper therefore uses it as a tool to reveal the deficit, traces the deficit to the NELBO objective, and proposes a simple fix.
Beyond PPL. Our evaluation covers three aspects:
- The deficit itself, independent of any sampler: reconstruction accuracy on six corpora, measured in a single forward pass (Figure 1, Table 9), and the NELBO decomposed by clean/corrupted position and correct/incorrect prediction, which localizes its cause (Figure 5).
- Its effect in generation: the number of positions changed per greedy step (Figure 2).
- Sample quality: the Gen PPL–entropy frontier, since PPL alone can be lowered by repetition (Figure 4); MAUVE, which compares generated and human text distributions (Figure 3); an LLM judge on five criteria (Appendix I, new); and Gen PPL and entropy across budgets of 16–1024 steps, under each model's default sampler and with a three-step greedy tail (Figure 8).
Standard training. CTR-Reg adds a single loss term to each model's own training recipe and leaves the sampler unchanged; we apply it to DUO, UDLM, and SEDD. It can also be added to an existing, conventionally trained checkpoint by a short fine-tune (2.4% of its pretraining compute; Appendix L).
Downstream tasks. To address your concern about real downstream evaluations, and following Reviewer SwXk's suggestion, we evaluate on two tasks increasingly used in recent evaluations of diffusion language models: training on TinyGSM (grade-school math problems with Python solutions) and testing on GSM8K, and Sudoku solving. We kindly ask you to refer to our response to Reviewer SwXk (W1) for the setup and results.
Dear Reviewer NQGV,
Thank you for your review and feedback on our submission.
We have posted a detailed response addressing the comments and concerns raised in your review. As the discussion period approaches its conclusion, we would greatly appreciate it if you could take a moment to review our response and let us know whether any questions or concerns remain. We would be more than happy to provide further clarification.
Thank you again for your time and consideration.
Best regards, The Authors
Part 1/2
We thank the reviewer for the review and for finding CTR-Reg clear and simple to follow. Your concerns prompted several new experiments, most importantly on the officially released, fully trained DUO checkpoints (169M on 524B tokens and 1.7B on 2.1T tokens); we address each concern below, and all revisions are marked in blue in the updated manuscript.
W1 & W2 (training scale; does the deficit exist in the normal setting; why it collapses diversity)
Thank you for these questions. To address them, we have fully rewritten Appendix F ("Scaling Data and Model Size Does Not Close the Retention Deficit") and added Appendix L (CTR-Reg fine-tuning of a pretrained checkpoint), which Reviewer UQDX requested and which also bears on your concern. All changes are marked in blue; the manuscript gives the full details, and we summarize them below. In short: the deficit is present in the official, fully trained DUO checkpoint you cite and at 1.7B/2.1T scale, and our experiments indicate that it comes from the training objective, not from limited data.
(1) Training scale, and whether the deficit exists in well-trained models. Our models use the same 169M backbone as the DUO and SEDD baselines, but a smaller token budget (20B) for compute reasons. To test whether the deficit is a small-scale artifact, Appendix F applies the corruption probe and greedy-tail sampling to the official DUO checkpoint trained on 524B tokens and to DUO-1.7B (1.7B parameters, 2.1T tokens). The 524B corruption probe was already in the original submission; the other results are new:
Accuracy in %, mean over six corpora (Table 7); entropy and changed positions from Tables 8 and 18 and Figure 10. †Not trained on OpenWebText; indicative only.
Scale makes DUO a far better denoiser but raises clean- and corrupted-token accuracy together: the official checkpoints keep only 52.7% and 72.5% of the clean tokens, barely more than the corrupted tokens they recover (51.6% and 70.5%), so clean-to-noisy conversion is not suppressed even after 524B–2.1T training tokens. Both checkpoints also collapse under the greedy tail, and our 20B DUO rewrites almost the same share of positions as the 524B one (47.7% vs. 43.9% at g = 1; 33.8% vs. 34.9% at g = 20), so our small-scale models are representative. Fine-tuning the 524B checkpoint with CTR-Reg for 12.6B tokens (2.4% of its pretraining compute) widens its gap from 0.4–1.4 to 27.8–30.5 points on OWT, AG News, and arXiv (Appendix L). Since 26× more NELBO training leaves the deficit intact while a short CTR-Reg fine-tune removes it, the cause is the objective, not the data: the NELBO barely asks for retention. For 1 − α_t < 0.4, a wrong prediction at a clean position costs only 0.09–1.32 nats more than a correct one, vs. 13–100 nats at a corrupted position, and mispredicted clean positions make up 23–31% of positions but only 3.5–5.2% of the NELBO (Figure 5, Appendix E.2).
(2) Why missing retention collapses diversity. We explain this in Section 3.1 ("Where the collapse comes from") and Section 4.3. In brief, a greedy step sets every position to its argmax at once (Eq. 7). Without retention, the argmax does not favor the token already there, so about half the sequence is rewritten at the first step and 173–270 of 512 positions are still rewritten at the 20th (Figure 2). These edits are made independently per position, so they can be mutually inconsistent (Gu et al., 2018), and repeated mode-seeking drifts into self-reinforcing repetition, as with greedy decoding in AR models (Holtzman et al., 2020). Entropy thus falls at every step, and samples degenerate (NELBO-only DUO, g = 20):
Retention is the cause: fine-tuning the same 524B checkpoint with CTR-Reg cuts the positions changed at the first step from 449 to 173 and keeps entropy at 5.33 instead of 2.14 at g = 20 (Appendix L).