- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper provides a theoretical analysis of the time sensitivity of uniform discrete diffusion models through the lens of match-count decomposition, and points out that the uniformly corrupted data can be easily separable in a finite-data setting. Experiments on various UDM frameworks have demonstrated the performance of time-agnostic UDMs on validation PPL and generative quality.
Critical strengths
-
The analysis framework (match-count decomposition, separability, Fisher time sensitivity, etc.) of UDM time sensitivity is rigorous and innovative.
-
The experimental results on validation PPL and generation quality of time-agnostic UDM are comprehensive.
Critical weaknesses
-
This paper only covers the analysis of full-sequence UDM. But in practice, the mainstream of DLM is block diffusion. As in Section 4 of this paper, the Delta-separability is defined with the minimum Hamming distance d(x), which should be positively correlated to sequence length, so what will be the case of block diffusion, whose block size is typically much smaller (e.g. 32) than the full-sequence setting in this paper (OWT with 1024 context length)? Does the conclusion on time sensitivity still hold for block diffusion, or we have to use a long sequence length to obtain the insensitivity of time?
-
This paper only removes the time in parameterization. However, in sampling, it still needs time to implement the reverse kernel and to determine when to stop the generation (at t=0), so it is not a completely time-independent framework. As in the standard practice of MDLM, confidence-based sampling algorithms are the most performative, which completely eliminates the usage of time, so is it possible for UDM sampling to be completely time-independent, with the ability to know when to stop the UDM generation?
-
The approach and the baselines achieve similar performance, which raises questions about the necessity of designing time-agnostic UDMs
-
This paper does not provide large-scale experiments, e.g. accuracy on generative tasks like GSM8K and HumanEval, making it unclear whether the time-agnostic UDMs are still performative in standard language benchmarks. Together with weakness 2, the generative evaluation of downstream tasks of DLMs is often not based on the reverse kernel sampler, as it is so random, but on more deterministic samplers such as the confidence-based ones without time.
Questions for authors
Please see the weaknesses section.
3 comments
Author Rebuttal (2/2)
Intuition behind sampler. Our sampler is motivated by two properties of the UDM reverse kernel (Eq. 3):
-
Token proposal: The distribution over replacement tokens contains a mixture of the uniform distribution and the model prediction , motivating our choice to sample candidates from .
-
Token replacement: For a transition from the current token to another token , the reverse kernel satisfies
= C_{s,t}\frac{1+\rho_s p_\theta^i(b\mid z_t)}{1+\rho_t p_\theta^i(a\mid z_t)},$$ where $C_{s,t}$ is independent of the token position and prediction. Thus, replacements are favored when the proposed token has high probability and the current token has low probability.
Inspired by these properties, we use for candidate sampling and the confidence difference for replacement prioritization. Although the original reverse kernel remains time-dependent and stochastic, our heuristic captures these preferences without explicitly using diffusion time.
W4. Provide large-scale experiments and sampler that satisfies the specified conditions in W2.
To answer this question with above W2, we (1) evaluate additional large-scale results on GSM8K and (2) evaluate a new sampling strategy that satisfies the properties highlighted by the reviewer.
Experimental Setup. We pretrain both 1.1B-parameter time-variant and time-agnostic UDMs on 445B tokens from the Nemotron subset of Nemotron-CC-v2, followed by 40 epochs of supervised fine-tuning on Augmented GSM8K. We then evaluate both models on the GSM8K test set with a generation length of 256 tokens.
Results. We compare ancestral sampling with our proposed Confidence Gain Sampler across different sampling budgets. Values represent GSM8K accuracy (%), with the average number of sampling steps shown in parentheses when early stopping is enabled.
| Sampling Method | 256 Steps | 128 Steps | 64 Steps | 32 Steps |
|---|---|---|---|---|
| Time-variant UDM (Ancestral) | 38.97 | 38.36 | 37.91 | 35.63 |
| Time-agnostic UDM (Ancestral) | 38.44 | 38.06 | 38.21 | 36.32 |
| Confidence Gain (w/o ES) | 51.33 | 51.10 | 51.33 | 48.14 |
| Confidence Gain (ES, ) | 51.55 (178.42) | 51.48 (90.62) | 51.48 (46.81) | 48.07 (24.86) |
ES denotes early stopping. Numbers in parentheses indicate the average number of sampling steps actually used.
- Even on downstream generation benchmarks with large-scale models, time-agnostic models achieve performance comparable to that of time-variant models.
- Further performance improvements can be achieved even without relying on ancestral sampling, which requires time information.
We believe this result highlights promising opportunities for further research on time-agnostic UDMs, while developing a time-homogeneous sampling process that provably preserves the target generative distribution remains an important direction for future work.
For further details on the experimental setup and sampling strategy, please refer to Appendices C.4 and C.3 of the revised paper, respectively.
W3. If the performance is similar, why should we use time-agnostic models?
We believe that similar performance is itself a strong motivation for using time-agnostic models. If explicit time conditioning provides little benefit, removing it simplifies the model and offers greater flexibility for future developments. We highlight two examples where this simplicity can be particularly useful:
- The improved sampler presented above is a simple example enabled by time-agnostic modeling. With a time-variant model, designing such a sampler would require careful consideration of the reverse kernel and marginal probability evolution, along with how the time input should be assigned throughout the sampling process. In contrast, a time-agnostic model allows greater freedom in designing sampling strategies without these constraints.
- A growing trend in diffusion language modeling is to initialize models from pretrained autoregressive LMs. Since autoregressive models do not take time as input, introducing time conditioning requires additional architectural choices. If time conditioning offers little practical benefit, time-agnostic modeling allows us to avoid considering about this ambiguous design choice entirely, making it more straightforward to leverage pretrained autoregressive models.
We believe that such simplicity can facilitate further advances in UDMs, beyond the examples discussed here. We hope these considerations help clarify the practical value of removing unnecessary time conditioning, even when the immediate performance gains are limited.
We sincerely thank the reviewer again for detailed and thoughtful review. We would be happy to address any further questions or concerns.
A Kind Reminder for Comments
We sincerely thank the reviewer for the thoughtful feedback. We highlight three key additions addressing the reviewer's concerns:
- Block diffusion: Added theoretical discussion and experiments with block size 16, showing competitive time-agnostic performance.
- Time-independent sampling: Proposed a Confidence Gain Sampler supporting adaptive early stopping without diffusion time.
- Large-scale evaluation: Added 1.1B-scale GSM8K experiments, where our proposed sampler improves accuracy from approximately 38% to over 51%.
We appreciate the reviewer’s time and would be happy to address any remaining questions.
Author Rebuttal (1/2)
We thank the reviewer for the valuable feedback! We have addressed all of the reviewer's questions and highlighted the changes in the revised paper in blue.
We thank the reviewer for raising this important question! While directly applying the Δ-separability analysis in Section 4 to shorter blocks may weaken the theoretical guarantee, block diffusion provides an additional source of separation through its clean prefix conditioning.
For block diffusion with block size B, let Z denote the corrupted sequence of the current block, spanning positions jB+1 to (j+1)B. For any position i within this block, the empirical-optimal predictor is \begin{equation} x_\star^i(z,t \mid x^{1:jB})=\mathbb{E}_{\hat{p}_N}[X^i | X^{1:jB}=x^{1:jB}, Z^{-i}=z^{-i}, T=t]. \end{equation}
Since the clean prefix must match exactly, only training sequences sharing x1:jB contribute to this expectation. Thus, this is equivalent to applying Theorem 2 to the subset of training data with the same prefix, where L=B, K is the vocabulary size, and N is the number of prefix-matched samples. Empirically, we found that distinct training samples almost never share identical prefixes once the prefix length exceeds 16 tokens in OWT dataset. Consequently, N=1 in nearly all such cases, making the empirical-optimal predictor exactly time-independent, regardless of the block size.
Nevertheless, as discussed in Section 5.1, the behavior of the empirical optimum does not necessarily extend to learned models. We therefore additionally train block uniform diffusion for 1M steps and evaluate on LM1B with a block size of 16:
The results show that time-agnostic models achieve comparable validation perplexity even with short diffusion blocks, suggesting that explicit time conditioning remains largely unnecessary in block diffusion.
We thank the reviewer again for this insightful question. We have added the corresponding discussion to Appendix B.7.
We thank the reviewer for this important suggestion. We would like to clarify that our primary goal is to understand whether explicit time conditioning is necessary for effective UDM learning, rather than to eliminate time dependence from the entire generative process.
Nevertheless, motivated by the reviewer's suggestion, we further investigate whether time-agnostic UDMs can support generation without an explicit diffusion-time schedule. To this end, we propose a simple confidence-guided sampler that prioritizes token updates based on model predictions and supports adaptive early stopping, without requiring diffusion time. Our experiments further show that the proposed sampler outperforms standard ancestral reverse-kernel sampling, as detailed in our response to W4.
Proposed Confidence Gain Sampler. Given a generation length L, a sampling budget of N steps, and an optional early-stopping threshold ϵ, we update k=⌊L/N⌋ tokens per step. We denote the model prediction xθ(z) by pθ(⋅∣z). Starting from a uniformly sampled sequence z, we repeat:
Sample candidates. Sample x^i∼pθi(⋅∣z) for each position i.
Compute confidence margins. Compute
Δi=pθi(x^i∣z)−pθi(zi∣z),
and select the top-k positions with the largest margins, denoted by I.
Early stopping (optional). Terminate if maxiΔi<ϵ.
Update. Set zi←x^i for all i∈I.
Repeat for at most N steps.