acceptodds
Under review as a conference paper at ICLR 2027

Does Uniform Discrete Diffusion Need Time?

Abstract

Uniform discrete diffusion models (UDMs) commonly use explicit time conditioning, but we find that it can often be unnecessary in practice. In this paper, we first show that the population-optimal UDM predictor generally depends on time: time controls how much the model should trust the observed context. We then show that this dependence can become negligible in finite-data settings relevant to language. When a corrupted training sequence remains much closer to its original clean sequence than to competing training sequences, the empirical-optimal predictor is nearly insensitive to time over most of the diffusion trajectory, where the guarantee weakens toward the high-noise endpoint. Empirically, trained language UDMs exhibit limited time sensitivity over most of the trajectory, while time-agnostic predictors remain competitive with, and often outperform, time-conditioned models across datasets and training objectives. These results challenge the use of explicit time conditioning in UDMs: although the population optimum depends on time, explicitly conditioning on it may often be unnecessary in practice.

Primary area:
generative models
Keywords:
Discrete diffusion models, uniform diffusion models, theory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_B7h3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper provides a theoretical analysis of the time sensitivity of uniform discrete diffusion models through the lens of match-count decomposition, and points out that the uniformly corrupted data can be easily separable in a finite-data setting. Experiments on various UDM frameworks have demonstrated the performance of time-agnostic UDMs on validation PPL and generative quality.

Critical strengths
  1. The analysis framework (match-count decomposition, separability, Fisher time sensitivity, etc.) of UDM time sensitivity is rigorous and innovative.

  2. The experimental results on validation PPL and generation quality of time-agnostic UDM are comprehensive.

Critical weaknesses
  1. This paper only covers the analysis of full-sequence UDM. But in practice, the mainstream of DLM is block diffusion. As in Section 4 of this paper, the Delta-separability is defined with the minimum Hamming distance d(x), which should be positively correlated to sequence length, so what will be the case of block diffusion, whose block size is typically much smaller (e.g. 32) than the full-sequence setting in this paper (OWT with 1024 context length)? Does the conclusion on time sensitivity still hold for block diffusion, or we have to use a long sequence length to obtain the insensitivity of time?

  2. This paper only removes the time in parameterization. However, in sampling, it still needs time to implement the reverse kernel and to determine when to stop the generation (at t=0), so it is not a completely time-independent framework. As in the standard practice of MDLM, confidence-based sampling algorithms are the most performative, which completely eliminates the usage of time, so is it possible for UDM sampling to be completely time-independent, with the ability to know when to stop the UDM generation?

  3. The approach and the baselines achieve similar performance, which raises questions about the necessity of designing time-agnostic UDMs

  4. This paper does not provide large-scale experiments, e.g. accuracy on generative tasks like GSM8K and HumanEval, making it unclear whether the time-agnostic UDMs are still performative in standard language benchmarks. Together with weakness 2, the generative evaluation of downstream tasks of DLMs is often not based on the reverse kernel sampler, as it is so random, but on more deterministic samplers such as the confidence-based ones without time.

Questions for authors

Please see the weaknesses section.

3 comments
AuthorsOpenReview ↗

Author Rebuttal (1/2)

We thank the reviewer for the valuable feedback! We have addressed all of the reviewer's questions and highlighted the changes in the revised paper in blue.

W1. Does the theory also hold for block diffusion?

We thank the reviewer for raising this important question! While directly applying the Δ\Delta-separability analysis in Section 4 to shorter blocks may weaken the theoretical guarantee, block diffusion provides an additional source of separation through its clean prefix conditioning.

For block diffusion with block size BB, let ZZ denote the corrupted sequence of the current block, spanning positions jB+1jB+1 to (j+1)B(j+1)B. For any position ii within this block, the empirical-optimal predictor is \begin{equation} x_\star^i(z,t \mid x^{1:jB})=\mathbb{E}_{\hat{p}_N}[X^i | X^{1:jB}=x^{1:jB}, Z^{-i}=z^{-i}, T=t]. \end{equation}

Since the clean prefix must match exactly, only training sequences sharing x1:jBx^{1:jB} contribute to this expectation. Thus, this is equivalent to applying Theorem 2 to the subset of training data with the same prefix, where L=BL=B, KK is the vocabulary size, and NN is the number of prefix-matched samples. Empirically, we found that distinct training samples almost never share identical prefixes once the prefix length exceeds 16 tokens in OWT dataset. Consequently, N=1N=1 in nearly all such cases, making the empirical-optimal predictor exactly time-independent, regardless of the block size.

Nevertheless, as discussed in Section 5.1, the behavior of the empirical optimum does not necessarily extend to learned models. We therefore additionally train block uniform diffusion for 1M steps and evaluate on LM1B with a block size of 16:

Time-variant (139M)Time-agnostic (131M)Time-agnostic (139M)
Val PPL32.1032.1831.86

The results show that time-agnostic models achieve comparable validation perplexity even with short diffusion blocks, suggesting that explicit time conditioning remains largely unnecessary in block diffusion.

We thank the reviewer again for this insightful question. We have added the corresponding discussion to Appendix B.7.

W2. The UDM reverse kernel itself is not time-agnostic. Can UDMs, like MDMs, (1) completely eliminate the use of time, (2) enable confidence-based like sampling algorithms, and (3) determine when to stop generation?

We thank the reviewer for this important suggestion. We would like to clarify that our primary goal is to understand whether explicit time conditioning is necessary for effective UDM learning, rather than to eliminate time dependence from the entire generative process.

Nevertheless, motivated by the reviewer's suggestion, we further investigate whether time-agnostic UDMs can support generation without an explicit diffusion-time schedule. To this end, we propose a simple confidence-guided sampler that prioritizes token updates based on model predictions and supports adaptive early stopping, without requiring diffusion time. Our experiments further show that the proposed sampler outperforms standard ancestral reverse-kernel sampling, as detailed in our response to W4.

Proposed Confidence Gain Sampler. Given a generation length LL, a sampling budget of NN steps, and an optional early-stopping threshold ϵ\epsilon, we update k=⌊L/N⌋k=\lfloor L/N \rfloor tokens per step. We denote the model prediction xθ(z)x_\theta(z) by pθ(⋅∣z)p_\theta(\cdot \mid z). Starting from a uniformly sampled sequence zz, we repeat:

  1. Sample candidates. Sample x^i∼pθi(⋅∣z)\hat{x}^i \sim p_\theta^i(\cdot \mid z) for each position ii.

  2. Compute confidence margins. Compute

    Δi=pθi(x^i∣z)−pθi(zi∣z),\Delta_i = p_\theta^i(\hat{x}^i \mid z) - p_\theta^i(z^i \mid z),

    and select the top-kk positions with the largest margins, denoted by I\mathcal{I}.

  3. Early stopping (optional). Terminate if max⁡iΔi<ϵ\max_i \Delta_i < \epsilon.

  4. Update. Set zi←x^iz^i \leftarrow \hat{x}^i for all i∈Ii \in \mathcal{I}.

Repeat for at most NN steps.

AuthorsOpenReview ↗

Author Rebuttal (2/2)

Intuition behind sampler. Our sampler is motivated by two properties of the UDM reverse kernel (Eq. 3):

  1. Token proposal: The distribution over replacement tokens contains a mixture of the uniform distribution and the model prediction pθ(⋅∣z)p_\theta(\cdot \mid z), motivating our choice to sample candidates from pθp_\theta.

  2. Token replacement: For a transition from the current token a=zia=z^i to another token b≠ab\neq a, the reverse kernel satisfies

    = C_{s,t}\frac{1+\rho_s p_\theta^i(b\mid z_t)}{1+\rho_t p_\theta^i(a\mid z_t)},$$ where $C_{s,t}$ is independent of the token position and prediction. Thus, replacements are favored when the proposed token has high probability and the current token has low probability.

Inspired by these properties, we use pθp_\theta for candidate sampling and the confidence difference pθi(b∣z)−pθi(a∣z)p_\theta^i(b\mid z)-p_\theta^i(a\mid z) for replacement prioritization. Although the original reverse kernel remains time-dependent and stochastic, our heuristic captures these preferences without explicitly using diffusion time.

W4. Provide large-scale experiments and sampler that satisfies the specified conditions in W2.

To answer this question with above W2, we (1) evaluate additional large-scale results on GSM8K and (2) evaluate a new sampling strategy that satisfies the properties highlighted by the reviewer.

Experimental Setup. We pretrain both 1.1B-parameter time-variant and time-agnostic UDMs on 445B tokens from the Nemotron subset of Nemotron-CC-v2, followed by 40 epochs of supervised fine-tuning on Augmented GSM8K. We then evaluate both models on the GSM8K test set with a generation length of 256 tokens.

Results. We compare ancestral sampling with our proposed Confidence Gain Sampler across different sampling budgets. Values represent GSM8K accuracy (%), with the average number of sampling steps shown in parentheses when early stopping is enabled.

Sampling Method256 Steps128 Steps64 Steps32 Steps
Time-variant UDM (Ancestral)38.9738.3637.9135.63
Time-agnostic UDM (Ancestral)38.4438.0638.2136.32
Confidence Gain (w/o ES)51.3351.1051.3348.14
Confidence Gain (ES, ϵ=10−4\epsilon=10^{-4})51.55 (178.42)51.48 (90.62)51.48 (46.81)48.07 (24.86)

ES denotes early stopping. Numbers in parentheses indicate the average number of sampling steps actually used.

  • Even on downstream generation benchmarks with large-scale models, time-agnostic models achieve performance comparable to that of time-variant models.
  • Further performance improvements can be achieved even without relying on ancestral sampling, which requires time information.

We believe this result highlights promising opportunities for further research on time-agnostic UDMs, while developing a time-homogeneous sampling process that provably preserves the target generative distribution remains an important direction for future work.

For further details on the experimental setup and sampling strategy, please refer to Appendices C.4 and C.3 of the revised paper, respectively.

W3. If the performance is similar, why should we use time-agnostic models?

We believe that similar performance is itself a strong motivation for using time-agnostic models. If explicit time conditioning provides little benefit, removing it simplifies the model and offers greater flexibility for future developments. We highlight two examples where this simplicity can be particularly useful:

  • The improved sampler presented above is a simple example enabled by time-agnostic modeling. With a time-variant model, designing such a sampler would require careful consideration of the reverse kernel and marginal probability evolution, along with how the time input should be assigned throughout the sampling process. In contrast, a time-agnostic model allows greater freedom in designing sampling strategies without these constraints.
  • A growing trend in diffusion language modeling is to initialize models from pretrained autoregressive LMs. Since autoregressive models do not take time as input, introducing time conditioning requires additional architectural choices. If time conditioning offers little practical benefit, time-agnostic modeling allows us to avoid considering about this ambiguous design choice entirely, making it more straightforward to leverage pretrained autoregressive models.

We believe that such simplicity can facilitate further advances in UDMs, beyond the examples discussed here. We hope these considerations help clarify the practical value of removing unnecessary time conditioning, even when the immediate performance gains are limited.

We sincerely thank the reviewer again for detailed and thoughtful review. We would be happy to address any further questions or concerns.

AuthorsOpenReview ↗

A Kind Reminder for Comments

We sincerely thank the reviewer for the thoughtful feedback. We highlight three key additions addressing the reviewer's concerns:

  • Block diffusion: Added theoretical discussion and experiments with block size 16, showing competitive time-agnostic performance.
  • Time-independent sampling: Proposed a Confidence Gain Sampler supporting adaptive early stopping without diffusion time.
  • Large-scale evaluation: Added 1.1B-scale GSM8K experiments, where our proposed sampler improves accuracy from approximately 38% to over 51%.

We appreciate the reviewer’s time and would be happy to address any remaining questions.

Reviewer_q6KoOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies whether explicit time conditioning is necessary in uniform discrete diffusion models (UDMs). The theoretical contributions are:

  • Proposition 1: The population-optimal predictor is a mixture over possible subsets of copied context positions, where time controls the relative weights assigned to these subsets.
  • Theorem 1: The entire time dependence of the optimal predictor can be characterized through the number of matching context tokens, with time reweighting otherwise time-independent component predictions.
  • Proposition 2: The derivative of the optimal prediction with respect to the noise parameter is exactly the covariance between the match count and the corresponding clean-token prediction.
  • Proposition 3: When training sequences are sufficiently separated in Hamming distance, their post-corruption match-count separation is preserved with high probability.
  • Theorem 2: A sufficiently large post-corruption separation margin guarantees small Fisher time sensitivity of the empirical-optimal predictor over most of the diffusion trajectory.

The experiments verify the separation assumptions on language data, measure time sensitivity in trained UDMs, and compare time-conditioned and time-agnostic models across datasets and objectives. The results generally support the theory, with time conditioning being most useful near the high-noise endpoint.

Critical strengths

S1. Well written. The paper is mostly clear, easy to follow, and the theoretical results are presented in a natural progression.

S2. Beautiful theory. I particularly liked Theorem 1 and Proposition 2, which give an elegant characterization of time dependence. The connection to finite-data separation in Theorem 2 is also insightful.

S3. Good experimental support. The experiments are well aligned with the theory and support most of its predictions. I especially liked the separation analysis and the hybrid time-conditioning experiment. Figure 2 (right) is quite insightful as well.

Critical weaknesses

W1. Connection to discrete flow matching. The uniform diffusion probability path is a convex interpolant with a uniform source, as studied by Gat et al. The paper should discuss this connection and clarify which theoretical results are specific to UDMs. In particular, could similar results hold for the full denoising posterior used in discrete flow matching? Or does the potential change of the generator break the theory?

W2. Experimental limitations.

  • The theory concerns the empirical oracle, but the experiments study learned networks. Could the authors provide a small-scale synthetic experiment where the oracle posterior and generative process can be computed exactly, and report the measured time sensitivity?
  • The experiments are limited somewhat in model scale, some comparisons change the architecture, and uncertainty estimates are missing. Could the authors test the agnostic model, by feeding a constant t=0t=0 throughout the process?
  • In Figure 2 (right), the authors refer to the plotted quantity as per-time NLL. Is this simply the NELBO integrand evaluated at a fixed time tt? If not, how is the NLL at a particular diffusion time defined and computed?

W3. Limited practical novelty. Time-agnostic discrete diffusion models are already widely used, including in recent UDMs. Removing an unnecessary conditioning variable can also improve optimization and generalization, so the empirical findings are not particularly surprising. The main contribution is therefore the theoretical explanation of an already established practice, rather than a new practical method.

Questions for authors
  • Q1. In Figure 2 (right), the authors refer to the plotted quantity as per-time NLL. Is this simply the NELBO integrand evaluated at a fixed time tt? If not, how is the NLL at a particular diffusion time defined and computed?
  • Q2. Could the authors provide a small-scale synthetic experiment where the oracle posterior and generative process can be computed exactly, and report the measured time sensitivity/dependency?
  • Q3. Could the authors test the agnostic model, by feeding a constant t=0t=0 throughout the process? Instead of changing the architecture, by removing the time component.
3 comments
AuthorsOpenReview ↗

Author Rebuttal (1/2)

We thank the reviewer for the valuable feedback! We have addressed all of the reviewer's questions and highlighted the changes in the revised paper in blue.

W1. Connection to discrete flow matching

Thank you for this insightful question. We have verified that our analysis extends to the full denoising posterior used in discrete flow matching, with only a one-token reduction in the effective separation margin.

Consider the empirical data distribution p^N:=1N∑n=1Nδxn\hat p_N := \frac{1}{N}\sum_{n=1}^N \delta_{\mathbf x_n} and its empirical oracle:

\begin{equation} \mu_{\theta^\star}^{\mathrm{Posterior},i}(\mathbf z,t)=\mathbb E[X^i \mid Z=\mathbf z,T=t]. \end{equation}

We define ItPosterior,i(z)\mathcal I_t^{\mathrm{Posterior},i}(\mathbf z) as the Fisher sensitivity of this oracle at position ii. We show that, for Δ≥3\Delta\ge3,

\begin{equation} \sup_{0<t\le1-K^{-1/4}} \mathcal I_t^{\mathrm{Posterior},i}(\mathbf z) \le NL^2K^{-(3\Delta-5)/4}, \end{equation} where Δ\Delta follows our original definition. Comparing this with Eq. (9) of Theorem 2, \begin{equation} \sup_{0<t\le1-K^{-1/4}} \mathcal I_t^i(\mathbf z) \le NL^2K^{-(3\Delta-2)/4}, \end{equation} we obtain a slightly looser bound, equivalent to reducing the separation margin Δ\Delta by exactly one. In our original analysis, we showed that Δ≥6\Delta\ge6 is sufficient to guarantee a Fisher sensitivity below 10−510^{-5} for the LOO posterior on OWT. For the full posterior, the same guarantee only requires Δ≥7\Delta\ge7. As shown in Figure 1 (Right), this slightly stronger separation condition is also satisfied for the vast majority of samples.

Connection to discrete flow matching and the sampling generator. As the reviewer noted, discrete flow matching with an independent uniform source and a convex interpolation path induces the same tokenwise conditional probability path as uniform diffusion, up to time reversal:

=\prod_{\ell=1}^{L} \left[ (1-t)\mathbf 1{z^\ell=x^\ell} +\frac{t}{K} \right].$$ **The choice of sampling generator does not affect the theoretical time sensitivity of the optimal denoising posterior.** Given this shared path and data distribution, the full denoising posterior serving as the population-optimal denoiser in discrete flow matching is uniquely determined. Its time sensitivity depends on the data distribution and this conditional probability path, not on the choice of generator used to realize the path. Therefore, changing the sampling generator while preserving the conditional probability path does not affect our result. We have incorporated this result into the revised paper. Please refer to Appendices B.7 and B.8 for detailed proofs and discussion. > **W2-1/Q2. Provide synthetic experiment validating Theorem** We thank the reviewer for this valuable suggestion. Following the reviewer's request, we conducted controlled synthetic experiments with an exactly computable empirical oracle. The results show that **(1) greater data separation leads to stronger oracle posterior concentration and lower oracle time sensitivity away from the high-noise endpoint, and (2) learned models exhibit consistently lower time sensitivity on both training and held-out data under the same change in data construction.** These findings provide empirical evidence connecting finite-data separation to learned-model time sensitivity. To keep our responses concise and avoid repetition, we kindly refer the reviewer to our response to Reviewer 9wkq (W1/Q1) for the detailed experimental setup and results, also included in Appendix C.2 of the revised paper.
AuthorsOpenReview ↗

Author Rebuttal (2/2)

W2-2/Q3. Experiments are limited, e.g., different architecture / model scale / uncertainty estimates. Please provide experimental results of time-agnostic UDM with feeding a constant t=0t=0.

Following the reviewer's suggestion, we find that time-agnostic UDM achieves comparable performance to time-variant UDM even with identical model architectures. Specifically, we retain the time-variant architecture but fix the time input to t=0t=0 for the time-agnostic model as suggested. Both models are trained for 1M steps using LOO+CE on LM1B, with and without sentence packing.

Time-variant UDMTime-agnostic UDM (t=0)
Val PPL on LM1B (Packed)33.9434.13
Val PPL on LM1B30.1529.32

These results confirm that removing time dependence does not compromise performance, even when controlling for architecture.

To further examine the effectiveness of time-agnostic UDMs at a larger scale and on downstream reasoning tasks, we pretrain both 1.1B-parameter time-variant and time-agnostic UDMs on 445B tokens from the Nemotron subset of Nemotron-CC-v2, followed by 40 epochs of supervised fine-tuning on Augmented GSM8K. We then evaluate both models on the GSM8K test set with a generation length of 256 tokens. The results are as follows:

Sampling Method256 Steps128 Steps64 Steps32 Steps
Time-variant UDM38.9738.3637.9135.63
Time-agnostic UDM38.4438.0638.2136.32

The two models achieve closely matched GSM8K accuracy across all sampling budgets, with absolute differences of at most 0.69 percentage points and no consistent advantage for either model. Together with our architecture-matched LM1B results, these experiments reinforce our central conclusion: explicit neural time conditioning is not necessary to achieve competitive UDM performance, even at the 1.1B-parameter scale.

Due to the limited rebuttal period, we were unable to conduct multiple training runs with different random seeds to report uncertainty estimates. Nevertheless, we hope the reviewer will find the consistently comparable performance across diverse experimental settings reassuring.

W3. Limited practical novelty.

We thank the reviewer for raising this point. While time-agnostic UDMs have been adopted in recent works, their empirical success does not establish when explicit time conditioning is useful or whether it can be reliably omitted across different training objectives, architectures, and model scales. We believe that establishing the viability of time-agnostic UDMs through systematic, controlled comparisons is itself a valuable practical contribution, even if it does not constitute a new practical method.

In particular, our analysis and experiments identify where time conditioning matters: its benefit is primarily concentrated near the high-noise endpoint, as further supported by our hybrid conditioning experiments. Moreover, our additional rebuttal experiments show that time-agnostic UDMs remain competitive (1) under identical architectures with constant time input, (2) at matched Transformer depth on OWT, and (3) at the 1.1B-parameter scale on GSM8K, with comparable performance across sampling budgets.

Thus, beyond demonstrating that time-agnostic UDMs can work, our contribution is to systematically establish when and how much explicit time conditioning matters, providing an evidence-based foundation for practical UDM design. Furthermore, during this rebuttal, we propose a heuristic time-independent sampler that substantially improves GSM8K accuracy over ancestral sampling, demonstrating the practical flexibility of time-agnostic UDMs. Please refer to our responses to Reviewer B7h3 (W2/W4) or to Appendix C.3 for details.

Q1. What is per-time NLL?

As the reviewer correctly noted, per-time NLL refers to the NELBO integrand evaluated at a fixed time tt. This definition was also included at the end of Appendix C.2 in the original submission.

We sincerely thank the reviewer again for detailed and thoughtful review. We would be happy to address any further questions or concerns.

AuthorsOpenReview ↗

A Kind Reminder for Comments

We sincerely thank the reviewer for the thoughtful feedback. We highlight three key additions addressing the reviewer's concerns:

  • Theoretical extension: Extended our Fisher sensitivity bound to the full denoising posterior used in discrete flow matching.
  • Synthetic validation: Added controlled experiments connecting empirical oracle posterior concentration to learned-model time sensitivity.
  • Additional evaluations: Added architecture-controlled comparisons and 1.1B-scale GSM8K experiments demonstrating competitive time-agnostic performance.

We appreciate the reviewer’s time and would be happy to address any remaining questions.

Reviewer_9wkqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper characterizes time dependence in the population-optimal predictor for uniform discrete diffusion, derives a separation-based sensitivity bound for a finite-data oracle, and evaluates time-conditioned, time-agnostic, and hybrid language UDMs.

Critical strengths

The match-count decomposition provides a useful characterization of oracle time dependence. The OWT separation measurements support the premises of the finite-data analysis, and the LM1B experiments include same-depth comparisons showing that time-agnostic models can remain competitive.

Critical weaknesses

The paper’s central explanation is not substantiated by its experiments. Theorem 2 concerns an empirical oracle whose time insensitivity follows from separation-induced posterior concentration. Figure 1 supports the separation conditions for corrupted training examples, whereas Figure 2 measures learned networks on held-out sequences without establishing that their sensitivity is governed by this mechanism. The authors acknowledge the distinction, but acknowledging it does not resolve the explanatory gap. The missing evidence concerns the paper’s main contribution, not merely an extension of the theorem to a broader setting.

The claimed theory–experiment agreement does not provide this evidence. Theorem 2 gives an upper bound that weakens toward high noise; it does not predict that actual sensitivity must increase there. Observing such an increase is therefore insufficient to validate the proposed mechanism. Likewise, competitive time-agnostic and hybrid models demonstrate the viability of reduced conditioning, not why it works. This distinction is particularly important because Appendix A.1 explicitly contrasts the proposed mechanism with implicit inference of the noise level, yet the experiments do not establish which explanation governs the learned models.

As a secondary issue, the OWT comparison changes depth from 12 to 13 layers to match parameter counts. Unlike the same-depth LM1B comparisons, it evaluates parameter reallocation rather than isolating the effect of removing time conditioning.

Overall, the paper establishes a conditional oracle property and a practical observation, but not the explanatory connection between them. I consider this a substantive gap requiring new evidence, rather than clarification alone.

Questions for authors

Clarification: What empirical evidence demonstrates that learned-model time sensitivity is governed by separation-induced posterior concentration, beyond qualitative compatibility with the oracle bound?

Suggested additional experiment: In a controlled setting with an exactly computable oracle, vary data separation while holding architecture and training budget fixed, and jointly measure oracle posterior concentration and learned-model sensitivity on training and held-out examples.

4 comments
AuthorsOpenReview ↗

Author Rebuttal (1/3)

We thank the reviewer for the valuable feedback! We have addressed all of the reviewer's questions and highlighted the changes in the revised paper in blue.

W1/Q1. Provide synthetic experiment validating Theorem.

We thank the reviewer for this valuable suggestion. To bridge our theoretical analysis and empirical observations, we conducted controlled synthetic experiments with an exactly computable empirical oracle. Our results show that (1) the more-separated empirical dataset exhibits stronger oracle posterior concentration and lower oracle time sensitivity over much of the diffusion trajectory, and (2) learned models exhibit a corresponding reduction in time sensitivity under the same change in training-data construction. These results provide new evidence linking empirical data geometry and learned-model behavior, beyond the qualitative agreement reported in our original submission.

Design of synthetic data experiments.

  • Data distribution. We consider sequences of length L=M+PL=M+P over a vocabulary of size KK, consisting of MM latent tokens and PP observation tokens. The latent tokens are sampled independently, and each observation token randomly copies one of two associated latent tokens:

    \qquad O_j \sim \mathrm{Uniform}(\lbrace H_{a_j}, H_{b_j}\rbrace)$$ Here, $a_j$ and $b_j$ are two distinct latent indices randomly selected once and fixed across all samples. The latent and observation positions are also randomly assigned and fixed. Intuitively, the data distribution consists of simple dependencies where each observation token copies one of two fixed latent tokens.
  • Varying data separation. We compare two empirical distributions of the same size NN: the standard empirical distribution p^N\hat p_N, consisting of NN independently sampled sequences from pdatap_{\mathrm{data}}, and an augmented empirical distribution p^NAug\hat p_N^{\mathrm{Aug}}. To construct the latter, we sample λN\lambda N independent sequences (anchors) and generate the remaining (1−λ)N(1-\lambda)N samples through augmentation. Specifically, each augmented sample is generated from a randomly selected anchor by preserving all latent tokens and independently resampling RR randomly selected observation positions from their original conditional distributions, while leaving the remaining positions unchanged. This introduces closely related training samples, allowing us to modify empirical data separation while keeping the total dataset size fixed.

  • Design rationale. This construction has two key properties. (1) Population preservation: Every original and augmented sample follows the same population distribution, yielding E[p^N]=E[p^NAug]=pdata.\mathbb{E}[\hat p_N]=\mathbb{E}[\hat p_N^{\mathrm{Aug}}]=p_{\mathrm{data}}. Thus, we can examine the effect of empirical data geometry while keeping the underlying population distribution and dataset size fixed. (2) Simplicity: The model only needs to learn the fixed dependencies between latent and observation tokens and how diffusion time affects their predictions, without requiring complex arithmetic or reasoning.

Experimental results. We set K=16K=16, L=128L=128, M=32M=32, P=96P=96, R=32R=32, and N=104N=10^4, with λ=0.1\lambda=0.1 for p^NAug\hat p_N^{\mathrm{Aug}}. For the learned models, we train a 3-layer Transformer with the same architecture as in our original paper, with the same training budget. We verify that our construction induces the intended difference in post-corruption separation margins Δ\Delta (Eq. 7). Across all evaluated corruption times, the standard empirical dataset exhibits substantially larger separation than its augmented counterpart. For example, at tsample=0.5t_{\mathrm{sample}}=0.5, the mean margin E[Δ]\mathbb{E}[\Delta] is 44.40 versus 25.01, while the mean minimum margin across positions E[Δmin⁡]\mathbb{E}[\Delta_{\min}] is 43.64 versus 5.90. Full results are provided in Appendix C.2.

Posterior concentration of the empirical oracle. We first examine how the empirical oracle's posterior concentration evolves over diffusion time, measured by the maximum predicted token probability. The table reports results for different corruption times tsamplet_{\mathrm{sample}}, with columns corresponding to evaluation-time intervals tevalt_{\mathrm{eval}}.

tsamplet_{\mathrm{sample}}Distribution0–0.600.60–0.650.65–0.700.70–0.750.75–0.800.80–0.850.85–0.900.90–0.950.95–0.99
0.3p^N\hat p_N1.00001.00001.00001.00001.00001.00001.00001.00001.0000
p^NAug\hat p_N^{\mathrm{Aug}}1.00001.00001.00001.00001.00001.00001.00000.99910.9749
0.7p^N\hat p_N1.00001.00001.00001.00001.00001.00001.00001.00000.9078
p^NAug\hat p_N^{\mathrm{Aug}}0.99990.99960.99930.99880.99760.99500.98850.96930.8915
AuthorsOpenReview ↗

Author Rebuttal (2/3)

For tsample=0.3t_{\mathrm{sample}}=0.3, both empirical oracles exhibit nearly deterministic predictions over most of the trajectory. At tsample=0.7t_{\mathrm{sample}}=0.7, the difference becomes more pronounced: the standard empirical oracle p^N\hat p_N, with greater data separation, exhibits faster posterior concentration as tevalt_{\mathrm{eval}} decreases from 1, whereas p^NAug\hat p_N^{\mathrm{Aug}} retains greater uncertainty. These observations provide direct evidence that empirical data separation promotes posterior concentration, consistent with the mechanism underlying Theorem 2.

Time sensitivity of the empirical oracle and learned models. We next investigate how this difference in posterior concentration relates to the time sensitivity of learned models. We report Fisher sensitivity for the empirical oracle on training inputs and for learned models on both training and held-out inputs.

Time sensitivity (tsample=0.3t_{\mathrm{sample}}=0.3).

ModelEval. SplitTraining distribution0–0.600.60–0.650.65–0.700.70–0.750.75–0.800.80–0.850.85–0.900.90–0.950.95–0.99
Emp.Trainp^N\hat p_N0000000041.623
p^NAug\hat p_N^{\mathrm{Aug}}000000.0070.1184.717162.460
LearnedTrainp^N\hat p_N1.8353.1463.7144.4465.4246.7198.1969.85411.342
p^NAug\hat p_N^{\mathrm{Aug}}1.9083.9525.0486.90410.05314.68419.75525.69228.628
Held-outp^N\hat p_N1.8023.1573.8044.6605.8397.4489.31811.32312.839
p^NAug\hat p_N^{\mathrm{Aug}}1.9383.9775.0766.8879.86314.10118.60724.05327.504

Time sensitivity (tsample=0.5t_{\mathrm{sample}}=0.5).

ModelEval. SplitTraining distribution0–0.600.60–0.650.65–0.700.70–0.750.75–0.800.80–0.850.85–0.900.90–0.950.95–0.99
Emp.Trainp^N\hat p_N000000001934.629
p^NAug\hat p_N^{\mathrm{Aug}}0.0040.0310.0570.1120.2590.7122.39511.823578.640
LearnedTrainp^N\hat p_N1.9672.8313.3203.9774.8926.1477.6289.32910.812
p^NAug\hat p_N^{\mathrm{Aug}}2.1483.5644.4725.9998.55312.26316.32321.21923.901
Held-outp^N\hat p_N1.7932.8743.4734.2915.4587.1059.08311.25612.905
p^NAug\hat p_N^{\mathrm{Aug}}2.0993.7744.7406.3969.24113.51118.37724.21126.769

Time sensitivity (tsample=0.7t_{\mathrm{sample}}=0.7).

ModelEval. SplitTraining distribution0–0.600.60–0.650.65–0.700.70–0.750.75–0.800.80–0.850.85–0.900.90–0.950.95–0.99
Emp.Trainp^N\hat p_N00000000.0049541.448
p^NAug\hat p_N^{\mathrm{Aug}}0.0060.0450.0880.1810.4401.2654.21215.2975854.230
LearnedTrainp^N\hat p_N1.9302.9453.4644.1455.0706.3077.7369.35310.811
p^NAug\hat p_N^{\mathrm{Aug}}2.0553.5934.4355.8448.19611.61815.40320.04622.903
Held-outp^N\hat p_N1.9112.8713.4014.1055.0746.3917.9329.69011.222
p^NAug\hat p_N^{\mathrm{Aug}}2.1773.5194.2955.5727.64410.57113.70417.49919.546

The empirical oracle exhibits nearly zero time sensitivity over most of the trajectory, with sensitivity concentrated near t=1t=1. Compared with p^N\hat p_N, the augmented oracle p^NAug\hat p_N^{\mathrm{Aug}} becomes time-sensitive at earlier evaluation times, consistent with its weaker posterior concentration. Near t=1t=1, however, the ordering can reverse: the more-separated oracle may undergo a sharper transition from its concentrated posterior to the unconditional distribution, resulting in higher Fisher sensitivity. This does not contradict Theorem 2, which bounds sensitivity without imposing a pointwise ordering.

Importantly, learned models exhibit the same qualitative trend as the empirical oracle across the two dataset constructions, despite differences in sensitivity magnitude. For instance, at tsample=0.5t_{\mathrm{sample}}=0.5 and teval∈[0.85,0.90]t_{\mathrm{eval}}\in[0.85,0.90], learned Fisher sensitivity decreases from 16.323 to 7.628 on training data and from 18.377 to 9.083 on held-out data when moving to the more-separated dataset. This consistent reduction provides evidence linking empirical data separation to learned-model time sensitivity.

AuthorsOpenReview ↗

Author Rebuttal (3/3)

Overall, these controlled experiments support our proposed mechanism beyond qualitative agreement with the theoretical upper bound. We provide full explanation in Appendix C.2 of the revised paper, including the full plots of the above experiments, the statistics of Δ\Delta, and training details. We have also added a brief discussion of the synthetic data experiments in Section 5.1.

W2. Missing OWT results for the 12-layer time-agnostic UDM.

To isolate the effect of time conditioning from the additional Transformer layer used in our original parameter-matched OWT comparison, we additionally trained a 12-layer time-agnostic UDM on OWT using LOO+CE, following the same experimental setup as the original submission. The results are as follows:

Time-variant UDM (12 Layer)Time-agnostic UDM (12 Layer)Time-agnostic UDM (13 Layer)
Val PPL25.2925.9225.29

At matched depth, the time-agnostic model achieves competitive validation PPL (25.92 versus 25.29), while the original 13-layer time-agnostic model matches the time-conditioned baseline at 25.29.

We further compare the generative perplexity across different numbers of sampling steps. Values in parentheses indicate the corresponding entropy.

Model / Steps32641282565121024
Time-variant UDM (12 Layer)114.04 (5.60)95.11 (5.56)89.79 (5.56)88.22 (5.56)84.39 (5.55)83.61 (5.54)
Time-agnostic UDM (12 Layer)98.98 (5.58)84.67 (5.56)80.78 (5.55)80.34 (5.56)76.51 (5.54)77.37 (5.54)
Time-agnostic UDM (13 Layer)93.92 (5.56)82.22 (5.54)78.13 (5.52)77.64 (5.54)73.87 (5.53)71.23 (5.52)

At matched model depth, the 12-layer time-agnostic UDM remains competitive with its time-conditioned counterpart, achieving comparable validation PPL (25.92 vs. 25.29) and lower generative PPL across all sampling budgets, with similar sample entropy. These results reinforce our main finding that time-agnostic UDMs can achieve competitive performance without explicit time conditioning, even when controlling for model depth.

We sincerely thank the reviewer again for detailed and thoughtful review. We would be happy to address any further questions or concerns.

AuthorsOpenReview ↗

A Kind Reminder for Comments

We sincerely thank the reviewer for the thoughtful feedback. We highlight two key additions addressing the reviewer's concerns:

  • Theory–experiment connection: Added controlled synthetic experiments demonstrating that greater data separation reduces time sensitivity in both the empirical oracle and learned models, including on held-out data.
  • Architecture-controlled comparison: Added matched-depth OWT experiments showing competitive validation PPL and improved generative PPL for time-agnostic UDMs.

We would appreciate the reviewer's consideration of these new results and welcome any further questions.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We sincerely thank the reviewers for their constructive feedback. In this work, we characterize time dependence in uniform discrete diffusion models (UDMs) and explains when explicit time conditioning can become unnecessary in finite-data settings.

Reviewers highlighted several strengths, including the elegant characterization of time dependence (q6Ko, 9wkq), the rigorous and innovative theoretical analysis (q6Ko, B7h3), the insightful separation analysis (q6Ko, 9wkq), and the comprehensive experiments demonstrating the competitiveness of time-agnostic UDMs (q6Ko, B7h3, 9wkq).

We appreciate that the suggestions provided by reviewers have meaningfully strengthened the paper, and we summarize the key improvements below:

  • Synthetic experiments connecting theory and learned models: Reviewers 9wkq and q6Ko requested controlled experiments bridging our theoretical analysis and learned-model behavior. We therefore conducted synthetic experiments (Appendix C.2) varying empirical data separation while preserving the underlying population distribution and dataset size. Using an exactly computable empirical oracle, we show that greater separation is associated with stronger posterior concentration and lower oracle time sensitivity away from the high-noise endpoint. Importantly, learned models also exhibit consistently lower time sensitivity on both training and held-out data.

  • Large-scale experiments and time-independent sampling: Reviewers q6Ko and B7h3 raised concerns about model scalability and downstream generation performance. We therefore trained 1.1B-parameter UDMs and demonstrated comparable GSM8K accuracy between time-variant and time-agnostic models (Section 5.3). Furthermore, motivated by reviewer B7h3, we propose a heuristic time-independent sampling strategy that substantially improves GSM8K accuracy (Appendix C.3).

All reviewer comments were addressed, and the requested experiments have been incorporated. Modified content is marked in blue text in the revised manuscript. We thank the reviewers again; feedback provided helped refine the paper and expand its scope in meaningful ways.

AuthorsOpenReview ↗

A Kind Reminder for Comments

Dear Area Chair and Reviewers,

We sincerely thank the reviewers for their constructive feedback and careful evaluation of our work. We have provided detailed point-by-point responses and updated the manuscript with additional experiments and clarifications addressing the reviewers’ concerns. The key changes are summarized in our individual reviewer responses, and all revisions in the manuscript are highlighted in blue.

We would greatly appreciate the Area Chair’s help in inviting reviewers to engage in further discussion and consider our responses. We welcome any remaining questions and would be happy to provide further clarification.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.