acceptodds
Under review as a conference paper at ICLR 2027

An ODE-based Sampler for Discrete Diffusion Models

Abstract

The marginals of a continuous diffusion model can be reproduced by a deterministic ordinary differential equation (ODE), known as the probability flow ODE, which provides a foundation for accelerated sampling and consistency distillation. In this paper we address the question whether an analogous formulation exists for discrete diffusion models and answer it affirmatively by introducing exponential residual flows (ERFs). Our framework augments the discrete state with a continuous residual variable for each token and position. These residuals evolve deterministically according to piecewise-defined ODEs whose vector fields depend on the current discrete state and the forward and backward rate matrices of the prescribed diffusion model. The discrete state changes whenever a residual reaches zero. We prove that this construction preserves the marginals of the original diffusion. We derive a practical discretization of our sampler and specialize it to masked and uniform diffusion. The deterministic trajectories of ERFs support consistency distillation and yield empirical improvements over previous approaches on across a variety of benchmarks including code and text generation tasks.

Primary area:
generative models
Keywords:
discrete diffusion models, continuous-time Markov chain, distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_RihfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces exponential residual flows (ERF), which augment discrete states with continuous residual variables to construct deterministic trajectories with the prescribed time marginals. It then uses these trajectories for consistency distillation into a residual-conditioned student.

Critical strengths

The deterministic construction provides an interesting basis for trajectory-based distillation in discrete diffusion. The distilled model shows promising few-step results, particularly for uniform diffusion.

Critical weaknesses

My main concern is whether the student actually learns the deterministic endpoint associated with its residual input. Changes in residuals can alter the order of token transitions and subsequent predictions, making this map potentially difficult to learn. Generation quality alone does not establish that the student reproduces this dependence.

The residual-conditioning ablation in Appendix H.10 makes this question particularly relevant. Removing residual inputs preserves similar performance in several settings and improves the reported TinyGSM MDM results. It therefore remains unclear whether the gains come from learning the deterministic coupling or primarily from soft-target self-distillation.

The smaller gains in masked diffusion also leave the practical benefit of the proposed construction less clear in that setting.

Questions for authors
  • With a fixed ERF teacher, how accurately does the student predict the endpoint associated with the same initial state and residual input?

  • Does matching the residual input to the teacher trajectory improve endpoint prediction compared with shuffled residual inputs?

  • How do the authors interpret the settings where removing residual conditioning maintains or improves performance?

3 comments

We thank the reviewer for these interesting questions and reviewing our paper carefully.

We agree that the actual learning of the theoretical map is not automatic and we do not claim that we learn the ideal mapping. However, we see two advantages to this formulation:

  1. It allows us to derive a principle trajectory-based distillation, closer to the standard continuous methods.
  2. As a byproduct the learned mapping, even it is not perfect is useful as a one-step approximation of p0∣tp_{0|t} (when marginalizing the residual), which is a central quantity is settings like guidance.

Changes in residuals can alter the order of token transitions and subsequent predictions, making this map potentially difficult to learn.

It is true that the map is a priori hard to learn but we do not believe it is impossible. It might be useful to notice that conceptually, this is very close to Strong Stochastic Flow Maps (McCallum et al) who learn a similar dependency in the continuous setting.

With a fixed ERF teacher, how accurately does the student predict the endpoint associated with the same initial state and residual input?

We ran an additional experiment on MNIST that shows that the network is able to approximately match the residual.

Pixel agreement (%) between one-step ERC predictions (argmax) and the 128-step teacher on binary MNIST.

Starting tUniform: matched EUniform: shuffled EMasked: matched EMasked: shuffled E
0.05099.1999.0699.9299.85
0.12599.1198.8399.8399.59
0.25099.1898.6199.6399.06
0.50098.6797.7498.7997.57
0.75096.3895.1394.1091.99
1.00094.4692.5577.8671.90

Results average over 16 held-out images and four exponential-clock draws per image and starting time. Shuffling changes E while holding X fixed.

Does matching the residual input to the teacher trajectory improve endpoint prediction compared with shuffled residual inputs?

The table, also shows that this agreement is higher when matching the input residual with the one the teacher starts with.

We want to emphasize that it is only possible to test this exact feature in the one-step setting, which explains why the results are not perfect. Indeed, when doing multiple steps with the student, we refresh the exponentials at each step so the final output does not correspond to the teacher trajectory started at that residual.

How do the authors interpret the settings where removing residual conditioning maintains or improves performance?

As mentioned by the reviewer, learning the map requires substantial computation inside the network. We hypothesize that it is sometimes easier (although suboptimal) for the network to just ignore it which may explain those results. However, by using the residuals it can theoretically reach better performance (and learn a useful object), hence the motivation of our method. The updated Appendix H.11 in the updated paper shows that changing the residuals while holding the tokens and noise time fixed changes ERC’s predictions. The student therefore learns a mapping that depends on the residuals. In principle, this gives it access to information about the endpoint of the teacher’s specific trajectory that is unavailable from the tokens and noise time alone.

An open question is how to process the exponential residuals so that the student more faithfully learns how the teacher’s endpoint depends on them. This could involve better flow-map losses that supervise the student along the teacher’s trajectories, or architectural changes that give the network a more effective way to use the residuals.

Thanks for rebuttal.

Yes, actually I already mentioned strong stochastic flowmap at section that is only visible to AC.

Actually I personally don't believe strong stochastic flowmap can really learn SDE map as well, since it is also be extremely hard to be trained, so again, lipshitz non-smooth. but since CTMC is much more chaotic compared to SDE, since we can't assume any local-smoothness of score function in CTMC compared to continuous space SDE.

I think binary MNIST example partially resolve my concern, especially large t regime. But still learnability of CTMC is exponential w.r.t. vocab size ^ sequence length, so considering simplicity of MNIST, not really sure I should believe this example as concrete evidence. Maybe this is my personal bias, but want to hear author's opinions on this.


To sum up, I want to see more evidence that (1) your model really learned deterministic mapping or (2) what mechanism that student model is using, if they are not learned exact mapping. Current score is kind of 2.5, but slightly lean to rejection.

We understand your point of view and agree that it is not obvious that a network could learn the ideal map because of the smoothness and generalization issue. However, we want to emphasize that even if the method is not able to recover this map, it is still learning some dependance on the residuals and not ignoring them as shown in the new experiment summed up in the additional figure that we put in Appendix H.11. Therefore, this mapping can be used as an approximation to a stochastic generator for p0∣tp_{0|t}.

Furthermore, just to clarify, we refer to Stochastic Flow Maps as relevant prior work illustrating the challenges associated with learning the flow of the ODE considered in our paper, but our framework is not exactly similar to theirs. Indeed, a genuine analogue of Stochastic Flow Maps in our setting would require learning a map that depends jointly on the initial condition and the realizations of LL Poisson processes, which is considerably more challenging than learning a map involving a finite number of exponential random variables.

In addition, we conducted extensive additional experiments on Fashion-MNIST, which presents a substantially more challenging setting than MNIST (with K=255K=255 for Fashion-MNIST, compared to K=2K=2 for MNIST). As shown in the table below, explicitly providing the exponential random variables as inputs significantly improves performance. Interestingly, we found that using Gumbel random variables instead of exponentials, obtained by simply applying a logarithmic transformation to the inputs, further improves the model's ability to learn the flow map. A similar observation was reported by Zhang et al. in Gumbel Distillation. Fashion-FD ↓

InputNFE 1NFE 2NFE 4NFE 8NFE 16
Gumbel155.21 ± 4.65119.06 ± 7.4697.33 ± 7.9796.10 ± 8.49126.96 ± 12.50
No noise input365.70 ± 4.54294.98 ± 4.31223.80 ± 5.31162.38 ± 5.12170.58 ± 5.62

KID ×1000 ↓

InputNFE 1NFE 2NFE 4NFE 8NFE 16
Gumbel463,677 ± 19,294341,718 ± 24,205256,276 ± 20,608241,563 ± 23,912325,890 ± 45,642
No noise input946,048 ± 5,792766,118 ± 11,491579,445 ± 12,474399,909 ± 14,857425,965 ± 17,005

It is worth noting that all the other distillation methods we tested (i.e., all baselines except ours) either collapsed or failed to produce satisfactory results on this dataset. In particular, despite extensive hyperparameter tuning, we were unable to obtain convincing results with these methods using only a few sampling steps. We find our latest results encouraging and believe that further methodological and algorithmic developments could make learning the flow map underlying ERC a viable and effective approach.

Similarly, we believe that our work opens the door to new methodological and algorithmic developments for discrete generative models. As was the case with early work on probability flow ODEs, we expect that further theoretical and practical investigations will be valuable in fully exploring the potential of this approach. Making it work better at scale is still an open problem but we hope this paper inspires some new works in the community that can figure out better ways.

Reviewer_HDTqOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes exponential residual flows (ERFs), a deterministic sampler for discrete diffusion models. The main idea is to augment the discrete state with exponential residual variables that evolve deterministically, with token changes triggered when residuals hit zero. The authors prove that this construction preserves the original diffusion marginals, and develop a practical sampler using leave-one-out denoiser freezing and net-flux rates. They further use these deterministic trajectories for consistency distillation (ERC), evaluating on text generation, GSM8K, Sudoku, and DiffusionGemma. The main empirical improvements are for uniform diffusion, while masked diffusion results are generally competitive with existing methods.

Critical strengths

[S1] Interesting and technically solid construction. I think the strongest contribution is the deterministic augmented flow itself. The idea of recycling exponential residuals instead of resampling after every jump is elegant, and Thm 3.1 gives a meaningful guarantee that the correct marginals are preserved. It's a long paper yet I found the detailed proof in App J to be informative, as it handles boundary crossings and nonexplosion rather than just relying on the continuity equation. While deterministic augmented samplers have been studied before (e.g., deterministic Gibbs ODEs and MAD Mix), the construction for time-varying discrete diffusion marginals seems novel and useful.

[S2] Thoughtful numerical development and promising empirical results. The leave-one-out freezing scheme is well motivated, especially the analysis of why naive score freezing can lead to infinitely many jumps. The net-flux implementation is also practical. I liked that the experiments cover different diffusion families and tasks, and compare sampling quality at matched NFE. The consistency distillation results for uniform diffusion are particularly promising, with noticeable gains at low NFE.

Critical weaknesses

[W1] The practical distillation gains are not sufficiently explained, and the connection to related consistency methods is incomplete. The exact ERF theorem is interesting, but the practical ERC procedure is quite different from what the theory directly justifies. In particular, the authors omit the endpoint boundary condition needed for the ideal consistency argument, and later refresh the transport teacher using the student itself. The paper acknowledges these choices, but it is unclear what prevents the learned endpoint map from drifting away from the desired distribution.

I also think Consistent Diffusion Language Models (ICML 2026) is an important missing reference. CDLM develops consistency in expectation over stochastic discrete trajectories, showing that deterministic trajectories are not necessary for principled discrete consistency learning. This does not invalidate ERF's marginal-preservation theorem, but it weakens the motivation that stochastic transitions are fundamentally incompatible with consistency distillation. CDLM's results on the importance of anchoring are also relevant here.

Finally, App H.10 shows that residual conditioning does not consistently improve results, especially for masked diffusion and some uniform-diffusion settings. Since this is the distinctive ingredient of ERC, I would like a clearer understanding of how much of the improvement comes from residual conditioning versus teacher refresh and the other training choices.

[W2] Some experimental choices make the reported improvements harder to assess. The TinyGSM configurations, including temperature, sampler, and ERC gap, appear to be selected on the same GSM8K test questions used for reporting the main results (App H.5). There is also only one run without confidence intervaals for the main distillation comparisons. This makes it difficult to know how robust the improvements are, particularly when some differences are relatively small.

I also found a few unclear details in the supplementary material. For example, the DiffusionGemma comparison in Table 1 is described around 48 NFEs, while Appendix I discusses a native sampler using only 11.1 evaluations on average. It is not fully clear which ancestral baseline appears in the table. The Sudoku teacher parameterizations and some baseline implementation details could also be clarified.

Questions for authors
  1. How does ERC compare with CDLM (Amin et al., 2026), especially for uniform diffusion? CDLM suggests that stochastic consistency objectives can already work well without deterministic trajectories. A comparison or clearer theoretical discussion would help establish what ERF adds for distillation.

  2. Can you clarify why removing the endpoint boundary condition is safe in practice, especially when the student is used to refresh the teacher? An anchoring ablation would be useful here.

  3. Do you have a matched ablation of residual conditioning under the main training setup? The current results in App H.10 seem mixed, and I am not convinced that conditioning on residuals is responsible for the main improvements.

1 comment

We would like to thank you for the thoughtful review of our paper and the positive remarks on the mathematical construction. We address the remaining concerns below.

How does ERC compare with CDLM (Amin et al., 2026), especially for uniform diffusion?

Regarding CDLM, we thank the reviewer for pointing out this work. The term consistency is used slightly differently in both papers, we used it following the approach of Consistency models from Song et al, which is different from the method of CDLM where trajectories remain stochastic. It is still interesting to compare to it as a baseline though, so we tested on fashion mnist and tinygsm.

On Fashion MNIST, CDLM collapses completely, or is very unstable to train (even with a sweep on the hyperparameters), while ERF default config produces recognizable images in few steps:

NFECDLMERC
11664.98 ± 176.391413.61 ± 167.02
21661.08 ± 176.261393.89 ± 167.86
41658.34 ± 175.981389.65 ± 167.68
81643.43 ± 175.171377.82 ± 166.03
161618.17 ± 174.371384.68 ± 169.16
321595.08 ± 171.631394.04 ± 165.65
641560.93 ± 169.691390.90 ± 163.44

Fashion-MNIST sample quality for the uniform CDLM and ERC models. Each entry uses 2,048 generated images and 10,000 official test images. Scores are polynomial MMD squared in a frozen Fashion-MNIST classifier's 128-dimensional features; lower is better.

On TinyGSM with uniform diffusion, ERC outperforms CDLM at 2–16 NFEs, while CDLM has higher accuracy at 32 and 64 NFEs (see updated Figure 3 in the paper, right panel).

Can you clarify why removing the endpoint boundary condition is safe in practice, especially when the student is used to refresh the teacher? An anchoring ablation would be useful here.

The endpoint boundary condition’s role is mostly to rule out trivial constant mapping (that would also be minimizers of the consistency loss). When including the anchor in preliminary experiments, it did not change our results. We hypothesize that this is because we initialize our student from the teacher network, so the mapping starts very far from those trivial solutions.

Answer to W2

For DiffusionGemma, the Ancestral row in Table 1 is the custom 48-NFE UDLM ancestral sampler, used for the comparison with the two 48-NFE ERF implementations. Appendix I also describes the model's official native sampler (which uses a variant of the entropy-bound sampler). That native procedure is not the ancestral row of Table 1, and its compute differs. The reported wall-clock numbers in Table 1 therefore should not be interpreted as a comparison to the native 11.1-NFE procedure. We will clarify this in the caption. For Sudoku, we train one MDM and one UDM teacher separately at each of three difficulty levels, for six teachers in total (Appendix G.1). All use the same DiT-style architecture (8 transformer blocks, hidden dimension 512, approximately 28.56M parameters), the same 20,000-step ELBO training recipe, and difficulty-specific datasets with 40, 35, or 30 clues (Tables 3–5). Tables 7–8 specify the evaluation and sampling selections. Appendices F.4–F.6 give the D-MMD, DCD, and iDLM implementations. We only modify the samplers as detailed in the experiment section. We will make these cross-references more visible in the main text.

Do you have a matched ablation of residual conditioning under the main training setup? The current results in App H.10 seem mixed, and I am not convinced that conditioning on residuals is responsible for the main improvements.

We want to emphasize that the goal is not only to say that the residual should improve the generative metrics but learning the dependence on the residuals, even if it is not perfect, is also a valuable byproduct of the method. Indeed, in the case of MDM for instance, at optimum, we get a stochastic one step generator for the posterior p0∣tp_{0|t} by calling the network with different exponential residuals, which is a quantity that is central in many settings such as guidance. To the best of our knowledge, ours is the first method that allows this (other methods do not necessarily target the original non-factorized p0∣tp_{0|t}). This property is another motivation for residual conditioning, in addition to yielding a principled trajectory distillation method. The mixed results in appendix H therefore do not remove the advantage of using the residuals, although they show that there is room for improvement. We have included in the updated appendix H.11 an experiment to show that we do learn some mapping involving the exponentials (even if it is not the ideal one). We will make that distinction clearer in the final version and highlight the added value of the residual conditioning.

Reviewer_btbHOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes exponential residual flows (ERF), a deterministic sampler for discrete diffusion that augments each (position, token) with an exponential residual clock and evolves all clocks by an ODE. The authors also show that the augmented marginals remain unchanged at every time step. A leave-one-out-frozen discretization is derived for masked and uniform diffusion, with a net-flux variant that has closed-form jump times. The deterministic transport is then used for consistency distillation. Experimentally, ERF matches ancestral sampling at equal NFE, and ERC gives clear few-step gains for uniform diffusion on TinyGSM and OpenWebText while being roughly on par with existing methods for masked diffusion.

Critical strengths
  • The ERF construction is theoretically novel and clean.
  • The authors provide extensive ablation studies, and negative results are also reported.
  • Writing in the main text is clear for a technically dense construction.
Critical weaknesses
  • Despite its theoretical novelty, ERF does not demonstrate large enough practical gain when compared with existing distillation baselines, with most of its performance gain coming from UDM.
  • NFE may not be a good proxy for sampling efficiency, since the auxilary variables may introduce extra computational overhead.
  • The protocol for distillation experiments is lenient: no careful tuning for temperature or the nucleus sampling probability. Besides, all the baseline results are based on the authors' reimplementation, and it is unclear what the replication quality is.
Questions for authors
  • Can the authors elaborate on the baseline replication quality?
  • Can the authors explain why in Table 1, the improvement is marginal and statistically insignificant? Given that this is the only experiment on billion-scale diffusion language models, can this method be applied to more practical real-world scenarios?
  • Can the authors report sampling wall-clock time as a better proxy for the inference speed?
1 comment

We would like to thank you for taking the time to review our paper. We address the remaining concerns below.

The protocol for distillation experiments is lenient: no careful tuning for temperature or the nucleus sampling probability.

We thank the reviewer for raising these points. Appendix G.9 (Figures 28–36) already reports the temperature and sampler sweeps for every distillation method and the shared teacher on TinyGSM and OpenWebText. We evaluate all the methods under both bridge and refresh sampling on the same eight-temperature grid for all methods within each dataset. On OpenWebText, Figure 4 shows the perplexity–entropy trade-off across temperatures using bridge sampling for all methods.

Regarding nucleus sampling, we fix it at p=1p=1 throughout the reported comparison for all methods.

Baseline replication

Regarding the quality of replications we would like to emphasize the following points:

  • Duo-DCD: We ran the authors’ official implementation directly from their codebase and obtained matching results in our replication check.
  • iDLM: in their released code their ancestral sampler can perform categorical draws in float32 even when the float64 option is enabled. This is a known source of bias that can lower both generative perplexity and sample entropy. The iDLM paper reports 41.29 GenPPL and 5.28 entropy for the 1,024-step MDLM teacher on OpenWebText. The corresponding MDLM reference in the Duo paper reports 104.85 and 5.63. This motivates our re-implementation.
  • D-MMD: We reimplemented the published algorithm because the code used for the paper’s experiments was unavailable. In a separate OpenWebText check at p=1p=1, our masked D-MMD student achieved a GPT-2 Gradient Moment of 0.716 at 16 NFEs, improving on 0.899 for its 256-NFE teacher. The published values were 0.558 and 0.672, respectively. We reproduced the improvement over the p=1p=1, with a gap in absolute scores which is seemingly not linked to the distillation procedure.

Overall we would like to insist that we have spent a significant time making sure that the baselines are well reproduced and fair. We actually even slightly modified the original methods to improve their results. For instance, the IDLM paper reports accuracies of 14.94% for MDM and 19.03% for UDM at temperature 1 and nfe 64, whereas with our modifications (different sampler adapted to their method), we reach over 30% accuracy at the same temperature and nfe. Similarly, for D-MMD sweeping the sampler gives big improvements as shown in the appendix.

Can the authors explain why in Table 1, the improvement is marginal and statistically insignificant? Given that this is the only experiment on billion-scale diffusion language models, can this method be applied to more practical real-world scenarios? Can the authors report sampling wall-clock time as a better proxy for the inference speed?

The goal of Table 1 is not to show that ERF improved over ancestral sampling, we do not claim this. Our goal is two-fold: Show that we do not lose quality compared to ancestral sampling, showing that ERF is a valid sampler to use for the purpose of distillation. And second, show that computationally, the ERF sampler does not introduce overhead even at large scale.

Our ERC student indeed has overhead at sampling time due to the projection layer in the network used to process the exponentials. However, as this overhead is only caused by the initial embedding layer multiplication (it is the only architectural change), it should get only smaller when scaling the network size (at fixed vocabulary size). We report below the wall-time for the tinygsm experiment for all methods

Method2 NFEs4 NFEs8 NFEs16 NFEs32 NFEs64 NFEs
Teacher37.674.9149.5298.9597.31,197.9
DCD37.675.2150.1300.1598.71,211.8
iDLM37.775.2150.2300.2600.11,211.4
D-MMD42.584.3168.5337.0675.91,352.0
ERC51.0102.1205.0411.6822.21,652.1

We thank the reviewer for pointing this out and will make sure to include a more comprehensive study and discussion of the wall-time in the final version.

Despite its theoretical novelty, ERF does not demonstrate large enough practical gain when compared with existing distillation baselines, with most of its performance gain coming from UDM.

We acknowledge that ERC’s empirical gains are concentrated in uniform diffusion. We view these results as a proof of concept that deterministic ERF trajectories can support effective few-step distillation. We see substantial room for improvement, particularly in developing flow-map objectives and better understanding how the student network should encode the exponential residuals.

Our gains over state-of-the-art baselines are smaller than those reported in some prior work. This partly reflects the effort we put into fair evaluation and strengthening the baselines.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.