acceptodds
Under review as a conference paper at ICLR 2027

Discrete Diffusion Models with Interpolation-Based Controllable Resampling

Abstract

Discrete diffusion models form a powerful class of generative models across diverse data modalities, including graphs and text. However, existing approaches face fundamental limitations. Masked diffusion models suffer from irreversible errors if tokens are unmasked too early in the reverse process, while uniform diffusion models, despite enabling self-correction, often yield low-quality samples due to their strong reliance on intermediate latent states. To address these limitations, we introduce IDDM, an Interpolation-Based Discrete Diffusion Model that improves diffusion by reducing dependence on intermediate latent states. Central to IDDM is a controllable resampling mechanism that partially resets probability mass to the prior distribution, mitigating error accumulation and enabling token corrections. IDDM defines a generative process whose transitions interpolate between staying at the current state, resampling from a prior, and flipping toward the target state, while enforcing marginal consistency and fully decoupling training from inference. We benchmark our model against state-of-the-art discrete diffusion models on molecular graph and text generation tasks, showing competitive performance.

Primary area:
generative models
Keywords:
discrete diffusion, generative model, graph generation, language modelling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wJsBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work introduces a discrete generative model with reverse transitions mixing the following actions: preserving the current state, predicting the clean token, and resampling from a categorical prior. The paper derives a continuous-time training objective for the zero-resampling setting which recovers masked diffusion as a special case. Experiments on molecular graph generation and text generation show that the proposed method achieves competitive performance with fewer sampling steps

Critical strengths
  1. Simple construction for controllable resampling.

This method uses the same transition rule with masking, uniform noise, or noise sampled according to category frequencies in the data. This is useful for graph generation where nonuniform priors matter.

  1. Strong results on both graphs and text.

The results show strong performance on both GuacaMol and GenPPL on OWT.

Critical weaknesses
  1. When do the theoretical results apply and what does decoupling guarantee are unclear.

Equation 14 is derived only for λ=0\lambda=0. The main text should make this clear. Extending this to positive resampling seems to need more analysis.

Also, changing the sampler without retraining does not guarantee good performance.

  1. Molecular results show improvements in some settings but not all.

On MOSES, the full step model has worse FCD than DiGress and Cometh, and lower validity than DeFoG. Best QM9 configuration uses no resampling, so it does not demonstrate a benefit from resampling itself.

Questions for authors

Please address the weakness.

3 comments

We thank the reviewer for their valuable comments. In the following, we address their points.

When do the theoretical results apply and what does decoupling guarantee are unclear. Equation 14 is derived only for λ=0\lambda=0. The main text should make this clear. Extending this to positive resampling seems to need more analysis. Also, changing the sampler without retraining does not guarantee good performance.

We thank the reviewer for raising this point.

The main benefit of decoupling is the flexibility to adjust sampling without retraining. In standard uniform and absorbing diffusion models, the reverse transition is fixed by the forward process through Bayes’ rule. In contrast, IDDM constructs a family of valid reverse kernels, parametrized by λ\lambda, that preserve the same prescribed marginal path. Because the training marginals are independent of λ\lambda, a single trained denoiser can be reused with different resampling schedules. This enables adaptation to the task and computational budget through inference-time tuning. Our construction applies to arbitrary categorical priors, including masking, uniform, and empirical priors.

We agree that this flexibility does not guarantee improved sample quality. However, our experiments demonstrate benefits across the evaluated priors and domains. These improvements are particularly pronounced in molecular generation at reduced sampling budgets. At 50 steps, compared with DeFoG, IDDM reduces QM9 FCD from 0.26 to 0.11. On GuacaMol, it improves validity from 91.7% to 98.1% and FCD score from 57.9 to 77.6. On MOSES, it improves validity from 83.9% to 87.1% while reducing FCD from 1.87 to 1.14. Our language experiments further show that controlled resampling improves generation quality under uniform priors and narrows the performance gap to masked diffusion language models.

Finally, we updated our manuscript to clarify that the continuous-time ELBO is derived for λ=0\lambda=0 under arbitrary categorical priors. We further updated our discussion to state that extending this analysis to positive resampling remains an interesting direction for future work.

Molecular results show improvements in some settings but not all. On MOSES, the full step model has worse FCD than DiGress and Cometh, and lower validity than DeFoG. Best QM9 configuration uses no resampling, so it does not demonstrate a benefit from resampling itself.

We agree that the improvements depend on the dataset, number of sampling steps, and evaluation metric. While on QM9, our best FCD is achieved without resampling, our ablations on MOSES and GuacaMol demonstrate benefits from positive λ\lambda at 50 steps. On MOSES at 500 steps, resampling improves validity and SNN by trading off FCD.

Further, IDDM provides a unified framework that connects existing methods by extending ReMDM-style correction [1] to arbitrary categorical priors and recovering SID [2] as the special case λ=1\lambda=1. Specifically, IDDM derives a family of marginal-consistent reverse kernels parameterized by λ\lambda, allowing the resampling strength to be flexibly adjusted at inference time using a single trained model.

We again thank the reviewer for their feedback and are happy to clarify any remaining questions or suggestions.

[1] Wang, Guanghan, et al. "Remasking discrete diffusion models with inference-time scaling." NeurIPS (2025).

[2] Boget, Yoann. "Simple and critical iterative denoising: A recasting of discrete diffusion in graph generation." ICML (2025).

Thank you for the clarifications and additional analyses. I see that the revised paper now makes the λ=0\lambda=0 restriction explicit, and the response clarifies that decoupling provides inference-time flexibility without guaranteeing improved sample quality. The response on dataset and budget dependent benefits also addresses my concern. Although the positive-resampling ELBO analysis remains open, these clarifications largely address my original concerns. Thus I maintain my score for weak acceptance.

We thank the reviewer for their response and are glad that their concerns have been addressed.

Reviewer_EXAUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposed an approach to improve the sampling process of discrete diffusion models. They propose to include controllable possibility of 3 actions, sampling from the prior, staying at current state, or choosing a data state during the denoising process. The main goal is to enable self-correction when an incorrect sampling occurs in diffusion language models. Furthermore, it is shown that MDLM is a special case of their proposed IDDM when the control is fully turned off.

Critical strengths

Proposed approach seems new, sound and interesting to me. Results are provided for both graph generation (molecule generation) and sequence generation (language modelling) and performance is shown to be competitive or better.

Critical weaknesses

Result analysis is lacking and results are generally not discussed well. On text generation specially, it would be beneficial to look at unconditional sampling since understanding sample quality from PPL and entropy alone is hard (there is usually a tradeoff) The role of the test time change of lambda is not clear. What can be achieved?

Please answer Q1 in questions.

Questions for authors

Q1: Are re-samplings from prior always useful to fix, mistakes? Sometimes they might just be added work without benefit.

L378: Explain more clearly what are IDDM (Mask) and IDDM (Uniform) and what is the difference to IDDM (ours) used in Tables 1 and 2.

nit: L142-144: In the middle of the paragraph, suddenly the context switches to graph generation and the discussion is no longer kept general. L377: It is better to include the general details of the model configuration in the main text.

2 comments

We thank the reviewer for their thoughtful feedback. In the following, we address their comments.

W1. Result analysis is lacking and results are generally not discussed well. On text generation specially, it would be beneficial to look at unconditional sampling since understanding sample quality from PPL and entropy alone is hard (there is usually a tradeoff) The role of the test time change of lambda is not clear. What can be achieved?

We thank the reviewer for this suggestion. We expand our analysis of unconditional text generation on OpenWebText by sweeping λ\lambda during inference for both IDDM (Mask) and IDDM (Uniform), which were trained with λ=0\lambda=0. We compare constant, linear-decay, and loop resampling schedulers across multiple sampling budgets; see Figure 5 in the updated manuscript.

At a fixed sampling budget, varying λ\lambda enables trading off generative perplexity against sample entropy without retraining. The loop schedule generally reduces perplexity while preserving entropy, whereas stronger resampling with the constant and linear-decay schedules can further reduce perplexity at the cost of a larger drop in entropy.

Q1. Are re-samplings from prior always useful to fix, mistakes? Sometimes they might just be added work without benefit.

As the reviewer noted, resampling often helps improve performance. However, while it provides opportunities to revise incorrect predictions from earlier steps, it can also corrupt correct states. Its benefit varies across datasets, number of steps, and evaluation metrics. Our MOSES ablations, for example, show that moderate resampling improves FCD at 50 steps, whereas at 500 steps positive λ\lambda improves validity and SNN by trading off FCD (see Fig. 8). Thus, λ\lambda provides a trade-off to be adjusted at inference without retraining.

Explain more clearly what are IDDM (Mask) and IDDM (Uniform) and what is the difference to IDDM (ours) used in Tables 1 and 2.

The only differences are the choices of the prior distribution π\pi. IDDM (Mask) and IDDM (Uniform) represent our text-generation models trained with a masking and a uniform prior, respectively. For graph generation, IDDM (ours) in Tables 1 and 2 uses the empirical marginal priors over node and edge attributes, estimated from the training data. This is done to enable a fair comparison with the baselines in each domain. We have renamed the corresponding row in the updated manuscript to IDDM (marginal).

nit: L142-144: In the middle of the paragraph, suddenly the context switches to graph generation and the discussion is no longer kept general.

We thank the reviewer for their suggestion. We revised this paragraph to introduce empirical marginal priors in general terms and keep the discussion of node and edge attributes in the graph-specific section. We are happy to make further changes if requested.

L377: It is better to include the general details of the model configuration in the main text.

We updated the manuscript to clarify the model configuration for the OWT experiment (See Section 4.2). We are happy to move further details from the appendix to the main text.

Again, we thank the reviewer for their feedback and are happy to address any remaining questions.

We thank the reviewer for their positive feedback on our work. We have addressed the concerns raised during the rebuttal and revised the manuscript accordingly. As the discussion period ends soon, we would appreciate feedback on whether our responses and revisions address the reviewer’s concerns. We are happy to clarify any remaining points.

Reviewer_RZpBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes IDDM, which allows discrete generative models to revise earlier mistakes during inference and supports categorical priors beyond masking. The method combines three actions: keeping the current state, moving toward the predicted clean state, and resampling from the prior. A parameter λ controls the strength of these updates. The same denoiser can use different λ values at inference without retraining.

Critical strengths
  1. The method is simple and intuitive, with an explicit derivation of marginal consistency for the ideal transition kernel.

  2. The results on GuacaMol and some low-step molecular generation settings are strong.

Critical weaknesses
  1. Section 4.3 counts state changes, but more changes do not necessarily mean more corrections. In controlled denoising experiments with known clean targets, it would help to separately measure how often incorrect states are corrected and correct states are corrupted.

  2. The baseline results in Tables 1 and 2 come from other papers, so the comparability of their settings and budgets needs further clarification. The paper also lacks error bars across repeated runs and comparisons of sampling time and network evaluations on the same hardware.

Questions for authors
  1. In Figure 6, QM9 generally achieves better FCD with λ=0. Why does resampling behave differently on QM9 than on datasets with larger molecules? Which tasks and sampling budgets benefit most from it?

  2. Where the configurations are compatible, could the authors fix the denoiser and compare IDDM with existing correction samplers? Examples include ReMDM, FB, and DFM correctors under absorbing priors, and DeFoG’s stochasticity adjustment for graph generation.

4 comments

We thank the reviewer for their feedback. In the following, we address their comments.

W1. Section 4.3 counts state changes, but more changes do not necessarily mean more corrections. In controlled denoising experiments with known clean targets, it would help to separately measure how often incorrect states are corrected and correct states are corrupted.

We agree with the reviewer that more changes do not necessarily mean more corrections.

Therefore, we follow the reviewer's suggestion and investigate how often flips from correct to incorrect and from incorrect to correct states happen in a controlled setting. For the kernel conditioned on the clean target (x), the corresponding probabilities can be derived in closed form using Eqs.9-11 given λ\lambda and the prior distribution π\pi, using the transition probabilities. For our linear schedule (γ(t)=1−t)(\gamma(t)=1-t), the probabilities for a reverse transition (t→s<t)(t\to s<t) are

$P(\mathrm{incorrect}\to\mathrm{correct}) =(1-\lambda)\left(1-\frac{s}{t}\right) +\lambda\left[1-s(1-\pi(x))\right]$, and $P(\mathrm{correct}\to\mathrm{incorrect}) =\lambda s(1-\pi(x)).$

The probabilities depend on λ\lambda, the diffusion time, and the prior probability of the ground truth class xx (π(x)\pi(x)). Increasing λ\lambda can increase both correction and corruption probabilities. We have included a figure in the updated manuscript illustrating this trade-off across λ\lambda and diffusion time.

To complement this controlled setting, we compute the same probabilities using the trained, imperfect QM9 denoiser. We corrupt the test graphs and compute correction and corruption probabilities from the learned reverse kernel, using the clean graphs as ground truth. We average these probabilities over initially incorrect and initially correct states, respectively. The corresponding figure shows the results as functions of λ\lambda and time, separately for each atom and edge category.

We included these figures in the updated manuscript (Figs 10-12).

W2. The baseline results in Tables 1 and 2 come from other papers, so the comparability of their settings and budgets needs further clarification. The paper also lacks error bars across repeated runs and comparisons of sampling time and network evaluations on the same hardware.

We thank the reviewer for raising this point.

Our implementation builds directly on DeFoG’s codebase and uses the same graph-transformer architecture and Relative Random Walk Probability (RRWP) features. We evaluate 10,000 generated graphs each for QM9 and GuacaMol and 25,000 for MOSES, matching the baselines reported evaluation sample counts. In Tables 1 and 2, we compare IDDM with the published DeFoG results at the same sampling-step budgets of 50 and 500 steps on all three datasets. The label “10% steps” refers to 50 steps out of 500. For DiGress, DisCo, and Cometh, we report the published results at a budget of 500 sampling steps. We updated our manuscript to explicitly state the number of sampling steps for each method. We also include additional experimental details (See Appendix F).

We reuse DeFoG's evaluation pipeline and the official GuacaMol and MOSES benchmark implementations. Additionally, we evaluated DeFoG’s available Moses checkpoints across various configurations to verify the values reported in their paper. While the baseline scores in Tables 1 and 2 are taken from the corresponding publications, we have made sure that our evaluation follows the standard benchmarking procedure.

We included the mean and standard deviation for the QM9 results computed across four independently trained models in the updated manuscript.

ModelStepsValidity ↑\uparrowRelaxed validity ↑\uparrowUniqueness ↑\uparrowFCD ↓\downarrow
IDDM (Marg.)50099.41 ±\pm 0.0999.54 ±\pm 0.0896.39 ±\pm 0.110.102 ±\pm 0.009
IDDM (Marg.)5099.25 ±\pm 0.0999.37 ±\pm 0.0996.48 ±\pm 0.190.109 ±\pm 0.011

The conclusion and interpretation remain unchanged. For GuacaMol and MOSES, we follow prior work and report standard deviations across sampling runs.

DatasetStepsValidity (%)FCD ↓\downarrow / FCD score ↑\uparrow
GuacaMol5097.80 ±\pm 0.01Score: 78.01 ±\pm 0.16
GuacaMol50099.60 ±\pm 0.04Score: 79.29 ±\pm 0.08
MOSES5086.88 ±\pm 0.33FCD: 1.136 ±\pm 0.012
MOSES50090.47 ±\pm 0.37FCD: 1.602 ±\pm 0.027

Finally, we report wall-clock comparison times in a direct comparison to DeFoG on an NVIDIA H100 80GB GPU. The table reports sampling time in seconds per 1,000 graphs normalized from five batches of 256 graphs.

DatasetSteps / NFEIDDM (Marg.)DeFoG
QM9503.073 ±\pm 0.0113.289 ±\pm 0.011
QM950030.742 ±\pm 0.01532.374 ±\pm 0.021
MOSES5014.841 ±\pm 0.00414.203 ±\pm 0.007
MOSES500148.344 ±\pm 0.026141.479 ±\pm 0.048

As we observe, the NFEs match and the runtimes are comparable.

We further report wall-clock sampling time in seconds for 1,000 OpenWebText sequences of length 1,024, generated on a single NVIDIA H100 80GB GPU with batch size 16 and top-p 0.9. We compare IDDM with MDLM using matching backbone architectures and approximately equal parameter counts.

StepsIDDM (Uniform)IDDM (Mask)MDLM
12837.5437.0034.73
25675.3374.0069.21
512150.12148.01138.19

Q1. In Figure 6, QM9 generally achieves better FCD with λ=0. Why does resampling behave differently on QM9 than on datasets with larger molecules? Which tasks and sampling budgets benefit most from it?

While a lower λ\lambda yields better FCD on QM9, our ablations in Figs.7-8 show benefits of resampling on other datasets. More specifically, we observe that the FCD performs best with a λ\lambda around 0.2-0.4 for large molecules when restricted to 50 steps. Further, the validity and SNN improve when using λ>0\lambda>0, for both 50 and 500 steps. Thus, λ\lambda provides a valuable trade-off parameter.

We assume this happens because QM9 consists of small molecules where the model has sufficient flips and resampling does not offer additional benefits.

Q2. Where the configurations are compatible, could the authors fix the denoiser and compare IDDM with existing correction samplers? Examples include ReMDM, FB, and DFM correctors under absorbing priors, and DeFoG’s stochasticity adjustment for graph generation.

Following the reviewer’s suggestion, we evaluate FB and DFM sampling using the same fixed IDDM denoiser trained with λ=0\lambda=0, and compare them with our resampling strategy. For each setting, we generate 1,000 unconditional sequences of length 1,024 using top-p sampling with p=0.9. Both FB and DFM reduce generative perplexity relative to the no-correction baseline (λ=0\lambda=0), however, they lead to lower sample entropy across all sampling budgets. Our loop schedule provides the best trade-off.

Samplerλ\lambdaGenPPL T=128GenPPL T=256GenPPL T=512Entropy T=128Entropy T=256Entropy T=512
Baseline0.0046.542.842.55.45.35.3
Cosine0.0537.239.927.45.3‾\underline{5.3}5.35.2‾\underline{5.2}
Linear0.0442.236.931.05.3‾\underline{5.3}5.35.3
Loop0.3032.0‾\underline{32.0}32.722.7‾\underline{22.7}5.45.35.2‾\underline{5.2}
FB—35.332.5‾\underline{32.5}31.55.3‾\underline{5.3}5.2‾\underline{5.2}5.2‾\underline{5.2}
DFM—22.919.417.95.05.05.0

Further, we evaluate IDDM using the pretrained DeFoG checkpoints. We reused the same λ\lambda values as in Table 1.

Samplerρ\rhoValidityFCD
DeFoG—83.91.87
IDDM288.39 ±\pm 0.091.179 ±\pm 0.011
IDDM488.92 ± 0.041.161 ± 0.015
Samplerρ\rhoValidityFCD score
DeFoG—91.757.9
IDDM296.20 ±\pm 0.0479.24 ±\pm 0.15
IDDM497.90 ±\pm 0.1480.13 ±\pm 0.20

As we observe, IDDM achieves similar results with their checkpoint and consistently outperforms DeFoG.

We thank the reviewer again for their valuable feedback. We are happy to address any remaining points.

We would like to thank the reviewer again for their insightful feedback, which helped us improve our paper. As the discussion period ends soon, we would greatly appreciate feedback on whether our rebuttal has addressed the concerns raised. We are happy to answer any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.