acceptodds
Under review as a conference paper at ICLR 2027

Simplex Diffusion Models

13 views

Abstract

Diffusion models have revolutionized generative modeling for continuous data through the gradual refinement of a belief state. This iterative refinement has not yet carried over to discrete diffusion models, which discard uncertainty at intermediate steps through categorical sampling (information collapse). We propose Simplex Diffusion Models (SDMs), a framework that lifts the diffusion process to the probability simplex to represent beliefs over categories. SDMs admit probability paths with closed-form reverse transitions and can be trained with a simple cross-entropy loss. Contrary to earlier proposals such as Dirichlet Flow Matching which requires integrating an ordinary differential equation, we introduce a DDIM-like sampler with a tunable level of stochasticity. Because SDMs operate on samples on the simplex, they can carry uncertainty across denoising steps, which mitigates information collapse. On OpenWebText, SDMs are competitive with strong discrete diffusion baselines, achieving 17.0 GenPPL at 5.46 unigram entropy in 64 sampling steps, close to real validation data. Even without self-conditioning (SC), SDMs outperform masked and uniform diffusion (with SC or predictor-corrector sampling) on code generation (TinyGSM, T=0.1; 49.0% vs. 45.8%). Distilled down to 8 steps, SDMs solve 32.1% of GSM8K problems, more than distilled discrete diffusion models with 128 steps (21.4%).

Primary area:
generative models
Keywords:
discrete diffusion, simplex, diffusion language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.80 over 5 reviews, above 85% of the others.
Reviewer_WZUfOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Simplex Diffusion Models (SDMs), which model categorical data through continuous states on the probability simplex. The forward process uses Dirichlet distributions with separately controlled means and concentrations. The main methodological contribution is a compatible backward bridge with closed-form sampling and adjustable stochasticity, enabling a DDIM-like sampler trained using cross-entropy. The paper also establishes connections to categorical diffusion and deterministic interpolation through concentration limits. Experiments cover Sudoku, TinyGSM/GSM8K, OpenWebText, and molecular generation. The strongest empirical evidence comes from code-generation accuracy, with additional promising results from few-step distillation.

Critical strengths
  1. A substantive and well-developed sampling construction. Proposition 3.3 provides a compatible Dirichlet backward bridge through thinning and innovation, with a tunable stochasticity parameter. This offers a concrete alternative to numerical ODE/SDE integration for simplex-based generative modeling. The concentration-limit analysis further clarifies how the construction relates to discrete and continuous representations.

  2. Meaningful evaluation beyond unconditional text metrics. The functional evaluation on TinyGSM/GSM8K provides useful evidence of practical value: at 512 NFEs and temperature 0.1, the expected-embedding SDM without self-conditioning achieves 49.0% accuracy, compared with 45.8% for MDM with self-conditioning. The extensive baseline sweeps and input/schedule ablations strengthen this evidence. I also appreciate the explicit discussion of the limitations of GenPPL/entropy frontiers and the diversity reduction caused by distillation.

Critical weaknesses
  1. The headline distillation result is selected on the test set. Section P.3 explicitly states that the reported 32.1% is selected from a hyperparameter sweep using the GSM8K test set, without a held-out validation split. The reported range across all 36 configurations is reassuring, but it does not make the selected result an unbiased estimate of generalization. Moreover, the IDLM results are taken from another paper with different teachers and distillation objectives, so the comparison does not isolate the benefit of the proposed simplex formulation.

  2. The explanation based on preserving uncertainty is not established for the strongest configurations. These generally use κ=1, where Proposition 3.3 gives a transition independent of the previous state conditional on the sampled clean token. Algorithm 2 first samples that token categorically, so, without self-conditioning, information from the previous state passes through a discrete bottleneck. The Argmax input further restricts what the denoiser observes. This does not invalidate the sampler, but it leaves unclear whether the gains arise from persistent uncertainty, the corruption distribution, or the sampling schedule.

Questions for authors
  1. Evaluation protocol: Can you report results with hyperparameters selected on a separate validation set and evaluated once on an untouched test set? Please also clarify how the non-distilled SDM and baseline configurations were selected, and provide uncertainty estimates for the principal TinyGSM comparisons.

  2. Mechanistic clarification: How should the uncertainty-preservation claim be interpreted when κ=1? In particular, with Argmax inputs and no self-conditioning, can the effective sampler be represented as a discrete-state chain? Which existing results distinguish benefits from retaining the previous simplex state from benefits of the Dirichlet corruption and adaptive schedule?

  3. Compute comparability: Section O.2 reports that expected embeddings add approximately 29% to forward-pass FLOPs. Does the training-budget matching account for this overhead, in addition to self-conditioning? Do the principal accuracy advantages persist at matched total compute or measured inference time?

3 comments

We are pleased that the reviewer found our sampler substantive and well-developed, and appreciated our evaluation beyond unconditional text generation. We appreciate their acknowledgement of our extensive baseline sweeps.

Important: We apologize for not stating which churn κ\kappa we use on OpenWebText (OWT). Unlike on Sudoku and TinyGSM, κ=1\kappa=1 hurts sample quality on OWT, consistent with prior work on predictor-corrector samplers [1]. A tuned schedule κt\kappa_t could outperform κ=0\kappa=0, but [1] shows that such tuning is costly, so we use κ=0\kappa=0 for simplicity. In Section 5.3, we now state "We train a 12-layer DiT with hidden dimension 768 and 12 attention heads (167.9M parameters) for 1M steps with a batch size of 512. In this scenario, setting κ=0\kappa=0 worked best."

The headline distillation results are selected on the test set.

We now select the configuration with the highest accuracy on the GSM8K training set at 8 NFEs, and report its test accuracy across NFEs: it solves 32.7% of the test problems with 8 NFEs and 39.7% with 64 NFEs (updated App. P.3, Table 7), more than IDLM with 128 NFEs (21.4%; Table 7).

How were the non-distilled SDMs and baseline configurations selected?

All models share the same pre-training, which follows recent diffusion language models [2, 3, 4]: the same architecture, optimizer and batch size, with SC variants trained for 77% of the 250k steps to match the training FLOPs (App. O.6). We then sweep the sampling grid for every method, the training time sampler and loss for the baselines, and the concentration schedule and churn for SDMs (Tables 28-35). As for distillation, Fig. 1 compares the best configuration of each method on the GSM8K test set, our evaluation split (App. O.6).

Uncertainty estimates for the main TinyGSM comparison.

We thank the reviewer for this suggestion. We now report 95% bootstrapped confidence intervals in Fig 1.

The IDLM results are obtained with a different teacher and objective. Can we isolate the benefits of the simplex formulation?

We thank the reviewer for this question. IDLM distills discrete diffusion teachers with a lower accuracy than our SDM teacher, so our comparison indeed cannot isolate the impact of the distillation algorithm.

We did not mean to imply that IDLM is a weaker method. IDLM was unstable in our early experiments. We have amended the paragraph "Distillation and Auto-Guidance" (Sec. 5.2) accordingly.

More broadly, our paper does not aim to provide a comprehensive study of distillation methods for SDMs. We provide one efficient distillation method for SDMs, and leave a comprehensive study of distillation for simplex-based models to future work.

The explanation of better performance based on preserving uncertainty does not hold for κ=1\kappa=1.

At κ=1\kappa=1, the next state depends only on the latest draw of x^0\hat{x}_0 (Prop. 3.3), so preserved uncertainty does not explain our results with κ=1\kappa=1. Only with κ<1\kappa<1 does the next state depends on the current state PtP_t. Importantly, with κ=0\kappa=0 and ct=ε/(1−αt)c_t = \varepsilon/(1-\alpha_t), PtP_t is a random convex combination of the initial noise and all past draws of x^0\hat{x}_0 (Sec. 4.1). We therefore only claim that the simplex state preserves uncertainty with κ<1\kappa<1.

The best churn depends on the task. On Sudoku and TinyGSM, κ=1\kappa=1 performs best, consistent with prior work on logic tasks: [5] report that "re-corruption alone markedly improves solve rates on logic tasks", and in our baselines, pure predictor-corrector sampling improves UDMs from 33.7% to 46.3% on TinyGSM (T=0.1T=0.1, 512 NFEs; Table 30). On OpenWebText, we use κ=0\kappa=0, which carries the state across steps.

At κ=1\kappa=1, with Argmax inputs and no SC, can the sampler be represented as a discrete-state chain?

Yes. At κ=1\kappa=1, PsP_s depends only on the sampled x^0\hat{x}_0 (Prop. 3.3), and with the Argmax input, x^0\hat{x}_0 depends only on arg⁡max⁡Pt\arg\max P_t, so the argmax tokens form a discrete-state Markov chain.

With the uniform prior, arg⁡max⁡Ps\arg\max P_s equals x^0\hat{x}_0 with a probability that depends only on ss, and is uniform over the other tokens otherwise. This chain therefore has the form of the predictor-corrector sampler of uniform diffusion (UDM + PC, App. N.2), with a different noise schedule.

Does the training budget matching account for the 29% FLOPs overhead of the Expected Embedding?

With Expected Embedding, we did not account for the 29% training FLOPs overhead, and train for 250k steps, like the baselines. We chose so because the overhead shrinks with model size (see next response). With the Argmax + SC input, we match the training and sampling FLOPs of discrete diffusion with SC.

Do the accuracy advantages persist at matched training and inference compute?

Matched training and sampling FLOPs: with the Argmax + SC input, SDMs cost as much as discrete diffusion with SC, in both training and sampling. At 512 NFEs and T=0.1T=0.1, they reach 53.6%, vs. 46.6% for MDM + SC (Fig. 1).

Matched sampling FLOPs: we added a version of Fig. 1 with inference FLOPs on the x-axis, and explain how we count the FLOPs in the new App. O.3. With the expected embedding, SDMs at 512 NFEs (89 TFLOPs) outperform the best baseline at 1k NFEs (139 TFLOPs): 50.0% vs. 47.8% at T=0.1T=0.1, and 46.6% vs. 40.1% at T=1T=1.

The 29% overhead of the expected embedding is large at our scale, but it shrinks with model size. With the Llama 3.1 architecture and vocabulary (~128k tokens), the overhead is 6.1% at 8B, 1.4% at 70B, and 0.5% at 405B (App. O.3).

Which existing results distinguish the benefits of retaining the previous simplex state from those of the Dirichlet corruption and the adaptive schedule?

We thank the reviewer for this question. The TinyGSM results below (T=0.1T=0.1, 512 NFEs) compare matched configurations:

Retaining the state (sweep the churn): retaining the state does not help on TinyGSM, as the accuracy increases with κ\kappa: 19.6% with κ=0\kappa=0, 44.7% with κ=0.2\kappa=0.2, and 50.0% with κ=1\kappa=1 (Table 34). Similarly, without SC, pure predictor-corrector sampling improves over ancestral sampling for discrete diffusion (UDM: 33.7% -> 46.3%; MDM: 32.6% -> 40.3%; Table 30).

Dirichlet corruption (κ=1\kappa=1): UDM + PC solves 40.9% of problems (46.4% with its best configuration; Table 30), vs. 43.7% for SDMs with the Argmax input and 50.0% with the expected embedding (Table 34).

Adaptive sampling grid (κ=1\kappa=1): the accuracy of SDMs increases from 41.2% with the linear grid to 50.0% with the adaptive grid (Table 34).

Finally, we emphasize that for OWT, we found that κ=0\kappa=0 was the best churn hyperparameter and that choosing pure renoising κ=1\kappa=1 was hurting performance.

[1] Deschenaux et al. The Diffusion Duality, Chapter II: Ψ-Samplers. ICLR, 2026.

[2] Sahoo et al. Simple and Effective Masked Diffusion Language Models. NeurIPS, 2024.

[3] Sahoo et al. The Diffusion Duality. ICML, 2025.

[4] Deschenaux and Gulcehre. Language Modeling with Hyperspherical Flows. TMLR, 2026.

[5] Noguerales et al. Binary Discrete Diffusion: Candidate-Set Diffusion for Logic and Planning Tasks. Preprint.

Reviewer_WZUfOpenReview ↗

my concerns have been resolved

Thanks to the author's rebuttal. My concerns have been resolved. I will keep my score 3, weak accept.

Reviewer_HabvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Simplex Diffusion Models (SDMs), which model categorical data via continuous probability vectors on the simplex rather than discrete tokens. Using a Dirichlet-based framework with flexible concentration schedules, SDMs achieve closed-form reverse transitions, while bypassing the ODE integration needed for Dirichlet Flow Matching (DFM). The authors also connect their approach to discrete and Gaussian diffusion, introduce a distribution-matching distillation method, and demonstrate strong empirical performance across datasets.

Critical strengths
  • [S1] Elegant, flexible formulation: Modeling directly on the simplex with closed-form reverse transitions is a clean alternative to DFM’s ODE-based sampling. The bridge construction is well-motivated, separating control over the mean and concentration adds useful flexibility. The theoretical results appear largely correct (although I couldn't rigorously go over the appendices).

  • [S2] Strong empirical results and honest ablations: The TinyGSM code-generation gains over masked/uniform diffusion and the few-step distilled performance are compelling. I also appreciate the extensive ablations (on churn, schedules, self-conditioning, and representations) and I appreciate the being transparent about some negative findings, including the limitations of OpenWebText evaluation metrics.

Critical weaknesses
  • [W1] Disconnect between core motivation and best empirical setting: The paper argues that simplex states preserve uncertainty across denoising steps, avoiding the information collapse of discrete diffusion. Yet, the best results consistently rely on maximum churn (κ=1\kappa=1), where the sampler draws a discrete clean token and independently re-noises it. This collapses the predicted distribution to a categorical sample and discards the previous simplex state. As noted in Appendix M.1, this κ=1\kappa=1 case closely mirrors Boget & Kalousis (2026). Because the more general state-dependent transitions (κ<1\kappa < 1) actually perform worse, it is hard to credit the gains to the new bridge or uncertainty preservation rather than the forward process, input representation, or schedule.
  • [W2] Some important experimental and implementation details weaken the comparisons. My biggest concern is the DFM baseline. App O3 describes using 48-node Gauss–Legendre quadrature for evaluating its velocity coefficient at large vocabulary sizes. This appears numerically unreliable. For example, at N=49k, t=1, and b=0.02, directly applying the stated quadrature gives approximately 8.77×10−168.77\times10^{-16}, whereas more accurate integration gives 1.32×10−41.32\times10^{-4}. This is a very large discrepancy and raises concerns about the reported DFM performance, although I couldnt verify whether the actual implementation contains additional numerical safeguards. There are also concerns about the distillation evaluation. App P3 explicitly mentions selecting hyperparameters using the GSM8K test set, without a separate validation split. Moreover, the comparison with IDLM uses different teachers, so it is difficult to attribute the improvement specifically to the distillation method. Finally, the claimed exact gradients through Gamma sampling (Appendix J) need clarification, since differentiating through the described rejection sampler does not automatically give an unbiased pathwise gradient. These issues are potentially fixable, but they affect the reliability of some central claims.
Questions for authors
  1. Can you clarify the uncertainty preservation interpretation for κ=1\kappa=1? Since this is consistently the best-performing configuration, what benefit is actually obtained from carrying a continuous simplex state across steps, compared to independent categorical resampling as in Boget & Kalousis (2026)?
  2. Does your DFM code directly apply the 48-node quadrature from App O3 at 50k vocabulary sizes, or did you use log-space transformations/other safeguards to prevent underflow?
  3. In App J, how are gradients computed through the Gamma rejection sampler during distillation? Is a cusotom or implicit reparameterization gradient used?
  4. How robust are the distillation improvements without selecting hyperparameters on the GSM8K test set? Also, how much of the improvement over IDLM can be attributed to the distillation method rather than the stronger teacher?
2 comments

We are pleased that the reviewer found our formulation elegant and flexible, and the experiments strong and honest (including regarding negative findings).

Important: We apologize for not stating which churn κ\kappa we use on OpenWebText (OWT). Unlike on Sudoku and TinyGSM, κ=1\kappa=1 hurts sample quality on OWT, consistent with prior work on predictor-corrector samplers [1]. A tuned schedule κt\kappa_t could outperform κ=0\kappa=0, but [1] shows that such tuning is costly, so we use κ=0\kappa=0 for simplicity. In Section 5.3, we now state "We train a 12-layer DiT with hidden dimension 768 and 12 attention heads (167.9M parameters) for 1M steps with a batch size of 512. In this scenario, setting κ=0\kappa=0 worked best."

With κ=1\kappa=1, the state is not preserved. What does the simplex state bring compared to Boget & Kalousis (2026)?

We thank the reviewer for this question. The sampler of Boget & Kalousis (2026) is the κ=1\kappa=1 case of our backward bridge, in which case the next state depends only on the latest prediction x^0\hat{x}_0 (Prop. 3.3). With κ<1\kappa<1, the next state also depends on the current state PtP_t. Importantly, with κ=0\kappa=0 and ct=ε/(1−αt)c_t = \varepsilon/(1-\alpha_t), PtP_t is a random convex combination of the initial noise and all past draws of x^0\hat{x}_0 (Sec. 4.1).

The best churn depends on the task. On Sudoku and TinyGSM, κ=1\kappa=1 performs best, consistent with prior work on logic tasks: [2] report that "re-corruption alone markedly improves solve rates on logic tasks", and in our baselines, pure predictor-corrector sampling improves UDMs from 33.7% to 46.3% on TinyGSM (T=0.1T=0.1, 512 NFEs; Table 30). On OpenWebText, we use κ=0\kappa=0, which carries the state across steps. We apologize again for omitting this in the original submission.

Beyond the sampler, our Dirichlet path decouples the evolution of the concentration and mean. It contains the path of Boget & Kalousis (2026) as a special case, and recovers discrete and Gaussian diffusion in the low- and high-concentration limits, respectively.

Does the DFM baseline apply the 48-node quadrature directly at 49k tokens, or does it use numerical safeguards?

We are using numerical safeguards. We are now explaining our procedure in appendix O.4:

  • "For b≳0.01b \gtrsim 0.01, we numerically integrate this integral using 48-node Gauss-Legendre quadrature instead of the numerical gradient approach. For numerical integration, we make the substitution (1−u)N−2=e−w(1-u)^{N-2} = e^{-w}   ⟹  \implies u=1−e−wN−2u = 1 - e^{\frac{-w}{N-2}} and integrate over the range w∈[0,50]w \in [0, 50]. This prevents standard quadrature on [0,1][0,1] from stepping over the narrow region near zero where (1−u)N−2(1-u)^{N-2} is concentrated before decaying to zero."

How robust are the distillation results without selecting hyperparameters on the GSM8K test set?

We now select the configuration with the highest accuracy on the GSM8K training set at 8 NFEs, and report its test accuracy across NFEs: it solves 32.7% of the test problems with 8 NFEs and 39.7% with 64 NFEs (updated App. P.3, Table 7), more than IDLM with 128 NFEs (21.4%; Table 7).

IDLM uses a different teacher. Do the gains come from the distillation algorithm or from the stronger teacher?

We thank the reviewer for this question. IDLM distills discrete diffusion teachers with a lower accuracy than our SDM teacher, so our comparison indeed cannot isolate the impact of the distillation algorithm. We did not mean to imply that IDLM is a weaker method. IDLM was unstable in our early experiments, hence we used a variant of the distribution-matching objective, described in App. J that does not backpropagate through the Gamma sampler (see our answer below). IDLM may reach a similar accuracy with more tuning. We have amended the paragraph "Distillation and Auto-Guidance" (Sec. 5.2) accordingly.

More broadly, our paper does not aim to provide a comprehensive study of distillation methods for SDMs. We provide one efficient distillation method for SDMs, and leave a comprehensive study of distillation for simplex-based models to future work.

Does differentiating through the Gamma rejection sampler give an unbiased pathwise gradient (App. J)?

We thank the reviewer for this question. We agree that differentiating directly through an accept-reject sampler does not give unbiased gradients. However, the algorithm behind all our reported distillation results does not differentiate through the sampler. We apologize for this confusion and have clarified our manuscript in this regard. We have also added a discussion regarding unbiased gradients for the Marsaglia-Tsang algorithm. To make this precise, we distinguish three generator updates for SDMs (App. J.2).

1. IDLM. The IDLM gradient has two terms. The direct term, DG⊤(log⁡D−log⁡Dteach)(Pt)\mathrm{D}G^\top (\log D - \log D_{\mathrm{teach}})(P_t), flows through the generator output P0sP_0^s. The indirect term flows through the noisy state Pt=Ft(P0s,V)P_t = F_t(P_0^s, V) and therefore contains the Jacobian DFt(P0s,V)\mathrm{D}F_t(P_0^s, V) of the Dirichlet corruption. This Jacobian depends on the Gamma draws, so it must be computed through the sampler.

2. IDLM with the direct gradient only, the update used in practice. We stop the gradient through PtP_t, which removes the indirect term exactly. The generator then receives gradient only through P0sP_0^s, and the Gamma sampler is never differentiated. All our reported distillation results use it, so they are unaffected by the bias raised by the reviewer. As discussed in App. J.2, it is not the exact gradient of the KL objective.

3. Simplex DMD. The exact gradient of the KL objective (Prop. J.1) is atDG⊤DFt⊤(Δ/Pt)a_t \mathrm{D}G^\top \mathrm{D}F_t^\top (\Delta / P_t), where Δ\Delta is the difference of the two denoisers evaluated at PtP_t. It flows through DFt\mathrm{D}F_t, so it is exact only with an unbiased pathwise derivative of the sampler, for instance implicit reparameterization [5].

We have corrected App. J.2, which incorrectly stated that DFt\mathrm{D}F_t can be computed exactly by automatic differentiation. We have also added the bias analysis and its figure, and we now state in App. J and P.3 that the reported results use IDLM with direct gradients.

[1] Deschenaux et al. The Diffusion Duality, Chapter II: Ψ-Samplers. ICLR, 2026.

[2] Noguerales et al. Binary Discrete Diffusion: Candidate-Set Diffusion for Logic and Planning Tasks. Preprint.

[3] Hoogeboom et al. Beyond Single Tokens: Distilling Discrete Diffusion Models via Discrete MMD. arXiv:2603.20155, 2026.

[4] Naesseth et al. Reparameterization Gradients through Acceptance-Rejection Sampling Algorithms. AISTATS, 2017.

[5] Figurnov et al. Implicit Reparameterization Gradients. NeurIPS, 2018.

Reviewer_LrCeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents simplex diffusion models (SDMs), a diffusion model that bridges discrete and continuous diffusion by diffusing directly on the probability simplex over discrete categories. SDM introduces novelty over existing methods via a different parameterization that allows for closed form reverse transitions and a flexible sampler that allows for differently levels of stochasticity during generation. SDMs are shown to outperform or to be competitive with discrete diffusion baselines on small text modeling and coding datasets.

Critical strengths

The paper proposes an elegant framework for diffusion on the probability simplex that unifies discrete and continuous diffusion. This framework makes some novel contributions over the existing work (DFM; Stark et al., 2024), and demonstrates that it is competitive with or outperforms state of the art methods on relevant datasets. In particular, the new method made substantial improvements over the existing DFM method on larger scale datasets with bigger vocabulary sizes.

Critical weaknesses

The main weakness of the paper is the potential concern over novelty and/or framing, as well as the strengths of some results.

  • The experiment section reports that the best results are achieved with κ=1\kappa=1, which is a regime in which the entire state is resampled rather than refined with a delta. This undermines the narrative that SDMs "carry uncertainty across denoising steps," since a smaller κ\kappa would keep more information about the current state and yet does not work better in practice.
  • The novelty claim of the unifying perspective is questionable / misleading. The contribution section says "we show that SDMs unify discrete and continuous diffusions." The conclusion on line 483-484 "SDMs suggest that there is a false dichotomy between discrete and continuous diffusion language models" also seems to suggest that SDM is the first to establish the unifying perspective, while Chandra et al. (2026) already states this perspective (in the paper title...!). The authors discuss this previous work at length in the appendix, but the framing in the main paper can be potentially misleading.
  • The dependence on self-conditioning further weaken the main claim. The introduction explicitly claims that categorical diffusion models suffer from information collapse and therefore need to rely on self-conditioning (SC). SDMs are proposed to address the information collapse problem. However, it is clear that SC still helps improve the performance, and that on sudoku SDM without SC is not on par with other methods that use it. In my opinion these results weaken the central claim that SDMs are effective because they address the information collapse problem.
Questions for authors
  • Could the authors clarify: 1) why SDMs work best with κ=1\kappa=1 2) the influence of SC on SDM results and how these are related to the information collapse problem?
  • Could the authors provide the expectation + SC results for TinyGSM or provide reasons for the omission?
  • Section O.2 reports that the expectation input adds ~29% FLOPs per forward pass on TinyGSM, and Section N.3 carefully matches training FLOPs for self-conditioning, but no corresponding adjustment appears to be made for the expectation input. Could the authors report the main comparison in Figure 1 at matched training and sampling FLOPs?
2 comments

We are glad that the reviewer LrCe found our framework elegant in unifying discrete and continuous diffusion on the simplex.

Important: We apologize for not stating which churn κ\kappa we use on OpenWebText (OWT). Unlike on Sudoku and TinyGSM, κ=1\kappa=1 hurts sample quality on OWT, consistent with prior work on predictor-corrector samplers [1]. A tuned schedule κt\kappa_t could outperform κ=0\kappa=0, but [1] shows that such tuning is costly, so we use κ=0\kappa=0 for simplicity. In Section 5.3, we now state "We train a 12-layer DiT with hidden dimension 768 and 12 attention heads (167.9M parameters) for 1M steps with a batch size of 512. In this scenario, setting κ=0\kappa=0 worked best."

Best results are achieved with κ=1\kappa=1, which re-samples the state. Since κ=1\kappa=1 works well, is the uncertainty-preservation framing misleading?

We thank the reviewer for this comment and clarify below which regimes and tasks support the uncertainty-preservation interpretation of the SDM sampling algorithm:

  • With κ=1\kappa=1, the bridge depends only on the sampled x^0\hat{x}_0 (Prop. 3.3) and not on the previous state PtP_t. Memory and uncertainty preservation is a property of the bridge when κ<1\kappa<1, which may or may not be important depending on the task. We emphasize that in some of the workloads we consider, such as OWT text generation, κ<1\kappa < 1 yields the best results, thereby supporting our uncertainty-preservation motivation.

  • Indeed with κ=1\kappa=1 only the information flowing through x^0\hat{x}_0 and SC is retained. On logic tasks, renoising (κ=1\kappa=1) at each step appears to be the strategy that works best across many different methods. For instance, TinyGSM, memoryless prediction and renoising (κ=1\kappa=1) improves the accuracy of the baselines (35.3% -> 46.4% for UDMs with T=0.1, 512 NFEs). Consistently, κ=1\kappa=1 works best for SDMs. This is consistent with prior works such as [2], in which authors state "Interestingly, re-corruption alone markedly improves solve rates on logic tasks", see also App. N.2 for an ablation where we show that pure Predictor-Corrector works best for all discrete diffusion baselines. Conversely, we found that language generation tasks such as OWT benefit from κ=0\kappa=0 and the additional memory it brings.

  • Prop. 4.1 shows that with κ=0\kappa=0 and ct=ε/(1−αt)c_t = \varepsilon/(1-\alpha_t), the state PtP_t is a random convex combination of the initial draw P1P_1 and all past draws of x^0\hat{x}_0 (see Fig. 5). Such combination of past states is what we meant to emphasize. In contrast, discrete diffusion models keep at most one past draw per position (unless they use SC). In that sense, using κ<1\kappa < 1 corresponds to an intrinsic self-conditioning property of SDMs. We have now updated the main manuscript to emphasize this point (see Section 4.1 on information collapse).

The contributions and the conclusion suggest that SDMs are the first to unify discrete and continuous diffusion, while Chandra et al. (2026) already do.

We did not mean to claim to be first: Chandra et al. (2026) do unify discrete, Gaussian and simplicial diffusion. In order to clarify our contribution we have added a sentence in the introduction to emphasize that SDMs provide one mechanism to connect discrete and Gaussian diffusion models and that we provide a detailed comparison with Chandra et al. (2026) in the appendix.

We meant to connect discrete diffusion to our simplex formulation (a Dirichlet path rather than a Wright-Fisher mixture; App. L.3), and have revised the contributions and conclusion to cite and contrast with Chandra et al. in the main body. We would like to highlight however that the link between continuous and discrete models is established through very different mechanisms. Indeed, in Chandra et al., the connection between continuous and discrete models is established through a particle limit whereas in our case the connection is established through the concentration parameter of a Dirichlet path.

Why is there no SDM with the Expected Embedding and SC on TinyGSM?

The main reason why we do not consider Expected Embedding and SC on TinyGSM is for computational fairness reasons:

  • On Sudoku (14 tokens), computing the Expected Embedding is cheap, so we can combine it with SC directly (Table 1).
  • On TinyGSM (49k tokens), it adds 29% to the FLOPs (App. O.2). We therefore combine SC with the Argmax input to match the training and sampling FLOPs of discrete diffusion with SC.

The Expected Embedding adds 29% FLOPs per forward pass, but unlike SC, its training steps are not reduced. How does Fig. 1 compare at matched training and sampling FLOPs?

Matched training and sampling FLOPs: with the Argmax + SC input, SDMs cost as much as discrete diffusion with SC, in both training and sampling. At 512 NFEs and T=0.1T=0.1, they reach 53.6%, vs. 46.6% for MDM + SC (Fig. 1).

Matched sampling FLOPs: we added a version of Fig. 1 with inference FLOPs on the x-axis, and explain how we count the FLOPs in the new App. O.3. With the expected embedding, SDMs at 512 NFEs (89 TFLOPs) outperform the best baseline at 1k NFEs (139 TFLOPs): 50.0% vs. 47.8% at T=0.1T=0.1, and 46.6% vs. 40.1% at T=1T=1.

The 29% overhead of the expected embedding is large at our scale, but it shrinks with model size. With the Llama 3.1 architecture and vocabulary (~128k tokens), the overhead is 6.1% at 8B, 1.4% at 70B, and 0.5% at 405B (App. O.3).

SC still improves SDMs. Does the benefit of SC weaken the claim that SDMs mitigate information collapse?

We would like to thank the reviewer for this insightful remark and we would like to take the opportunity to clarify our statements.

  • First, our main claim is that the SDM bridge can keep a memory of past predictions, which discrete diffusion cannot, not that SC cannot help. Indeed, the effects are not yet fully understood and its memory preserving effect is only one part of the explanation of the success of self-conditioning.
  • Second, we are confident in our claim as it seems that SC helps SDMs the least:
    • +10.7% on Sudoku (vs. +15.7% for UDMs and +25.3% for MDMs at 180 NFEs; Table 1)
    • +4.6% on TinyGSM (vs. +5.2% for UDMs and +12.5% for MDMs at 512 NFEs, T=0.1T=0.1; Tables 30 and 34).
Reviewer_jTNcOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces Simplex Diffusion Models (SDMs), a generative framework for discrete data that performs diffusion directly on the probability simplex, so intermediate states retain distributions over categories rather than collapsing to sampled discrete tokens. The main contributions are: (1) a Dirichlet-based forward process with closed-form reverse transitions and a DDIM-like sampler whose stochasticity is controlled by a churn parameter; (2) theoretical results connecting SDMs to both discrete and continuous diffusion, including discrete diffusion as a high-temperature limit and deterministic/Gaussian behavior at low temperature; and (3) a simple cross-entropy training objective that allows uncertainty to persist across denoising steps without requiring self-conditioning.

Validation spans Sudoku, code generation, language modeling, and molecular generation. On TinyGSM, SDMs outperform strong masked and uniform diffusion baselines, reaching 49.0% accuracy without self-conditioning and 57.0% with self-conditioning at higher sampling budgets; an 8-step distilled SDM achieves 32.1% on GSM8K versus 21.4% for a 128-step distilled discrete-diffusion baseline. On Sudoku, SDMs reach 99.1%, close to the best discrete model at 99.3%. On OpenWebText, they are competitive with strong diffusion baselines in the GenPPL–entropy tradeoff, while the authors also show that sampling heuristics substantially affect this benchmark. Overall, the experiments support both the practical competitiveness and the proposed theoretical interpretation of SDMs, though the authors note that evaluation is limited to relatively small models.

Critical strengths
  1. Principled and novel formulation. SDMs preserve a distribution over discrete categories throughout denoising, directly addressing information collapse while retaining a simple cross-entropy training objective and closed-form reverse transitions. The accompanying theory also connects simplex diffusion to both discrete and continuous diffusion limits. This is consequential because it is not just an empirical variant; it offers a coherent framework with clear conceptual and mathematical justification.

  2. Broad and convincing empirical validation. The method is evaluated across Sudoku, code generation, language modeling, and molecular generation, with strong results against masked, uniform, and continuous-diffusion baselines. Particularly compelling are the TinyGSM/GSM8K results, where SDMs outperform stronger discrete-diffusion baselines even with substantially fewer sampling steps after distillation. This breadth makes the contribution more credible and suggests the method generalizes beyond a single benchmark.

Critical weaknesses
  1. Limited evidence at meaningful model scale. The experiments are restricted to relatively small models, and the authors explicitly acknowledge that it remains unknown whether the conclusions hold for large-scale language generation. This is consequential because scalability is central to the practical significance of a new diffusion framework for text; without larger-scale evidence, it is difficult to know whether the reported advantages survive realistic vocabulary, model-size, and generation-cost regimes.

  2. Some empirical gains are entangled with substantial sampling and tuning choices. On TinyGSM, performance depends strongly on adaptive time schedules and churn, while on OpenWebText the strongest results require several logit-shaping interventions that also substantially improve competing models. This makes it harder to isolate how much of the empirical improvement is attributable specifically to the simplex formulation rather than to inference-time engineering. That attribution issue weakens the strength of the empirical case for the core method.

Questions for authors
  1. Scalability: The current experiments use relatively small models, and the paper notes that large-scale viability remains open. Do the authors have evidence—empirical or computational—indicating that SDM’s advantages and sampling efficiency persist as model size, vocabulary size, and sequence length increase? This would materially affect my assessment of the method’s practical significance.

  2. Attribution of empirical gains: Several of the strongest results rely on adaptive time schedules, churn, self-conditioning, distillation, or logit-shaping interventions. Can the authors more clearly quantify how much of the improvement comes from the simplex formulation itself versus these auxiliary sampling/training choices, ideally under matched settings across baselines? This would materially affect how strongly the experiments support the core methodological claim.

3 comments

We appreciate that the reviewer found our experiments support both the practical competitiveness and theoretical foundations of SDMs. We are pleased to see that the reviewer found our draft clear and easy to follow, and that it offered a substantially deeper understanding of the problem.

Evidence at larger vocabulary and model sizes, and generation cost.

Indeed, we have not evaluated our method at large scale and acknowledge this limitation in the submitted manuscript. However, we would like to emphasize the following points which we think strengthen our claims even at a small scale:

  • Prior simplex models were mostly evaluated on tasks with reduced vocabularies, such as DNA [1, 2]. SDMs scale to the 49k-token vocabulary of TinyGSM, where they reach 50.0% at 512 NFEs, versus 13.6% for Dirichlet Flow Matching (Table 6).
  • We follow the standard setting of recent diffusion language models [3] (170M parameters), and spent the available compute on exploring the design space. Thus, we did not make claims about the performance at larger scales.
  • Like other diffusion language models, SDMs can only use a KV cache with block-wise generation [4, 5]. With full-sequence generation, diffusion language models are generally slower than AR.

How important is the backward bridge for performance? What is the impact of the churn on performance?

The backward bridge lets us tune the sampling stochasticity through the churn κ\kappa. We have identified two settings:

  • κ=1\kappa=1 for logic tasks. The churn κ\kappa has a large impact. On TinyGSM (T=0.1T=0.1, 512 NFEs), SDMs solve 19.6% of problems with κ=0\kappa=0, 44.7% with κ=0.2\kappa=0.2, and 50.0% with κ=1\kappa=1. Figs. 10-13 and Tables 24-26 and 32-35 show the same trend across schedules, temperatures and step budgets.
  • κ=0\kappa=0 for language modeling tasks. In particular, we now state that all results on OWT use κ=0\kappa=0 as this gave us the best settings.

More precisely, with κ=1\kappa=1, the bridge depends only on the sampled x^0\hat{x}_0 (Prop. 3.3). Memory and uncertainty preservation is a property of the bridge when κ<1\kappa<1, which may or may not be important depending on the task. We emphasize that in some of the workloads we consider, such as OWT text generation, κ<1\kappa < 1 yields the best results, thereby supporting our uncertainty-preservation motivation.

Finally, on TinyGSM (T=0.1T=0.1, 512 NFEs), Dirichlet Flow Matching (DFM), which integrates an ODE, solves 13.6% of problems. SDMs solve 19.6% with κ=0\kappa=0, 44.7% with κ=0.2\kappa=0.2, and 50.0% with κ=1\kappa=1 (Tables 6, 34).

How important is the adaptive time schedule for performance?

Adaptive time sampling during training and adaptive time schedule greatly improves accuracy of SDMs and baselines on logic tasks. More precisely:

  • Adaptive time samplers are important for continuous diffusion language models [3, 6, 7, 8], and help SDMs too. On TinyGSM (T=0.1T=0.1, 512 NFEs), the adaptive sampling grid improves the accuracy of SDMs from 41.2% -> 50.0% (Table 34).
  • For discrete diffusion, the adaptive time sampler improves MDM + SC slightly (44.8% -> 46.6%) but hurts UDM + PC (46.3% -> 40.9%; Table 30).

What is the impact of SC on performance?

We do not claim that SDMs can entirely discard SC. In fact, we show that SDMs can still benefit from SC in specific settings. Our main contribution is to show that some of the benefits of SC can be obtained by choosing the simplex representation in order to prevent the information collapse of discrete representation. In particular:

  • Our headline TinyGSM result (50.0%) feeds the expected embedding of PtP_t to the denoiser and does not use SC.
  • Since the expected embedding costs 29% more FLOPs per forward pass (App. O.2), we also train SDMs with the Argmax + SC input, whose denoiser matches that of discrete diffusion with SC, such as loopholing [9]. Note that the cost of the expected embedding denoiser input method is mitigated at scale, as the FLOPs of the rest of the network grow larger (see the new App. O.3). For example, at the 70B scale, with the Llama 3 architecture, the overhead is of 1.4%.
  • Finally, it seems that SC helps SDMs the least when compared with other discrete diffusion baselines:
    • +10.7% on Sudoku (vs. +15.7% for UDMs and +25.3% for MDMs at 180 NFEs; Table 1)
    • +4.6% on TinyGSM (vs. +5.2% for UDMs and +12.5% for MDMs at 512 NFEs, T=0.1T=0.1; Tables 30 and 34).

Impact of Logit shaping on OWT: how much comes from the formulation vs from the inference-time interventions?

On OWT, most of the gain comes from the logit shaping interventions. We consider those logit shaping interventions to be one of the contributions of our paper as they allow us to get close to validation data GenPPL/Entropy for all diffusion baselines and to go beyond that for the AR baseline.

With no intervention, at T=1T=1 and 64 / 1024 steps,

  • SDMs (Expected Embedding) 215.0 (H1=5.59H_1=5.59) / 125.4 (H1=5.51H_1=5.51),
  • SDMs (Argmax+SC) 107.1 (H1=5.52H_1=5.52) / 95.1 (H1=5.49H_1=5.49),
  • MDMs 131.4 (H1=5.68H_1=5.68) / 98.9 (H1=5.62H_1=5.62),
  • UDMs 78.8 (H1=5.52H_1=5.52) / 74.8 (H1=5.52H_1=5.52),

With the nucleus sampling, temperature annealing and sequence frequency penalty, MDMs reach 16.9 and UDMs 17.3 at H1≥5.46H_1 \geq 5.46 (Table 10). The local frequency penalty, which only helps SDMs, brings SDMs to 17.0 (Table 10). We therefore only claim that SDMs are competitive on OWT. After our interventions the AR Pareto frontier scores beyond real validation data. We interpret this result as a limitation of Pareto frontier on OWT as a meaningful benchmark for text generation.

[1] Stark et al. Dirichlet Flow Matching with Applications to DNA Sequence Design. ICML, 2024.

[2] Davis et al. Fisher Flow Matching for Generative Modeling over Discrete Data. NeurIPS, 2024.

[3] Deschenaux and Gulcehre. Language Modeling with Hyperspherical Flows. TMLR, 2026.

[4] Han et al. SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control. ACL, 2023.

[5] Arriola et al. Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models. ICLR, 2025.

[6] Dieleman et al. Continuous Diffusion for Categorical Data. arXiv:2211.15089, 2022.

[7] Lee et al. Flow Map Language Models: One-step Language Modeling via Continuous Denoising. NeurIPS, 2026.

[8] Raya et al. Noise Scheduling as Information-Guided Allocation in Diffusion Training. ICML, 2026.

[9] Jo et al. Loopholing Discrete Diffusion: Deterministic Bypass of the Sampling Wall. ICLR, 2026.

I thank the authors for adding this additional context. I remain clearly accepting this paper.

Reviewer_4metOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Simplex Diffusion Models for categorical generation using Dirichlet probability paths on the simplex, aiming to mitigate information collapse. The authors derive a DDIM-like family of conditional backward bridges with tunable stochasticity and cross-entropy training. Experiments cover Sudoku, code generation after training on TinyGSM, OpenWebText, and molecule generation, with additional results on few-step distillation.

Critical strengths
  • The explicit conditional backward bridges for flexible Dirichlet paths constitute a substantive contribution. They support sampling without numerical ODE integration and allow independent control of mean and concentration.

  • The paper provides a thorough empirical evaluation that demonstrates the practical potential of simplex diffusion models. The experiments are well designed and covered diverse generation tasks, offering valuable insights into the behavior and effectiveness of the proposed framework.

Critical weaknesses
  • At kappa=1, Algorithm 2 samples a clean categorical token and redraws the next simplex state independently of the previous state conditional on that token. Without SC, information from the previous predictive distribution therefore passes through a categorical bottleneck. The strongest low-temperature Argmax variant additionally uses SC. In this configuration, the process can be represented as categorical denoising and re-corruption augmented by SC memory. These results do not establish that retaining the continuous simplex state across steps explains the improvement. The paper should distinguish the benefits of its corruption path, denoiser inputs, and SC from those of the proposed state-preserving bridge.

  • Appendix P.3 explicitly selects the distilled configuration using GSM8K test performance after sweeping 36 configurations and multiple churn settings. Consequently, the reported 32.1% is a best-of-sweep result rather than an independently evaluated generalization estimate. Figure 19 similarly selects calibration weights on evaluation splits. The fact that all 36 configurations at kappa=1 achieve 27.4–32.1% is encouraging and suggests the improvement is not confined to one configuration, but it does not remove selection bias. Independent evaluation after fixing the settings is needed to substantiate the headline estimates.

Questions for authors
  1. In the Argmax + SC setting with kappa=1, what additional cross-step information does the simplex state retain beyond the categorical input and SC hidden state? Please clarify how this configuration supports the uncertainty-preservation explanation, distinguishing it from the expectation-input and kappa<1 settings.
  2. Can you provide distillation and language-understanding results with configurations and calibration weights selected on a separate validation set, followed by evaluation on examples not used for selection? Please specify the selection and evaluation protocol.
  3. Section H.3 shows that fixed positive churn does not admit a continuous time Markov limit, yet the best results use κ = 1. What does increasing the number of sampling steps approximate in this regime? Does the sampling distribution converge as the time grid is refined?

+) I am open to raising my score if the author can address these questions particularly by clarifying the role of simplex state information preservation and providing independently evaluated results that support the main empirical claims

7 comments

We appreciate the reviewer's comment highlighting our substantial contribution of the backward bridge, our comparison against strong baselines, and extensive ablations. We address the reviewer concerns and questions below.

Important: We apologize for not stating which churn κ\kappa we use on OpenWebText (OWT). Unlike on Sudoku and TinyGSM, κ=1\kappa=1 hurts sample quality on OWT, consistent with prior work on predictor-corrector samplers [1]. A tuned schedule κt\kappa_t could outperform κ=0\kappa=0, but [1] shows that such tuning is costly, so we use κ=0\kappa=0 for simplicity. In Section 5.3, we now state "We train a 12-layer DiT with hidden dimension 768 and 12 attention heads (167.9M parameters) for 1M steps with a batch size of 512. In this scenario, setting κ=0\kappa=0 worked best."

In the Argmax + SC setting with κ=1\kappa=1, what cross-step information does the simplex state retain beyond the categorical input and the SC input, and how does this support the uncertainty-preservation explanation?

We thank the reviewer for this comment and clarify below which regimes and tasks support the uncertainty-preservation interpretation of the SDM sampling algorithm:

  • With κ=1\kappa=1, the bridge depends only on the sampled x^0\hat{x}_0 (Prop. 3.3) and not on the previous state PtP_t. Memory and uncertainty preservation is a property of the bridge when κ<1\kappa<1, which may or may not be important depending on the task. We emphasize that in some of the workloads we consider, such as OWT text generation, κ<1\kappa < 1 yields the best results, thereby supporting our uncertainty-preservation motivation.

  • Indeed with κ=1\kappa=1 only the information flowing through x^0\hat{x}_0 and SC is retained. On logic tasks, renoising (κ=1\kappa=1) at each step appears to be the strategy that works best across many different methods. For instance, TinyGSM, memoryless prediction and renoising (κ=1\kappa=1) improves the accuracy of the baselines (35.3% -> 46.4% for UDMs with T=0.1, 512 NFEs). Consistently, κ=1\kappa=1 works best for SDMs. This is consistent with prior works such as [2], in which authors state "Interestingly, re-corruption alone markedly improves solve rates on logic tasks", see also App. N.2 for an ablation where we show that pure Predictor-Corrector works best for all discrete diffusion baselines. Conversely, we found that language generation tasks such as OWT benefit from κ=0\kappa=0 and the additional memory it brings.

  • Prop. 4.1 shows that with κ=0\kappa=0 and ct=ε/(1−αt)c_t = \varepsilon/(1-\alpha_t), the state PtP_t is a random convex combination of the initial draw P1P_1 and all past draws of x^0\hat{x}_0 (see Fig. 5). Such combination of past states is what we meant to emphasize. In contrast, discrete diffusion models keep at most one past draw per position (unless they use SC). In that sense, using κ<1\kappa < 1 corresponds to an intrinsic self-conditioning property of SDMs. We have now updated the main manuscript to emphasize this point (see Section 4.1 on information collapse).

How does the Argmax + SC setting with κ=1\kappa=1 differ from the case κ<1\kappa<1?

With κ<1\kappa<1, the current state PtP_t depends on the previous ones (Prop. 4.1), even if the denoiser only sees the argmax. The Argmax input is a choice of denoiser architecture and does not change the mathematical framework. We have now clarified in the manuscript that Argmax is a denoiser input choice and not a modeling choice, see the revised Section 3.2.

The distillation hyperparameters are selected on the evaluation split

We now select the top variant on the GSM8K training set at 8 NFEs, and report its GSM8K test accuracy across NFEs (updated App. P.3, Table 8). This does not change our conclusions: the selected variant solves 32.7% of the test problems with 8 NFEs, more than IDLM with 128 NFEs (21.4%), and 39.7% with 64 NFEs (Table 7).

The calibration weights are selected on the evaluation splits.

We did not re-select ww on held-out data, because the accuracy is nearly flat around the best values. Additionally, with w=0w=0, i.e., without any tuning, SDMs match or outperform MDMs.

Compared with w=0w=0, the best ww improves PIQA by 0.3% (54.3% vs. 54.6%) and HellaSwag by 2.3% (35.6% vs. 37.9%), and mainly helps on ARC-Easy (36.3% vs. 43.2%; Table 12). Any w∈[0.5,0.8]w \in [0.5, 0.8] gives 42.6% to 43.2% on ARC-Easy, and any w∈[0.3,0.7]w \in [0.3, 0.7] gives 37.2% to 37.8% on HellaSwag (Fig. 21, Fig. 19 in the submission).

Section H.3 shows that fixed positive churn does not admit a continuous time Markov limit, yet the best results use κ = 1. What does increasing the number of sampling steps approximate in this regime? Does the sampling distribution converge as the time grid is refined?

We have shown that when κ>0\kappa > 0, there is no stochastically continuous Markov process in the small stepsize limit, but this does not rule out the existence of a non-stochastically continuous dynamics. In fact, the same question holds for remasking in classical Masked Diffusion Models, as in [3]. We leave this theoretical question for future work.

[1] The Diffusion Duality, Chapter II: Ψ-Samplers, Deschenaux et al., ICLR 2026.

[2] Binary Discrete Diffusion: Candidate-Set Diffusion for Logic and Planning Tasks, Noguerales et al., Preprint.

[3] Remasking Discrete Diffusion Models with Inference-Time Scaling, Wang et al., NeurIPS 2025

Thank you for the response and the revision. Some of my concerns remain. At κ = 1, as you confirm, nothing is carried across steps beyond x̂₀ and SC, so the headline TinyGSM and distillation results are not evidence for uncertainty preservation. The contributions and conclusion still claim it unconditionally, supported only by an unquantified sentence that κ = 0 worked best on OWT. The calibration weights are still selected on the evaluation splits, and Q3 is unanswered.

Overall, the backward bridge is novel and the framework may prove impactful, but the claims still exceed the evidence.

Comments on the paper

3 comments
~Yoann_Boget1OpenReview ↗

Connection to earlier work

Dear Authors,

Congratulations on your article.

I want to draw your attention to a connection to our work (Unrestrained Simplex Denoising [UNSID], Boget and Kalousis, NeurIPS 2026) that remains unclear in your text. We believe that when the churn parameter is set to κ=1\kappa = 1, which gives the best experimental results in your evaluation, your approach becomes equivalent to ours. The community would benefit from commenting on this relationship and making the potential equivalence explicit.

Also, our method does not integrate an ODE or SDE, as described in the current draft; instead, it draws a clean sample and re-noises it independently.

Your work thoughtfully generalizes and extends our framework, and I think it is an important contribution to the field. Please consider these remarks as constructive suggestions to help situate your work more clearly within the existing literature. Thank you for this stimulating work.

Dear Yoann Boget,

Congratulations on your NeurIPS publication. You are correct that UNSID [1] does not integrate an ODE / SDE; instead, it draws a clean sample and re-noises it independently. We will amend the introduction and cite the NeurIPS version. With κ=1\kappa=1, our bridge indeed only depends on P0P_0 (as stated in Prop. 3.3): the reverse transition draws x^0\hat{x}_0 from the denoiser and re-noises it independently, which is the UNSID sampler applied to our Dirichlet path. With the DFM / UNSID path given below, the two samplers coincide. This correspondence is noted in App. M.1, where we state "Note that very recently Boget & Kalousis (2026) have proposed the same forward process as Stark et al. (2024). They introduce an inference procedure which, like ours, allows for stochasticity but assumes ps∣0,t(Ps∣P0,Pt)=ps∣0(Ps∣P0)p_{s|0,t}(P_s|P_0, P_t) = p_{s|0}(P_s|P_0) as in star-shaped diffusion models (Okhotin et al., 2023) (corresponding to κ=1\kappa=1 in our case).". However, it is not highlighted in the main text; we will make the equivalence explicit in the introduction.

Regarding experimental results, the best κ\kappa is task dependent. For Sudoku and TinyGSM, κ=1\kappa=1 performs best for SDMs (Tables 24 to 26 and 32 to 35, Figures 10 to 13); in this case, the sampler only depends on P0P_0, as in UNSID. The fact that re-noising seems to be a strong sampler choice for these logical tasks is widely acknowledged in the literature. For instance, the authors of [2] state that "re-corruption alone markedly improves solve rates on logic tasks". Consistently, in our baselines (Tables 27 to 31), the pure Predictor-Corrector sampler of App. N.2, which also uses the bridge ps∣0,t=ps∣0p_{s|0,t} = p_{s|0}, is the best sampler without Self-Conditioning for uniform diffusion on both tasks and for masked diffusion on Sudoku.

On OpenWebText, we use κ=0\kappa=0, following prior work [3], which observed that using many corrector steps can hurt sample quality. Therefore, our main theoretical and methodological contribution, which exhibits a continuum of possible reverse transitions from κ=0\kappa=0 to κ=1\kappa=1 (UNSID), comes with practical applications. We omitted from the main text that we use κ=0\kappa=0 on OWT, and we will state it in the experimental section.

Finally, an important difference with UNSID is that our Dirichlet path is more general, as we decouple the concentration (t↦ctt \mapsto c_t), the mean weight (t↦αtt \mapsto \alpha_t), and the prior distribution π\pi. Specifically, if we denote by t↦htt \mapsto h_t the schedule in DFM [4] and UNSID, and by NN the vocabulary size, we recover the DFM [4] / UNSID path with αt=htht+N\alpha_t = \frac{h_t}{h_t + N}, ct=N1−αtc_t = \frac{N}{1-\alpha_t} and π=1N\pi=\frac{1}{N}. This generality also allows us to recover both discrete and Gaussian diffusion models in the low and high concentration limits, respectively.

Thank you for your questions and the kind words.

Best,

Authors

[1] Unrestrained Simplex Denoising for Discrete Data. A Non-Markovian Approach Applied to Graph Generation, Boget et al., NeurIPS 2026

[2] Binary Discrete Diffusion: Candidate-Set Diffusion for Logic and Planning Tasks, Noguerales et al., Preprint.

[3] The Diffusion Duality, Chapter II: Ψ-Samplers, Deschenaux et al., ICLR 2026.

[4] Dirichlet Flow Matching with Applications to DNA Sequence Design, Stark et al., ICML 2024.

AuthorsOpenReview ↗

General Response

Dear Area Chair and Reviewers,

We are grateful for the detailed feedback and questions. Reviewers found the backward bridge substantive and well-developed (4met, WZUf), the framework elegant (LrCe, Habv), the experiments strong and honest (Habv), and the paper clear (jTNc). Below, we summarize the main questions, our answers, and the changes to the manuscript, which are marked in red in the PDF file.

Churn κ\kappa and uncertainty preservation (4met, LrCe, Habv, WZUf, and the public comment by Yoann Boget). At κ=1\kappa=1, the next state depends only on the sampled x^0\hat{x}_0 (Prop. 3.3). The best churn depends on the task: κ=1\kappa=1 performs best on Sudoku and TinyGSM, consistent with prior work on logic tasks, while we use κ=0\kappa=0 on OpenWebText. In the manuscript, we:

  • moved proposition E.1 to the main text, proving that, with κ=0\kappa=0, the simplex state is a random convex combination of the initial noise and past predictions (Sec. 4.1), and revised the abstract accordingly,
  • stated that we use κ=0\kappa=0 on OpenWebText (Sec. 5.3), which we forgot at submission time,
  • clarified in the introduction that Boget & Kalousis (2026) sample without integrating an ODE or SDE,
  • clarified that the Argmax input is a design choice of the denoiser, independent of the backward bridge (Sec. 3.2).

Compute (LrCe, WZUf, jTNc). The expected embedding adds 29% FLOPs per forward pass at our scale. At the same time, with the Argmax + SC input, SDMs match the training and sampling FLOPs of discrete diffusion with SC, and reach 53.6% on TinyGSM, vs. 46.6% for MDM + SC (T=0.1T=0.1, 512 NFEs). We created App. O.3, to explain how we count FLOPs and shows that the overhead shrinks with model size (1.4% at 70B parameters). Finally, we created a version of Fig. 1 with the inference FLOPs on the x-axis instead of the NFEs (Fig. 9).

Evaluation protocol (4met, Habv, WZUf). We now select the distillation hyperparameters on the GSM8K training set (our models are trained on TinyGSM), and report the accuracy on the GSM8K test set (updated App. P.3). This does not change our conclusions: the selected model solves 32.7% of the test problems with 8 NFEs, more than IDLM with 128 NFEs (21.4%).

Distillation (Habv, WZUf). IDLM distills weaker teachers, hence our comparison does not isolate the distillation algorithm. We do not claim that IDLM is a weaker method, although we found it unstable in preliminary experiments. We have amended Sec. 5.2 accordingly. Our reported distillation results use DMD rather than Simplex DMD and do not backpropagate through the Gamma sampler, as we now state in App. P.3. App. J.1 now quantifies the bias of differentiating through the sampler, which only concerns Simplex DMD.

Relation to Chandra et al. (2026) (LrCe). We do not claim to be the first to connect discrete and continuous diffusion. We now cite and contrast against Chandra et al. (2026) in the introduction and the conclusion.

Dirichlet Flow Matching baseline (Habv). We describe our numerical safeguards in App. O.4.

Best,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.