acceptodds
Under review as a conference paper at ICLR 2027

Simplex Relaxation for Discrete Diffusion

10 views

Abstract

Discrete diffusion models for categorical generation are defined by a corruption kernel, which determines the intermediate state space and the associated reverse prediction problem. Within this family, uniform diffusion has been extensively developed, with recent work connecting its categorical corruption process to continuous representations and dynamics. Motivated by this view, we ask whether uniform discrete diffusion can be augmented with an explicit continuous state while leaving its categorical corruption process unchanged. We introduce Simplax, an exact Dirichlet–categorical augmentation that couples each corrupted categorical state with an auxiliary simplex-valued variable while preserving the uniform diffusion process as its categorical marginal. This augmentation yields a tractable Rao–Blackwellized reverse-bridge objective and a stochastic reverse sampler, while retaining the corrupted categorical state as the denoiser input. Empirically, Simplax improves the generative perplexity–entropy tradeoff on unconditional OpenWebText generation. On Sudoku, a model trained exclusively on -clue puzzles achieves the highest accuracy among the compared methods across all evaluated clue densities, including the minimum uniquely solvable -clue regime, and also achieves the highest validity in unconditional generation.

Primary area:
generative models
Keywords:
discrete diffusion, simplex relaxation, categorical diffusion, Dirichlet distribution, generative modeling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_KF8LOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Simplax, an augmentation of uniform discrete diffusion that attaches an auxiliary Dirchlet-distributed simplex variable wtw_t. Marginalizing out wtw_t exactly recovers the original uniform discrete diffusion process, and the authors use this structure to derive a Rao–Blackwellized training objective and a wtw_t-based reverse sampler. Since the denoiser still takes the categorical token ztz_t as input, the efficiency of token-based denoising is preserved. Empirically, Simplax improves generative perplexity on OpenWebText language modeling and improves both solving accuracy and unconditional validity on Sudoku.

Critical strengths
  • The proposed method is clean and novel. Attaching wtw_t preserves the marginal distribution of ztz_t exactly, and the resulting reverse conditionals are available in closed form.
  • The method is practical. By keeping categorical token as inputs, it is faster than continuous-state methods, with only about 10–20% overhead relative to UDLM.
  • The empirical results are strong. Simplax shows good results on language modeling, and on Sudoku task.
Critical weaknesses
  • The Rao–Blackwellization applies only to the auxiliary anchor z~t\tilde z_t drawn from wtw_t. However, relative to UDLM, Simplax actually introduces an additional source of randomness, wtw_t. Nevertheless, the paper provides no analysis or discussion about why attaching wtw_t leads to improved performance.
  • The numerical precision of categorical sampling should be clearly specified for all baselines in the language modeling experiments, and ideally unified to FP64. Prior work has shown that categorical sampling in float32 can be inaccurate and can artificially lower entropy and generative perplexity [1]. Appendix A.4 states that LangFlow uses an FP32 sampler and S-FLM an FP64 sampler, but the precision used for the other methods, including Simplax’s categorical sampling, is not stated.
  • The paper attributes the benefit of UDLM initialization to a staged strategy similar in spirit to Duo (in line 323). However, Duo’s curriculum uses a simplex relaxation to first solve an easier, soft problem and then a harder one (soft to hard), which is why it can be viewed as curriculum learning. In contrast, the staged Simplax training first learns with a hard anchor ztz_t (UDLM) and then with a soft, weighted-sum anchor (Simplax), i.e., from hard to soft. Since the direction is reversed, the analogy to Duo does not seem appropriate.
  • In Fig. 4, Simplax’s generative perplexity appears to degrade more quickly as generative entropy increases. This behavior is not discussed.
  • In several figures (e.g., Fig. 4 and Fig. 3 etc), multiple graphs share similar line styles and are distinguished mainly by color, which makes them difficult to read for color-blind readers.

[1] Zheng, Kaiwen, et al. "Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling." International Conference on Learning Representations. Vol. 2025. 2025.

Questions for authors
  • Given that Rao–Blackwellization only removes the variance from sampling z~t\tilde z_t, while wtw_t adds an extra random variable relative to UDLM, what is the mechanism behind the performance improvement?

  • Duo’s curriculum proceeds from soft to hard, whereas the staged Simplax (UDLM init) training proceeds from a hard anchor (UDLM) to a soft weighted-sum anchor (Simplax). Could the authors explain why this reversed schedule is described as similar to Duo, and what actually explains the benefit of UDLM initialization?

  • What numerical precision is used for categorical sampling for each method? Do the results still hold even when all methods use float64 sampling?

  • Why does Simplax’s generative perplexity degrade more quickly at higher entropy than the baselines?

I think the method is simple, novel, and effective. If these questions and weaknesses are adequately addressed, I’m willing to change my assessment.

10 comments
AuthorsOpenReview ↗

Comment 1/4 to Reviewer KF8L

We thank the reviewer for the thoughtful questions on the mechanism behind the improvement, the sampling precision, the staged training, and the shape of the frontiers. We address each point below and indicate the corresponding changes in the revised paper.

W1. The Rao–Blackwellization applies only to the auxiliary anchor drawn from wtw_t. However, relative to UDLM, Simplax actually introduces an additional source of randomness, wtw_t. Nevertheless, the paper provides no analysis or discussion about why attaching wtw_t leads to improved performance.

Q1. Given that Rao–Blackwellization only removes the variance from sampling z~t\tilde{z}_t, while wtw_t adds an extra random variable relative to UDLM, what is the mechanism behind the performance improvement?

Thank you for this question on the mechanism behind the improvement. The revision adds the analysis below to Sec. 3.2 and App. C.

wtw_t changes the objective that the network minimizes. The network input is ztz_t, and the objective is linear in wtw_t, so the randomness of wtw_t only adds noise to an unbiased estimate of a fixed objective. Averaging over wtw_t gives the new Eq. 17:

Ewt[L]=11+ηLUDLM(anchor zt)+η1+ηEz~t∼q(zt∣x)[LUDLM(anchor z~t)].\mathbb{E}_{w_t}\big[\mathcal{L}\big] = \frac{1}{1+\eta}\mathcal{L}_{\mathrm{UDLM}}(\text{anchor } z_t) + \frac{\eta}{1+\eta}\mathbb{E}_{\tilde{z}_t \sim q(z_t \mid x)}\big[\mathcal{L}_{\mathrm{UDLM}}(\text{anchor } \tilde{z}_t)\big].

The first term is the UDLM objective, and the second term is new. The Rao–Blackwellization only makes this objective computable from a single network evaluation.

The new term supervises the transitions that the sampler adds. By Eq. 10, a Simplax reverse step draws an anchor from Cat(wt)\mathrm{Cat}(w_t) and takes a uniform-diffusion reverse step from it. Since wt=(zt+ηp^t)/(1+η)w_t = (z_t + \eta \hat{p}_t)/(1+\eta), with p^t\hat{p}_t the previous clean-token prediction mapped to time tt, the anchor differs from the input token ztz_t with probability up to η/(1+η)\eta/(1+\eta), while the network still sees ztz_t. The UDLM term evaluates the prediction only through the transition out of the input token, whereas the cross-anchor term evaluates the same prediction through the transition out of an independently corrupted anchor, which is exactly this input–anchor mismatch.

The gain requires both the training term and the sampler. In the new Tab. 18, 50k-step checkpoints sampled at NFE 1024 give the following Gen. PPL under GPT-2 Large:

Trainingηsample=0\eta_{\mathrm{sample}} = 0 (UDLM transitions)ηsample=10−2\eta_{\mathrm{sample}} = 10^{-2}
ηtrain=0\eta_{\mathrm{train}} = 0 (UDLM objective)79.087.2
ηtrain=10−2\eta_{\mathrm{train}} = 10^{-2} (Simplax objective)74.158.2

Without the cross-anchor term, the added transitions raise Gen. PPL by 10% (from 79.0 to 87.2); with it, they lower Gen. PPL by 21% (from 74.1 to 58.2). The added transitions by themselves therefore do not improve generation at NFE 1024.

Off-input anchors keep revising tokens at high NFE. As the step size shrinks, a UDLM transition keeps the current token with probability approaching one, whereas an off-input anchor still replaces it. On the main checkpoint, Simplax (UDLM init) in Tab. 1, the sampler takes 0.12, 0.97, and 7.7 off-input anchors per token at NFE 16, 128, and 1024 (new Tab. 19). At NFE 1024, it changes each token 14.2 times, compared with 6.6 under UDLM transitions with the same predictions, and 71% of the final tokens were last set by an off-input anchor. Since the denoiser conditions on a context that keeps developing during generation, these updates let earlier token choices be revised with later predictions, which UDLM transitions rarely do.

The gain comes from revisions late in generation. In the new Tab. 20, we use off-input anchors in only one half of the trajectory, on the same checkpoint at NFE 1024 with float64 sampling and entropy matched to 5.44. Gen. PPL:

Off-input anchorsEntropyGPT-2 LargeLlama-2 7B
None5.44057.4030.40
Only for t>0.5t > 0.55.44861.9232.37
Only for t≤0.5t \le 0.55.44245.7525.68
Whole trajectory5.43845.8425.74

Using them only in the late half reproduces the full gain, whereas using them only in the early half, when the context is still largely corrupted, does not help. The benefit therefore comes from revisions made once most of the context has been denoised.

We have added the decomposition as the new paragraph "What the objective trains" in Sec. 3.2 at L.243, with the proof of Eq. 17 in the new App. E.1, and the analysis above as the new paragraph "Why off-input anchors help" in App. C at L.1752, with the counts in the new Tab. 19 and the restricted-sampling results in the new Tab. 20.

AuthorsOpenReview ↗

Comment 2/4 to Reviewer KF8L

W2. The numerical precision of categorical sampling should be clearly specified for all baselines in the language modeling experiments, and ideally unified to FP64. Prior work has shown that categorical sampling in float32 can be inaccurate and can artificially lower entropy and generative perplexity [1]. Appendix A.4 states that LangFlow uses an FP32 sampler and S-FLM an FP64 sampler, but the precision used for the other methods, including Simplax's categorical sampling, is not stated.

Q3. What numerical precision is used for categorical sampling for each method? Do the results still hold even when all methods use float64 sampling?

Thank you for raising the precision of categorical sampling. In short, float32 sampling did not systematically favor Simplax, and the conclusions of Tab. 1 hold when every method uses float64 sampling. We give the details below, and the revision states the precision of every method.

Only Simplax, FLM, and LangFlow used float32. In the submitted results, AR, CANDI, UDLM, MDLM, and Duo use float64 categorical sampling at every step. FLM and LangFlow draw tokens with a categorical sample only once, at the final step, and this draw was in float32. S-FLM decodes its final state deterministically by argmax, so it involves no categorical sampling; its FP64 sampler refers to the ODE integration. Both Simplax models used float32 categorical sampling at every step. We now state this in App. A.4, L.1056.

We re-evaluated Simplax, Simplax (UDLM init), FLM, and LangFlow with float64 categorical sampling: at every step for both Simplax models, and at the final step for FLM and LangFlow. In this re-evaluation, every method of Tab. 1 therefore uses float64 sampling. We follow the protocol of Tab. 1, with 1024 samples and the same scorer. For Simplax we additionally refined the temperature on a 0.003 grid so that the generated entropy matches the data entropy to within ±0.003\pm 0.003 nats.

Each cell shows the submitted float32 value → the value with float64 sampling. Time is seconds per batch of 16 sequences of length 1024, measured with the protocol of Tab. 1; the two Simplax models share the same sampler and therefore the same time.

MethodNFEENTGPT-2 LLlama-2Time (s)
FLM165.58† → 5.58†259.5 → 258.0139.3 → 138.80.896 → 0.908
1285.42 → 5.42112.0 → 110.866.5 → 66.77.12 → 7.12
10245.45 → 5.45125.6 → 125.274.4 → 74.457.0 → 56.9
LangFlow165.42 → 5.42115.1 → 114.659.9 → 59.21.15 → 1.17
1285.42 → 5.4460.2 → 60.830.0 → 30.09.38 → 9.38
10245.41† → 5.43†68.3 → 64.428.2 → 27.775.3 → 75.2
Simplax165.46 → 5.4498.7 → 92.954.2 → 51.60.814 → 0.859
1285.46 → 5.4463.4 → 59.835.0 → 33.86.09 → 6.46
10245.45 → 5.4452.6 → 51.429.8 → 29.248.4 → 51.3
Simplax (UDLM init)165.42 → 5.4481.5 → 84.945.2 → 46.40.814 → 0.859
1285.45 → 5.4456.5 → 54.331.1 → 30.36.09 → 6.46
10245.42 → 5.4442.8 → 45.824.4 → 25.748.4 → 51.3

† The temperature sweep does not reach the data entropy 5.44.

With float64 for all methods, Tab. 1's conclusions hold. Float64 sampling raised the Gen. PPL of Simplax (UDLM init) at NFE 16 and 1024 (e.g., from 42.8 to 45.8 at NFE 1024 under GPT-2 Large) and lowered that of Simplax at every NFE, so float32 did not favor Simplax in one direction; these changes likely reflect the entropy at which each point was matched rather than the precision, as explained below. With float64 sampling, Simplax (UDLM init) is the best diffusion model on GPT-2 Large at every NFE and on Llama-2 at NFE 16 and 1024; at NFE 128 under GPT-2 Large, its float64 value, 54.3, is now also below UDLM (55.0), which was ahead of the submitted 56.5. LangFlow remains best on Llama-2 at NFE 128. Gen. PPL rises steeply with entropy near 5.44, by about 2–3% per 0.01 nats in our sweeps, and the submitted points of both Simplax models lay between 5.42 and 5.46. Float64 sampling adds about 6% to the sampling time of Simplax; at NFE 1024 it takes 51.3 s and remains faster than FLM, LangFlow, and S-FLM, which takes 80.5 s. Our other replies quote the submitted Tab. 1; we will update all results in the paper to float64 sampling in the camera-ready version.

AuthorsOpenReview ↗

Comment 3/4 to Reviewer KF8L

W3. The paper attributes the benefit of UDLM initialization to a staged strategy similar in spirit to Duo (in line 323). However, Duo's curriculum uses a simplex relaxation to first solve an easier, soft problem and then a harder one (soft to hard), which is why it can be viewed as curriculum learning. In contrast, the staged Simplax training first learns with a hard anchor (UDLM) and then with a soft, weighted-sum anchor (Simplax), i.e., from hard to soft. Since the direction is reversed, the analogy to Duo does not seem appropriate.

Q2a. Duo's curriculum proceeds from soft to hard, whereas the staged Simplax (UDLM init) training proceeds from a hard anchor (UDLM) to a soft weighted-sum anchor (Simplax). Could the authors explain why this reversed schedule is described as similar to Duo, ...

Thank you for the careful comparison with Duo. We agree that the submitted sentence did not state what the analogy refers to, and we have clarified it in the revision.

The comparison concerns how a fixed budget is allocated. In both Duo and Simplax, the same network is trained with one objective for part of the budget and then switched to the final objective. We do not claim that the UDLM stage is easier or that the schedule proceeds from soft to hard. The revised sentence at L.377 now states this directly: Duo likewise switches the training objective partway through a fixed budget.

Soft and hard mean different things in Duo and Simplax. The soft/hard distinction in Duo refers to the denoiser input, a tempered softmax of the Gaussian latent versus its argmax (Sahoo et al., 2025, Sec. 4.1), whereas in Simplax the denoiser input is the hard token ztz_t in both stages and the weighted-sum anchor appears only inside the loss.

Q2b. ... and what actually explains the benefit of UDLM initialization?

Thank you for this question on the benefit of UDLM initialization. The revision adds the explanation below to Sec. 3.2 and App. C.

The two terms of the objective converge at different speeds. By the new Eq. 17 in Sec. 3.2, it decomposes exactly as 11+ηLUDLM+η1+ηLcross\frac{1}{1+\eta}\mathcal{L}_{\mathrm{UDLM}} + \frac{\eta}{1+\eta}\mathcal{L}_{\mathrm{cross}}, and exactness keeps q(zt∣x)q(z_t \mid x) identical across stages, so a UDLM checkpoint has already optimized the dominant term. Measuring both terms on held-out data shows that they converge on different time scales. Starting from UDLM 800k, 200k Simplax steps improve Lcross\mathcal{L}_{\mathrm{cross}} by 0.350±0.0150.350 \pm 0.015 while changing LUDLM\mathcal{L}_{\mathrm{UDLM}} by only 0.001±0.0020.001 \pm 0.002, and 94% of this gain is reached within 50k steps; LUDLM\mathcal{L}_{\mathrm{UDLM}}, in contrast, still improves between 400k and 1M UDLM steps. The staged schedule therefore spends most of the budget on the slowly converging main term and needs only a short Simplax stage for the cross-anchor term, consistent with Tab. 21. We have added this explanation to Sec. 4.2 at L.377, which corresponds to L.323 of the submission, and the full analysis to App. C as the new paragraph "Decomposition of the training objective" with Fig. 18 and Tab. 22.

Reviewer_ftgCOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies discrete diffusion with a uniform corruption process. Specifically, the submission aims to augment the discrete diffusion process with an explicit continuous state, while leaving the discrete process unchanged.

Critical strengths
  • Introducing a new factorization into discrete diffusion is timely.
Critical weaknesses
  • The paper focuses solely on discrete diffusion with a uniform corruption process which limits its impact.
  • According to my opinion, the paper lacks a clear motivation. While the probabilistic framework can be introduced, the paper does not indicate why this framework is beneficial. E.g., the paper states: "we seek an exact augmentation ..." (l.79) but doesn't answer why this is beneficial. Similarly, it reads: "we consider the following joint factorization" (l.158) but doesn't answer why exactly this factorization. Throughout the paper, I find points are stated without providing a clear motivation. This makes the paper not very insightful because I'm essentially reading one statement after another without motivation and intuition. This is the reason why I rated the paper's informativeness as 1: the facts are well established and the motivation to use them in the provided context is not provided.
  • Similarly to the previous point, the authors introduce the continuous time limit but its motivation remains unclear.
  • Overall, I also think the writing of the theoretical part could be improved significantly. It is not easy for a reader to verify some of the facts. E.g., people not intimately familiar with the literature may question Eq. (9).
  • Experimental validation should also show other techniques beyond discrete diffusion with a uniform corruption process. E.g., Tab. 1 shows autoregressive results but doesn't show discrete diffusion with other corruption processes.
  • On OpenWebText, the best results are achieved when using prior work (UDLM) for 800k steps and the proposed approach for 200k steps (Simplax (UDLM init))? The method on its own is competitive but not necessarily state-of-the-art on OpenWebText. This, in combination with the missing motivation, is the main reason why I rate this submission as a weak reject: unclear benefits and unclear motivation.
Questions for authors

see above

9 comments
AuthorsOpenReview ↗

Comment 1/3 to Reviewer ftgC

We thank the reviewer for the detailed feedback on the motivation and presentation of the construction. Below we explain why each part of the construction is needed, and we indicate where the revised paper now states this.

W1. The paper focuses solely on discrete diffusion with a uniform corruption process which limits its impact.

Thank you for this comment on the scope of the paper. We respectfully disagree with the assertion that focusing on uniform diffusion limits the impact of our work.

Uniform diffusion is a promising and growing family. As discussed in Sec. 1 at L.039, uniform diffusion corrupts tokens within the original vocabulary instead of toward an absorbing mask (Schiff et al., 2025), so every token stays revisable during sampling. Recent work exploits this for self-correction (von Rütte et al., 2025) and few-step generation (Sahoo et al., 2025), and connects the corruption process to an underlying Gaussian diffusion (Sahoo et al., 2025), which is the view Simplax builds on. Uniform diffusion remains competitive on likelihood-based benchmarks at 1.7B parameters (Sahoo et al., 2026) and already scales to 10B and 25B parameters (von Rütte et al., 2026; Google DeepMind, 2026). Since any UDLM checkpoint is a valid Simplax model, Simplax can improve such models with a short additional training stage: in Tab. 21, replacing the last 100k–400k of the 1M UDLM steps with Simplax steps lowers NFE-1024 Gen. PPL under GPT-2 Large by 23–28% (from 59.2 to 42.8–45.6) under the same total budget.

W2. According to my opinion, the paper lacks a clear motivation. While the probabilistic framework can be introduced, the paper does not indicate why this framework is beneficial. E.g., the paper states: "we seek an exact augmentation ..." (l.79) but doesn't answer why this is beneficial.

Thank you for this comment on the motivation. The revised Sec. 3 now states it before the construction.

Our goal is to use a continuous simplex state while keeping discrete diffusion exact. Continuous diffusion language models carry the uncertainty among tokens in a continuous state, but feed it to the denoiser through a vocabulary-wide matrix product at every step. Discrete diffusion embeds a single token by lookup, but its state carries no such uncertainty. Uniform corruption already contains a simplex variable wtw_t: by Eq. 8, it is exactly a Dirichlet draw wt∼Dir(ηpt)w_t \sim \mathrm{Dir}(\eta p_t) followed by a categorical draw zt∼Cat(wt)z_t \sim \mathrm{Cat}(w_t). An exact augmentation lets the reverse process use wtw_t while the discrete process and the token input of the denoiser stay unchanged, which is why we seek it: an approximate one would change the discrete process.

Exactness gives three properties used in the revised paper.

  1. Compatibility (C1). Marginalizing the auxiliary variable recovers uniform diffusion exactly, so any UDLM checkpoint is a valid Simplax model. This enables the sampler swap in Tab. 6 and the staged training of Tab. 21, where UDLM training followed by a short Simplax stage improves over either objective alone at low and high NFE under the same total budget. An approximate relaxation would change the marginal and lose this.
  2. Efficiency is preserved (C2). Because the categorical marginal is unchanged, the denoiser still takes ztz_t via token lookup. Feeding the simplex point wtw_t instead requires a vocabulary-wide dense product wt⊤Ew_t^\top E, and Tab. 3 shows that it also gives worse generation quality.
  3. Tractability without approximation (C3). The reverse posterior of zsz_s given wtw_t, Eqs. 9–10, is categorical in closed form, which yields the exact Rao–Blackwellized objective of Sec. 3.2 and the exact reverse sampler of Sec. 3.3.

Using wtw_t improves generation through the reverse steps it adds. Because wtw_t carries the previous prediction, the Simplax sampler can take reverse steps from anchors other than the input token, and the new term of the objective trains exactly these steps (Eq. 17, "What the objective trains" at L.243). These steps keep revising tokens late in generation, once most of the context has been denoised, which accounts for the gain (Tabs. 18–20, "Why off-input anchors help" at L.1752; see also our reply to Reviewer KF8L, W1/Q1).

We have added these as explicit design requirements (C1)–(C3) at the start of Sec. 3 at L.162, together with the reasons for them, and Secs. 3.1–3.3 mark where each is satisfied or used at L.176, L.193, L.220, L.235, and L.304.

AuthorsOpenReview ↗

Comment 2/3 to Reviewer ftgC

W3a. Similarly, it reads: "we consider the following joint factorization" (l.158) but doesn't answer why exactly this factorization.

Thank you for this question on the factorization. The revised Sec. 3.1 now explains it.

The factorization follows from Dirichlet–categorical conjugacy. Read wtw_t as the latent probability vector from which the token ztz_t is drawn: placing the prior Dir(ηpt)\mathrm{Dir}(\eta p_t) on it and observing one draw ztz_t gives the posterior Dir(ηpt+zt)\mathrm{Dir}(\eta p_t + z_t), which is exactly Eq. 7. Each wuw_u depends only on zuz_u and xx, so the categorical chain x→zs→ztx \to z_s \to z_t is untouched and marginalizing ww recovers uniform diffusion exactly. Conjugacy then gives both the Dirichlet marginal q(wt∣x)=Dir(ηpt)q(w_t \mid x) = \mathrm{Dir}(\eta p_t) and the trivial decoder q(zt∣wt)=Cat(wt)q(z_t \mid w_t) = \mathrm{Cat}(w_t), which together form Eq. 8. We have added this reading after Eq. 8 in Sec. 3.1 at L.192, together with a sentence at Eq. 6 at L.175 stating that the factorization keeps the discrete chain intact and lets each wtw_t depend only on ztz_t and xx.

W4. Similarly to the previous point, the authors introduce the continuous time limit but its motivation remains unclear.

Thank you for this comment on the continuous-time limit. The revised Sec. 3.2 now states its motivation.

It lets one trained model be sampled at any NFE. The limit removes the dependence of training on a fixed discretization, so that, as in UDLM and MDLM, one model can be sampled at any NFE; Fig. 4 evaluates the same model from NFE 8 to 1024.

It singles out the objective. App. G shows that the natural alternatives either vanish at O(Δ2)O(\Delta^2) or diverge, whereas ours admits a finite, nonzero limit, as summarized under "Alternative bridge objectives" in Sec. 3.2.

We have added this motivation at the start of the paragraph "Continuous-time limit" in Sec. 3.2 at L.252.

W5. Overall, I also think the writing of the theoretical part could be improved significantly. It is not easy for a reader to verify some of the facts. E.g., people not intimately familiar with the literature may question Eq. (9).

Thank you for this comment on the theoretical part.

Eq. 9 follows in three lines. We have added the derivation to Sec. 3.1 after Eq. 10 at L.199:

  • By Bayes' rule and Corollary 1 in App. D, q(zt=ek∣wt,x)∝pt,k Dir(wt;ηpt+ek)=wt,k Dir(wt;ηpt)q(z_t = e_k \mid w_t, x) \propto p_{t,k}\, \mathrm{Dir}(w_t; \eta p_t + e_k) = w_{t,k}\, \mathrm{Dir}(w_t; \eta p_t), so q(zt∣wt,x)=Cat(wt)q(z_t \mid w_t, x) = \mathrm{Cat}(w_t).
  • Since zs⊥wt∣(zt,x)z_s \perp w_t \mid (z_t, x) by the factorization in Eq. 6, q(zs∣wt,x)=∑kwt,k q(zs∣zt=ek,x)q(z_s \mid w_t, x) = \sum_k w_{t,k}\, q(z_s \mid z_t = e_k, x).
  • Substituting Eq. 4, q(zs∣ek,x)=[αt∣sek+(1−αt∣s)πk1]⊙ps/pt,kq(z_s \mid e_k, x) = [\alpha_{t\mid s} e_k + (1 - \alpha_{t\mid s}) \pi_k \mathbf{1}] \odot p_s / p_{t,k}, and summing over kk gives ps⊙[αt∣s(wt⊘pt)+(1−αt∣s)⟨wt,π⊘pt⟩1]p_s \odot [\alpha_{t\mid s} (w_t \oslash p_t) + (1 - \alpha_{t\mid s}) \langle w_t, \pi \oslash p_t \rangle \mathbf{1}], which is Eq. 10.

Each equation now points to its proof. The full proofs of Eqs. 8, 9, and 11 are items 1–2, 3, and 5 of Proposition 1 in App. D, and each of these equations in Sec. 3.1 now points to its item at L.189, L.194, and L.203.

W6. Experimental validation should also show other techniques beyond discrete diffusion with a uniform corruption process. E.g., Tab. 1 shows autoregressive results but doesn't show discrete diffusion with other corruption processes.

Thank you for this suggestion. We believe there is a slight misunderstanding about the baselines in Tab. 1.

Tab. 1 already includes other corruption processes. These are MDLM for absorbing corruption, the continuous diffusion and flow models FLM, LangFlow, and S-FLM, and the hybrid discrete–continuous model CANDI. Tab. 2 on Sudoku also includes MDLM, FLM, and S-FLM.

AuthorsOpenReview ↗

Comment 3/3 to Reviewer ftgC

W7. On OpenWebText, the best results are achieved when using prior work (UDLM) for 800k steps and the proposed approach for 200k steps (Simplax (UDLM init))? The method on its own is competitive but not necessarily state-of-the-art on OpenWebText. This, in combination with the missing motivation, is the main reason why I rate this submission as a weak reject: unclear benefits and unclear motivation.

Thank you for this comment on the OpenWebText results. We respectfully disagree that the benefits are unclear.

Simplax (UDLM init) uses the same budget. As stated in Sec. 4.1, every OpenWebText model, including all baselines, uses the same 179M-parameter backbone and a total budget of 1M optimizer steps, and UDLM, Simplax, and Simplax (UDLM init) also see the same number of training tokens. Replacing the last 200k of the 1M UDLM steps with Simplax steps lowers Gen. PPL at NFE 1024 under GPT-2 Large by 28% (from 59.2 to 42.8). In Tab. 21, all four allocations from 600k/400k to 900k/100k UDLM/Simplax steps give 42.8–45.6 at NFE 1024, improving over both UDLM 1M and Simplax 1M. Any UDLM checkpoint is thus a valid Simplax initialization that a short Simplax stage improves. In the new paragraph "Decomposition of the training objective" in App. C at L.1890, we show that this stage learns the cross-anchor term of Eq. 17 within about 50k steps, while the slowly converging UDLM term is already trained by the UDLM stage.

Simplax from scratch already surpasses existing methods in several settings. It lowers the Gen. PPL of UDLM by 11% (from 59.2 to 52.6) at NFE 1024 under GPT-2 Large. At NFE 16, UDLM stays at entropy 5.53 or above at every temperature of the sweep, whereas Simplax reaches the data entropy. Among all diffusion baselines in Tab. 1, it ranks first in two of the six settings, second in two, and third in two.

The motivation is in our replies to W2 and W3b/W8.

W3b. Throughout the paper, I find points are stated without providing a clear motivation.

W8. This makes the paper not very insightful because I'm essentially reading one statement after another without motivation and intuition. This is the reason why I rated the paper's informativeness as 1: the facts are well established and the motivation to use them in the provided context is not provided.

Thank you for this comment on the motivation and informativeness. We respectfully disagree that the paper only restates established facts, and we have updated our revision to make the motivation explicit.

The contribution is an exact continuous augmentation of discrete diffusion. Dirichlet–categorical conjugacy is classical, and we use it as an ingredient, but conjugacy alone does not give this. The construction must recover uniform diffusion exactly when the auxiliary variable is marginalized (C1), keep the denoiser's token input and its input distribution (C2), and keep the reverse posterior of zsz_s given wtw_t in closed form (C3), and the resulting objective must have the well-defined continuous-time limit derived in Sec. 3.2. The revised Sec. 3 now states (C1)–(C3) up front at L.162, marks where each requirement is satisfied or used in Secs. 3.1–3.3 at L.176, L.193, L.220, L.235, and L.304, and explains what the resulting objective trains beyond UDLM in the new paragraph "What the objective trains" in Sec. 3.2 at L.243.

References

  • DiffusionGemma Team, Google DeepMind. DiffusionGemma Technical Report. arXiv:2608.00146, 2026.

  • Sahoo, S. S., Deschenaux, J., Gokaslan, A., Wang, G., Chiu, J. T., and Kuleshov, V. The Diffusion Duality. ICML, 2025.

  • Sahoo, S. S., Lemercier, J.-M., Yang, Z., Deschenaux, J., Liu, J., Thickstun, J., and Jukic, A. Scaling Beyond Masked Diffusion Language Models. ICML, 2026.

  • Schiff, Y., Sahoo, S. S., Phung, H., Wang, G., Boshar, S., Dalla-torre, H., de Almeida, B. P., Rush, A. M., Pierrot, T., and Kuleshov, V. Simple Guidance Mechanisms for Discrete Diffusion Models. ICLR, 2025.

  • von Rütte, D., Fluri, J., Ding, Y., Orvieto, A., Schölkopf, B., and Hofmann, T. Generalized Interpolating Discrete Diffusion. ICML, 2025.

  • von Rütte, D., Fluri, J., Pooladzandi, O., Schölkopf, B., Hofmann, T., and Orvieto, A. Scaling Behavior of Discrete Diffusion Language Models. ICLR, 2026.

Reviewer_oKoJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Simplax, a Dirichlet–categorical augmentation of discrete diffusion with a uniform corruption kernel. The method preserves the categorical corruption process while introducing an auxiliary simplex-valued variable into the training objective and reverse sampler. The authors evaluate it on text generation and Sudoku.

Critical strengths
  • The evaluation covers several practical dimensions, including generation quality, sampling speed, and memory usage, alongside theoretical analysis.

  • At larger NFEs, Simplax appears competitive with methods such as UDLM.

  • The Dirichlet–categorical construction is relatively simple and may be useful more broadly within uniform categorical diffusion.

Critical weaknesses
  • Compared with recent baselines such as LangFlow, Simplax’s empirical advantage appears limited. The contribution therefore seems somewhat incremental relative to UDLM.
  • The strongest reported results appear to come from continuing training from a UDLM checkpoint at a particular stage, while training Simplax from scratch does not clearly surpass existing methods. A comparison under matched total training compute would help isolate the benefit of the proposed method.
  • It remains unclear whether the gains reflect a substantive improvement or are sensitive to numerical choices and training details. Whether the same benefits persist with more advanced models and larger training scales also remains open.
  • The Dirichlet concentration η\eta adds a hyperparameter used in both training and reverse sampling. Its sensitivity, including possible interactions with sampling settings, warrants further study.
Questions for authors
  1. The comparison with ELF is appreciated. Why does ELF not reach the reported OpenWebText entropy level? Could differences in the training data, preprocessing, tokenization, or evaluation protocol explain this gap?

  2. Simplax shows a substantial advantage on Sudoku, but the source of this improvement is not sufficiently clear. Can the authors clarify whether it arises from the proposed method itself, training hyperparameters, or differences in how the baselines were reproduced?

  3. Does continuing training from a UDLM checkpoint also improve performance on Sudoku? How does it compare with training Simplax from scratch under the same total training budget?


I'm open to modify the scoring based on the discussion with authors.

9 comments
AuthorsOpenReview ↗

Comment 1/6 to Reviewer oKoJ

We thank the reviewer for the careful reading and for the concrete questions on the comparison with LangFlow and ELF, the training budget, the robustness of the gains, and the role of the concentration η\eta. We address each point below and indicate the corresponding changes in the revised paper.

W1a. ... The contribution therefore seems somewhat incremental relative to UDLM.

Thank you for this comment on the contribution relative to UDLM. We respectfully disagree.

Simplax uses a continuous state that UDLM discards. By Eq. 8, uniform corruption is exactly a two-stage process: a simplex point wt∼Dir(ηpt)w_t \sim \mathrm{Dir}(\eta p_t) is drawn given xx, and the token is drawn from it, zt∼Cat(wt)z_t \sim \mathrm{Cat}(w_t). UDLM models only ztz_t and marginalizes wtw_t out, discarding what wtw_t carries about xx beyond the token. Simplax keeps it: wtw_t enters the training target, which adds the cross-anchor term of the new Eq. 17, and the sampler carries it across steps, as described in Sec. 3.3. Since the state still contains ztz_t, the denoiser keeps its token input.

This improves on UDLM under the same training budget. In Tab. 1, under the same 1M-step budget, Simplax from scratch improves NFE-1024 Gen. PPL under GPT-2 Large from 59.2 to 52.6.

W1b. Compared with recent baselines such as LangFlow, Simplax's empirical advantage appears limited.

Thank you for this comment on the comparison with LangFlow. We respectfully disagree with this assessment.

Simplax (UDLM init) lowers the Gen. PPL of LangFlow by over 25% under GPT-2 Large. In Tab. 1, Simplax (UDLM init) lowers the Gen. PPL of LangFlow (Chen et al., 2026) by 29% and 37% at NFE 16 and 1024 under GPT-2 Large, and by 25% and 14% under Llama-2 7B, with the two within 6% at NFE 128. Simplax achieves this without self-conditioning (Chen et al., 2023), which LangFlow uses at extra training compute.

Simplax also samples faster. In Tab. 1, one batch at NFE 1024 takes 48.4 s for Simplax and 75.3 s for LangFlow. LangFlow carries the uncertainty among tokens in a continuous input, which requires a vocabulary-wide matrix product at every step. Simplax carries it in the exact simplex variable wtw_t of uniform corruption (Eq. 8), while the denoiser still embeds the token ztz_t by lookup.

AuthorsOpenReview ↗

Comment 2/6 to Reviewer oKoJ

W2. The strongest reported results appear to come from continuing training from a UDLM checkpoint at a particular stage, while training Simplax from scratch does not clearly surpass existing methods. A comparison under matched total training compute would help isolate the benefit of the proposed method.

Thank you for suggesting a comparison under matched training compute. We would first like to clarify that the training budget is already matched across all models.

All OpenWebText models use the same training budget. As stated in Sec. 4.1, every model, including both Simplax variants and all baselines, uses the same 179M-parameter backbone and a total budget of 1M optimizer steps. Simplax (UDLM init) is not trained longer: it uses 800k UDLM steps followed by 200k Simplax steps, 1M steps in total. Simplax, Simplax (UDLM init), and UDLM are trained in our codebase with batch size 512 and 1024-token sequences, so they also see the same number of training tokens.

The gain is not tied to a particular split. Tab. 21 in App. C compares four allocations of UDLM and Simplax steps: 600k/400k, 700k/300k, 800k/200k, and 900k/100k. All of them perform similarly and improve over Simplax 1M, and at NFE 16 and 1024 they also clearly improve over UDLM 1M. GPT-2 Large Gen. PPL at NFE 16 / 128 / 1024:

Training (1M steps total)NFE 16NFE 128NFE 1024
UDLM 1M186.0†55.059.2
Simplax 1M (from scratch)98.763.452.6
UDLM 600k → Simplax 400k84.252.645.6
UDLM 700k → Simplax 300k84.052.445.4
UDLM 800k → Simplax 200k81.556.542.8
UDLM 900k → Simplax 100k84.553.045.2

† The sweep does not reach the data entropy 5.44 ± 0.05, as marked in Tab. 21.

Simplax from scratch already surpasses existing methods in several settings. Under the same 1M-step budget, it lowers the Gen. PPL of UDLM, from which it differs only by wtw_t, by 11% (from 59.2 to 52.6) at NFE 1024 under GPT-2 Large. At NFE 16, UDLM stays at entropy 5.53 or above at every temperature of the sweep, whereas Simplax reaches the data entropy. Among all diffusion baselines in Tab. 1, it ranks first in two of the six settings, second in two, and third in two. UDLM initialization lowers its NFE-1024 Gen. PPL by a further 19% (from 52.6 to 42.8). This is because the two terms of the Simplax objective in Eq. 17 converge at different speeds: the UDLM term converges slowly and is already trained by the UDLM stage, whereas a short Simplax stage learns the cross-anchor term within about 50k steps. We show this in the new paragraph "Decomposition of the training objective" in App. C at L.1890, with Fig. 18 and Tab. 22, and summarize it in our reply to Reviewer KF8L (Q2b).

W3a. It remains unclear whether the gains reflect a substantive improvement or are sensitive to numerical choices and training details.

Thank you for raising the robustness of the gains. We respectfully disagree based on the ablations already reported in the paper.

The gains over UDLM are robust. We varied each numerical and training choice with everything else fixed:

  • Training stage. In Tab. 21, every allocation from 600k UDLM + 400k Simplax to 900k + 100k gives 42.8–45.6 at NFE 1024, versus 52.6 for Simplax from scratch and 59.2 for UDLM, so the gain is not tied to a particular stage.
  • Concentration η\eta. In Tab. 5, both η=10−3\eta = 10^{-3} and 10−210^{-2} improve over η=0\eta = 0 at NFE 1024, by 4% and 29% (from 78.7 to 75.4 and 56.2). The joint sweep over training and sampling η\eta is in our reply to W4.
  • Sampler. In Tab. 4, the stochastic and ww-deterministic samplers give similar results: 105.1 / 73.6 / 56.2 vs. 106.0 / 75.1 / 60.7 at NFE 16 / 128 / 1024.
  • Evaluation noise. The bootstrap standard errors in App. A.6 are about 1% of Gen. PPL, well below the gaps to UDLM at NFE 1024 under GPT-2 Large: 42.77±0.4342.77 \pm 0.43 for Simplax (UDLM init) and 52.55±0.4452.55 \pm 0.44 for Simplax, vs. 59.24±0.5759.24 \pm 0.57 for UDLM.
AuthorsOpenReview ↗

Comment 3/6 to Reviewer oKoJ

W3b. Whether the same benefits persist with more advanced models and larger training scales also remains open.

Thank you for raising the question of scale. We address the theory and the experiments separately, and we have updated our revision accordingly.

The theoretical contribution does not depend on model scale. The exact recovery of uniform diffusion as the categorical marginal, the closed-form reverse conditionals, the Rao–Blackwellized objective, and its continuous-time limit are properties of the construction and hold for any denoiser. In particular, any UDLM checkpoint, at any scale, is a valid Simplax model.

We follow the standard 179M setting of all compared methods. This allows a direct comparison with their released checkpoints. Experiments at larger scales are not feasible with our computational resources. Scaling Simplax to larger models and training budgets is a question for future work, and we have added it to the Limitation paragraph of Sec. 6 at L.536.

W4. The Dirichlet concentration adds a hyperparameter used in both training and reverse sampling. Its sensitivity, including possible interactions with sampling settings, warrants further study.

Thank you for suggesting a study of the sensitivity to η\eta. We ran this study by decoupling the training η\eta from the sampling η\eta. Each 50k-step checkpoint of Tab. 5, trained with ηtrain∈{0,10−3,10−2,10−1}\eta_{\mathrm{train}} \in \{0, 10^{-3}, 10^{-2}, 10^{-1}\}, is sampled with every ηsample\eta_{\mathrm{sample}} in the same set, using the ww-deterministic sampler with float64 sampling and a temperature tuned so that the entropy matches the data entropy, 5.44. Gen PPL under GPT-2 Large on 1,024 samples:

ηtrain\eta_{\mathrm{train}} \ ηsample\eta_{\mathrm{sample}}0010−310^{-3}10−210^{-2}10−110^{-1}
NFE 16
00104.7106.7110.8169.2
10−310^{-3}101.4104.8111.7164.2
10−210^{-2}100.4103.0106.8135.3
10−110^{-1}106.2111.6113.2133.6
NFE 128
0078.281.485.7142.3
10−310^{-3}79.178.979.2126.1
10−210^{-2}75.474.670.674.7
10−110^{-1}83.281.676.969.0
NFE 1024
0079.077.687.21273.3
10−310^{-3}80.983.988.71061.4
10−210^{-2}74.171.758.230.2†
10−110^{-1}81.574.262.333.2†

† The temperature sweep does not reach the data entropy 5.44. The best value in each NFE block, excluding †, is shown in bold, as in Tab. 18.

The same η\eta in training and sampling works best at NFE ≥128\ge 128. The paper uses the same η\eta for both. At NFE 1024, using 10−210^{-2} for both lowers Gen PPL by 21% (from 74.1 to 58.2) relative to sampling the same checkpoint with ηsample=0\eta_{\mathrm{sample}} = 0, and at NFE 128 the two lowest values, 70.6 and 69.0, lie where the training and sampling η\eta are equal. At NFE 16, ηsample=0\eta_{\mathrm{sample}} = 0 is best for every checkpoint.

Up to 10−210^{-2}, only a larger sampling η\eta hurts. With ηsample≤ηtrain≤10−2\eta_{\mathrm{sample}} \le \eta_{\mathrm{train}} \le 10^{-2}, every cell lies within 100–107 at NFE 16, 71–79 at NFE 128, and 58–84 at NFE 1024. Above the training η\eta, Gen PPL rises with the gap: two orders of magnitude above, it reaches 126–169 at NFE 16 and 128 and exceeds 1,000 at NFE 1024. The sampler then takes more reverse steps from anchors other than the input token, which such checkpoints have barely trained, since the cross-anchor term of Eq. 17 has weight ηtrain/(1+ηtrain)\eta_{\mathrm{train}}/(1+\eta_{\mathrm{train}}).

We have added this table to App. C as Tab. 18, with the discussion above as the new paragraph "Training versus sampling concentration" at L.1709.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.