- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Simplex Diffusion Models (SDMs), which model categorical data through continuous states on the probability simplex. The forward process uses Dirichlet distributions with separately controlled means and concentrations. The main methodological contribution is a compatible backward bridge with closed-form sampling and adjustable stochasticity, enabling a DDIM-like sampler trained using cross-entropy. The paper also establishes connections to categorical diffusion and deterministic interpolation through concentration limits. Experiments cover Sudoku, TinyGSM/GSM8K, OpenWebText, and molecular generation. The strongest empirical evidence comes from code-generation accuracy, with additional promising results from few-step distillation.
Critical strengths
-
A substantive and well-developed sampling construction. Proposition 3.3 provides a compatible Dirichlet backward bridge through thinning and innovation, with a tunable stochasticity parameter. This offers a concrete alternative to numerical ODE/SDE integration for simplex-based generative modeling. The concentration-limit analysis further clarifies how the construction relates to discrete and continuous representations.
-
Meaningful evaluation beyond unconditional text metrics. The functional evaluation on TinyGSM/GSM8K provides useful evidence of practical value: at 512 NFEs and temperature 0.1, the expected-embedding SDM without self-conditioning achieves 49.0% accuracy, compared with 45.8% for MDM with self-conditioning. The extensive baseline sweeps and input/schedule ablations strengthen this evidence. I also appreciate the explicit discussion of the limitations of GenPPL/entropy frontiers and the diversity reduction caused by distillation.
Critical weaknesses
-
The headline distillation result is selected on the test set. Section P.3 explicitly states that the reported 32.1% is selected from a hyperparameter sweep using the GSM8K test set, without a held-out validation split. The reported range across all 36 configurations is reassuring, but it does not make the selected result an unbiased estimate of generalization. Moreover, the IDLM results are taken from another paper with different teachers and distillation objectives, so the comparison does not isolate the benefit of the proposed simplex formulation.
-
The explanation based on preserving uncertainty is not established for the strongest configurations. These generally use κ=1, where Proposition 3.3 gives a transition independent of the previous state conditional on the sampled clean token. Algorithm 2 first samples that token categorically, so, without self-conditioning, information from the previous state passes through a discrete bottleneck. The Argmax input further restricts what the denoiser observes. This does not invalidate the sampler, but it leaves unclear whether the gains arise from persistent uncertainty, the corruption distribution, or the sampling schedule.
Questions for authors
-
Evaluation protocol: Can you report results with hyperparameters selected on a separate validation set and evaluated once on an untouched test set? Please also clarify how the non-distilled SDM and baseline configurations were selected, and provide uncertainty estimates for the principal TinyGSM comparisons.
-
Mechanistic clarification: How should the uncertainty-preservation claim be interpreted when κ=1? In particular, with Argmax inputs and no self-conditioning, can the effective sampler be represented as a discrete-state chain? Which existing results distinguish benefits from retaining the previous simplex state from benefits of the Dirichlet corruption and adaptive schedule?
-
Compute comparability: Section O.2 reports that expected embeddings add approximately 29% to forward-pass FLOPs. Does the training-budget matching account for this overhead, in addition to self-conditioning? Do the principal accuracy advantages persist at matched total compute or measured inference time?
3 comments
Does the training budget matching account for the 29% FLOPs overhead of the Expected Embedding?
With Expected Embedding, we did not account for the 29% training FLOPs overhead, and train for 250k steps, like the baselines. We chose so because the overhead shrinks with model size (see next response). With the Argmax + SC input, we match the training and sampling FLOPs of discrete diffusion with SC.
Do the accuracy advantages persist at matched training and inference compute?
Matched training and sampling FLOPs: with the Argmax + SC input, SDMs cost as much as discrete diffusion with SC, in both training and sampling. At 512 NFEs and , they reach 53.6%, vs. 46.6% for MDM + SC (Fig. 1).
Matched sampling FLOPs: we added a version of Fig. 1 with inference FLOPs on the x-axis, and explain how we count the FLOPs in the new App. O.3. With the expected embedding, SDMs at 512 NFEs (89 TFLOPs) outperform the best baseline at 1k NFEs (139 TFLOPs): 50.0% vs. 47.8% at , and 46.6% vs. 40.1% at .
The 29% overhead of the expected embedding is large at our scale, but it shrinks with model size. With the Llama 3.1 architecture and vocabulary (~128k tokens), the overhead is 6.1% at 8B, 1.4% at 70B, and 0.5% at 405B (App. O.3).
Which existing results distinguish the benefits of retaining the previous simplex state from those of the Dirichlet corruption and the adaptive schedule?
We thank the reviewer for this question. The TinyGSM results below (, 512 NFEs) compare matched configurations:
Retaining the state (sweep the churn): retaining the state does not help on TinyGSM, as the accuracy increases with : 19.6% with , 44.7% with , and 50.0% with (Table 34). Similarly, without SC, pure predictor-corrector sampling improves over ancestral sampling for discrete diffusion (UDM: 33.7% -> 46.3%; MDM: 32.6% -> 40.3%; Table 30).
Dirichlet corruption (): UDM + PC solves 40.9% of problems (46.4% with its best configuration; Table 30), vs. 43.7% for SDMs with the Argmax input and 50.0% with the expected embedding (Table 34).
Adaptive sampling grid (): the accuracy of SDMs increases from 41.2% with the linear grid to 50.0% with the adaptive grid (Table 34).
Finally, we emphasize that for OWT, we found that was the best churn hyperparameter and that choosing pure renoising was hurting performance.
[1] Deschenaux et al. The Diffusion Duality, Chapter II: Ψ-Samplers. ICLR, 2026.
[2] Sahoo et al. Simple and Effective Masked Diffusion Language Models. NeurIPS, 2024.
[3] Sahoo et al. The Diffusion Duality. ICML, 2025.
[4] Deschenaux and Gulcehre. Language Modeling with Hyperspherical Flows. TMLR, 2026.
[5] Noguerales et al. Binary Discrete Diffusion: Candidate-Set Diffusion for Logic and Planning Tasks. Preprint.
my concerns have been resolved
Thanks to the author's rebuttal. My concerns have been resolved. I will keep my score 3, weak accept.
We are pleased that the reviewer found our sampler substantive and well-developed, and appreciated our evaluation beyond unconditional text generation. We appreciate their acknowledgement of our extensive baseline sweeps.
Important: We apologize for not stating which churn κ we use on OpenWebText (OWT). Unlike on Sudoku and TinyGSM, κ=1 hurts sample quality on OWT, consistent with prior work on predictor-corrector samplers [1]. A tuned schedule κt could outperform κ=0, but [1] shows that such tuning is costly, so we use κ=0 for simplicity. In Section 5.3, we now state "We train a 12-layer DiT with hidden dimension 768 and 12 attention heads (167.9M parameters) for 1M steps with a batch size of 512. In this scenario, setting κ=0 worked best."
We now select the configuration with the highest accuracy on the GSM8K training set at 8 NFEs, and report its test accuracy across NFEs: it solves 32.7% of the test problems with 8 NFEs and 39.7% with 64 NFEs (updated App. P.3, Table 7), more than IDLM with 128 NFEs (21.4%; Table 7).
All models share the same pre-training, which follows recent diffusion language models [2, 3, 4]: the same architecture, optimizer and batch size, with SC variants trained for 77% of the 250k steps to match the training FLOPs (App. O.6). We then sweep the sampling grid for every method, the training time sampler and loss for the baselines, and the concentration schedule and churn for SDMs (Tables 28-35). As for distillation, Fig. 1 compares the best configuration of each method on the GSM8K test set, our evaluation split (App. O.6).
We thank the reviewer for this suggestion. We now report 95% bootstrapped confidence intervals in Fig 1.
We thank the reviewer for this question. IDLM distills discrete diffusion teachers with a lower accuracy than our SDM teacher, so our comparison indeed cannot isolate the impact of the distillation algorithm.
We did not mean to imply that IDLM is a weaker method. IDLM was unstable in our early experiments. We have amended the paragraph "Distillation and Auto-Guidance" (Sec. 5.2) accordingly.
More broadly, our paper does not aim to provide a comprehensive study of distillation methods for SDMs. We provide one efficient distillation method for SDMs, and leave a comprehensive study of distillation for simplex-based models to future work.
At κ=1, the next state depends only on the latest draw of x^0 (Prop. 3.3), so preserved uncertainty does not explain our results with κ=1. Only with κ<1 does the next state depends on the current state Pt. Importantly, with κ=0 and ct=ε/(1−αt), Pt is a random convex combination of the initial noise and all past draws of x^0 (Sec. 4.1). We therefore only claim that the simplex state preserves uncertainty with κ<1.
The best churn depends on the task. On Sudoku and TinyGSM, κ=1 performs best, consistent with prior work on logic tasks: [5] report that "re-corruption alone markedly improves solve rates on logic tasks", and in our baselines, pure predictor-corrector sampling improves UDMs from 33.7% to 46.3% on TinyGSM (T=0.1, 512 NFEs; Table 30). On OpenWebText, we use κ=0, which carries the state across steps.
Yes. At κ=1, Ps depends only on the sampled x^0 (Prop. 3.3), and with the Argmax input, x^0 depends only on argmaxPt, so the argmax tokens form a discrete-state Markov chain.
With the uniform prior, argmaxPs equals x^0 with a probability that depends only on s, and is uniform over the other tokens otherwise. This chain therefore has the form of the predictor-corrector sampler of uniform diffusion (UDM + PC, App. N.2), with a different noise schedule.