- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Simplax, an augmentation of uniform discrete diffusion that attaches an auxiliary Dirchlet-distributed simplex variable . Marginalizing out exactly recovers the original uniform discrete diffusion process, and the authors use this structure to derive a Rao–Blackwellized training objective and a -based reverse sampler. Since the denoiser still takes the categorical token as input, the efficiency of token-based denoising is preserved. Empirically, Simplax improves generative perplexity on OpenWebText language modeling and improves both solving accuracy and unconditional validity on Sudoku.
Critical strengths
- The proposed method is clean and novel. Attaching preserves the marginal distribution of exactly, and the resulting reverse conditionals are available in closed form.
- The method is practical. By keeping categorical token as inputs, it is faster than continuous-state methods, with only about 10–20% overhead relative to UDLM.
- The empirical results are strong. Simplax shows good results on language modeling, and on Sudoku task.
Critical weaknesses
- The Rao–Blackwellization applies only to the auxiliary anchor drawn from . However, relative to UDLM, Simplax actually introduces an additional source of randomness, . Nevertheless, the paper provides no analysis or discussion about why attaching leads to improved performance.
- The numerical precision of categorical sampling should be clearly specified for all baselines in the language modeling experiments, and ideally unified to FP64. Prior work has shown that categorical sampling in float32 can be inaccurate and can artificially lower entropy and generative perplexity [1]. Appendix A.4 states that LangFlow uses an FP32 sampler and S-FLM an FP64 sampler, but the precision used for the other methods, including Simplax’s categorical sampling, is not stated.
- The paper attributes the benefit of UDLM initialization to a staged strategy similar in spirit to Duo (in line 323). However, Duo’s curriculum uses a simplex relaxation to first solve an easier, soft problem and then a harder one (soft to hard), which is why it can be viewed as curriculum learning. In contrast, the staged Simplax training first learns with a hard anchor (UDLM) and then with a soft, weighted-sum anchor (Simplax), i.e., from hard to soft. Since the direction is reversed, the analogy to Duo does not seem appropriate.
- In Fig. 4, Simplax’s generative perplexity appears to degrade more quickly as generative entropy increases. This behavior is not discussed.
- In several figures (e.g., Fig. 4 and Fig. 3 etc), multiple graphs share similar line styles and are distinguished mainly by color, which makes them difficult to read for color-blind readers.
[1] Zheng, Kaiwen, et al. "Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling." International Conference on Learning Representations. Vol. 2025. 2025.
Questions for authors
-
Given that Rao–Blackwellization only removes the variance from sampling , while adds an extra random variable relative to UDLM, what is the mechanism behind the performance improvement?
-
Duo’s curriculum proceeds from soft to hard, whereas the staged Simplax (UDLM init) training proceeds from a hard anchor (UDLM) to a soft weighted-sum anchor (Simplax). Could the authors explain why this reversed schedule is described as similar to Duo, and what actually explains the benefit of UDLM initialization?
-
What numerical precision is used for categorical sampling for each method? Do the results still hold even when all methods use float64 sampling?
-
Why does Simplax’s generative perplexity degrade more quickly at higher entropy than the baselines?
I think the method is simple, novel, and effective. If these questions and weaknesses are adequately addressed, I’m willing to change my assessment.
10 comments
Comment 2/4 to Reviewer KF8L
W2. The numerical precision of categorical sampling should be clearly specified for all baselines in the language modeling experiments, and ideally unified to FP64. Prior work has shown that categorical sampling in float32 can be inaccurate and can artificially lower entropy and generative perplexity [1]. Appendix A.4 states that LangFlow uses an FP32 sampler and S-FLM an FP64 sampler, but the precision used for the other methods, including Simplax's categorical sampling, is not stated.
Q3. What numerical precision is used for categorical sampling for each method? Do the results still hold even when all methods use float64 sampling?
Thank you for raising the precision of categorical sampling. In short, float32 sampling did not systematically favor Simplax, and the conclusions of Tab. 1 hold when every method uses float64 sampling. We give the details below, and the revision states the precision of every method.
Only Simplax, FLM, and LangFlow used float32. In the submitted results, AR, CANDI, UDLM, MDLM, and Duo use float64 categorical sampling at every step. FLM and LangFlow draw tokens with a categorical sample only once, at the final step, and this draw was in float32. S-FLM decodes its final state deterministically by argmax, so it involves no categorical sampling; its FP64 sampler refers to the ODE integration. Both Simplax models used float32 categorical sampling at every step. We now state this in App. A.4, L.1056.
We re-evaluated Simplax, Simplax (UDLM init), FLM, and LangFlow with float64 categorical sampling: at every step for both Simplax models, and at the final step for FLM and LangFlow. In this re-evaluation, every method of Tab. 1 therefore uses float64 sampling. We follow the protocol of Tab. 1, with 1024 samples and the same scorer. For Simplax we additionally refined the temperature on a 0.003 grid so that the generated entropy matches the data entropy to within nats.
Each cell shows the submitted float32 value → the value with float64 sampling. Time is seconds per batch of 16 sequences of length 1024, measured with the protocol of Tab. 1; the two Simplax models share the same sampler and therefore the same time.
| Method | NFE | ENT | GPT-2 L | Llama-2 | Time (s) |
|---|---|---|---|---|---|
| FLM | 16 | 5.58† → 5.58† | 259.5 → 258.0 | 139.3 → 138.8 | 0.896 → 0.908 |
| 128 | 5.42 → 5.42 | 112.0 → 110.8 | 66.5 → 66.7 | 7.12 → 7.12 | |
| 1024 | 5.45 → 5.45 | 125.6 → 125.2 | 74.4 → 74.4 | 57.0 → 56.9 | |
| LangFlow | 16 | 5.42 → 5.42 | 115.1 → 114.6 | 59.9 → 59.2 | 1.15 → 1.17 |
| 128 | 5.42 → 5.44 | 60.2 → 60.8 | 30.0 → 30.0 | 9.38 → 9.38 | |
| 1024 | 5.41† → 5.43† | 68.3 → 64.4 | 28.2 → 27.7 | 75.3 → 75.2 | |
| Simplax | 16 | 5.46 → 5.44 | 98.7 → 92.9 | 54.2 → 51.6 | 0.814 → 0.859 |
| 128 | 5.46 → 5.44 | 63.4 → 59.8 | 35.0 → 33.8 | 6.09 → 6.46 | |
| 1024 | 5.45 → 5.44 | 52.6 → 51.4 | 29.8 → 29.2 | 48.4 → 51.3 | |
| Simplax (UDLM init) | 16 | 5.42 → 5.44 | 81.5 → 84.9 | 45.2 → 46.4 | 0.814 → 0.859 |
| 128 | 5.45 → 5.44 | 56.5 → 54.3 | 31.1 → 30.3 | 6.09 → 6.46 | |
| 1024 | 5.42 → 5.44 | 42.8 → 45.8 | 24.4 → 25.7 | 48.4 → 51.3 |
† The temperature sweep does not reach the data entropy 5.44.
With float64 for all methods, Tab. 1's conclusions hold. Float64 sampling raised the Gen. PPL of Simplax (UDLM init) at NFE 16 and 1024 (e.g., from 42.8 to 45.8 at NFE 1024 under GPT-2 Large) and lowered that of Simplax at every NFE, so float32 did not favor Simplax in one direction; these changes likely reflect the entropy at which each point was matched rather than the precision, as explained below. With float64 sampling, Simplax (UDLM init) is the best diffusion model on GPT-2 Large at every NFE and on Llama-2 at NFE 16 and 1024; at NFE 128 under GPT-2 Large, its float64 value, 54.3, is now also below UDLM (55.0), which was ahead of the submitted 56.5. LangFlow remains best on Llama-2 at NFE 128. Gen. PPL rises steeply with entropy near 5.44, by about 2–3% per 0.01 nats in our sweeps, and the submitted points of both Simplax models lay between 5.42 and 5.46. Float64 sampling adds about 6% to the sampling time of Simplax; at NFE 1024 it takes 51.3 s and remains faster than FLM, LangFlow, and S-FLM, which takes 80.5 s. Our other replies quote the submitted Tab. 1; we will update all results in the paper to float64 sampling in the camera-ready version.
Comment 3/4 to Reviewer KF8L
W3. The paper attributes the benefit of UDLM initialization to a staged strategy similar in spirit to Duo (in line 323). However, Duo's curriculum uses a simplex relaxation to first solve an easier, soft problem and then a harder one (soft to hard), which is why it can be viewed as curriculum learning. In contrast, the staged Simplax training first learns with a hard anchor (UDLM) and then with a soft, weighted-sum anchor (Simplax), i.e., from hard to soft. Since the direction is reversed, the analogy to Duo does not seem appropriate.
Q2a. Duo's curriculum proceeds from soft to hard, whereas the staged Simplax (UDLM init) training proceeds from a hard anchor (UDLM) to a soft weighted-sum anchor (Simplax). Could the authors explain why this reversed schedule is described as similar to Duo, ...
Thank you for the careful comparison with Duo. We agree that the submitted sentence did not state what the analogy refers to, and we have clarified it in the revision.
The comparison concerns how a fixed budget is allocated. In both Duo and Simplax, the same network is trained with one objective for part of the budget and then switched to the final objective. We do not claim that the UDLM stage is easier or that the schedule proceeds from soft to hard. The revised sentence at L.377 now states this directly: Duo likewise switches the training objective partway through a fixed budget.
Soft and hard mean different things in Duo and Simplax. The soft/hard distinction in Duo refers to the denoiser input, a tempered softmax of the Gaussian latent versus its argmax (Sahoo et al., 2025, Sec. 4.1), whereas in Simplax the denoiser input is the hard token in both stages and the weighted-sum anchor appears only inside the loss.
Q2b. ... and what actually explains the benefit of UDLM initialization?
Thank you for this question on the benefit of UDLM initialization. The revision adds the explanation below to Sec. 3.2 and App. C.
The two terms of the objective converge at different speeds. By the new Eq. 17 in Sec. 3.2, it decomposes exactly as , and exactness keeps identical across stages, so a UDLM checkpoint has already optimized the dominant term. Measuring both terms on held-out data shows that they converge on different time scales. Starting from UDLM 800k, 200k Simplax steps improve by while changing by only , and 94% of this gain is reached within 50k steps; , in contrast, still improves between 400k and 1M UDLM steps. The staged schedule therefore spends most of the budget on the slowly converging main term and needs only a short Simplax stage for the cross-anchor term, consistent with Tab. 21. We have added this explanation to Sec. 4.2 at L.377, which corresponds to L.323 of the submission, and the full analysis to App. C as the new paragraph "Decomposition of the training objective" with Fig. 18 and Tab. 22.
Comment 1/4 to Reviewer KF8L
We thank the reviewer for the thoughtful questions on the mechanism behind the improvement, the sampling precision, the staged training, and the shape of the frontiers. We address each point below and indicate the corresponding changes in the revised paper.
Thank you for this question on the mechanism behind the improvement. The revision adds the analysis below to Sec. 3.2 and App. C.
wt changes the objective that the network minimizes. The network input is zt, and the objective is linear in wt, so the randomness of wt only adds noise to an unbiased estimate of a fixed objective. Averaging over wt gives the new Eq. 17:
Ewt[L]=1+η1LUDLM(anchor zt)+1+ηηEz~t∼q(zt∣x)[LUDLM(anchor z~t)].
The first term is the UDLM objective, and the second term is new. The Rao–Blackwellization only makes this objective computable from a single network evaluation.
The new term supervises the transitions that the sampler adds. By Eq. 10, a Simplax reverse step draws an anchor from Cat(wt) and takes a uniform-diffusion reverse step from it. Since wt=(zt+ηp^t)/(1+η), with p^t the previous clean-token prediction mapped to time t, the anchor differs from the input token zt with probability up to η/(1+η), while the network still sees zt. The UDLM term evaluates the prediction only through the transition out of the input token, whereas the cross-anchor term evaluates the same prediction through the transition out of an independently corrupted anchor, which is exactly this input–anchor mismatch.
The gain requires both the training term and the sampler. In the new Tab. 18, 50k-step checkpoints sampled at NFE 1024 give the following Gen. PPL under GPT-2 Large:
Without the cross-anchor term, the added transitions raise Gen. PPL by 10% (from 79.0 to 87.2); with it, they lower Gen. PPL by 21% (from 74.1 to 58.2). The added transitions by themselves therefore do not improve generation at NFE 1024.
Off-input anchors keep revising tokens at high NFE. As the step size shrinks, a UDLM transition keeps the current token with probability approaching one, whereas an off-input anchor still replaces it. On the main checkpoint, Simplax (UDLM init) in Tab. 1, the sampler takes 0.12, 0.97, and 7.7 off-input anchors per token at NFE 16, 128, and 1024 (new Tab. 19). At NFE 1024, it changes each token 14.2 times, compared with 6.6 under UDLM transitions with the same predictions, and 71% of the final tokens were last set by an off-input anchor. Since the denoiser conditions on a context that keeps developing during generation, these updates let earlier token choices be revised with later predictions, which UDLM transitions rarely do.
The gain comes from revisions late in generation. In the new Tab. 20, we use off-input anchors in only one half of the trajectory, on the same checkpoint at NFE 1024 with float64 sampling and entropy matched to 5.44. Gen. PPL:
Using them only in the late half reproduces the full gain, whereas using them only in the early half, when the context is still largely corrupted, does not help. The benefit therefore comes from revisions made once most of the context has been denoised.
We have added the decomposition as the new paragraph "What the objective trains" in Sec. 3.2 at L.243, with the proof of Eq. 17 in the new App. E.1, and the analysis above as the new paragraph "Why off-input anchors help" in App. C at L.1752, with the counts in the new Tab. 19 and the restricted-sampling results in the new Tab. 20.