acceptodds
Under review as a conference paper at ICLR 2027

Alpha Diffusion Language Models: Factorization Alone Is Not the Problem

Abstract

Discrete diffusion language models can generate multiple tokens in parallel, but reducing the number of denoising steps can lead to inconsistent predictions. Standard cross-entropy training fits conditional token marginals, whereas parallel generation requires consistent joint predictions. We introduce *Alpha Diffusion Language Models (AlphaDLM)*, trained with a sequence-level alpha loss that recovers cross-entropy in the limit of vanishing alpha and has a joint-mode optimum at alpha one. Our analysis characterizes how the objective and factorization jointly determine the fitted distribution. We identify conditions under which intermediate alpha preserves multiple valid completions while excluding invalid token combinations. Trained on TinyGSM, AlphaDLM achieves *34.6%* accuracy on GSM8K with only four model evaluations. We further scale the method to *the larger SDAR model* and evaluate it on code and mathematics benchmarks. These results show that changing the training objective can improve the accuracy-computation trade-off of factorized diffusion language models.

Primary area:
generative models
Keywords:
Diffusion language models, Discrete diffusion

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_pbzGOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper describes a new alpha cross-entropy loss for training diffusion language models. They show that this improves performance on math and coding benchmarks.

Critical strengths
  • Empirical results show improvement over baseline
  • The method seems to remove incorrect parallel predictions in diffusion language models, especially at low NFE, due to their sequence-level alpha loss.
Critical weaknesses
  • I think the learned distribution pθp_\theta is different than the actual ground truth joint data distribution pdatap_\text{data}. It is not obvious to me why equation (12) (or any of the other equations, for that matter), would support learning to decode the joint data distribution. Fundamentally, it still makes mean-field token-wise predictions whose log-probs are simply added together. That is, the actual predictor model still does not seem to have interdependence. I can believe that this learns something different than the naive token-wise marginal distributions, but I still don't think that actually gives the ground truth joint data distribution. What distribution is actually being learned here? Can you characterize it in terms of the joint data distribution? Is it some tilted version of the ground truth joint data distribution? This is a serious problem because if our model isn't trained to learn the underlying ground truth distribution, then we can't really rely on it to make valid predictions in general.
  • Proposition 1 seems to have an unrealistic assumption that the different completions will be disjoint token-wise ("different groups use disjoint token sets at every position"). For instance, with the prefix "Hi! My name is", we could have valid completions "John Smith Wang" or "Julia Smith Patel". The middle token of both of those completions is still Smith. What would your loss do under this scenario?
Questions for authors

See weaknesses. To address them, could you try the following experiment: Construct a synthetic low-dimensional dataset for which the joint probabilities are known (e.g., sequences of length 8, with vocabulary of A, B, C, D). Report how the learned probabilities differ from the ground truth probabilities. I also want more rigorous theoretical analysis of your loss's learned distributions.

3 comments
AuthorsOpenReview ↗

Rebuttal 1

W.1

I think the learned distribution pθp_\theta is different than the actual ground truth joint data distribution pdatap_{\mathrm{data}}. ...

We agree: AlphaDLM does not learn the joint data distribution qq, and we do not claim that it does. But neither does a perfectly trained MDLM once several tokens are revealed in one step. Let qq be the data distribution of the tokens revealed together and SS its support (the valid completions). For a distribution π\pi, let pπ=∏iπip_\pi=\prod_i\pi_i be the product of its marginals πi\pi_i: the same token frequencies, but independent tokens. The total correlation TC(π)=KL(π ∥ pπ)\mathrm{TC}(\pi)=\mathrm{KL}(\pi\,\|\,p_\pi) measures the dependence between tokens under π\pi and is zero exactly when they are independent.

MDLM. The cross-entropy optimum is pqp_q, so its divergence from the data is KL(q ∥ pq)=TC(q)\mathrm{KL}(q\,\|\,p_q)=\mathrm{TC}(q). It equals qq only if the tokens are independent, and it generates invalid outputs whenever SS is not a Cartesian product.

AlphaDLM. By the characterization in Appendix A (Eq. 16), for 0<α<10<\alpha<1 the optimal factorized predictor is pα∗=pπ∗p^\ast_\alpha=p_{\pi^\ast}, where

π∗∈arg⁡min⁡π KL(π ∥ q)+α H(pπ),\pi^\ast\in\arg\min_{\pi}\ \mathrm{KL}(\pi\,\|\,q)+\alpha\,H(p_\pi),

with π\pi supported on SS and HH the entropy. The loss keeps the data but charges α\alpha for the entropy of the generated outputs. Two consequences follow (new Proposition 2 in Section 3.2, proved in Appendix G).

  • Diversity decreases: H(pα2∗)≤H(pα1∗)H(p^\ast_{\alpha_2})\le H(p^\ast_{\alpha_1}) for α1<α2\alpha_1<\alpha_2. This follows by adding the optimality conditions at α1\alpha_1 and α2\alpha_2.
  • Invalid outputs are bounded by token dependence, and hence by the same entropy: pα∗(S)≥e−TC(π∗)≥e−H(pα∗)p^\ast_\alpha(S)\ge e^{-\mathrm{TC}(\pi^\ast)}\ge e^{-H(p^\ast_\alpha)}. The first inequality is Jensen's, using that π∗\pi^\ast is supported on SS. The second holds because TC(π∗)=H(pα∗)−H(π∗)\mathrm{TC}(\pi^\ast)=H(p^\ast_\alpha)-H(\pi^\ast).

The entropy that bounds invalid outputs can only decrease as α\alpha grows. In the experiment of Q.1 the first bound is close to the measured value (0.820.82 against 0.880.88 at α=0.5\alpha=0.5).

So α\alpha trades diversity for validity, and kk selects the point. Without factorization the optimum is the tilted distribution q1/(1−α)q^{1/(1-\alpha)} that the reviewer mentions (Eq. 8, due to [1]); the display above is its factorized counterpart (equivalent to Eq. 9, following [2]). We do not try to recover qq in few steps, which no factorized model can do. We aim at valid outputs in few steps by changing only the loss, and the revised Discussion states the price in diversity.

W.2

Proposition 1 seems to have an unrealistic assumption that the different completions will be disjoint token-wise ...

The shared token is not an obstacle. The loss maximizes ∑yq(y)∏ipi(yi)α\sum_yq(y)\prod_ip_i(y_i)^\alpha. Since every valid completion has Smith in the middle, its factor is common to all terms: the objective equals p2(Smith)αp_2(\mathrm{Smith})^\alpha times the same objective for the first and last positions. Every optimum therefore predicts Smith with probability one, and the other two positions are solved as if the middle one were absent. Proposition 1 applies to them, with the two groups (John, Wang) and (Julia, Patel), which share no token at either position. If the names have probabilities 0.65 and 0.35, cross-entropy assigns 2⋅0.65⋅0.35=45.52\cdot0.65\cdot0.35=45.5% to the mixed names. For α>1/2\alpha>1/2 the optimum of our loss is one complete name, so John Smith Patel has probability zero. It cannot keep both names: any factorized distribution that does so must also produce the mixed names.

We agree that general overlapping supports are outside Proposition 1, and this is a real limitation. For q(AA)=0.4q(\mathrm{AA})=0.4 and q(AB)=q(BA)=0.3q(\mathrm{AB})=q(\mathrm{BA})=0.3, the optimum at α=0.5\alpha=0.5 still assigns 4.96% to the invalid BB, compared with 9% for cross-entropy (Appendix A.1). In this general case exact validity is not guaranteed, but the new Proposition 2 holds for any support and bounds the probability of invalid outputs. Our answer to Q.1 measures it on a distribution with an overlapping support.

References

[1] Sypherd et al. A Tunable Loss Function for Robust Classification. 2022.

[2] Lapidoth, Pfister. Two Measures of Dependence. Entropy, 2019.

AuthorsOpenReview ↗

Rebuttal 2

Q.1

... could you try the following experiment: Construct a synthetic low-dimensional dataset for which the joint probabilities are known (e.g., sequences of length 8, with vocabulary of A, B, C, D). ... I also want more rigorous theoretical analysis of your loss's learned distributions.

We ran this experiment (new Appendix G). Sequences have length 8 over the vocabulary A, B, C, D, and all 65,536 probabilities are enumerated exactly, so there is no sampling error. The data distribution is a Markov chain in which adjacent letters must differ: the first letter has probabilities (0.4, 0.3, 0.2, 0.1) for A, B, C, D, and each next letter is drawn from the same weights restricted to letters different from the previous one. A sequence is valid if no two adjacent letters are equal, which holds for 8,748 sequences. It does not satisfy the assumptions of Proposition 1.

One parallel step. All eight tokens are predicted at once, independently. We use the exact cross-entropy optimum and numerically optimize the alpha loss over factorized distributions. We enumerate all 65,536 sequences to calculate validity and total variation distance to the data.

We also calculate how many different valid sequences we expect to see in 1,000 independent generations. Repeated answers count only once, and invalid answers do not count. The column "Distinct valid / 1,000" therefore measures both correctness and variety. We compute this expectation directly from the probabilities, without sampling.

PredictorValid (%)Distinct valid / 1,000TV to data
MDLM (cross-entropy)10.91080.891
MDLM, token temperature 0.57.2700.928
MDLM, greedy tokens0.001.000
AlphaDLM, α=0.4\alpha=0.467.42010.806
AlphaDLM, α=0.5\alpha=0.588.01430.851
AlphaDLM, α=0.7\alpha=0.798.6520.923
AlphaDLM, α=1\alpha=110010.995

In one step, changing the loss raises validity from 10.9% to 88.0% at α=0.5\alpha=0.5, with 143 different valid answers expected in 1,000 generations. Neither predictor reproduces the data distribution. Sharpening the tokens of MDLM does not help: greedy selection returns AAAAAAAA, which is invalid. For α=0.4\alpha=0.4 and 0.50.5, AlphaDLM is also closer to the data in total variation.

Several steps, trained denoisers. We train two neural denoisers on sequences from the same distribution above, masking tokens and predicting the missing ones. Both use the same architecture and training setup: MDLM uses cross-entropy, and AlphaDLM uses sequence-level alpha loss with α=4/m\alpha=4/m for mm masked tokens.

Generation starts with all eight positions masked and takes 1, 2, 4, or 8 calls. "MDLM, exact" uses the true conditional token probabilities at each call and requires no training. With one call, trained AlphaDLM reaches 92.1% validity. All eight positions are masked, so its loss uses α=0.5\alpha=0.5, as in the one-step calculation above.

Valid (%) / calls1248
MDLM, exact10.916.734.1100
MDLM, trained10.815.530.495.5
AlphaDLM, trained (k=4k=4)92.196.098.299.8

The exact MDLM recovers the data distribution with eight calls. AlphaDLM already produces mostly valid outputs with one call, but its outputs become less diverse with more calls. With eight calls, MDLM is the better choice: it is valid and recovers the data distribution. The loss is useful when few calls are available and a valid output matters more than covering all valid outputs.

Concluding remarks. We would be grateful to know whether our answers address your concerns. If so, we kindly ask you to consider increasing your rating. We are happy to discuss any remaining questions.

Thanks for the thorough rebuttal. I've increased my rating.

Reviewer_y3vGOpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The authors introduce Alpha Diffusion Language Models (AlphaDLMs), DLMs that are trained with the generalized cross-entropy loss (also called alpha loss). For alpha -> 0, the alpha loss reduces to the standard cross-entropy loss, but for alpha > 0, the alpha loss is lower bounded by 1/alpha and more generally becomes mode-seeking rather than support covering. Without a factorized inductive prior, this results in sharpening of the sequence-level distribution by the exponent 1/(1-alpha). With a factorized inductive prior, this results in the token-level marginals of a similar sequence-level sharpened distribution (where correlation across token positions is additionally suppressed). The authors apply this loss to small language model (SLM) training on TinyGSM and fine-tuning of SDAR (a medium-scale DLM) on math and code. On TinyGSM (with SLM) and math (with SDAR) they report better accuracy at few-step generation, while on code (with SDAR) a beneficial transfer largely fails.

Critical strengths

Using the alpha loss for diffusion model training is an elegant alternative for improving few-step generation capabilities via policy sharpening, especially compared to previous methods based on trajectory supervision or self-distillation that require multi-stage training pipelines.

Critical weaknesses
  1. Much of the paper is prior art. It does not introduce the alpha loss and the vast majority of the presented analysis/theory is from prior art, except maybe Proposition 1, which adds only limited value in my opinion.

  2. There is no comparison to alternative methods for policy sharpening, in particular the cited works by the authors on trajectory supervision and self-distillation are completely missing from the experiments. The experiments also have additional issues, see my questions.

Questions for authors

Clarifications and minor concerns:

  1. On line 211-212, the authors write: "at α=1\alpha=1, any distribution supported on joint modes is optimal". But the formula (Eq. 8) indicates that the optimal distribution is exactly uniform over the modes of q. The description for the case α>1\alpha > 1 is similarly confusing.

  2. In Proposition 1, are completions from non-selected groups assigned probability zero? If so, can this be formally added to the proposition? If not, then isn't the statement rather trivial? E.g., standard cross-entropy gives the same guarantees for universal support of positive probability.

  3. Section 3.3 is not properly motivated. Why can one not use a fixed α\alpha but rather has to define α=k/m\alpha = k/m? This is also exacerbated since much of the theory is developed for 1/m<α<11/m < \alpha < 1, which cannot be guaranteed for a fixed k>1k > 1.

  4. What do you mean by "you evaluate Python solutions on all 1319 GSM8K test problems" on line 318?

  5. Can the authors repeat Table 1 with greedy decoding (T=0) for all methods that admit it? This would better capture the gains achieved by AlphaDLM and complement Figure 5.

  6. The values of k differ across domains, e.g., using k=0.8 for mathematics and k=0.4 for code. Can the authors provide a principled approach for choosing this?

  7. In Section 4.2, the authors write: "we apply Equation 12 separately to each four-token response block". Why is this required? This indicates a limitation in the alpha loss, potentially with stability? The authors should elaborate on this and defend their choice, provide a stability analysis, or indicate stability as a severe limitation of their loss.

  8. Can the authors repeat Figure 5 with temperature T=0 decoding? Right now it is unclear how much of the gains in the experiments on SDAR stem from token-level sharpening vs sequence-level.

  9. The authors write on lines 464-467: "AlphaDLM achieves higher TPF than CE+CAP with similar accuracy on both code benchmarks (Figure 6(c,d)). Both achieve 60.61% pass@1 on HumanEval-Instruct, while AlphaDLM increases TPF from 1.74 to 1.94. On MBPP-Instruct, TPF rises from 1.51 to 1.68 while pass@1 drops by 0.19 percentage points.". These claims are not substantiated or clear from the Figure. If anything, the Pareto front is spanned by CE+CAP rather than AlphaDLM.

Additional Remarks:

  1. In Section 2.1, the authors cite only Wu et al., 2026 for the confidence-based decoding, while there is a zoo of competing methods to provide attribution to, such as entropy-based, max-probability-based, and max-margin-based decoding. Also, confidence-based decoding originates back to at least Ghazvininejad et al. 2019.

  2. In the introduction, the authors cite Liu et al., 2025 and Xu et al., 2025 for method that explicitly model token dependencies, while there are more recent works, e.g., Bansal et al., 2025 and Hersche et al., 2026.

  3. The y-label for Figure 5 has a stray "(" bracket.

  4. In Section 4.2, adding how many tokens are contained in Nemotron-SFT-Math-v4 and OpenCodeInstruct, respectively, would help the reader.

  5. Model architecture (including # parameters) for TinyGSM is missing from the main text.

Ghazvininejad, M., Levy, O., Liu, Y., & Zettlemoyer, L. (2019, November). Mask-predict: Parallel decoding of conditional masked language models. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP) (pp. 6112-6121).

Bansal, P., & Sanghavi, S. (2025). Enabling approximate joint sampling in diffusion LMs. arXiv. https://arxiv.org/abs/2509.22738

Hersche, M., Menet, N., Tanios, R., & Rahimi, A. (2026). Locally coherent parallel decoding in diffusion language models. In Forty-third International Conference on Machine Learning. https://openreview.net/forum?id=hn4ndkaD6x

4 comments
AuthorsOpenReview ↗

Rebuttal 1

W.1

Much of the paper is prior art. It does not introduce the alpha loss and the vast majority of the presented analysis/theory is from prior art, except maybe Proposition 1, which adds only limited value in my opinion.

We agree that the alpha loss is not new and do not claim it. Our contribution is to apply it to the joint probability that a factorized denoiser assigns to all masked tokens and to analyze how the objective interacts with factorization. Eq. 8 is due to [1], Eq. 9 specializes [2], and Proposition 1 is ours. Section 3.2 now says this explicitly. The known case has two tokens and one completion per group, where selecting a group means collapsing to one answer. Proposition 1 shows that for 1/m<α<11/m<\alpha<1 the optimal factorized predictor excludes all invalid combinations while keeping every valid completion of the selected group, that is, before the limit α≥1\alpha\ge1 and without collapse. The same loss applied token by token only sharpens each marginal: at 4 NFE on TinyGSM it reaches 19.86%, against 34.65% for sequence-level training.

W.2

There is no comparison to alternative methods for policy sharpening, in particular the cited works by the authors on trajectory supervision and self-distillation are completely missing from the experiments. The experiments also have additional issues, see my questions.

We trained TinyGSM adaptations of T3D [3], dParallel [4], CDLM [5], and CE+CAP [6] (Table 1, Appendix F.1). All rows use the same architecture, confidence sampler, exact NFE, and 1,319 GSM8K problems.

NFE1248163264
CE0.450.535.6918.5036.3244.3540.64
CE+CAP0.610.616.7524.1839.7345.4942.38
dParallel0.762.1214.0331.3941.8543.5244.43
T3D9.4818.6520.4720.6220.6220.6220.62
CDLM0.300.7610.7725.5535.1037.2338.51
Ours, k=8k=82.737.7328.8943.9749.2049.8950.04
Ours, k=16k=166.2918.0434.6539.0440.5640.8640.94
Ours, k=32k=327.6619.4825.3225.7025.7025.7025.70

AlphaDLM is best at 4–64 NFE, and T3D is best at 1 NFE. Rows with different kk are separate checkpoints. The setup favors the baselines: T3D and dParallel start from CE at 250k updates and train 50k more on 100k verified teacher solutions, while AlphaDLM uses 250k updates in total, the original data, and no teacher. CE+CAP follows the AlphaDLM recipe with a different loss. CDLM uses 1,248 updates, as longer training did not help. On SDAR the comparison with CDLM is mixed (Figure 6, Appendix F.4): AlphaDLM is better on MBPP and at higher TPF on mathematics, CDLM on HumanEval.

Q.1

On line 211-212, the authors write: "at α\alpha = 1, any distribution supported on joint modes is optimal". But the formula (Eq. 8) indicates that the optimal distribution is exactly uniform over the modes of q. The description for the case α\alpha > 1 is similarly confusing.

Eq. 8 holds for 0<α<10<\alpha<1. As α→1−\alpha\to1^- it converges to the uniform distribution over joint modes. Exactly at α=1\alpha=1 the expected loss 1−∑yq(y)p(y)1-\sum_y q(y)p(y) is linear in pp, so every distribution supported on the joint modes is optimal, including the uniform one. For α>1\alpha>1, ∑yq(y)p(y)α≤∑yq(y)p(y)≤max⁡yq(y)\sum_y q(y)p(y)^\alpha\le\sum_y q(y)p(y)\le\max_y q(y), with equality only for a point mass on one joint mode, so mixtures of tied modes are not optimal. Section 3.2 now states the three cases separately.

Q.2

In Proposition 1, are completions from non-selected groups assigned probability zero? If so, can this be formally added to the proposition? If not, then isn't the statement rather trivial? E.g., standard cross-entropy gives the same guarantees for universal support of positive probability.

Yes. Every globally optimal factorized predictor assigns zero probability to all completions outside the selected group (proof in Appendix A.1), and the statement of Proposition 1 now says so. Cross-entropy instead yields the product of marginals, which keeps all valid completions but also gives positive probability to cross-group combinations such as I is.

References

[1] Sypherd et al. A Tunable Loss Function for Robust Classification. 2022.

[2] Lapidoth, Pfister. Two Measures of Dependence. Entropy, 2019.

[3] Zhang et al. T3D. 2026.

[4] Chen et al. dParallel. ICLR 2026.

[5] Kim et al. CDLM. MLSys 2026.

[6] Bie et al. LLaDA2.0. 2025.

AuthorsOpenReview ↗

Rebuttal 2

Q.3

Section 3.3 is not properly motivated. Why can one not use a fixed α\alpha but rather has to define α\alpha = k/m? This is also exacerbated since much of the theory is developed for 1/m < α\alpha < 1, which cannot be guaranteed for a fixed k > 1.

A fixed α\alpha is possible but treats inputs with different numbers of masked tokens mm unequally. (1) The gradient is the CE gradient times pθ(xM)α=exp⁡(αmsˉ)p_\theta(x_M)^\alpha=\exp(\alpha m\bar s), where sˉ\bar s is the mean token log-probability. For a fixed α\alpha this weight vanishes exponentially in mm, so heavily masked inputs barely contribute. With α=k/m\alpha=k/m it equals exp⁡(ksˉ)\exp(k\bar s) for every mm. (2) The range 1/m<α<11/m<\alpha<1 itself depends on mm, and a fixed α\alpha violates it whenever m≤1/αm\le1/\alpha. With α=k/m\alpha=k/m it becomes 1<k<m1<k<m, which for a fixed k>1k>1 fails only for inputs with at most kk masked tokens. There the optimum is a point mass on a joint mode, which is still a valid completion. Section 3.3 now explains both points.

Q.4

What do you mean by "you evaluate Python solutions on all 1319 GSM8K test problems" on line 318?

We follow the standard TinyGSM training / GSM8K evaluation setup used in S-FLM and other reported diffusion baselines, including MDLM and DUO. The model is trained on TinyGSM question-Python solution pairs. At evaluation time, it generates one Python solution for each of the 1,319 GSM8K test problems; we execute the generated program and compare its numerical output with the ground-truth answer.

Q.5

Can the authors repeat Table 1 with greedy decoding (T=0) for all methods that admit it? This would better capture the gains achieved by AlphaDLM and complement Figure 5.

We ran T=0T=0 ancestral sampling for all methods with available checkpoints (new Appendix F.2, Table 9): argmax clean-token predictions, random reveal positions, exact NFE. AlphaDLM is evaluated here without confidence-based unmasking.

NFE48163264
MDLM2.056.6013.8722.3728.66
DUO5.0812.9624.4930.1731.39
IDLM (MDLM)1.906.9015.3920.3924.87
IDLM (DUO)6.3715.5426.0030.7133.59
Ours, k=16k=1614.9425.4731.1637.9141.17

AlphaDLM is best at every budget. The other rows of Table 1 already have greedy counterparts: the rows with confidence-based unmasking use argmax tokens, S\mathbb{S}-FLM (top-1) is greedy, and the FMLM+ and DBTM values are published results with greedy refinement. AlphaDLM is higher in Table 1 (34.6% at 4 NFE) because confidence-based position selection also contributes.

Q.6

The values of k differ across domains, e.g., using k=0.8 for mathematics and k=0.4 for code. Can the authors provide a principled approach for choosing this?

kk sets the trade-off in Eq. 9 between fidelity to the data distribution and independence of tokens, so its best value depends on how many tokens are decoded per evaluation. On TinyGSM, k=16k=16 is best at 4 NFE and k=8k=8 at 16. On SDAR, larger kk raises TPF and lowers peak accuracy (Figure 6(e,f)). We chose the reported values from this sweep, evaluated on the same benchmarks, as those increasing TPF without a noticeable loss of peak accuracy (now stated in Section 4.2). Both k=0.4k=0.4 and k=0.8k=0.8 improve over CE. In practice we suggest starting from a small kk and increasing it while accuracy at the target budget does not drop. Fewer evaluations favor larger kk (see Q.8).

Q.7

In Section 4.2, the authors write: "we apply Equation 12 separately to each four-token response block". Why is this required? This indicates a limitation in the alpha loss, potentially with stability? The authors should elaborate on this and defend their choice, provide a stability analysis, or indicate stability as a severe limitation of their loss.

It is not required for stability. SDAR predicts the four tokens of a block in parallel given the completed previous blocks, and tokens of different blocks are never predicted in the same evaluation. Invalid combinations can therefore arise only within a block, and the block-level loss couples exactly the tokens decoded jointly (now explained in Section 4.2). On TinyGSM the loss is applied to all masked tokens of 512-position sequences without blocks, for kk up to 64.

AuthorsOpenReview ↗

Rebuttal 3

Q.8

Can the authors repeat Figure 5 with temperature T=0 decoding? Right now it is unclear how much of the gains in the experiments on SDAR stem from token-level sharpening vs sequence-level.

Figure 6 (SDAR) already uses T=0T=0 (argmax tokens), which its caption now states, and Figure 5 includes a T=0T=0 curve. Since the confidence threshold still depends on probabilities, we also evaluated the SDAR mathematics models with a fixed number of steps per four-token block, T=0T=0, and no confidence threshold (new Appendix F.3, Table 10). With one step per block, all four tokens come from one evaluation, so rescaling the logits cannot change the output.

kkGSM8K, 2 stepsMATH500, 2 stepsGSM8K, 1 stepMATH500, 1 step
0 (CE)49.2022.6022.828.40
0.456.8627.4032.6812.60
0.659.4429.6038.2917.40
0.860.8831.6045.3418.60
1.062.0935.0044.3521.60
1.262.2431.8048.3721.00
1.664.2930.8049.9627.40
2.060.9631.8050.8026.40

Without confidence, one-step accuracy more than doubles (22.82% to 50.80% on GSM8K, 8.40% to 27.40% on MATH500), which token-level sharpening cannot explain. Consistently, on TinyGSM the token-wise alpha loss reaches 19.86% at 4 NFE against 34.65% for the sequence-level loss.

Q.9

The authors write on lines 464-467: "AlphaDLM achieves higher TPF than CE+CAP with similar accuracy on both code benchmarks (Figure 6(c,d)). Both achieve 60.61% pass@1 on HumanEval-Instruct, while AlphaDLM increases TPF from 1.74 to 1.94. On MBPP-Instruct, TPF rises from 1.51 to 1.68 while pass@1 drops by 0.19 percentage points.". These claims are not substantiated or clear from the Figure. If anything, the Pareto front is spanned by CE+CAP rather than AlphaDLM.

We agree. The quoted numbers compare both methods at the same threshold (0.99), which corresponds to different operating points. At similar TPF they are on par: 60.61% at TPF 1.94 for AlphaDLM against 60.61% at 1.97 for CE+CAP on HumanEval-Instruct, and 66.37% at 2.01 against 66.56% at 2.08 on MBPP-Instruct, within one standard deviation. We revised this paragraph and the contribution list: on code AlphaDLM is on par with CE+CAP.

R.1

In Section 2.1, the authors cite only Wu et al., 2026 for the confidence-based decoding, while there is a zoo of competing methods to provide attribution to, such as entropy-based, max-probability-based, and max-margin-based decoding. Also, confidence-based decoding originates back to at least Ghazvininejad et al. 2019.

Section 2.1 now credits Mask-Predict [1] and distinguishes the max-probability (used here), entropy, and margin criteria.

R.2

In the introduction, the authors cite Liu et al., 2025 and Xu et al., 2025 for method that explicitly model token dependencies, while there are more recent works, e.g., Bansal et al., 2025 and Hersche et al., 2026.

We added [2] and [3] to the introduction and Related Work. These works change the sampling procedure, whereas we change only the training objective.

R.3

The y-label for Figure 5 has a stray "(" bracket.

The label "GSM8K accuracy (%)" was clipped. It is fixed.

R.4

In Section 4.2, adding how many tokens are contained in Nemotron-SFT-Math-v4 and OpenCodeInstruct, respectively, would help the reader.

Mathematics: 272,872 examples, 186.1M tokens. Code: 1,489,929 examples, 573.2M tokens. Fine-tuning processes 349.2M and 197.0M tokens, respectively (added to Appendix C).

R.5

Model architecture (including # parameters) for TinyGSM is missing from the main text.

The model is a bidirectional DiT with 12 blocks, 12 heads, width 768, and 167.9M parameters. This is now stated in Section 4.1.

Concluding remarks. We would be grateful to know whether our answers address your concerns. If so, we kindly ask you to consider increasing your rating.

References

[1] Ghazvininejad et al. Mask-Predict. EMNLP 2019.

[2] Bansal, Sanghavi. Enabling Approximate Joint Sampling in Diffusion LMs. 2025.

[3] Hersche et al. Locally Coherent Parallel Decoding in Diffusion Language Models. ICML 2026.

Thank you for your detailed response. You have addressed all of my questions to my full satisfaction. With my concerns regarding the soundness of the study now resolved, I am happy to recommend the paper for acceptance.

Reviewer_9yQKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper changes the training loss to improve parallel generation in factorized diffusion models. It uses a sequence-level alpha loss and reports gains with few decoding steps.

Critical strengths
  1. Separating the effect of factorization from the training objective is interesting.
  2. The matched comparison shows a large gain at evaluations.
  3. The experiments also include a 1.7B model, so the results go beyond small toy settings.
Critical weaknesses
  1. The experiments are limited to math and coding. It is unclear whether the gains generalize to other tasks.
Questions for authors
  1. The gains on SDAR-1.7B are mostly in efficiency, with only small gains in performance. Would the method still help on a larger model such as LLaDA [1]?

[1] Nie, Shen, et al. "Large language diffusion models." Advances in Neural Information Processing Systems 38 (2026): 50608-50646.

2 comments

W.1

The experiments are limited to math and coding. It is unclear whether the gains generalize to other tasks.

We chose these domains because correctness is verified automatically, by executing the program or checking the final answer, so an invalid combination of tokens shows up directly as an error. The objective itself does not rely on these domains: it changes only the training loss and applies to any set of masked tokens. Whether it helps on other tasks, in particular open-ended generation where the diversity of outputs matters, remains open. The revised Discussion (Section 5) states this limitation.

Q.1

The gains on SDAR-1.7B are mostly in efficiency, with only small gains in performance. Would the method still help on a larger model such as LLaDA [1]?

We expect the method to help on LLaDA [1], and most of all when it is sampled with few steps. We address the two parts of the question in turn.

Gains on SDAR. The standard SDAR decoder leaves little room for accuracy gains: with four-token blocks and confidence threshold 0.99, CE commits only about 1.4 tokens per forward, so generation is close to sequential. In this regime the gain appears as speed. AlphaDLM matches the best CE accuracy with about 43–44% more tokens per forward (TPF): 76.66% at TPF 2.00 against 76.35% at TPF 1.38 on GSM8K, and 51.04% at TPF 2.06 against 50.60% at TPF 1.44 on MATH500. When more tokens are generated per forward, the gain appears in accuracy: if every four-token block is decoded in a single step, GSM8K accuracy rises from 22.82% for CE to 45.34% for AlphaDLM (new Appendix F.3).

Larger models such as LLaDA. The objective requires only a factorized denoiser that predicts masked tokens, as in LLaDA, and can be applied by fine-tuning the pretrained model, as we did for SDAR. The problem it addresses comes from the factorized output rather than from model size: even exact token marginals assign probability to invalid combinations of tokens (Section 3.2). LLaDA denoises the whole response rather than four-token blocks, so it reveals many more tokens per step than SDAR, which is the regime where our gains are largest. In the rebuttal we focused on adding baselines to the TinyGSM and SDAR setups (T3D, dParallel, CDLM, and CE+CAP, see the general response), so that the comparison on the existing settings is more complete, and we leave the evaluation on LLaDA to future work.

Concluding remarks. We would be grateful to know whether our answers address your concerns. We are happy to discuss any remaining questions.

References

[1] Nie et al. Large Language Diffusion Models. NeurIPS 38.

Thank you for the efforts. While I find the explanations reasonable, some of my main concerns are not directly addressed with new experiments. I still believe evaluation on LLaDA-scale models is important. Moreover, LLaDA has been evaluated on tasks beyond math and coding. I will keep my score unchanged.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

Dear reviewers, thank you for your thoughtful feedback. We appreciate that you found separating the effect of factorization from the training objective interesting (9yQK), the method an elegant alternative to multi-stage trajectory supervision and self-distillation (y3vG), and the removal of incorrect parallel predictions at low NFE supported by the results (9yQK, pbzG). Below we summarize the main additions.

(1) Comparison with trajectory supervision and self-distillation. We trained TinyGSM versions of T3D [1], dParallel [2], CDLM [3], and CE+CAP [4] (Figure 1, Table 1, and Appendix F.1). AlphaDLM is the most accurate method at 4–64 NFE, for example 34.65% at 4 NFE against 20.47% for the best baseline, and only T3D is better at one NFE. AlphaDLM obtains this by changing only the loss, without the teacher, generated trajectories, or extra training stage that T3D, dParallel, and CDLM require. On SDAR-1.7B we added CDLM (Figure 6 and Appendix F.4): AlphaDLM gives the better trade-off on MBPP-Instruct and at higher TPF on mathematics, and CDLM is better on HumanEval-Instruct.

(2) Which distribution is learned. Our loss does not recover the joint data distribution, but neither does a perfectly trained MDLM once several tokens are revealed in one step. The new Proposition 2 (Section 3.2) shows that α\alpha trades the diversity of the outputs for a bound on the probability of invalid outputs, so the loss extends cross-entropy with a controlled trade-off.

Revised paper. All changes in the revised paper are marked in blue.

  • Sections 1, 2.1 and Related Work. Earlier work on confidence-based decoding and the suggested references are now cited.
  • Section 3.2. We state which results are known and which are new. We separate the cases α→1−\alpha\to1^-, α=1\alpha=1, and α>1\alpha>1. Proposition 1 now states that completions outside the selected group have zero probability. We discuss shared tokens and overlapping supports. The new Proposition 2 gives the validity–diversity trade-off.
  • Section 3.3. We explain why α=k/m\alpha=k/m is used rather than a fixed α\alpha, and where the range of Proposition 1 holds in practice.
  • Section 4. We state the TinyGSM architecture and parameter count. The new baselines are added to Figures 1 and 6 and Table 1. We explain the SDAR block-level loss and the selection of kk, and we revised the paragraph on code. The caption of Figure 6 states T=0T=0. The axis label of Figure 5 is fixed.
  • Section 5. We state the limitations: the scope of the experiments and the trade-off between validity and diversity.
  • Appendix C. We report the token counts of the SDAR training sets.
  • Appendix F (new). Baselines on TinyGSM (F.1), greedy ancestral decoding (F.2), SDAR with a fixed number of steps per block (F.3), and CDLM on SDAR (F.4).
  • Appendix G (new). Proof of Proposition 2, the analysis of cross-entropy with several tokens per step, and an experiment on a known joint distribution.

References

[1] Zhang et al. Few-Step Diffusion Language Models via Trajectory Self-Distillation. 2026.

[2] Chen et al. dParallel. ICLR 2026.

[3] Kim et al. CDLM. MLSys 2026.

[4] Bie et al. LLaDA2.0. 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.