acceptodds
Under review as a conference paper at ICLR 2027

Revisiting the 16× Pretraining Compute Gap between Diffusion and Autoregressive Language Models

Abstract

Diffusion language models (DLMs) have attracted wide attention for their fast, parallel decoding, but they are believed to be far less efficient to train than autoregressive language models (ARMs): a widely cited study reports that a DLM needs roughly 16× the training compute of an ARM to match its likelihood, a figure that has made pretraining DLMs from scratch look prohibitive at scale. We find that the methodology behind this figure has fallen out of step with current DLM practice: it scores the model with its evidence lower bound (ELBO) and trains a full-sequence, decoder-only model. We therefore revisit the scaling laws of DLMs, adopting the exact likelihood under the model’s own decoding order, block diffusion training, an encoder–decoder architecture, and a sweep over the block sizes used in practice. Under this new protocol, we study across ten model variants, about 1,700 models in total, and find that pretraining DLMs is far cheaper than previously thought. Concretely, scoring with the exact likelihood reduces the compute multiple to 12.8×; block diffusion training and an encoder–decoder architecture further reduce it to 7.8× at block size 32, and smaller block sizes narrow it to 4.6× at block size 8 and 3.0× at block size 4. We complement these scaling analyses with an empirical rule for the optimal decoder depth of encoder–decoder block diffusion, a held-out compute budget beyond the fitting range at which the gap between variants persists, and downstream benchmark results where likelihood improvements carry over to accuracy. Overall, the 16× figure substantially overstates the pretraining cost gap between DLMs and ARMs, and pretraining DLMs from scratch is far more viable than previously assumed.

Primary area:
foundation or frontier models, including LLMs
Keywords:
diffusion language models, masked diffusion, scaling laws, IsoFLOP analysis, pretraining efficiency, block diffusion, encoder-decoder architecture, exact likelihood

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zd8QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Rcent work SMDM found that masked diffusion LMs need about 16x the training compute of autoregressive models (ARMs) to reach the same likelihood. This paper re-measures that gap with about 1,700 compute-matched models, using four update rules: exact likelihood (DUEL) instead of the ELBO, block diffusion, an encoder–decoder architecture (E2D2), and block sizes 4 to 32. The multiple falls from 17.8× (reproducing SMDM) to 12.8× with exact likelihood, further going down to 7.8× with E2D2 at block size 32, and ultimately 3.0× at block size 4. The paper also gives a rule for E2D2 decoder depth, a held-out budget check, and downstream accuracy that tracks likelihood.

Critical strengths

The paper reports actual FLOPs across different dLM training recipes. The paper also provides a clean decomposition of where the the gap comes from at a held-out budget, separating the evaluation effect (ELBO vs. exact likelihood on the same checkpoints) from the modeling effects shows which choice buys how much. The decoder-depth rule is directly reusable.

Critical weaknesses

W1. I have concern that the “far cheaper” conclusion mostly holds only when DLMs are made more autoregressive. Diffusion LMs are attractive because they decode several tokens per step. Every likelihood here, though, assumes one token per step, which is fully sequential. Block size 1 is just an AR model. The paper never measures the speed given up, and perplexity alone can mislead across model families.

W2. Each model is trained once with a single learning rate shared from 19M to 2.1B parameters, and more importantly no training curves are shown. The paper’s own tables show unexplained jumps. Moving one layer between E2D2’s encoder and decoder swings loss by up to 80 m-nats, far beyond the 5 m-nat “flat basin” the paper describes. Several IsoFLOP curves are also non-convex at the large-model end. Because each E2D2 point is the best of up to 15 runs, while ARM and BD3LM points are single runs, chances are part of E2D2’s advantage may come from selecting lucky runs. Repeating a few cells with 2–3 seeds and reporting confidence intervals on the multiples would make the conclusions more solid.

Questions for authors
  1. To what extent does the "cheap training" advantage hold when the model decodes fast? Every likelihood in the paper comes from revealing one token per step, which is as slow as an autoregressive model. Diffusion models are attractive because they can reveal several tokens per step. Could the authors report the compute gap when the model reveals 2, 4 or 8 tokens per step, so readers can see the trade-off between training cost and decoding speed?
  2. What is the gap in real training time? The paper compares models at equal FLOPs, but Table 8 suggests block-diffusion runs take about twice as long on the same hardware as autoregressive runs. Could the authors report the gap in GPU hours as well? And how they are computed?
  3. How much do results vary between training runs? Each model was trained only once, and some neighboring results differ by large, unexplained amounts. Could the authors retrain a few settings with different random seeds, and show training curves for the largest models?
1 comment

We thank the reviewer for their advice. We provide our rebuttal as follows:

W1, Q1. One token per step.

  • First, the 16× figure for the MDM is also measured one token at a time: the ELBO is the likelihood of one-by-one decoding in a random order. Our protocol is no more sequential than the original.
  • Second, parallel decoding need not lose this quality: Theorem 1 of Fast-dLLM (Wu et al., 2026b) shows that when the selected tokens have high confidence, decoding them in parallel gives the same result as decoding them one by one.

W2, Q3. Run-to-run variation.

  • Training curves. They are smooth, with no spikes, for all models including the largest. We will add them to the appendix.
  • Loss near the optimal decoder depth. It is flat there: within one layer of the optimum the median change is 4.1 m-nats. The 80 m-nat swing occurs in one cell (b=4, 6×10^18, N=336M) and does not affect the scaling-law measurement: dropping any single cell for E2D2 b=4 leaves the multiple between 2.96× and 3.04×.
  • Convexity. Based on our observation, we do not observe any IsoFLOP curves in Figure 2 to be non-convex.
  • The decoder-depth sweep is not cherry-picking. It studies a degree of freedom that only the encoder–decoder architecture has, and the resulting rule (Eq. 2) is itself a useful finding.
  • Multiple seeds. The grid already costs about 136k GPU hours, so repeating it across seeds to recompute the multiples is not feasible. As a substitute, a residual bootstrap over the existing cells gives 95% intervals of [7.6, 8.1] for E2D2 b=32 (7.8×), [9.1, 10.3] for BD3LM b=32 (9.6×), and [2.8, 3.2] for E2D2 b=4 (3.0×). These compute multiple values are rather stable.

Q2. Training time.

You are right that the block-diffusion and E2D2 runs in Table 8 take longer per FLOP. This is an implementation artifact: unlike our ARM and MDM runs, our BD3LM and E2D2 implementations do not use the FlashAttention backend. With the same backend, we expect no such difference in training time at equal FLOPs. We will update the appendix to state this and avoid misleading readers.

Reviewer_iGBROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper revisits the finding that masked DLLMs need about 16x the training FLOPs of ARMs to reach similar likelihood. The authors claim that this number is mostly outdated with the new progress of techniques being applied to DLLMs after that. With a new protocol, the authors conducted iso-flop sweeps over different variants up to 2.1B non-embedding parameters on SlimPajama. They show that with things like replacing ELBO with DUEL for scoring, changing to BD3LM and E2D2 with different block sizes, the actual compute needed for matching likelihood DLLMs can be as low as 3x. They also provide empirical evidence beyond NLL.

Critical strengths
  1. A careful and systematic study on a widely cited claim. Not only do the authors reproduce the original results but also break down where this 16x (17x) factor comes from through ablation.
  2. The methodological modifications are well motivated. FLOP calculations are careful and detailed too.
  3. I respect the transparency and there are more details in the appendix that I did not read but the main paper provides enough information and supportive experiments.
  4. The decoder-depth rule for E2D2 is practical and well supported.
Critical weaknesses
  1. Correct me if I’m wrong but it seems like DUEL measures the likelihood of one-token-per-step decoding, which is the mode that is not usually how DLLMs decode. Of course with aggressive distillation and some other works that try to do one-step decoding, this would make sense.
  2. Shrinking the block size on both BD3LM and E2D2 makes the training FLOPs cheaper but it ends up being more AR-like. So I feel like this part is rather obvious and does not stand as the main claim. Instead, I’d love to see a Pareto Frontier being compared. Speed v.s. Training FLOPs under the same decoding quality. Basically given a target decoding throughput and training FLOPs, what is the better model, or given a training FLOPs, can DLLMs decode faster while maintaining similar quality.
  3. The tuning effort is slightly asymmetric across variants. Most of the runs are E2D2 due to grid search. ARMs only have a few runs and no arch tuning. Learning rates stay the same if I’m not wrong, which is a quite important axis. Moreover, prior papers show that even changing the weight decay factor makes AR still great under data-constraint regime compared to DLLMs, which has been considered as a core strength (a, b). So i’m slightly concerned about this due to less heavily tuned hyperparams.
  1. When the block size is a decently chosen one (so we have enough speedup) like 32, E2D2 still needs around 7.8x the ARM’s compute. Although it is true that it’s way cheaper than the 16x, stating that “pretraining DLMs from scratch is far more viable than previously assumed” needs to be slightly more precise.
Questions for authors
  1. What happens when batch size becomes larger? Or the slope when we go from bs=1 to bs=4 for example compared to ARMs.
  2. Is there a way to measure the “likelihood” for multi-token-per-step decoding?
1 comment

We thank the reviewer for their advice. We provide our rebuttal as follows:

W1, Q2. One-token-per-step decoding and multi-token likelihood.

  • First, the 16× figure for the MDM is also measured one token at a time: the ELBO is the likelihood of one-by-one decoding in a random order. Our protocol is no more sequential than the original.
  • Second, parallel decoding need not lose this quality: Theorem 1 of Fast-dLLM (Wu et al., 2026b) shows that when the selected tokens have high confidence, decoding them in parallel gives the same result as decoding them one by one.

W2. Pareto frontier.

This is a good idea. A frontier of decoding speed against training FLOPs at matched quality would show directly which model is better at a given throughput and budget. It is beyond the scope of this paper, which measures training compute, and we will pursue it in follow-up work.

W3. Asymmetric tuning.

  • The decoder-depth sweep is not extra tuning. It studies a separate degree of freedom that only the encoder–decoder architecture has, and the resulting rule (Eq. 2) is itself a useful finding.
  • All variants share the SMDM recipe for comparability with the 16× study. We keep this as a stated limitation.
  • Both references study fixed data with repeated epochs, where regularization controls overfitting. They are out of scope of this work since we study compute-constrained regime.

W4. We agree with the reviewer, and will update the abstract and conclusion, explicitly stating that DLM pretraining remains several times costlier than ARM pretraining.

Q1. Reading "bs" as block size: from the ARM (b=1, 1×) the multiple rises to 3.0× at b=4. From BD3LM at b=32 to the full-sequence MDM adds only 1.33×. Most of the cost is paid in the first few tokens of parallelism. If you meant serving batch size B: by Sec. 3.1, the useful block size is about 256/B, so larger batches need smaller blocks.

Reviewer_sUamOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Prior scaling studies of discrete diffusion have shown that discrete diffusion requires 16× more training compute than autoregressive models to achieve the same negative log-likelihood (NLL). This paper argues that this gap is overstated. Using DUEL-style NLL approximation, block diffusion, and the efficient E2D2 architecture, the paper claims that this gap can be narrowed to as little as 3×.

Critical strengths

It is good to confirm that E2D2 consistently outperforms block diffusion and to identify an effective ratio between the encoder and decoder in E2D2.

Critical weaknesses

I appreciate the authors' efforts to analyze scaling laws, but the main claims of the paper are overstated and potentially misleading. The overall framing of the paper should be revised to focus on the scaling laws of block diffusion.

  1. The findings of the paper are too trivial, and the claims are too strong.
  • As the block size of discrete diffusion decreases, the model becomes increasingly similar to an autoregressive model. This observation is not insightful, and I would even think that it instead highlights how inefficient the training of discrete diffusion is: even with a block size of 4, discrete diffusion requires 3 times more training compute than autoregressive models.
  • However, the authors make overly strong claims throughout the paper. For example, in the abstract, they state that “pretraining DLMs from scratch is far more viable than previously assumed.” Most of the recent large discrete diffusion models have been initialized from autoregressive models. However, if discrete diffusion still requires 3 times more compute than autoregressive models even with a block size of 4, it is not convincing that from-scratch pretraining of DLMs is truly viable. I think such autoregressive model initialization should be considered a more reasonable approach.
  1. The paper only considers training compute.
  • A smaller block size essentially means a smaller gain in inference speed. The inference speed of discrete diffusion benefits from parallel generation but also suffers from its attention mechanisms. Thus, it is questionable whether block sizes of 4 or 8 are truly practical settings.
  1. The interpretation of downstream benchmarks is misleading.
  • The authors state that “Likelihood improvements carry over to downstream accuracy.” However, this is potentially misleading because the main cause of the performance improvement seems to be the block size, which affects both likelihood and downstream accuracy. Again, it is trivial that more autoregressive-style diffusion models achieve better performance.
  • The authors states “DUEL is the better scoring rule,” as a novel finding, but this is also natural and is neither a novel finding of this paper nor of DUEL. The authors use top-1 confidence sampling here, and many prior works have already shown that top-1 confidence sampling performs better.
Questions for authors

Please see weaknesses.

1 comment

We thank the reviewer for their advice. We provide our rebuttal as follows:

W1. Trivial findings, overstated claims.

We will remove the claim that pretraining DLMs from scratch is "far more viable". The abstract will instead give the gap by block size (7.8× at b=32, 3.0–4.6× at b=4–8) and state that DLM pretraining remains several times costlier than ARM pretraining. Our contributions are three measurements, none of which comes from shrinking the block:

  1. Evaluation. Scoring the same MDM checkpoints with the exact likelihood instead of the ELBO lowers the multiple from 17.8× to 12.8×. Our paper is the first to notice that about a third of the 16× is an evaluation artifact.
  2. Block training. All prior work compares block diffusion with other models at the same number of training tokens. This is not a compute-fair comparison, because one block diffusion forward pass costs about 2× the FLOPs. To our knowledge, this paper is the first to evaluate block diffusion rigorously at matched FLOPs, and block training still wins there: 9.6× at b=32 against 12.8× for the full-sequence MDM.
  3. Encoder–decoder. To our knowledge, this paper is the first to show explicitly and rigorously that the encoder–decoder architecture is more training-efficient than standard block diffusion.

On AR initialization: the transfer looks cheap today because existing AR checkpoints are heavily overtrained, far beyond the compute-optimal point, and their pretraining compute is not counted. Our paper studies the compute-optimal regime, where all training compute is accounted for.

W2. Only training compute.

  • First, the 16× figure for the MDM is also measured one token at a time: the ELBO is the likelihood of one-by-one decoding in a random order. Our protocol is no more sequential than the original.
  • Second, parallel decoding need not lose this quality: Theorem 1 of Fast-dLLM (Wu et al., 2026b) shows that when the selected tokens have high confidence, decoding them in parallel gives the same result as decoding them one by one.

W3. Downstream benchmarks. We will change the heading to "Downstream accuracy is consistent with the likelihood ranking". Besides shrinking block size, each modeling step that improves the likelihood also improves accuracy:

  • BD3LM vs. MDM. BD3LM is more accurate than the full-sequence MDM in 15 of 20 (block size, budget) cells, and in all 12 cells at the three largest budgets.
  • E2D2 vs. BD3LM. At the same block size, E2D2 is more accurate than BD3LM in 17 of 20 cells.

Block size plays no role in these two comparisons. We agree that the DUEL-vs-ELBO scoring result is expected; it will be presented as a consistency check.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.