- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Rcent work SMDM found that masked diffusion LMs need about 16x the training compute of autoregressive models (ARMs) to reach the same likelihood. This paper re-measures that gap with about 1,700 compute-matched models, using four update rules: exact likelihood (DUEL) instead of the ELBO, block diffusion, an encoder–decoder architecture (E2D2), and block sizes 4 to 32. The multiple falls from 17.8× (reproducing SMDM) to 12.8× with exact likelihood, further going down to 7.8× with E2D2 at block size 32, and ultimately 3.0× at block size 4. The paper also gives a rule for E2D2 decoder depth, a held-out budget check, and downstream accuracy that tracks likelihood.
Critical strengths
The paper reports actual FLOPs across different dLM training recipes. The paper also provides a clean decomposition of where the the gap comes from at a held-out budget, separating the evaluation effect (ELBO vs. exact likelihood on the same checkpoints) from the modeling effects shows which choice buys how much. The decoder-depth rule is directly reusable.
Critical weaknesses
W1. I have concern that the “far cheaper” conclusion mostly holds only when DLMs are made more autoregressive. Diffusion LMs are attractive because they decode several tokens per step. Every likelihood here, though, assumes one token per step, which is fully sequential. Block size 1 is just an AR model. The paper never measures the speed given up, and perplexity alone can mislead across model families.
W2. Each model is trained once with a single learning rate shared from 19M to 2.1B parameters, and more importantly no training curves are shown. The paper’s own tables show unexplained jumps. Moving one layer between E2D2’s encoder and decoder swings loss by up to 80 m-nats, far beyond the 5 m-nat “flat basin” the paper describes. Several IsoFLOP curves are also non-convex at the large-model end. Because each E2D2 point is the best of up to 15 runs, while ARM and BD3LM points are single runs, chances are part of E2D2’s advantage may come from selecting lucky runs. Repeating a few cells with 2–3 seeds and reporting confidence intervals on the multiples would make the conclusions more solid.
Questions for authors
- To what extent does the "cheap training" advantage hold when the model decodes fast? Every likelihood in the paper comes from revealing one token per step, which is as slow as an autoregressive model. Diffusion models are attractive because they can reveal several tokens per step. Could the authors report the compute gap when the model reveals 2, 4 or 8 tokens per step, so readers can see the trade-off between training cost and decoding speed?
- What is the gap in real training time? The paper compares models at equal FLOPs, but Table 8 suggests block-diffusion runs take about twice as long on the same hardware as autoregressive runs. Could the authors report the gap in GPU hours as well? And how they are computed?
- How much do results vary between training runs? Each model was trained only once, and some neighboring results differ by large, unexplained amounts. Could the authors retrain a few settings with different random seeds, and show training curves for the largest models?
We thank the reviewer for their advice. We provide our rebuttal as follows:
W1, Q1. One token per step.
W2, Q3. Run-to-run variation.
Q2. Training time.
You are right that the block-diffusion and E2D2 runs in Table 8 take longer per FLOP. This is an implementation artifact: unlike our ARM and MDM runs, our BD3LM and E2D2 implementations do not use the FlashAttention backend. With the same backend, we expect no such difference in training time at equal FLOPs. We will update the appendix to state this and avoid misleading readers.