- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors tackle the setting of pretraining continuous diffusion language models with a downstream evaluation of mathematical reasoning. They combine three existing methods, ELF, REPA, and REG, into a pretraining recipe that performs well in absolute terms, e.g., their largest ~800M parameter model achieves 13.39% MATH pass@1. They also provide favorable comparisons to dLMs of similar scale, although these comparisons are not training FLOP-matched. Lastly, they find that their ELF-REG recipe supports early stopping by decoding tokens from intermediate continuous states without any explicit few-shot distillation.
Critical strengths
- The most encouraging part of this paper is that it could serve as a "hero run", showing that combining a bag of tricks such as ELF, REPA, and REG together yields a model that is closer to the ballpark of decent AR LLMs. e.g., 13.39% MATH pass@1, compared to 24.8% for Llama-3.2-1B-Instruct.
- It is interesting that the authors find that early exiting works even without dedicated few-shot distillation. Ultimately, few-shot inference is one of the main motivations for exploring (continuous) diffusion LMs at all, so this is a nice result.
Critical weaknesses
- The critical weakness here is that the comparisons are not compute-controlled. This leads to a number of questions: (1) compared to the strongest baseline recipe (such as PlaidQ+CFG), how does ELF-REG compare (at various inference budgets) when one carefully controls training FLOPs? The authors provide the "Task-training tokens" in Fig. 1(d), but this is not a standard approach to comparing pretrained LMs and so I have a hard time buying this as a replacement for an experiment that controls data and compute. This is ultimately the relevant experiment in deciding whether one pretraining recipe is better than another. (2) Similarly, it is unclear if the variant with REG+REPA is even better than ELF alone, when one adjusts for the extra training FLOPs of REG+REPA. Given that this paper combines existing methods ELF+REG+REPA and therefore provides little methodological novelty, I place more weight on this experiment setup being sound.
- I found the writing in places to be extremely terse and AI-sounding. e.g., the writing in Sections 3.2 and 3.3.
Questions for authors
- Can you fix a data distribution and a set of FLOP counts, and build pretraining scaling laws for ELF-REG, ELF, and the strongest baseline (e.g., PlaidQ)? This approach has been taken in the continuous diffusion literature since Plaid [1], and has been standard in language modeling since Kaplan et al. [2]. e.g., you could follow the isoFLOP analysis of Chinchilla [3]. This would be the proper way to compare scaling recipes and would make the tentative results here sound.
- Can you take a pass through the paper and clean up the writing, which sounds terse and AI-generated in many places such as S3.2, S3.3?
[1] Ishaan Gulrajani, Tatsunori B Hashimoto. Likelihood-Based Diffusion Language Models. Advances in Neural Information Processing Systems 36 (NeurIPS 2023). [2] Jared Kaplan et al. Scaling Laws for Neural Language Models, 2020. https://arxiv.org/abs/2001.08361 [3] Jordan Hoffman et al. Training Compute-Optimal Large Language Models, 2022. https://arxiv.org/abs/2203.15556
4 comments
Response to reviewer Xn8R (2)
We provide a comprehensive comparison of ELF-REG with baselines regarding training and inference FLOPs, consistently showing superior ELF-REG performance at lower compute. On GSM8K, ELF-REG-B achieves higher accuracy with fewer training and inference FLOPs. On code, ELF-REG-L achieves higher accuracy with fewer inference FLOPs. These comparisons are also included in Table 32 in the updated manuscript.
GSM8K:
| Model | NFE | Pass@1 (%) | Training FLOPs () | Inference TFLOPs/answer |
|---|---|---|---|---|
| ELF-REG-B, epoch 9 | 16 | 37.91 | 5.081 | 4.691 |
| FMLM+ (Init) | 64 | 33.60 | 13.993 | 11.616 |
| MDLM | 1024 | 33.90 | 5.196 | 138.555 |
| Duo | 1024 | 36.02 | 5.196 | 138.554 |
| S-FLM (top-1) | 512 | 18.18 | 5.195 | 69.272 |
ELF training FLOPs include one-time encoder/teacher feature preparation, assuming we generate text features from Qwen encoder/teacher and reuse them.
Code (HE = HumanEval; HE+ = HumanEval+):
| Model | NFE | HE (%) | HE+ (%) | MBPP-378 (%) | TFLOPs/HE | TFLOPs/MBPP-378 |
|---|---|---|---|---|---|---|
| ELF-REG-L, epoch 12 | 16 | 24.24 | 23.17 | 27.18 | 24.954 | 24.954 |
| Open-dCoder | 128 | 20.80 | 17.60 | 23.90 | 33.946 | 31.719 |
| oDLM | 128 | 17.87 | 16.40 | 22.99 | 42.127 | 39.323 |
| PlaidQ | 129 | 19.02 | 18.05 | 22.18 | 42.654 | 39.816 |
| PlaidQ+CFG | 257 | 22.04 | 19.70 | 24.58 | 84.977 | 79.324 |
ELF-REG uses consistency sampling, and its inference FLOPs include the frozen encoder and final decoding. Accuracies are averaged over 4 generation seeds, using EvalPlus for code evaluation. Baseline accuracies are values extracted from the papers and denotes estimated training/inference costs or accuracy read from figures. Bold marks the highest accuracy or lowest compute in each column.
For example, ELF-REG-L exceeds PlaidQ+CFG on all three code metrics at NFE 16 compared to PlaidQ+CFG's NFE 257, while using fewer inference FLOPs.
[W2] [Q2] terse and AI-sounding writing.
We recognise that section 3 is more terse than intended and we have edited section 3 to make the writing sound less terse. We experimented with writing section 3 using a LLM, but it was too long so we wrote it from scratch. The terseness was due to aggressive space saving, and we should have written more natural prose instead.
[1] Likelihood-Based Diffusion Language Models, https://arxiv.org/pdf/2305.18619
[2] Training Compute-Optimal Large Language Models, https://arxiv.org/pdf/2203.15556
[3] Scaling Laws Are Unreliable for Downstream Tasks: A Reality Check, https://arxiv.org/abs/2507.00885
[4] Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution, https://arxiv.org/pdf/2310.16834
[5] Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models, https://arxiv.org/pdf/2503.09573
[6] The Diffusion Duality, https://arxiv.org/pdf/2506.10892
I thank the authors for running a compute-controlled pretraining experiment. Unfortunately the setup is still quite non-standard for LLM pretraining, where it appears the authors perform epoched pretraining on GSM8K directly rather than pretraining on a standard corpus such as DCLM or FineWeb. I therefore have a hard time trusting that their approach actually generalizes to relevant pretraining settings, in which models are pretrained from scratch on mixes of web and high-quality data, and later finetuned/post-trained on tasks such as GSM8K.
Given that this paper is a recipe combining several established methods in the literature, and hence has limited methodological novelty, I believe the bar should be higher for a sound empirical evaluation. Because of the issues in soundness above, I cannot justify giving an accept recommendation.
We thank the reviewer for the response. We don't claim generalisation to general-corpus LLM pretraining, which is beyond our available compute. Instead, we study task-specific math reasoning and code generation with fully continuous dLMs. We follow existing works in training on task-specific data instead of general-corpus pretraining, including S-FLM, FMLM+, and DBTM [1-3], which we already cited. The table contains recent papers that do GSM8K task-specific training:
| Study | General pretraining? | Benchmark Task |
|---|---|---|
| S-FLM [1] | No | GSM8K |
| FMLM+ [2] | No | GSM8K |
| DBTM [3] | No | GSM8K |
| DoT (from scratch) [4] | No | GSM8K |
| CoBit-S [5] | No | GSM8K |
| Simplex Diffusion Models [6] | No | GSM8K |
| AlphaDLM [7] | No | GSM8K |
| ClockDLM [8] | No | GSM8K |
| DoubleCat / SemiCat [9] | No | GSM8K |
| H-CoBit [10] | No | GSM8K |
The reviewer cited Plaid [11], which performs isoFLOP analysis of language-model likelihood without directly evaluating math/code reasoning. Moreover, [12] shows that better perplexity doesn't reliably translate into better downstream performance. We therefore measure reasoning accuracy directly when controlling for training FLOPs.
Our primary contribution is establishing an effective recipe for training fully continuous dLMs for reasoning that achieves higher GSM8K accuracy with fewer training FLOPs than ELF baseline, S-FLM (top-1) [1], and FMLM+ (Init) [2]. REPA supplies supervision at individual token positions, while REG requires the shared denoiser to predict a representation of the entire prompt and response. With consistency sampling at NFE 64, ELF-REG-L reaches 45.26% at 4.66e19 training FLOPs, exceeding ELF-L baseline's 43.33% at 6.65e19.
The reasoning-specific motivation for early-stop is that obtaining the correct answer does not require every reasoning token to stabilize. ELF-REG-B reaches 37.64% accuracy at NFE 64 versus 38.29% at NFE 512, although complete-response agreement is only 8.61% (Table 11). Thus, one eighth of the NFE budget already provides near-final reasoning accuracy without dedicated few-step training. This concerns answer correctness, which unconditional generation perplexity does not measure.
[1] Language Modeling with Hyperspherical Flows, https://arxiv.org/abs/2605.11125
[2] Posterior Refinement: Fast Language Generation via Any-Order Flow Maps, https://arxiv.org/abs/2606.24773
[3] Discrete Beckmann Transport Models for One-Step Language Modeling and Reasoning, https://arxiv.org/abs/2609.15903
[4] Diffusion of Thoughts: Chain-of-Thought Reasoning in Diffusion Language Models, https://arxiv.org/abs/2402.07754
[5] CoBit: Language Modeling with Bitstream Diffusion, https://arxiv.org/abs/2605.07013
[6] Simplex Diffusion Models, https://arxiv.org/abs/2609.35553
[7] Alpha Diffusion Language Models: Factorization Alone Is Not the Problem, https://arxiv.org/abs/2609.38066
[8] Clock Diffusion: Efficient Semi-Autoregressive Continuous Diffusion Language Models, https://arxiv.org/abs/2610.00894
[9] On the Design Space of Continuous Diffusion Language Models, https://arxiv.org/abs/2610.13834
[10] Denoising Hierarchical Representations: Joint Continuous Diffusion for Language Modeling, https://arxiv.org/abs/2610.08738
[11] Likelihood-Based Diffusion Language Models, https://arxiv.org/abs/2305.18619
[12] Scaling Laws Are Unreliable for Downstream Tasks: A Reality Check, https://arxiv.org/abs/2507.00885
Response to reviewer Xn8R (1)
We added consistency-sampler results, comparisons of accuracy at lower training and inference cost, and edited section 3 for readability in the updated version of the paper. Below, we report training and inference FLOPs, including the frozen encoder's inference cost, and GSM8K isoFLOP comparisons. Additions to the paper are coloured blue.
To further demonstrate the superior performance of our recipe, we added consistency-sampler results without additional training. Using the consistency sampler and same NFE, ELF-REG-L outperforms ELF-L baseline across all six benchmarks (Table 4). ELF-REG's highest pass@1 accuracy using the consistency sampler are 61.50% on GSM8K (NFE 128) and 20.15% on MATH-500 (NFE 128), compared with 58.40% and 16.85% for the ELF baseline. On HumanEval and HumanEval+, ELF-REG reaches 33.08% and 31.40% (both NFE 128), versus 28.35% and 26.83%. On MBPP-378 and MBPP-500, ELF-REG reaches 38.76% and 25.60% (both NFE 128), versus 36.51% and 24.55%. Please see Table 4 for more details.
[W1] [Q1] Training FLOPs- We add GSM8K isoFLOP comparisons.We agree that task-training tokens alone do not control training compute; we note that the ELF paper, which we build on, exclusively reports training tokens as a proxy for training compute. We have therefore added isoFLOP experiments comparing ELF baseline and ELF-REG on the same training distribution across four model sizes, together with training and inference FLOP comparisons.
Following your suggestions and the isoFLOP approach in Plaid [1] and Chinchilla [2], we evaluate four model sizes at four training FLOP budgets on our GSM8K training data (Figure 10). We evaluate performance using the consistency sampler with early stop at NFE 64. We report reference training FLOPs that count one full-dataset encoder/teacher feature-preparation pass per training run and assume reuse of those cached features.
ELF-REG improves accuracy at the same model size and training FLOPs in almost all cases. The table below reports the highest accuracy across the model sizes evaluated at each budget:
Each entry is the highest mean accuracy chosen from the model sizes evaluated at that budget (averaged over 4 generation seeds). Thus, the improvement persists after accounting for the extra training FLOPs of REPA+REG. We report reasoning accuracy directly because improvements in pretraining loss or NLL for scaling laws don't reliably predict downstream performance [3].
Figure 10 also includes base training loss and GPT-2-large response NLL. The response NLL scores ELF's generated responses under GPT-2-large, conditioned on the prompt and it is not ELF's own likelihood. GPT-2-large is also used to evaluate generated text in prior dLM papers, including SEDD [4], Block Diffusion [5] and Duo [6]. Our study compares ELF-baseline and ELF-REG at fixed training budgets and we don't extrapolate a general pretraining scaling law to larger models from these results.
The new efficiency comparisons also extend to MATH and code (Table 31). For example, ELF-REG-L achieves 17.40% on MATH-500 compared with 16.85% for ELF-L baseline, using 19.1% fewer training FLOPs and 12.3% fewer inference FLOPs.
We have not included PlaidQ in a scaling law experiment because the repository linked in PlaidQ is not accessible and returns HTTP 404. Without its implementation, we can't faithfully implement PlaidQ's training on same data and FLOP budgets, and we can't directly measure the training FLOP count. However, based on the architecture using dense attention, training settings, and other hyperparameters reported by PlaidQ, we estimate PlaidQ's training FLOPs to be approximately ≈22.82×1020 (we assume 380k optimiser steps, 512 batch size, and 11.731 TFLOPs per input), whereas ELF-REG uses 2.861×1020 FLOPs, almost an order of magnitude less compute than PlaidQ to achieve superior performance when considering training compute.
Additionally, we also estimate the inference FLOPs of PlaidQ from its architecture and sampling settings, including CFG. On HumanEval, ELF-REG-L achieves 24.24% at NFE 16, using 24.954 TFLOPs per answer, including the frozen encoder. PlaidQ+CFG achieves 22.04% at NFE 257 on HumanEval, with around 84.977 TFLOPs per answer. Therefore, ELF-REG out-performs PlaidQ+CFG at fewer inference FLOPs.