acceptodds
Under review as a conference paper at ICLR 2027

Measuring Progress in Diffusion Language Model Pretraining

Abstract

In contrast to autoregressive models, it is an open question how to meaningfully compare training runs of diffusion language models (DLMs). In this paper, we propose an empirical protocol for choosing a performance metric for early DLM web pretraining, so that training runs can be compared by the time they take to reach a certain target. We ask three questions of each possible metric: 1) Does the metric rank training recipes robustly under slight changes to the performance target? 2) Does the metric consistently improve during DLM pretraining, so that several options for setting a target are possible? And 3) Does the metric recover the known rankings of autoregressive models, whose text likelihood can be evaluated directly? We apply this protocol to the following metrics from the DLM literature: accuracy in several downstream tasks, generative perplexity and bits per byte, distributional similarity, decoded-text validity, repetition and diversity. We do so across seven popular masked and uniform DLM training recipes. In our experiments, downstream performance provides the clearest stable target for tracking progress. Generative perplexity and bits per byte also give comparatively stable rankings and consistently improve, but every tested autoregressive reference model incorrectly ranks at least one pair from the GPT-2 family (even after entropy matching). MAUVE also improves persistently and gives stable local rankings, but reverses two GPT-2 comparisons. Unigram and bigram JS preserve the entropy-matched AR orders tested here, although their local DLM rankings change more often. Validity and repetition mainly change early in training or favor random text, so they do not provide natural stopping targets. Alongside diversity measures, they remain useful checks on the DLM's samplers. Based on our findings, we release an open-source speedrun built around downstream HellaSwag accuracy, one of the metrics that meets our checks most consistently here.

Primary area:
generative models
Keywords:
diffusion language models, evaluation metrics, pretraining efficiency, benchmarking

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AYgFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how to measure pretraining progress across different diffusion language modeling recipes, for which exact likelihoods are intractable and cannot be directly compared across modeling methods. The paper considers stability, monotonicity, and recovery when evaluating different models, aiming to establish fair comparisons between diffusion language models. Furthermore, this work finds that HellaSwag can serve as a stable, common metric for diffusion language models and introduces the HellaSwag speedrun to guide the early training.

Critical strengths
  1. Important research topic. Diffusion language models are emerging as a new modeling paradigm, and exploring how to scale them is valuable.

  2. The paper conducts many experiments to validate its claims, including training models with different random seeds, which makes the experimental results more reliable.

Critical weaknesses
  1. The writing and presentation of the paper are poor. Although this work presents many technical findings, it does not organize them well. For example, the abstract lists many key experimental results, but as a reader, I cannot gain a clear understanding of the main motivation and insights. The paper needs polishing. Furthermore, the citations are too coarse-grained: many are placed at the ends of paragraphs, making it difficult to identify the most relevant reference for a particular statement.

  2. This work introduces the HellaSwag speedrun without clearly explaining why the benchmark and threshold of 27.5% was chosen. In Appendix F.1, the authors state that scoring 2.5 percentage points above random guessing corresponds to about 251 additional correct answers and refer to the nanoGPT speedrun, but this still does not directly explain the choice of 27.5%.

  3. The study does not seem to provide clear evidence about how to evaluate the scaling ability of different diffusion language models. Reaching 27.5% on HellaSwag does not imply that a model is better. What would happen if we continued training these models longer? Would the trends observed in the HellaSwag speedrun persist? I'm primarily interested in the scaling ability of models. I think the work would be stronger if the authors could demonstrate that the observed scaling trends extend to larger model sizes or greater computational budgets.

Questions for authors
  1. Could you explain the choice of metric for the speedrun? HellaSwag may be more stable than PIQA and ARC-Easy, but we could establish a systematic evaluation metric covering knowledge, mathematics, and coding tasks, or use average accuracy across a wide range of benchmarks to evaluate models. Furthermore, the nanoGPT speedrun may test different optimizers and architectures within the same autoregressive modeling paradigm. For diffusion language models, however, evaluating masked and uniform diffusion language models using the same protocol with a 27.5% HellaSwag threshold may not be appropriate for different modeling methods.

  2. As discussed in Weakness 3, could the authors provide more evidence that the early rankings in the HellaSwag speedrun remain consistent when the models are trained on more tokens or scaled to larger sizes?

1 comment

Thank you for the feedback on the presentation. In the revision, we brought the motivation forward in the abstract and placed citations next to the claims they support.

“without clearly explaining why the benchmark and threshold of 27.5% was chosen”

Right, in general, any threshold would be somewhat arbitrary. Originally, the nanoGPT targets were also adjusted to what reasonable runs could reach. We were looking for a task that shows progress early and is cheap enough to evaluate repeatedly. With four endings per question, chance gives 25%. On the 10,042 validation questions, 27.5% means about 251 extra correct answers, nearly six standard deviations above random guessing. The fastest recipe reaches it in about 12 minutes. The ranking also doesn't critically depend on the exact target value: masked recipes come out ahead of uniform ones at every target we tested, from 26.5% to 28%. In the revision, we moved this justification from Appendix F.1 into the main text.

“Could you explain the choice of metric for the speedrun? […] average accuracy across a wide range of benchmarks”

Yes, and we first want to stress that the main contribution is the audit. It asks what properties a progress metric should have at a small scale, and checks which metrics have them. The speedrun is one concrete instantiation built on the audit's results. For simplicity, we built it on a single metric. We chose HellaSwag because it is interpretable and was a strong contender on all three criteria in our setting. Other variants are possible, too. As our limitations section notes, a well-chosen mixture of metrics may be better in the long run, and our audit can be used to evaluate such mixtures directly.

“evaluating masked and uniform diffusion language models using the same protocol […] may not be appropriate”

This is an important point. Each model uses its own loss to pick one of the four endings, and across recipes we only compare how often that pick is correct. Raw losses are never compared across recipes, so differences in how tight each bound is matter much less. This is also the standard protocol for downstream evaluation of DLMs. Since our audit is about existing metrics, we used it as is. Designing protocols specific to each recipe is an interesting direction, which currently sits beyond the scope of this paper.

“What would happen if we continued training these models longer? […] scaled to larger sizes?”

We did the check in Appendix F.3.1 using the runs from the speedrun that went over the budget cap used in the audit. We compared their average HellaSwag accuracy at the saved 30-, 40-, 50-minute checkpoints and so on, whenever at least two recipes remained. Their order agreed with the speedrun ranking in 94 of 95 pairwise comparisons. The only exception was a near-tie between D3PM mask and SEDD mask at 40 minutes, differing by 0.02 percentage points. These comparisons are all at the same model size. We completely agree that testing whether the ordering carries over to larger models would be a super valuable extension.

Reviewer_NG6fOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines how to measure progress during early pretraining of diffusion language models, where likelihood-based losses are difficult to compare across training recipes. The authors propose candidate metrics using 3 checks: whether time-to-target rankings are stable under small changes in the target, whether the metric improves over training, and whether it preserves likelihood-based rankings within established autoregressive model families.

The metric covers downstream accuracy, reference-model scores, and generated-text statistics across 7 DLM training recipes under same setup. Several downstream tasks pass all three checks, whereas generative perplexity and some distributional metrics fail to recover the autoregressive reference ordering. The authors then release a HellaSwag-based pretraining speedrun and report that masked recipes reach the target sooner in their experiments.

Critical strengths

S1. The paper addresses a practical problem in DLM research: how to compare early training progress when the native objectives are not directly comparable. The motivation is consistent with known limitations of likelihood comparisons across diffusion families [R1] and entropy-sensitive sample-based evaluation [R2]. More importantly, the authors test their proposed criteria systematically, with shared data, aligned checkpoints, repeated training runs, and documented implementation choices. This makes the results easier to inspect and reuse.

S2. A usable benchmark with informative negative findings: The released code of speedrun turns the audit into a concrete benchmark rather than stopping at a metric comparison. It uses training seeds separate from the audit, verifies the final crossing points with 128 corruptions, and checks rankings at nearby targets. I also find the negative results useful, particularly the failure of generative perplexity and MAUVE to preserve certain autoregressive-model orderings after entropy matching. These findings should help practitioners choose what to monitor during early DLM development.

[R1] Sahoo, S. S., et al. (2026). Scaling Beyond Masked Diffusion Language Models. ICML 2026. https://proceedings.mlr.press/v306/sahoo26a.html

[R2] Pynadath, P., et al. (2026). Generative Frontiers: Why Evaluation Matters for Diffusion Language Models. arXiv:2604.02718. https://arxiv.org/abs/2604.02718

Critical weaknesses

W1. The recovery criterion may be overly strict: The recovery check requires every metric to preserve the FineWeb likelihood ordering within autoregressive model families. I am not fully convinced that this should be a strict requirement for downstream accuracy. For example, Table 9 shows that ARC-Easy norm slightly reverses the Llama 3.1 ordering, while its unnormalized version preserves it. The paper treats such reversals as failures without reporting uncertainty in the downstream comparisons. It would be useful to distinguish reliable reversals from small or uncertain differences and examine whether this changes the selected metrics. More importantly, the authors should clarify why a disagreement with FineWeb likelihood necessarily makes a downstream task unsuitable for tracking early DLM pretraining.

W2. The stability check relies on interpolated training curves: Stability is one of the main criteria used to select a speedrun metric. However, the analysis relies on linear interpolation between only ten evaluated checkpoints per recipe (Appendices B.3 and D.1). This may hide short-term fluctuations or ranking changes between observed checkpoints. Since stability plays an important role in the metric selection, I would like to see whether the conclusions are sensitive to the interpolation method or checkpoint spacing. The more fine-grained HellaSwag evaluations from the speedrun could provide a useful check.

Questions for authors

Q1. Why should agreement with the FineWeb likelihood ordering be a strict requirement for downstream evaluation metrics? For example, ARC-Easy norm reverses the Llama 3.1 ordering, while its unnormalized version preserves it (Table 9). Could this reflect differences in what the metrics measure rather than a failure to track training progress? Would relaxing this requirement change the final metric selection?

Q2. The stability analysis relies on linear interpolation between ten evaluated checkpoints per recipe (Appendices B.3 and D.1). How sensitive are the stability results to this choice? Could the authors use the more finely evaluated HellaSwag trajectories from the speedrun as a sanity check, or otherwise show that the interpolation does not materially affect the metric selection?

Q3. How do the three checks relate to FineTasks [R3], Signal and Noise [R4], and DataDecide [R5]? Do the longer trajectories already available in this study offer any indication that the early finish-line ranking remains informative later in training?

[R3] Kydlíček, H., et al. (2024). Scaling FineWeb to 1000+ Languages: Step 1 — Finding Signal in 100s of Evaluation Tasks (FineTasks). Hugging Face research blog. https://huggingface.co/spaces/HuggingFaceFW/blogpost-fine-tasks

[R4] Heineman, D., et al. (2025). Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation. NeurIPS 2025. https://papers.neurips.cc/paper_files/paper/2025/hash/18b3b2947174ae9b1a8fc91ff89f4eb6-Abstract-Conference.html

[R5] Magnusson, I., et al. (2025). DataDecide: How to Predict Best Pretraining Data with Small Experiments. ICML 2025. https://proceedings.mlr.press/v267/magnusson25a.html

1 comment

Thank you for the constructive review and the concrete suggestions.

"Why should agreement with the FineWeb likelihood ordering be a strict requirement [...]?"

Our reasoning here is as follows: First: Monotonicity and stability only look at the shape of the curves. They say nothing about what the curves measure. A metric can be perfectly monotone and stable and still rank models for reasons unrelated to quality. This calls for an external grounding that anchors the audit in a case where we know how recipes compare. Second: However, there are not many universally agreed-upon comparisons of early pretraining quality, even in the autoregressive case. The absolute standard for modern language models here is text likelihood (or, equivalently, cross-entropy loss). This is why we choose recovering the likelihood-based ordering as the minimum requirement, which is the one that would very likely reach the highest consensus among researchers.

In practice, dropping this recovery check entirely would leave GenPPL, BPB and MAUVE in contention on the strength of their training curves alone. We believe that it is important to keep the check (and perhaps distinguish a clear disagreement from a close call).

"distinguish reliable reversals from small or uncertain differences"

Fair point, we distinguished the two in the revised version (Appendix E.3). MNLI shows a clear Pythia 160M/410M reversal. For the other tasks with a reversal, we quantify the uncertainty (five of the seven reversals have 95% intervals allowing either order, we report those intervals) and propose to retain only ARC-Easy norm, while leaving out of the picture BLiMP and direct MNLI since they are also less promising with respect to the other criteria (their DLM scores improve less consistently). Table 10 gives the intervals.

“linear interpolation between only ten evaluated checkpoints […] sensitive to the interpolation method or checkpoint spacing”

We checked both and added both sensitivity checks in Appendix E.1.2.

  • Spacing. On the ten-seed HellaSwag runs, we compared all 18–42 measured checkpoints per recipe with a subset of 9–10. Thinning the grid moved the fraction of comparable pairs changing order from 2.1% to 3.5%. Coverage stayed close to 90%.
  • Interpolation. Across the 40 metrics in the paper, we replaced linear interpolation with shape-preserving cubic interpolation on each seed’s curve before averaging. Changes were small, averaging 0.36 percentage points for order changes and 0.88 points for coverage. The corresponding rank correlations across metrics were 0.953 and 0.995.

“How do the three checks relate to FineTasks, Signal and Noise, and DataDecide?”

Thank you for the pointers, we now discuss all three works in Appendix A. These works study autoregressive models. For diffusion models, exact sequence likelihood is generally intractable, which is why we use AR families for the likelihood check.

  • FineTasks also checks monotonicity, noise, performance above chance and consistency of model orderings between consecutive checkpoints. Our stability check asks whether the same recipes still get first to a target finish line when it moves a little. FineTasks
  • Signal and Noise measures how well a benchmark separates models relative to fluctuations across late checkpoints. It also checks whether the order of training-data recipes at small scale agrees with their order at larger scale. It is independent and complementary to our stability check that verifies if a ranking of methods is a stable signal with respect to the perturbations of the finish line that takes into account the noise during training. Signal and Noise
  • DataDecide directly tests whether small experiments identify the pretraining data that work best for larger models. DataDecide

So these works apply similar checks in spirit, with differences here and there in the methodology, with the main difference that they are applied to autoregressive models. Our contribution is to audit the case of DLMs, where the likelihood is not available exactly. That gap is what motivates our audit for DLMs.

Reviewer_Tau3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies common diffusion LM evaluation metrics (downstream accuracy, GenPPL, MAUVE, etc) against a three-part protocol (stability, monotonicity during training, AR-model size consistency) for ranking training recipes in the context of a small-scale (170M parameters) speedrun. The paper dismisses the ELBO as a metric, and finds that sample-based metrics like GenPPL and MAUVE misorder GPT-2 sizes even after entropy matching. The paper proposes a speedrun with a finish line of 27.5% HellaSwag accuracy.

Critical strengths

The critical study of evaluation metrics for small-scale diffusion metrics has some value. The community is well aware of the deficiencies of these metrics, so the primary contribution of this study is to quantify and analyze the problems with these metrics. I broadly agree that these metrics have problems and I appreciate the effort to systematically document these deficiencies.

Critical weaknesses

The paper dismisses the ELBO in words, but doesn't empirically evaluate it using the methodology it applies to every other metric. The ELBO is the most natural metric for a speed run (it is easy to compute, and most closely related to the optimization objective) so the burden is on the authors to prove that it is inadequate. Every model's ELBO is an upper bound on the NLL, so the only confounding factor is the looseness of the bound. This looseness can vary a bit between model families, but I would need evidence that looseness is a problem wrt to the methodology described in this paper.

27.5% accuracy on HellaSwag is a strange target. HellaSwag is a multiple-choice benchmark, so random guessing gets 25% accuracy. The "finish line" for this speed run is a performance just barely better than random chance. HellaSwag test set contains ~10k examples so technically this is statistically significant, but it seems like more of a binary indicator that the model has begun learning rather than a meaningful threshold on a continuous accuracy target. Is that the right way to think about this? It seems unlikely that this target can resolve the small performance differences that a speedrun is typically used to tease out. It looks to me like this only passes the stability check because the differences in training recipes studied in the paper are so large.

There's no consideration of scaling laws here, which are a critical methodological tool for learning from the behavior of small models. This is more of a critique of the nanoGPT speedrun itself, but the present work adopts this evaluation setting and is subject to the same critique. Near-chance HellaSwag accuracy is a poor target for fitting a scaling law, and the burden is on the authors to argue that modeling choices made against this benchmark can be predictably scaled.

Questions for authors

Doesn't HellaSwag's scoring rule in Eq. (1) use the model ELBO? How does this metric avoid your stated concerns about ELBO looseness?

1 comment

Thank you for the careful review. We agree with both of your main points (ELBO and scaling laws), and we address them below.

"dismisses the ELBO in words, but doesn't empirically evaluate it using the methodology it applies to every other metric"

We agree, and we have now evaluated the ELBO on the full audit grid (Section 3.2 and C.2.1) and made light additions to the relevant places in the revised paper. It does well. It improves across checkpoints after averaging seeds (Table 8 and 10), which is expected since it is closely related to the training objective. It is cheap to compute, and its rankings are stable when we vary the target (Figure 1) and across the estimation budgets we tested (Appendix C.2.1). The recovery check on a family of AR models doesn't apply directly, because the bounds are recipe-specific.

In this setup, ELBO rankings also show some agreement with downstream rankings (Figure 14). The Spearman correlations are 0.67 for LAMBADA, 0.46 for HellaSwag, 0.31/0.52 for ARC-Easy no-norm/norm, and 0.30/0.37 for PIQA no-norm/norm. Agreement is stronger with Gen. PPL (0.80–0.84 across scorers) and BPB (0.68–0.69), although their AR reversals limit what we can conclude from this. We therefore retain ELBO as a useful progress metric on the strength of its improvement, stability and low cost.

"Doesn't HellaSwag's scoring rule in Eq. (1) use the model ELBO? How does this metric avoid your stated concerns about ELBO looseness?"

Yes, it does. However, the ELBO is only used as a score to pick the argmin among the candidate endings of a single context, and absolute ELBO values are never compared across recipes or across questions (which is one motivation for adding this in the audit and comparing it to the ELBO under our three checks: in an ideal world, if a recipe's looseness gap is the same amount per character across the endings of a question, the argmin picks the same ending that the true likelihood would, which avoids comparing different ELBOs with potentially different gaps across questions and across recipes). This is one reason for which one might hope that downstream accuracy are less directly impacted by different bound tightness across recipes (but, of course, this cannot be checked directly). Here, we decided to keep HellaSwag for one simple instantiation of the speedrun with a number having a direct task interpretation.

"27.5% accuracy on HellaSwag is a strange target [...] more of a binary indicator that the model has begun learning [...] unlikely that this target can resolve the small performance differences"

We partly agree, and we would not monitor runs on HellaSwag alone either. Our claim is not that HellaSwag accuracy is an excellent metric. It is that the other metrics we audited do not especially do better on our three criteria (monotonicity, stability, recovery). On the threshold itself (App. F.1), any threshold is somewhat arbitrary. We intended 27.5 as an early milestone: It is 10% above the 25% guessing rate. For context, autoregressive nanoGPT speedrun models scored around 29.5–30% on HellaSwag at their validation-loss finish line. The fastest recipe reaches it in about 12 minutes (Figure 3a), short enough to iterate. In Figure 8, perturbing the HellaSwag target changed fewer than 1% of comparable recipe-pair orders at every calibration budget from 10 to 20 minutes (Figure 8). In the separate ten-seed speedrun, the three masked recipes took the first three places at every tested target from 26.5% to 28% (Figure 3).

"There's no consideration of scaling laws here"

Ideally we’d compare training FLOPs against quality across model sizes, but such an experimental study takes considerably more computation. Importantly, the question of which quality metric to use in such a study would remain. The latter is exactly what we investigate here, at a small scale. Does the metric keep improving with training? If we move the target a little, do the same recipes still get there first? (We translated our audit to quality-vs-training FLOPs instead of training time, and both ELBO and HellaSwag remained stable under nearby targets.) Our premise is that a cheap setup has value for iteration. We’ll make it clearer that the speedrun is just one illustration of the metric audit, that this audit also supports several reasonable choices, and that testing transfer across model sizes is the next step.

To give at least some idea how our findings would scale with training time (Appendix F.3.1): We compared the recipes still running every ten minutes from minute 30 onward. Their order by HellaSwag accuracy agreed with their 27.5% finish order in 94 of 95 pairwise comparisons. At about 9.8B tokens, for example, SEDD mask still led SEDD uniform, 30.3% versus 27.7% across ten runs each. These comparisons are all at the same model size. We acknowledge that testing whether the ordering carries over to larger models would be valuable and would require training at several sizes.

Comments on the paper

1 comment

Thank you again for the feedback. We have uploaded a revised paper addressing the reviewers’ comments. In particular, we added held-out negative ELBO to the metric audit, reported uncertainty in the comparisons with autoregressive likelihood rankings, and checked sensitivity to checkpoint spacing and interpolation. We also clarified the choice of 27.5% HellaSwag accuracy as the speedrun target and examined whether the recipe rankings persist with longer training. Our replies below give the results and point to the relevant sections.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.