- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies how to measure pretraining progress across different diffusion language modeling recipes, for which exact likelihoods are intractable and cannot be directly compared across modeling methods. The paper considers stability, monotonicity, and recovery when evaluating different models, aiming to establish fair comparisons between diffusion language models. Furthermore, this work finds that HellaSwag can serve as a stable, common metric for diffusion language models and introduces the HellaSwag speedrun to guide the early training.
Critical strengths
-
Important research topic. Diffusion language models are emerging as a new modeling paradigm, and exploring how to scale them is valuable.
-
The paper conducts many experiments to validate its claims, including training models with different random seeds, which makes the experimental results more reliable.
Critical weaknesses
-
The writing and presentation of the paper are poor. Although this work presents many technical findings, it does not organize them well. For example, the abstract lists many key experimental results, but as a reader, I cannot gain a clear understanding of the main motivation and insights. The paper needs polishing. Furthermore, the citations are too coarse-grained: many are placed at the ends of paragraphs, making it difficult to identify the most relevant reference for a particular statement.
-
This work introduces the HellaSwag speedrun without clearly explaining why the benchmark and threshold of 27.5% was chosen. In Appendix F.1, the authors state that scoring 2.5 percentage points above random guessing corresponds to about 251 additional correct answers and refer to the nanoGPT speedrun, but this still does not directly explain the choice of 27.5%.
-
The study does not seem to provide clear evidence about how to evaluate the scaling ability of different diffusion language models. Reaching 27.5% on HellaSwag does not imply that a model is better. What would happen if we continued training these models longer? Would the trends observed in the HellaSwag speedrun persist? I'm primarily interested in the scaling ability of models. I think the work would be stronger if the authors could demonstrate that the observed scaling trends extend to larger model sizes or greater computational budgets.
Questions for authors
-
Could you explain the choice of metric for the speedrun? HellaSwag may be more stable than PIQA and ARC-Easy, but we could establish a systematic evaluation metric covering knowledge, mathematics, and coding tasks, or use average accuracy across a wide range of benchmarks to evaluate models. Furthermore, the nanoGPT speedrun may test different optimizers and architectures within the same autoregressive modeling paradigm. For diffusion language models, however, evaluating masked and uniform diffusion language models using the same protocol with a 27.5% HellaSwag threshold may not be appropriate for different modeling methods.
-
As discussed in Weakness 3, could the authors provide more evidence that the early rankings in the HellaSwag speedrun remain consistent when the models are trained on more tokens or scaled to larger sizes?
Thank you for the feedback on the presentation. In the revision, we brought the motivation forward in the abstract and placed citations next to the claims they support.
Right, in general, any threshold would be somewhat arbitrary. Originally, the nanoGPT targets were also adjusted to what reasonable runs could reach. We were looking for a task that shows progress early and is cheap enough to evaluate repeatedly. With four endings per question, chance gives 25%. On the 10,042 validation questions, 27.5% means about 251 extra correct answers, nearly six standard deviations above random guessing. The fastest recipe reaches it in about 12 minutes. The ranking also doesn't critically depend on the exact target value: masked recipes come out ahead of uniform ones at every target we tested, from 26.5% to 28%. In the revision, we moved this justification from Appendix F.1 into the main text.
Yes, and we first want to stress that the main contribution is the audit. It asks what properties a progress metric should have at a small scale, and checks which metrics have them. The speedrun is one concrete instantiation built on the audit's results. For simplicity, we built it on a single metric. We chose HellaSwag because it is interpretable and was a strong contender on all three criteria in our setting. Other variants are possible, too. As our limitations section notes, a well-chosen mixture of metrics may be better in the long run, and our audit can be used to evaluate such mixtures directly.
This is an important point. Each model uses its own loss to pick one of the four endings, and across recipes we only compare how often that pick is correct. Raw losses are never compared across recipes, so differences in how tight each bound is matter much less. This is also the standard protocol for downstream evaluation of DLMs. Since our audit is about existing metrics, we used it as is. Designing protocols specific to each recipe is an interesting direction, which currently sits beyond the scope of this paper.
We did the check in Appendix F.3.1 using the runs from the speedrun that went over the budget cap used in the audit. We compared their average HellaSwag accuracy at the saved 30-, 40-, 50-minute checkpoints and so on, whenever at least two recipes remained. Their order agreed with the speedrun ranking in 94 of 95 pairwise comparisons. The only exception was a near-tie between D3PM mask and SEDD mask at 40 minutes, differing by 0.02 percentage points. These comparisons are all at the same model size. We completely agree that testing whether the ordering carries over to larger models would be a super valuable extension.