- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper provides an interesting analysis of why unconditional text generation by diffusion language models (DLMs) needs broader evaluation metrics than gen-PPL alone. The methods include analyzing the theoretical properties of gen-PPL and constructing four naive samplers that match or outperform trained DLMs on gen-PPL. Experiments evaluate DLMs with a profile of complementary measurements, including distributional discrepancies and blinded pairwise LLM judgments, which collectively identifies all four constructed failures. Overall, this paper clearly describes the problems with the gen-PPL metric and provides a practical evaluation approach.
Critical strengths
I summarize the notable strengths of the paper as follows:
- The topic is practical and timely. Gen-PPL is a major metric in the current evaluation of unconditional text generation by diffusion language models, and several works [1, 2] have shown that it can be unreliable and easy to hack. Thus, providing solutions to the problems of gen-PPL is useful and important for the fair evaluation of DLMs.
- The provided solution is simple and practical. As shown in Table 1, using multiple complementary metrics produces substantially different rankings from those implied by gen-PPL and clearly identifies the failures of the constructed samplers, providing a practical approach to better evaluation of DLMs.
- The constructed counterexamples illustrate the current limitations of diversity checks. The four training-free samplers distinguish different failure modes, including cross-sample collapse and artificial repetition. These examples make the difference between unigram entropy and joint sequence entropy clear and provide useful stress tests for evaluation metrics.
Critical weaknesses
I summarize the notable weaknesses of the paper as follows:
- The contribution lies more in systematic analysis than in identifying a previously unrecognized problem. This means that the core contribution of the paper is not identifying the problem for the first time, but systematically analyzing an acknowledged problem. Another work [1] also discusses the KL–entropy decomposition and ranking changes caused by temperature. Although the authors clearly acknowledge the community’s awareness of these problems and make further contributions, this may still weaken the novelty of the paper. Distinguishing the known conclusions from the new theoretical and empirical contributions would make the contributions clearer.
- The paper lacks a broader evaluation of the models after the ranking changes. The paper does not clearly establish to what extent the evaluation suite would improve the quality of the selected model/sampler. Providing stronger evidence connecting the rankings to human scores or downstream task performance would help address this problem.
- The human study is limited in scale. As mentioned by the authors in the limitations, the current human study would benefit from a larger scale. Currently, there are only 9 raters and 300 valid comparisons. Also, each time, the rater only looks at the first 35 words, which makes it difficult to assess long-range coherence and repetition. Thus, whether the current human study can support fine-grained model rankings and comparisons of text quality is still unclear.
Questions for authors
- Could you provide the downstream task performance of the selected models/samplers based on the ranking?
- Could you provide a sensitivity analysis of the text length used in the human study? For example, would the results change if we increased the excerpt length from the first 35 words to 100 words or more?
2 comments
Dear reviewer eCMH,
We are grateful for your time and insightful review of our submission. We hope that our rebuttal and revisions, particularly the substantially expanded human evaluation with 1,698 comparisons from 205 annotators, and the clarification of our contributions relative to prior work, have addressed your concerns.
As the end of the rebuttal period is approaching, we would be happy to answer any remaining questions. If our responses and additional experiments address your concerns, we would be grateful if you would consider updating your assessment.
We thank the reviewer for highlighting the practical importance of the evaluation problem and the usefulness of our counterexamples.
Relationship to Generative Frontiers
Thank you for bringing this work to our attention. We now credit its KL–entropy analysis in Related Work. Its proposed frontier comparison assumes that unigram entropy preserves joint-entropy rankings. Our constructions show that this assumption can fail within the entropy ranges used to compare trained models.
Figure 6(b) shows that this failure extends across a curve. On OpenWebText, one of our naive generators configurations achieve gen-PPL of 4.73–7.14 at token entropies of 5.30–5.55, below the reference-text score of 17.1 and every trained-model score in Table 1. A curve in the gen-PPL/token-entropy plane can therefore look favorable despite complete collapse. We also identify increasing repetition across samples in trained CoBit-M that token entropy misses.
These findings motivate our recommendation to include an across-sample diversity check such as Self-BLEU. For evaluating DLM efficiency, we also prioritize plotting the metric profile against NFE rather than temperature. This addresses a central motivation for DLMs: whether generation quality and diversity can be maintained with fewer model evaluations.
Human evaluation and excerpt length
We agree that our original study is too small for fine-grained rankings and that 35-word excerpts limit the assessment of coherence and repetition. We have therefore added an evaluation using MAUVE’s released human judgments: 1,698 retained comparisons from 205 annotators. This is now our primary human-alignment benchmark, with our smaller study retained as further evidence.
Downstream performance
We have not added downstream task experiments. Our focus is evaluating pretrained DLMs before task-specific adaptation, to help compare model families, study scaling, and select base models for further post-training. The added human evaluation provides stronger evidence that the recommended metrics reflect perceived text quality.