acceptodds
Under review as a conference paper at ICLR 2027

Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs New Metrics

Abstract

Diffusion language models (DLMs) have emerged as the leading non-autoregressive alternatives to language modeling. Because their likelihoods are intractable, generative perplexity (gen-PPL) has become the de facto standard for reporting unconditional generation quality: the length-normalized likelihood of generated samples under a frozen autoregressive scorer, typically paired with a per-sample unigram entropy guardrail. We give an information-theoretic account of why this metric is unsound. Scoring model samples places the joint entropy of the generator inside gen-PPL, making collapse optimal for any fixed scorer. The reported per-sample unigram entropy ignores token order and cross-position dependence and therefore cannot correct this failure. Four training-free naive samplers reach or beat trained-model gen-PPL on LM1B and OpenWebText at non-degenerate unigram entropy, while producing text that is incoherent by construction. We then rebenchmark recent DLMs using distributional discrepancies and diversity diagnostics, and compare their rankings with LLM judgments. As part of this evaluation, we introduce a compression-based discrepancy that fits frozen byte-level coders to real and generated samples and measures local coverage and fidelity in both directions. The resulting metric profile provides a more informative view of unconditional text generation. We argue that DLM progress should be evaluated using a set of metrics that expose different properties of generated text.

Primary area:
generative models
Keywords:
generative perplexity, evaluation, diffusion language models, distributional metrics, non-autoregressive generation, benchmark design, MAUVE, entropy, language model evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_eCMHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper provides an interesting analysis of why unconditional text generation by diffusion language models (DLMs) needs broader evaluation metrics than gen-PPL alone. The methods include analyzing the theoretical properties of gen-PPL and constructing four naive samplers that match or outperform trained DLMs on gen-PPL. Experiments evaluate DLMs with a profile of complementary measurements, including distributional discrepancies and blinded pairwise LLM judgments, which collectively identifies all four constructed failures. Overall, this paper clearly describes the problems with the gen-PPL metric and provides a practical evaluation approach.

Critical strengths

I summarize the notable strengths of the paper as follows:

  1. The topic is practical and timely. Gen-PPL is a major metric in the current evaluation of unconditional text generation by diffusion language models, and several works [1, 2] have shown that it can be unreliable and easy to hack. Thus, providing solutions to the problems of gen-PPL is useful and important for the fair evaluation of DLMs.
  2. The provided solution is simple and practical. As shown in Table 1, using multiple complementary metrics produces substantially different rankings from those implied by gen-PPL and clearly identifies the failures of the constructed samplers, providing a practical approach to better evaluation of DLMs.
  3. The constructed counterexamples illustrate the current limitations of diversity checks. The four training-free samplers distinguish different failure modes, including cross-sample collapse and artificial repetition. These examples make the difference between unigram entropy and joint sequence entropy clear and provide useful stress tests for evaluation metrics.

[1] https://arxiv.org/abs/2409.02908

[2] https://arxiv.org/abs/2503.00307

Critical weaknesses

I summarize the notable weaknesses of the paper as follows:

  1. The contribution lies more in systematic analysis than in identifying a previously unrecognized problem. This means that the core contribution of the paper is not identifying the problem for the first time, but systematically analyzing an acknowledged problem. Another work [1] also discusses the KL–entropy decomposition and ranking changes caused by temperature. Although the authors clearly acknowledge the community’s awareness of these problems and make further contributions, this may still weaken the novelty of the paper. Distinguishing the known conclusions from the new theoretical and empirical contributions would make the contributions clearer.
  2. The paper lacks a broader evaluation of the models after the ranking changes. The paper does not clearly establish to what extent the evaluation suite would improve the quality of the selected model/sampler. Providing stronger evidence connecting the rankings to human scores or downstream task performance would help address this problem.
  3. The human study is limited in scale. As mentioned by the authors in the limitations, the current human study would benefit from a larger scale. Currently, there are only 9 raters and 300 valid comparisons. Also, each time, the rater only looks at the first 35 words, which makes it difficult to assess long-range coherence and repetition. Thus, whether the current human study can support fine-grained model rankings and comparisons of text quality is still unclear.

[1] https://arxiv.org/abs/2604.02718

Questions for authors
  1. Could you provide the downstream task performance of the selected models/samplers based on the ranking?
  2. Could you provide a sensitivity analysis of the text length used in the human study? For example, would the results change if we increased the excerpt length from the first 35 words to 100 words or more?
2 comments

We thank the reviewer for highlighting the practical importance of the evaluation problem and the usefulness of our counterexamples.

Relationship to Generative Frontiers

Thank you for bringing this work to our attention. We now credit its KL–entropy analysis in Related Work. Its proposed frontier comparison assumes that unigram entropy preserves joint-entropy rankings. Our constructions show that this assumption can fail within the entropy ranges used to compare trained models.

Figure 6(b) shows that this failure extends across a curve. On OpenWebText, one of our naive generators configurations achieve gen-PPL of 4.73–7.14 at token entropies of 5.30–5.55, below the reference-text score of 17.1 and every trained-model score in Table 1. A curve in the gen-PPL/token-entropy plane can therefore look favorable despite complete collapse. We also identify increasing repetition across samples in trained CoBit-M that token entropy misses.

These findings motivate our recommendation to include an across-sample diversity check such as Self-BLEU. For evaluating DLM efficiency, we also prioritize plotting the metric profile against NFE rather than temperature. This addresses a central motivation for DLMs: whether generation quality and diversity can be maintained with fewer model evaluations.

Human evaluation and excerpt length

We agree that our original study is too small for fine-grained rankings and that 35-word excerpts limit the assessment of coherence and repetition. We have therefore added an evaluation using MAUVE’s released human judgments: 1,698 retained comparisons from 205 annotators. This is now our primary human-alignment benchmark, with our smaller study retained as further evidence.

Downstream performance

We have not added downstream task experiments. Our focus is evaluating pretrained DLMs before task-specific adaptation, to help compare model families, study scaling, and select base models for further post-training. The added human evaluation provides stronger evidence that the recommended metrics reflect perceived text quality.

Dear reviewer eCMH,

We are grateful for your time and insightful review of our submission. We hope that our rebuttal and revisions, particularly the substantially expanded human evaluation with 1,698 comparisons from 205 annotators, and the clarification of our contributions relative to prior work, have addressed your concerns.

As the end of the rebuttal period is approaching, we would be happy to answer any remaining questions. If our responses and additional experiments address your concerns, we would be grateful if you would consider updating your assessment.

Reviewer_Rf86OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper clearly explains the serious problems with gen-PPL as a scoring metric for diffusion language models (DLMs). The theoretical analysis is neat and concise. Experimentally, under the gen-PPL evaluation criterion, without training any model and using only simple statistics from the corpus, the authors construct four samplers (high-frequency word sampling, mirror copying, cyclic padding, and phrase concatenation) that “beat” advanced DLMs trained with considerable effort, thereby demonstrating the problems with gen-PPL. At the same time, they provide an alternative to gen-PPL, proposing a “metric combination” scheme for evaluating DLMs.

Critical strengths

A comprehensive re-evaluation of mainstream DLMs reveals several facts that gen-PPL had masked (for example, the model with the best gen-PPL ranks at the bottom on distribution metrics). These findings have practical reference value and can help researchers avoid being misled by a single metric.

Critical weaknesses

Using too many metrics means higher evaluation cost and a more complex pipeline; moreover, LLM judges themselves may also favor “predictable, safe” text, exhibiting a similar bias to gen-PPL. The paper recommends simultaneously reporting multiple metrics such as LLM judge Elo, unigram entropy, positional entropy, GM, MAUVE, ED, and JCD, and also plotting sampling budget curves. This substantially increases the experimental cost for researchers. The paper does not discuss in depth: if resources are limited, which metrics are “must-report” and which are “nice-to-have”? Nor does it conduct ablation experiments to demonstrate the marginal contribution of each metric.

Questions for authors
  1. When discussing alternative metrics, the paper mainly focuses on JCD and the family of distribution discrepancy metrics. However, the NLP community already has a large body of work on text generation evaluation (e.g., alternatives to BLEU, improvements to ROUGE, and BERTScore). Could the paper provide a more systematic review of the relationship between this work and the claims made in this paper?

  2. Regarding the cost issue, JCD in the metric combination is fitted based on a byte-level compressor (CTW), which theoretically does not rely on neural networks—this is an advantage. However, the computational overhead of CTW on large-scale text may be high, and the compressor is sensitive to text length. The paper does not discuss in detail the computational cost and feasibility of deploying JCD on large-scale datasets (e.g., at the Common Crawl level). By contrast, gen-PPL only requires a frozen GPT-2 for scoring and is very lightweight.

2 comments

We thank the reviewer for this valuable feedback. We are pleased that the reviewer found our work clear and the theoretical analysis neat and concise. We now address each point in turn.

Metric recommendations

Our original version emphasized metric limitations but could have been strengthened with clearer practical recommendations. To address this, we have revised Section 7 to make stronger recommendations based on human preference alignment and sample-count stability. Generating samples is the main computational bottleneck in our evaluation; the measured automatic metrics have median runtimes below 40 seconds at 1,024 texts per source. Metrics that remain informative with fewer samples are therefore particularly useful.

We have revised Section 7 to give a more focused recommendation: add Self-BLEU and Gradient Moment (GM) to the commonly reported gen-PPL and token entropy. LLM judging is optional when resources permit.

The recommendation follows two findings. First, our CoBit-M analysis shows that Self-BLEU detects increasing repetition across samples that token entropy misses. Token entropy measures diversity within individual samples and does not establish diversity across the generator’s samples. Second, GM combines strong agreement with human rankings and stable rankings at reduced sample counts in our experiments. Together, these findings explain what the recommended additions contribute.

Relationship to alternative metrics

Regarding existing evaluation methods, our suite is a non-exhaustive selection within an extensive literature. Self-BLEU directly applies BLEU among generated samples, while our character n-gram MMD uses lexical features related to BLEU and ROUGE, aggregated into a discrepancy between collections. FBD and MAUVE share BERTScore’s use of pretrained representations, but compare collections differently: BERTScore matches contextual tokens between a candidate and reference; FBD compares feature means and covariances; MAUVE compares distributions over quantized features. Other metrics could fill similar roles.

Computational cost of JCD

We have added direct runtime and memory measurements in Appendix D.4. For 1,024 texts per source, our JCD implementation takes 0.85 seconds on CPU, including fitting and scoring for one split, and uses 0.31 GiB of memory. Gen-PPL takes 20.5 seconds with GPU scoring in the same experiment. JCD therefore adds little computational overhead at the sample sizes used in our study.

Dear reviewer Rf86,

We are thankful for your time and constructive review, particularly your suggestions regarding the practical evaluation recommendations and computational costs. We hope that our revised metric recommendations (Section 7), additional human evaluation, and computational benchmarks (Appendix D.4) have addressed your concerns.

As the end of the rebuttal period is approaching, we would be happy to answer any remaining questions. If our responses and additional analyses address your concerns, we would be grateful if you would consider updating your assessment.

Reviewer_vBA6OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper discusses the evaluation of unconditional text evaluation for diffusion language models. The authors argue that gen-PPL alone is not a suitable evaluation regime for model evaluation, and design naïve samplers to show that gen-PPL can be easily hacked. Finally, the authors propose to use a set of metrics for overall evaluation, including preference-based metrics, diversity-based metrics and distribution-based metrics.

Critical strengths
  1. This paper re-emphasizes an important problem that gen-PPL alone is not sufficient for evaluation of diffusion language models.
  2. This paper reports the performance of language models and agreement of metrics under a large set of evaluation metrics.
Critical weaknesses
  1. The contributions of this work are trivial and not new to me. Gen-PPL is a canonical metric that is well-known by researchers years ago, including its “collapse” risk during evaluation. The analysis of its behavior and construction of hacking examples brings nothing new to my knowledge. As for the preference-diversity tradeoff and evaluation with a bunch of metrics, are already consensuses in the community of text generation evaluation. I list some early works [1-2] for reference.
  2. There is no “related work” section, the relations with existing works are seldomly discussed.

[1] Texygen: A Benchmarking Platform for Text Generation Models. SIGIR 2018.

[2] Language GANs Falling Short. ICLR 2020.

[3] On the Relation between Quality-Diversity Evaluation and Distribution-Fitting Goal in Text Generation. ICML 2020.

Questions for authors
  1. Please discuss the relation to existing works on LLM evaluation.
4 comments

We thank the reviewer for the comments and references and agree that Gen-PPL is a well known metric. We would like to clarify that our contribution concerns the failure of gen-PPL even when controlled with token-entropy. To our knowledge, this is the first presentation of naive samplers which outperform all models on the pareto frontier of Gen-PPL and entropy. While Gen-PPL has known weaknesses with regards to mode collapse, the extent of these weaknesses even when controlled with token-entropy, is one of our main novel contributions. We believe this is important to explore as this pair remains the primary basis for reporting improvements in unconditional generation in many recent DLM papers, including CoBit, LangFlow, FLM/FMLM, ELF, and Duo.

We construct incoherent text samplers that achieve gen-PPL competitive with leading trained DLMs while maintaining great token entropy. One construction always emits the same sample, despite having high token entropy. The relevant distinction is between diversity of tokens within a sample and diversity of samples across generations. Our constructions exploit this distinction to obtain favorable evaluation results despite failures of both coherence and diversity. To our knowledge, achieving this combination with naive samplers is new; and it goes beyond restating the general risk of collapse.

We further show that this blind spot matters for real-world DLMs. CoBit-M achieves the best gen-PPL among the evaluated OpenWebText generators, with token entropy slightly better than Plaid, but in practice produces much worse results which can be seen in other distributional metrics such as Self-BLEU (0.545 vs. 0.051) and GM (19.3 vs. 1.7). Our sampling-budget analysis also finds increasing cross-sample repetition in CoBit-M while token entropy remains nearly unchanged. The issue therefore affects as well the evaluation of trained-model performance.

This motivates our recommendation to report diversity both within and across samples. We are not claiming that Self-BLEU or evaluation with multiple metrics is new. We establish why the particular diversity safeguard used in contemporary DLM evaluation is insufficient, demonstrate its failure empirically, and identify a practical correction.

We thank the reviewer for pointing out additional related work, we have added a Related Work and an updated Evaluation as a Metric Profile sections discussing all three suggested references and positioning these contributions of our work with more detail.

Dear reviewer vBA6,

We are grateful for your time and feedback on our submission. We hope that our rebuttal and revisions, particularly the new Related Work section and clarification of our novel contributions regarding the failure of gen-PPL even when controlled with token entropy, have addressed your concerns.

As the end of the rebuttal period is approaching, we would be happy to answer any remaining questions. If our responses and revisions address your concerns, we would be grateful if you would consider updating your assessment.

Thanks for the rebuttal. The emphasized token-entropy control in the revised paper has alleviated my concerns over the novelty.

However, I still have concerns over the proposed evaluation protocol. A classical solution for evaluation is including both preference and diversity (as in my previously suggested references). In my view, this paper proposes the same solution with slightly different choices of metrics. If time permits, please explain your contribution over evaluation. For example, I would pick gen-PPL and Self-BLEU together without reading this paper, are there any flaws for this simple solution?

I will reconsider my score if this problem is addressed.

We thank the reviewer for the follow-up and are glad that the novelty concern has been alleviated. To answer the question directly: gen-PPL + Self-BLEU is not enough. Naive samplers can also outperform trained DLMs on this pair. For example, our Mirror sampler beats, on OpenWebText, SDTT Teacher, FLM, and FMLM on both gen-PPL (40.8 vs. 45.1–62.8) and Self-BLEU (0.053 vs. 0.058–0.137).

This points to a more general problem we face: joint sample diversity is not trivial to report. As we study in Section 4, the gen-PPL cross-entropy decomposes into the entropy over samples and a KL term, so gen-PPL can improve when this entropy falls. However, this entropy is not estimable from finite samples in the ambient space VLV^L, with VV the vocabulary size and LL the sequence length.

The entropy that is computable, and that has captured the attention of DLM practitioners, is per-sample token entropy, which measures token (1-gram) diversity within samples. However, this measure can be highly decorrelated from the unknown, non-estimable sample entropy of the text generator. As we show empirically, with a naive generator outperforming the whole gen-PPL–token-entropy Pareto frontier of trained DLMs, and as we indeed find in trained DLMs, token entropy is insufficient for capturing the loss of sample diversity. Token entropy is still valuable, however, because it captures a different, somewhat orthogonal form of diversity loss: repeating tokens within a sample makes next-token prediction artificially more predictable, lowering gen-PPL without the text conveying a better message.

Self-BLEU, in turn, is a proxy for sample diversity: it measures n-gram diversity across samples. In particular, we find it effective at capturing the loss of sample diversity in CoBit-M, which seems to be steered towards generating n-grams that are predictable by GPT-2 Large, and thus lower gen-PPL.

We therefore still believe it is worth reporting a quality metric (e.g., LLM-as-a-judge Elo or gen-PPL) plus the weak guardrails for diversity (token entropy for within sample token diversity, and Self-BLEU for across samples n-gram diversity). These are measured in the ambient space and carry useful signal on unconditional text quality, but we need something else that more closely captures sample diversity. That is why we also advocate for distributional similarity metrics, as they bake in some notion of quality and diversity in one. Within this family, we recommend GM for its human alignment and sample-count stability (Section 7).

To summarize our contribution over evaluation: we agree that quality and diversity should both be reported, but in text generation, diversity can be lost in two orthogonal ways, and both can artificially improve gen-PPL: losing diversity across samples, which lowers the sample entropy, and losing token diversity within samples, which makes next tokens more predictable. Token entropy, used in current DLM papers, measures only the second; Self-BLEU is a proxy for the first, but can also be fooled. We show both failures with naive generators and in trained DLMs, and therefore recommend reporting diversity within and across samples, together with a distributional similarity metric such as GM, as sample entropy is not estimable and distributional metrics capture quality and diversity in feature space in one. We thank the reviewer again for this thoughtful follow-up, and would be grateful if they would reconsider their score in light of this clarification.

Comments on the paper

1 comment

We thank all reviewers for their constructive feedback. We have revised the paper to clarify what our work adds to known concerns about gen-PPL, strengthen the evidence behind our recommendations, and make the proposed evaluation more practical. The main concerns and changes are:

  1. Novelty and relationship to prior work. We added a Related Work section that distinguishes established concerns—including the KL–entropy decomposition discussed in Generative Frontiers—from our contribution: showing that the commonly used token-entropy safeguard can fail alongside gen-PPL. Our incoherent generators achieve favorable scores across a range of token entropies, including configurations that always emit the same sequence (Figure 6). We also identify increasing repetition across samples in trained CoBit-M that token entropy misses. These findings explain why the continued use of this metric pair remains problematic despite general awareness of collapse.

  2. Stronger evidence for the evaluation recommendations. The original human study was small and used short excerpts. We added an evaluation using MAUVE’s released texts and human judgments: 1,698 retained comparisons from 205 annotators, covering human and GPT-2 continuations. This is now our primary human-alignment benchmark, with the original study retained as further evidence. We also examine how metric rankings change with fewer samples. Together, human alignment and sample-count stability motivate our recommendation of gradient moment (GM) in the tested settings. Figure 3 and Appendix D report the results, protocols, and uncertainty.

  3. Clear priorities and computational costs. Revised Section 7 recommends adding Self-BLEU and GM to the commonly reported gen-PPL and token entropy. Self-BLEU checks diversity across samples, which token entropy within individual samples can miss. LLM judging is optional when resources permit, and the remaining metrics provide additional diagnostics. Appendix D.4 now reports runtime and memory measurements: the benchmarked automatic metrics each have median runtimes below 40 seconds at 1,024 texts per source. Our proposed JCD takes 0.85 seconds on CPU for one fitting/scoring split, supporting its feasibility at our evaluation scale.

  4. Scope and sampling efficiency. We clarified that our focus is evaluating pretrained DLMs before task-specific adaptation, to inform model comparisons, scaling studies, and base-model selection. For the efficiency question motivating DLMs, we emphasize metric profiles plotted against NFE: these show whether quality, diversity, and distributional agreement with real text are maintained as the number of model evaluations decreases. The added human evaluation strengthens the evidence for text-quality assessment; predicting downstream task performance remains a separate question.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.