- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper mainly proposes SOL. Its core idea is that existing likelihood-based and sample-based evaluation metrics on text generation are relatively poor, so instead, it compared how close the distribution of model-generated text is to the distribution of real text. To this end, the paper proposes a new metric for evaluating text quality. Since computing this metric using Wasserstein-on-Wasserstein is relatively difficult, it uses Double-Sliced Wasserstein to compute it.
The paper then conducts several experiments. The first examines the mode collapse problem of generation perplexity. The second reuses the human judgments from the MAUVE paper to compare how well automatic metrics agree with human rankings of eight GPT-2 generation configurations. Finally, it reevaluates DLMs against AR models and finds that the ordering of Gen. PPL does not always agree with that of SOL.
Critical strengths
Generative perplexity is indeed a poor metric, although this is probably already a consensus in the field[1][2]. However, there is still no widely accepted, better alternative to the existing automatic evaluation metrics. Therefore, I think this is a good problem to address, and it is indeed worth solving.
[1] Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs Distributional Metrics. arXiv 2606.08417
[2] On the Relation between Quality-Diversity Evaluation and Distribution-Fitting Goal in Text Generation. ICML 2020
Critical weaknesses
- My first concern is that, if this metric is intended to be widely adopted, I think its many complicated design choices may hinder its adoption. For example, many widely used metrics, such as entropy, perplexity, have relatively simple designs, and it is fairly intuitive why they should work. Although some of them were later shown to be vulnerable to hacking, they were still relatively reliable metrics for a long time. With such a complicated design, I suspect there may be many potential vulnerabilities to exploit(like [3][4]). In other words, we do not know how robust SOL is against hacking. This also leads to a related question: SOL introduces a substantial number of design choices, but have the authors sufficiently demonstrated the necessity of these choices and the robustness of the resulting model rankings to them?
- My next concern is that the evidence for alignment between SOL and human judgments is insufficient. The paper reuses human judgments from the MAUVE paper, but ultimately, I think a new metric still needs to be validated through human evaluation[5]. Without new human evaluations, it is difficult to determine how useful this score actually is. Even if SOL is indeed a good metric at the distributional level, it would still be of little use if the generated text looks very poor to humans. The fact that SOL does not agree with Gen. PPL does not necessarily mean that SOL itself is a good metric.
- Furthermore, the results in Appendix D.9 do not seem to demonstrate that SOL is particularly stable.(correct me if i am wrong with that) Simply changing the encoding from [x] to [x, x] changes the optimal sampling temperature from 0.9 to 1.3, which raises further concerns about the robustness of the metric.
[3] On the Blind Spots of Model-Based Evaluation Metrics for Text Generation. ACL 2023
[4] MENLI: Robust Evaluation Metrics from Natural Language Inference. TACL 2023
[5] SummEval: Re-evaluating Summarization Evaluation. TACL 2021
Questions for authors
-
The first question is whether the model rankings produced by SOL have any practical significance for evaluation. Since the paper proposes SOL as a practical language model evaluation metric, what sufficient evidence do the authors provide to demonstrate that a better SOL score leads to a more meaningful comparison of model quality? also, is there any evidence that SOL in Table 3/22 also align with human judgments across these models?
-
The second question is related to the weakness mentioned above. The authors' own experiments have already shown that one representation choice can change which model is selected as the best. What about the other design choices? How robust are the resulting model rankings to these choices?
3 comments
Thank you for the clarifications and additional experiments. However, the new human study uses only 163 judgments (77 from one annotator), and Opus 5.5 reverses SOL on MDLM versus LangFlow and DUO. Rankings of close models also flip across encoders (such as LDDM-U versus AR in Table 26), leaving my robustness concerns unresolved. Therefore, I will maintain my current score.
Thank you for the reply. We see the remaining concerns differently. Below we lay out our reasoning step by step:
-
In your original review, you agree that Gen. PPL is a poor metric and that a better alternative is worth developing. A main reason is that Gen. PPL is not a distributional metric. It scores single sequences and is minimized by a single sequence. A better alternative therefore has to compare distributions.
-
Token sequences carry no meaningful geometry, so in practice this comparison needs an encoder. Every distributional metric for text therefore depends on an encoder, e.g. MAUVE. FID depends on an encoder in the same way in the image setting. Although FID has well known problems, we would argue that it is still useful for the image community.
-
The existing encoder-based metrics for text show pathological behavior. We highlight this throughout the paper and the revision.
-
Opus 5.5 reverses the ranking of SOL only within the middle group of LangFlow, DUO, and MDLM. In the human study, these three models are close, so some disagreement between judges is to be expected. Opus agrees with SOL on all other pairs. We do not think that these reversals invalidate the method.
-
SOL avoids the pathological behavior of Gen. PPL and MAUVE described in steps 1 and 3, and its ranking agrees with human judgment ( on OpenWebText). Close pairs such as LDDM-U and AR can change order across encoders, as for every encoder-based metric (step 2). Still, 96.7% to 98.3% of model pairs keep their order. As for FID, the encoder is fixed and reported together with the score.
Taken together, the current choice is between Gen. PPL, which does not compare distributions, existing encoder-based metrics with failures such as those described above, and SOL. SOL is the only option that compares distributions while avoiding these failures. In particular SOL is theoretically justified and is an actual distance under light assumptions on the encoder. We believe it is a clear improvement over the metrics in current use.
If you disagree with one of the steps above, we would be grateful to know which one, so that we can address it directly. Otherwise, we kindly ask you to reconsider your assessment of our work.
We thank the reviewer for the careful reading and the detailed comments.
We agree that the practical setup should be simple. A user chooses only the encoder and the reference set. Parameters have fixed defaults: 5,000 samples, Lξ=Lg=8192 slices and the bidirectional Dream encoder. Our implementation uses these defaults.
Gen. PPL can be improved without matching the data because its optimum is not the data distribution. It is minimized by a single sequence, the most likely one under the reference model. SOL is different. For an injective encoder, SOL is a metric, and its only minimizer is the data distribution. Lowering SOL therefore means moving closer to the data in the encoder's representation. Both examples in G2 reach very good Gen. PPL and entropy, and SOL does not reward them. We do not claim robustness against adversarial attacks on a specific encoder.
The design choices fall into two groups.
The first group consists of modeling choices, and we show why each is needed. A single last-token state misses changes at the beginning of a text (Figure 3). A mean-pooled vector is less sensitive than the full set of token states in nine of ten structural perturbations. Causal encoders react mainly to changes early in the text, while the bidirectional Dream encoder reacts evenly. The model rankings, however, barely depend on the encoder. Across GPT-2 Large, OLMo, and Dream, the rank correlations over 36 model configurations are 0.990 to 0.996.
The second group consists of numerical choices, which only control how accurately SOL is computed. Going from 1,024 to 8,192 slices changes SOL by at most 1.5%, and the quantile resolution changes it by less than 0.1%. At the defaults, these approximations have converged.
We agree. We ran a new human study directly on the OpenWebText models from our experiments (G1, Section 5.4, Appendix E.2). The human ranking and SOL have a Spearman correlation of 0.97.
The [x,x] representation of [1] is not part of SOL. It is an alternative that we tested and rejected. SOL uses [x].
Appendix D.8 explains the shift. In the second copy, the encoder predicts each token from the first copy with 99% top-1 accuracy. High-temperature samples are then no longer surprising, and the score mainly reflects word choice. With a separator, [x,sep,x] again prefers temperature 0.9. We will rename this appendix section to make clear that [x,x] is not used by SOL.
The study in G1 tests this directly on the OpenWebText benchmark. It covers real OpenWebText, Soft-Masked, AR, LangFlow, DUO, MDLM, and FMLM. The human annotators and SOL clearly agree on the rankings of the tested models.
Furthermore, the new baselines in G3 show a case that Gen. PPL and entropy cannot resolve. MDLM-MMD [2] samples in 16 steps, while MDLM with ReMDM [3] uses 1,024 steps with remasking. We therefore expect a clear quality gap. However, both have almost the same Gen. PPL (28.6 and 28.9), and MDLM-MMD also matches the data entropy. SOL separates them clearly, with 56.8 against 23.6 for Dream.
For the encoder, the model rankings are almost unchanged: across GPT-2 Large, OLMo, and Dream, the rank correlations over 36 model configurations are 0.990 to 0.996. The numerical parameters such as number of slices have converged at the defaults we suggest.
We are happy to answer any further questions. If our responses resolve the reviewer's concerns, we kindly ask the reviewer to raise the score.
[1] Repetition Improves Language Model Embeddings, Springer et al., 2025.
[2] Representation-Space MMD for Diffusion Language Models, Drobyshevskiy et al., 2026.
[3] Remasking Discrete Diffusion Models with Inference-Time Scaling, Wang et al., 2025.