acceptodds
Under review as a conference paper at ICLR 2027

How Evaluation Choices Affect MAUVE Scores and Generator Rankings

Abstract

Since its introduction, has been widely adopted for evaluating open-ended text generation, serving both to compare language models and to design and select decoding strategies. Because it compares generated and human-written text in a learned feature space, its scores depend on evaluation choices such as the representation model, the evaluated text length, and the quantization settings. These choices are known to shift scores, but less is known about whether they also change which generation model is preferred. We study this question across seven generation models, three datasets, and 180 evaluation settings, separating variation due to evaluation choices from variation due to decoding configurations. For identical generations, evaluation choices produce a wider score spread than decoding changes, and they alter model rankings even after aggregating over decoding strategies: two settings that differ only in the representation model select the same top-ranked model in 30% of comparisons. Among the choices examined, the representation model has the largest effect on ranking stability, and it also affects how closely rankings agree with human preference judgments. These findings suggest that interpreting a MAUVE comparison requires considering its evaluation configuration and the sensitivity of model rankings to that configuration, and show that human evaluation can identify representation models whose rankings align more closely with human preferences, providing practical guidance for choosing among them.

Primary area:
foundation or frontier models, including LLMs
Keywords:
evaluation, open-ended text generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8t7VOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how sensitive MAUVE is to different evaluation choices, including the representation model, text length, and bucket count. Using fixed generated text from seven models, three datasets, and 180 evaluation settings, the authors find that changing the evaluation setup shifts scores about 4.6× more than changing the decoding method, indicating a high sensitivty to evaluation setup changes that affect not only the absolute metric scores but the rankings produces based on these scores. The choice of the representation model is detected as the main contributor to this sensitivity. It also affects agreement with human preferences (T5-based MAUVE 0.70 vs. GPT-2-based 0.51 on Wikitext). Based on the analysis, the authors provide actionalble guidelines on how to correctly report the results when using the metric, namely, MAUVE results should always be reported with their full evaluation configuration.

Critical strengths
  • The paper addresses an important question of the reliability and stability of text generation metrics (specificlly MAUVE, which is widely used).Instead of reporting the absolute score differences, the authors examine whether model rankings are preserved under changes to the evaluation configuration, which is the more consequential and practically relevant for amyone using the metric
  • The empirical study is broad, covering seven generation models, three datasets, twelve representation models, and a total of 180 evaluation settings. This gives the analysis a good empirical basis and allows the authors to draw their conclusions.
  • The authors discuss their findings and provide concrete reporting recommendations (e.g., specifying the representation model, pooling, normalisation, text-length budget, and bucket count), which makes the work useful for future evaluation practice.
Critical weaknesses
  • The main metrics are reported as point estimates without confidence intervals, significance tests, or other forms of uncertainty quantification. This makes it difficult to assess whether the observed differences such as 0.51 vs. 0.70 (human agreement) or 0.395 vs. 0.646 (representation vs. bucket-count sensitivity) are statistically reliable, and it weakens the strength of the conclusions.
  • No details on the human study are provided in the paper. The authors do not report the number of annotators, the number of judgments per cell or per pair, recruitment or screening procedures, annotation instructions, or how pairwise judgments were aggregated into a ranking. Although the Ethics Statement claims that details on recruitment, consent, compensation, and ethical review are "provided according to the study protocol," none of this information appears anywhere in the main text or appendices. Since RQ4 and the claim of practical guidance rest on this study, this omission limits reproducibility and the verifiability of the findings.
  • The human data itself appears noisy with order consistency being 76.0%, and split-half ranking reliability - 0.64. This adds a fairly low upper bound on the achievable agreement between any automatic metric and the human ranking, which the authors do not discuss. Combined with the lack of the study design description, this limits confidence in the conclusions drawn.
  • The paper writing is at times hard to follow, with lots of hedged and short bullet-point style sentences. Substantial revision to make the text more readable is required in my opinion. The text reads very AI-generated at its current version.
Questions for authors

Suggestions:

  • I would reorder the last two related work paragraphs. The discussion would flow more naturally (and be easier to follow) if it first covered work on the metric itself (MAUVE, studies of its sensitivity to evaluation choices, and alternative distributional metrics) and only then - decoding strategies.
  • I think a paper would benefit from a more detailed explanation of IQR and how it is used here. What is it? Why was it chosen over alternatives such as the standard deviation? What values do we expect (e.g. for RQ1: a good metric that is stable and insensitive to evaluation variation should have a lower IQR). Making this explicit would help readers to interpret the reported value and connect the findings.
  • The Finding statements should be limited to the empirical result of the corresponding research question. Currently they also include a concluding sentence that motivates the next research question, which should be placed in the surrounding text (e.g., in Finding 1, "RQ2 tests whether this variation changes which generation model is preferred")
Reviewer_A61mOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates how MAUVE evaluation configurations affect text-generator rankings when generated outputs are held fixed. It compares twelve representations, three truncation budgets, and five bucket counts across three domains and seven generators. The analysis examines score variation, ranking stability, and agreement with pairwise human preferences. The authors report that configuration choices, particularly the representation, can substantially alter generator rankings and the selected best model. The study aims to assess the robustness of MAUVE-based model comparisons and highlight the practical consequences of evaluation configuration choices.

Critical strengths
  1. The fixed-output experimental design addresses a practically relevant evaluation question. Holding generated texts constant helps isolate the effects of MAUVE configuration choices on model comparisons.

  2. The archived example provides concrete evidence that changing representations can reverse generator rankings. This observation matters for model selection, since evaluation choices may affect which generator is considered best.

Critical weaknesses
  1. The headline quantitative results are not sufficiently reproducible from the supplied artifacts. The paper reports 1,175 generation cells, whereas the archive contains 1,146, including conflicting duplicate evaluation keys. The available analysis does not recover the reported aggregate ranking statistics. A complete revised score table, analysis index, and executable scripts are needed to substantiate the paper’s central quantitative contribution.

  2. The interpretation and practical recommendations lack sufficient validation. The normalization explanation requires controlled comparisons and clarification of whether the executed MAUVE pipeline already applies internal L2 normalization. Furthermore, the human-preference study lacks the judgments and protocol needed for independent auditing, and configuration selection is not validated on independent preference data. These limitations weaken both the proposed explanation for ranking changes and the recommendations about evaluation configurations.

Questions for authors
  1. Can you provide the exact score table, analysis index, and scripts underlying the headline results? Please reconcile the reported 1,175 generation cells with the 1,146 cells in the archive and explain the conflicting duplicate evaluation keys.

  2. Does the MAUVE implementation used in the experiments apply internal L2 normalization before PCA? What preprocessing steps differ between the normalization conditions, and which controlled comparisons support the normalization explanation?

  3. Can you clarify the human-evaluation protocol, including annotator allocation, presentation randomization, consent, compensation, and any applicable ethics approval? Please also provide anonymized judgments and preference-ranking code where permitted.

  4. What is the paired uncertainty interval for the reported alignment improvement of Δτ = 0.19? Was configuration selection evaluated on independent judgments, and does the uncertainty analysis account for shared prompts and annotators?

Reviewer_beEYOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper study how MAUVE evaluation choices affect the resulting evaluation scores and generator rankings. The seven generators on three datasets and find that evaluation choices produce greater score variation than decoding change, alter model rankings, and the representation modle has the largest effect on ranking stability.

Critical strengths
  • The paper study an important problem of measuring the reliability of evaluation metric of open-ended text generation. As LLMs gets widely deployed for a wide range of practical writing tasks, reliably evaluating their text quality is crucial.
Critical weaknesses
  • The paper’s findings provide limited practical value on top of existing work. Prior works (https://aclanthology.org/2023.acl-long.674/, https://openreview.net/forum?id=bvpkw7UIRdU, https://aclanthology.org/2025.coling-main.668/) already show that decoding methods, representation choice, and hyperparameters substantially affect generated-text quality and MAUVE scores, so little new insights are discovered in this work.
  • The scope of this works is narrowly focused on MAUVE and older models. For example, the generation are used from the Decoding Decoded paper and all generator models are several years old. This severely limits the practical impact of this study as current frontier models and newest open-weight models, especially reasoning models, are likely much stronger models and may observe a different trend than studied in this work.
Questions for authors
  • What are the major new findings that differentiate your work from prior works on this topic?
  • Does the same result trend hold under modern models?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.