acceptodds
Under review as a conference paper at ICLR 2027

Scoring Rules! Statistical and Strategic Alignment for Text Evaluation Metrics

Abstract

Reference-based text evaluation metrics, which are widely used to assess natural language generation systems, score a candidate response by comparing it with a reference response. The reliability of an evaluation metric is usually judged by the statistical correlation with human ratings. However, as these metrics are increasingly used as optimization objectives, this alignment by correlation is no longer sufficient: agents may strategize to game the evaluation metric. We study this issue through two complementary notions of alignment. A metric is statistically aligned if it correlates with human ratings, and strategically aligned if it resists strategic perturbations that do not add task-relevant information. We make two contributions. First, we propose test principles for reference-based metrics, consisting of human-rating correlation, degradation sensitivity, and manipulation robustness. These principles evaluate whether a metric agrees with human judgments, penalizes low-effort information loss, and resists strategic score inflation. Second, we develop a unified design framework for mutual-information-based metrics that are designed to resist strategic perturbations. The framework decomposes existing and new metrics into four design choices: information measure, estimation method, text representation, and prediction mechanism. Across peer review, summarization, and question answering tasks, we find that strong human-rating correlation does not imply strategic alignment: LLM-as-a-Judge achieves high correlation but is susceptible to manipulations. In contrast, mutual-information-based metrics substantially improve manipulation robustness. Our framework also uncovers a new metric that achieves the strongest overall robustness in our experiments while remaining competitive on human-rating correlation.

Primary area:
datasets and benchmarks
Keywords:
Robust NLG Evaluation Metrics, Statistical and strategic alignment, Mutual Information Estimation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2mgyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors decompose the reliability of reference-based text evaluation metrics into statistical alignment (correlation with human ratings) and strategic alignment (robustness to degradations and manipulations), proposing three unified test principles. They then build a design framework for mutual-information-based metrics, decomposing each into a four-component tuple (F, E, R, Π), which recovers GEM/GPPM/TVD-MI as special cases and exposes new combinations. Experiments across peer review, summarization, and QA on 7 datasets show that LLM-as-a-Judge achieves the highest human correlation (ρ=0.45–0.63) but fails 18–26/30 manipulation tests, while the newly identified statement-level TV f-variational metric achieves 0/30 manipulation failures and only 3/37 degradation failures.

Critical strengths
  1. Constructive design framework. The four-component decomposition not only unifies GEM/GPPM/TVD-MI (Table 1) but exposes the statement-level TV f-var metric, which empirically outperforms all prior MI-based methods (Table 3-4), demonstrating that the framework is constructive rather than merely taxonomic.

  2. Well-controlled experimental design. 7 datasets spanning peer review/summarization/QA, 5-6 strategies each for degradation and manipulation, unified quantification via paired t-test + SMD (Table 3-4), achieving a level of cross-method comparability rare in robustness evaluation studies.

  3. Practically relevant core observation. The finding that "statistical alignment does not imply strategic alignment" (LLM-as-Judge has high correlation but is highly manipulable) carries genuine warning value for the current widespread practice of using LLM-as-Judge as RL rewards.

Critical weaknesses
  1. The evaluation principles are circularly justified. The paper observes that MI naturally satisfies the data processing inequality (post-processing cannot increase mutual information), then repackages this mathematical property as a "principle good metrics should satisfy" (manipulation robustness), then uses this principle to demonstrate MI-based metrics are superior. This is reverse-engineering evaluation criteria from the method's known strengths, not independently justifying the criteria. The paper provides no evidence that metrics satisfying these principles actually produce better outcomes in downstream tasks (e.g., RL training, model selection, deployment).

  2. The implicit assumption "information content = quality" is unexamined. Degradation sensitivity requires that scores strictly decrease when information is removed; manipulation robustness requires that non-informational modifications cannot increase scores. This equates text quality entirely with information content. But rephrasing can improve readability, and removing redundancy can improve conciseness — both are quality improvements that do not add information. LLM-as-a-Judge scoring rephrase higher may be correct behavior, not a vulnerability. The paper uniformly treats such cases as manipulation failures without discussing this possibility.

  3. No downstream validation. The paper uses its own proposed test principles to evaluate all metrics, including its own. This is structurally equivalent to grading one's own homework. The most convincing experiment would be: train the same model using LLM-as-Judge and MI-based metrics as RL rewards, respectively, then evaluate the final model quality with independent human evaluation. If the MI-based reward produces a demonstrably better model (less reward hacking, better actual quality), the principles' value is established. The paper contains no such experiment.

Questions for authors

The paper implicitly assumes "text quality ≈ information content." This may partially hold for peer review, but clearly breaks down for summarization (where conciseness is a core metric) and creative writing. How do you delineate the boundary conditions of your principles?

Reviewer_PK94OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to evaluate reference-based text evaluation metrics beyond their correlation with human judgments. It distinguishes statistical alignment, measured by correlation with human ratings, from strategic alignment, measured through two controlled tests: degradation sensitivity and manipulation robustness. The paper proposes three evaluation principles and a four-component design framework for mutual-information (MI)-based metrics, decomposing each metric into an information measure (F), an estimation method (E), a text representation (R), and a prediction mechanism (Π\Pi). This framework recovers existing methods such as GEM, GPPM-Judgment, and TVD-MI as special cases and enables the construction of new combinations.

The authors evaluate metrics across seven datasets spanning peer review, summarization, and question answering. Their results suggest that LLM-as-a-Judge achieves stronger correlation with human ratings but is vulnerable to the tested manipulations. The proposed statement-level, TV-based, (f)-variational MI metric exhibits stronger resistance to these manipulations than the tested LLM judges and has fewer degradation-test failures than the selected MI baselines. The paper's central message is that correlation with human judgments alone is insufficient to characterize an evaluation metric.

Critical strengths
  1. The paper addresses an important problem. As evaluation metrics increasingly influence model development and may be optimized against, correlation with human ratings alone may not reveal whether a metric rewards genuinely better outputs. Separating ordinary evaluation quality from behavior under controlled perturbations is a useful perspective.
  2. The MI design framework offers a clear way to organize existing methods. Decomposing metrics into (F), (E), (R), and (Π\Pi) makes their differences more explicit and provides a systematic vocabulary for investigating new combinations. The statement-level metric is a concrete outcome of this framework rather than merely a conceptual proposal.
  3. The empirical study covers multiple tasks and metric families. The experiments compare MI-based methods with conventional metrics and several LLM judges across peer review, summarization, and question answering. The manipulation results are particularly notable: the proposed metric and report-level TVD-MI have zero manipulation-test failures out of 30, whereas the tested LLM judges have 18–26 failures. This is meaningful evidence that evaluation metrics with similar high-level purposes can behave very differently under the tested score-inflating transformations.
Critical weaknesses

CW1: The knowledge contribution of the three Test Principles is unclear.

Prior work has already studied the constituent evaluation dimensions, including degradation sensitivity and manipulation robustness in GPPM-Judgment, GEM, and TVD-MI, while ITEM (Yari et al., 2025) evaluates human-judgment alignment alongside robustness to controlled perturbations. APPLS (Guo et al., 2024) and Stress Testing Factual Consistency Metrics (Mujahid et al., 2025) also use controlled transformations to assess metric behavior. These works differ in their tasks and specific tests, but together they suggest that this paper primarily consolidates existing evaluation criteria into a unified protocol. The paper does not sufficiently clarify what new knowledge this unification provides beyond the existing principles and findings.

CW2. The experiments do not fully answer the stated metric-selection research question.

The RQ asks which (F,E)(F,E) configuration yields the best-behaved metric for a fixed (R,Π)(R,\Pi), but only human-rating correlation is evaluated across all 12 configurations; degradation and manipulation tests cover only three selected configurations. Consequently, the experiments cannot establish which configuration best balances all three principles across the stated design space. The paper also provides no clear criterion for choosing among configurations when human-rating agreement and robustness conflict.

References

Guo, Y., August, T., Leroy, G., Cohen, T., & Wang, L. (2024). APPLS: Evaluating Evaluation Metrics for Plain Language Summarization. Proceedings of EMNLP 2024, 9194–9211. https://doi.org/10.18653/v1/2024.emnlp-main.519

Mujahid, Z. M., Wright, D., & Augenstein, I. (2025). Stress Testing Factual Consistency Metrics for Long-Document Summarization. arXiv:2511.07689. https://arxiv.org/abs/2511.07689

Yari, A. H., Kulkarni, K., Khan, A. R., & Koto, F. (2025). Revisiting Metric Reliability for Fine-grained Evaluation of Machine Translation and Summarization in Indian Languages. arXiv:2510.07061. https://arxiv.org/abs/2510.07061

Questions for authors

Q1. The optimization motivation is not followed through in the experiments. The Introduction motivates the work partly through reward hacking when evaluation metrics become optimization objectives, but the experiments use predefined perturbations rather than metric-driven optimization or adaptive attacks. Could the authors either provide evidence that the proposed framework addresses these optimization-related risks or narrow the motivation and claims to match the current experimental scope?

Q2. The results do not provide a clear selection rule between MI-based metrics and LLM-as-a-Judge. The proposed metric is substantially more robust to the tested manipulations, but its human-rating correlations are generally lower than those of LLM judges. Given these conflicting results, how should practitioners decide which approach to use, and under what conditions does the robustness gain justify the lower human-rating agreement?

Q3. The perturbation construction and validation are insufficiently clear to establish that the tests measure their intended properties. Could the authors provide representative examples and more details about how the perturbations are constructed and validated? In particular, how do they verify that degradation perturbations genuinely reduce task-relevant quality and that manipulation perturbations can increase metric scores without improving the underlying output?

Q4. The no-op fallback may affect the reported manipulation-robustness results. The supplementary material states that perturbation-generation failures may fall back to a no-op, which leaves the input unchanged and could make a manipulation test easier to pass. Could the authors report the no-op rate by dataset and perturbation strategy, and show whether the main results remain unchanged when these cases are excluded?

Q5. The directional hypothesis tests are under-specified. Could the authors explicitly state the null and alternative hypotheses, and whether the paired (t)-tests are one-sided or two-sided, for both degradation and manipulation? In particular, the absence of a statistically significant score increase does not by itself establish that score inflation is negligible.

Reviewer_B5mxOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on text evaluation metrics: it argues that metrics should be assessed on (1) agreement with human judgments (statistical alignment) and (2) robustness to strategic, score-seeking perturbations (strategic alignment). It argues that prior mutual-information (MI) metrics focus on different sized text chunks, and propose a unified MI-based evaluation framework, introducing a statement-level TV f-variational metric. Across seven datasets, the metric outperfomrs prior MI-based metrics, and find that LLM judges is prone to manipulation failures.

Critical strengths
  • The paper proposes a new statement-level TV metric that empirically outperform MI baselines.such as GEM and TVD-MI.
Critical weaknesses
  • Limited novelty. The new proposed metric, statement-level TV, is a combination or variation of existing metrics. It achieves marginally better resistance to strategic manipulation, but still lags far behind LLM judges in terms of human rating alignments. Therefore, it is not a new metric that is both statistically aligned and strategically aligned.
  • The manipulation strategies need independent validation. Some rewrites or rephrasing can improve the quality, clarity, or overall quality of an answer. Thus, a score increase may not solely be a score inflation. Since the purpose of these manipulation is to keep the faithful scoring unchanged, some human validation is needed to validate that the quality of text is unchanged. Similarly, it needs to be verified that the degradation actually reduce text quality.
  • Presentation: the majority of implementation details and design choices such as scoring model details, degradation/manipulation strategies, effect sizes, etc. are put in appendix, while they are essential to the understanding of the work.
Questions for authors
  • Currently, a fixed set of perturbation is evaluated. Will the metrics be robust to an adversary that actively optimizing the metric?
  • How do one verify that the manipulations add no task-relevant information? Similarly, how do one verify that the degradations actually reduce text quality?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.