acceptodds
Under review as a conference paper at ICLR 2027

Evaluating Textual Diversity Metrics with Controlled Mixtures of Diverse Perspectives

Abstract

The prevalence of large language models (LLMs) as tools for information access and creativity necessitates the development of rigorous methods to assess the diversity of LLM-generated content in open-ended contexts. While many diversity metrics have been proposed in the literature, incorporating both varied representations of text and methods for measuring diversity, we identify a significant gap between existing abstract studies of diversity, and the practical realities of measuring diversity in text. Drawing on the interdisciplinary framework proposed by Stirling, which decomposes diversity into three necessary dimensions (variety, balance, and disparity), we add a fourth requirement specific to open-ended natural language, invariance to composition, and operationalize the framework to develop a generalizable procedure for validating proposed diversity measurements of natural language grounded in real datasets of diverse text. Concretely, we: (1) Demonstrate how to generate controlled mixtures of diverse real-world, open-ended text from existing datasets with processes that make the four fundamental diversity criteria directly accessible; and (2) Empirically measure and statistically test which diversity metrics correspond with the underlying conceptual diversity. Our evaluation bridges the gap between theoretical notions of diversity and empirical work in natural language processing, providing grounding for diversity measurement in open-ended text and a methodology for validating future proposed metrics.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
diversity, homogeneity, metrics, natural language processing

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_Rz7aOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how to evaluate existing diversity metrics and investigates what lies behind diversity metrics. Building on an existing study (Stirling), the paper proposes the fourth dimension, i.e., invariance to surface-level structural composition. Based on four dimensions, including variety, balance, disparity, and invariance, the paper further introduces a meta-evaluation framework to validate the sensitivity of existing diversity metrics with respect to these four dimensions from an empirical perspective. The experimental results show that many existing diversity metrics cannot meet these desired criteria simultaneously.

Critical strengths
  1. The paper studies the evaluation of diversity metrics. This topic is valuable because it is difficult to evaluate a diversity metric to determine whether it reflects characteristics of the evaluated data due to the lack of diversity ground truth.

  2. The paper is well-organized and easy to follow.

  3. The evaluation design of the paper is explainable. Based on the labeled groups and hierarchical structures in Perspectrum and DiverseSumm, the paper can connect metric responses to known data manipulations, which helps to analyze results and offer explainability.

Critical weaknesses
  1. There needs to be further clarification on whether compositional effects have an influence on the composition experiment. Table 1 requires comparisons under the same concept frequency, whereas Section 4.4 conducts testing by appending texts of known concepts. The latter will change concept frequency, balance, and set size simultaneously. Thus, score changes may not derive solely from composition, and flatter curves do not establish invariance.

  2. Some experimental conclusions are overstated compared to evidence in figures and tables. For example, in Figure 3, MPNet+Vendi and Qwen3+Vendi show significant differences for all three adjacent comparisons. However, Section 4.3 states that disparity tests rarely reach significance.

  3. The paper does not sufficiently clarify why a smaller regression coefficient is insensitive to changes in diversity. It may be quite different in magnitude and rules of change for different metrics. Thus, a smaller score change may not demonstrate that metrics fail to identify the number of concepts and balance. The authors should explain how these coefficients can be compared in a fairer manner and validate whether the analysis correctly identifies metrics known to capture these properties.

Questions for authors

Refer to Critical Weaknesses for more details.

Reviewer_9cBPOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on text diversity measurement in NLP: it questions whether existing diversity metrics faithfully measure the underlying conceptual variation rather than surface-level variations. Following Stirling, it decomposes diversity into four components: variety, balance, disparity, and invariance to composition. Several findings emerge in the experiments comparing lexical, embedding, and entailment-based diversity representation.

Critical strengths
  • Diversity evaluation is a well-motivated problem. Variationi n wording or surface-level text features may not necessarily measure diversity in concepts.
  • Decomposing diversity evaluation to quantity (variety), balance, disparity, and invariance to composition makes it easier to diagnose metric behavior and point to where a diversity metric may succeed or fail.
Critical weaknesses
  • The evaluation is limited in scale. Although the paper reports repeated sampleing, each evaluated collection of the four dimensions contains only 4 to 10 texts. The controlled setting provides limited evidence about behavior on large collections of long passages in real use scenarios. The paper should report text-length distribution, concept coverage, and the design choices of the collection size in evaluation.
  • Some finding claims are not what the evaluation results show. For example, in Disparity section, it is claimed that "most metrics are insensitive to disparity." However, embedding metrics such as qwen3-8b and mpnet can separate topic/claim/instance strongly.
  • While the diversity metrics are intended for LLM generations, the evaluation never uses actual LLM generations.
Questions for authors
  • How is distance between groups defined in Section 4.3 Disparity?
  • Do the findings in Section 4 persist with larger collection sizes? Would there be more natural compositions than simply concatenating sentences?

Comments on the paper

1 comment
Area_Chair_MC5FOpenReview ↗

Review #3

Paper Summary: The paper focuses on LLM output diversity evaluation metrics. The main motivation is that existing diversity measures may not capture conceptual nuance and necessary dimensions of output diversity. Therefore, the paper suggests breaking down diversity (conceptually) based on four properties that diversity metrics should have: variety, balance and disparity, compositional invariance. In experiments, they use datasets where text perspectives are grouped by concept and build synthetic text sets where the properties have ground truth to test existing metrics against the four properties. They are interested in whether existing diversity metrics are sensitive to each property. The claim is that this bridges the gap between concepts underlying diversity and practical experiments to measure output diversity.

Strengths:

  • The area of focus is well motivated. As LLMs are used more widely, there are growing concerns about the homogenization in model outputs and the downstream impacts of models lacking diversity. This makes measuring diversity particularly important.
  • The paper is careful not to overstate the contribution. They note that the first three properties are derived from [Stirling 2007] which means the fourth property is the novelty here.
  • I think the experimental approach is interesting. This is not how I would have naturally designed experiments. It seems the authors could have used the properties to develop a better diversity metric that actually covers the four properties, then ran more standard experiments to show that the metric is competitive with existing measures. Instead, they seem to present a more general framework for testing any given diversity metric, which is kind of an interesting framing.

Weaknesses:

  • The contribution is inconsistently described throughout the paper. At one point, the paper positions its contribution as a framework. At another point, the contributionh is described as the method for building controlled text mixtures. Then, at another point, the paper points to the empirical comparison of metrics as the main novelty. Even if we take the framework as the main contribution, its not clear how useful this is to the broader research community. We could do this test and find that all measures fall short... and then what?
  • The four properties are not quite new. Three of them come directly from Stirling 2007 as noted. And the fourth one does not include operationalization so this is a major weakness.
  • Positioning: while the paper is clear about the dependence on Stirling 2007, its not quite as clear about how its differentiated from Tevet 2021. There is a brief comment but that paper also introduces a validation approach. so the claim that this paper identifies a significant gap seems like an overstatement. The research community is quite aware of the limitations of diversity metrics, improvements and alternatives are much harder, which this paper does not seem to go into.

Questions For Authors: (see weaknesses outline above)

Flag For Ethics Review: No

Rating: 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.