- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies how to evaluate existing diversity metrics and investigates what lies behind diversity metrics. Building on an existing study (Stirling), the paper proposes the fourth dimension, i.e., invariance to surface-level structural composition. Based on four dimensions, including variety, balance, disparity, and invariance, the paper further introduces a meta-evaluation framework to validate the sensitivity of existing diversity metrics with respect to these four dimensions from an empirical perspective. The experimental results show that many existing diversity metrics cannot meet these desired criteria simultaneously.
Critical strengths
-
The paper studies the evaluation of diversity metrics. This topic is valuable because it is difficult to evaluate a diversity metric to determine whether it reflects characteristics of the evaluated data due to the lack of diversity ground truth.
-
The paper is well-organized and easy to follow.
-
The evaluation design of the paper is explainable. Based on the labeled groups and hierarchical structures in Perspectrum and DiverseSumm, the paper can connect metric responses to known data manipulations, which helps to analyze results and offer explainability.
Critical weaknesses
-
There needs to be further clarification on whether compositional effects have an influence on the composition experiment. Table 1 requires comparisons under the same concept frequency, whereas Section 4.4 conducts testing by appending texts of known concepts. The latter will change concept frequency, balance, and set size simultaneously. Thus, score changes may not derive solely from composition, and flatter curves do not establish invariance.
-
Some experimental conclusions are overstated compared to evidence in figures and tables. For example, in Figure 3, MPNet+Vendi and Qwen3+Vendi show significant differences for all three adjacent comparisons. However, Section 4.3 states that disparity tests rarely reach significance.
-
The paper does not sufficiently clarify why a smaller regression coefficient is insensitive to changes in diversity. It may be quite different in magnitude and rules of change for different metrics. Thus, a smaller score change may not demonstrate that metrics fail to identify the number of concepts and balance. The authors should explain how these coefficients can be compared in a fairer manner and validate whether the analysis correctly identifies metrics known to capture these properties.
Questions for authors
Refer to Critical Weaknesses for more details.
Review #3
Paper Summary: The paper focuses on LLM output diversity evaluation metrics. The main motivation is that existing diversity measures may not capture conceptual nuance and necessary dimensions of output diversity. Therefore, the paper suggests breaking down diversity (conceptually) based on four properties that diversity metrics should have: variety, balance and disparity, compositional invariance. In experiments, they use datasets where text perspectives are grouped by concept and build synthetic text sets where the properties have ground truth to test existing metrics against the four properties. They are interested in whether existing diversity metrics are sensitive to each property. The claim is that this bridges the gap between concepts underlying diversity and practical experiments to measure output diversity.
Strengths:
Weaknesses:
Questions For Authors: (see weaknesses outline above)
Flag For Ethics Review: No
Rating: 2.