- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposed a metric for evaluating the pluralism of responses. The authors project sentences embedding into a 2D topic space, learns a scoring field from a model-level pluralism ranking, and scores a response by averaging its sentences’ nearest anchor scores. It outperforms diversity and llm as judge baselines in perturbation tests and correlation with human ratings.
Critical strengths
s1: The research question is timely and important. Measuring how pluralistic a response is a pain point in the pluralism domain, since the evaluated responses usually involve multiple topics and arguments are mixed with unclear boundaries. Currently there is no good solution for this.
s2: The evaluation includes both perturbation tests and a human study, and the human study reports annotator agreements,
Critical weaknesses
w1: The metric scores each sentence independently and then takes the average, so it ignores the relations between sentences. If several sentences express the same viewpoint, or the same sentence is repeated, the score can be inflated.
w2: The paper only mentioned that the 16,007 anchor points are drawn from the topic clusters of the five corpora, but didn't mention how they are selected and filtered, as this is a crucial step in the pipeline (wrong anchor gives wrong score), it needs to be clarify in the paper.
w3: The whole pipeline is built on one embedding model, so all calculations are based on one semantic space, if apply to another model that with differen pretain data or even different architecture, will it work? It needs to test another embedding model and run the pertubation test and everything else.
Questions for authors
see weakness
7 comments
Thank you for the clarifications. I appreciate OvERALE as a valuable initial step toward automated pluralism evaluation. Although the authors clarify that OvERALE is a sentence-level scorer by design, practical applications typically involve paragraphs or multi-sentence responses, where relationships among sentences matter. Nevertheless, I consider this work a promising starting point. For future work, I encourage the authors to explore modeling relationships among the viewpoints expressed across sentences, incorporating information such as viewpoint overlap and topical complementarity to develop a more robust metric better suited to multi-sentence responses.
Thank you for your comment and for considering OvERALE a promising starting point. We would like to clarify one point, because we think it is affecting how the work is being assessed. OvERALE does not score sentences. It scores whole responses, and the sentence is the unit from which the response score is computed.
1. What OvERALE scores
The paper states this in several places. The Abstract describes "a framework that assigns a score to long-form responses". Section 3.3 defines the score of a response x, split into sentences u_1, ..., u_n, as "the mean, over sentences, of each sentence's average gold score among its K nearest neighbors" (Eq. 10). Using a smaller unit to score a larger text is a common design for long-form evaluation. FActScore [1] scores a generation as the share of its atomic facts that are supported. SelfCheckGPT [2] obtains a passage-level score by averaging sentence-level scores over all sentences of the passage. In each case the output is a score for the whole text.
2. Every experiment is at response level
- Perturbations (Section 4.3). All 18 perturbations edit complete responses: appending or prepending paragraphs, deleting perspective sentences, shuffling sentence order, and so on. The shifts are measured on the response score. They were run on the full Philosophia-QA corpus of n = 57,191 and on n = 10,000 responses from strix-philosophy-QA, both made of long-form argumentative answers. OvERALE passes 18/18 perturbations on Philosophia-QA and 17/18 on strix-philosophy-QA.
- Human validation (Section 4.4). Ten annotators rated whole responses on a 1,000-item pool and a shared 50-item pool. OvERALE's response score correlates with the human composite at r = 0.580 (p < 0.001). The eight LLM judges and AJ(FS+FR), which read the entire response, reach |r| < 0.06.
- Human study of the perturbations (Appendix K). Annotators rated the original and all 18 perturbed versions of 13 complete responses (247 items). 16 of the 18 perturbations behave as expected in the human ratings.
3. Relationships among sentences
Our diversity baselines (Section 4.2) already score a response from how its sentences relate to each other, and during the review we also tested response-level scorers at the request of other reviewers (Appendix H). They did not perform better than OvERALE.
- Pairwise and dispersion measures (Section 4.2). Max Pairwise Distance and Centroid Dispersion pass only 1/5 decrease perturbations on both datasets, and correlate with the human composite at r = −0.006 and r = −0.018.
- Response-level aggregation (Appendix H). Scoring one centroid per response with the same trained field passes 0/5 decrease perturbations and reaches r = 0.084 with human ratings, against r = 0.580 for OvERALE.
These measures reward added off-topic content as extra diversity, because they have no reference for what counts as a legitimate topic. OvERALE gets this reference from its gold corpus. We agree that modelling relationships among viewpoints is an interesting direction (Appendix F), and we think it would need a reference of this kind, as OvERALE has.
4. Practical use
OvERALE needs one lookup against a fixed gold corpus per sentence, with no generative-model calls at inference (Introduction, contributions). The paper presents it as a proxy for pluralism, and the evidence above (large-scale perturbation tests on whole responses and agreement with human ratings of whole responses) supports its use in that role. We have already made it clearer in the paper that the score is a response-level score computed from sentences as an unit.(Section 3.3).
We hope this clarifies the design, and we are grateful for your encouragement and your suggestion for future work.
References
[1] Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12076–12100, Singapore. Association for Computational Linguistics.
[2] Potsawee Manakul, Adian Liusie, and Mark Gales. 2023. SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9004–9017, Singapore. Association for Computational Linguistics.
W1 (sentence-level averaging).
We agree that OvERALE scores each sentence on its own and averages the scores, so it does not model relations between sentences. This is a response-level aspect of pluralism, and it falls outside what a sentence-level score is designed to capture.
OvERALE is a sentence-level scorer by design. Section 2.2 now defines the scored object as a sentence's projection in the shared topic space ("Let x ∈ R² be a sentence's projection in the shared topic space"), so the hypothesis, the learned field and the scoring rule all operate at the same granularity. We acknowledge this in Appendix F: "Sentence and response level scoring may be better suited to different types of responses". A system that can switch between these granularities might capture pluralism more effectively.
The responses we evaluate are natural model outputs, which do not contain deliberate redundancy. We agree that a response built to repeat highly pluralistic sentences could raise its score, because the score is an average over sentences. We have therefore added a note to Appendix F that responses should be checked for redundancy before scoring.
W2 (anchor construction)
The submission gave only the size of the gold corpus (N_g = 16,007). We have added Appendix G (Gold Anchor Construction), which describes the candidate pool, the embedding, the cluster assignment, the per-cluster cap and the projection.
We think this construction is a sound choice for three reasons:
W3 (single embedding model).
We thank the reviewer for this point. OvERALE is defined on a fixed shared topic space built with one sentence encoder (
all-MiniLM-L6-v2), and we do not claim that the space or the reported results carry over to other encoders. To remove any ambiguity, we clarify the implicit part of the hypothesis in Section 2.2 so that it reads "pluralism can be approximated as a function of position in this shared two-dimensional topic space alone", instead of "pluralism can be approximated as a function of position alone,"This encoder is part of the scorer's definition: the topic space, the gold anchors and the trained field are all learned in its coordinates. A different encoder is therefore a new instance of the method and a separate study is required, and not a test of the one we present. Nevertheless we used 10 emdbedding models to verify the possibility of " topic structure may generalize beyond the corpora used to construct it.", mentioned in the introduction, for different embedding spaces.
We added Appendix J with two parts: