acceptodds
Under review as a conference paper at ICLR 2027

OvERALE: A Heuristic Approach to Pluralism Evaluation

Abstract

Large Language Models (LLMs) have long been criticised for their unidimensionality and systemic biases. Incorporating such systems into society risks deepening ideological polarization and fostering tunnel vision. We require responses that balance diverse perspectives while carefully filtering out viewpoints that fall outside the bounds of societal acceptability. A term used to describe this notion is *Overton Pluralism*. This domain is still in its nascent stage and suffers from critical bottlenecks, such as the lack of a reasonably defined Overton window, the disruption of objective evaluation by long-form text, and the limitations of human-in-the-loop evaluation that constrain progression towards generalizability and efficiency in terms of cost and speed. In our attempt to address these limitations, we propose **OvERALE** (**Ov**erton-ground**E**d, **R**anking-**A**ligned corpus **L**attice **E**valuator), a framework that assigns a score to long-form responses, used as a proxy for pluralism. It is assessed by projecting the Overton window of individual human viewpoints (*Micro* lens) into a topic-level domain (*Meso* lens), situating pluralism within this broader topic space for downstream evaluation. OvERALE trains a rank-consistent pluralism lattice over a static gold corpus, which then serves as a static reference for evaluating new responses. Empirical sensitivity analysis shows OvERALE's score moves reliably in the expected direction across targeted perturbations, outperforming diversity and coverage based baselines as well as LLM-as-judge evaluators in directional alignment and stability. Our framework marks a meaningful step toward automated Overton pluralism evaluation, offering a scalable alternative to human annotation that extends naturally to large bodies of text.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Pluralism, Alignment

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_wnKdOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposed a metric for evaluating the pluralism of responses. The authors project sentences embedding into a 2D topic space, learns a scoring field from a model-level pluralism ranking, and scores a response by averaging its sentences’ nearest anchor scores. It outperforms diversity and llm as judge baselines in perturbation tests and correlation with human ratings.

Critical strengths

s1: The research question is timely and important. Measuring how pluralistic a response is a pain point in the pluralism domain, since the evaluated responses usually involve multiple topics and arguments are mixed with unclear boundaries. Currently there is no good solution for this.

s2: The evaluation includes both perturbation tests and a human study, and the human study reports annotator agreements,

Critical weaknesses

w1: The metric scores each sentence independently and then takes the average, so it ignores the relations between sentences. If several sentences express the same viewpoint, or the same sentence is repeated, the score can be inflated.

w2: The paper only mentioned that the 16,007 anchor points are drawn from the topic clusters of the five corpora, but didn't mention how they are selected and filtered, as this is a crucial step in the pipeline (wrong anchor gives wrong score), it needs to be clarify in the paper.

w3: The whole pipeline is built on one embedding model, so all calculations are based on one semantic space, if apply to another model that with differen pretain data or even different architecture, will it work? It needs to test another embedding model and run the pertubation test and everything else.

Questions for authors

see weakness

7 comments

W1 (sentence-level averaging).

We agree that OvERALE scores each sentence on its own and averages the scores, so it does not model relations between sentences. This is a response-level aspect of pluralism, and it falls outside what a sentence-level score is designed to capture.

OvERALE is a sentence-level scorer by design. Section 2.2 now defines the scored object as a sentence's projection in the shared topic space ("Let x ∈ R² be a sentence's projection in the shared topic space"), so the hypothesis, the learned field and the scoring rule all operate at the same granularity. We acknowledge this in Appendix F: "Sentence and response level scoring may be better suited to different types of responses". A system that can switch between these granularities might capture pluralism more effectively.

The responses we evaluate are natural model outputs, which do not contain deliberate redundancy. We agree that a response built to repeat highly pluralistic sentences could raise its score, because the score is an average over sentences. We have therefore added a note to Appendix F that responses should be checked for redundancy before scoring.

W2 (anchor construction)

The submission gave only the size of the gold corpus (N_g = 16,007). We have added Appendix G (Gold Anchor Construction), which describes the candidate pool, the embedding, the cluster assignment, the per-cluster cap and the projection.

We think this construction is a sound choice for three reasons:

  • Same corpora and clusters for anchors and boundaries. The anchors are the projected member points of the same topic clusters, drawn from the same five corpora, that define the region Ω and the hulls. The anchors and the boundaries therefore describe the same topic structure.
  • No overlap with the evaluation data. No OvertonBench sentence and no evaluated model response is used to build the anchors. Their scores come only afterwards, from the trained field, so the anchors cannot leak evaluation data into the score.
  • Balanced and reproducible. Uniform subsampling at both the corpus level and the cluster level, plus the per-cluster cap, probably keeps any single large corpus or cluster from dominating the anchor set. A fixed random seed makes the sample reproducible.

W3 (single embedding model).

We thank the reviewer for this point. OvERALE is defined on a fixed shared topic space built with one sentence encoder (all-MiniLM-L6-v2), and we do not claim that the space or the reported results carry over to other encoders. To remove any ambiguity, we clarify the implicit part of the hypothesis in Section 2.2 so that it reads "pluralism can be approximated as a function of position in this shared two-dimensional topic space alone", instead of "pluralism can be approximated as a function of position alone,"

This encoder is part of the scorer's definition: the topic space, the gold anchors and the trained field are all learned in its coordinates. A different encoder is therefore a new instance of the method and a separate study is required, and not a test of the one we present. Nevertheless we used 10 emdbedding models to verify the possibility of " topic structure may generalize beyond the corpora used to construct it.", mentioned in the introduction, for different embedding spaces.

We added Appendix J with two parts:

  • Topic structure. We rebuilt the shared space with ten encoders and repeated Figure 2(d). For every encoder, the OvertonBench cluster centroids lie predominantly within the union hull and the estimated pluralistic window overlaps substantially with the intersection hull. This supports the claim in the paper that the topic structure may generalize beyond the corpora used to construct it. The hull shapes differ across encoders, so the space remains encoder-specific.
  • Scoring. We make no claim about downstream pluralism evaluation with other encoders. In a preliminary attempt with several of them, perturbation pass rates reached at most 14/18, which matches the default encoder at larger neighbourhood sizes but is below its 18/18 and 17/18 at K = 7. We do not draw conclusions from this. The hyperparameter space is large: the few hyperparameters we sweep in the paper alone gives nearly thousand combinations, and with the architecture and optimization settings it is far larger, each with many possible values. We varied only a small subset of these, one at a time, over a few values. Training and evaluating one configuration is costly (the field is trained on a 2048 × 2048 lattice, and the perturbation suite scores about 19 variants of tens of thousands of responses), so tuning every encoder over this space was not feasible within our compute budget. A fair conclusion requires a complete, encoder-specific study, which is outside the scope of this work and left to future work.

Thank you for the clarifications. I appreciate OvERALE as a valuable initial step toward automated pluralism evaluation. Although the authors clarify that OvERALE is a sentence-level scorer by design, practical applications typically involve paragraphs or multi-sentence responses, where relationships among sentences matter. Nevertheless, I consider this work a promising starting point. For future work, I encourage the authors to explore modeling relationships among the viewpoints expressed across sentences, incorporating information such as viewpoint overlap and topical complementarity to develop a more robust metric better suited to multi-sentence responses.

Thank you for your comment and for considering OvERALE a promising starting point. We would like to clarify one point, because we think it is affecting how the work is being assessed. OvERALE does not score sentences. It scores whole responses, and the sentence is the unit from which the response score is computed.

1. What OvERALE scores

The paper states this in several places. The Abstract describes "a framework that assigns a score to long-form responses". Section 3.3 defines the score of a response x, split into sentences u_1, ..., u_n, as "the mean, over sentences, of each sentence's average gold score among its K nearest neighbors" (Eq. 10). Using a smaller unit to score a larger text is a common design for long-form evaluation. FActScore [1] scores a generation as the share of its atomic facts that are supported. SelfCheckGPT [2] obtains a passage-level score by averaging sentence-level scores over all sentences of the passage. In each case the output is a score for the whole text.

2. Every experiment is at response level

  • Perturbations (Section 4.3). All 18 perturbations edit complete responses: appending or prepending paragraphs, deleting perspective sentences, shuffling sentence order, and so on. The shifts are measured on the response score. They were run on the full Philosophia-QA corpus of n = 57,191 and on n = 10,000 responses from strix-philosophy-QA, both made of long-form argumentative answers. OvERALE passes 18/18 perturbations on Philosophia-QA and 17/18 on strix-philosophy-QA.
  • Human validation (Section 4.4). Ten annotators rated whole responses on a 1,000-item pool and a shared 50-item pool. OvERALE's response score correlates with the human composite at r = 0.580 (p < 0.001). The eight LLM judges and AJ(FS+FR), which read the entire response, reach |r| < 0.06.
  • Human study of the perturbations (Appendix K). Annotators rated the original and all 18 perturbed versions of 13 complete responses (247 items). 16 of the 18 perturbations behave as expected in the human ratings.

3. Relationships among sentences

Our diversity baselines (Section 4.2) already score a response from how its sentences relate to each other, and during the review we also tested response-level scorers at the request of other reviewers (Appendix H). They did not perform better than OvERALE.

  • Pairwise and dispersion measures (Section 4.2). Max Pairwise Distance and Centroid Dispersion pass only 1/5 decrease perturbations on both datasets, and correlate with the human composite at r = −0.006 and r = −0.018.
  • Response-level aggregation (Appendix H). Scoring one centroid per response with the same trained field passes 0/5 decrease perturbations and reaches r = 0.084 with human ratings, against r = 0.580 for OvERALE.

These measures reward added off-topic content as extra diversity, because they have no reference for what counts as a legitimate topic. OvERALE gets this reference from its gold corpus. We agree that modelling relationships among viewpoints is an interesting direction (Appendix F), and we think it would need a reference of this kind, as OvERALE has.

4. Practical use

OvERALE needs one lookup against a fixed gold corpus per sentence, with no generative-model calls at inference (Introduction, contributions). The paper presents it as a proxy for pluralism, and the evidence above (large-scale perturbation tests on whole responses and agreement with human ratings of whole responses) supports its use in that role. We have already made it clearer in the paper that the score is a response-level score computed from sentences as an unit.(Section 3.3).

We hope this clarifies the design, and we are grateful for your encouragement and your suggestion for future work.

References

[1] Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12076–12100, Singapore. Association for Computational Linguistics.

[2] Potsawee Manakul, Adian Liusie, and Mark Gales. 2023. SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9004–9017, Singapore. Association for Computational Linguistics.

Reviewer_GodAOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper introduces an automated score (OvERALE) intended as a proxy for pluralism in long form responses. The paper presents a three-stage methodology: (1) project topic clusters from five human-written datasets into a shared two-dimensional space and check where OVERTONBENCH falls within it. (2) learn a scoring field using OVERTONBENCH model rankings and a prior that favors points near the center, then freeze the resulting scores into a reference corpus. (3) split new responses into sentences, project them into this space, score each sentence using its nearest reference points, and average across sentences. This way, new responses can be scored without further generative judge calls. The authors evaluate the method using perturbations on two philosophy QA datasets and a human study. They report stronger performance than the evaluated baselines on the perturbation tests and moderate correlation with human pluralism ratings (r = 0.58).

Critical strengths
  • The topic space uses five human-written datasets while evaluation uses two independently sourced QA datasets. This is a useful starting point for testing transfer beyond the construction data.
  • The human evaluation adds meaningful evidence. The reported correlation with human pluralism ratings is moderate (r = 0.58), and agreement on the shared 50 responses is reasonably strong (ICC = 0.76). This supports the method beyond its performance on synthetic edits.
Critical weaknesses

The approach is interesting, however, the paper appears to draw stronger conclusions than the experiments support.

For example, the authors claim that “correct ranking order is enough to recover the same region” although training also explicitly favors the center of the topic space. The resulting shape probably cannot be attributed to ranking supervision alone. The rank-only ablation performs worse on the perturbation tests, although this does not directly test whether it recovers the same spatial region.

The paper also states that “Indicating the agreement is driven predominantly by genuine viewpoint coverage rather than relevance or fluency alone”. A higher correlation with pluralism ratings than with relevance or fluency does not establish what drives the agreement.

Section 4.3 states that K=7 was selected based on performance on the same perturbation tests used for evaluation. This makes the reported performance less convincing as evidence of generalization. Moreover the methods are evaluated on different sample sizes, which complicates comparisons when passing depends on statistical significance.

Also, a nonsignificant score change does not prove stability. Individual scores could change considerably in opposite directions.

Overall, I think the datasets and human study support a narrower conclusion than general pluralism evaluation.

Questions for authors

[Clarification] What explains the near zero human correlations for all judge baselines?

6 comments

Ranking Order and Region

We have revised the sentence in the manuscript, which now reads:

"It is therefore notable that the field trained to respect the correct ranking order occupies approximately the same region: compare the OvertonBench bracket in Figure 2(d) with the area the trained field actually fills in Figure 8 and ranking violations in Figure 4-7."

This observation was based on the figures cited there, which show how closely the field trained with our ranking heuristic traces the OvertonBench region. It was not written from the perspective of the ablation tables, so the original wording "is enough to recover" went further than the comparison supports. We agree that both components of the objective, the ranking loss and the center-favoring regularizer, contribute to the performance (Table 3), and we have updated the phrase so that it is consistent with these results as well.

Correlation Clarification

We agree that a higher correlation with the pluralism ratings than with the controls does not establish what drives the agreement. We have revised the sentence in the manuscript, which now reads:

"OvERALE's correlations with Q3 and Q4 are positive and significant (r=0.300r=0.300 and r=0.307r=0.307, respectively) but clearly lower than its correlation with the composite score (r=0.580r=0.580), which suggests that the agreement is probably not explained by relevance or fluency alone."

We no longer use the word "driven" and make no causal claim.

Selection of K. We thank the reviewer for this point. We clarify that K appears in two places: the neighborhood size used when training the field (Eq. 6) and the neighborhood size of the scoring-time lookup (Eq. 10). The sweep in Table 7 (and Tables 4–6) varies only the second, on a field trained at K = 7. We missed describing the selection of the training K in the submitted version, although the sweep had been computed, and we now add it as Appendix I. The training K is chosen with the same criterion as the other training hyperparameters (λreg\lambda_{\text{reg}}, μ\mu, γ\gamma; Tables 8–10), the symmetry of the score distribution over the gold set, and not by the perturbation tests. K = 7 gives the smallest mean–median gap (−0.043, against −0.059 at K = 14, −0.110 at K = 20, −0.127 at K = 10, and −0.354 at K = 5, where the median saturates at 1). The perturbations have no role in any form during training as can be seen in the methodology, and is only used during sensitivity evaluation, the scoring-time K is the one selected by pass rate on the perturbation tests (Table 7).

Different sample sizes. We thank the reviewer for raising this point and clarify why the samples differ. OvERALE and the embedding-based baselines (Max Pairwise Distance, Centroid Dispersion, MODPLURAL-adapted) run locally at negligible cost, so we evaluate them on the full Philosophia-QA set (n = 57,191) and on a sample of n = 10,000 from strix. The LLM-judge baselines and AJ(FS+FR) require one API call for every response and every perturbed variant, i.e. 19 variants for each of 2,000 responses, about 38,000 calls per judge. We limited them to n = 2,000 on both datasets due to cost considerations.

We believe the comparison remains fair. A pass is a within-method verdict: each method's perturbed scores are compared with its own unperturbed scores on the same responses, using the same paired Wilcoxon test, the same Benjamini–Hochberg correction and the same expected direction. A method's verdict therefore never depends on another method's sample, and since the sample is a uniform random draws from the same data, the verdicts compare how each method responds to the same set of edits.

Stable Category

We would like to clarify what the stable category tests. In our setting, "stable" means insensitivity: these perturbations are pure surface-form edits (whitespace, capitalization, sentence order, boilerplate, connective synonyms, spelling variants) that touch no viewpoint content (Table 26), so a pluralism metric should not respond to them systematically in either direction. A non-significant paired test is the evidence for that, and it does show that the population-level shift is absent, i.e. the scores do not move consistently up or down. We agree that it does not show that every individual score is unchanged: single responses may move in both directions without a net shift, and we do not claim per-response invariance. The claim is that OvERALE shows no systematic shift under edits that should not matter for pluralism. To make this clear, we changed one line in Section 4.3, where the stable category is now described as "pure surface-form edits that touch no viewpoint content, so the score should be insensitive to them on average".

OvERALE as a proxy for Pluralism

We agree that our evidence supports a narrower conclusion than general pluralism evaluation. OvERALE is intended as a proxy for pluralism, as stated in Appendix F ("OvERALE currently serves as a proxy for pluralism rather than measuring full Overton pluralism"), and we have now made this explicit in the Conclusion. We now describe OvERALE as providing "an efficient proxy for long-form pluralism" and as "a start toward scalable proxy-based pluralism evaluation", and we added the sentence: "Pluralism is subjective and varies across contexts, so no single framework can capture it fully."

Near-zero judge correlations. Humans agree on these items (ICC = 0.76) and OvERALE correlates with them at r=0.580r = 0.580, so the human scores contain a usable signal. We observe that six of the nine judge variants assign the lowest score to 87–95% of the responses (Claude-Sonnet-5: 94.5%), and the other three assign it to 54–72%. All nine have ∣r∣≤0.03|r| \le 0.03 with the human composite on the 1,000 items, and none is significant on the 50 shared items. This is the known problem of direct LLM scoring, where one score dominates the distribution, giving low variance and many ties, and correlations with human judgments are low (G-Eval [1]). In our setting, the judges were not able to capture the differences in grades of pluralism perceived by humans. Human ratings, in contrast, are spread over the whole scale: all five values are used for Q1 and Q2, no single value accounts for more than 29% of the ratings, and the composite has mean 2.78 and standard deviation 0.96 (30% of items at or below 2, 16% at or above 4).

References

[1] Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. EMNLP 2023.

AuthorsOpenReview ↗

Requst to review our responses

Dear Reviewer GodA,

As we approach the end of the discussion phase, we sincerely request you to review our responses. We have made every effort to carefully address all your concerns and provide detailed clarifications.

If you find our responses satisfactory, we would be grateful if you could consider reassessing your evaluation of our paper. We remain fully committed to addressing any additional questions or concerns you may have.

Thank you for your time, thoughtful feedback, and consideration.

Thanks

Reviewer_5HuxOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper proposes OvERALE, a metric that scores how pluralistic a long-form response is. Specifically, the authors first project sentences from 5 human-written corpora into a shared 2D PCA space. Then the authors train a convolutional lattice on the OvertonBench model ranking, with a regularizer that favors the center. To score a new response, the authors find the nearest corpus sentences for each of its sentences, and average their trained scores. From 18 perturbations on Philosophia-QA and strix-philosophy-QA, and human ratings of 1,050 Philosophia-QA responses, the authors find that (1) OvERALE passes more perturbations than diversity, coverage, and LLM-judge baselines; (2) only OvERALE correlates significantly with human ratings (r = 0.58).

Critical strengths
  1. The problem of evaluating pluralism in long-form responses without costly human annotation or repeated LLM-judge calls is important.

  2. The authors report per-perturbation results for every method on Philosophia-QA, which makes the analysis easy to check.

Critical weaknesses
  1. In Section 2.2, the authors claim that “a set of sentences drawn from many diverse topics has a centroid that moves toward the middle.” However, Equation 10 never computes this centroid. It only averages the scores of single sentences, and by the paper’s own argument, each sentence of a response covering many topics would still sit near the edge. For example, if we join two responses on different topics, the score is a weighted average of their two scores, which can never be higher than the better one. In other words, what the authors measure is where each sentence falls, instead of how many topics a response covers. These two are very different. One possible way would be to use the centroid in both training and scoring.

  2. The authors validate OvERALE mainly with targeted edits, and I think simply comparing the metrics on such toy examples is not that convincing. Each edit that should increase pluralism only adds a generic sentence, such as a hedge, which mentions other perspectives without discussing them. The only edit that removes the response’s own viewpoints is “delete perspective marking sentences”. However, it changes the score by only 0.00013, which is even smaller than the change from reformatting the response as markdown bullets. Also, humans only rate unperturbed responses. So how do we know that the edits change pluralism at all? Edits of the same length that add a real opposing argument should be added, and humans should also rate the perturbed responses.

  3. One related work is missing [1], which also studies Overton pluralism and finds that aligned responses tend to cover information spread across several responses from the base model. The authors should discuss how OvERALE differs from it.

[1] Lake, Thom, Eunsol Choi, and Greg Durrett. "From distributional to overton pluralism: Investigating large language model alignment."

Questions for authors
  1. In Table 2, the correlations of all baselines are close to zero, even for the LLM judges, whose rubric asks about the same two aspects as the human ratings. Is each correlation computed over the 1,050 rated responses or over all the ratings? Also, how is the sample of responses scored by the judges matched to the human-rated ones?

  2. From my understanding, the regularizer alone gives higher scores near the center of the topic space, and the ranking loss is never removed in the ablations. Is this correct? The results without the ranking loss should be added. A training-free scorer that simply averages how close each sentence is to the center should also be compared. If such a simple scorer can achieve close performance, then the contributions of this paper may be impacted.

  3. How many responses are actually changed by the deletion edit? What is the score change on those responses only?

11 comments

Section 2.2. Section 2.2 is titled "A Hypothesis for Scoring Pluralism in this Space", and the centroid sentence is a motivating intuition for Figure 1(b): where content sits in the shared topic space relates to how pluralistic it is. It is not a claim that Eq. 10 computes a response-level centroid. We use it to justify scoring by position alone, s=f(x)s = f(x), with ff learned from the OvertonBench ranking and the center-favoring prior (Eqs. 6-7). To remove the ambiguity, we revised the definition in Section 2.2 from "Let x∈R2x \in \mathbb{R}^2 be a response's projected centroid in the shared topic space" to "Let x∈R2x \in \mathbb{R}^2 be a sentence's projection in the shared topic space", and we replace response with sentence in connecting points in this subsection.

Sentence-level scoring. We agree that Eq. 10 averages sentence scores, as OvERALE scores where a response's sentences fall in this shared space rather than counting the topics it covers directly. This is a deliberate design to test the hypothesis, and our whole experimental setup follows it, including the choice of perturbations. The lack of granularity switching is acknowledged in Appendix F ("sentence and response level scoring may be better suited to different types of responses"). Given the diversity of natural language, the right granularity likely varies across responses, and a scorer that adapts it to the response is an important direction for future work.

Using the centroid in scoring. We tested this directly (Appendix H). We average a response's sentence projections into one centroid and score that point by its nearest gold points, using either the trained field or the training-free center-distance scores. Tables 1 and 2 compare these two centroid scorers with OvERALE under the same protocol as the main paper.

Table 1. Perturbation pass rate by category.

MethodPhil-QA IncreasePhil-QA DecreasePhil-QA Stablestrix Increasestrix Decreasestrix Stable
OvERALE (K=7K=7)6/65/57/76/64/57/7
Response-centroid (trained field)6/60/55/73/60/57/7
Response-centroid (center distance)5/61/51/75/61/53/7

Table 2. Pearson correlation (rr) and significance (pp) with human ratings (1,000 single-rated items).

MethodComposite rrComposite ppQ3 rrQ3 ppQ4 rrQ4 pp
OvERALE (K=7K=7)0.580<0.0010.300<0.0010.307<0.001
Response-centroid (trained field)0.0840.0080.0160.6070.0140.663
Response-centroid (center distance)0.118<0.0010.119<0.0010.0370.245

Both centroid scorers fail the decrease perturbations (Table 1), because appending unrelated content moves the centroid toward the center, which both fields reward. This matches Max Pairwise Distance and Centroid Dispersion, which pass 6/6 of the increase perturbations but only 1/5 of the decrease perturbations on both datasets (Table 1 of the paper). The failure appears under both scoring functions, so we believe it comes from averaging positions rather than from the trained weights. The centroid scorers also agree much less with human ratings than OvERALE does (Table 2).

Two separate evaluations. The sensitivity analysis and the human validation answer two different questions. The human study asks how closely OvERALE's score agrees with human judgment of pluralism on natural responses, so it uses unperturbed responses by design (Section 4.4). The sensitivity analysis asks whether the score moves in the intended direction under controlled edits (Section 4.3), and it does not depend on human ratings. This two-part protocol is how evaluation metrics are commonly validated. For example, ROSCOE [1] evaluates its metrics on diagnostic datasets with programmatically introduced errors and, separately, on human-annotated datasets. Rating perturbed responses would not add evidence to either question. Agreement on natural responses is already what the human study measures.

References

[1] Olga Golovneva, Moya Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning. ICLR 2023.

Opposing-argument edits. We agree that such edits are interesting, but we believe they fall outside the scope of this work. A real opposing argument depends on the content of the whole response, so it has to be generated separately for each response, and whether it is a genuine opposition is a subjective judgment. The edit would then differ from response to response, and its quality would become a confound. This makes it a poor controlled perturbation. Our suite instead uses fixed, rule-based edits that add the same text to every response, which keeps the edit identical across all responses, and tests for the direction of change rather than magnitude.

  • Closest perturbations in our suite. Four of the six increase perturbations (stakeholder perspectives, short- vs long-term framing, local vs broader framing, multi-stance overview) introduce an explicit second analytical dimension on the question, and the other two introduce explicit stance-marking language (Appendix A.4, Table 24).

Not a length effect. The decrease perturbations also add text: off-topic paragraphs, anecdotes and a reference list. Their scores go down while the increase perturbations' scores go up, so length alone does not explain the direction.

Perturbation design. Our perturbations are designed to assess sentence-level pluralism scoring: each adds, removes or alters whole sentences, such as appending a hedging or multi-perspective sentence, appending off-topic or anecdotal sentences, or deleting every sentence containing a perspective-marking cue (e.g. "some", "others", "however", "on the other hand"). We have in Appendix A.4 grounded these perturbations in form of an intuition derived from prior work on pluralism, stance, diversity, or topic structure.

The deletion edit. We would like to clarify what this edit is and how it fits into the suite. It deletes only the sentences containing one of a fixed set of cue words (e.g. "some", "however", "on the other hand"), which plausibly signal an expressed position or an acknowledgment of an alternative perspective (Appendix A.4, Table 25). It is a lexical-cue rule, and it does not remove "the response's own viewpoints" in general: a viewpoint stated without such a cue is left untouched, and a cue word does not always mark a viewpoint. It is therefore not the only edit that acts on pluralism content, and it is not a special case. It is the mirror image of the increase edits that add stance-marking language (append balanced conclusion, append uncertainty acknowledgment, e.g. "emphasis may reasonably vary by context"), and it sits alongside other rule-based, localized edits in the suite. The other decrease edits lower pluralism content by dilution: they add content that is novel but gives no substantive diversity on the question, such as off-topic paragraphs, narrow anecdotes ("a single household ... does not generalize") and a bare reference list. The increase edits add a second analytical dimension on the question (stakeholder, short- vs long-term, local vs broader framing, multi-stance overview). Every perturbation in the suite is a fixed, rule-based edit of this kind.

As stated in Section 4.3, all perturbations are "targeted, localized" edits "designed to marginally decrease" or "marginally increase" pluralism content. What we test is therefore whether the score moves consistently in the expected direction across the dataset, and not how far it moves. This is why Table 1 reports pass counts, where a perturbation passes if the Benjamini-Hochberg-adjusted Wilcoxon test is significant in the predicted direction, and "passed reflects direction of change".

The 0.00013 mean shift for the deletion edit should be read in that light. The shift is small, as expected for a marginal edit that changes few sentences, and it is consistently negative across the 57,191 paired responses (p = 0.023, Table 11). The comparison with markdown reformatting does not favor the larger number. Reformatting shifts the mean by more, but the shift is not significant, meaning its direction is not consistent across responses. A mean of that size can come from a few large, anomalous changes, so it does not show which way the score moves. Magnitude alone does not tell us the direction, and consistency of direction is what the test measures.

On strix the deletion edit is not significant, which is the one miss for OvERALE in Table 1. The decrease category does not rest on this edit: the other four decrease perturbations shift the score by −0.029 to −0.055 on Philosophia-QA and by −0.063 to −0.094 on strix, and all are significant.

New Reference We thank the reviewer for pointing us to this work and will add it to the related work (Appendix A.1). Lake et al. [1] study why aligned models appear less diverse and find that the drop is largely explained by quality control and information aggregation: an aligned model's response tends to combine information that a base model spreads across several samples. This is a shift from distributional to Overton-style coverage, and it is an analysis of how alignment produces such responses.

OvERALE addresses a different problem and is orthogonal to it. We do not study how models acquire Overton behavior. We provide a score for a single response, without sampling other responses or calling a generative model at inference time: its sentences are projected into a shared topic space and scored by a field learned once from the OvertonBench ranking. Their finding is complementary, since it implies that aligned responses are expected to cover several viewpoints in one response, which is the setting OvERALE evaluates.

Unit of analysis. Each correlation is computed per response, not per rating. Table 2 uses the 1,000 single-rated responses (n=1000n = 1000). The 50 responses rated by all ten annotators are analyzed separately, with the mean of the ten ratings as the human score.

Matching. There is no sampling step. Every method, including each judge, scores exactly the same human-rated responses, and scores are joined to ratings by response ID.

Near-zero judge correlations. Humans agree on these items (ICC = 0.76) and OvERALE correlates with them at r=0.580r = 0.580, so the human scores contain a usable signal. We observe that six of the nine judge variants assign the lowest score to 87–95% of the responses (Claude-Sonnet-5: 94.5%), and the other three assign it to 54–72%. All nine have ∣r∣≤0.03|r| \le 0.03 with the human composite on the 1,000 items, and none is significant on the 50 shared items. This is the known problem of direct LLM scoring, where one score dominates the distribution, giving low variance and many ties, and correlations with human judgments are low (G-Eval [2]). In our setting, the judges were not able to capture the differences in grades of pluralism perceived by humans. Human ratings, in contrast, are spread over the whole scale: all five values are used for Q1 and Q2, no single value accounts for more than 29% of the ratings, and the composite has mean 2.78 and standard deviation 0.96 (30% of items at or below 2, 16% at or above 4).

New Baselines and Ablations

The reviewer's understanding of the earlier ablations is correct: the λ_reg sweep in Table 3 (Appendix D.1) keeps λ_rank = 1 fixed, so the ranking loss is never removed there. In order to address these points we have added both requested comparisons and saw the results.

Without the ranking loss. Table 3 now has a row with λ_rank = 0 and λ_reg = 1 (regularizer only). It passes 11/18 perturbations on both datasets (Philosophia-QA: 6/6, 1/5, 4/7; strix: 6/6, 0/5, 5/7), against 18/18 and 17/18 for the full objective. The decrease perturbations fail, and every perturbation shifts the score by only about 0.001, which probably means that the regularizer alone yields a nearly flat field that does not separate the categories. Together with the rank-only row (λ_reg = 0), this suggests that both loss terms contribute.

Training-free center-distance scorer. We compare this scorer in Appendix H. Each sentence is scored by the average center-distance score of its nearest gold points, and the response score is the mean over sentences. It passes 6/18 perturbations on Philosophia-QA and 7/18 on strix, and it fails all five decrease perturbations on both datasets (OvERALE: 18/18 and 17/18). Its correlation with the human composite is r=0.123r = 0.123, against 0.580 for OvERALE. The simple scorer is therefore not close in performance, and the trained field contributes to the results.

How many responses change. The deletion edit changes 15,176 of 57,191 Philosophia-QA responses (26.5%) and 1,120 of 9,996 strix responses (11.2%). The remaining responses contain no cue sentence and are returned unchanged.

Score change on the changed responses only.

DatasetChanged responsesMean changeMedian changeShare decreasedWilcoxon pp
Philosophia-QA15,176−0.0005−0.00452.1%0.014
strix1,120−0.0040.00049.6%0.58

References

[1] Thom Lake, Eunsol Choi, and Greg Durrett. From Distributional to Overton Pluralism: Investigating Large Language Model Alignment.

[2] Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. EMNLP 2023.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.