acceptodds
Withdrawn from ICLR 2027

Efficient Elicitation of Collective Disagreements

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

This paper focuses on how to measure disagreements in a population that faces alternative proposals, be them political candidates, policy issues, or projects to be funded. Disagreement measures require increasing levels of information to be elicited from individuals, from one single pairwise comparison to complete rankings over the alternatives. In this paper, we propose a stratified framework that classifies disagreement measures according to this information, allowing us in turn to design efficient elicitation protocols for their computation. Specifically, we introduce the plurality matrix, an aggregated preference representation that records, for every subset of alternatives, the probability that each ranks first in . We define the level of a disagreement measure as the smallest subset size needed to compute it. We prove that rank variance and divisiveness require exactly level and that the hierarchy is strict, with skewness and excess kurtosis at levels and , respectively. Electoral data illustrate the contrasting rank distributions that these higher moments describe. To make our results practically actionable, we design two elicitation protocols to estimate the various levels of the plurality matrix, exploring the trade-off between the number of required participants and the cognitive load requested to each of them.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Preferences, Computational Social Choice, Digital Democracy

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pNFrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper sorts disagreement measures by "level": the smallest subset size you need to compute the measure. It introduces a plurality matrix that records, for every subset of alternatives, how likely each alternative is to rank first. It shows that rank variance and divisiveness have level 3, and that skewness and excess kurtosis have levels 4 and 5. It then gives 2 elicitation protocols that trade off how many people you ask against how much you ask each person, with bounds on both. It illustrates the results on real election data and synthetic preference profiles.

Critical strengths
  • I like the focus on cognitive load. Revealing richer kinds of disagreement means collecting more preference information, and people get tired of answering lots of pairwise questions. I think this is the most practical part of the paper.
  • I appreciate the combinatorial structure of the plurality matrix. Because it covers every subset, I imagine it can catch effects like this one: the alternative that wins every head-to-head may not be the top choice in a bigger set. Say 40 people rank A > B > C, 35 rank B > C > A, and 25 rank C > B > A. B beats A and C head-to-head, but A gets the most first-place votes.
Critical weaknesses
  1. Gap Between Theory and Motivation. I agree that it's important to reveal disagreement and help people address it. But I don't see how the contributions do that. The paper introduces the plurality matrix, defines levels, proves hierarchy results, and gives elicitation protocols. I still don't understand why it matters that a measure has level 3, 4, or 5, why I should care about any one measure, or what the plurality matrix tells me that a simple matrix of pairwise preference counts would miss. The A/B/C example shows that 2 populations can look the same at level 2 and differ at level 3, but not why that difference matters. The elicitation bounds tell me how many people and comparisons I need, but not how much elicitation I'd actually have to do for a real problem, or what I'd get for it. There's a lot of theory here and a missing connection to something practical.

  2. Inconsistencies in the Formalization. The plurality matrix covers every subset of alternatives, but the model assumes every person has a strict ranking. That rules out ties and intransitive preferences. If this is meant to be a foundational, general-purpose object for disagreement, I think it should handle the most general case: pairwise preferences, ties, and intransitivity. Restricting to strict rankings might be fine, but then the authors should justify it. Right now the limitation seems to defeat the purpose of the thoroughness in the plurality matrix.

Questions for authors

Clarification Questions

  1. What does the plurality matrix tell us about disagreement that a matrix of pairwise preference counts doesn't?
  2. Why should we care that a measure has level 3, 4, or 5? What does knowing that let someone conclude or do?
  3. For a real problem, how much elicitation would I actually need to do?
  4. Existing measures, such as the diversity measures of Hashemi & Endriss (2014) and Karpov (2017) or the polarization measure of Can et al. (2015), are defined on full preference profiles. What do they miss? What concrete problem does that cause?
  5. The plurality matrix covers every subset, but the model only allows strict rankings. What's the value of being thorough in one place but not the other? Can the framework handle ties or intransitive preferences? If not, why are strict rankings the right scope?

Comments

  1. Concrete Application. You'll notice that a lot of my clarification questions are really about motivation. What I'm really missing is how the plurality matrix ties back to the broader motivation of revealing and understanding disagreement. I do like that there are real-world experiments. What's missing is how a theoretical study could address the concerns you raise in the introduction:

    "Such a process is crucial for deliberation, since identifying disagreements is the first step to resolve them (Waldron, 1999). However, a solid theoretical study of disagreement measures, and how to elicit them, seems to be lacking."

    One way to answer all of these questions is to present a concrete application.

  2. Cognitive Load. I genuinely liked the points you make about the cognitive load of elicitation. Right now they're kind of squirreled away in the related work and in Section 4. I've had a lot of firsthand experience with this and find it compelling. I'd encourage you to try using it as a central motivation, at least for some of the elicitation results. Some ways of eliciting preferences, like asking people to rank things, are much easier than answering lots of pairwise questions. Stacks could work as a running example: people there are asked to make many, many comparisons and often get tired quickly.

  3. Selective Rankings. One line of work I think is relevant is Selective Preference Aggregation (Kadekodi et al., ICML 2025). It's recent work on building rankings that respond to disagreement and reveal it. The idea is to return a tiered ranking that groups items together instead of trying to arbitrate disagreement. I took a look at it, and I think it's probably worth more than a citation. In particular, do the 2 papers use some of the same primitives? It could also work as a motivating example that shows how the plurality matrix captures something about the structure of disagreement that a tiered ranking reveals.

  4. Active Learning for Ranking. One other line of work I thought about here is active learning for ranking. To be fair, you do mention some of this in the related work (e.g., Saha & Gopalan, 2019). Papers like An Active Learning Algorithm for Ranking from Pairwise Preferences with an Almost Optimal Query Complexity (Ailon, JMLR 2012) and Preference Elicitation as Average-Case Sorting (Peters and Procaccia, AAAI 2021) give concrete bounds on how many pairwise questions a person would need to answer. I get that your setting is different, but I'd like to see a few sentences in your related work on how your protocols differ from theirs and whether their results carry over to your setting. Right now it's hard to tell how your work relates to these results.

Minor Comments

  • The A/B/C example in the introduction would be easier to follow with a picture.
  • I would present the 3 disagreement measures in Section 2.1 in a table - it's easier to parse and will save space.

Nits

Page numbers are PDF pages.

Main text

  • p. 1: "be them political candidates" → "be they political candidates".
  • p. 1: "some of them focused at identifying" → "focused on identifying".
  • p. 1: "most-agreed upon" → "most-agreed-upon".
  • p. 2: "our framework allows to identify" → "allows us to identify".
  • p. 2: "the number of users that needs to be queried" → "need to be queried".
  • p. 3: "one of the main concept of our paper" → "concepts".
  • p. 3: "we refer to the data or information of degree k to the sub-matrix" → "refer to … as the sub-matrix".
  • p. 3: "a real-value statistic" → "real-valued".
  • p. 4: "the preferences profiles" → "the preference profiles".
  • p. 4: "struggles to differ a uniform profile from" → "struggles to distinguish a uniform profile from".
  • p. 4: "the other metrics manages to do it" → "manage".
  • p. 4: "dedicated to study the hierarchy of the different degrees information" → "dedicated to studying the hierarchy of the different degrees of information".
  • p. 5: "straighforward" → "straightforward".
  • p. 5: "actually exists, rises" → "arises".
  • p. 6: "distinct structures of disagreement that is detectable" → "that are detectable".
  • p. 6: the Figure 2 caption says "Speakman Mark" but the text says "Mark Speakman".
  • p. 7: "The proof of the following result is Appendix B" → "is in Appendix B".
  • p. 7: "need not to be accessed" → "need not be accessed".
  • pp. 7 and 18, Proposition 3.7: the single-peaked formula uses s_{j−1} where it should be s_{i_{j−1}} (and the endpoint conventions drop the i_j indices too). In the proof, the three-element base case is circular, and the Plackett–Luce reconstruction divides by p_rr, which isn't defined.
  • p. 8: "see Appendix ??" is an unresolved reference.
  • p. 9: "an hierarchy" → "a hierarchy".
  • p. 9: "Extending our framework … suggest interesting directions" → "suggests".
  • p. 9: the last two sentences of the Table 3 paragraph say the same thing; one can go.

Appendix

  • p. 15: "X_b X_b" should be "X_b X_c" in the rank-variance proof.
  • p. 16: the heading "Computing E[r_a | b ≻ a]" appears twice; the second should be "Computing E[r_a | a ≻ b]".
  • p. 16: "Plugging these both expectation terms back" → "Plugging both of these expectation terms back".
  • p. 18: "strenght" → "strength".
  • p. 19: "can not" → "cannot".
  • p. 20: "how the notions of 'central' and 'extreme' translates" → "translate".
  • p. 20: "their preference decrease" → "their preferences decrease".
  • p. 20: "corresponding to middle alternative of the axis" → "the middle alternative".
  • p. 20: "c_m ≻ c_{m−2} ≻ … ≻ c_1" → "c_m ≻ c_{m−1} ≻ c_{m−2} ≻ … ≻ c_1".
  • p. 21: "In contrasts, Mallows mixtures" → "In contrast".
  • p. 21: "The plurality matrix of the five distributions … are identical" → "is identical".
  • p. 21: "Subsec. D.1 gives the long-form expressions" → "Subsec. D.2".
  • p. 22: "seven different preferences profiles" → "preference profiles".
  • p. 22: "Eq. equation 10" and "equation 10–equation 12" → "Eq. 10" and "Eqs. 10–12".
Reviewer_za7dOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary
  • The paper looks at how much preference information is needed to compute measures of collective disagreement. Its main tool is the plurality matrix, where each entry is the probability that an alternative is ranked first within a subset.
  • They prove some technical results, and use electoral and synthetic data to motivate skewness and kurtosis to describe different rank distributions
Critical strengths

They give an interesting way to compare disagreement measures by the amount of preference information they require. The agreement index, rank variance, divisiveness, and central moments, explain why pairwise data can miss important differences in rank distributions. This is useful application to a broad range of existing voting rules and disagreement measures.

Critical weaknesses
  • There are numerous errors throughout the document (e.g., line 393: Appendix ??, C1 and C2 are both Synthetic Data, proof of Prop 3.2 has typos, along with many others); it doesn't seem like the paper was checked carefully. The authors should review the document to fix these (very obvious) errors; there may be other ones affecting correctness of proofs that I may not have identified.
  • There are also some correctness issues. Thm 3.6 allows k = 1 and k = m, but the first central moment is always 0 and level m+1 is not defined. Also, πmin\pi_\text{min} in eg 2.5 does not exist: 14 other alternatives would reach have mean rank 8, while a has mean rank 14.3, so expected ranks sum to 126.3 (and not 120). The correctness of results needs to be verified more carefully.
  • elicitation results estimate every entry of the plurality matrix, but they don't actually give an accuracy bound for the disagreement measures themselves. I think this is important because divisiveness contains pabp_{ab} and 1−pab1-p_{ab} in the denominator, and skewness and kurtosis divide by powers of the variance; small errors can then cause large errors in the final value.
Questions for authors
  • See the points listed under critical weaknesses.
Reviewer_NFRiOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks how much preference information is needed to compute a measure of disagreement over a set of alternatives. This information is organized in a so called plurality matrix, which gives, for every subset of alternatives, the probability that each member is ranked first within it. Disagreement measures are classified into levels according to the largest subset size whose choice probabilities they need (eg, rank variance and divisiveness have level 3, and the k-th moment of an alternative's rank has level k+1). The hierarchy is strict in general: each level contains information that the lower levels do not. Under Plackett-Luce (since item strengths are determined) or single-peaked preferences pairwise comparisons determine all levels. The paper then bounds the number of voters two elicitation protocols need, and validates its results on election datasets, synthetic profiles and simulations.

Critical strengths
  • the level of a disagreement measure is a simple and usable notion with a clear practical consequence. Namely, by collecting only pairwise votes one cannot compute rank variance or divisiveness, and both become computable once triples are elicited
  • in general, the paper is clear, formal, and precise.
Critical weaknesses

a) The paper failed to compare with the closest related work, which significantly reduces the claimed novelty.

  • The plurality matrix is exactly the object studied by Chierichetti et al. (Tight Bounds for Learning RUMs from Small Slates, NeurIPS 2024), which is never cited. Chierichetti et al winning distribution of a slate SS is the row pS(⋅)p_S(\cdot), and their slates of size at most kk are the paper's degree-kk information. Furthermore, their Theorem 12 uses a more general form of the construction in Prop. 3.4 (a focal alternative with the rest placed at random) to build profiles that agree on all subsets up to size Θ(m)\Theta(\sqrt{m}) but differ on the full set, and their Theorem 3 shows that the entries on subsets of size up to O(mlog⁡(1/ε))O(\sqrt{m\log(1/\varepsilon)}) determine all larger entries within ε\varepsilon, so higher-degree information is unrecoverable (Section 3.3) only in the exact sense.

  • The strict hierarchy and the moment levels (Prop. 3.4, Thm. 3.6, Cor. B.3) follow from Halpern et al. (Computing Voting Rules with Elicited Incomplete Votes, EC 2024, which is cited only in passing and not really compared) and Bentert and Skowron (Comparing Election Methods Where Each Voter Ranks Only Few Candidates, AAAI 2020, again completely not cited).

b) the elicitation part does not support the claim of efficient elicitation of disagreement. Theorem 4.5 bounds the error of each matrix entry, not of the measures, which sum many entries, so the per-entry guarantee does not translate into a useful guarantee on the quantity of interest. The protocols estimate all k(mk)k\binom{m}{k} entries of degree kk, so the voter counts grow like mkm^k, while the measures depend on the higher-degree entries only through sums, such as positional scores, which Bentert and Skowron already estimate directly. The protocols are evaluated only at m=10m=10, where rankings need fewer voters and fewer comparisons in total than chains

Questions for authors

Please discuss the weaknesses and compare with the articles above.

Minors:

  • How are the inexpressible entries of Table 4 proved? and which definition of the distance-based diversity measures did you use? Under the definition implicit in the ΔKT\Delta_{\mathrm{KT}} entry (Kendall-tau diversity, ie expected distance between two independent voters), the Spearman entry seems to be wrong. The expected squared Spearman distance equals 2∑aVar(ra)2\sum_a \mathrm{Var}(r_a), and since ∑aE[ra2]\sum_a \mathbb{E}[r_a^2] is the same for every profile, it is a function of the Borda scores alone, i.e. level 2. Please clarify.

  • In the proof of Prop. 3.7, the display for p{a,b,c}(a)p_{\{a,b,c\}}(a) ends with pab−(pab−p{a,b,c}(a))p_{ab}-(p_{ab}-p_{\{a,b,c\}}(a)), which is circular. Is the intended argument that c≻a≻bc\succ a\succ b has probability zero under single-peakedness, so that p{a,b,c}(a)=pabp_{\{a,b,c\}}(a)=p_{ab} directly? Please clarify this as well.

(- the fit to ICLR is a bit weak: no learning is involved and the natural readers are in computational social choice)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.