- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper sorts disagreement measures by "level": the smallest subset size you need to compute the measure. It introduces a plurality matrix that records, for every subset of alternatives, how likely each alternative is to rank first. It shows that rank variance and divisiveness have level 3, and that skewness and excess kurtosis have levels 4 and 5. It then gives 2 elicitation protocols that trade off how many people you ask against how much you ask each person, with bounds on both. It illustrates the results on real election data and synthetic preference profiles.
Critical strengths
- I like the focus on cognitive load. Revealing richer kinds of disagreement means collecting more preference information, and people get tired of answering lots of pairwise questions. I think this is the most practical part of the paper.
- I appreciate the combinatorial structure of the plurality matrix. Because it covers every subset, I imagine it can catch effects like this one: the alternative that wins every head-to-head may not be the top choice in a bigger set. Say 40 people rank A > B > C, 35 rank B > C > A, and 25 rank C > B > A. B beats A and C head-to-head, but A gets the most first-place votes.
Critical weaknesses
-
Gap Between Theory and Motivation. I agree that it's important to reveal disagreement and help people address it. But I don't see how the contributions do that. The paper introduces the plurality matrix, defines levels, proves hierarchy results, and gives elicitation protocols. I still don't understand why it matters that a measure has level 3, 4, or 5, why I should care about any one measure, or what the plurality matrix tells me that a simple matrix of pairwise preference counts would miss. The A/B/C example shows that 2 populations can look the same at level 2 and differ at level 3, but not why that difference matters. The elicitation bounds tell me how many people and comparisons I need, but not how much elicitation I'd actually have to do for a real problem, or what I'd get for it. There's a lot of theory here and a missing connection to something practical.
-
Inconsistencies in the Formalization. The plurality matrix covers every subset of alternatives, but the model assumes every person has a strict ranking. That rules out ties and intransitive preferences. If this is meant to be a foundational, general-purpose object for disagreement, I think it should handle the most general case: pairwise preferences, ties, and intransitivity. Restricting to strict rankings might be fine, but then the authors should justify it. Right now the limitation seems to defeat the purpose of the thoroughness in the plurality matrix.
Questions for authors
Clarification Questions
- What does the plurality matrix tell us about disagreement that a matrix of pairwise preference counts doesn't?
- Why should we care that a measure has level 3, 4, or 5? What does knowing that let someone conclude or do?
- For a real problem, how much elicitation would I actually need to do?
- Existing measures, such as the diversity measures of Hashemi & Endriss (2014) and Karpov (2017) or the polarization measure of Can et al. (2015), are defined on full preference profiles. What do they miss? What concrete problem does that cause?
- The plurality matrix covers every subset, but the model only allows strict rankings. What's the value of being thorough in one place but not the other? Can the framework handle ties or intransitive preferences? If not, why are strict rankings the right scope?
Comments
-
Concrete Application. You'll notice that a lot of my clarification questions are really about motivation. What I'm really missing is how the plurality matrix ties back to the broader motivation of revealing and understanding disagreement. I do like that there are real-world experiments. What's missing is how a theoretical study could address the concerns you raise in the introduction:
"Such a process is crucial for deliberation, since identifying disagreements is the first step to resolve them (Waldron, 1999). However, a solid theoretical study of disagreement measures, and how to elicit them, seems to be lacking."
One way to answer all of these questions is to present a concrete application.
-
Cognitive Load. I genuinely liked the points you make about the cognitive load of elicitation. Right now they're kind of squirreled away in the related work and in Section 4. I've had a lot of firsthand experience with this and find it compelling. I'd encourage you to try using it as a central motivation, at least for some of the elicitation results. Some ways of eliciting preferences, like asking people to rank things, are much easier than answering lots of pairwise questions. Stacks could work as a running example: people there are asked to make many, many comparisons and often get tired quickly.
-
Selective Rankings. One line of work I think is relevant is Selective Preference Aggregation (Kadekodi et al., ICML 2025). It's recent work on building rankings that respond to disagreement and reveal it. The idea is to return a tiered ranking that groups items together instead of trying to arbitrate disagreement. I took a look at it, and I think it's probably worth more than a citation. In particular, do the 2 papers use some of the same primitives? It could also work as a motivating example that shows how the plurality matrix captures something about the structure of disagreement that a tiered ranking reveals.
-
Active Learning for Ranking. One other line of work I thought about here is active learning for ranking. To be fair, you do mention some of this in the related work (e.g., Saha & Gopalan, 2019). Papers like An Active Learning Algorithm for Ranking from Pairwise Preferences with an Almost Optimal Query Complexity (Ailon, JMLR 2012) and Preference Elicitation as Average-Case Sorting (Peters and Procaccia, AAAI 2021) give concrete bounds on how many pairwise questions a person would need to answer. I get that your setting is different, but I'd like to see a few sentences in your related work on how your protocols differ from theirs and whether their results carry over to your setting. Right now it's hard to tell how your work relates to these results.
Minor Comments
- The A/B/C example in the introduction would be easier to follow with a picture.
- I would present the 3 disagreement measures in Section 2.1 in a table - it's easier to parse and will save space.
Nits
Page numbers are PDF pages.
Main text
- p. 1: "be them political candidates" → "be they political candidates".
- p. 1: "some of them focused at identifying" → "focused on identifying".
- p. 1: "most-agreed upon" → "most-agreed-upon".
- p. 2: "our framework allows to identify" → "allows us to identify".
- p. 2: "the number of users that needs to be queried" → "need to be queried".
- p. 3: "one of the main concept of our paper" → "concepts".
- p. 3: "we refer to the data or information of degree k to the sub-matrix" → "refer to … as the sub-matrix".
- p. 3: "a real-value statistic" → "real-valued".
- p. 4: "the preferences profiles" → "the preference profiles".
- p. 4: "struggles to differ a uniform profile from" → "struggles to distinguish a uniform profile from".
- p. 4: "the other metrics manages to do it" → "manage".
- p. 4: "dedicated to study the hierarchy of the different degrees information" → "dedicated to studying the hierarchy of the different degrees of information".
- p. 5: "straighforward" → "straightforward".
- p. 5: "actually exists, rises" → "arises".
- p. 6: "distinct structures of disagreement that is detectable" → "that are detectable".
- p. 6: the Figure 2 caption says "Speakman Mark" but the text says "Mark Speakman".
- p. 7: "The proof of the following result is Appendix B" → "is in Appendix B".
- p. 7: "need not to be accessed" → "need not be accessed".
- pp. 7 and 18, Proposition 3.7: the single-peaked formula uses s_{j−1} where it should be s_{i_{j−1}} (and the endpoint conventions drop the i_j indices too). In the proof, the three-element base case is circular, and the Plackett–Luce reconstruction divides by p_rr, which isn't defined.
- p. 8: "see Appendix ??" is an unresolved reference.
- p. 9: "an hierarchy" → "a hierarchy".
- p. 9: "Extending our framework … suggest interesting directions" → "suggests".
- p. 9: the last two sentences of the Table 3 paragraph say the same thing; one can go.
Appendix
- p. 15: "X_b X_b" should be "X_b X_c" in the rank-variance proof.
- p. 16: the heading "Computing E[r_a | b ≻ a]" appears twice; the second should be "Computing E[r_a | a ≻ b]".
- p. 16: "Plugging these both expectation terms back" → "Plugging both of these expectation terms back".
- p. 18: "strenght" → "strength".
- p. 19: "can not" → "cannot".
- p. 20: "how the notions of 'central' and 'extreme' translates" → "translate".
- p. 20: "their preference decrease" → "their preferences decrease".
- p. 20: "corresponding to middle alternative of the axis" → "the middle alternative".
- p. 20: "c_m ≻ c_{m−2} ≻ … ≻ c_1" → "c_m ≻ c_{m−1} ≻ c_{m−2} ≻ … ≻ c_1".
- p. 21: "In contrasts, Mallows mixtures" → "In contrast".
- p. 21: "The plurality matrix of the five distributions … are identical" → "is identical".
- p. 21: "Subsec. D.1 gives the long-form expressions" → "Subsec. D.2".
- p. 22: "seven different preferences profiles" → "preference profiles".
- p. 22: "Eq. equation 10" and "equation 10–equation 12" → "Eq. 10" and "Eqs. 10–12".