- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper addresses the problem that traditional preference aggregation methods tend to impose a deterministic ranking even when there is substantial disagreement among individuals. The authors argue that conventional ranking methods may produce a definitive ordering even when the support for competing preferences is nearly equal, potentially obscuring differences in population preferences.
To address this issue, the paper proposes SE-RO, a statistically grounded selective ranking method. SE-RO introduces two separate thresholds: a population support floor (σ) and a posterior evidence threshold (α). A pairwise preference is admitted only when there is sufficient posterior probability that its population support exceeds the specified support floor.
Specifically, SE-RO first selects pairwise comparisons according to these thresholds, constructs a directed graph, and then applies strongly connected component consolidation and topological sorting to produce a tiered partial ranking. The paper also provides theoretical analyses of ranking recovery, population support guarantees, and robustness to recording bias.
Critical strengths
1. The paper addresses a meaningful problem in preference aggregation. The paper highlights that existing ranking methods may overlook heterogeneous preferences and disagreements among different groups. This is a relevant issue, particularly in the context of pluralistic alignment.
2. The proposed method is simple, intuitive, and interpretable. SE-RO has a straightforward design, and the two parameters, σ and α, have clear interpretations. They provide an intuitive way to control the trade-off between population support, statistical evidence, and ranking coverage.
Critical weaknesses
1. The necessity and practical value of the proposed research objective are not sufficiently justified. The primary objective of this paper is not to improve ranking accuracy, but to selectively retain comparisons with sufficiently broad population support and statistical evidence by introducing additional thresholds. However, the authors do not sufficiently justify why this selective ranking objective is more appropriate than traditional majority-based ranking for downstream alignment tasks. For example, consider a pairwise comparison where 55% of the population prefers A over B, while 45% prefers B over A. Given representative sampling and sufficient statistical evidence, ranking A above B is not statistically incorrect. However, SE-RO would withhold this comparison when σ = 0.60. This introduces an additional constraint on population consensus rather than directly improving the accuracy of majority-preference ranking. Although the paper provides a theoretical interpretation of σ in terms of minority-group protection, it does not sufficiently discuss how σ and α should be selected in practical applications or what concrete benefits different threshold choices bring to downstream models. Furthermore, while the paper experimentally explores threshold values close to 0.5, the theoretical relationship between SE-RO and classical majority-based ranking methods as both thresholds approach 0.5 deserves further investigation.
2. The methodological novelty is relatively limited. The main methodological contribution is the integration of population support and statistical evidence requirements into a selective ranking framework. However, the overall technical approach is relatively straightforward and largely builds upon existing techniques, including Beta-Binomial posterior inference and graph-based tier construction. The current experiments do not sufficiently demonstrate that this combination offers substantial advantages over simpler statistical confidence-based filtering approaches. Therefore, the methodological contribution appears to be primarily an incremental extension and combination of existing techniques, with limited algorithmic novelty.
3. The experimental comparisons and ablation studies are insufficient. Although the authors compare SE-RO with classical ranking methods such as Borda, Copeland, Rank Centrality, and MB–K, as well as selective rules such as Vote Share and BT-Margin, the evaluation lacks stronger baselines that more directly address the proposed research objective. For example, selective ranking methods based on exact binomial tests or statistical confidence intervals would provide more direct comparisons. Additional comparisons with classical tournament-based ranking approaches, such as Kemeny ranking where appropriate, could also provide useful insights. Moreover, the experiments do not sufficiently isolate the contributions of the individual components. In particular, the necessity of constructing a tiered partial order, rather than directly retaining statistically supported pairwise comparisons or using a more general partial order, is not adequately investigated.
Questions for authors
1. Further clarify the research motivation and practical value of selective ranking. The authors should better explain why statistically reliable majority preferences should be withheld when they do not meet a predefined population support threshold, and how this choice benefits real-world alignment applications. The paper would also benefit from more principled guidance on selecting σ and α, together with a more systematic analysis of the theoretical relationship between SE-RO and classical majority-based ranking as these thresholds approach 0.5.
2. Include stronger and more directly relevant baselines and ablation studies. I suggest including selective ranking baselines based on exact binomial tests and statistical confidence intervals, and comparing them with SE-RO at matched coverage levels. Furthermore, ablation studies should be conducted to evaluate the individual contributions of the posterior-based admission rule and the graph-based tier construction. In particular, comparing tiered partial rankings with more general partial-order representations would help justify the necessity of the proposed ranking structure.
3. Strengthen the evaluation of downstream alignment performance. The authors should consider using SE-RO-filtered preference labels in actual preference-learning pipelines, such as DPO or reward model training. The resulting models could then be compared against models trained on the original preference data or data filtered by simpler confidence-based methods. Such experiments would provide stronger evidence of whether SE-RO offers practical advantages for downstream alignment beyond improving the coverage–agreement trade-off in selective ranking tasks.
3 comments
Follow up to reviewers
As discussion closes today, we would welcome any feedback on whether our responses addressed your concerns.
Thank you for your detailed response and additional experiments. I appreciate the authors' efforts to address my concerns. However, I believe my main concerns remain unresolved. While the authors clarify the motivation and theoretical properties of SE-RO, they do not convincingly demonstrate why selective withholding leads to better practical LLM alignment outcomes than conventional preference aggregation methods. Furthermore, the methodological novelty remains limited, and the learned withholding experiment does not directly validate the effectiveness of SE-RO in actual DPO or reward model training. Therefore, despite the additional clarifications, I will maintain my original scores.
Response to Reviewer FggF
Thank you for your time and detailed review! We would like to address your points below:
> …55% of the population prefers A over B … SE-RO would withhold this comparison when σ = 0.60.
We agree, and this is why σ is a choice. At σ and α of 0.5⁺, SE-RO is nearly a majority vote, as we now prove in Proposition 14. With σ below 0.55, SE-RO ranks A above B. In Figure 4, we show what SE-RO displays and overrules as σ moves from 0.75 to 0.51. A floor of 0.60 is for uses where overruling 45% of people has an unacceptable cost.
We provide an example on Sushi where we may wish to avoid overruling a group. Overall, 58.8% of participants prefer sea urchin to egg, but 63.5% of those aged 19 or younger prefer egg. All four baselines rank sea urchin above egg in all 200 samples, while SE-RO withholds the pair at σ = 0.60.
> …how σ and α should be selected in practical applications …
σ sets the level of consensus a displayed comparison needs. A practitioner may raise σ when overruling a large minority is costly, for example on political or safety questions. They may lower σ toward ½ when a majority answer is good enough. A floor of σ = 0.80 excludes any comparison opposed by most of a group holding at least 40% of the population.
α sets the level of certainty SE-RO needs before it displays a comparison. In Corollary 3, we bound the posterior chance that a displayed comparison lacks support by 1 − α. A practitioner may raise α when a wrong display is costly. They may lower α to display more when some errors are acceptable.
Both can also be chosen after seeing the output. In Proposition 6, we prove that raising σ or α never adds a display. A user can run SE-RO at several values and keep the ranking that suits the task, as we show in Figure 4.
> …the theoretical relationship between SE-RO and classical majority-based ranking methods as both thresholds approach 0.5 …
We've added this to the paper as Proposition 14 and Corollary 15. SE-RO admits a direction only if it wins a strict majority of the observed votes. Once σ and α are close enough to ½, it admits every such direction. Every Kemeny ranking places the resulting tiers in the same order. Kemeny therefore differs from SE-RO only on the pairs where SE-RO abstains.
> The methodological novelty is relatively limited. …
SE-RO is a simple idea, but it is novel. It separates the support a comparison needs, σ, from the evidence required, α. This separation makes SE-RO pluralistic. A confidence filter on the majority direction treats disagreement as noise that more votes will resolve. With enough votes it orders every 55/45 pair and overrules the 45%. SE-RO keeps such a pair unordered as panels grow. More votes only confirm that 45% of people disagree.
> …selective ranking methods based on exact binomial tests or statistical confidence intervals … at matched coverage levels.
These results are in the Appendix, and Experiments now points to them. In Table 12, we show an exact test of the majority direction ordering 55/45 pairs as panels grow. We've added the same comparison on our four real datasets as Table 13. A rule based on the exact confidence interval admits the same directions as the exact test.
These comparisons are also the ablation of our admission rule. They replace it with an exact test and keep the tier step.
> …the necessity of constructing a tiered partial order …
Admitted pairs can include i ≻ j and j ≻ k without i ≻ k, or form cycles, as in our Cyclical population. A partial order is transitive, so it must then drop an admitted pair. The data do not say which. In Theorem 2, we prove that SE-RO returns the finest valid tiered ranking. We've added a paragraph on this to the Algorithm section. For the tier step, we report admitted, hidden and displayed pairs on every dataset in Table 9.
In settings where coverage is critical or ratings are unevenly sampled, practitioners are free to drop the tier step. Those who expect transitive preferences can also display the comparisons that admitted pairs imply. We note that transitivity does not hold in general. Individual preferences can be intransitive (Tversky, 1969), and majorities can cycle, as in a 2021 Minneapolis election (McCune and McCune, 2026).
> …SE-RO-filtered preference labels in actual preference-learning pipelines, such as DPO …
In Learned Withholding, we show the input process to DPO and reward models. SE-RO labels each pair of responses with its admitted direction, or withholds the pair. An admitted direction gives a prompt with a preferred and a rejected response. A model trained on these labels learns to withhold, and its rankings overrule fewer held-out voters than BT.
Thank you again for the detailed review. In your review, you list soundness at 1. We hope our clarifications have addressed your concerns. If not, please point us to the claim, proof or experiment you have in mind.