acceptodds
Under review as a conference paper at ICLR 2027

Selective Rankings from Heterogeneous Incomplete Preferences

Abstract

Many model alignment and recommendation systems aggregate diverse human preferences into a single ranking, concealing substantial disagreement and favoring a narrow majority. When preferences are only partially observed, even the direction of the population majority may be uncertain. We introduce SE-RO (Selective Evidence-based Rank Ordering), a method for inferring selective rankings from incomplete pairwise preferences. SE-RO constructs tiered partial orders with separate requirements for how broadly a preference must be shared and how much posterior evidence is needed to display it. Under representative sampling, we bound the risk of displaying comparisons with insufficient population support and give conditions for recovering the population ranking. We also quantify how much recording bias a fixed ranking can tolerate. We use real-world preference tasks and synthetic populations to test common ranking failures and characterize the trade-off between population support, evidence, and coverage. Our results show how selective rankings can avoid overruling divided groups while retaining comparisons with broad population support.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
pluralistic alignment, preference aggregation, abstention, incomplete preferences, uncertainty quantification, social choice, pairwise comparisons

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_FggFOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the problem that traditional preference aggregation methods tend to impose a deterministic ranking even when there is substantial disagreement among individuals. The authors argue that conventional ranking methods may produce a definitive ordering even when the support for competing preferences is nearly equal, potentially obscuring differences in population preferences.

To address this issue, the paper proposes SE-RO, a statistically grounded selective ranking method. SE-RO introduces two separate thresholds: a population support floor (σ) and a posterior evidence threshold (α). A pairwise preference is admitted only when there is sufficient posterior probability that its population support exceeds the specified support floor.

Specifically, SE-RO first selects pairwise comparisons according to these thresholds, constructs a directed graph, and then applies strongly connected component consolidation and topological sorting to produce a tiered partial ranking. The paper also provides theoretical analyses of ranking recovery, population support guarantees, and robustness to recording bias.

Critical strengths

1. The paper addresses a meaningful problem in preference aggregation. The paper highlights that existing ranking methods may overlook heterogeneous preferences and disagreements among different groups. This is a relevant issue, particularly in the context of pluralistic alignment.

2. The proposed method is simple, intuitive, and interpretable. SE-RO has a straightforward design, and the two parameters, σ and α, have clear interpretations. They provide an intuitive way to control the trade-off between population support, statistical evidence, and ranking coverage.

Critical weaknesses

1. The necessity and practical value of the proposed research objective are not sufficiently justified. The primary objective of this paper is not to improve ranking accuracy, but to selectively retain comparisons with sufficiently broad population support and statistical evidence by introducing additional thresholds. However, the authors do not sufficiently justify why this selective ranking objective is more appropriate than traditional majority-based ranking for downstream alignment tasks. For example, consider a pairwise comparison where 55% of the population prefers A over B, while 45% prefers B over A. Given representative sampling and sufficient statistical evidence, ranking A above B is not statistically incorrect. However, SE-RO would withhold this comparison when σ = 0.60. This introduces an additional constraint on population consensus rather than directly improving the accuracy of majority-preference ranking. Although the paper provides a theoretical interpretation of σ in terms of minority-group protection, it does not sufficiently discuss how σ and α should be selected in practical applications or what concrete benefits different threshold choices bring to downstream models. Furthermore, while the paper experimentally explores threshold values close to 0.5, the theoretical relationship between SE-RO and classical majority-based ranking methods as both thresholds approach 0.5 deserves further investigation.

2. The methodological novelty is relatively limited. The main methodological contribution is the integration of population support and statistical evidence requirements into a selective ranking framework. However, the overall technical approach is relatively straightforward and largely builds upon existing techniques, including Beta-Binomial posterior inference and graph-based tier construction. The current experiments do not sufficiently demonstrate that this combination offers substantial advantages over simpler statistical confidence-based filtering approaches. Therefore, the methodological contribution appears to be primarily an incremental extension and combination of existing techniques, with limited algorithmic novelty.

3. The experimental comparisons and ablation studies are insufficient. Although the authors compare SE-RO with classical ranking methods such as Borda, Copeland, Rank Centrality, and MB–K, as well as selective rules such as Vote Share and BT-Margin, the evaluation lacks stronger baselines that more directly address the proposed research objective. For example, selective ranking methods based on exact binomial tests or statistical confidence intervals would provide more direct comparisons. Additional comparisons with classical tournament-based ranking approaches, such as Kemeny ranking where appropriate, could also provide useful insights. Moreover, the experiments do not sufficiently isolate the contributions of the individual components. In particular, the necessity of constructing a tiered partial order, rather than directly retaining statistically supported pairwise comparisons or using a more general partial order, is not adequately investigated.

Questions for authors

1. Further clarify the research motivation and practical value of selective ranking. The authors should better explain why statistically reliable majority preferences should be withheld when they do not meet a predefined population support threshold, and how this choice benefits real-world alignment applications. The paper would also benefit from more principled guidance on selecting σ and α, together with a more systematic analysis of the theoretical relationship between SE-RO and classical majority-based ranking as these thresholds approach 0.5.

2. Include stronger and more directly relevant baselines and ablation studies. I suggest including selective ranking baselines based on exact binomial tests and statistical confidence intervals, and comparing them with SE-RO at matched coverage levels. Furthermore, ablation studies should be conducted to evaluate the individual contributions of the posterior-based admission rule and the graph-based tier construction. In particular, comparing tiered partial rankings with more general partial-order representations would help justify the necessity of the proposed ranking structure.

3. Strengthen the evaluation of downstream alignment performance. The authors should consider using SE-RO-filtered preference labels in actual preference-learning pipelines, such as DPO or reward model training. The resulting models could then be compared against models trained on the original preference data or data filtered by simpler confidence-based methods. Such experiments would provide stronger evidence of whether SE-RO offers practical advantages for downstream alignment beyond improving the coverage–agreement trade-off in selective ranking tasks.

3 comments
AuthorsOpenReview ↗

Response to Reviewer FggF

Thank you for your time and detailed review! We would like to address your points below:

> …55% of the population prefers A over B … SE-RO would withhold this comparison when σ = 0.60.

We agree, and this is why σ is a choice. At σ and α of 0.5⁺, SE-RO is nearly a majority vote, as we now prove in Proposition 14. With σ below 0.55, SE-RO ranks A above B. In Figure 4, we show what SE-RO displays and overrules as σ moves from 0.75 to 0.51. A floor of 0.60 is for uses where overruling 45% of people has an unacceptable cost.

We provide an example on Sushi where we may wish to avoid overruling a group. Overall, 58.8% of participants prefer sea urchin to egg, but 63.5% of those aged 19 or younger prefer egg. All four baselines rank sea urchin above egg in all 200 samples, while SE-RO withholds the pair at σ = 0.60.

> …how σ and α should be selected in practical applications …

σ sets the level of consensus a displayed comparison needs. A practitioner may raise σ when overruling a large minority is costly, for example on political or safety questions. They may lower σ toward ½ when a majority answer is good enough. A floor of σ = 0.80 excludes any comparison opposed by most of a group holding at least 40% of the population.

α sets the level of certainty SE-RO needs before it displays a comparison. In Corollary 3, we bound the posterior chance that a displayed comparison lacks support by 1 − α. A practitioner may raise α when a wrong display is costly. They may lower α to display more when some errors are acceptable.

Both can also be chosen after seeing the output. In Proposition 6, we prove that raising σ or α never adds a display. A user can run SE-RO at several values and keep the ranking that suits the task, as we show in Figure 4.

> …the theoretical relationship between SE-RO and classical majority-based ranking methods as both thresholds approach 0.5 …

We've added this to the paper as Proposition 14 and Corollary 15. SE-RO admits a direction only if it wins a strict majority of the observed votes. Once σ and α are close enough to ½, it admits every such direction. Every Kemeny ranking places the resulting tiers in the same order. Kemeny therefore differs from SE-RO only on the pairs where SE-RO abstains.

> The methodological novelty is relatively limited. …

SE-RO is a simple idea, but it is novel. It separates the support a comparison needs, σ, from the evidence required, α. This separation makes SE-RO pluralistic. A confidence filter on the majority direction treats disagreement as noise that more votes will resolve. With enough votes it orders every 55/45 pair and overrules the 45%. SE-RO keeps such a pair unordered as panels grow. More votes only confirm that 45% of people disagree.

> …selective ranking methods based on exact binomial tests or statistical confidence intervals … at matched coverage levels.

These results are in the Appendix, and Experiments now points to them. In Table 12, we show an exact test of the majority direction ordering 55/45 pairs as panels grow. We've added the same comparison on our four real datasets as Table 13. A rule based on the exact confidence interval admits the same directions as the exact test.

These comparisons are also the ablation of our admission rule. They replace it with an exact test and keep the tier step.

> …the necessity of constructing a tiered partial order …

Admitted pairs can include i ≻ j and j ≻ k without i ≻ k, or form cycles, as in our Cyclical population. A partial order is transitive, so it must then drop an admitted pair. The data do not say which. In Theorem 2, we prove that SE-RO returns the finest valid tiered ranking. We've added a paragraph on this to the Algorithm section. For the tier step, we report admitted, hidden and displayed pairs on every dataset in Table 9.

In settings where coverage is critical or ratings are unevenly sampled, practitioners are free to drop the tier step. Those who expect transitive preferences can also display the comparisons that admitted pairs imply. We note that transitivity does not hold in general. Individual preferences can be intransitive (Tversky, 1969), and majorities can cycle, as in a 2021 Minneapolis election (McCune and McCune, 2026).

> …SE-RO-filtered preference labels in actual preference-learning pipelines, such as DPO …

In Learned Withholding, we show the input process to DPO and reward models. SE-RO labels each pair of responses with its admitted direction, or withholds the pair. An admitted direction gives a prompt with a preferred and a rejected response. A model trained on these labels learns to withhold, and its rankings overrule fewer held-out voters than BT.

Thank you again for the detailed review. In your review, you list soundness at 1. We hope our clarifications have addressed your concerns. If not, please point us to the claim, proof or experiment you have in mind.

AuthorsOpenReview ↗

Follow up to reviewers

As discussion closes today, we would welcome any feedback on whether our responses addressed your concerns.

Thank you for your detailed response and additional experiments. I appreciate the authors' efforts to address my concerns. However, I believe my main concerns remain unresolved. While the authors clarify the motivation and theoretical properties of SE-RO, they do not convincingly demonstrate why selective withholding leads to better practical LLM alignment outcomes than conventional preference aggregation methods. Furthermore, the methodological novelty remains limited, and the learned withholding experiment does not directly validate the effectiveness of SE-RO in actual DPO or reward model training. Therefore, despite the additional clarifications, I will maintain my original scores.

Reviewer_W8xmOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes SE-RO, a method for constructing selective rankings from incomplete pairwise preferences.

Critical strengths

The paper examines recording bias, small support margins, prior sensitivity, and the loss of otherwise admissible comparisons during tier construction.

Critical weaknesses
  1. The methodological novelty over SPA is limited. The main methodological change is a Beta-posterior admission rule, concentration bounds and sensitivity analysis

  2. The main LM application doesn't follow representative-sampling assumption. In Community Alignment, comparisons are derived from each evaluator's top choice, so the observed comparisons depend on their preferences These experiments do not directly validate the population-support guarantees.

  3. The paper was extremely difficult to read. The authors need to rethink the exposition and find a clearer way to present their ideas. The current version is very hard to read therefore I might miss a lot of important details, and I think it would be hard to improve this draft’s readability during the rebuttal week.

Questions for authors

What is the methodological novelty of the paper?

3 comments
AuthorsOpenReview ↗

Response to Reviewer W8xm

Thank you for your review. We address your points below.

> The methodological novelty over SPA is limited. …

> What is the methodological novelty of the paper?

SPA ranks a finite panel and says nothing about people outside it. It is largely theoretical and has limited to no practical ML application. SE-RO applies across many domains. It does not need the same raters on every pair, and it does not need every pair voted on. It therefore handles data SPA cannot, such as RLHF preference labels and recommendation logs, where each person rates only a few pairs.

Our method creates a ranking with guarantees. The ranking represents the population the votes come from. In Corollary 3 and Theorem 7, we bound how often it displays a comparison the population does not support, and in Proposition 13 and Algorithm 2, we measure how much recording bias it withstands.

> The main LM application doesn't follow representative-sampling assumption … do not directly validate the population-support guarantees.

We agree that Learned Withholding does not follow our sampling assumptions, and that is why we include it. We cannot guarantee that real-world data follows our assumptions. On this data SE-RO beats Vote Share, agreeing with held-out voters 4.8 points more often on 598 further prompts.

This is also why we include Γ, our measure of how much biased recording each displayed comparison withstands. For experiments that follow our assumptions, see Experiments and its appendix.

> The paper was extremely difficult to read. …

We've updated the writing for clarity. In particular, we've updated Problem Statement to define the target ranking in plainer words, and Algorithm to better explain how the tiers form. We note that Reviewer J89Z found the paper clear and easy to follow. If parts of the paper remain unclear to you, please point us to them.

Thank you again for your time. Please let us know if you have further questions. If we have addressed your concerns, please consider updating your score.

AuthorsOpenReview ↗

Follow up to Reviewers

As discussion closes today, we would welcome any feedback on whether our responses addressed your concerns.

Thank you for the response and the revisions. My main concerns remain as follows.

Practical relevance and novelty. The response distinguishes SE-RO from SPA, but I remain unconvinced about the concrete benefits of this distinction in practical ML settings, particularly when the sampling assumptions underlying the guarantees do not hold. This concern is related to Reviewer FggF’s first question regarding practical value and second weakness regarding limited methodological novelty. The response does not sufficiently establish that the proposed approach offers meaningful advantages over simpler alternatives.

Presentation. I do not think citing another reviewer’s positive assessment of the writing adequately addresses my concern about readability. Reviewer FggF also described the paper as requiring “substantial effort in places.” Different reviewers can reasonably have different experiences, and another reviewer finding the paper clear does not invalidate the difficulties I encountered. Nevertheless, I appreciate your efforts to revise the presentation.

In recognition of these revisions, I have decided to raise my rating to 2. This reflects the effort to address the presentation concerns, rather than a resolution of my substantive concerns about novelty and practical relevance. My concern is not that theoretical work must include an immediate application, but that the contribution and its implications for machine learning remain insufficiently established to justify acceptance at ICLR.

Reviewer_J89ZOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SE-RO, a method for constructing selective tiered rankings from incomplete and heterogeneous pairwise preferences. It separates a population support threshold from a posterior evidence threshold, admitting comparisons only when there is sufficient evidence of broad population support. The admitted comparisons are consolidated into tiers using a graph-based construction. The paper provides finite-sample recovery guarantees under representative sampling, analyzes sensitivity to preference-dependent recording, and evaluates the method on synthetic and real-world preference datasets. It also demonstrates learned withholding on the Community Alignment dataset.

Critical strengths
  1. Well-motivated problem. The paper addresses an important limitation of conventional preference aggregation, which can obscure population disagreement by imposing nearly complete rankings. Separating population support from statistical evidence is conceptually clear and useful.
  2. Substantial theoretical analysis. The paper provides formal guarantees for tier construction, safe displays, and finite-evidence recovery, together with sensitivity analysis under bounded recording bias. The assumptions and limitations are generally stated explicitly.
  3. Broad empirical evaluation. The experiments cover several synthetic population structures and real-world preference datasets. The Community Alignment experiment provides evidence that SE-RO can achieve higher held-out agreement than Vote Share and BT-Margin at comparable coverage.
Critical weaknesses
  1. Limited methodological novelty relative to existing techniques. The method combines standard Beta–Binomial inference with the graph construction inherited from Selective Preference Aggregation (SPA). The comparison with an exact binomial test against the support threshold in Appendix C.6 is particularly relevant: both methods withhold contested comparisons, and their recall differs relatively modestly. The paper would benefit from a clearer demonstration of the additional methodological or practical value beyond this combination.
  2. Potentially substantial information loss from tier construction. The graph consolidation step can hide comparisons that individually satisfy the admission criterion. For example, at the default thresholds, the admitted and displayed fractions are 71.2% and 34.3% on EnviroSound, and 53.6% and 17.2% on ERS Set 77. On Jester, the method displays no comparisons despite admitting some directions. It remains unclear whether the restriction to tiered rankings is necessary for downstream preference-learning applications, compared with retaining a more general partial order.
  3. Limited validation of downstream language-model alignment. Section 5 trains a linear predictor over frozen language-model embeddings rather than a generative model using preference optimization. Although the learned predictor achieves higher agreement than the BT model, their coverage differs substantially (42.3% versus 100%). A coverage-matched selective learned baseline, and ideally a downstream preference-training experiment, would strengthen the alignment-related contribution.
  4. Dependence on representative observation assumptions. The population-support interpretation relies on representative pairwise sampling. The Community Alignment top-choice observation mechanism does not fully satisfy this assumption, as the authors acknowledge. The recording-bias experiments also demonstrate that additional data may reinforce incorrect decisions under preference-dependent recording. The proposed sensitivity analysis is useful, but real-world bias bounds remain unknown.
Questions for authors
  1. What practical or theoretical advantages does SE-RO provide over an exact binomial test against the same support threshold combined with the SPA tier-construction procedure? Can this comparison be extended beyond the synthetic Two Groups setting?
  2. Could the authors evaluate the use of admitted pairwise relations directly, without requiring a complete tiered ordering? How much downstream utility is lost by withholding already-admitted comparisons during graph consolidation?
  3. Can the learned withholding model be compared against a coverage-matched selective BT predictor or another uncertainty-aware learned baseline? Is there evidence that SE-RO-derived labels improve actual downstream preference optimization or pluralistic alignment of a generative model?
2 comments
AuthorsOpenReview ↗

Response to Reviewer J89Z

Thank you for your time and feedback! We appreciate your comments and concerns, which have helped us to refine our paper. We would like to address them below:

> What practical or theoretical advantages does SE-RO provide over an exact binomial test against the same support threshold combined with the SPA tier-construction procedure? …

With the exact test against σ in place of our admission rule, the ranking displays fewer supported comparisons. The test is nearly our admission rule, and we know of no ranking method that uses it. In Proposition 5, we prove that SE-RO needs the same number of supporting votes as the test, or one fewer. With Holm's adjustment, the test recalls 95.1% of supported comparisons in Table 12, against 98.4% for SE-RO.

> Can this comparison be extended beyond the synthetic Two Groups setting?

Yes. Proposition 5 holds on any data, so on every dataset the exact test at level 1 − α admits only comparisons that SE-RO admits. We've added the comparison on our four real datasets as Table 13. The test against σ never displays more than SE-RO, and on EnviroSound it displays nothing. A majority test orders 43% of the contested pairs on Sushi, where SE-RO orders none. One such pair is sea urchin over egg, which 58.8% of participants prefer and SE-RO withholds in every sample.

> It remains unclear whether the restriction to tiered rankings is necessary … compared with retaining a more general partial order.

Admitted pairs are not always transitive. A partial order would then have to drop one, and the data cannot say which. In Theorem 2, we prove that SE-RO returns the finest valid tiered ranking. We've added a paragraph on this to the Algorithm section.

In settings where coverage is critical or ratings are unevenly sampled, practitioners are free to drop the tier step. Those who expect transitive preferences can also display the comparisons that admitted pairs imply. These choices can remove much of the withholding from tiering. We note that transitivity does not hold in general, as Tversky (1969) shows.

> Potentially substantial information loss from tier construction. … On Jester, the method displays no comparisons despite admitting some directions.

We've added two columns to Table 9 that split the pairs SE-RO does not display by reason. The split at σ = 0.60 and α = 0.90 is below, as a percent of all pairs.

DisplayedHidden by the tier stepContestedToo few votes
EnviroSound34.337.00.028.8
Sushi37.821.035.06.1
Jester0.016.471.112.5
ERS Set 7717.236.430.515.9

Jester users disagree on most pairs of jokes. At σ = 0.60, 71.1% of Jester pairs are contested, and SE-RO keeps them unordered. A practitioner who wants a ranking of jokes may lower σ and α. At σ = 0.51 and α = 0.5⁺, SE-RO displays 78.0% of Jester pairs, as we show in Table 1.

> …rather than a generative model using preference optimization.

SE-RO changes the labels that preference optimization methods (e.g. DPO, PPO, GRPO) learn from, and our demo is meant to show that a model can learn from them when to withhold. SE-RO withholds the comparisons that people do not support, and this necessarily changes what these methods optimize.

> …their coverage differs substantially (42.3% versus 100%). A coverage-matched selective learned baseline … would strengthen the alignment-related contribution.

SE-RO is about when and why to withhold. Its coverage follows from the support a comparison needs, σ, and the evidence required, α. Matching that coverage would be an arbitrary choice, because the number of pairs withheld has no meaning apart from SE-RO. A matched baseline withholds the same number of pairs without requiring any level of support, so its labels can overrule a minority of any size.

> The Community Alignment top-choice observation mechanism does not fully satisfy this assumption … real-world bias bounds remain unknown.

We cannot guarantee that real-world data follows our assumptions, and this is why we include Γ. In Theorem 12, we prove that votes alone cannot identify population support when recording bias is unrestricted. Γ therefore reports the largest bias under which every displayed comparison still meets σ. At σ = 0.60 and α = 0.90, the median Γ lets supporters be 11% more likely to be recorded than nonsupporters on EnviroSound, 17% on ERS Set 77 and 20% on Sushi.

Thank you again for the careful review. Please let us know if this resolves your concerns. If it does, we would be grateful if you would consider raising your score.

Thank you for the detailed rebuttal and additional analyses.

I appreciate the new comparison with the exact binomial test on real-world datasets, as well as the clarification regarding the theoretical role of tier construction. These additions help strengthen the paper.

However, some of my concerns remain. In particular, the additional breakdown of hidden comparisons highlights the substantial information loss introduced by tier construction, while its necessity for downstream preference-learning applications remains unclear.

I also believe that a coverage-matched selective baseline would still be informative. Such a comparison would help distinguish improvements due to the quality of the withholding decisions from improvements arising simply from evaluating a smaller, potentially easier subset of comparisons.

Finally, the learned withholding experiment remains a limited demonstration of downstream language-model alignment, rather than a direct evaluation of generative preference optimization.

Overall, the rebuttal addresses several points constructively, but the remaining limitations prevent me from increasing my rating. I will maintain my original score of 3 (Weak Acceptance).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Thank you all for your time and feedback! We would like to address the shared concerns below:

1. What is novel over SPA and over an exact binomial test.

  • Over SPA. SPA ranks a finite panel and has limited to no practical ML application. SE-RO ranks the population the votes come from, and it works when each person rates only a few pairs, as in RLHF preference labels. In Corollary 3 and Theorem 7, we bound how often it displays a comparison the population does not support.

  • Over an exact test. A test of the majority direction orders every 55/45 pair once it has enough votes. A test against σ keeps the floor, and in Proposition 5 we prove that it is our admission rule in frequentist form. We know of no ranking method that uses it.

2. SE-RO's withholding has meaning. SE-RO's admission rule withholds a comparison for one of two reasons. SE-RO's coverage follows from σ, α and the data. Either the votes are too few to confirm support of at least σ, or support is below σ. A baseline can match that coverage only by using SE-RO as the reference, and its withholding then has no meaning of its own.

Summary of changes. We've updated the writing for clarity. In particular, we've updated Problem Statement to define the target ranking in plainer words, and Algorithm to better explain how the tiers form. We've updated figures and captions, and simplified Assumption 1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.