Selective Rankings from Heterogeneous Incomplete Preferences

Many model alignment and recommendation systems aggregate diverse human preferences into a single ranking, concealing substantial disagreement and favoring a narrow majority. When preferences are only partially observed, even the direction of the population majority may be uncertain. We introduce SE-RO (Selective Evidence-based Rank Ordering), a method for inferring selective rankings from incomplete pairwise preferences. SE-RO constructs tiered partial orders with separate requirements for how broadly a preference must be shared and how much posterior evidence is needed to display it. Under representative sampling, we bound the risk of displaying comparisons with insufficient population support and give conditions for recovering the population ranking. We also quantify how much recording bias a fixed ranking can tolerate. We use real-world preference tasks and synthetic populations to test common ranking failures and characterize the trade-off between population support, evidence, and coverage. Our results show how selective rankings can avoid overruling divided groups while retaining comparisons with broad population support.