Reimagining Meaningful Model Multiplicity

Predictive model multiplicity is when multiple models in a hypothesis class $H$ disagree on their predictions while having similar overall error to the risk minimizer $h\ast$. This has substantial implications for algorithmic fairness: if such multiplicity exists, then there may be a model competitive with the risk minimizer of the target hypothesis class which does substantially better in terms of target fairness metrics; some argue that searching for such a less discriminatory model is a legal duty. Standard approaches formalize this duty via search over a Rashomon set $R(H)$ of models, but this search is computationally expensive and does not easily provide generalization guarantees. We propose a reframing to “meaningful" model multiplicity that avoids both obstacle—rather than asking how to find the most fair model in $R(H),$ we ask whether any $h\in H$ can outperform $h\ast$ on a target group $g$. We provide efficient ensembling techniques over models which witness such multiplicity which simultaneously will outperform $h^{\ast }$ on all affected groups, and show that models which are suitably multiaccurate preclude meaningful multiplicity. Finally, for constraint-based fairness metrics for classification tasks, we show that building a multicalibrated predictor for the label probability $\mathbb{E}[y|x]$ and then postprocessing it to satisfy the fairness guarantees will Pareto dominate approaches based on searching the Rashomon set of classifiers.