Predictive Multiplicity Is Mostly Margin: Decomposing Robustness and Distribution Shift

Predictive multiplicity, how much a prediction can change across a set of nearoptimal models, is often treated as a distinct, principled signal for group-label-free spurious-correlation robustness, on par with training loss (AFR) or early-training disagreement (SELF). We make this signal concrete for the last layer: freezing a pretrained encoder, characterizing the ϵ-Rashomon set of near-optimal linear heads over its features, and deriving a closed-form estimator of a per-sample multiplicity score m(x). Under the Laplace approximation used to obtain it, m(x) decomposes exactly into a margin term and a leverage term, and its ability to localize minority examples is carried overwhelmingly by margin; leverage, the part specific to multiplicity, is uninformative about mean group membership, though a small residual signal remains recoverable from its distribution (Sec. 4). This explains, and we confirm on three benchmarks, why multiplicity, loss, disagreement, and softmax confidence are interchangeable for last-layer group robustness. The decomposition is not purely negative: leverage is precisely what a covariateshift detector needs, since it is uninformative for the failure mode multiplicity was proposed for but informative for a different one. Isolating it gives a near-perfect detector on an idealized synthetic shift and replicates across two encoders on Waterbirds, but not on a second dataset (CelebA), where combining leverage with margin wins instead, tracking whether margin itself carries real shift signal there. Isolating leverage is therefore dataset-dependent, not a universal prescription, and neither it nor any fixed combination we tried clearly beats standard Mahalanobis or k-NN distance baselines. The decomposition’s value is mainly explanatory: it clarifies why a classifier’s own leverage term carries shift information, rather than offering a new state-of-the-art detector. In short, multiplicity is mostly margin for robustness, and the leverage it discards for that purpose is informative about a different failure mode, even if not uniquely so.