acceptodds
Under review as a conference paper at ICLR 2027

Predictive Multiplicity Is Mostly Margin: Decomposing Robustness and Distribution Shift

Abstract

Predictive multiplicity, how much a prediction can change across a set of nearoptimal models, is often treated as a distinct, principled signal for group-label-free spurious-correlation robustness, on par with training loss (AFR) or early-training disagreement (SELF). We make this signal concrete for the last layer: freezing a pretrained encoder, characterizing the ϵ-Rashomon set of near-optimal linear heads over its features, and deriving a closed-form estimator of a per-sample multiplicity score m(x). Under the Laplace approximation used to obtain it, m(x) decomposes exactly into a margin term and a leverage term, and its ability to localize minority examples is carried overwhelmingly by margin; leverage, the part specific to multiplicity, is uninformative about mean group membership, though a small residual signal remains recoverable from its distribution (Sec. 4). This explains, and we confirm on three benchmarks, why multiplicity, loss, disagreement, and softmax confidence are interchangeable for last-layer group robustness. The decomposition is not purely negative: leverage is precisely what a covariateshift detector needs, since it is uninformative for the failure mode multiplicity was proposed for but informative for a different one. Isolating it gives a near-perfect detector on an idealized synthetic shift and replicates across two encoders on Waterbirds, but not on a second dataset (CelebA), where combining leverage with margin wins instead, tracking whether margin itself carries real shift signal there. Isolating leverage is therefore dataset-dependent, not a universal prescription, and neither it nor any fixed combination we tried clearly beats standard Mahalanobis or k-NN distance baselines. The decomposition’s value is mainly explanatory: it clarifies why a classifier’s own leverage term carries shift information, rather than offering a new state-of-the-art detector. In short, multiplicity is mostly margin for robustness, and the leverage it discards for that purpose is informative about a different failure mode, even if not uniquely so.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
predictive multiplicity, distribution shift, robustness, machine learning, linear classifiers, margins, leverage, representation learning, model disagreement, fairness

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sKHWOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigated the Predictive Multiplicity, in classification setting. They mainly showed that Predictive Multiplicity relies on model's prediction confidence to identify minority groups. and more broadly, what makes it useful and mostly reflect how confident a model is in its own predictions.

Critical strengths

1- Interpretable decomposition of predictive multiplicity 2- Valuable negative findings and transparent evaluation

Critical weaknesses

1- I would say one critical weakness is that the paper is hard to follow and read. I mean despite having interesting results and experiments, the presentation is not good. This might be my personal opinion, so I tried not to include it as a negative factor in my scoring, but the main goal of having a good presentation is that people could easily understand the paper. For instances:
1.1- The Abstract should answer three main questions, WHY, WHAT, HOW. WHY the things you do matter, WHAT are the things that other people couldn't or haven't done in this filed, or WHAT distinguished your work form other, and HOW you have done it. I couldn't find the answers.
1.2- In Abstract, Why CelebA is in parentheses, and why Waterbird is not? AFR is a known technique but why suddenly mentioned in the first of Abstract?
1.3 Introduction is a place to make wide ranger of reader familiar with the problem, setting, ... . not allocating half of it to your contributions.
1.4 if you assume that a reader should be super onboard with the setting, why spending half of it to Worst-group robustness?

Due to not having a good presentation, I found it really hard to understand your paper, so weaknesses below, might be my wrong understanding. I would be happy to read authors responses regarding the notes bellow:
2- As far as I understand your proof, the proof seems to ignore the interactions between features. This comes from Appendix a, lemma 4 that claims the leverage term has equal group means. but in it, the argument ignores cross-feature terms in the "every quadratic form".
3- Due to limited practical contribution, I would say your work is making it difficult to justify its additional complexity, even though that you have mentioned in your intro that this is an analysis paper. The paper should more clearly show that what new understanding or practical guidance your analysis is providing, especially beyond the existing approaches.

Questions for authors

See the Weaknesses, but I would appreciate if you can answer the three main questions I mentioned in the weaknesses, that might help me understand your work better. (WHY, WHAT, HOW)

5 comments

Thank you. We used your list as the checklist for the rewrite.

Your three questions, briefly. Why: robustness methods that work without group labels need a score that flags minority examples, and multiplicity looks like a more principled choice than loss or disagreement, but it costs more. What: for a last layer, multiplicity is the margin plus a leverage correction. The minority signal sits in the margin, reweighting by either one gives the same accuracy, and neither helps. How: a closed form from a quadratic approximation of the Rashomon set, theory in a Gaussian model, and experiments on five datasets.

On 1.1 to 1.4: the introduction now follows that why, what, how order, the long contribution list is gone, and CelebA is out of the parentheses in the abstract.

W2. You're right. The lemma is replaced by an exact identity that keeps the cross term. The leverage gap is about 8% of the total at ε = 0.05 rather than zero, and it comes out negative in all 64 runs of a new benchmark with learned features.

W3. The introduction now ends with what someone should actually do. Multiplicity gives you what margin gives you, at more than 1000 times the cost. Always run a uniform baseline and a random score baseline, because choosing a strength on validation WGA handed random scores up to 4.8pp. And in the one place reweighting helped, it was the error set that did it, not multiplicity.

Thank you for checking, and we're glad the why, what, how framing helped.

The revised version is uploaded. We also uploaded it again today to be safe, so the submission page should now show a modification date of 16 Nov. A quick way to confirm you have the right file is that the revised PDF has a new Section 6, "A controlled benchmark with an ERM-trained encoder", and the abstract mentions a Hessian cross term. The revision history may not be visible from reviewer accounts, which could be why it showed as empty. If the PDF you get still has no Section 6, please let us know and we will ask the chairs to look into it.

For the changes most relevant to your review, the corrected lemma is Lemma 3 in App. A and the practical takeaways are at the end of the introduction.

Thank you again for your earlier comment. We wanted to follow up now that the revised PDF is visible on the submission page (it includes the new Section 6, "A controlled benchmark with an ERM-trained encoder"). In it, we restructured the introduction around your why, what, how questions, replaced the flawed lemma with an exact result that keeps the cross term (App. A, Lemma 3), and added the practical takeaways you asked for at the end of the introduction. As the discussion period ends tomorrow, if the revision addressed your concerns, we would be grateful if you would consider updating your assessment. We're glad to clarify anything that is still unclear.

Reviewer_NKGcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies what predictive multiplicity captures in a setting with a frozen encoder and a linear classification head. Using a quadratic approximation to the ε-Rashomon set of near-optimal heads, the authors derive a closed-form score, m(x)=hw(x)−∣z(x)∣m(x)=h_w(x)-|z(x)|, that separates a leverage term from the absolute logit margin.

The authors argue that the usefulness of this score for robustness to spurious correlations is largely driven by the margin, while leverage provides information relevant to covariate-shift detection. They investigate these claims through Gaussian simulations, direct constrained optimization in low-dimensional settings, and frozen-feature experiments on Waterbirds, CelebA, and CivilComments. The experiments compare scoring methods for last-layer retraining and examine the two components under distribution shifts. The results also show that the benefit of using leverage alone depends on the dataset and that it does not consistently outperform standard distance-based detectors such as Mahalanobis distance or k-NN. The paper therefore positions its contribution primarily as an explanation of the score’s behavior rather than a new state-of-the-art method.

Critical strengths

1. The paper offers a concrete lens for analyzing the role of predictive multiplicity.

Equation (1) separates the range over which the logit can vary within the quadratically approximated Rashomon set from the absolute margin of the reference classifier. This makes it easy to see what the multiplicity score captures beyond a simple margin-based measure. The direct constrained optimization and independent Hessian verification in Appendix C also provide useful checks of these computations, albeit in low-dimensional settings.

2. The paper explicitly reports the limitations of the method and how results vary across datasets.

The authors show that the benefit of using leverage alone appears on Waterbirds but does not carry over in the same way to CelebA, and that it is not clearly better than Mahalanobis or k-NN. Such results are valuable for an analysis paper and help readers avoid mistaking observations on a particular dataset for a general prescription.

Critical weaknesses

1. The argument for Lemma 4, which states that mean leverage is identical across groups, does not appear to hold.

Appendix A concludes that the expectation of the quadratic form involving the inverse Hessian is equal across the two groups, based on the fact that the square of each feature follows the same distribution in both groups. However, the quadratic form also contains cross-terms between features. Under the generative model given in the paper,

\mathbb{E}[x_c x_s \mid \text{minority}] = -\mu_c\mu_s$$ so the conclusion does not follow without additional justification that $(H^{-1})_{cs}=0$. Moreover, even equal expectations of the quadratic form would not imply equal expectations of its square root, to which the leverage term is proportional. In fact, computing the population optimum and the full Hessian by numerical integration under the setting specified in the paper, namely $\mu_c=\mu_s=\sigma=1$, $\alpha=0.1$, nuisance dimension 8, and $\epsilon=0.05$, gives $\mathbb{E}[h_w \mid \text{majority}] \approx 2.1299$ and $\mathbb{E}[h_w \mid \text{minority}] \approx 2.0345$, which is inconsistent with the claim that $\Delta h_w = 0$. In this numerical check, the bias and nuisance weights were fixed at zero by symmetry, and the core and spurious weights were jointly optimized with respect to the unregularized population cross-entropy. The full population Hessian was then used, including the bias, nuisance dimensions, and core–spurious cross terms. Increasing the number of Gauss–Hermite quadrature nodes along each of the two feature axes from 60 to 120 left the values reported above unchanged to four decimal places. This does not refute the weaker empirical conclusion that the margin plays the dominant role. However, the central claim that the margin alone exactly accounts for the difference in group means seems to need revision. **2. The scope of the conclusion that the various scores are interchangeable is broader than the experimental evidence supports.** The direct reason the methods yield identical results on Waterbirds is that all of them select $\lambda=0$ and therefore perform the same uniform retraining. The authors also acknowledge this in Appendix C. These results can thus serve as evidence that reweighting did not help in this setting, but they are hard to view as evidence that the scores are interchangeable in situations where the scores are actually used. For CelebA and CivilComments, the selected values of $\lambda$ are not sufficiently reported, and no equivalence test is provided for the original Table 2. Under a separate bootstrap protocol, RLM performs worse than AFR and confidence. In addition, the worst group accuracy under uniform retraining on Waterbirds is 79.9% in Table 2 but 67.6% in the later experiments, and it is unclear what protocol difference accounts for this gap, which makes it difficult to tie the results together into a single coherent conclusion.
Questions for authors

1. Could you explain how the leverage in Lemma 4 and Table 1 was computed?

I would like to know how the cross terms between the core and spurious features in the full Hessian were handled. If a diagonal approximation or some other variant was used, please state this explicitly. It would also be helpful to explain which parts of Proposition 1 still hold when the full population Hessian is used.

2. Could you clarify the protocol differences between Table 2 and the later Waterbirds experiments?

In particular, I would like to understand why the worst group accuracy under uniform retraining differs between 79.9% and 67.6%, which values of λ\lambda were selected for each method and each seed, and how the data used for last-layer retraining was separated from the data used for hyperparameter selection. A per-seed comparison under the original protocol would also be important for judging the scope of the conclusions.

2 comments

Thank you for the counterexample. You were right about all of it.

W1 and Q1. We get your numbers exactly (2.1299 and 2.0345). The old proof left out the cross term 2 (H inverse)_cs x_c x_s. The new Lemma 3 keeps it, and we now compute Δhw directly by quadrature rather than reading it off E[q], since as you said the square root breaks that step. To answer your question about Table 1: it did use the full Hessian, but summed over examples while the loss was a mean, and that mismatch hid the gap. The margin result, Δ(−|z|) > 0, is now proved at the joint optimum and no longer needs the reduction to separate blocks. The sign of Δhw is numerical only. We checked 189 configurations and we say that in the paper.

W2. We agree. On Waterbirds every method picks λ = 0, so the honest reading is "reweighting did not help", and that's what the paper says now. For CelebA and CivilComments we never saved the selected λ, so we only claim that no row separates from the others. The new Sec. 6 was built for your point. Nonzero strengths get selected there (Table 15 has the values for each seed), multiplicity tracks margin, neither beats a random control, and the error set does far better.

Q2. The 79.9% is a uniform refit on the validation split, which on Waterbirds is balanced across backgrounds. The 67.6% is a uniform refit on the training split. Table 3 lays out the protocols. In the first one λ was chosen on the same split the head was fit on. That's a weakness and we say so. Sec. 6 uses three splits that don't overlap.

Thank you for taking the time to write such a detailed response. Your clarifications and additional results have addressed my concerns. I believe the paper makes a valuable contribution, and the revisions will strengthen it further.

I will maintain my current score.

Reviewer_KQTbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates predictive multiplicity, i.e., variation in predictions across near-optimal models, as a signal for robustness to spurious correlations. Focusing on linear classifiers with frozen pretrained representations, the authors derive a closed-form approximation of multiplicity and decompose it into a margin and leverage term. They find that its ability to identify minority-group examples is largely driven by margin, while leverage can be useful for distribution-shift detection. The authors validate their approximation and investigate these findings through experiments on synthetic and real-world datasets, including comparisons with existing reweighting and OOD detection methods.

Critical strengths
  1. Theoretical insights: The authors derive a closed-form approximation of predictive multiplicity that decomposes it into margin and leverage, providing an interpretable characterization of these components within the fixed-representation, linear-classifier setting.
  2. Acknowledgment of limitations and negative results: The authors openly acknowledge negative results and discuss the scope of their conclusions.
Critical weaknesses
  1. Dense presentation and assumed familiarity with prior work: The paper is difficult to follow, as it introduces several concepts and methods without sufficient explanation, making the overall discussion less accessible.
  2. Limited scope: The analysis is restricted to linear classifiers operating on fixed pretrained representations, limiting the applicability of the findings.
Questions for authors

At line 251, the authors state that AFR, SELF, and RLM select λ=0\lambda = 0 on Waterbirds, achieving 79.9% WGA in Table 2. However, at lines 259–260, when referring to the group-labeled selection used in Table 2, they report λ=16\lambda = 16 for AFR (65.4% WGA) and λ=1\lambda = 1 for SELF (67.3% WGA). Could the authors clarify this inconsistency?

3 comments

Thank you for catching the inconsistency.

On your question: the two statements came from two different protocols, and our text wrongly pinned both on Table 2. In the table, the head is refit on the validation split and every method ends up choosing λ = 0, which gives 79.9%. The λ = 16 and λ = 1 values are from a second experiment where the head is refit on the training split. Table 3 now says which protocol each result uses.

W1. We rewrote the introduction for someone outside this corner of the field. A new Sec. 2 explains worst group accuracy, retraining the last layer, each baseline, and the Rashomon set before any of them get used. Some secondary results moved to the appendix to make room.

W2. This is a real limit and we tried to push on it in two ways. Sec. 6 looks at the last layer of encoders trained by ERM on spuriously correlated data, which is the setting DFR, AFR and SELF were designed for. The decomposition holds in all 64 runs. App. H goes further and takes the Rashomon set over every parameter of a small MLP. There, margin stops explaining the variance of the score (R² is about 0.1), though leverage still carries no minority signal. So our claim is about the last layer, and Sec. 8 says plainly that deep networks are untested.

Thank you again for your review. As the discussion period ends tomorrow, we wanted to check whether our response resolved your question about the λ inconsistency (now explained in Table 3) and your concerns about presentation and scope. If it did, we would be grateful if you would consider updating your assessment. We're glad to clarify anything that is still unclear.

I thank the authors for the clarification on the λ protocols, which resolves my question, and for the substantial and transparent revision. I particularly appreciate that the authors disclosed the errors found during the rebuttal (the cross term in the former Lemma 4, the √n Hessian scaling, the μ_s mismatch, and the mislabeled RLM row) and narrowed their claims accordingly. The new controlled benchmark with an ERM-trained encoder and the random-score control is a clear improvement.

At the same time, the extent of the changes (corrected theory, re-computed tables, a new benchmark and a restructured manuscript uploaded at the end of the discussion) means that the current version would benefit from a full new round of review. On scope, App. H indicates that the margin no longer explains the score when the Rashomon set spans all parameters of a small MLP (R² ≈ 0.1), which confirms that the conclusions are limited to the last-layer setting. Regarding presentation, I share sKHW’s view that the core sections remain dense and hard to parse, although the introduction is much improved.

I believe this is a valuable and honest analysis, but in its current form I maintain my score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of the revision: corrected Lemma 4, two disclosed bugs, and a new benchmark

Thank you all for the reviews. We've uploaded a revised PDF and supplement. Here is what changed.

Lemma 4 was wrong, and two of you caught it. Our proof dropped the cross term between the core and spurious features. The corrected result (Prop. 1(iii), App. A) says the group difference in E[q] equals minus 4 μ_c μ_s times the (c,s) entry of the inverse Hessian, and that entry is not zero. So we take back "Δhw = 0" and "entirely margin". What we can say now is that the leverage gap is negative and about 8% of the multiplicity gap at ε = 0.05. It ranges from 3% to 17% as ε goes from 0.01 to 0.2. Margin still explains 94 to 97% of the variance of the score. Table 1 and Fig. 1 are redone.

While fixing this we found two bugs of our own, and the paper now says so. The original Table 1 used a Hessian summed over examples together with a mean loss, which made hw too small by a factor of √n. It also used μ_s = 2 when the text said 1. The same half width code produced the RLM row of the WGA table (now Table 4), so that row was really a margin row. Waterbirds isn't affected because λ = 0 gets selected there. The shift detection results came from separate code that was correct.

We added a new benchmark (Sec. 6, App. I): Colored MNIST and Colored Fashion MNIST with an encoder trained by ERM on the biased data, 64 runs, a separate split for choosing the reweighting strength, and a random score control. The leverage gap is negative in all 64 runs. Reweighting by multiplicity and reweighting by margin stay within 1pp of each other across the whole strength grid. Neither one beats the random control. A score based on the error set gains 8 to 39pp when the minority is small, so we stopped calling the scores interchangeable. Multiplicity is interchangeable with margin, and that's the claim we make now.

On presentation: the introduction is rewritten, there is a new background section, and Table 3 lists the protocols. We kept the abstract close to the original.

Replies to each reviewer are posted under the individual reviews.

Thank you to all three reviewers for your time and careful feedback during the discussion. Your comments led directly to the main improvements in the paper: the corrected analysis of the leverage term, the clearer protocols, the new controlled benchmark, and a substantially rewritten presentation. We have uploaded a final revision tonight that shortens the abstract and restructures Sections 3 to 8 so that each paragraph opens with a direct answer. We appreciate your engagement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.