- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how model multiplicity can be used to improve subgroup predictive performance without explicitly enumerating a Rashomon set. It defines meaningful multiplicity through subgroup loss improvements, develops decision-list and linear-ensemble constructions for combining improvement witnesses, and establishes sufficient conditions based on multiaccuracy and groupwise self-orthogonality that exclude such improvements. It further applies existing multicalibrated postprocessing results to fairness-constrained classification. The contribution is theoretical, supported by population-level guarantees and a finite-sample analysis of the decision-list procedure.
Critical strengths
-
An interesting connection between model multiplicity and exploitable subgroup prediction error. The paper focuses on alternative models that improve predictive performance on identifiable groups, giving a concrete interpretation of when multiplicity can support model improvement.
-
Constructive guarantees for combining improvements across overlapping groups. The proposed ensemble procedures address the substantive difficulty that improving performance on one group can affect other overlapping groups. The decision-list construction approximately matches all supplied witnesses on their respective groups. The sequential linear construction provides a complementary approach without explicitly enumerating every group intersection.
-
The analysis connects population guarantees with finite-sample learning and downstream fairness. This work provides a finite-sample generalization guarantee for the decision-list procedure, giving statistical support to the finite-witness setting.
Critical weaknesses
-
The technical contribution beyond existing multigroup learning frameworks is insufficiently distinguished. The subgroup improvement criterion in Definition 2.1 closely resembles the certificates of suboptimality and multigroup optimality targets already studied in the bias-bounty framework (An Algorithmic Framework for Bias Bounties, FAccT 2022), which also supplies the underlying decision-list update mechanism. Moreover, the main downstream classification guarantee in Theorem 6.8 is explicitly inherited from prior multicalibrated postprocessing work.
-
Lack of empirical validation of the proposed algorithms. The paper does not demonstrate whether the methods achieve substantial improvements for underperforming subgroups at a reasonable computational cost, or how their performance changes when subgroup risks must be estimated from limited data and many groups overlap. Comparisons with existing multigroup learning methods and Rashomon-set selection would help establish whether the proposed approach offers a meaningful practical advantage.
-
The claimed superiority over Rashomon-set search depends on changing the admissible model class. The ensemble constructions explicitly allow predictors outside the original hypothesis class, while the cited downstream postprocessing procedure can also produce classifiers outside the convex hull of the comparator class. This limits the replacement claim when the original class encodes substantive requirements such as interpretability, model complexity, or deployment restrictions.
Questions for authors
-
Which algorithmic or theoretical guarantees are new relative to the bias-bounty framework and prior multicalibrated postprocessing work? Please clarify which results require substantive new arguments beyond adapting these existing frameworks to model multiplicity.
-
Can the authors provide empirical comparisons with existing multigroup learning methods and Rashomon-set selection to demonstrate subgroup improvements at a reasonable computational cost? How do these benefits vary with group overlap and the amount of data available for estimating subgroup risks?
-
Does the claimed advantage over Rashomon-set search persist when both approaches must satisfy the same restrictions on the admissible model class? Please clarify how the replacement claim should be qualified when the original class encodes interpretability, complexity, or deployment requirements.
Thank you for your review and thoughtful response! To your questions:
More genrally, we think of this work as a translational one: in our minds the interesting thing right now for ML-theory adjacent work is how it can be leveraged to do something useful, especially given how human-written technical work is getting swiftly suplanted. I.e., the thing that is valuable to us in this moment is making connections between theory and other areas and asking the right questions. The model multiplicity literature is actively shaping government policy and legal conversations around AI, so it seems very worthwhile to us to point out that the idea of predictive diversity within the Rashomon set that underpins it is itself suspect.
Your point about empirical validation is well-taken; we have some simple simulated results we can include in an appendix. As mentioned in the response to reviewer MDo8, the statements hold, for instance, any time a minority class has a different relationship to the label than the majority. In terms of computational cost: the existing guarantees for Rashomon set search do not have sample complexity guarantees, because there can be an exponential number of models in the set, it is NP hard to search over, and the search cannot be reduced to a standard squared error oracle call as our approaches can be. In terms of comparison to other multigroup learning methods: when groups are disjoint they are generally equivalent to learning separate minimizers per group. When they are overlapping, however, we get 1/eps^2 complexity with low swap regret per group, while naive ensembling will not get you any form of swap regret. (ie you have to handle the intersections of groups, and naive approaches will lead to a sample complexity blow-up.)
You are correct that the results do require increased complexity in the model, though we emphasize that our algorithms are all on the simple side (i.e., adding in a decision list/tree over models)---as we also mention in the discussion to reviewer MDo8, in practice even the most complex multicalibration processes do not greatly increase model complexity [1] Fig. 5 (e.g., looking at the decision tree of depth 6 the boosting process has created after round 6 to round, and how much the MSE and MSCE drop).
We can add discussion on this to the appendix with a specific example demonstrating how explainability can be maintained to some degree---i.e. generating a concrete example demonstrating the effect seen in Fig 5 of the above.
As a side note: one of the reasons why in practice one might not want to use a more complex model is not just that it may not be explainable, but also that very simple models tend to in practice be more robust to distribution shift etc---anecdotally, this is the case in medical settings where a model may have to be deployed across very different hospitals and patient populations. A nice thing about the multicalibration-style approaches we use here is that they are also more robust to these kinds of effects---this is the observation underpinning the omniprediction literature, and is one reason why these approaches have been used in medical settings [2].
[1] Globus-Harris et al, Multicalibration as Boosting for Regression, ICML 2023. https://arxiv.org/pdf/2301.13767
[2] Barda et al, Performing risk stratification for Covid-19 when individual level data is not available - the experience of a large healthcare organization https://www.medrxiv.org/content/10.1101/2020.04.23.20076976v1.full.pdf