- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigated the Predictive Multiplicity, in classification setting. They mainly showed that Predictive Multiplicity relies on model's prediction confidence to identify minority groups. and more broadly, what makes it useful and mostly reflect how confident a model is in its own predictions.
Critical strengths
1- Interpretable decomposition of predictive multiplicity 2- Valuable negative findings and transparent evaluation
Critical weaknesses
1- I would say one critical weakness is that the paper is hard to follow and read. I mean despite having interesting results and experiments, the presentation is not good. This might be my personal opinion, so I tried not to include it as a negative factor in my scoring, but the main goal of having a good presentation is that people could easily understand the paper. For instances:
1.1- The Abstract should answer three main questions, WHY, WHAT, HOW. WHY the things you do matter, WHAT are the things that other people couldn't or haven't done in this filed, or WHAT distinguished your work form other, and HOW you have done it. I couldn't find the answers.
1.2- In Abstract, Why CelebA is in parentheses, and why Waterbird is not? AFR is a known technique but why suddenly mentioned in the first of Abstract?
1.3 Introduction is a place to make wide ranger of reader familiar with the problem, setting, ... . not allocating half of it to your contributions.
1.4 if you assume that a reader should be super onboard with the setting, why spending half of it to Worst-group robustness?
Due to not having a good presentation, I found it really hard to understand your paper, so weaknesses below, might be my wrong understanding. I would be happy to read authors responses regarding the notes bellow:
2- As far as I understand your proof, the proof seems to ignore the interactions between features. This comes from Appendix a, lemma 4 that claims the leverage term has equal group means. but in it, the argument ignores cross-feature terms in the "every quadratic form".
3- Due to limited practical contribution, I would say your work is making it difficult to justify its additional complexity, even though that you have mentioned in your intro that this is an analysis paper. The paper should more clearly show that what new understanding or practical guidance your analysis is providing, especially beyond the existing approaches.
Questions for authors
See the Weaknesses, but I would appreciate if you can answer the three main questions I mentioned in the weaknesses, that might help me understand your work better. (WHY, WHAT, HOW)
5 comments
Thank you for checking, and we're glad the why, what, how framing helped.
The revised version is uploaded. We also uploaded it again today to be safe, so the submission page should now show a modification date of 16 Nov. A quick way to confirm you have the right file is that the revised PDF has a new Section 6, "A controlled benchmark with an ERM-trained encoder", and the abstract mentions a Hessian cross term. The revision history may not be visible from reviewer accounts, which could be why it showed as empty. If the PDF you get still has no Section 6, please let us know and we will ask the chairs to look into it.
For the changes most relevant to your review, the corrected lemma is Lemma 3 in App. A and the practical takeaways are at the end of the introduction.
Thank you again for your earlier comment. We wanted to follow up now that the revised PDF is visible on the submission page (it includes the new Section 6, "A controlled benchmark with an ERM-trained encoder"). In it, we restructured the introduction around your why, what, how questions, replaced the flawed lemma with an exact result that keeps the cross term (App. A, Lemma 3), and added the practical takeaways you asked for at the end of the introduction. As the discussion period ends tomorrow, if the revision addressed your concerns, we would be grateful if you would consider updating your assessment. We're glad to clarify anything that is still unclear.
Thank you. We used your list as the checklist for the rewrite.
Your three questions, briefly. Why: robustness methods that work without group labels need a score that flags minority examples, and multiplicity looks like a more principled choice than loss or disagreement, but it costs more. What: for a last layer, multiplicity is the margin plus a leverage correction. The minority signal sits in the margin, reweighting by either one gives the same accuracy, and neither helps. How: a closed form from a quadratic approximation of the Rashomon set, theory in a Gaussian model, and experiments on five datasets.
On 1.1 to 1.4: the introduction now follows that why, what, how order, the long contribution list is gone, and CelebA is out of the parentheses in the abstract.
W2. You're right. The lemma is replaced by an exact identity that keeps the cross term. The leverage gap is about 8% of the total at ε = 0.05 rather than zero, and it comes out negative in all 64 runs of a new benchmark with learned features.
W3. The introduction now ends with what someone should actually do. Multiplicity gives you what margin gives you, at more than 1000 times the cost. Always run a uniform baseline and a random score baseline, because choosing a strength on validation WGA handed random scores up to 4.8pp. And in the one place reweighting helped, it was the error set that did it, not multiplicity.