- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper shows that selectors with identical group-level statistics can yield different worst-group accuracy (WGA) due to differences in which samples are flagged. This sensitivity is pronounced on CelebA but limited on Waterbirds and MetaShift. However, the underlying mechanism and practical utility of the proposed diagnostic remain unclear.
Critical strengths
1.The idea of permuting majority scores within each class to investigate the sensitivity of WGA to sample-level flag assignments is interesting and potentially useful for diagnosing the limitations of debiasing selectors.
2.The authors conduct extensive experiments to investigate how WGA responds to changes in which majority-group samples are flagged, while keeping group-level statistics largely unchanged.
Critical weaknesses
1.The manuscript would benefit from careful language editing to improve clarity and readability. Some sentences are abrupt or awkward, such as “These are dataset labels, not descriptions of identity” and “all four groups enter WGA.” Other expressions are overly compressed or difficult to interpret, such as “holds 82–86% of the flags a CelebA reassignment changes” and “The control is weaker than equating the whole training problem.” These passages should be revised for clarity and precision.
2.The novelty and broader significance of the work appear limited. Overall, the study remains largely exploratory, as the findings vary substantially across datasets and experimental settings, without yielding a consistent explanation of the observed effects. Consequently, it provides limited actionable guidance for future debiasing research.
Questions for authors
- Title: The title is somewhat awkward and could be revised to improve clarity and readability.
2.Abstract: The abstract is overly dense, with excessive experimental details and numerical results. The authors should streamline it to emphasize the central motivation, key findings, and main implications.
3.Notation (Line 149): The variables introduced in line 149 are not clearly defined or explained, making the subsequent discussion difficult to follow. The authors should explicitly define these variables and clarify their roles.
4.Correction estimators: In the section “Two Correction Estimators,” equations are introduced before the relevant notation and concepts are adequately defined. The authors should introduce the necessary definitions before presenting the equations to improve readability.
5.Experimental presentation (Section 5): The experimental results are difficult to follow because the main findings, implementation details, exceptional cases, and statistical interpretations are interleaved. The authors should reorganize this section around the key research questions, present the main findings more clearly, distinguish empirical observations from interpretations, and move technical details and exceptional cases to the appendix where appropriate.
6.Interpretation of the H–D control experiments: The interpretation of the H–D control experiments is not sufficiently supported. Although the authors acknowledge that the H–D effects differ between selectors and that low ranks do not imply good absolute correction performance, the mechanism underlying these observations remains unclear. In particular, the reversal of confidence H–D contrasts on the two additional backbones weakens the claim that sample difficulty consistently explains the observed effects. Moreover, matching intervention dose does not fully isolate sample difficulty from differences in feature composition and optimization impact. The authors should clarify the scope of the conclusions supported by these controls and avoid overinterpreting the role of sample difficulty.