acceptodds
Under review as a conference paper at ICLR 2027

Identification Is Not Enough: Which Majority Examples Are Flagged Changes Last-Layer Correction

Abstract

Stratified last-layer correction fixes worst-group failures by upweighting examples a selector scores as likely minority, and selectors are compared by how well those flags recover the annotated minority. We test whether these summaries predict the outcome of the correction. Our matched control permutes, within each class, the scores of majority examples and leaves every minority example's score untouched. This provably fixes AUROC, precision, recall, the detected minority set and the expected weight each group receives, and changes only which majority examples carry the flagged weight. It needs group labels: a diagnostic, not a method. On CelebA, with two selectors (prediction confidence and a patch-blur score), random reassignment raises test worst-group accuracy on all five backbones under both balanced-sampling and expected-weight fitting. On the three original backbones the selectors' own assignments rank in the bottom 0–2.5% of 200 random reassignments on test, though higher (up to 44%) on two later backbones: the majority examples a selector flags, boundary cases frozen ERM often misclassifies, are worse to upweight than random ones. This is not only a threshold shift: after re-tuning each head's threshold on validation, reassignment still gains 2.7–3.2 worst-group accuracy points with no group losing more than 0.2, and a threshold-free worst-group-pair AUC agrees. Flagging the easiest majority examples helps about as much; fresh sampling seeds alone move accuracy under 1.4 points. The effect depends on the dataset: on Waterbirds and MetaShift, whose selectors flag few majority false positives, it is near zero under expected-weight fitting. At the default setting even a test-selected best reassignment trails frozen ERM; at validation-tuned settings random reassignment beats tuned class balancing in 7 of 10 backbone–selector pairs on test but only 5 of 10 on validation. Identification summaries and expected group weights are therefore not sufficient to predict the outcome of stratified last-layer correction.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
spurious correlations, group robustness, worst-group accuracy, last-layer retraining, pseudo-group inference, evaluation methodology

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_HEm6OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper shows that selectors with identical group-level statistics can yield different worst-group accuracy (WGA) due to differences in which samples are flagged. This sensitivity is pronounced on CelebA but limited on Waterbirds and MetaShift. However, the underlying mechanism and practical utility of the proposed diagnostic remain unclear.

Critical strengths

1.The idea of permuting majority scores within each class to investigate the sensitivity of WGA to sample-level flag assignments is interesting and potentially useful for diagnosing the limitations of debiasing selectors.

2.The authors conduct extensive experiments to investigate how WGA responds to changes in which majority-group samples are flagged, while keeping group-level statistics largely unchanged.

Critical weaknesses

1.The manuscript would benefit from careful language editing to improve clarity and readability. Some sentences are abrupt or awkward, such as “These are dataset labels, not descriptions of identity” and “all four groups enter WGA.” Other expressions are overly compressed or difficult to interpret, such as “holds 82–86% of the flags a CelebA reassignment changes” and “The control is weaker than equating the whole training problem.” These passages should be revised for clarity and precision.

2.The novelty and broader significance of the work appear limited. Overall, the study remains largely exploratory, as the findings vary substantially across datasets and experimental settings, without yielding a consistent explanation of the observed effects. Consequently, it provides limited actionable guidance for future debiasing research.

Questions for authors
  1. Title: The title is somewhat awkward and could be revised to improve clarity and readability.

2.Abstract: The abstract is overly dense, with excessive experimental details and numerical results. The authors should streamline it to emphasize the central motivation, key findings, and main implications.

3.Notation (Line 149): The variables introduced in line 149 are not clearly defined or explained, making the subsequent discussion difficult to follow. The authors should explicitly define these variables and clarify their roles.

4.Correction estimators: In the section “Two Correction Estimators,” equations are introduced before the relevant notation and concepts are adequately defined. The authors should introduce the necessary definitions before presenting the equations to improve readability.

5.Experimental presentation (Section 5): The experimental results are difficult to follow because the main findings, implementation details, exceptional cases, and statistical interpretations are interleaved. The authors should reorganize this section around the key research questions, present the main findings more clearly, distinguish empirical observations from interpretations, and move technical details and exceptional cases to the appendix where appropriate.

6.Interpretation of the H–D control experiments: The interpretation of the H–D control experiments is not sufficiently supported. Although the authors acknowledge that the H–D effects differ between selectors and that low ranks do not imply good absolute correction performance, the mechanism underlying these observations remains unclear. In particular, the reversal of confidence H–D contrasts on the two additional backbones weakens the claim that sample difficulty consistently explains the observed effects. Moreover, matching intervention dose does not fully isolate sample difficulty from differences in feature composition and optimization impact. The authors should clarify the scope of the conclusions supported by these controls and avoid overinterpreting the role of sample difficulty.

Reviewer_SMGdOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

From my limited understanding, this paper looks at last-layer re-training, and analyses which majority examples are up-weighted. They do experiments in 3 classic image datasets, CelebA, Waterbirds, and MetaShift. I have not been able to understand the objective of this.

Critical strengths

Code is shared. Figures are esthetically pleasing.

Critical weaknesses

This paper is extremely difficult to understand. It sounds (from my personal opinion) AI-generated from start to finish. It reads as a list of facts about a few experiments that were done but it does not at all explain why this is done, what the broader aim is, and what these experiments actually mean. A few examples of sections that are hard to understand:

  • TLDR: " A matched control that fixes precision, recall, AUROC and group weights shows that which majority examples a selector flags alone moves worst-group accuracy of last-layer correction: identification metrics are not enough to judge selectors.". What does this mean? The TLDR should be understandable to anyone in the ML field, but this is impossible to understand.
  • Abstract: reads as a summary of findings but there is no contextualisation of the work, the problem it's tackling, and its potential impact.
  • Contributions: these should be readable standalone as a summary of the paper's work. Currently they are very difficult to understand.
  • Related work: please explain how you've grouped these different methods / how they fit together instead of just starting directly with: "DFR establishes .... JTT upsamples ..."
  • Results section as a whole is very difficult to understand. As just one example, this does not make sense as a heading by itself "What counts as independence evidence."

Also, your appendix should be part of the main pdf.

Questions for authors

See weaknesses

Reviewer_Lw47OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper tries to do a case-study-like analysis on stratified last layer re-training methods and mostly analyses pseudo-group selectors and how the specific majority false positives they flag affect last-layer retraining. While the core experimental idea which is permuting majority scores within each class while holding minority scores fixed, is clean and well-motivated, the manuscript is written in a highly compressed, ledger-like style that requires substantial effort to parse.

Critical strengths

Using a proxy to select minority/majority examples within each class is a common technique in the last layer retraining and spurious correlation literature. Analyzing different aspects of these pipelines is a good idea to understand the underlying mechanism. I value the paper's effort to isolate some of these aspects and experiments on different methods.

Critical weaknesses

The presentation of the paper needs substantial improvement. The text in uploaded PDF is not selectable and even the references and links are not accessible. The Appendix is presented in the supplementary material as a separate PDF. There is no new novel method proposed as a result of the whole analysis to enhance the performance on minority groups. Table 1 presents notations in a very confusing manner and all of those single-letter notations should be fixed and properly explained in text. The flow of the paper is totally hard to follow.

Additionally,

(1) the deterministic effect (U−OU - O) only appears on CelebA and is near zero on Waterbirds and MetaShift (Table 3),

(2) at the default k=0.10,C=1k=0.10, C=1 setting on CelebA, expected-weight correction is ~20 points worse than frozen ERM and simple class balancing (Table 5), and when hyperparameters are tuned on validation, 4 of 6 RN50 runs ignore the selector's flags altogether (Section 7.1), and

(3) the proposed loss-decile mechanism (H−DH - D) reverses sign for the confidence selector on the two replication seeds (Section 6).

Questions for authors

See Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.