- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Geometry-Guided Error-Aware Robustness (GEAR), a method for improving worst-group accuracy (WGA) under spurious correlations without requiring group annotations during training or validation. The central observation is that difficult examples within the same class may differ from easy examples because of either misleading contextual information or useful task-related information. Consequently, simply suppressing the associated feature directions can discard discriminative information, while relearning from difficult examples may preserve reliance on spurious features.
With a regular-trained model, GEAR further fits suppression-based classifiers that reduce sensitivity to selected directions and relearning-based classifiers that emphasize difficult regions through adaptive training weights.
Experiments on six image and text benchmarks are conducted, including Waterbirds, CelebA, SpuCoBirds, SpuCoDogs, CivilComments, and MultiNLI. The improvements over existing methods validate the effectiveness of GEAR.
Critical strengths
(1) The paper addresses a fundamental challenge in distributionally robust learning: improving minority-group performance without access to group annotations. This setting is practically relevant because group labels can be expensive, unavailable, or difficult to define.
(2) The paper correctly identifies that feature variations associated with difficult examples can encode both spurious context and useful task information. Rather than assuming that all such variations should be removed or amplified, GEAR explicitly considers both possibilities. This is a compelling motivation for the proposed framework.
(3) Broad evaluation across image and text benchmarks. The six benchmarks cover different modalities and sources of spurious correlations, including image backgrounds, demographic attributes, and lexical cues. This breadth provides stronger evidence of applicability than experiments on a single dataset.
Critical weaknesses
(1) Dependence on candidate selection. The proposed suppression and relearning strategies generate separate candidate models, while the final model is selected from the initial, suppression, and relearning models. This raises the question of whether the effectiveness of GEAR primarily comes from its feature-geometry-based interventions or from validation-based candidate selection. The authors should clarify why neither intervention can consistently improve upon the initial model and quantify the contribution of the selection procedure.
(2) Lack of a unified suppression-and-relearning strategy. Suppression modifies only the classifier head while preserving the feature extractor, whereas relearning allows the model to adapt its representations. These strategies appear complementary, yet the method treats them as separate alternatives. Could suppression be incorporated as a constraint or initialization for relearning to retain its benefits while enabling representation adaptation? A unified approach, or a clearer justification for why the two strategies should remain separate, would strengthen the method.
(3) Limited to classifier-level correction. GEAR operates on frozen representations and enforces to suppress selected feature directions at the linear classifier level. However, this intervention does not modify or disentangle the underlying representations, which may encode task-relevant and spurious information in entangled directions. Consequently, the method's gains may depend heavily on the quality of the pretrained representation and the existence of a suitable linear decision boundary. It remains unclear whether GEAR can address spurious correlations that are deeply embedded in the representation and cannot be resolved by adjusting the classifier alone. Evaluating representation-level robustness, or comparing against methods that explicitly regularize or relearn the encoder, would strengthen the paper's claims about its broader applicability.
(4) Scalability to complex data distributions. Since GEAR does not require group annotations, it should be evaluated on a large-scale benchmark such as ImageNet to verify its effectiveness and scalability under more complex data distributions.
Questions for authors
See weaknesses.
3 comments
Thank you for the detailed clarification. Your responses address some of my concerns, but the interaction between suppression and representation relearning remains unclear.
If suppression directions are recomputed from the relearned representation, the weaker performance of the combined approach cannot be explained solely by changes in the original feature directions. Instead, it raises the question of whether the suppression criterion reliably distinguishes spurious information from useful task information. If relearning removes spurious dependence, further suppression should be unnecessary; if harmful information remains and is correctly identified, suppression should help. Could you explain why applying suppression to the updated representation reduces performance, and whether this reflects a limitation of the proposed criterion for identifying suppression directions?
Thank you for this perceptive follow-up. We fully agree with your analytical framework, and we are glad to clarify that the results actually support the reliability of our suppression criterion. As you pointed out, if our suppression criterion works properly, applying suppression to the relearned features should give either neutral results (when the relearned features already discard the shortcut) or better results (when residual shortcut dependence remains), so it should not be worse than the relearned features without suppression. This is in fact what we observe. Our previous table omitted the intermediate baseline of relearned features without suppression, and here we include it as Row 2 below to make this direct comparison clearer.
| # | Method / Feature set | Waterbirds | CelebA | SpuCoBirds | SpuCoDogs |
|---|---|---|---|---|---|
| 1 | Source features + Suppression head | 91.8 ± 0.8 | 77.0 ± 0.6 | 85.9 ± 1.1 | 46.8 ± 1.0 |
| 2 | Relearned features, no suppression (new) | 62.7 ± 1.9 | 87.2 ± 1.9 | 95.0 ± 0.6 | 78.5 ± 3.1 |
| 3 | Relearned features + Suppression head (Combined) | 88.3 ± 1.3 | 85.7 ± 2.2 | 95.0 ± 0.6 | 78.5 ± 3.1 |
| 4 | Suppressed features + Relearned head (Combined) | 69.6 ± 9.4 | 78.3 ± 1.4 | 94.8 ± 0.4 | 80.4 ± 1.4 |
| 5 | GEAR (separate branches) | 91.9 ± 0.9 | 87.2 ± 1.9 | 95.0 ± 0.6 | 83.6 ± 1.2 |
Comparing Rows 2 and 3 shows that our suppression criterion can distinguish spurious directions from useful task information even on the updated representations. Both rows use the exact same relearned features, while Row 3 recomputes the suppression directions from them and selects the suppression strength via validation. On Waterbirds, where relearning leaves strong residual shortcuts, our criterion successfully identifies these harmful directions in the new feature space and boosts WGA from 62.7% to 88.3% (+25.6 points). On SpuCoBirds and SpuCoDogs, where relearning already removes the spurious dependence, validation sets the suppression strength to zero and the two rows are identical. On CelebA, the small difference is within one standard deviation and nearly neutral.
Why is Row 3 still below full GEAR (Row 5) on Waterbirds and SpuCoDogs? The gap does not come from suppression hurting the relearned features, but because forcing representation relearning as the first step can hurt performance on some datasets. On Waterbirds, this step is unnecessary and degrades the pretrained features (Row 2 drops to 62.7%). Our suppression criterion then recovers performance to 88.3% (Row 3), it remains slightly below applying suppression directly to the clean source features (Row 1 at 91.8%). Similarly, on SpuCoDogs, full-network relearning (78.5%) falls below GEAR's head-only relearning branch (83.6%) on the source features. Reversing the order by fitting a relearned head on suppressed features (Row 4) also underperforms the separate branches.
We also agree that one may form a unified method by making representation relearning optional via validation. This is equivalent to adding Row 3 as an extra candidate alongside GEAR's existing branches. However, in our observed benchmark datasets, this candidate does not improve over GEAR (Row 5) as it generally needs only one of the two steps. When representation relearning succeeds, validation usually sets the suppression strength to zero, so Row 3 reduces to Row 2. When representation relearning is less effective, Row 3 falls below Row 1. Row 3 would add value on a dataset where representation relearning improves over the source features while still leaving residual shortcuts for suppression to remove. However, we have not yet observed this case based on our current benchmarks.
To summarize, our suppression criterion are in fact reliable on updated representations, while the lower performance of the combined pipeline mainly stems from unconditionally applying representation relearning first. We thank you again for helping us clarify this important interaction.
Thank you for your time and effort in reviewing our paper! We appreciate your recognition of our work and provide point-by-point responses below.
W1. Thank you for this thoughtful comment. You are right that neither correction dominates everywhere, and in GEAR the geometric interventions and validation selection play distinct, complementary roles.
First, no single intervention works everywhere because shortcuts and task signals interact differently across datasets. When the easy-to-hard direction cleanly isolates the shortcut, suppression removes it. However, when this direction mixes the shortcut with task signal, suppression can also remove useful task information, making representation relearning under our geometric weights the better choice (Proposition 4.1).
Second, while the geometric interventions create the raw gains over f0, validation selection makes GEAR work across diverse datasets. Below, each intervention already matches full GEAR on its suited benchmark, so the large gains over f0 come from the interventions. However, since a new dataset's feature structure is unknown, committing to one branch can cause a sharp drop when it mismatches the data. Validation selection is thus crucial to identify the right branch, and its contribution is quantified by the gap between full GEAR and each single-branch variant below.
W2. Thank you for this insightful suggestion! We keep the two strategies separate because they rest on opposite premises. Suppression assumes the initial features are sufficient and only the classifier head relies on wrong directions, whereas relearning assumes the features themselves are entangled and must be updated. On datasets suited for suppression (e.g., Waterbirds), relearning unnecessarily alters good pretrained features; conversely, where relearning already fixes the representation, adding suppression can remove useful task signal.
Following your suggestion, we tested two combined approaches: our suppression head on the relearned features, and a relearned head on the suppressed features. As shown in rows 2-4 below, neither improves over keeping the branches separate in GEAR. We would like to explore other ways to unify them in future work.
W3. We appreciate this comment and have highlighted this more clearly in the revision. GEAR is not restricted to frozen representations, as our representation relearning branch fine-tunes the full encoder end-to-end.
Your intuition about entangled directions is spot on. When spurious and task signals are deeply entangled, adjusting only the head is insufficient, and GEAR switches to representation relearning to update the encoder. Comparing rows 1 and 2 in the table above under the same suppression head, updating the representation brings large gains on all three entangled datasets (+8.7 points on CelebA, +9.1 on SpuCoBirds, and +31.7 on SpuCoDogs). GEAR outperforms encoder-retraining baselines in both Table 1 (XRM, MaskTune, CCDB) and Table 7 (JTT) on most benchmarks.
W4. Thank you for this helpful suggestion! Scaling to larger and more complex distributions is a great direction. While GEAR needs no group labels for training or validation, evaluating test WGA still requires ground-truth group labels on the test set, which standard ImageNet does not provide. We follow the standard protocol in the literature. SpuCoBirds and SpuCoDogs are built from ImageNet images with annotated spurious groups. We tested six benchmarks which exceed the four to five typically used in prior work (e.g., JTT, DFR, AFR), span vision and language, and scale up to 269,038 training examples on CivilComments. We will explore larger group-annotated benchmarks (such as ImageNet-9) in future work.
We have revised our manuscript based on your comments and believe that these points address your concerns. We appreciate the opportunity to improve the paper and would be happy to provide any further clarifications.