acceptodds
Under review as a conference paper at ICLR 2027

Suppress or Relearn? Feature Geometry for Robustness to Spurious Correlations

Abstract

Deep neural networks can achieve high average accuracy in classification tasks by learning spurious correlations between class labels and context. This leads to poor performance when these correlations break down. Reducing the model's reliance on contextual features can help, but may also suppress useful task information. To address this trade-off, we propose Geometry-Guided Error-Aware Robustness (GEAR), which uses validation errors to decide when to reduce the model's sensitivity along selected feature directions and when to relearn. GEAR identifies directions of feature change from easy to hard examples within each class, using them to guide suppression and define regions for relearning. Both fitting and selection require only task labels. In an additive Gaussian model, we show that neither suppression nor relearning uniformly dominates and give conditions under which GEAR asymptotically achieves optimal worst-group accuracy among linear heads on fixed features. GEAR improves mean worst-group accuracy over its initial classifiers on all six image and text benchmarks.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Deep Learning, Spurious Correlations, Robust

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_py9EOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Geometry-Guided Error-Aware Robustness (GEAR), a method for improving worst-group accuracy (WGA) under spurious correlations without requiring group annotations during training or validation. The central observation is that difficult examples within the same class may differ from easy examples because of either misleading contextual information or useful task-related information. Consequently, simply suppressing the associated feature directions can discard discriminative information, while relearning from difficult examples may preserve reliance on spurious features.

With a regular-trained model, GEAR further fits suppression-based classifiers that reduce sensitivity to selected directions and relearning-based classifiers that emphasize difficult regions through adaptive training weights.

Experiments on six image and text benchmarks are conducted, including Waterbirds, CelebA, SpuCoBirds, SpuCoDogs, CivilComments, and MultiNLI. The improvements over existing methods validate the effectiveness of GEAR.

Critical strengths

(1) The paper addresses a fundamental challenge in distributionally robust learning: improving minority-group performance without access to group annotations. This setting is practically relevant because group labels can be expensive, unavailable, or difficult to define.

(2) The paper correctly identifies that feature variations associated with difficult examples can encode both spurious context and useful task information. Rather than assuming that all such variations should be removed or amplified, GEAR explicitly considers both possibilities. This is a compelling motivation for the proposed framework.

(3) Broad evaluation across image and text benchmarks. The six benchmarks cover different modalities and sources of spurious correlations, including image backgrounds, demographic attributes, and lexical cues. This breadth provides stronger evidence of applicability than experiments on a single dataset.

Critical weaknesses

(1) Dependence on candidate selection. The proposed suppression and relearning strategies generate separate candidate models, while the final model is selected from the initial, suppression, and relearning models. This raises the question of whether the effectiveness of GEAR primarily comes from its feature-geometry-based interventions or from validation-based candidate selection. The authors should clarify why neither intervention can consistently improve upon the initial model and quantify the contribution of the selection procedure.

(2) Lack of a unified suppression-and-relearning strategy. Suppression modifies only the classifier head while preserving the feature extractor, whereas relearning allows the model to adapt its representations. These strategies appear complementary, yet the method treats them as separate alternatives. Could suppression be incorporated as a constraint or initialization for relearning to retain its benefits while enabling representation adaptation? A unified approach, or a clearer justification for why the two strategies should remain separate, would strengthen the method.

(3) Limited to classifier-level correction. GEAR operates on frozen representations and enforces WUr=0WU_r=0 to suppress selected feature directions at the linear classifier level. However, this intervention does not modify or disentangle the underlying representations, which may encode task-relevant and spurious information in entangled directions. Consequently, the method's gains may depend heavily on the quality of the pretrained representation and the existence of a suitable linear decision boundary. It remains unclear whether GEAR can address spurious correlations that are deeply embedded in the representation and cannot be resolved by adjusting the classifier alone. Evaluating representation-level robustness, or comparing against methods that explicitly regularize or relearn the encoder, would strengthen the paper's claims about its broader applicability.

(4) Scalability to complex data distributions. Since GEAR does not require group annotations, it should be evaluated on a large-scale benchmark such as ImageNet to verify its effectiveness and scalability under more complex data distributions.

Questions for authors

See weaknesses.

3 comments

Thank you for your time and effort in reviewing our paper! We appreciate your recognition of our work and provide point-by-point responses below.

W1. Thank you for this thoughtful comment. You are right that neither correction dominates everywhere, and in GEAR the geometric interventions and validation selection play distinct, complementary roles.

First, no single intervention works everywhere because shortcuts and task signals interact differently across datasets. When the easy-to-hard direction cleanly isolates the shortcut, suppression removes it. However, when this direction mixes the shortcut with task signal, suppression can also remove useful task information, making representation relearning under our geometric weights the better choice (Proposition 4.1).

Second, while the geometric interventions create the raw gains over f0f_0, validation selection makes GEAR work across diverse datasets. Below, each intervention already matches full GEAR on its suited benchmark, so the large gains over f0f_0 come from the interventions. However, since a new dataset's feature structure is unknown, committing to one branch can cause a sharp drop when it mismatches the data. Validation selection is thus crucial to identify the right branch, and its contribution is quantified by the gap between full GEAR and each single-branch variant below.

Candidate BranchWaterbirdsCelebASpuCoBirdsSpuCoDogs
Initial model f0f_074.2 ± 3.631.9 ± 1.781.5 ± 1.449.7 ± 2.3
Suppression only91.9 ± 0.976.0 ± 1.185.6 ± 0.846.8 ± 1.0
Head relearning only67.8 ± 7.378.0 ± 2.295.2 ± 0.283.6 ± 1.2
Repr. relearning only72.2 ± 6.387.2 ± 1.995.0 ± 0.678.5 ± 3.1
GEAR (full)91.9 ± 0.987.2 ± 1.995.0 ± 0.683.6 ± 1.2

W2. Thank you for this insightful suggestion! We keep the two strategies separate because they rest on opposite premises. Suppression assumes the initial features are sufficient and only the classifier head relies on wrong directions, whereas relearning assumes the features themselves are entangled and must be updated. On datasets suited for suppression (e.g., Waterbirds), relearning unnecessarily alters good pretrained features; conversely, where relearning already fixes the representation, adding suppression can remove useful task signal.

Following your suggestion, we tested two combined approaches: our suppression head on the relearned features, and a relearned head on the suppressed features. As shown in rows 2-4 below, neither improves over keeping the branches separate in GEAR. We would like to explore other ways to unify them in future work.

Method / Feature setWaterbirdsCelebASpuCoBirdsSpuCoDogs
Source features + Suppression head91.8 ± 0.877.0 ± 0.685.9 ± 1.146.8 ± 1.0
Relearned features + Suppression head (Combined)88.3 ± 1.385.7 ± 2.295.0 ± 0.678.5 ± 3.1
Suppressed features + Relearned head (Combined)69.6 ± 9.478.3 ± 1.494.8 ± 0.480.4 ± 1.4
GEAR (separate branches)91.9 ± 0.987.2 ± 1.995.0 ± 0.683.6 ± 1.2

W3. We appreciate this comment and have highlighted this more clearly in the revision. GEAR is not restricted to frozen representations, as our representation relearning branch fine-tunes the full encoder end-to-end.

Your intuition about entangled directions is spot on. When spurious and task signals are deeply entangled, adjusting only the head is insufficient, and GEAR switches to representation relearning to update the encoder. Comparing rows 1 and 2 in the table above under the same suppression head, updating the representation brings large gains on all three entangled datasets (+8.7 points on CelebA, +9.1 on SpuCoBirds, and +31.7 on SpuCoDogs). GEAR outperforms encoder-retraining baselines in both Table 1 (XRM, MaskTune, CCDB) and Table 7 (JTT) on most benchmarks.

W4. Thank you for this helpful suggestion! Scaling to larger and more complex distributions is a great direction. While GEAR needs no group labels for training or validation, evaluating test WGA still requires ground-truth group labels on the test set, which standard ImageNet does not provide. We follow the standard protocol in the literature. SpuCoBirds and SpuCoDogs are built from ImageNet images with annotated spurious groups. We tested six benchmarks which exceed the four to five typically used in prior work (e.g., JTT, DFR, AFR), span vision and language, and scale up to 269,038 training examples on CivilComments. We will explore larger group-annotated benchmarks (such as ImageNet-9) in future work.

We have revised our manuscript based on your comments and believe that these points address your concerns. We appreciate the opportunity to improve the paper and would be happy to provide any further clarifications.

Thank you for the detailed clarification. Your responses address some of my concerns, but the interaction between suppression and representation relearning remains unclear.

If suppression directions are recomputed from the relearned representation, the weaker performance of the combined approach cannot be explained solely by changes in the original feature directions. Instead, it raises the question of whether the suppression criterion reliably distinguishes spurious information from useful task information. If relearning removes spurious dependence, further suppression should be unnecessary; if harmful information remains and is correctly identified, suppression should help. Could you explain why applying suppression to the updated representation reduces performance, and whether this reflects a limitation of the proposed criterion for identifying suppression directions?

Thank you for this perceptive follow-up. We fully agree with your analytical framework, and we are glad to clarify that the results actually support the reliability of our suppression criterion. As you pointed out, if our suppression criterion works properly, applying suppression to the relearned features should give either neutral results (when the relearned features already discard the shortcut) or better results (when residual shortcut dependence remains), so it should not be worse than the relearned features without suppression. This is in fact what we observe. Our previous table omitted the intermediate baseline of relearned features without suppression, and here we include it as Row 2 below to make this direct comparison clearer.

#Method / Feature setWaterbirdsCelebASpuCoBirdsSpuCoDogs
1Source features + Suppression head91.8 ± 0.877.0 ± 0.685.9 ± 1.146.8 ± 1.0
2Relearned features, no suppression (new)62.7 ± 1.987.2 ± 1.995.0 ± 0.678.5 ± 3.1
3Relearned features + Suppression head (Combined)88.3 ± 1.385.7 ± 2.295.0 ± 0.678.5 ± 3.1
4Suppressed features + Relearned head (Combined)69.6 ± 9.478.3 ± 1.494.8 ± 0.480.4 ± 1.4
5GEAR (separate branches)91.9 ± 0.987.2 ± 1.995.0 ± 0.683.6 ± 1.2

Comparing Rows 2 and 3 shows that our suppression criterion can distinguish spurious directions from useful task information even on the updated representations. Both rows use the exact same relearned features, while Row 3 recomputes the suppression directions from them and selects the suppression strength via validation. On Waterbirds, where relearning leaves strong residual shortcuts, our criterion successfully identifies these harmful directions in the new feature space and boosts WGA from 62.7% to 88.3% (+25.6 points). On SpuCoBirds and SpuCoDogs, where relearning already removes the spurious dependence, validation sets the suppression strength to zero and the two rows are identical. On CelebA, the small difference is within one standard deviation and nearly neutral.

Why is Row 3 still below full GEAR (Row 5) on Waterbirds and SpuCoDogs? The gap does not come from suppression hurting the relearned features, but because forcing representation relearning as the first step can hurt performance on some datasets. On Waterbirds, this step is unnecessary and degrades the pretrained features (Row 2 drops to 62.7%). Our suppression criterion then recovers performance to 88.3% (Row 3), it remains slightly below applying suppression directly to the clean source features (Row 1 at 91.8%). Similarly, on SpuCoDogs, full-network relearning (78.5%) falls below GEAR's head-only relearning branch (83.6%) on the source features. Reversing the order by fitting a relearned head on suppressed features (Row 4) also underperforms the separate branches.

We also agree that one may form a unified method by making representation relearning optional via validation. This is equivalent to adding Row 3 as an extra candidate alongside GEAR's existing branches. However, in our observed benchmark datasets, this candidate does not improve over GEAR (Row 5) as it generally needs only one of the two steps. When representation relearning succeeds, validation usually sets the suppression strength to zero, so Row 3 reduces to Row 2. When representation relearning is less effective, Row 3 falls below Row 1. Row 3 would add value on a dataset where representation relearning improves over the source features while still leaving residual shortcuts for suppression to remove. However, we have not yet observed this case based on our current benchmarks.

To summarize, our suppression criterion are in fact reliable on updated representations, while the lower performance of the combined pipeline mainly stems from unconditionally applying representation relearning first. We thank you again for helping us clarify this important interaction.

Reviewer_PfUfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GEAR to improve robustness to spurious correlations without group annotations. GEAR estimates within-class feature directions from easy to difficult examples and constructs candidates through directional suppression and weighted relearning. It then selects a model using class-balanced and difficulty-weighted validation errors. Experiments on six image and text benchmarks show improved worst-group accuracy over the corresponding initial models. Under a fixed-feature Gaussian model, the paper also analyzes when each correction is preferable and provides conditional guarantees for model selection.

Critical strengths

1.GEAR requires no group annotations for training or selection. It improves worst-group accuracy on all six benchmarks and achieves the highest reported mean among the compared task-label-only methods on five. 2.The corrections are complementary: suppression performs better on Waterbirds, while representation relearning performs better on CelebA. Random-direction controls also support the value of the learned geometry.

Critical weaknesses
  1. The benefit of geometric partitioning over ordinary difficulty weighting is not fully isolated. Table 3 removes both regional and within-region weighting; a matched control retaining difficulty weighting is missing.
  2. The asymptotic optimality conditions require the reference to ignore useful task signal. The reliability of selection with practical references needs stronger empirical support.
Questions for authors

1.How does a variant without geometric partitioning perform when difficulty weighting, training budget, and model selection remain unchanged? This would help isolate the contribution of geometry. 2.What is the worst-group accuracy gap between the selected model and the best candidate on each dataset? How sensitive is this gap to the reference model’s training duration? 3.Please clarify the initialization and trainable parameters for relearning. Section 2 presents the initial model as the starting point, whereas Appendix A.4.3 describes starting image and CivilComments relearning from the pretrained backbone and freezing the MultiNLI encoder. How do these settings correspond to the main description?

3 comments

Thank you for dedicating your time and effort to review our paper. We sincerely appreciate your recognition of the merits of our work and we have carefully considered your comments and made revisions accordingly. We provide point-by-point clarifications to your questions below.

W1. Thank you for your careful reading and for this helpful suggestion. Table 3 originally did not include this matched control, so we ran it using the exact same training budget and validation selection. Specifically, we kept the difficulty weighting and removed only the geometric partition.

As shown in the table below, the geometric partition brings a clear benefit on its own. Removing the partition reduces WGA by 3.9 points on CelebA and 3.7 points on SpuCoDogs, while performance on SpuCoBirds remains similar. On Waterbirds, the results are identical because validation selects the suppression branch, which does not use the partition. We have added this control to the revision in Table 16 and referenced it in Table 3.

WaterbirdsCelebASpuCoBirdsSpuCoDogs
GEAR91.9 ± 0.987.2 ± 1.995.0 ± 0.683.6 ± 1.2
Without the partition91.9 ± 0.983.3 ± 1.996.0 ± 0.379.9 ± 0.9

W2. We appreciate you raising this point. The condition in our analysis serves only as a sufficient condition in a simplified model to make the asymptotic analysis tractable. In practice, GEAR requires much less from the reference model. The reference model only needs to make some errors within each class, which holds under standard training durations and can be checked directly. To provide the empirical support you asked for, we measured the selection loss under practical reference model (see first table in our response to Q2 below). Across these benchmarks, the selected candidate performs nearly as well as the best candidate in hindsight. This confirms that the practical reference model can reliably guide validation selection.

Q1. Please refer to our response to W1.

Q2. Thank you for this great question. As shown in the first table below, the WGA gap between the selected model and the best candidate in hindsight is very small across all four datasets.

mean test WGA (%) (Five candidates per run)WaterbirdsCelebASpuCoBirdsSpuCoDogs
Selected91.987.295.083.6
Best candidate in hindsight92.087.295.383.6

In the second table, we vary the reference model's training duration from 0.5 × to 4 × the default duration. In most cases, the selection gap remains minimal across durations. Extreme durations can still have an effect, which matches our expectation. If the reference model is trained too briefly (e.g., 0.5 × on SpuCoDogs), its difficulty scores are less informative and overall performance drops slightly, though selection itself remains accurate. Conversely, if it is trained too long (e.g., 2 × and 4 × on CelebA), it begins to memorize the hard examples. This leaves too few errors in the hard region for reliable validation selection, and in the extreme case where the hard region becomes empty, GEAR automatically detects this and stops.

Mean test WGA (%) by reference training duration (as a multiple of the default duration in our paper)½×1×2×4×
Waterbirds, selected92.491.991.591.2
Waterbirds, best candidate in hindsight92.592.091.691.8
CelebA, selected86.987.281.077.3
CelebA, best candidate in hindsight86.987.288.688.6
SpuCoBirds, selected93.795.095.195.3
SpuCoBirds, best candidate in hindsight93.895.095.295.4
SpuCoDogs, selected77.683.683.582.6
SpuCoDogs, best candidate in hindsight78.183.684.682.6

Q3. Thank you for your careful reading on our paper. Our wording of "starting point" in Section 2 could have been clearer. There, we meant that the initial model f0f_0 serves as the first step of the pipeline by providing the easy-to-hard directions, regions, and sample weights for all candidates, rather than serving as the weight initialization for every branch. Regarding parameter initialization and training, the two settings in Appendix A.4.3 correspond to our two branches. For head-level corrections (and on MultiNLI), we freeze the encoder of f0f_0 and refit only the linear head. For representation relearning on the image benchmarks and CivilComments, we initialize from the pretrained backbone and train the full network using the regions and weights derived from f0f_0. We have revised Section 2 to state this distinction clearly.

Thank you again for your feedback. We have incorporated your suggestions to strengthen our work. We appreciate the opportunity to improve the paper and look forward to your final evaluation and would be happy to provide any further clarifications if needed.

Thank you for the clarifications and additional experiments, which have addressed my questions. Based on my overall assessment of the paper, I will maintain my original score of 3.

We sincerely appreciate your feedback and support! Again, thank you for your dedication to review and your invaluable insights. We look forward to submitting the revised version of our work.

Reviewer_CrTxOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The deep neural networks can learn spurious correlations between class labels and context to achieve a high average accuracy for classification tasks. However, when these correlations break, it leads to poor performance. The paper proposes a Geometry-Guided Error-Aware Robustness (GEAR) method to address the problem that arises when reducing the reliance on context features may also suppress useful task information. GEAR uses validation errors to decide when to reduce sensitivity along selected feature directions and when to relearn. Further stating that none of them uniformly dominates either suppression or relearning to adopt, rather the choice of correction to adopt varies based on the datasets.

Critical strengths
  • ⁠The paper proposes a novel method called GEAR, which improves worst-group robustness without group annotations after identifying difficult examples by linking their errors to feature variation.
  • The paper shows theoretically that in a simplified model with fixed features, neither suppression nor relaring is always better. It also empirically shows that which correction works better is dependent on the dataset.
  • ⁠The paper highlights GEAR’s importance by demonstrating experiments across different modalities, including various image and text benchmarks.
Critical weaknesses
  • The predictions obtained from the reference model do not guarantee that hard samples obtained with lower confidence are mainly due to reliance on context information and limited reliability on target information. Thus, it does not indicate reliance on spurious correlation features. The low confidence can also be due to image variations in quality and lighting. Since GEAR uses this confidence score probability to determine the suppression directions, and difficulty-based weighting and relearning, the authors must evaluate the sensitivity of GEAR to these image variations.

  • ⁠The GEAR-based suppression fits a head insensitive to the selected directions, which points to the feature change from easier to hard examples. It can reduce reliance on misleading cues, but can also discard useful information. The background interventions (shown in Table 13) confirm that the learned directions respond to both the background and bird removal and also contain task related variations. However, the paper do not measure how much of the useful task information the suppression discards.

  • Results in Table 14 show that the proposed GEAR method performs worse than the source classifier, GEAR: 88.66% WGA and source classifier: 92.97% WGA, under the background removed intervention, where only the bird remains. This suggests that even the GEAR method discarded some of the task related information. The authors must provide a brief discussion and justifications on the results.

  • There is no related work in the paper. The paper should describe clearly how GEAR is different from existing methods. Or is there a claim that there is no literature review?

Questions for authors
  1. See above (critical weaknesses)
  2. There is a lot of per-dataset configuration chosen without group labels. I would have preferred one fixed configuration (or a few fix configurations) or at least detailed parameter ablations.
  3. How sensitive is GEAR to corruptions variation?
  4. From Table 15, representation relearning seems to work through the head, not the features. Is that understanding correct?
  5. Which of the three error terms in Theorem 4.2 binds in practice?

I am at the boundary of weak accept vs weak reject. I will finalize my ratings post rebuttal.

5 comments

Thank you for dedicating your time and effort to review our paper! We sincerely appreciate your positive feedback and have carefully considered your comments and revised the manuscript accordingly. We provide point-by-point clarifications below.

W1. Thank you for this insightful comment. We fully agree that low confidence from the reference model can arise from non-spurious variations or corruptions (e.g., image quality and lighting). Following your suggestion, we tested GEAR on all four image benchmarks by corrupting 10%, 20%, or 40% of training and validation images with one of five random medium-severity corruptions (Gaussian noise, Gaussian blur, brightness, contrast, or JPEG). As shown in the table below, GEAR remains consistently robust across five random seeds.

We believe this robustness stems from how group-free methods (e.g., JTT, AFR) work. Without group labels, reference confidence serves as a proxy for sample difficulty, and GEAR mitigates this difficulty. When the reference model is trained under spurious correlations, its systematic low-confidence predictions still naturally reflect shortcut reliance, which GEAR handles effectively.

WGA (%), mean ± stdWaterbirdsCelebASpuCoBirdsSpuCoDogs
Clean91.9 ± 0.987.2 ± 1.995.0 ± 0.683.6 ± 1.2
10% corrupted92.0 ± 0.885.7 ± 2.094.8 ± 0.582.1 ± 2.1
20% corrupted91.3 ± 1.686.7 ± 2.494.2 ± 0.384.0 ± 1.1
40% corrupted90.4 ± 1.183.3 ± 1.294.3 ± 0.481.0 ± 2.2

W2. We appreciate this thoughtful comment. As you point out, the learned directions can respond to both background and task-related changes, so suppression may remove some task-relevant information. However, GEAR selects the suppression branch only when validation error shows that the removed variation is redundant or recoverable from the remaining features. When this information is needed (as on CelebA and the two SpuCo datasets), validation selects relearning instead. Following your insightful suggestion to quantify information loss in suppression, we masked out the background on Waterbirds (leaving only the bird) and compared a suppressed head against a normal head. As shown in the table below, the impact of suppressing the learned directions is negligible.

WGA on bird-only images, mean ± std (%)Waterbirds
Normal feature vector92.87 ± 0.97
With the learned directions suppressed92.83 ± 1.11

W3. Thank you for this careful observation. While GEAR's WGA falls slightly below that of the source classifier under background removal in Table 14, this gap does not stem from a loss of task-relevant information, as masking affects the two classifiers differently. The source classifier relies on background shortcuts, so removing the background eliminates these shortcuts and improves its WGA. Since GEAR already avoids shortcuts, masking mainly shifts the feature distribution and can mismatch the default classification threshold. Specifically, GEAR predicts waterbird when its score exceeds zero, which no longer separates the classes well after the shift. Recalibrating only this threshold on validation (with all weights fixed) restores GEAR's WGA to 93.34% versus 92.98% for the source classifier (Appendix B.6.1). Since adjusting a scalar threshold adds no new information, it confirms that the task signal was intact in the features. We are happy to highlight this clarification in our revision.

W4. Thank you for pointing this out. Due to space constraints, we originally discussed related methods in the Introduction. We agree that a dedicated section makes our positioning clearer, and we have added a dedicated Related Work section (Section 5).

Methods for spurious correlations fall into two classes: group-annotated methods (e.g., GroupDRO, DFR), which use group labels to target worst-group loss directly, and group-free methods (e.g., JTT, AFR, XRM, MaskTune, CCDB), which use no group labels in training. GEAR belongs to the latter and differs in two ways:

  1. A geometric view of difficulty. Whereas most existing methods use sample difficulty to decide which examples to emphasize, GEAR treats it as a geometric signal to estimate the feature directions separating easy from hard samples within each class.

  2. Adaptive selection rather than a single fixed correction. Prior methods commit to one correction strategy across datasets, yet the right choice depends on how shortcut and task signals align in feature space. Since neither suppression nor relearning dominates across distributions (Proposition 4.1, Figure 3(a)), GEAR incorporates both and uses validation to select the better branch.

To our knowledge, both points are new among group-free methods, and we now state this comparison explicitly in Section 5.

Due to the input length limit, we continue our clarification in the next text box.

Q1. Please refer to our responses to W1-W4 above.

Q2. Fixed configuration and parameter ablations. Thank you for this constructive suggestion. For context, per-dataset selection is standard practice in this literature (e.g., GroupDRO, JTT, DFR, AFR) because the benchmarks differ substantially in modality, scale, distribution, and shortcut strength. Rather than relying on heavy manual tuning or group-labeled validation sets (as in JTT and AFR), GEAR determines most configurations automatically via standard validation error using only inputs and task labels. At the same time, we fully agree on the value of evaluating a unified setup alongside parameter sensitivity, and both confirm GEAR's stability. First, regarding parameter ablations, Section 4.3 (Figure 3(b)) reports sensitivity to the suppression strength and demonstrates stable performance across a wide range. We would be glad to extend this analysis to any specific parameters you have in mind. Second, following your suggestion for a single fixed configuration, we applied the setup already shared by the two SpuCo datasets unchanged across all four image benchmarks. As shown in the table below, performance remains very close to the per-dataset results.

GEARWaterbirdsCelebASpuCoBirdsSpuCoDogs
Per-dataset configuration91.9 ± 0.987.2 ± 1.995.0 ± 0.683.6 ± 1.2
Fixed configuration92.0 ± 0.986.2 ± 1.895.0 ± 0.683.6 ± 1.2

Q3. Please refer to our response to W1.

Q4. Role of features vs. classifier head in representation relearning (Table 15). Thank you for your careful reading and for bringing this up. You make a great point that the classifier head plays an essential role, and in fact representation relearning works through both the features and the head together. In Table 15, we originally evaluated the features alone using a standard generic linear head, without GEAR's head. The "Relearned" row only showed how the relearned features perform under a generic head, which cannot make full use of refined representation. We have renamed the rows to make this design clearer. Below, we extend this ablation to all four image datasets and also evaluate GEAR's suppression head on both the source and relearned features. With GEAR's head, the relearned features outperform every generic-head baseline across all four datasets and beat the source features under the same head on three of the four. This shows that the relearned features do bring a clear gain, and a matching head is needed to capture that structure.

Feature pipeline, headWaterbirdsCelebASpuCoBirdsSpuCoDogs
Source features, generic head69.4 ± 5.766.2 ± 1.489.0 ± 0.360.6 ± 0.3
Source features, random subspace removed, generic head69.3 ± 5.866.2 ± 1.389.0 ± 0.460.5 ± 0.3
Source features, learned subspace removed, generic head82.2 ± 4.770.7 ± 1.590.6 ± 0.458.8 ± 0.6
Relearned features, generic head65.1 ± 1.565.0 ± 2.891.8 ± 0.257.6 ± 1.2
Source features, GEAR suppression head91.8 ± 0.877.0 ± 0.685.9 ± 1.146.8 ± 1.0
Relearned features, GEAR suppression head88.3 ± 1.385.7 ± 2.295.0 ± 0.678.5 ± 3.1

Q5. Thank you for this thoughtful question. In practice, εfitε_{fit} can be the term that binds. Theorem 4.2 bounds the WGA gap between GEAR's selected candidate and the optimal linear head by three terms. The first term, εfitε_{fit} measures the gap from the best fitted candidate to the optimal linear head, which cannot be measured directly since the optimal linear head is unknown. The other two terms are the validation sampling noise εvalε_{val} (Eq. 6) and the proxy weighting bias εrefε_{ref} from using the reference model instead of true group labels. Because εvalε_{val} and εrefε_{ref} only affect candidate selection, their combined cost is simply the WGA gap between the selected candidate and the best fitted candidate in hindsight. We measured this gap across all four image datasets in the table below, and it is at most 0.3 percentage points. This shows that εvalε_{val} and εrefε_{ref} can be small in practice, leaving εfitε_{fit} as the main binding term.

Five candidates per run, mean test WGA (%):

CandidateWaterbirdsCelebASpuCoBirdsSpuCoDogs
Selected91.987.295.083.6
Best candidate in hindsight92.087.295.383.6

We believe the above points strengthen our work and effectively address your concerns. Thank you for your valuable feedback and suggestions again. We look forward to your final evaluation and would be happy to provide any further clarifications if needed.

Reviewer_CrTxOpenReview ↗

Additional Questions

Thanks for your response. I have a few questions based on the responses and revised paper.

  • Why do Table 1 and Tables 16 to 21 report different GEAR and initial-model numbers? Also, FG-CCDB at 1.90% on MultiNLI and BAM-CM at 82.20 ± 29.07 remain same in Table 1. Are these reproductions correct?
  • On CelebA, selection loses 7.6 and 11.3 points at 2× and 4× reference duration. How is the duration chosen without group labels?
  • Please report the source classifier with its threshold recalibrated in the same way as GEAR’s.
  • On SpuCo, the suppression head on relearned features has α = 0. Is there any common-head evidence that relearned features are better?
  • Please add a corruption test with a single corruption type correlated with class, plus the corrupted ERM baseline.
  • Does the fixed configuration same on CivilComments and MultiNLI?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.