Suppress or Relearn? Feature Geometry for Robustness to Spurious Correlations

Deep neural networks can achieve high average accuracy in classification tasks by learning spurious correlations between class labels and context. This leads to poor performance when these correlations break down. Reducing the model's reliance on contextual features can help, but may also suppress useful task information. To address this trade-off, we propose Geometry-Guided Error-Aware Robustness (GEAR), which uses validation errors to decide when to reduce the model's sensitivity along selected feature directions and when to relearn. GEAR identifies directions of feature change from easy to hard examples within each class, using them to guide suppression and define regions for relearning. Both fitting and selection require only task labels. In an additive Gaussian model, we show that neither suppression nor relearning uniformly dominates and give conditions under which GEAR asymptotically achieves optimal worst-group accuracy among linear heads on fixed features. GEAR improves mean worst-group accuracy over its initial classifiers on all six image and text benchmarks.