GRIT: Out-of-Distribution Robustness from Finite and Noisy Invariant Pairs

Invariant pairs provide information about variation that should not affect prediction, but a finite, noisy collection can leave relevant directions unseen or suppress predictive information. We study out-of-distribution (OOD) robustness on fixed pretrained features, assuming that the label distribution given a linear projection of these features is invariant across environments. We propose Geometric Robustness Invariant Training (GRIT), which estimates a linear subspace from invariant pair differences and projects it out before predictor fitting, allowing the same constraint to accompany ERM or compatible robustness objectives. Our theoretical analysis bounds target risk through source prediction risk, a term related to covariate shift, and residual sensitivity along the unstable subspace. For GRIT, the bound depends on pair count, noise relative to the weakest clean signal, and selected rank. A parallel guarantee uses the fitted optimum to characterize quadratic prediction consistency under the same noise and shift assumptions. On ColoredMNIST-INV and Waterbirds-INV, reported means improve over each of eight evaluated objectives and exceed prediction consistency in 15 of 16 settings. On UrbanCars, combining pairs with background or object changes improves robustness to two spurious features.