When to Switch: From Adversarial Training to ERM for Out-of-Distribution Generalization

A key challenge in out-of-distribution (OOD) learning is that models trained with empirical risk minimization (ERM) may learn spurious features. It is well known that adversarial training (AT) can suppress such learning, but OOD generalization may deteriorate in later training stages. In the present paper, we propose an adaptive two-stage AT-to-ERM strategy which uses early AT to encourage invariant feature learning, followed by ERM for empirical fitting based on the learned representation. The key is to determine when to switch. We introduce class conditional cross domain variance (CDV) to quantify cross domain discrepancies in the within-class variances of normalized features. We switch from AT to ERM when CDV shifts from decreasing to increasing. In the invariant spurious feature model, we prove that CDV is proportional to the fourth power of the normalized spurious feature coefficient, and gradient flow analysis shows that, beyond a finite time, the decay rate of CDV is at most a prescribed threshold. Experiments on five benchmarks with convolutional and Transformer backbones show that our strategy improves mean OOD accuracy by $0.93-2.70$ percentage points over AT and by $1.17-1.60$ percentage points over ERM, while outperforming MAT, Fish, and IRM. The strategy is also competitive when integrated into the strong ERM++ baseline, and CDV guided switching yields better average OOD performance than CORAL or MMD guided switching.