Gradient Principal Component Analysis for Domain Generalization

Domain generalization aims to learn a robust model from multiple source domains that can perform well on target domain. Existing methods usually regularize model learning by reshaping forward feature distributions, thereby learning representations that are both domain-invariant and class-discriminative. These methods ultimately achieve the desired objective through regularization losses. However, during the optimization process, the parameter update directions are inevitably entangled with domain-specific directions. To address this issue, we propose a new framework that disentangles shared principal directions from domain-specific directions in the gradient space, and suppresses the latter to promote cross-domain generalization. Our method first estimates a cross-domain shared principal subspace from aggregated uncentered second-order feature statistics. Under shared-structure assumption, this subspace approximates principal task-relevant structures that recur across source domains. During backpropagation, raw gradients are projected along the input feature dimension onto this feature derived subspace. Comprehensive experiments across multiple benchmarks show that our method achieves superior generalization performance compared to state-of-the-art methods. Moreover, we provide theoretical analyses of the validity and effectiveness of principal-subspace-based gradient surgery, offering theoretical foundations for multiple domain generalization settings.