acceptodds
Under review as a conference paper at ICLR 2027

Gradient Principal Component Analysis for Domain Generalization

Abstract

Domain generalization aims to learn a robust model from multiple source domains that can perform well on target domain. Existing methods usually regularize model learning by reshaping forward feature distributions, thereby learning representations that are both domain-invariant and class-discriminative. These methods ultimately achieve the desired objective through regularization losses. However, during the optimization process, the parameter update directions are inevitably entangled with domain-specific directions. To address this issue, we propose a new framework that disentangles shared principal directions from domain-specific directions in the gradient space, and suppresses the latter to promote cross-domain generalization. Our method first estimates a cross-domain shared principal subspace from aggregated uncentered second-order feature statistics. Under shared-structure assumption, this subspace approximates principal task-relevant structures that recur across source domains. During backpropagation, raw gradients are projected along the input feature dimension onto this feature derived subspace. Comprehensive experiments across multiple benchmarks show that our method achieves superior generalization performance compared to state-of-the-art methods. Moreover, we provide theoretical analyses of the validity and effectiveness of principal-subspace-based gradient surgery, offering theoretical foundations for multiple domain generalization settings.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Domain Generalization, Principal Component Analysis, Gradient Projection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gszoOpenReview ↗
Rating
2
Confidence
3(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the problem of domain generalization. The authors argue that model updates can be decomposed into domain-invariant and domain-specific components, and that the model should be guided toward domain-invariant update directions. To achieve this, they first sum the feature matrices across different domains at each network layer. They then sort the eigenvalues of the resulting matrix in descending order and mask the gradients associated with tail eigenvalues. The experimental results demonstrate the effectiveness of the proposed method.

Critical strengths
  1. The introduction and the proposed method are easy to read and understand.
  2. The paper provides extensive mathematical derivations.
  3. The experimental results demonstrate the effectiveness of the proposed method.
Critical weaknesses
  1. The literature review is insufficient. Addressing domain generalization from an optimization perspective should not be presented as a revisit, given the substantial body of existing related work. The most relevant studies appear to be those based on gradient and masking, including, but not limited to, the following:

[1]Learning to Generalize: Meta-Learning for Domain Generalization.

[2] Invariant Gradient Variances for Out-of-Distribution Generalization.

[3] Gradient Matching for Domain Generalization.

[4] Arithmetic Meta-Learning for Domain Generalization.

[5] Understanding Hessian Alignment for Domain Generalization.

[6] Learning Explanations that Are Hard to Vary.

[7] Sand-Mask: An Enhanced Gradient Masking Strategy for the Discovery of Invariances in Domain Generalization.

[8] Self-Challenging Improves Cross-Domain Generalization.

The authors should clarify how their method differs from these and other related approaches, and explain its advantages over existing methods.

  1. I have listed my other concerns in the Questions section below. I will make my final assessment based on the authors' responses.
Questions for authors
  1. The paper states that "our method does not simply force gradients from different domains to become similar." This statement brings to mind meta-learning-based approaches, including [1]–[4], as well as some SAM-based methods. These approaches do not directly modify the gradients; instead, they use regularization techniques to guide the model toward a parameter space that satisfies their desired properties. In comparison, explicitly masking gradients, as proposed in this paper, seems to impose a stronger constraint and arguably involves more of a forcing mechanism. How would they differentiate their approach from these existing methods in this regard?

  2. The relationship between the motivation, the proposed method, and the mathematical derivations is unclear. The motivation is that model updates can be decomposed into domain-invariant and domain-specific components, and that the model should be guided toward domain-invariant update directions. The proposed method sorts the eigenvalues in descending order and masks the gradients associated with tail eigenvalues. However, I could not find evidence to support the assumption that eigenvectors associated with larger eigenvalues are more likely to capture domain-invariant information, whereas those associated with smaller eigenvalues are more heavily influenced by domain-specific information. Could the authors provide evidence to support this assumption?

  3. Have the authors conducted experiments on the DomainNet dataset?

Reviewer_yoNdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Gradient Principal Component Analysis (GPCA), aiming to improve domain generalization from the perspective of the optimization process rather than traditional feature distribution alignment. GPCA mainly consists of three steps. The first is Online Covariance Statistics: extracting intermediate features from different source domains, computing uncentered second-order matrices, and maintaining feature statistics using Exponential Moving Average (EMA). The second is Principal Subspace Construction: aggregating the second-order matrices from different source domains and selecting the principal feature subspace whose cumulative eigenvalue energy reaches a given threshold. The third is Gradient Surgery: projecting the gradients of selected network layers onto the constructed principal feature subspace to suppress domain-specific gradient components. The authors present the idea and explain the method in a relatively clear and intuitive way. The proposed method is evaluated on four DomainBed datasets, including PACS, VLCS, OfficeHome, and TerraIncognita, and domain-label-free experiments are also reported on PACS. The results show certain competitiveness.

Critical strengths

The authors present the idea and explain the method in a relatively clear and intuitive way. The proposed method is evaluated on four DomainBed datasets, including PACS, VLCS, OfficeHome, and TerraIncognita, and domain-label-free experiments are also reported on PACS. The results show certain competitiveness.

Critical weaknesses
  1. The authors assume that the principal directions extracted from aggregated source-domain feature statistics mainly capture shared task-relevant information. However, high-energy principal directions do not necessarily correspond to domain-invariant information, as they may also contain domain-specific features. Although the authors' theoretical analysis is based on this assumption and the proposed method does achieve better generalization performance, these results do not directly demonstrate that the suppressed gradient components are indeed domain-specific information. From the existing content of the paper, it is difficult to confirm whether the improvements actually come from the PCA principal directions, how much independent improvement is brought by gradient projection, and whether high-energy directions are indeed more beneficial for DG than low-energy directions.
  2. In addition to the main experiments, the authors conduct Parameter Sensitivity Analysis experiments, and claim that "our method performs consistently under different warm-up steps, showing that it does not rely on a carefully tuned warm-up stage." However, warmup steps show the largest performance variation among the parameters in Table 3 (87.84–88.91). Moreover, the best performance of GPCA on the PACS dataset is 88.9, which is only 0.2 higher than the best baseline compared by the authors. Therefore, I think there are some somewhat exaggerated descriptions in the paper. This includes the authors'claim in the abstract that they achieved"superior generalization performance over state-of-the-art", but most of the methods compared in Table 1 were published between 2016 and 2022, with only two methods from 2025. There are still many relevant studies from the past three years that are worth considering, such as PGrad (ICLR 2023), which also proposes using PCA principal directions to improve gradient updates, and ERM++ (WACV 2025), an improved ERM baseline. In addition, related studies in this field generally provide standard deviations in their experimental results. The authors mention in the appendix that they evaluate the method using three random seeds. If the authors could also provide the standard deviation results and more comprehensive experimental evidence, the credibility of the reported performance could be improved.
Questions for authors
  1. Could the authors further explain why the principal directions extracted from aggregated source-domain feature statistics should mainly capture domain-invariant and task-relevant information? How can the authors distinguish the benefits of PCA principal directions from the general regularization effect of gradient projection?
  2. Why are some closely related or stronger recent methods, such as PGrad (ICLR 2023) and ERM++ (WACV 2025), not included in the comparisons? Could the authors provide more comprehensive comparisons to support the claimed state-of-the-art performance?
  3. The authors mention in the appendix that the experiments are conducted with three random seeds. Could the authors also provide the standard deviations or standard errors to better demonstrate the reliability of the reported improvements?
Reviewer_ELVtOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2(+1)
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a domain generalization method that projects gradients onto a shared parameter subspace across domains. Unlike previous methods that guide optimization indirectly through regularization terms in the loss function, the proposed method directly guides the optimization trajectory within this subspace using principal components of feature statistics. Empirical evaluations on various domain generalization benchmarks suggest the effectiveness of the proposed method in improving generalization performance.

Critical strengths

The method is clear and easy to follow. It can be potentially used in a plug-and-play manner.

Critical weaknesses

(1) Validation of the Shared Subspace

Using SVD to identify optimization subspaces is already established in domain generalization and optimization. The authors do not explain the similarities and differences between their method and existing SVD-based methods [1, 2]. The paper argues that adding regularization terms to the loss is insufficient to guide optimization toward a shared parameter subspace. However, no experiment demonstrates that this issue actually occurs in previous regularization-based methods.

(2) Justification of Assumption 1

If activations specific to one domain have high energy, the corresponding directions may be retained even if they are not shared across domains. Assumption 1 assumes that these domain-specific variations are small. However, the paper does not provide empirical analysis showing whether this assumption holds in realistic domain generalization scenarios. Furthermore, previous work [3] illustrates in Figure 1 that a small subset of parameters can dominate optimization and argues that this imbalance may reinforce spurious correlations. This raises concerns about whether Assumption 1 holds in practice, particularly when high energy feature directions reflect domain specific information.

Minor Issues

  • The hyperparameter sensitivity analysis is limited to PACS. I am concerned that the performance gains may partly come from hyperparameter tuning rather than the method itself.
  • Variability across random seeds is reported only for PACS, and detailed SAGM results are not reported.
  • In Figure 3, the improvement in intra-class compactness is not particularly clear.

Reference

[1] Wang, Zhe, Jake Grigsby, and Yanjun Qi. "PGrad: Learning Principal Gradients for Domain Generalization." International Conference on Learning Representations (ICLR), 2023. OpenReview, https://openreview.net/forum?id=CgCmwcfgEdH.

[2] Saha, Gobinda, Isha Garg, and Kaushik Roy. "Gradient Projection Memory for Continual Learning." International Conference on Learning Representations (ICLR), 2021. OpenReview, https://openreview.net.

[3] Cho, Sumin, Dongwon Kim, and Kwangsu Kim. "One-Step Generalization Ratio Guided Optimization for Domain Generalization." Proceedings of the 42nd International Conference on Machine Learning, PMLR, vol. 267, 2025.

Questions for authors

(1) Does the proposed method improve performance when combined with previous regularization-based methods? Following the paper’s argument that implicit regularization does not guarantee optimization within a shared subspace, combining the proposed method with these methods should improve performance.

(2) How does the proposed method differ from PGrad [1]? Is there any benefit to using principal components of activations rather than directly analyzing gradients or update trajectories? The performance comparison with PGrad is also needed.

(3) Please also address the weaknesses above.

Reference

[1] Wang, Zhe, Jake Grigsby, and Yanjun Qi. "PGrad: Learning Principal Gradients for Domain Generalization." International Conference on Learning Representations (ICLR), 2023. OpenReview, https://openreview.net/forum?id=CgCmwcfgEdH.

1 comment
Reviewer_ELVtOpenReview ↗

Follow-up on the Eigengap in Assumption 1

Thank you for the detailed responses and additional experiments. Having read all the reviews and the authors’ responses to each, I think most concerns have been addressed satisfactorily.

My remaining question concerns the eigengap in Assumption 1. Although aligning updates with principal directions has precedent, I find the Davis–Kahan interpretation in Theorem 1 interesting. It shows that the estimated subspace can remain close to the shared subspace even with domain-specific contributions, provided the shared eigengap is sufficiently large relative to the aggregate perturbation. Table 4 shows that projecting onto the principal subspace improves DG accuracy over ERM on PACS, raising the possibility that retaining some domain-specific information may help generalization.

To connect this observation with Theorem 1, could the authors give a simple example or theoretical intuition for why the shared eigengap might be sufficiently large at the rank selected by GPCA? If the authors commit to adding this explanation to the revised manuscript, I would be willing to raise my score from 2 to 3.

One small note on the response itself. The two parts were posted outside my review thread, and some equations did not render correctly on my end. I would appreciate a reply directly under this review so I can follow the discussion more easily.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.