- Rating
- 2
- Confidence
- 3(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the problem of domain generalization. The authors argue that model updates can be decomposed into domain-invariant and domain-specific components, and that the model should be guided toward domain-invariant update directions. To achieve this, they first sum the feature matrices across different domains at each network layer. They then sort the eigenvalues of the resulting matrix in descending order and mask the gradients associated with tail eigenvalues. The experimental results demonstrate the effectiveness of the proposed method.
Critical strengths
- The introduction and the proposed method are easy to read and understand.
- The paper provides extensive mathematical derivations.
- The experimental results demonstrate the effectiveness of the proposed method.
Critical weaknesses
- The literature review is insufficient. Addressing domain generalization from an optimization perspective should not be presented as a revisit, given the substantial body of existing related work. The most relevant studies appear to be those based on gradient and masking, including, but not limited to, the following:
[1]Learning to Generalize: Meta-Learning for Domain Generalization.
[2] Invariant Gradient Variances for Out-of-Distribution Generalization.
[3] Gradient Matching for Domain Generalization.
[4] Arithmetic Meta-Learning for Domain Generalization.
[5] Understanding Hessian Alignment for Domain Generalization.
[6] Learning Explanations that Are Hard to Vary.
[7] Sand-Mask: An Enhanced Gradient Masking Strategy for the Discovery of Invariances in Domain Generalization.
[8] Self-Challenging Improves Cross-Domain Generalization.
The authors should clarify how their method differs from these and other related approaches, and explain its advantages over existing methods.
- I have listed my other concerns in the Questions section below. I will make my final assessment based on the authors' responses.
Questions for authors
-
The paper states that "our method does not simply force gradients from different domains to become similar." This statement brings to mind meta-learning-based approaches, including [1]–[4], as well as some SAM-based methods. These approaches do not directly modify the gradients; instead, they use regularization techniques to guide the model toward a parameter space that satisfies their desired properties. In comparison, explicitly masking gradients, as proposed in this paper, seems to impose a stronger constraint and arguably involves more of a forcing mechanism. How would they differentiate their approach from these existing methods in this regard?
-
The relationship between the motivation, the proposed method, and the mathematical derivations is unclear. The motivation is that model updates can be decomposed into domain-invariant and domain-specific components, and that the model should be guided toward domain-invariant update directions. The proposed method sorts the eigenvalues in descending order and masks the gradients associated with tail eigenvalues. However, I could not find evidence to support the assumption that eigenvectors associated with larger eigenvalues are more likely to capture domain-invariant information, whereas those associated with smaller eigenvalues are more heavily influenced by domain-specific information. Could the authors provide evidence to support this assumption?
-
Have the authors conducted experiments on the DomainNet dataset?
Follow-up on the Eigengap in Assumption 1
Thank you for the detailed responses and additional experiments. Having read all the reviews and the authors’ responses to each, I think most concerns have been addressed satisfactorily.
My remaining question concerns the eigengap in Assumption 1. Although aligning updates with principal directions has precedent, I find the Davis–Kahan interpretation in Theorem 1 interesting. It shows that the estimated subspace can remain close to the shared subspace even with domain-specific contributions, provided the shared eigengap is sufficiently large relative to the aggregate perturbation. Table 4 shows that projecting onto the principal subspace improves DG accuracy over ERM on PACS, raising the possibility that retaining some domain-specific information may help generalization.
To connect this observation with Theorem 1, could the authors give a simple example or theoretical intuition for why the shared eigengap might be sufficiently large at the rank selected by GPCA? If the authors commit to adding this explanation to the revised manuscript, I would be willing to raise my score from 2 to 3.
One small note on the response itself. The two parts were posted outside my review thread, and some equations did not render correctly on my end. I would appreciate a reply directly under this review so I can follow the discussion more easily.