- Rating
- 3(+2)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Due to the presence of spurious correlations in training data, domain generalization methods make use of multiple training distributions (environments) to steel models against learning predictive rules that will not generalize. Across different methods, the choice of environments and their weight in the training loss is often chosen heuristically, despite the fact that different groups of environments and weighings over those environments can significantly impact training. In this work, the authors present UAED, a method for adaptively reweighing a family of environments’ contribution to the training loss. The work presents a variety of theoretical results predicting the nature of the learned policy over environments, and applies this method to two popular domain generalization benchmarks.
Critical strengths
- The paper is well written and easy to follow. Figures and tables are clearly labelled. Theoretical results are well formatted and are followed by clear explanations of their significance.
- Extensive experiments across two popular datasets provide ample evidence for the empirical findings of the work.
Critical weaknesses
- [W1] The theoretical results, while intuitive, are also unsurprising. The results regarding the cooperative policy, namely that it will favour shortcuts if the KL penalty is not high enough and that it will defer to the reference distribution when paired with an invariant learner, are to be expected given the set up. Similarly with the robust policy providing larger weights to environments with higher scores.
- [W2] The empirical results seem to demonstrate that the methods presented in the paper are do not provide any gains with respect to simple baselines. Common methods like IRM match UAED’s performance. Furthermore, among different UAED methods, simply using a static environment policy produces the same performance as learning the policy. The authors state that this points to the importance of the family of environments the policy refers to. However, this result begs the question as to what readers are supposed to take away from the paper.
Questions for authors
- [Q1] Are there experiments on ColorMNIST with the robust policy?
- [Q2] Is there an intuition as to why the cooperative and robust policies tend to attain the same performance? Is it that they both provide some weighting over all environments in the family due to the KL term with the reference?
- [Q3] Is there a way that the findings of this work can be applied to other domain generalization methods and inform their construction?
Reminder to engage in reviewer-author discussion
Hi gGkV,
As there are only two days left in the author-reviewer discussion period, I would like to remind you to kindly respond to the authors' rebuttal, posted over a week ago. After November 18, reviewer responses are not be allowed and you will no longer be able to ask any follow-up questions. If you have no follow-ups, a short and polite response acknowledging you have read and understood the rebuttal and have no further questions is sufficient.
Thanks, AC