- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
They propose an approach to detect the domain shift from the training domain, by using the discrepancy between the outputs of the standard-classifier and prototype based classifier. They measure the performance of the proposed approach by using the correlation between their proposed score and the domain-shift labels. Empirically, they show that the proposed approach outperforms baseline approaches in many cases, where they test their approach in the conventional domain adaptation benchmark such as domainnet, office, office-home.
Critical strengths
- The presentation is clear. Mostly, it is easy to follow their approach.
- The proposed approach is reasonable.
- The approach is working in domains that are unseen during training.
Critical weaknesses
- The novelty is limited because identifying the shift or the confidence of the prediction by using discrepancy of the classifiers is already proposed as indicated by the submission as in [1]. They also do not have the comparison with [1].
- Their proposed method is a bit heuristic, and there is no theoretical guarantee.
- They test their approach only on the conventional object-centric dataset. The results might lack generality. It is better to include datasets such as WilDs. Considering the goal of the task, they should include tasks such as weather-shift or noise-shift.
- The tested dataset looks a bit too easy. F-1 score in Table 1 is almost 100% in the baseline approach. That is also why they should test their approach in more diverse settings.
[1] Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shift, Baek et al., NeurIPS 2022
Questions for authors
All weaknesses above need to be addressed to increase my rating. If you have any clarification, please mention that in rebuttal.