- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a test-time adaptation (TTA) method that aim to learn good features by seeking an important subspace. Unlike existing TTA methods that seeks such subspace with maximum variance, the proposed method seeks more informative subspace by using optimal transport (OT) framework with 2-Wasserstein distance. To avoid computational complexity of discrete OT solver, it uses normalize flows (NFs) for approximating OT map. The proposed method can be applied to both classification and regression tasks. Experiments on classification and regression tasks show the effectiveness of the proposed method compared to existing TTA methods.
Critical strengths
- TTA is a practically important research topic, and the proposed approach, applicable to both classification and regression tasks, is useful.
- Experiments with two classification and two regression tasks show the effectiveness of the proposed method compared to various existing TTA methods.
Critical weaknesses
- There are several unclear and problematic points in the proposed method. First, Eq. (5) contains obvious errors: the matrix directly acts on the norm, thus, the objective function becomes a matrix (not a scalar), and the argmin operator is not the objective function. In addition, although the proposed method approximates the OT map with NF, it is unclear whether the important constraint that the OT map should satisfy (i.e., the density-preservation constraint described below Eq. 5) can be met by using NF alone. Furthermore, does not appear to be updated in Algorithm 1. However, it should be updated in Eq. 5. The paper also does not provide a concrete procedure for obtaining from the target sample with NF , and does not appear in Algorithm 1. More importantly, although learning the subspace is a key point in the proposed framework, how to learn it is not described in Algorithm 1. The only explanation is briefly described in Line 209: "We address this by explicitly optimizing on the Stiefel manifold with Riemannian updates." Because of these issues, the optimization procedure is difficult to understand and reproduce, raising concerns about the soundness of the proposed method.
- There are concerns about whether experimental settings are fair. As described in Lines 278-284, the learnable parameters of the proposed method are different for each dataset. It also appears that the values were selected based on the best test performance for each dataset. It is also unclear why different evaluation metrics are used for Camelyon17 and Birdcalls (i.e., the former uses accuracy, but the latter uses F1 score). I would like the authors to justify the validity of these settings.
- Unlike standard TTA methods, the proposed method requires additional training for the source NF, which may limit its practical applicability, as it does not directly apply to off-the-shelf classification and regression models.
Questions for authors
- (Clarification requests) Please see the comments in the Critical Weaknesses section.
- Although the proposed method requires covariate shift-like assumptions for distribution shift, can it handle other distribution shifts, such as concept shift?