- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
-
The paper is clearly written. It is clear what problem is addressed, why the existing approaches fail, and how the proposed method works.
-
The proposed solution is straightforward. Training a classifier to distinguish the two datasets provides a simple way to estimate the importance weights, making it easy to understand and use in practice.
-
The experiments cover relevant applications and models. The comparisons with unweighted PPI and known weights are useful, as well as the ablations on the classifier choice and clipping. Even though the coverage of the method is still below desired in proteingym, it is much better than existing approaches.
Critical strengths
-
The paper is clearly written. It is clear what problem is addressed, why the existing approaches fail, and how the proposed method works.
-
The proposed solution is straightforward. Training a classifier to distinguish the two datasets provides a simple way to estimate the importance weights, making it easy to understand and use in practice.
-
The experiments cover relevant applications and models. The comparisons with unweighted PPI and known weights are useful, as well as the ablations on the classifier choice and clipping. Even though the coverage of the method is still below desired in proteingym, it is much better than existing approaches.
Critical weaknesses
-
The methodological novelty is somewhat limited. Estimating importance weights by classifying source and target examples is an established approach (e.g., Section 3 of Sugiyama et al. (2010)), as the paper itself acknowledges. The contribution is therefore combining this idea with PPI and showing when it works. I think this is a useful contribution, but the paper should make this distinction more consistently. In particular, the claim that prior methods only target training under covariate shift seems incorrect; these methods can also be used for evaluation.
-
The paper could do more to establish the relevance of the correction for real-world cases. In both datasets, the shift is introduced artificially. These are useful controlled experiments, but I would like to see an example where the labeled and target datasets differ because of a real data-collection process. This would also make it more convincing that the method is useful when the shift is not artificial by the experimenter. The title of Section 6.3, “Real-world shift,” is misleading given that the shift is simulated.
-
I find the consistency result not very informative. Theorem 1 assumes that the estimated weights converge uniformly to the true weights; given this and boundedness, convergence of the weights seems fairly straightforward (yet important to establish). The harder question is when the classifier provides the correct probability that a datapoint was drawn from one of the two distributions. Theorem 2 does provides some insight into this, but assumes a correctly specified logistic model, no Platt scaling, no regularization, and no clipping (different from the experiments). It would be useful to study, perhaps via simulations, what effect these choices have on the estimated probabilities.
-
I thought the experiment varying shift is extremely insightful and deserves more attention. At no shift, the reported coverage is 90.5% for ADAEVAL and 90.9% for classical estimation. However, Table 5 does not include unweighted PPI. Ideally, these three methods should be compared across (a) shift severity, and (b) number of labels, and (c) classifier choice combined (rather than separately as in Figure 3). This would make it even more convincing that using the weight estimator in PPI is a 'free lunch'.
-
A few (minor) parts of the writing could be improved:
- Section 3 defines but also uses . Both should be defined.
- Section 4.1 should briefly explain Platt scaling.
- The plots are small, and legends overlap the plotted results. I would encourage larger panels and legends outside the plotting area.
- Coverage estimates should include confidence intervals.
- The unresolved “Appendix ??” reference in Section 6.1 should be corrected.
Questions for authors
-
How important is Platt scaling for the results? Would the coverage and estimation error change substantially without it?
-
What is going on in the relationship between coverage and sample size?
- On ImageNet, ADAEVAL coverage falls from about 87% at to about 81% at . On ProteinGym, it falls from 42.6% at to 0.8% at . I appreciate that the paper reports this openly.
- My intuition would be that as sample size increases, coverage should improve? as then we can better estimate the weights.
- Why do you think the coverage in Proteingym is so much lower than the desired 90%? It would be good to include your best attempt at understanding this in the paper.