acceptodds
Under review as a conference paper at ICLR 2027

AdaEval: Adaptive Importance Weighting for Reliable Model Evaluation under Unknown Covariate Shift

Abstract

Reliable model evaluation in the wild often relies on the AUTOEVAL framework, which augments a small human-labeled dataset with massive synthetic labels to produce low-variance performance estimates. A critical limitation is that AUTOEVAL assumes either that the labeled and unlabeled data share the same distribution or that the true importance weights are known, both of which are unrealistic under real-world covariate shift. We propose ADAEVAL, a method that learns importance weights directly from unlabeled data by training a discriminative classifier to distinguish labeled from target inputs. The resulting weighted estimator re-targets the labeled sample to the shifted distribution without any prior knowledge of the shift. We prove consistency and asymptotic normality under mild conditions on the weight estimation error. On ImageNet with simulated covariate shift, standard AUTOEVAL collapses to a coverage of just with labeled points; ADAEVAL achieves – coverage against PPI++'s –, and reduces MSE by – relative to classical estimation. On ProteinGym, under a strong fitness-biased shift, ADAEVAL reduces absolute MSE bias by – while coverage decays to near zero as grows large, a residual gap we characterize explicitly. ADAEVAL reduces bias and MSE under unknown covariate shift, with a fully characterized residual coverage gap; it is not a general solution for statistically valid confidence intervals across all regimes.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Model evaluation, covariate shift, importance weighting, distribution shift, semi-supervised inference, prediction-powered inference, asymptotic normality, domain adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_H9EROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  1. The paper is clearly written. It is clear what problem is addressed, why the existing approaches fail, and how the proposed method works.

  2. The proposed solution is straightforward. Training a classifier to distinguish the two datasets provides a simple way to estimate the importance weights, making it easy to understand and use in practice.

  3. The experiments cover relevant applications and models. The comparisons with unweighted PPI and known weights are useful, as well as the ablations on the classifier choice and clipping. Even though the coverage of the method is still below desired in proteingym, it is much better than existing approaches.

Critical strengths
  1. The paper is clearly written. It is clear what problem is addressed, why the existing approaches fail, and how the proposed method works.

  2. The proposed solution is straightforward. Training a classifier to distinguish the two datasets provides a simple way to estimate the importance weights, making it easy to understand and use in practice.

  3. The experiments cover relevant applications and models. The comparisons with unweighted PPI and known weights are useful, as well as the ablations on the classifier choice and clipping. Even though the coverage of the method is still below desired in proteingym, it is much better than existing approaches.

Critical weaknesses
  1. The methodological novelty is somewhat limited. Estimating importance weights by classifying source and target examples is an established approach (e.g., Section 3 of Sugiyama et al. (2010)), as the paper itself acknowledges. The contribution is therefore combining this idea with PPI and showing when it works. I think this is a useful contribution, but the paper should make this distinction more consistently. In particular, the claim that prior methods only target training under covariate shift seems incorrect; these methods can also be used for evaluation.

  2. The paper could do more to establish the relevance of the correction for real-world cases. In both datasets, the shift is introduced artificially. These are useful controlled experiments, but I would like to see an example where the labeled and target datasets differ because of a real data-collection process. This would also make it more convincing that the method is useful when the shift is not artificial by the experimenter. The title of Section 6.3, “Real-world shift,” is misleading given that the shift is simulated.

  3. I find the consistency result not very informative. Theorem 1 assumes that the estimated weights converge uniformly to the true weights; given this and boundedness, convergence of the weights seems fairly straightforward (yet important to establish). The harder question is when the classifier provides the correct probability that a datapoint was drawn from one of the two distributions. Theorem 2 does provides some insight into this, but assumes a correctly specified logistic model, no Platt scaling, no regularization, and no clipping (different from the experiments). It would be useful to study, perhaps via simulations, what effect these choices have on the estimated probabilities.

  4. I thought the experiment varying shift is extremely insightful and deserves more attention. At no shift, the reported coverage is 90.5% for ADAEVAL and 90.9% for classical estimation. However, Table 5 does not include unweighted PPI. Ideally, these three methods should be compared across (a) shift severity, and (b) number of labels, and (c) classifier choice combined (rather than separately as in Figure 3). This would make it even more convincing that using the weight estimator in PPI is a 'free lunch'.

  5. A few (minor) parts of the writing could be improved:

    • Section 3 defines E^i\hat E_i but also uses E^ju\hat E_j^u. Both should be defined.
    • Section 4.1 should briefly explain Platt scaling.
    • The plots are small, and legends overlap the plotted results. I would encourage larger panels and legends outside the plotting area.
    • Coverage estimates should include confidence intervals.
    • The unresolved “Appendix ??” reference in Section 6.1 should be corrected.
Questions for authors
  1. How important is Platt scaling for the results? Would the coverage and estimation error change substantially without it?

  2. What is going on in the relationship between coverage and sample size?

  • On ImageNet, ADAEVAL coverage falls from about 87% at n=300n=300 to about 81% at n=500n=500. On ProteinGym, it falls from 42.6% at n=200n=200 to 0.8% at n=1500n=1500. I appreciate that the paper reports this openly.
  • My intuition would be that as sample size increases, coverage should improve? as then we can better estimate the weights.
  1. Why do you think the coverage in Proteingym is so much lower than the desired 90%? It would be good to include your best attempt at understanding this in the paper.
Reviewer_NKJVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper considers model evaluation under covariate shift with unlabeled label when the density ration is unknown. They propose to estimate the ratio and plug in the resulting weights into the PPI++ estimator. On various experiments, they show that this estimate achieves better coverage, although coverage appears to remain below nominal.

Critical strengths

I find the exposition of the problem formulation to be clean, i.e., it is very easy to comprehend that estimating the density ration is the goal of this paper.

Critical weaknesses
  1. Can you clarify when does Assumption A2 hold? I.e., uniform consistency is a very strong assumption and there does not appear to be any justification for the implemented estimator or discussion surrounding this.
  2. Can you provide more insight behind Assumption A5? I.e., it seems that this is a workaround around parametric restrictions of your classifier. I wonder if the authors can provide more insights as to why should I expect this to hold in any realistic settings? I.e., if it does not hold, then I believe the theory should provide some justification of how well it does under misspecification :-)
  3. None of the experiments seem to have any standard error bars, since the numbers are often very close to each other it is hard to tell whether the proposed method is in fact significant.
Questions for authors

See weaknesses section.

Reviewer_vgPbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper extends AutoEva to the scenario in presence of covariate shift. The authors proposes an estimator using importance weights and plugging them into the PPI++ bias correction. They establish theoretical results on consistency, asymptotic normality and an ESS bound.

Critical strengths
  • It is a practically relevant problem, and the proposed method is easy to implement.
  • Rigorous theoretical properties are provided.
Critical weaknesses
  • Although framed as a model evaluation problem, it is in spirit same as a PPI estimation problem under covariate shift, which has been widely studied in literature. The weighting approach is fairly standard in covariate shift/missing-data literature. The theoretical results follow from standard arguments used in PPI++. Overall, the contribution is really incremental.
  • The experiments are not convincing. In particular, the coverage keeps decreasing as the sample size increases. It looks like the estimator has some systematic bias that does not vanish with increasing sample size.
Questions for authors

See above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.