acceptodds
Under review as a conference paper at ICLR 2027

Unsupervised Domain Shift Detection via Discrepancy Between Pointwise and Prototype-Refined Pseudo-Labels

Abstract

We study unsupervised domain-shift detection for image classifiers in the realistic setting where target labels are unavailable. We propose a simple detector based on the discrepancy between two label assignments on unlabeled target data: the classifier’s pointwise pseudo-labels and prototype-refined pseudo-labels obtained through batch-level prototype refinement in feature space. Our key observation is that these two labelings largely agree when the source classifier remains well aligned with the target domain, but diverge under harmful shift, when the source decision boundary becomes mismatched with the target structure. This leads to a shift detection score that can be computed directly from a deployed model, without training an auxiliary detector or explicitly estimating source–target divergence. Experiments on standard visual domain-shift benchmarks show that the proposed score accurately detects batch-level shift and tracks accuracy degradation better than confidence-based and distribution-based baselines. These results suggest that pseudo-label self-consistency provides a simple and effective criterion for detecting performance-relevant distribution shift.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
domain shift detection, pseudo-labels

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Z6KLOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

They propose an approach to detect the domain shift from the training domain, by using the discrepancy between the outputs of the standard-classifier and prototype based classifier. They measure the performance of the proposed approach by using the correlation between their proposed score and the domain-shift labels. Empirically, they show that the proposed approach outperforms baseline approaches in many cases, where they test their approach in the conventional domain adaptation benchmark such as domainnet, office, office-home.

Critical strengths
  1. The presentation is clear. Mostly, it is easy to follow their approach.
  2. The proposed approach is reasonable.
  3. The approach is working in domains that are unseen during training.
Critical weaknesses
  1. The novelty is limited because identifying the shift or the confidence of the prediction by using discrepancy of the classifiers is already proposed as indicated by the submission as in [1]. They also do not have the comparison with [1].
  2. Their proposed method is a bit heuristic, and there is no theoretical guarantee.
  3. They test their approach only on the conventional object-centric dataset. The results might lack generality. It is better to include datasets such as WilDs. Considering the goal of the task, they should include tasks such as weather-shift or noise-shift.
  4. The tested dataset looks a bit too easy. F-1 score in Table 1 is almost 100% in the baseline approach. That is also why they should test their approach in more diverse settings.

[1] Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shift, Baek et al., NeurIPS 2022

Questions for authors

All weaknesses above need to be addressed to increase my rating. If you have any clarification, please mention that in rebuttal.

Reviewer_u4x4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces PLSC (Pseudo-Label Self-Consistency), an unsupervised domain-shift detector that measures disagreement between pointwise pseudo-labels (PL) and their prototype-refined counterparts (PPL). It operates without target labels, model retraining, or an auxiliary detector. Its claimed novelty is based on pseudo-label refinement, commonly used in source-free adaptation, to detect domain shifts, with a particular emphasis on harmful shifts. Experiments on Office-31, Office-Home, VisDA, and DomainNet126 show the highest average detection F1 among the tested MMD, KS, and C2ST configurations, and report correlations between p-values and model accuracy across source and target batches. Ablation studies further favor Euclidean distance over cosine distance and soft clustering over hard clustering.

Critical strengths
  • A simple diagnostic without additional model training. PLSC requires only a trained source model, an unlabeled source reference set and an unlabeled target batch. It does not update the deployed model or train an auxiliary detector. Repurposing prototype refinement as a diagnostic signal is practically attractive, and the competitive results across multiple benchmarks offer encouraging evidence of its utility. A direct runtime comparison would, however, help substantiate its computational advantage.

  • A permutation procedure that preserves batch dependence. Because prototype-refined labels are computed from the full batch, the individual agreement indicators cannot be assumed independent. Recomputing prototypes for each permutation accounts for this dependence under the null hypothesis of source–target exchangeability. This is a methodological strength, although the resulting calibration applies to distributional equality, not to the absence of harmful performance degradation.

Critical weaknesses
  • The experiments do not adequately support the claim of harmful-shift detection. The paper does not define harmful shift using an explicit threshold for performance degradation, and its test is calibrated to detect distributional differences rather than harm. Table 1 evaluates whether source and target batches can be distinguished. The low-noise experiments do consider approximately accuracy-preserving shifts, but they are limited to Gaussian noise and report mean p-values instead of alert frequencies. These results therefore leave open whether PLSC avoids unnecessary alerts on benign shifts while still detecting harmful ones. This is a central limitation given the paper’s stated contribution.
  1. The incremental contribution is not sufficiently established against relevant alternatives. Lee et al. (ICCV 2023) already use disagreement with an adapted target pseudo-labeling function for label-free accuracy estimation. PLSC replaces model adaptation with prototype refinement and adds a permutation test, but the paper should explain more clearly what these differences contribute. MaNo is cited but not evaluated, and the accuracy correlations in Table 2 are computed by pooling source and target batches. Comparisons with relevant performance-estimation approaches and correlations computed on target batches alone would help determine whether PLSC captures performance degradation beyond simply distinguishing domains.
Questions for authors

Questions that could change my assessment

  1. How do you operationally define harmful shifts? Can you report alert frequencies at p < 0.05, with uncertainty, for both approximately benign and harmful corruption settings?
  2. Among PL–PPL disagreements, how often does PPL correct an incorrect prediction versus change a correct prediction into an error? Comparing overall PL and PPL accuracy would be useful, but this decomposition would more directly test the proposed mechanism.
  3. Does the accuracy correlation remain strong on target batches alone and when measured against accuracy degradation? Please clarify the observations and aggregations used in Table 2.
  4. How does the approach differ from Lee et al. (ICCV 2023), and how does it compare with MaNo on performance tracking? If you compare with Lee et al., please distinguish the additional adaptation cost.

Clarifications

  1. Which MMD kernel and bandwidth were used, and how many permutations were performed? Clarify what the parenthesized values in Table 1 represent.
  2. How does PLSC’s runtime compare with the baselines, including the full permutation procedure?
  3. The boldface convention in Table 1 is unclear: KS-features’ average of 98.1 is highlighted despite higher averages for KS-softmax and PLSC.
Reviewer_TkaCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces PLSC, an unsupervised detector that operates at the batch level and aims to identify harmful domain shift. Given a target batch, PLSC contrasts the pseudo-labels produced by the source classifier with labels reassigned according to the closest probability-weighted class prototype in feature space, in the spirit of SHOT. A permutation test then assesses whether this agreement is significantly weaker than on an unlabeled source reference batch. The prototypes are re-estimated under every permutation. Across Office-31, Office-Home, VisDA and DomainNet126, the authors report a 99.0% average detection F1 and an 84.5% average absolute Spearman correlation between p-values and accuracy.

Critical strengths

(1) The approach is straightforward and inexpensive to deploy, and its statistical test is carefully designed. All it requires is a trained model, unlabeled source reference data, and an unlabeled target batch. No auxiliary training stage and no parameter updates are involved.

(2) Because the prototypes are re-estimated inside each permutation, the statistic's dependence on the batch is properly taken into account. Provided the advertised advantage holds up, this renders PLSC a plausible low-cost monitoring tool.

Critical weaknesses

Weaknesses

  • The central claim that PLSC selectively detects harmful rather than generic distribution shift is not adequately established. Section 3 describes harmful shift as distributional change that substantially reduces the source classifier's accuracy, but provides neither an operational degradation threshold nor an independent evaluation that separates harmful from harmless shifts. The formal null hypothesis is distributional exchangeability (Ps=PtP_s=P_t), while the test statistic measures reduced PL–PPL agreement rather than accuracy degradation directly. The main cross-domain experiments generally couple distribution shift with lower accuracy, and the controlled Gaussian-noise experiments also vary shift severity and degradation together. At near-zero degradation, KS-softmax and C2ST-softmax are less reactive than PLSC on all three evaluated domains (e.g., mean p-values of 100.0% versus 29.0% on Office-Home A at the lowest noise level), while KS-softmax detects smaller degradations earlier on R and P. Moreover, Table 2 computes absolute Spearman correlations over both source and target batches, which may conflate source–target separation with accuracy tracking within shifted target data. Overall, the evidence supports a model-aware shift detector, but does not yet establish the claimed harm-selective behavior.

  • The empirical gains are modest and the comparison is incomplete. Detection F1 is near-saturated among the strongest baselines (99.0% for PLSC versus 98.5% for KS-softmax). Although Table 1 reports variability in parentheses, no paired significance tests are provided for this small margin. Table 2 reports a 1.4-point average absolute Spearman advantage without uncertainty estimates, and PLSC is not the best method on Office-Home or VisDA. Important related work is missing: Detectron (Ginsberg et al., ICLR 2023) tests for harmful covariate shift via classifier disagreement with a permutation test, while Dispersion (Xie et al., NeurIPS 2023) estimates accuracy from label-free feature separation; together with MaNo, these are natural baselines for the accuracy-tracking claim.

Questions for authors

Clarifications: What empirical Type-I error (rejection rate) is observed at α=0.05\alpha=0.05 on source-vs-source batches? Please also report the false-positive and false-negative counts underlying Table 1. The nominal 5% significance level does not require the realized false-positive rate to equal 5%, since permutation tests can be conservative. Nevertheless, the reported 99.0% average detection F1, including 100.0% on VisDA, warrants an explicit breakdown of false positives, false negatives, and detection power. How many (p-value, accuracy) pairs contribute to each correlation in Table 2? What signed correlation is obtained when computed over target batches alone, and what are the corresponding uncertainty estimates?

Experiments that would change my assessment: Please report per-trial rejection rates for shifts that modify p(x) while leaving accuracy intact, in addition to the harmful pairs.Candidate cases include Office-31 D↔W, provided that the specific source–target direction is empirically verified to preserve model accuracy, or label-preserving transformations under which the model's accuracy remains approximately unchanged. Please benchmark against Detectron, and against Dispersion and MaNo on the accuracy-tracking metric.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.