acceptodds
Under review as a conference paper at ICLR 2027

EXACT: Estimation–Support Asymmetry in Test-Time Adaptation

12 views

Abstract

Test-time adaptation (TTA) uses unlabeled target observations to improve model performance under distribution shift, often by estimating target-side statistics from the observed data. Existing methods mainly focus on how these statistics should be estimated or optimized, while paying less attention to whether the available target data are sufficient for reliable estimation. Different target statistics, however, have different data requirements. Global statistics are estimated from all observed target samples, whereas class-specific statistics depend on samples from the corresponding classes. We refer to this difference as estimation–support asymmetry. Our analysis and experiments show that their relative reliability varies with the amount of available target data: global statistics are more reliable when data are limited, whereas class-specific statistics become more effective as target samples accumulate. These findings motivate an inference strategy that ties the use of target statistics to the data available for their estimation. We instantiate this principle in EXACT, a framework that uses global statistics when target data are limited and enables target prototype estimation as observations accumulate. A label-free, coverage-based criterion derived from our class coverage analysis determines when class-specific statistics are used. EXACT requires no source data, target labels, backpropagation, or updates to the pretrained model. Across seven corruption and natural-shift benchmarks, EXACT achieves the highest average accuracy among the compared methods in all four backbone/inference settings. Reliability and ablation analyses provide further evidence for the estimation–support perspective and its coverage-based implementation.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Test-time adaptation, Distribution shift, Estimation-support asymmetry, Class coverage, Prototype estimation

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_AeuLOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studied frozen model, backpropagation-free test-time adaptation. In this paper, the authors argue that global target statistics and class-specific target statistics differ in their sample requirements, which it calls estimation support asymmetry. The paper reports a reliability reversal. A branch using the global logit statistics PuP_{u} is more accuracte with few target samples, while a prototype-based branch PtP_{t} becomes better as samples accumulate. EXACT swithes from PuP_{u} to PtP_{t} once the sample count reaches C⋅HCC\cdot H_{C} which is the coupon-collector expectation under uniform classes. It achieves the best average on seven benchmarks across ViT/CLIP and online/transductive settings, and improves 35 additional backbones.

Critical strengths
  • The paper is well-written and the motivation is clear. Also, the method is simple, trainging-free, and nearly hyperparameter-free.

  • The evaluation is broad, covering seven benchmarks, two protocols, and 35 extra backbones, with consistent gains.

  • The diagnostics in Figures 3 and 4 are informative, and the authors are candid that C⋅HCC\cdot H_C is not an exact crossover.

Critical weaknesses

W1. The analysis compares the estimation variance of each statistic around its own population value and never relates it to accuracy. A “variance only” argument can’t produce a crossover, since it favors the global statistic at every NN. The large NN advantage of PtP_{t} requires a bias term, such as the global centering being unable to correct class-conditional shifts, and no such term is given.

W2. In the transductive setting N≥C⋅HCN\geq C\cdot H_C always holds, so EXACT reduces to PtP_{t}. On the online experiments, the switch occurs early in the stream, at about 5%5\% on CIFAR-100-C and 15%15\% on ImageNet-C, which means that most prediction come from PtP_{t}. On ImageNet-C, most of the gain comes from PuP_{u} alone. Figure 12 confounded in the same way, since replacing early predictions of any method with a stronger predictor would improve it.

Questions for authors

Q1. The paper attributes the small NN failure of PtP_{t} to limited estimation support, but Figure 4 shows only that absent class weight and prototype error decrease together with NN. Could the authors run two interventions at fixed NN, (a) building PtP_t from oracle-label prototypes and (b) restricting PtP_t to classes with non-negligible soft count, and report whether the collapse persists?

Q2. Two components could cause the small NN collapse independently of support. The weight normalization in Eq. (14) turns every absent-class prototype into a full-magnitude average of other classes. The balanced Sinkhorn in Eqs. (12) and (16) enforces a uniform marginal even when N<CN<C. Could the authors report Always PtP_{t} accuracy versus NN, as in Figure 3, with the Sinkhorn steps replaced by softmax and with prototypes initialized from the classifier weights as in T3A?

Q3. TDA (Karmanov et al., CVPR 2024) builds class-wise feature caches. It is training-free, backpropagation-free, and source-free, so it appears to meet the baseline criteria stated in App. A.3, and it is cited in the paper. The reviewer would like it to be added to Table 1.

Reviewer_y5USOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EXACT studies how the usefulness of global and class-specific target statistics changes as unlabeled samples accumulate. It first predicts using standardized logit statistics, then switches at a class-count-based coverage reference to soft-weighted target prototypes. The model stays frozen. Online evaluation scores each arriving batch without revising earlier predictions; transductive evaluation uses the full target set. The paper reports the highest average among compared methods in four backbone/protocol settings. Its uniform-sampling analysis motivates a reference threshold, with explicit limits on coverage, assignment error, and the location of the accuracy crossover.

Critical strengths
  • The branch comparison identifies a useful empirical principle. Figure 3 and Section 4.3, shows global statistics working better at low support and class-specific inference becoming more useful later. The theory does not locate the exact accuracy crossover, which the paper acknowledges.

  • The empirical scope is broad within frozen-model adaptation. Table 1, reports online ViT/CLIP averages of 67.24/52.55 and transductive averages of 68.15/53.65 across seven benchmarks. These exceed the strongest alternative averages by 2.88, 1.04, 3.79, and 1.25 percentage points. No comparison to parameter-updating methods outside the stated scope follows.

Critical weaknesses
  • Missing numerical temperatures leave a central inference operator underspecified. Eqs. 12 and 16, uses Sinkhorn normalization for logits, cosine scores, and fused standardized scores. Appendix A.3 specifies temperature scaling, uniform marginals, and 20 iterations, but gives no numerical temperatures. These values affect assignments and predictions, so the manuscript alone does not fix the deployed procedure.

  • The practical comparison with adaptive fusion remains incomplete. The paper describes that comparator correctly. However, trailing an oracle branch does not determine whether a mixture trails the deployed count-based switch. Direct cumulative scoring of the existing predictions would better establish the design choice. This is a minor comparison gap, not evidence that fusion wins.

Questions for authors
  1. What temperature does each Sinkhorn call use? Are these values fixed across datasets and backbones, and what numerical safeguards accompany them?
  2. Using the existing predictions, what are the stream-average accuracies of the tested fusion rules against EXACT, with paired seed differences?
  3. Can the five-seed main means include variability? This would help assess close differences.
Reviewer_qL2YOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how the amount of available target data affects the reliability of statistics used for frozen-model test-time adaptation. Its central observation is that global statistics use all observed samples, whereas class-specific statistics depend on per-class support. The proposed EXACT framework initially uses global logit normalization and switches to soft-weighted target prototypes once the sample count reaches the expected uniform coupon-collector coverage time. Experiments cover seven benchmarks with supervised and CLIP backbones, showing the effectiveness of EXACT.

Critical strengths
  1. The paper identifies a useful practical question and investigates it systematically. Evaluating adaptation across target sample sizes is informative beyond reporting accuracy on complete datasets.

  2. The empirical results are broad and encouraging.

Critical weaknesses

Section 3.1 provides only qualitative support for the proposed reliability reversal. The rates compare different population quantities with different variance constants, so they do not establish that global statistics are more reliable at low support. Moreover, the analysis concerns true-label means, whereas EXACT uses soft-weighted prototypes. Appendix A.1.3 addresses assignment error, but its additional terms need not vanish as samples accumulate.

Questions for authors
  1. What assumptions would connect the estimation-error rates in Section 3.1 to the classification errors of PuP_u and PtP_t? Please clarify which aspects of the reliability reversal are theoretically supported and which are empirical observations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.