acceptodds
Under review as a conference paper at ICLR 2027

When Is Test-Time Adaptation Identifiable from Unlabeled Evidence?

Abstract

Test-time adaptation (TTA) offers many ways to update a deployed model without labels, but choosing the wrong update can make a strong source model worse. Recent methods therefore try to predict which adaptation will work from unlabeled test data. Yet the difficulty is not merely that different deployments can prefer different TTA actions: two deployments can be indistinguishable under the selector's allowed pre-action evidence while requiring opposite actions. We therefore ask a prior question: does the evidence given to the selector contain enough information to determine the best action at all? We show that this is not guaranteed, even with a perfect selector. If an observation channel makes two deployments look the same while their TTA rankings differ, reliable selection is impossible from that channel; richer evidence can restore the decision only when it resolves the relevant ambiguity. TTA selection failure can therefore reflect an information-channel failure, not only a selector-capacity failure. We make this boundary explicit in a finite-batch Gaussian TTA model, where doing nothing beats mean recentering for small shifts, recentering wins beyond a unique critical shift, and the boundary shrinks with batch size as . Controlled experiments on CIFAR-100-C and DomainNet-126 instantiate the same obstruction with modern TTA methods: across both synthetic corruptions and natural domain shifts, changing only deployment stream structure can reverse the oracle action while global order-blind evidence remains essentially invariant. Graded correlation sweeps show these reversals emerging progressively as temporal structure increases, while order-blind evidence barely changes. This separates two failure modes that are usually mixed together—a weak selector and an information channel that cannot support the desired decision—and suggests a simple question before building a stronger selector: can its inputs support the decision at all?

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
test-time adaptation, identifiability, unsupervised model selection, distribution shift, domain adaptation, temporal correlation, unlabeled data

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_VgeUOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies the problem of which TTA method to choose when receiving new data. It asks when it even is possible to distinguish the best action to take, under the evidence the selector is given. The paper shows that even a perfect selector cannot always choose the best action, given the evidence.

Critical strengths
  1. The paper explores an interesting research direction of when it even is possible to choose the correct adaptation method.

  2. There are experiments on a variety of datasets and different selector are used in experiments to compare against.

Critical weaknesses
  1. The paper is extremely difficult to follow. It presents a quite simple problem setting and method, in an extremely convoluted way. Extremely many keywords are used and the language is structured very complicated. Formatting is also off in many places. It makes it hard to understand what the main contribution of the paper is.

  2. Much of the theory that is presented, is trivially derived from the problem formulation. They also do not seem to provide a foundation for deciding when an adptation method will be helpful and are seemingly detached from the experiments.

Questions for authors

No further questions.

1 comment
Reviewer_VgeUOpenReview ↗

Response to the authors

Thank you for the response and revisions. The new matched-pair regret analysis strengthens the connection between the theoretical formulation and the experiments. In particular, showing that the order-blind selectors perform near the minimum regret achievable under a shared-action constraint helps distinguish insufficient information from limitations of a fitted selector. The matched-regressor comparison and clarification of the RF-versus-Ridge results are also useful.

I appreciate the clearer distinction between the familiar impossibility principle and the claimed TTA-specific contribution. I also recognize that an analysis of information limits need not provide a universal rule for choosing an adaptation method.

My remaining reservation concerns the significance and practical scope of the resulting insights. The strongest evidence comes from controlled reorderings and selected strong pairs, leaving unclear how frequently consequential action ambiguities arise in typical deployments. The natural-data experiments provide a useful boundary case, but SOURCE remains sufficient throughout the tested action menu, so they do not establish a practical need for stream-aware selection in those settings.

Overall, the rebuttal meaningfully addresses my concern that the theory and experiments were disconnected. I am therefore raising my score from 1 (clear rejection) to 2 (weak rejection). My remaining concern is whether the TTA-specific findings provide sufficient insight beyond the general observation that order-blind evidence cannot distinguish streams requiring different actions.

Reviewer_AuU5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper is tackling the problem of when and how (which TTA method) to adapt when distribution shift is present. In particular, it attempts at answering the question: does the stream of unlabeled data have enough evidence to dictate when and how to adapt? The paper presents some theoretical analysis along with small scale experiments on CIFAR100 and DomainNet.

Critical strengths

The main strength of this paper is the problem that it attempts to tackle. It is worth exploring how one can identify from the unlabeled stream of data at test time when and how to adapt.

Critical weaknesses

The main weaknesses of this paper are:

  1. The paper is very hard to read, follow, and understand. It also feels that AI was heavily used in the writing.

  2. The theoretical analysis presented in this work are also hard to follow, despite the simplicity of some of the points (e.g. definition 1). The paper introduces a large amount of notations and terminologies along with distinctions between theoretical and empirical setting that makes it difficult to follow.

  3. The experimental setup is synthetic and does not follow the standard practice (neither in terms of chosen benchmark, nor the evaluation protocol and metric). The strongest results arise when the authors deliberately reorder the same samples to create IID versus temporally correlated streams. This is a clean way of demonstrating the claimed information-channel obstruction, but it is less clear how often such strong oracle-action reversals arise naturally under commonly used TTA evaluation protocols (episodic/continual setting, ImagNet-C/R benchamrks, average online error rate metric, ...etc).

Questions for authors

refer to the weaknesses section

Reviewer_vniyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether pre-action unlabeled evidence is sufficient to identify the optimal test-time adaptation (TTA) action from a fixed set of methods, including KEEP (no adaptation). It establishes identifiability conditions and impossibility results, derives a finite-batch KEEP-versus-RECENTER boundary, and demonstrates controlled deployments in which changing test-stream order reverses the best action while leaving global order-blind summaries essentially unchanged.

Critical strengths
  1. The distinction between selector limitations and insufficient observational evidence provides a useful perspective on TTA action selection.

  2. The matched-world experiments are carefully designed and convincingly demonstrate oracle-action reversals using identical images with different orderings.

  3. The paper includes valuable robustness analyses and acknowledges settings where stream-aware features provide little benefit.

Critical weaknesses
  1. The experiments could more directly quantify the unavoidable regret associated with the order-blind observation channel. The matched pairs establish that a permutation-invariant selector cannot always choose the oracle action, but the learned selectors’ regret also reflects their fitting and generalization performance. Since the candidate-action errors are already available, the authors could compute, for each pair, the minimum average regret attainable when both members must receive the same action, optimizing over all five candidates. Comparing this lower bound with Z2 and MORPHEUS-NC on the same matched-pair subset would quantify the unavoidable pairwise cost and the remaining excess regret. This would strengthen the empirical connection to the paper’s central distinction between information limitations and selector limitations.

  2. The interpretation of the correlation-sweep selector results needs clarification. Figure 5 shows substantial Z3-RF regret at intermediate correlation strengths on DomainNet, while Ridge-ORDER11 performs substantially better there. Section 4.1 describes the general evaluation protocol, but Appendix B.5 does not explicitly state whether the sweep selectors are transferred from the main experiments or refitted, which conditions enter training and evaluation, or what “held-in” transfers means. Figure 5 already includes ORDER11 RF, but its window size is not specified alongside Ridge’s W=32. Clarifying the sweep protocol and confirming that the RF-versus-Ridge comparison uses identical ORDER11 features, window sizes, and training/evaluation splits would help interpret these differences. Numerical regret summaries and variability across the three existing data-generation seeds would also be useful. High regret from one fitted selector alone does not establish insufficiency of its evidence.

Questions for authors
  1. How are the selectors in Figure 5 trained and evaluated? Are they transferred from the main experiments or refitted on sweep worlds? Which correlation strengths and corruption/transfer conditions enter training and evaluation, and what does “held-in” mean for the DomainNet sweeps?

  2. Does the ORDER11 RF curve in Figure 5 use W=32 and the same training/evaluation splits as Ridge-ORDER11? If so, what explains their different behavior at intermediate correlation strengths? Please report numerical regret summaries and variability across the three existing data-generation seeds.

  3. For each matched pair, what is the minimum average oracle regret when both worlds must receive the same action, optimizing over all five candidates? How close are Z2 and MORPHEUS-NC to this lower bound on the same matched-pair subset?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.