acceptodds
Under review as a conference paper at ICLR 2027

Intervention-Dependent Selection Credit in Stateful Test-Time Adaptation

Abstract

Selective test-time adaptation (TTA) methods choose which test samples drive each update. Their gains are often credited to reliable samples, but selection also changes sample counts and loss weights, while each update changes the model that scores later samples. Standard ablations conflate these effects. We measure the value of sample identity by recording native per-stage counts and rerunning each stream with those counts, choosing samples by each run's own native ranking or at random. Crossing these choices at the two stages of DeYO, EATA, and SAR yields a factorial that splits identity value between stages and measures their interaction. Native selection has positive identity value in 164 of 180 ImageNet-C cells. Yet stage credit changes with the comparison: on a five-corruption panel with 10 seeds, first-stage dominance is resolved in 31 of its 35 cells under the original comparison; matching mean loss weights leaves 19 of 40 cells first-stage dominant, with 8 resolved reversals (9 with joint resampling), and removes most of the ViT-B/16 identity value. In 8 of 12 candidate SAR cells, 20-seed intervals show that stages helping individually can hurt together. Waterbirds further separates identity-driven worst-group gains from harms that weaken under uniform loss weights. Selection credit is therefore intervention-dependent: it describes a stage's marginal effect within a specified stateful comparison, rather than an intrinsic property of its criterion.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
test-time adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_zhqKOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how performance gains should be attributed to individual filtering stages in selective test-time adaptation (TTA). The authors first record the number of samples retained by the original method at each stage of each batch. Under the same sample-count constraints, they then replace each stage's sample-selection rule with either selection based on the original score ranking or random selection. For a two-stage pipeline, the authors construct four crossed combinations and decompose the stage contributions using two-player Shapley values and an interaction term. The experiments analyze widely used TTA methods, including DeYO, EATA, and SAR, on ImageNet-C. The authors find that, under most test conditions, the original selection rules outperform random selection with matched sample counts. They also observe that two-stage filtering can exhibit a reversal upon combination: each stage contributes positively in isolation, but their combination makes a negative contribution.

Critical strengths

The experimental setup is relatively novel. By separating the filtering stages of TTA methods and analyzing their individual contributions, the paper offers a viable approach to studying sample selection in TTA and designing ablation experiments.

Critical weaknesses
  1. The paper has limited novelty. Its main contribution is an experimental framework for analyzing the contributions of filtering stages in TTA, but it neither proposes a new method nor improves an existing one.

  2. The intervention used to analyze stage contributions is artificial: a filtering stage in the original TTA method is directly replaced with random selection while the number of selected samples is held fixed. This experimental design does not reflect standard TTA settings.

  3. The paper's contribution is limited to empirical observations, without rigorous theoretical analysis. The conclusion of Proposition 1 is also relatively straightforward: it merely shows that negative interactions can arise between two filtering stages.

Questions for authors
  1. Could you clearly state the paper's specific contributions? What insights does the analysis of two-stage filtering provide for designing and improving TTA methods?

  2. The paper analyzes the effects of sample selection across different experimental cells for methods such as SAR and DeYO. Can you derive a unified, actionable conclusion that generalizes across methods?

Reviewer_wBeuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This work investigates selective test-time adaptation. In particular, the authors use a type of intervention testing framework to investigate multi-stage TTA approaches, enabling per stage contributions to be observed that are obscured (or unobservable) in typical ablation studies.

The authors investigate their framework on 2 model architectures, 3 datasets, and 3 TTA approaches. Their results indicate that "native" selection is better than random / uniform sampling, first stage selection is more important than second stage selection, and that stage selection that are beneficial separately, may be detrimental collectively.

Critical strengths

I would first state that I am not a domain expert, and thus some of my comments may come from ignorance of the research domain, rather than the quality of the paper itself.

  1. I cannot speak to the paper's novelty, but I can see why a multi-stage decomposition framework via interventions such as the one proposed could be useful.

  2. The loss weights and composition reversal results are quite interesting, and I like that the authors challenged the notion that "more selection stages are better" and demonstrated under which methods it could occur.

Critical weaknesses

The paper is very hard to read

For lack of a better term, the paper is obtuse. I do not understand why the authors repeatedly choose to describe their methods and results in as complicated a manner as possible.

For example, the introduction and abstract report on very specific results with no context. Using an excerpt from the abstract:

We measure the value of sample identity by recording native per-stage counts and rerunning each stream with those counts, choosing samples by each run’s own native ranking or at random. Crossing these choices at the two stages of DeYO, EATA, and SAR yields a factorial that splits identity value between stages and measures their interaction.

What is sample identity or native per-stage counts? What choices are being crossed? There are two stages of what? I acknowledge that might be missing some knowledge about the jargon used in this domain. However, the abstract doesn't mention that you are looking at multi-stage selection processes until "per-stage counts" and even then it is implied.

The next sentence in the abstract is:

Native selection has positive identity value in 164 of 180 ImageNet-C cells.

As a first time reader of this paper, I am asking myself, "what is a cell?", "why are there 180 of them?", "is a positive identity value good?".

I urge the authors to consider the perspective of the reader when engaging with their research. As best I can tell, the authors are trying to say "the sample selection procedures of DeYO, EATA, and SAR for TTA are better than a random sampling procedure." Why not just say that?

The whole paper is like this, and, as a result, I think a significant round of editing is needed to make the paper more concise.

I also think the authors skirt the line of what is acceptable regarding the page limit. The paper, and several of its main results, are only comprehensible if one consults the appendix. For example, the ColoredMNIST experiments are barely a paragraph with the bulk of the results in the appendix. Almost every single result in the main paper relies on some knowledge or evidence present in the appendices.

Some results are not surprising

As mentioned earlier, some of the results of the paper are interesting. However, some of them are not. It is not surprising that native selection outperforms random sampling. Neither is it surprising that stage 1 is typically given more credit than stage 2.

Overall, I lean towards a reject given the quality of the paper's writing. I am open to adjusting my recommendation based on the rebuttal of the authors.

Questions for authors

I did not quite get how the number of samples per stage were selected, I assume it is somewhat dependent on the TTA method.

  1. Is this assumption correct (i.e., that there is no restriction on the number of samples that must be selected in a given stage)?
  2. How do you expect your results to differ if you were to consider that each stage needs to select some smaller subset of the samples from the previous stage. The paper mentions that credit assignment in stage 2 can be lower because the pool of candidates doesn't necessarily decrease by much. What if you required the pool to decrease by some factor at each stage?
Reviewer_zsaVOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents an ablation study for TTA methods that selects a subset of samples to do the adaptation. The main finding is that the selection strategies proposed in these TTA methods are better than random selection of samples.

Critical strengths

The problem area of test time adaptation is useful. Understanding how to learn an optimal set of samples to use for adaptation is important.

Critical weaknesses
  • The paper in the current state seems like a big ablation studies for a handful of TTA methods. The conclusions (especially the first and third in intro) drawn in the paper are broadly known in the literature and are not new.
  • The second conclusion is unclear. Why RQ2 is important to understand? The paper could start with the rationale for the sample selection, e.g., is it compute efficiency or is it unavailability of reliable samples, etc. Depending on these considerations the sample selection method might change between stages. If the paper had described how far a sample selection is from the optimal, that would have been interesting too. However, it is hard to understand why one should study the case of what happens when alternating between random and a heuristic based sample selection.
  • Another interesting aspect of the sample selection is sequential dependency which could have been explored better.
  • The paper compares fairly dated TTA methods, there are a lot of recent methods in the literature. Essentially one can run every TTA method augmented with a sample selection method. It is unclear how the authors have chosen the methods to compare
Questions for authors

As above

Reviewer_7XBWOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper examines the contribution of sample selection to performance in stateful test-time adaptation. The authors fix the number of samples selected at each batch and stage, cross native ranking-based selection with random selection, and develop an experimental approach to quantify each selection stage's contribution and the interactions between stages. They conclude that the contribution of sample selection depends on the specific intervention and loss-weighting settings.

Critical strengths
  1. The paper proposes a simple and effective method for measuring the contribution of sample selection. By matching selection counts for each batch and stage and using tools such as Shapley values, the method quantifies the contribution of sample selection to TTA performance and the interactions between stages.
  2. The experiments are extensive and provide strong support for the paper's observations.
Critical weaknesses
  1. The additional scientific insight remains limited. The paper uses established tools such as the Shapley decomposition, with its main contribution being more fine-grained controlled experiments on existing TTA methods. However, it remains unclear what additional insights into selective TTA this refinement provides. The interpretation of the results largely remains at the general conclusion that the contribution of sample selection varies with the intervention and loss-weighting settings.

  2. The mechanisms underlying the key observations remain insufficiently explained. The experiments show how stage contributions change, but offer limited explanation of why they change. For example, swapping the stage order also changes the samples available to each stage and the number retained, making it difficult to determine whether differences in contribution arise from stage order, the range of samples available for selection, or the selection criteria themselves. SAR also exhibits cases in which retaining native selection at only one stage improves performance, whereas retaining it at both stages reduces performance. However, the paper does not clearly explain this reversal or provide a meaningful investigation into its causes.

Questions for authors

What additional scientific insights does this paper provide beyond standard ablation studies? Could the authors illustrate this with specific findings that advance our understanding of sample selection mechanisms in TTA?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.