- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how performance gains should be attributed to individual filtering stages in selective test-time adaptation (TTA). The authors first record the number of samples retained by the original method at each stage of each batch. Under the same sample-count constraints, they then replace each stage's sample-selection rule with either selection based on the original score ranking or random selection. For a two-stage pipeline, the authors construct four crossed combinations and decompose the stage contributions using two-player Shapley values and an interaction term. The experiments analyze widely used TTA methods, including DeYO, EATA, and SAR, on ImageNet-C. The authors find that, under most test conditions, the original selection rules outperform random selection with matched sample counts. They also observe that two-stage filtering can exhibit a reversal upon combination: each stage contributes positively in isolation, but their combination makes a negative contribution.
Critical strengths
The experimental setup is relatively novel. By separating the filtering stages of TTA methods and analyzing their individual contributions, the paper offers a viable approach to studying sample selection in TTA and designing ablation experiments.
Critical weaknesses
-
The paper has limited novelty. Its main contribution is an experimental framework for analyzing the contributions of filtering stages in TTA, but it neither proposes a new method nor improves an existing one.
-
The intervention used to analyze stage contributions is artificial: a filtering stage in the original TTA method is directly replaced with random selection while the number of selected samples is held fixed. This experimental design does not reflect standard TTA settings.
-
The paper's contribution is limited to empirical observations, without rigorous theoretical analysis. The conclusion of Proposition 1 is also relatively straightforward: it merely shows that negative interactions can arise between two filtering stages.
Questions for authors
-
Could you clearly state the paper's specific contributions? What insights does the analysis of two-stage filtering provide for designing and improving TTA methods?
-
The paper analyzes the effects of sample selection across different experimental cells for methods such as SAR and DeYO. Can you derive a unified, actionable conclusion that generalizes across methods?