- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies the acceptance of candidate updates in test-time adaptation (TTA): even when a test input warrants an adaptation attempt, the resulting parameter update may not be worth retaining. The authors propose a four-stage pipeline that screens inputs, generates candidate updates, evaluates them, and either commits or rolls them back. First, predictive entropy and quota rules are used to allocate the budget for generating candidate updates. Next, one step of entropy minimization is performed on the update views. Finally, independently augmented views of the same batch of original images are used to compare prediction consistency before and after the update, with an additional penalty for prediction drift relative to a frozen pretrained model. The candidate parameters and optimizer state for the current batch are committed only if the acceptance score exceeds a threshold; otherwise, the pre-adaptation state is restored. The paper reports experiments on datasets including ImageNet-C and Office-Home, using models such as CLIP and DINOv2. The improvements over the baselines are limited.
Critical strengths
The paper addresses a practically important problem. TTA updates can degrade model performance or even cause model collapse, making it necessary to evaluate the updates themselves. Compared with the pre-update sample filtering commonly used in prior TTA methods, this paper introduces a mechanism for accepting or rejecting candidate updates. This is a general strategy that could potentially be integrated into existing TTA methods to improve their robustness.
Critical weaknesses
-
The experimental results do not provide sufficient evidence that the proposed method achieves state-of-the-art performance. The most recent baseline is SAR, published in 2023, and the average improvement over SAR is less than . These gains are marginal, and comparisons with more recent methods are missing.
-
The paper frequently uses defensive formulations such as "rather than," making the writing difficult to follow and disrupting its flow. The main figure consists almost entirely of text, making it difficult to understand the rationale behind the method's design.
-
The proposed gating mechanism has limited novelty. Multi-view consistency across augmented views and regularization that penalizes drift from a pretrained model are common components of TTA methods. Although the paper explicitly acknowledges that consistency alone does not guarantee correctness, it still uses consistency on held-out views as the basis for gating, without providing a theoretical justification for this choice.
-
The experimental assessment of harmful updates considers only samples in the current batch. It does not account for the long-term effects of TTA updates on predictions for subsequent samples. Therefore, the reported reduction in harmful updates does not directly establish the effectiveness of the gating mechanism.
Questions for authors
-
Could you include comparisons with more recent baselines, particularly the latest TTA methods, to better demonstrate the effectiveness of the proposed method?
-
Why do you adopt the gating mechanism despite explicitly acknowledging that it does not guarantee correctness? Could you provide theoretical or experimental evidence to justify this choice?
-
Equation (6) implies that when the number of probe views is M = 1, inter-view disagreement should always be zero. Consequently, all candidate updates would be rejected, and the model should remain in its original state. This is inconsistent with the gain of 2.04 reported in Appendix Table 12. Could you explain this discrepancy?