- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents the RASD approach for online test-time adaptation of vision-language models. The weak internal perturbation signals may support the robust full-view teachers selection and the soft correction target construction. Selected teachers are more likely to supervise the masked student views via self-distillation. And a batch marginal-entropy loss is likely to prevent the collapse of predictions. Experiments on common corruptions, natural Image Net shifts, recognition datasets and cross-domain benchmarks are conducted.
Critical strengths
The results about the corruption are encouraging. The results show that RASD is 4.06pp better than LMR on CIFAR-10-C and 3.95pp better on Image Net-C. These gains are impressive and additional experiments with ViT-L/14 suggest the benefits are probably not constrained to a single backbone.
The diagnosis of reliability seems to justify the proposed method. Furthermore, as shown in Table C, predictions in agreement with the internal consensus tend to be more accurate on all datasets. This justifies the idea that internal consistency can be used as an internal indicators to choose the teacher instead of each prediction for adaptation.
Critical weaknesses
-
The comparison does not resolve a number of important issues around tuning and computational budget. The RASD approach uses dataset specific learning rates, masking settings, and loss weights; however, the procedure used to determine these is not explained. A number of baseline approaches make use of transferred settings without a target-label search, which presents an additional difficulty in interpreting the generalization results. For each batch RASD performs 10 adaptation steps, while BITTA and LMR perform one. Without results controlling for the computational budget or wall clock time, the benefit of this additional computation is unclear.
-
The ablation experiments do not completely ablate out the proposed reliability mechanism. As shown in Table 5, consensus correction with marginal-entropy regularization reaches 80.62/52.25, and the full method reaches 81.97/53.69. While the entropy term plays a large role in bringing this improvement, the authors do not use an entropy-only baseline. While the removal of the reliability gate in Table H is interesting, it does not differentiate whether the consensus mechanism is better than filtering solely by confidence, or simple average aggregation with only mean-on-average calibration, which would bring out the main contribution from the stabilization components.
Questions for authors
-
What data and selection criteria were used to determine the dataset-specific settings given in Table A? Were any labels from the target-test set excluded from any procedure that pertains to hyper-parameter selection?
-
How does RASD fare compared to LMR for the same adaptation budget? Let us evaluate in terms of end-to-end processing time and maximum memory usage, what is the cost of the extra probe and student branch.
-
What causes the differences to seemingly contradict each other (Table 5 showing 72.85/40.66 for gated selfdistillation and Table G showing 64.34/38.75 with learnable filling vector with seemingly the same aims)? Please specify if I missed configuration difference between these experiments.
Dear Author, is there anything you'd like to discuss? Please note that the discussion period is coming to an end. We highly value your explanations and comments.