acceptodds
Under review as a conference paper at ICLR 2027

Reliability-Aware Self-Distillation for Test-Time Adaptation of Vision-Language Models

Abstract

Test-time adaptation (TTA) improves the robustness of vision-language models under distribution shifts by exploiting unlabeled test data. However, most existing methods derive supervision from the model's current predictions, which may themselves be unreliable under distribution shift and reinforce prediction errors during adaptation. We revisit masked self-distillation as a source of test-time supervision and observe that teacher reliability can be effectively indicated by consistency under mild internal perturbations. Based on this observation, we propose Reliability-Aware Self-Distillation (RASD). Specifically, RASD constructs multiple weak internal probes and aggregates their class-wise evidence into a vulnerability-aware consensus that captures both evidence strength and perturbation sensitivity. The consensus validates full-view predictions for teacher selection and provides a soft correction target, while the selected full-view predictions supervise strongly masked student views through self-distillation. To further stabilize continuous adaptation, we introduce a batch-level marginal-entropy regularization that mitigates prediction collapse. Extensive experiments demonstrate that RASD consistently improves over recent VLM-TTA methods on common corruption benchmarks, while retaining strong generalization across natural ImageNet shifts, diverse recognition datasets, and cross-domain settings.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Self-Distillation, Test-Time Adaptation, Vision-Language Models, CLIP

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Qic1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents the RASD approach for online test-time adaptation of vision-language models. The weak internal perturbation signals may support the robust full-view teachers selection and the soft correction target construction. Selected teachers are more likely to supervise the masked student views via self-distillation. And a batch marginal-entropy loss is likely to prevent the collapse of predictions. Experiments on common corruptions, natural Image Net shifts, recognition datasets and cross-domain benchmarks are conducted.

Critical strengths

The results about the corruption are encouraging. The results show that RASD is 4.06pp better than LMR on CIFAR-10-C and 3.95pp better on Image Net-C. These gains are impressive and additional experiments with ViT-L/14 suggest the benefits are probably not constrained to a single backbone.

The diagnosis of reliability seems to justify the proposed method. Furthermore, as shown in Table C, predictions in agreement with the internal consensus tend to be more accurate on all datasets. This justifies the idea that internal consistency can be used as an internal indicators to choose the teacher instead of each prediction for adaptation.

Critical weaknesses
  1. The comparison does not resolve a number of important issues around tuning and computational budget. The RASD approach uses dataset specific learning rates, masking settings, and loss weights; however, the procedure used to determine these is not explained. A number of baseline approaches make use of transferred settings without a target-label search, which presents an additional difficulty in interpreting the generalization results. For each batch RASD performs 10 adaptation steps, while BITTA and LMR perform one. Without results controlling for the computational budget or wall clock time, the benefit of this additional computation is unclear.

  2. The ablation experiments do not completely ablate out the proposed reliability mechanism. As shown in Table 5, consensus correction with marginal-entropy regularization reaches 80.62/52.25, and the full method reaches 81.97/53.69. While the entropy term plays a large role in bringing this improvement, the authors do not use an entropy-only baseline. While the removal of the reliability gate in Table H is interesting, it does not differentiate whether the consensus mechanism is better than filtering solely by confidence, or simple average aggregation with only mean-on-average calibration, which would bring out the main contribution from the stabilization components.

Questions for authors
  1. What data and selection criteria were used to determine the dataset-specific settings given in Table A? Were any labels from the target-test set excluded from any procedure that pertains to hyper-parameter selection?

  2. How does RASD fare compared to LMR for the same adaptation budget? Let us evaluate in terms of end-to-end processing time and maximum memory usage, what is the cost of the extra probe and student branch.

  3. What causes the differences to seemingly contradict each other (Table 5 showing 72.85/40.66 for gated selfdistillation and Table G showing 64.34/38.75 with learnable filling vector with seemingly the same aims)? Please specify if I missed configuration difference between these experiments.

1 comment

Dear Author, is there anything you'd like to discuss? Please note that the discussion period is coming to an end. We highly value your explanations and comments.

Reviewer_G7LDOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a test-time adaptation method (RASD) for vision-language models such as CLIP. RASD estimates prediction reliability by perturbing internal features and checking if resulting predictions remain consistent. It combines these predictions into a consensus that favours classes with strong, stable support. The consensus selects reliable teachers and refines their predictions, while masked self-distillation encourages consistency under stronger feature perturbations. A batch-level entropy regularizer helps prevent predictions from collapsing to a few classes.

Experiments cover image corruptions, natural ImageNet shifts, ten recognition datasets, and four domain benchmarks. RASD achieves the best reported average accuracy among the evaluated methods in each benchmark group, with particularly strong gains on CIFAR-10-C and ImageNet-C.

Critical strengths
  1. The motivation is clear and relevant. Incorrect predictions can be amplified during test-time adaptation. Estimating the teacher's reliability using responses to internal perturbation is a reasonable way to mitigate this risk.

  2. The evaluation covers a broad range of settings, including different corruptions, natural shifts, recognition datasets, and multiple domains. Moreover, experiments using the ViT-L/14 show that the benefits are not limited to a specific backbone. The paper also includes extensive component analyses and implementation details.

  3. The proposed method outperforms compared approaches in most settings. Results on corruption benchmarks are particularly strong.

  4. The reliability analysis provides useful supporting evidence. In particular, Table C shows that predictions agreeing with the internal consensus are substantially more accurate than disagreeing predictions across several datasets.

Critical weaknesses
  1. Although RASD differs in its overall design, perturbing internal features for test-time adaptation has already been explored in “FATA: Feature Augmentation Based Test-Time Adaptation,” which is neither discussed nor evaluated in the paper. A clearer comparison is needed to establish what RASD’s masking and reliability-aware consensus mechanisms add beyond existing feature-augmentation-based approaches.

  2. The extensive dataset-specific configuration makes generalization difficult to assess: Table A changes learning rates, masking strengths, student-view counts, consensus penalties, and loss weights across datasets. The paper does not explain how these settings were selected without target-label access. Meanwhile, several baselines use transferred configurations on unsupported benchmarks. The reported averages therefore do not clearly establish how RASD performs under a comparable tuning budget or with one transferable configuration.

  3. The computational cost is not measured. RASD uses sixteen consensus probes, up to sixteen masked students, and ten adaptation steps per batch. Reusing earlier visual features reduces the cost of additional views, but the method still performs repeated tail computations and backpropagation. Runtime, throughput, peak memory, and comparisons at matched computational budgets are needed to assess the practical value of gains.

  4. The comparison omits important related methods, particularly CLIPTTA which reports higher accuracies on some overlapping datasets: 80.3% on EuroSAT, 80.1% on ImageNet-R, and 50.8% on ImageNet-Sketch, compared with RASD’s 78.78%, 79.46%, and 49.95%, respectively.

Questions for authors
  1. What distinguishes RASD from FATA beyond the choice of feature perturbation? Can you compare the methods under the same adaptation protocol and evaluate whether the proposed masking and reliability-aware consensus improve over FATA-style feature augmentation with comparable regularization?

  2. How were the dataset-specific settings in Table A selected? Were separate validation data used? How does RASD perform with a single configuration transferred across datasets, and were competing methods given comparable tuning opportunities?

  3. What is the actual adaptation cost? Please report runtime, throughput, and peak memory, along with comparisons against strong baselines at matched adaptation-step or computational budgets.

  4. Why is CLIPTTA excluded despite being cited and reporting stronger results on some overlapping datasets? Can you evaluate it under the same backbone, pre-processing, and online prediction protocol?

Reviewer_rggTOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the unreliability of teacher predictions used in masked self-distillation in vision-language models. The authors find that prediction consistency under mild internal perturbations can serve as a proxy for teacher reliability. They construct a consensus distribution from weakly masked probes and select full-view predictions that agree with the consensus and satisfy a reliability threshold. Their objectives align the full-view prediction with the consensus probability distribution and use the selected full-view prediction to supervise strongly masked student views. A batch-level entropy regularization term is introduced to mitigate prediction collapse during adaptation.

Critical strengths
  • The paper addresses an important problem in masked self-distillation: unreliable teacher predictions can reinforce errors during test-time adaptation. Using consistency across weakly perturbed probes to select reliable teachers is an interesting approach that helps reduce prediction drift without requiring labeled test data.

  • The experimental results show improvements over recent methods across multiple benchmarks. In particular, the improvements on CIFAR-10-C (+4.06%) and ImageNet-C (+3.95%) over the strongest baseline are notable.

Critical weaknesses

Weaknesses

  • The central assumption that consistency under mild internal perturbations indicates teacher reliability is not sufficiently justified.

  • Most experiments are conducted using CLIP ViT-B/16 and masked self-distillation. It is unclear whether the proposed reliability measure generalizes to other vision-language models (e.g., SigLIP [1]), self-distillation approaches (e.g., EMA-based teachers [2]), or modalities (e.g., audio).

  • The experiments use only a single random seed. On some benchmarks, the improvements over the strongest baseline are marginal (e.g., +0.40% on CIFAR-100-C). Without reporting variance across multiple runs, it is difficult to determine whether these improvements are statistically meaningful or due to random variation.

  • Important ablation results are placed in the appendix. The paper could improve its presentation by including these results in the main text, particularly the analysis of internal consistency and prediction reliability, and the results on larger backbones.

  • The paper does not explicitly discuss the limitations of the proposed method or scenarios where it may fail. Furthermore, while the experiments consider online test-time adaptation, they do not evaluate continual test-time adaptation across changing domains without model resets.

  • It is unclear why the authors encourage high batch-level entropy through LbmeL_{\mathrm{bme}}. For instance, if most samples in a test batch belong to the same class, this objective may penalize correct predictions and potentially hurt adaptation.

References:

[1] Zhai, X., et al. Sigmoid Loss for Language Image Pre-Training. (ICCV 2023).

[2] Tarvainen, A., and Valpola, H. Mean Teachers Are Better Role Models: Weight-Averaged Consistency Targets Improve Semi-Supervised Deep Learning Results. (NeurIPS 2017).

Questions for authors
  1. Why do the authors choose to mask features before the final Transformer block? Would alternative perturbation strategies, such as image-level augmentations (e.g., cropping) or masking intermediate-layer features, provide similar benefits?

  2. Why is the proposed vulnerability-aware consensus preferred over simpler alternatives? It is unclear whether combining log-odds with a mean-minus-standard-deviation penalty provides advantages over averaging probe predictions or using an agreement score.

  3. Why not use the consensus prediction from weakly perturbed probes directly as the final prediction, for example, through majority voting? Since the consensus is considered more reliable than the original prediction, it is unclear what additional benefit masked self-distillation provides over directly using the consensus.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.