acceptodds
Under review as a conference paper at ICLR 2027

Intrinsic Self-Distillation for Test-Time Adaptation

Abstract

Test-time adaptation has emerged as a promising approach for addressing distribution shifts in target domains by improving predictions during inference on online data streams. For decades, the utilization of unlabeled data has been widely explored in semi-supervised and self-supervised learning, and provides a blueprint for test-time adaptation. Building on pioneering insights, recent solutions achieve notable improvements with data augmentation or auxiliary modules. However, such approaches demand additional computational complexity and reduce practical applicability. In this paper, we introduce intrinsic self-distillation for adapting to the target domain using intermediate representations, without augmentation or auxiliary modules. Our method leverages intermediate representations as intrinsic learning signals, motivated by two empirical observations: first, the outputs of deeper layers exhibit progressively higher accuracy. Second, predictions from intermediate representations are more robust to noise. We demonstrate the efficiency and effectiveness of our approach across various architectures and benchmarks.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
test-time adaptation, self-distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_YwV2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper considers test-time adaptation (TTA) based on two observations: deeper representations provide more accurate predictions, while intermediate representations are more robust to noisy supervision. Based on these observations, the authors propose Intrinsic Self-Distillation (ISD), which consists of Gradual Feature Distillation (GFD) and Debiased Logit Distillation (DLD). GFD progressively distills predictions from deeper to shallower representations, while DLD performs debiased logit distillation using a feature-level bias that is updated via an exponential moving average. ISD updates only affine parameters and does not require additional networks or input augmentation. Experiments across several TTA settings and backbones show competitive accuracy, strong stability, and low computational overhead.

Critical strengths
  1. The proposed ISD method is simple and computationally efficient. It uses intrinsic intermediate representations for self-distillation without requiring data augmentation or auxiliary modules, and requires only a single forward pass. This makes the approach practically attractive compared with more computationally expensive TTA methods.

  2. The paper provides broad experimental evaluations across different TTA settings and architectures. ISD is evaluated under standard, continual, label-shift, and batch-size-one TTA settings across ResNet50-GN, ViT-Base, and ConvNeXt-Tiny, and generally achieves competitive or improved accuracy and calibration.

Critical weaknesses
  1. The benefit of gradual feature distillation is not fully demonstrated. As shown in Figure 5, Shallow ISD, which applies GFD to a single layer, performs close to the full progressive two-layer ISD in several settings. This makes the additional benefit of progressive distillation across multiple intermediate layers unclear.

  2. The computational analysis is limited. Although ISD requires only a single forward pass and reports latency close to Tent, GFD applies the shared classifier to multiple intermediate representations and requires additional loss computation and backpropagation. The paper does not report FLOPs, memory usage, or how the computational cost changes with the number of intermediate layers.

Questions for authors
  1. Could the authors provide a systematic analysis of the number of intermediate layers used for GFD? The current ablation compares only single-layer Shallow ISD with progressive two-layer ISD. Comparisons among single-layer, progressive two-layer, and progressive three-layer GFD, together with their accuracy, latency, and memory usage, would help clarify whether the benefit increases with additional intermediate layers and whether the current two-layer design is optimal.

  2. Could the authors provide additional computational analysis for Shallow ISD and full ISD? If the authors included the number of FLOPs, peak memory usage, and computational cost of Shallow ISD and full ISD, that would help clarify the efficiency trade-off introduced by progressive GFD.

5 comments

We thank the reviewer for the constructive comments. We have carefully revised the manuscript to address each of the points raised.

[W1/Q1] Benefit of Gradual Feature Distillation

To evaluate the effect of distillation depth, we compare single-layer, two-layer, three-layer, and four-layer configurations of ISD. As shown in Table 1, accuracy improves with increasing distillation depth on ViT-Base, with the largest gain occurring when moving from single-layer to two-layer distillation, followed by marginal improvements with additional layers. On ConvNeXt-Tiny, two-layer distillation also improves accuracy from 49.19% to 51.91%.

Based on these results, we adopt two-layer GFD as the default to ensure a consistent distillation strategy across architectures, since the final stages of ResNet50-GN and ConvNeXt-Tiny contain three blocks, allowing two progressive distillation steps.

Table 1. Average accuracy across different distillation depths.

Distillation DepthsSingle LayerTwo LayerThree LayerFour Layer
ViT-Base69.6869.9970.0770.11
ConvNeXt-Tiny49.1951.91--

[W2/Q2] Computational Efficiency

We provide a computational analysis of Shallow ISD and progressive two-layer and three-layer ISD in Table 2, reporting latency, backward cost, throughput, peak GPU memory, and GFLOPs. The results show that increasing the distillation depth improves adaptation accuracy with only modest additional computational overhead.

Table 2. Average accuracy and computational efficiency across different distillation depths.

MethodLatency (ms/batch)Backward (ms/batch)Throughput (images/s)Peak Memory (GB)GFLOPsAcc. (%)
Tent296.1112.40215.935.3772.17060.37
SPA833.93128.4976.6718.63179.70669.95
ZeroSiam299.0013.29213.845.4572.86967.59
ISD (Shallow single-layer)297.1713.47215.165.5572.17669.68
ISD (Gradual two-layer)300.5214.39212.765.5972.17969.99
ISD (Gradual three-layer)302.8615.42211.125.6272.18270.07

Dear Reviewer YwV2,

Thank you again for your thoughtful feedback. We have carefully responded to your comments and updated our manuscript.

As the discussion period is coming to a close, we would appreciate it if you could let us know whether our responses have addressed your concerns. We would also be grateful if you could reconsider your current rating based on our responses and additional results.

We would be happy to address any remaining questions or concerns, and we will do our best to respond within the remaining discussion period.

Thank you for the additional experiments. The computational analysis addresses my efficiency concern and shows that progressive GFD introduces little additional overhead. However, the benefit of progressive GFD is still not fully clear. The new depth analysis appears to focus on Standard-TTA with ViT-Base, while Figure 5 shows that full ISD performs similarly to Shallow ISD in several continual and realistic TTA settings. Therefore, I will keep my current rating.

Reviewer_NGAcOpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Intrinsic Self-Distillation (ISD), a test-time adaptation method that uses intermediate representations within the same network as intrinsic learning signals. ISD combines gradual feature distillation (GFD), which transfers supervision from deeper to shallower layers, with debiased logit distillation (DLD), which reduces prediction bias and helps prevent model collapse. The method requires neither data augmentation nor auxiliary teacher networks, and updates only the affine parameters of the feature extractor. Experiments on ImageNet-C and several natural domain-shift benchmarks cover ResNet-50, ViT-Base, ConvNeXt-Tiny, MobileNetV3, and MLP-Mixer under standard, continual, label-shift, and batch-size-1 TTA settings. ISD generally improves accuracy and calibration while maintaining a single-model adaptation pipeline.

Critical strengths
  1. The method is conceptually simple and easy to integrate into existing TTA pipelines. It reuses intermediate features from the adapted model and does not require an additional teacher network, augmentation branch, or auxiliary classifier.

  2. The separation between GFD and DLD is well motivated. GFD provides a more structured learning signal than direct entropy minimization, while DLD addresses feature-level bias and reduces the risk of collapse. The component ablations support the complementary roles of the two losses.

  3. The experimental evaluation is broad. The paper considers CNNs, ViTs, ConvNeXt, MobileNet, and MLP-Mixer, and evaluates both accuracy and expected calibration error under standard, continual, realistic, and mixed-domain TTA settings.

  4. The method achieves strong results on ImageNet-C. In several settings, ISD improves both accuracy and calibration over competitive baselines, while remaining close to single-forward-pass methods in the reported latency measurements.

Critical weaknesses
  1. The efficiency claim is only partially supported. ISD avoids an additional teacher or augmentation branch, but it still stores multiple intermediate representations, computes several adjusted logits, updates an exponential moving average, and performs backpropagation through the affine parameters. Table 6 reports latency for one ImageNet-C setting, but does not provide peak memory, backward cost, throughput, or end-to-end FLOPs across batch sizes.

  2. The method has a nontrivial architectural constraint. The selected intermediate features must have the same dimensionality as the final representation so that they can share the classifier. This requires architecture-specific layer selection and prevents direct application to models whose intermediate representations have incompatible dimensions. The claim that ISD applies to “most modern DNNs” should therefore be qualified.

  3. The central idea of using a network’s own intermediate representations as supervision is not new. For example, Learning CNNs by Attention Distillation performs top-down and layer-wise distillation of attention maps within a single network, using deeper-layer attention as supervision for shallower layers [1], while related work has also explored feature- and prediction-level self-distillation [2, 3]. Therefore, the novelty of ISD does not come from self-distillation itself, but must be demonstrated through its specific formulation for fully test-time adaptation. The paper should clarify how GFD and DLD differ fundamentally from existing layer-wise self-distillation objectives and include controlled comparisons against generic intermediate-layer distillation under the same TTA protocol. Without such comparisons, the contribution may be viewed as an application and combination of established self-distillation techniques rather than a clearly new adaptation principle.

[1] Hou, Y. et al. Learning Lightweight Lane Detection CNNs by Self Attention Distillation. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019. [2] Zhang, L. et al. Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self-Distillation. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019. [3] Xu, TB. et al. Data-Distortion Guided Self-Distillation for Deep Neural Networks. Proceedings of the AAAI Conference on Artificial Intelligence, 2019.

Questions for authors

Refer to the Weaknesses

5 comments

We appreciate the reviewer's valuable feedback. We have added additional experiments and revised the manuscript accordingly.

[W1] Computational efficiency

We report latency, backward cost, throughput, peak GPU memory, and GFLOPs in Table 1 to provide a more comprehensive efficiency evaluation. As shown in the table, ISD achieves computational efficiency comparable to Tent and ZeroSiam while providing higher accuracy. Furthermore, increasing the distillation depth improves adaptation performance with a modest increase in computational cost. Additionally, Table 2 reports computational efficiency across different batch sizes, with GFLOPs measured per image.

Table 1. Computational efficiency comparison with existing TTA methods.

MethodLatency (ms/batch)Backward (ms/batch)Throughput (images/s)Peak Memory (GB)GFLOPsAcc. (%)
Tent296.1112.40215.935.3772.17060.37
SPA833.93128.4976.6718.63179.70669.95
ZeroSiam299.0013.29213.845.4572.86967.59
ISD (Shallow single-layer)297.1713.47215.165.5572.17669.68
ISD (Gradual two-layer)300.5214.39212.765.5972.17969.99
ISD (Gradual three-layer)302.8615.42211.125.6272.18270.07

Table 2. Computational efficiency comparison across different batch sizes.

Batch SizeLatency (ms/batch)Backward (ms/batch)Throughput (images/s)Peak Memory (GB)GFLOPs
32143.8214.21222.423.1372.179
64300.5214.39212.765.5972.179
128611.1514.36209.2410.5272.179

[W2] Architectural applicability

We acknowledge that ISD requires dimensionally compatible intermediate representations to share the pretrained classifier without introducing additional learnable modules. We have revised the Limitations section to explicitly clarify this architectural constraint and provide a more appropriate description of the applicability of ISD.

[W3] Novelty and comparison with conventional self-distillation

Conventional self-distillation methods are primarily designed for supervised representation learning. Consequently, they may not be well suited to the adaptation and stability requirements of TTA. Specifically, BYOT [1] introduces additional learnable modules that require separate initialization and optimization, while DDSD [2] requires additional forward passes to process multiple augmented views, increasing computational overhead during adaptation.

In contrast, GFD utilizes a shared pretrained classifier without introducing additional learnable modules and incorporates logit adjustment to reduce sensitivity to initialization, enabling efficient adaptation and improving adaptability. Moreover, DLD compensates for accumulated prediction biases to promote stable adaptation under distribution shifts. These designs distinguish ISD from conventional layer-wise self-distillation by explicitly addressing the challenges of adaptability, efficiency, and stability in TTA.

To assess the effectiveness of conventional self-distillation methods under TTA, we evaluate BYOT and DDSD under the same TTA protocol, replacing ground-truth supervision with pseudo-labels. As shown in Table 3, GFD achieves higher average accuracy than both methods. Furthermore, incorporating DLD into BYOT and DDSD improves their performance, suggesting that DLD can enhance adaptation stability across different self-distillation approaches. Combining GFD and DLD achieves the best average accuracy, demonstrating their effectiveness for TTA.

Table 3. Comparison with conventional self-distillation methods.

MethodAverage Accuracy (%)
DDSD49.42
BYOT54.80
GFD (Ours)60.43
DDSD w/ DLD64.11
BYOT w/ DLD66.15
GFD w/ DLD (Ours)69.99

[1] Zhang, L. et al. Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self-Distillation. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.

[2] Xu, TB. et al. Data-Distortion Guided Self-Distillation for Deep Neural Networks. Proceedings of the AAAI Conference on Artificial Intelligence, 2019.

Thanks to the authors for the response and the additional experiments. One clarification concerns the experiments on Office-Home and DomainNet. ImageNet-C primarily introduces changes in imaging conditions, such as noise, blur, and weather, whereas Office-Home and DomainNet involve larger appearance and representation shifts across photographs, paintings, and sketches. These benchmarks involve substantially different shifts from ImageNet-C, whereas the motivating layer-wise analysis is conducted on corrupted ImageNet-C. Could the authors clarify whether ISD is applied unchanged to these datasets, or whether any modifications are made to the intermediate-layer selection, GFD/DLD objectives, or adaptation procedure?

Thank you for your response and valuable comments. For practical applicability, we use the same ISD settings across all datasets without modifying the intermediate-layer selection, GFD/DLD objectives, or adaptation procedure.

Specifically, for Office-Home and DomainNet, we use λ\lambda=1, α\alpha=1, mm=0.95, and two-layer GFD, consistent with ImageNet-C. However, following the experimental protocol of PALSE, we search for the learning rate in the range of 1e-3 to 1e-6 for all methods on Office-Home and DomainNet and select the learning rate that achieves the best accuracy (Appendix A.2).

Moreover, we evaluated the robustness of intermediate representations to noisy supervision on DomainNet, which involves larger appearance and representation shifts. Table 1 presents the results under 95% and 50% noisy supervision. In both settings, adaptation with supervision applied to intermediate layers consistently achieves higher accuracy than adaptation with supervision applied to the final layer, demonstrating the robustness of intermediate representations to noisy supervision under natural domain shifts. Notably, under 50% noisy supervision on Clipart, applying supervision to the final layer reduces accuracy from 64.98% to 63.15%. In contrast, supervision applied to the last two intermediate layers improves accuracy to 65.76% and 65.33%, respectively, despite the supervision signals having lower accuracy than the initial predictions.

Table 1. Average accuracy (%) under noisy supervision on DomainNet.

Supervision LayerClipartInfographPaintingQuickdrawRealSketchAvg.
No Adaptation64.9815.2650.4511.8563.7154.0043.38
95% Noisy Adaptation
Last Layer55.789.9636.689.2956.2435.6333.93
Last - 1 Layer60.7710.2842.9612.0158.8943.5938.08
Last - 2 Layer63.9513.4147.1511.1561.7250.4341.30
50% Noisy Adaptation
Last Layer63.1516.1647.7413.1661.7649.8641.97
Last - 1 Layer65.3318.7951.4814.7263.4254.2044.66
Last - 2 Layer65.7618.8752.2614.8164.5355.5645.30
Reviewer_7fSoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes intrinsic self-distillation (ISD) for test-time adaptation. Motivated by the observed differences in noise sensitivity and predictive accuracy across network depths, ISD distills predictions from deeper to shallower layers using a shared classifier. It combines this objective with debiased logit distillation to improve adaptation stability, without data augmentation or auxiliary trainable modules.

Critical strengths
  1. The investigation of intermediate representations under noisy supervision is interesting in the TTA setting.
  2. The empirical evaluation is extensive, covering multiple architectures, adaptation protocols, and distribution shifts. The reported results suggest a promising balance between adaptation performance and computational efficiency.
Critical weaknesses

Although Section 3.2 provides empirical motivation, the mechanism underlying Figure 3 remains insufficiently established. The experiment shows that applying noisy supervision at intermediate layers better preserves final prediction accuracy, but it is unclear whether this reflects greater robustness of the representations or differences in the parameters and gradient paths involved in adaptation. This limits the explanatory strength and generality of the paper’s central motivation.

Questions for authors

Could the authors clarify how the incorrect labels in Figure 3 are generated and which parameters are updated for each loss location? Can you provide a controlled analysis distinguishing representation robustness from differences in optimization, and explain how this supports the effectiveness of deeper-to-shallower distillation?

3 comments

We appreciate the reviewer's insightful feedback. We have conducted additional experiments and updated the manuscript to address the concerns raised.

[W1/Q1] Noisy label generation and controlled analysis of updated parameters

In Figure 3, we randomly replace the labels of 95% of samples in each batch with uniformly sampled incorrect classes, excluding the ground-truth class. Although all ViT LayerNorm (LN) parameters are initially set as trainable, supervision at intermediate layers does not propagate gradients to subsequent layers, resulting in different sets of updated parameters depending on the supervision layer.

To distinguish the robustness of intermediate representations from differences in updated parameters, we conduct a controlled analysis by progressively freezing the LN parameters in the final ViT blocks. In particular, freezing the LNs in blocks 10–12 and the final LN ensures identical sets of updated parameters across different supervision layers. As shown in Table 1, intermediate supervision consistently preserves higher final prediction accuracy than supervision at the final layer, even when the same set of parameters is updated. These results suggest that the robustness of intermediate representations to noisy supervision is not solely attributable to differences in updated parameters.

Table 1. Average accuracy (%) under 95% noisy supervision.

Supervision LayerFrozen LN: 10–12Frozen LN: 11–12Frozen LN: 12All LN Trainable
10th58.1559.4159.4159.51
11th58.5959.7560.2460.19
12th28.7729.9431.7545.26

[Q2] Effectiveness of deeper-to-shallower distillation

Based on the observations, we considered which supervision signals should be provided to which layers for effective TTA. Deeper representations exhibit higher predictive accuracy, making them suitable sources of supervision, while shallower representations are less sensitive to noisy supervision, making them suitable targets for adaptation. Considering the distinct characteristics of deeper and shallower representations, we design GFD to progressively transfer supervision from deeper to shallower layers, leveraging their respective strengths for effective adaptation under distribution shifts.

Dear Reviewer 7fSo,

Thank you again for your thoughtful feedback and for your positive assessment of our work. We have carefully responded to your comments and updated our manuscript.

As the discussion period is coming to a close, we would appreciate it if you could let us know whether our responses have addressed your remaining concerns. We would be happy to provide any further clarification if needed.

Thank you for clarifying the noisy-label generation procedure and providing the controlled experiment. These clarifications alleviate my original concern, and I will maintain my original positive assessment.

Comments on the paper

1 comment

Thank your for your constructive comments and for taking the time to review our paper. We have carefully revised the manuscript to address the reviewers' comments. All changed in the revised manuscript (revised PDF) are highlighted in blue for clarity.

  1. Noisy supervision analysis: Clarification of the noisy-label generation process and additional controlled experiments investigating the effect of updated parameter sets (Appendix A.3, Table 8).

  2. Computational efficiency: Expanded analysis of backward cost, throughput, peak GPU memory, and GFLOPs across existing TTA methods (Section 5.3, Table 6).

  3. Deep and shallow ISD: Additional analysis of the trade-off between accuracy and computational efficiency across different distillation depths, along with an explanation of the selection of the default two-layer configuration. (Section 5.3, Table 6).

  4. Comparison with self-distillation methods: Additional experiments with BYOT and DDSD under the TTA protocol and further discussion on the novelty of ISD. (Section 5.4, Table 7).

  5. Limitations: Clarification of the architectural requirements and revision of the limitation statement to more accurately reflect the applicability of ISD (Section 6, Limitation).

If you have any further concerns or questions, we would be happy to address them and incorporate additional discussions in a subsequent revision.

Thank you again for your time and consideration.
Sincerely.
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.