- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper considers test-time adaptation (TTA) based on two observations: deeper representations provide more accurate predictions, while intermediate representations are more robust to noisy supervision. Based on these observations, the authors propose Intrinsic Self-Distillation (ISD), which consists of Gradual Feature Distillation (GFD) and Debiased Logit Distillation (DLD). GFD progressively distills predictions from deeper to shallower representations, while DLD performs debiased logit distillation using a feature-level bias that is updated via an exponential moving average. ISD updates only affine parameters and does not require additional networks or input augmentation. Experiments across several TTA settings and backbones show competitive accuracy, strong stability, and low computational overhead.
Critical strengths
-
The proposed ISD method is simple and computationally efficient. It uses intrinsic intermediate representations for self-distillation without requiring data augmentation or auxiliary modules, and requires only a single forward pass. This makes the approach practically attractive compared with more computationally expensive TTA methods.
-
The paper provides broad experimental evaluations across different TTA settings and architectures. ISD is evaluated under standard, continual, label-shift, and batch-size-one TTA settings across ResNet50-GN, ViT-Base, and ConvNeXt-Tiny, and generally achieves competitive or improved accuracy and calibration.
Critical weaknesses
-
The benefit of gradual feature distillation is not fully demonstrated. As shown in Figure 5, Shallow ISD, which applies GFD to a single layer, performs close to the full progressive two-layer ISD in several settings. This makes the additional benefit of progressive distillation across multiple intermediate layers unclear.
-
The computational analysis is limited. Although ISD requires only a single forward pass and reports latency close to Tent, GFD applies the shared classifier to multiple intermediate representations and requires additional loss computation and backpropagation. The paper does not report FLOPs, memory usage, or how the computational cost changes with the number of intermediate layers.
Questions for authors
-
Could the authors provide a systematic analysis of the number of intermediate layers used for GFD? The current ablation compares only single-layer Shallow ISD with progressive two-layer ISD. Comparisons among single-layer, progressive two-layer, and progressive three-layer GFD, together with their accuracy, latency, and memory usage, would help clarify whether the benefit increases with additional intermediate layers and whether the current two-layer design is optimal.
-
Could the authors provide additional computational analysis for Shallow ISD and full ISD? If the authors included the number of FLOPs, peak memory usage, and computational cost of Shallow ISD and full ISD, that would help clarify the efficiency trade-off introduced by progressive GFD.
5 comments
Dear Reviewer YwV2,
Thank you again for your thoughtful feedback. We have carefully responded to your comments and updated our manuscript.
As the discussion period is coming to a close, we would appreciate it if you could let us know whether our responses have addressed your concerns. We would also be grateful if you could reconsider your current rating based on our responses and additional results.
We would be happy to address any remaining questions or concerns, and we will do our best to respond within the remaining discussion period.
Thank you for the additional experiments. The computational analysis addresses my efficiency concern and shows that progressive GFD introduces little additional overhead. However, the benefit of progressive GFD is still not fully clear. The new depth analysis appears to focus on Standard-TTA with ViT-Base, while Figure 5 shows that full ISD performs similarly to Shallow ISD in several continual and realistic TTA settings. Therefore, I will keep my current rating.
We thank the reviewer for the constructive comments. We have carefully revised the manuscript to address each of the points raised.
[W1/Q1] Benefit of Gradual Feature Distillation
To evaluate the effect of distillation depth, we compare single-layer, two-layer, three-layer, and four-layer configurations of ISD. As shown in Table 1, accuracy improves with increasing distillation depth on ViT-Base, with the largest gain occurring when moving from single-layer to two-layer distillation, followed by marginal improvements with additional layers. On ConvNeXt-Tiny, two-layer distillation also improves accuracy from 49.19% to 51.91%.
Based on these results, we adopt two-layer GFD as the default to ensure a consistent distillation strategy across architectures, since the final stages of ResNet50-GN and ConvNeXt-Tiny contain three blocks, allowing two progressive distillation steps.
Table 1. Average accuracy across different distillation depths.
[W2/Q2] Computational Efficiency
We provide a computational analysis of Shallow ISD and progressive two-layer and three-layer ISD in Table 2, reporting latency, backward cost, throughput, peak GPU memory, and GFLOPs. The results show that increasing the distillation depth improves adaptation accuracy with only modest additional computational overhead.
Table 2. Average accuracy and computational efficiency across different distillation depths.