acceptodds
Withdrawn from ICLR 2027

Where to Adapt Matters: Online Depth-wise Routing for Test-Time Adaptation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Test-Time Adaptation (TTA) adapts a deployed model during online inference to mitigate the impact of domain shift. Existing methods concentrate on designing what to optimize and how to regularize the update, yet where to update is fixed in advance: they adapt a parameter subset chosen before deployment. We show that no single fixed choice covers diverse test conditions well: the preferred adaptation depth varies across settings, for example with the test batch size, where normalization groups at different depths take turns being optimal. Based on this insight, we propose Adaptive Group-wise Online Experts (AGOE), which turns where to update into an online decision. AGOE partitions normalization parameters into depth-wise experts and reallocates a fixed update budget among them at each step via per-depth learning-rate scaling, guided by a label-free reliability signal and a Hedge-style online rule. Routing preserves the base TTA objective, adds no extra backward pass, and acts as a plug-in complement to existing adapters. Experiments show that AGOE consistently improves its base adapters, especially in regimes where fixed-depth adaptation degrades severely, such as small-batch adaptation.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Test-Time Adaptation, Domain Shift, Online Learning, Adaptive Routing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_oqT2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies where to update a model during online test-time adaptation. The proposed AGOE partitions trainable normalization parameters into depth-wise groups and allocates a fixed sum of learning rates among them. It compares each group's proposed update direction with its recent parameter trajectory, then uses a Hedge rule to adjust the allocation for the next batch. AGOE preserves the base adaptation objective and requires one backward pass per batch. Experiments with Tent, DeYO, and ZeroSiam report gains across several batch-size regimes and distribution shifts, particularly where all-layer adaptation is unstable.

Critical strengths
  1. The paper provides useful evidence that suitable adaptation depths vary across batch-size regimes.
  2. The method offers a lightweight plug-in for several base adapters.
Critical weaknesses
  1. The distinction from closely related work is not sufficiently established. GALA [1] is repeatedly described as one-time selection, although the original method selects layers dynamically. A direct comparison with LAW [2], which also studies layer-wise learning-rate adaptation, is also missing. Meanwhile, the idea of adapting learning rates for TTA has also been widely explored [3][4], but the paper fails to properly discuss them.
  2. The propose method builds on the assumption that more consistent update directions should be encouraged. However, in TTA directional consistency does not establish update usefulness. The accumulated update direction might be noisy due to the unsupervised nature of TTA. Potential beneficial updates that can correct the update direction might be penalized by the method’s design. Existing results do not directly show how the signal distinguishes these potentially helpful cases during distribution transitions.

References:
[1] Sahoo, Sabyasachi, et al. "A layer selection approach to test time adaptation." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 19. 2025.
[2] Park, Junyoung, et al. "Layer-wise auto-weighting for non-stationary test-time adaptation." 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE, 2024.
[3] Lee, Jae-Hong. "Bayesian weight enhancement with steady-state adaptation for test-time adaptation in dynamic environments." Forty-second International Conference on Machine Learning. 2025.
[4] Shaoran, Lv, Xinyao Li, and Jingjing Li. "CONGA: Confidence-and-Gradient-Aware Learning Rate Schedule for Test Time Adaptation." Forty-third International Conference on Machine Learning.

Questions for authors
  1. Please clarify the description of GALA and confirm whether its implementation in Tables 2, 3, and 6 follows the original dynamic layer-selection procedure?
  2. Could the authors explain the methodological differences from LAW and compare the approaches under matched backbone, base objective, stream, and tuning conditions to establish AGOE's additional contribution?
  3. Eqs. (1) and (3) keep the total learning-rate scale fixed. Could this limit the update efficiency? Could the authors compare the such LR-fixed setting method with settings that allow total LR to change?
  4. Could the authors analyze whether the proposed mechanism can distinguish beneficial direction changes from noisy updates, and identify cases where consistently aligned updates reinforce errors?
Reviewer_aZfPOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper claims that prior work in TTA does not consider that adaptively selecting components, and it is crucial for TTA robustness across test-time conditions. Based on the observation experiments that the optimal layer to adapt varies with conditions such as test-time batch size, the author proposed a method that adaptively adjust the depth-wise learning rate, which could be integrated with some existing baselines.

Critical strengths

The paper have a clear motivation and research focus on stabilizing TTA by adaptively adjusting the components to update.

Critical weaknesses

The paper have significant hallucinations regarding its closest related work GALA [1] . For example, in line 47 it introduces GALA as the closest line of work in layer selection but "commits to a fixed subset once". This is wrong, since the method GALA proposed is a TTA method that adaptively select layers for each example in stream, and also uses a reset window to periodically reset the anchor (see Figure 2 of the paper). The mistakes of mischaracterizing GALA as static TTA method have been made repeatedly through out the paper (line. 47–48, 232–234, 668–669, 430–431). Since this is the main stated distinction in the paper, it significantly undermines the claimed contribution. Furthermore, the reference "Minsoo Kim et al. Buffer layers for test-time adaptation. arXiv:2601.00000, 2026" is also fabricated. The correct paper is "Buffer layers for Test-Time Adaptation" from "Hyeongyu Kim, Geonhui Han, Dosik Hwang" with arxiv number 2510.21271 [2]. Lastly, the proposed adaptive method and the observation of noisy gradients have strong similarities to prior work [1, 3, 4]. It is very difficult to access the novelty and contribution of this paper with little to no effort put in explaining the differences.

[1] Sahoo et al., "A Layer Selection Approach to Test Time Adaptation." AAAI 2025.
[2] Kim et al., "Buffer layers for Test-Time Adaptation." NeurIPS 2025.
[3] Niu et al., "Towards Stable Test-Time Adaptation in Dynamic Wild World." ICLR 2023.
[4] Park et al., "Layer-wise Auto-Weighting for Non-Stationary Test-Time Adaptation." WACV 2024.

Questions for authors

See weaknesses.

Reviewer_smpuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces AGOE, which dynamically allocates learning rates across depth-wise groups of normalization parameters during test-time adaptation. Combining update–trajectory alignment with a Hedge weighting rule to reduce potentially unstable updates. Experiments on CIFAR-10-C, CIFAR-100-C, and ImageNet-C report improvements over several base adaptation methods, particularly under small-batch conditions.

Critical strengths

The paper addresses a practical TTA problem with a lightweight method compatible with multiple adaptation objectives and evaluated across several challenging settings.

Critical weaknesses
  1. The methodological novelty over prior alignment-based adaptation is insufficiently established. Equation (1) implements group-wise learning-rate scaling, while GALA already uses update–trajectory alignment for online layer selection. Soft Hedge allocation and EMA anchors are identifiable differences, but their contribution beyond existing mechanisms needs to be established.

  2. The experiments do not adequately isolate the benefit of continuously changing the allocation. In Figure 5, the best dynamic configuration achieves 33.09%, versus 32.45% for the learning-rate-swept uniform control (η=0). In Table 1, AGOE-DeYO averages 74.06%, versus 73.32% for fixed middle-depth adaptation. These results support some benefit, but do not establish superiority over tuned fixed nonuniform learning rates or initially estimated, subsequently frozen weights. These controls require a disclosed, matched tuning protocol.

  3. The GALA comparison relies on an incorrect characterization, and its implementation fidelity remains unclear. Section 3.2 describes GALA as one-time scoring followed by fixed selection, whereas the original method reselects layers online and periodically resets its anchor (GALA, Figure 2 and Algorithm 1). The description error is confirmed; an implementation error is not. The authors should clarify whether their baseline preserves these mechanisms and report its loss, trainable parameters, reset settings, and learning-rate search.

Questions for authors
  1. What is the substantive advance over GALA’s online alignment-based selection, and which ablations isolate the contributions of Hedge allocation and EMA anchors?

  2. Do the gains persist against tuned fixed nonuniform learning rates and initially estimated, then frozen allocations under the same tuning budget and selection protocol?

  3. Was GALA implemented with online layer reselection and periodic anchor resets, despite Section 3.2 describing one-time selection? Please provide its exact configuration and learning-rate search.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.