Where to Adapt Matters: Online Depth-wise Routing for Test-Time Adaptation

Test-Time Adaptation (TTA) adapts a deployed model during online inference to mitigate the impact of domain shift. Existing methods concentrate on designing what to optimize and how to regularize the update, yet where to update is fixed in advance: they adapt a parameter subset chosen before deployment. We show that no single fixed choice covers diverse test conditions well: the preferred adaptation depth varies across settings, for example with the test batch size, where normalization groups at different depths take turns being optimal. Based on this insight, we propose Adaptive Group-wise Online Experts (AGOE), which turns where to update into an online decision. AGOE partitions normalization parameters into depth-wise experts and reallocates a fixed update budget among them at each step via per-depth learning-rate scaling, guided by a label-free reliability signal and a Hedge-style online rule. Routing preserves the base TTA objective, adds no extra backward pass, and acts as a plug-in complement to existing adapters. Experiments show that AGOE consistently improves its base adapters, especially in regimes where fixed-depth adaptation degrades severely, such as small-batch adaptation.