GoTTA be Diverse: Rethinking Memory Policies for Test-Time Adaptation

Test-time adaptation (TTA) adapts pretrained models online to unlabeled test streams under distribution shift. Under temporally correlated and label-skewed streams, memory buffers determine which samples are repeatedly used for adaptation, yet memory selection is often coupled with a specific TTA method, making its contribution difficult to isolate. We decouple memory selection from the adaptation objective and benchmark six memory policies across eleven TTA methods under temporally correlated streams, covering both episodic and continual adaptation. Our analysis shows that memory selection can substantially affect TTA performance and that its impact depends on the underlying adaptation mechanism: pseudo-labeling methods can be sensitive to redundant samples, while normalization and RoTTA-based methods exhibit different responses to diversity-aware retention. We identify within-class redundancy as a complementary limitation of class-aware memory selection. Although class-balanced policies regulate how memory capacity is distributed across pseudo-classes, they do not explicitly prevent multiple slots from containing highly similar samples. Motivated by this observation, we introduce Guided Observational Test-Time Adaptation (GOTTA), a memory policy that combines class-aware allocation with within-class diversity filtering. GOTTA measures diversity in softmax predictive distribution space while retaining the existing retention-utility mechanism and leaving the underlying adaptation objective unchanged. Across our experiments, diversity-aware memory selection improves a broad range of TTA methods under temporally correlated streams, with the magnitude of the benefit depending on the adaptation method and available memory capacity. Experiments on CIFAR-10-C demonstrate the importance of memory composition under constrained budgets, while evaluations on ImageNet-C and ITD examine its behavior across datasets and stream settings. Overall, our study identifies memory selection as a distinct design axis of TTA and shows that how samples are retained can substantially shape online adaptation. These findings motivate treating memory policy as a first-class design choice alongside the adaptation objective.