acceptodds
Under review as a conference paper at ICLR 2027

GoTTA be Diverse: Rethinking Memory Policies for Test-Time Adaptation

Abstract

Test-time adaptation (TTA) adapts pretrained models online to unlabeled test streams under distribution shift. Under temporally correlated and label-skewed streams, memory buffers determine which samples are repeatedly used for adaptation, yet memory selection is often coupled with a specific TTA method, making its contribution difficult to isolate. We decouple memory selection from the adaptation objective and benchmark six memory policies across eleven TTA methods under temporally correlated streams, covering both episodic and continual adaptation. Our analysis shows that memory selection can substantially affect TTA performance and that its impact depends on the underlying adaptation mechanism: pseudo-labeling methods can be sensitive to redundant samples, while normalization and RoTTA-based methods exhibit different responses to diversity-aware retention. We identify within-class redundancy as a complementary limitation of class-aware memory selection. Although class-balanced policies regulate how memory capacity is distributed across pseudo-classes, they do not explicitly prevent multiple slots from containing highly similar samples. Motivated by this observation, we introduce Guided Observational Test-Time Adaptation (GOTTA), a memory policy that combines class-aware allocation with within-class diversity filtering. GOTTA measures diversity in softmax predictive distribution space while retaining the existing retention-utility mechanism and leaving the underlying adaptation objective unchanged. Across our experiments, diversity-aware memory selection improves a broad range of TTA methods under temporally correlated streams, with the magnitude of the benefit depending on the adaptation method and available memory capacity. Experiments on CIFAR-10-C demonstrate the importance of memory composition under constrained budgets, while evaluations on ImageNet-C and ITD examine its behavior across datasets and stream settings. Overall, our study identifies memory selection as a distinct design axis of TTA and shows that how samples are retained can substantially shape online adaptation. These findings motivate treating memory policy as a first-class design choice alongside the adaptation objective.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Test-time adaptation, memory, diversity, TTA benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Dx1sOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper revisits replay-memory policies for test-time adaptation under temporally correlated, label-skewed streams. It separates the memory policy from the adaptation objective and evaluates six policies (none, FIFO, reservoir, PBRS, CSTU, DAB-buffer, plus the proposed version) across eleven TTA methods. It argues that class-balanced policies leave within-class redundancy unaddressed, and proposes GOTTA: an incoming sample is admitted only if its softmax vector is more than ε (L2) away from every stored sample with the same pseudo-label, and when memory is full it may replace its nearest stored neighbor in the majority class, subject to the CSTU age/uncertainty score. Validation is on CIFAR-10-C (continual, M=32, three seeds), a 10-class ImageNet-C subset, and ITD.

Critical strengths
  • Decoupled evaluation of memory policies:

Evaluating each policy under many adaptation methods with shared streams is a sensible protocol, and the paper reports unfavorable cases rather than hiding them. The finding that the effect of memory depends strongly on the adaptation mechanism is worth having in the literature and is the main reason I see value in this work.

  • Simplicity:

The policy is a drop-in module with small overhead and no change to the adaptation objective, which would make it easy to adopt if the benefit were established.

Critical weaknesses
  • The evidence does not support the central claims.

    • Mechanism: In Figure 2, Norm, RoTTA and DAB peak at ε = 0 to 0.25 and degrade as ε grows. These are the columns where GoTTA leads in Table 1. At ε = 0 the filter admits essentially everything, so the gain over CSTU must come from another change, presumably the nearest-neighbor replacement rule, which is never ablated. At the other end, two softmax vectors with the same argmax cannot be far apart (about 0.95 at most for C=10 by my calculation), so ε ≥ 1.0 appears to freeze the memory at one sample per pseudo-class. The "redundancy-sensitive" methods peak exactly there, and their accuracy stays below the unadapted source model (56.49%). This suggests they benefit from adapting less, not from diversity. No direct measurement of memory content (redundancy, pseudo-label accuracy, confidence) is given.

    • Sensitivity and fairness: Averaged over ε (Table 6), the full method is below CSTU for Norm, NOTE and RoTTA. The ε used in each table is not stated, and the Table 1 numbers appear to correspond to different ε for different methods. If ε was selected per method on the evaluation stream, the comparison with untuned baselines is not fair.

    • Generality: On ITD, the benchmark closest to the motivating scenario, DAB-buffer is far ahead (34.44 / 25.38 / 43.91 vs 14.20 / 13.43 / 29.28). On full ImageNet-C GoTTA is not the best in any column, and at M = 8 PBRS is best for all three methods. ImageNet-C* accuracies are near chance for a 10-class task. The text discusses these cases mainly relative to no memory, and only Table 1 uses more than one seed.

  • The paper does not engage with the closest prior work, which narrows its novelty and leaves the benchmark incomplete. The two framing claims are that memory policies have not been studied independently of the adaptation method, and that within-class redundancy is an unaddressed limitation. Both need to be qualified.

    • Redundancy filtering: EATA, used here as a baseline, already excludes samples whose model output is too similar (cosine similarity against a threshold ε) to a moving average of previous outputs. It explicitly mentions comparing against stored outputs of earlier samples as the direct alternative. DAB carries this idea into a buffer. GOTTA's filter is close to that alternative applied per pseudo-class, yet the relation is not discussed. DANCE (Findings of ACL 2026) uses within-class nearest-neighbor novelty for cache admission in VLM test-time adaptation, with the same motivation of near-duplicate confident samples.

    • Memory selection as a module: SNAP (Neurips 2025) proposes a class- and domain-representative memory and evaluates it with five TTA methods. The multi-cluster memory of Tseng et al. (arXiv 2603.21135) is a plug-and-play memory for the same PTTA setting. The paper cited for ITD itself equips existing TTA methods with memory. SoTTA (Neurips 2023; high-confidence, class-balanced sampling), STAMP (cited but not benchmarked) and ResiTTA (ICASSP 2025) propose further selection rules. Several of these prefer confident or representative samples, the opposite of what a softmax-distance filter tends to admit, so they matter as baselines and not only as citations.

    • Coreset framing. The continual-learning literature on buffer selection (CBRS, Rainbow Memory, OCS) is not discussed, and Aljundi et al. (2019), a sample-selection method, is cited as parameter regularization.

A paper whose contribution is a systematic study of memory policies should cover the existing policies and state precisely what is new relative to them.

Questions for authors

Which ε is used in each table? Is it fixed or chosen per adaptation method, and on what data? Were hyperparameters of the baseline policies tuned in the same way?

At ε = 0, what differs between GoTTA and CSTU, and which difference explains the gain for Norm, RoTTA and DAB? Please ablate the admission filter and the nearest-neighbor replacement separately.

For ε ≥ 1.0, how many samples are admitted over the stream, and what is the memory occupancy over time? Is my reading correct that the memory is frozen after the first sample per pseudo-class?

What is the source-model accuracy on ImageNet-C* and ITD? Why are ImageNet-C* accuracies near chance when the 1000-class results are much higher?

Can you report, for each policy, the pseudo-label accuracy, mean confidence, and a redundancy measure of the stored samples?

What do the "DAB" column and the no-memory RoTTA row correspond to?

How does the admission filter differ in effect from EATA's redundancy criterion and from DAB's, beyond the choice of reference (nearest stored neighbor vs. moving average)? An experiment replacing one with the other inside the same buffer would clarify this.

Reviewer_MUhZOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper systematically investigates how memory-selection policies affect test-time adaptation under temporally correlated, non-i.i.d. streams. It identifies within-class redundancy as a limitation of existing class-balanced memories and proposes GOTTA, a plug-in memory policy that combines class-aware allocation with diversity filtering in predictive-distribution space. Experiments across multiple TTA methods and benchmarks examine its effectiveness under different memory capacities and stream settings.

Critical strengths

The paper provides a broad and systematic empirical study covering 11 TTA algorithms and seven memory configurations. This controlled evaluation helps isolate the impact of memory selection from the underlying adaptation objective.

Critical weaknesses
  1. Limited methodological novelty. The proposed method is technically simple, primarily adding a distance-based diversity filter within pseudo-class partitions on top of existing class-aware memory allocation and eviction mechanisms. This contribution appears incremental for an ICLR paper.

  2. Unclear advantage in large-class settings. GOTTA’s core mechanism is most meaningful when the memory can retain multiple samples per pseudo-class. Accordingly, its strongest results are obtained on a specially constructed 10-class ImageNet-C subset with more than six memory slots per class. On the full 1,000-class ImageNet-C benchmark, GOTTA remains functional and can improve over the no-memory baseline, but it is not consistently superior to existing memory policies. It is also substantially outperformed by DAB-buffer on the 21-class ITD benchmark. Therefore, the current results do not establish a clear practical advantage for GOTTA when the memory-to-class ratio becomes less favorable.

Questions for authors

When the number of classes substantially exceeds the memory capacity, as in the full ImageNet-C setting (C=1000C = 1000, M=64M = 64), each occupied pseudo-class can retain at most one sample. In this regime, the memory cannot meaningfully maintain intra-class diversity. What advantage is GOTTA expected to provide over standard class-balanced memory policies under this condition, and how do the full ImageNet-C results support this claimed advantage?

2 comments

Dear Reviewer MUhZ,

Thank you for your feedback. We would like to kindly follow up on our detailed rebuttal, in which we provide further experimental evidence and clarify the scientific contributions of our work.

We hope you will have the opportunity to revisit our responses. We believe our clarifications provide a more comprehensive perspective on our contributions and hope they will be helpful in reconsidering your initial assessment.

Reviewer_MUhZOpenReview ↗

Official Comment by Reviewer MUhZ

Thank you for the detailed rebuttal and additional experiments. The response substantially improves the clarity of the method and its empirical positioning, and I therefore raise my rating from 1 to 2.

Nevertheless, my main concerns remain. The principal results use method-specific thresholds selected on the same test streams, while the independently transferred ITD results do not establish an advantage over DAB-buffer. Moreover, under large-class, low-memory settings, the proposed within-class diversity mechanism has limited opportunity to operate, and the PL results primarily demonstrate the benefit of restrictive or frozen replay rather than the proposed diversity criterion. Overall, I recognize the value of the systematic empirical study, but I still find the methodological novelty and practical advantage insufficient for acceptance.

Reviewer_inCxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies memory selection separately from the test-time adaptation objective, comparing six policies across eleven methods under correlated streams. It also proposes GOTTA: a pseudo-class-aware buffer that filters nearby predictive distributions and selects an eligible replacement using age and uncertainty. The base adaptation objective stays unchanged. Experiments show substantial method-dependent effects of memory choice, including useful negative results. The proposed policy helps several constrained-memory settings but is not consistently the best policy on video or full-class ImageNet-C. The paper largely acknowledges this conditional scope.

Critical strengths
  • The matched policy/objective study identifies a meaningful experimental factor. GOTTA improves Norm over CSTU by 4.00 points and RoTTA by 6.04 points in this comparison. Table 4, p. 14, supplies three-seed deviations. Positive and negative outcomes show that memory choice can substantially change adaptation behavior; a contribution here need not make every configuration exceed frozen inference.

  • Threshold, capacity, and refresh results expose useful limits. Figure 2 and Table 3, shows different objectives favoring different diversity constraints and memory capacities. These findings support the paper's central claim that policy effects depend on objective and capacity.

Critical weaknesses
  • The exact main-table threshold choices and selection rule are missing. The paper introduces the admission threshold and Figure 2 demonstrates substantial objective-dependent sensitivity. The setup do not map all main-table rows to selected thresholds or explain their selection. Table 6 averages five thresholds, which does not resolve that mapping. This impedes reproduction and assessment of a usable default. No unfair or outcome-dependent tuning is established. Original CSTU coefficients have published defaults of 1.0 each; the authors need only confirm whether those values are inherited here.

  • The mechanism is not fully isolated from occupancy and eviction changes. Figure 2, p. 9, varies a filter that can also change achieved occupancy. Table 6, p. 15, provides a useful score ablation but averages over thresholds without occupancy or acceptance-distance diagnostics. The policy effect is supported; attribution solely to reduced within-class redundancy is less established. This is a minor explanatory gap.

Questions for authors
  • What exact threshold produced each row in Tables 1-3 , and what data or rule selected it? Were both inherited CSTU coefficients fixed at 1.0?
  • Can existing logs report achieved occupancy, acceptance rates, and within-class distances? Can the zero-threshold variant keep the same replacement rule to separate admission from eviction effects?
  • Is Figure 1 an observed DAB failure case or a schematic? This is an unresolved clarification, not an accepted claim that DAB lacks balancing or that its implementation is wrong.
1 comment

Dear Reviewer inCx,

Thank you for your constructive feedback and for recognizing the scientific contributions of our work. We would like to kindly follow up on our rebuttal.

We hope our responses have addressed your concerns and would greatly appreciate your consideration of these clarifications in your final assessment. We remain happy to address any outstanding questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.