- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper revisits replay-memory policies for test-time adaptation under temporally correlated, label-skewed streams. It separates the memory policy from the adaptation objective and evaluates six policies (none, FIFO, reservoir, PBRS, CSTU, DAB-buffer, plus the proposed version) across eleven TTA methods. It argues that class-balanced policies leave within-class redundancy unaddressed, and proposes GOTTA: an incoming sample is admitted only if its softmax vector is more than ε (L2) away from every stored sample with the same pseudo-label, and when memory is full it may replace its nearest stored neighbor in the majority class, subject to the CSTU age/uncertainty score. Validation is on CIFAR-10-C (continual, M=32, three seeds), a 10-class ImageNet-C subset, and ITD.
Critical strengths
- Decoupled evaluation of memory policies:
Evaluating each policy under many adaptation methods with shared streams is a sensible protocol, and the paper reports unfavorable cases rather than hiding them. The finding that the effect of memory depends strongly on the adaptation mechanism is worth having in the literature and is the main reason I see value in this work.
- Simplicity:
The policy is a drop-in module with small overhead and no change to the adaptation objective, which would make it easy to adopt if the benefit were established.
Critical weaknesses
-
The evidence does not support the central claims.
-
Mechanism: In Figure 2, Norm, RoTTA and DAB peak at ε = 0 to 0.25 and degrade as ε grows. These are the columns where GoTTA leads in Table 1. At ε = 0 the filter admits essentially everything, so the gain over CSTU must come from another change, presumably the nearest-neighbor replacement rule, which is never ablated. At the other end, two softmax vectors with the same argmax cannot be far apart (about 0.95 at most for C=10 by my calculation), so ε ≥ 1.0 appears to freeze the memory at one sample per pseudo-class. The "redundancy-sensitive" methods peak exactly there, and their accuracy stays below the unadapted source model (56.49%). This suggests they benefit from adapting less, not from diversity. No direct measurement of memory content (redundancy, pseudo-label accuracy, confidence) is given.
-
Sensitivity and fairness: Averaged over ε (Table 6), the full method is below CSTU for Norm, NOTE and RoTTA. The ε used in each table is not stated, and the Table 1 numbers appear to correspond to different ε for different methods. If ε was selected per method on the evaluation stream, the comparison with untuned baselines is not fair.
-
Generality: On ITD, the benchmark closest to the motivating scenario, DAB-buffer is far ahead (34.44 / 25.38 / 43.91 vs 14.20 / 13.43 / 29.28). On full ImageNet-C GoTTA is not the best in any column, and at M = 8 PBRS is best for all three methods. ImageNet-C* accuracies are near chance for a 10-class task. The text discusses these cases mainly relative to no memory, and only Table 1 uses more than one seed.
-
-
The paper does not engage with the closest prior work, which narrows its novelty and leaves the benchmark incomplete. The two framing claims are that memory policies have not been studied independently of the adaptation method, and that within-class redundancy is an unaddressed limitation. Both need to be qualified.
-
Redundancy filtering: EATA, used here as a baseline, already excludes samples whose model output is too similar (cosine similarity against a threshold ε) to a moving average of previous outputs. It explicitly mentions comparing against stored outputs of earlier samples as the direct alternative. DAB carries this idea into a buffer. GOTTA's filter is close to that alternative applied per pseudo-class, yet the relation is not discussed. DANCE (Findings of ACL 2026) uses within-class nearest-neighbor novelty for cache admission in VLM test-time adaptation, with the same motivation of near-duplicate confident samples.
-
Memory selection as a module: SNAP (Neurips 2025) proposes a class- and domain-representative memory and evaluates it with five TTA methods. The multi-cluster memory of Tseng et al. (arXiv 2603.21135) is a plug-and-play memory for the same PTTA setting. The paper cited for ITD itself equips existing TTA methods with memory. SoTTA (Neurips 2023; high-confidence, class-balanced sampling), STAMP (cited but not benchmarked) and ResiTTA (ICASSP 2025) propose further selection rules. Several of these prefer confident or representative samples, the opposite of what a softmax-distance filter tends to admit, so they matter as baselines and not only as citations.
-
Coreset framing. The continual-learning literature on buffer selection (CBRS, Rainbow Memory, OCS) is not discussed, and Aljundi et al. (2019), a sample-selection method, is cited as parameter regularization.
-
A paper whose contribution is a systematic study of memory policies should cover the existing policies and state precisely what is new relative to them.
Questions for authors
Which ε is used in each table? Is it fixed or chosen per adaptation method, and on what data? Were hyperparameters of the baseline policies tuned in the same way?
At ε = 0, what differs between GoTTA and CSTU, and which difference explains the gain for Norm, RoTTA and DAB? Please ablate the admission filter and the nearest-neighbor replacement separately.
For ε ≥ 1.0, how many samples are admitted over the stream, and what is the memory occupancy over time? Is my reading correct that the memory is frozen after the first sample per pseudo-class?
What is the source-model accuracy on ImageNet-C* and ITD? Why are ImageNet-C* accuracies near chance when the 1000-class results are much higher?
Can you report, for each policy, the pseudo-label accuracy, mean confidence, and a redundancy measure of the stored samples?
What do the "DAB" column and the no-memory RoTTA row correspond to?
How does the admission filter differ in effect from EATA's redundancy criterion and from DAB's, beyond the choice of reference (nearest stored neighbor vs. moving average)? An experiment replacing one with the other inside the same buffer would clarify this.
Dear Reviewer MUhZ,
Thank you for your feedback. We would like to kindly follow up on our detailed rebuttal, in which we provide further experimental evidence and clarify the scientific contributions of our work.
We hope you will have the opportunity to revisit our responses. We believe our clarifications provide a more comprehensive perspective on our contributions and hope they will be helpful in reconsidering your initial assessment.