ENGRAM: Associative Recall for Prolonged Test-Time Adaptation

Test-time adaptation (TTA) must cope with conditions that disappear and return. Updating a single model can overwrite useful adaptations, while test-time mixtures of experts maintain separate parameter states and require input-dependent routing. We propose ENGRAM, a plug-in framework that enables a single model to recall earlier adaptations through associative memory without changing its test-time adaptation objective. A visual descriptor of each test batch retrieves a linear combination of shared parameter-adjustment directions. After adaptation, an error-correcting write updates memory. A compensating residual retains adjustments not reproduced by the updated memory read, preserving the adapted parameters exactly. Our analysis examines how descriptor similarity and limited memory capacity affect recall. The experiments span image corruptions (ImageNet-C and CIFAR-10/100-C), object-level style shifts (DomainNet-126 and PACS), and semantic segmentation (Cityscapes → ACDC), under recurring and evolving conditions. In recurring corruption tests, ENGRAM improves on expert-based baselines using the same TTA host in 26 of 30 comparisons and ties the remainder; it attains the lowest final error among compared methods in all six recurring corruption settings. With LoRA-based adaptation, it improves ACDC segmentation by 2.7 mIoU points over the corresponding expert-based baseline at comparable computation and peak GPU memory. Long-horizon tests show sustained recall, and held-out evaluations demonstrate transfer to unseen corruptions. Ablation studies show how the different components help preserve useful adaptations and recall them when conditions return.