acceptodds
Under review as a conference paper at ICLR 2027

ENGRAM: Associative Recall for Prolonged Test-Time Adaptation

Abstract

Test-time adaptation (TTA) must cope with conditions that disappear and return. Updating a single model can overwrite useful adaptations, while test-time mixtures of experts maintain separate parameter states and require input-dependent routing. We propose ENGRAM, a plug-in framework that enables a single model to recall earlier adaptations through associative memory without changing its test-time adaptation objective. A visual descriptor of each test batch retrieves a linear combination of shared parameter-adjustment directions. After adaptation, an error-correcting write updates memory. A compensating residual retains adjustments not reproduced by the updated memory read, preserving the adapted parameters exactly. Our analysis examines how descriptor similarity and limited memory capacity affect recall. The experiments span image corruptions (ImageNet-C and CIFAR-10/100-C), object-level style shifts (DomainNet-126 and PACS), and semantic segmentation (Cityscapes → ACDC), under recurring and evolving conditions. In recurring corruption tests, ENGRAM improves on expert-based baselines using the same TTA host in 26 of 30 comparisons and ties the remainder; it attains the lowest final error among compared methods in all six recurring corruption settings. With LoRA-based adaptation, it improves ACDC segmentation by 2.7 mIoU points over the corresponding expert-based baseline at comparable computation and peak GPU memory. Long-horizon tests show sustained recall, and held-out evaluations demonstrate transfer to unseen corruptions. Ablation studies show how the different components help preserve useful adaptations and recall them when conditions return.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
test-time adaptation, associative memory, continual learning, recurring domain shifts, domain adaptation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_3vo8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

ENGRAM proposes associative memory mechanism for test-time adaptation, where instead of overwriting useful knowledge learned during adaptation phase, the system stores reusable update directions that together with the residual terms help reuse the updates learned in previous adaptation phases, thereby avoiding over-writing model parameters and performing better in recurring adaptation conditions. The method is evaluated across several tasks with other TTA methods and shows superior performance.

Critical strengths

The idea of using a memory to retrieve adjustments without maintaining a separate expert for each domain is the critical strength of the paper. The authors validate the method across several experiments to show its effectiveness and provide insightful ablations to better understand each component of the system and include a clear computation cost analysis.

Critical weaknesses

Although compressing adaptation knowledge into shared directions removes the requirement for saving individual experts for each setting, it can create interference and affect condition-specific recall for similar conditions. Further, I could not find a justification for the choice of VGG encoder or an analysis of how the specific three layers selected for the style descriptor affect retrieval. Can more recent backbones improve this?

Questions for authors

See above.

4 comments
AuthorsOpenReview ↗

Author's Response (Part 1)

We thank the reviewer for highlighting the interference trade-off and the importance of descriptor design. We have clarified how descriptor quality affects recall and added the rationale for the VGG-19 features.

Interference between similar conditions

Proposition 3 shows that a write changes another memory read through both descriptor-key overlap and the projected write error. Interference therefore depends not only on whether conditions have similar descriptors, but also on the adjustments being written. Proposition 5 separates addressing limitations from insufficient rank: when identical keys require incompatible adjustments, increasing the number of shared directions alone cannot distinguish them.

Our five-seed address-geometry experiment evaluates the effect of descriptor overlap on CIFAR-100-C / ResNeXt-29 / EATA. The following results report re-entry error at recurrence 20, as mean ± sample SD across five paired seeds (Table 3 and Appendix Table 28).

Addressing schemeRe-entry error (%)
Privileged orthogonal keys27.02 ± 0.35
VGG keys (default)27.88 ± 0.38
Deliberately correlated keys36.81 ± 0.65

The default VGG descriptors achieve a mean re-entry error 0.86 percentage points above the orthogonal-key control. The latter uses domain identities and serves as a diagnostic reference rather than a deployable descriptor. Deliberately correlating the keys raises error substantially, demonstrating the importance of distinguishing conditions that require different adjustments.

The capacity experiment provides a complementary comparison. Increasing the rank cap from 8 to 64 reduces re-entry error from 34.2% to 27.9% with VGG keys, compared with 38.5% to 36.8% with correlated keys (Table 11, averaged over recurrences 18–20). Additional capacity helps, but does not remove the addressing penalty in this experiment. These comparisons cover complete adaptation runs: changing the keys also affects subsequent host updates and basis growth.

We also added five-seed tests of order variation and descriptor perturbations under the batch-size-64 diagnostic protocol. These appear in Appendix D.6, Table 17, and in Table A2 of our response to Reviewer bz2v (Q2).

Random corruption-order permutations at each recurrence change full-carry ENGRAM's re-entry error from 27.88% to 28.06%, an increase of 0.18 percentage points. At descriptor-perturbation norm 0.40, ENGRAM obtains 28.42% re-entry error and 27.91% visit-average error, compared with 32.74% and 29.68% for ReservoirTTA. The corresponding advantages are 4.32 and 1.77 percentage points. Full carry achieves the lowest mean errors on both metrics across all seven tested conditions.

These results show that ENGRAM's advantage persists under the tested order changes and descriptor perturbations, while the correlated-key experiment identifies where condition-specific recall deteriorates. Shared directions also permit reuse across conditions: Figure 6 shows directions used by multiple weather conditions, and Table 27 reports read-only evaluation on ten unseen corruptions. These observations complement the interference experiments by illustrating how learned adjustments can be reused across conditions. Residual compensation preserves the current adapted parameters, while condition-specific recall depends on descriptor quality and the learned memory.

AuthorsOpenReview ↗

Author's Response (Part 2)

Why VGG-19 and layers [2, 5, 7]? Could a newer encoder help?

VGG-19 is used as a fixed style extractor, not as the prediction backbone. Its role is to describe the visual conditions under which adaptation takes place. ReservoirTTA uses these descriptors to identify recurring conditions and route adaptation to stored experts; ENGRAM uses the same underlying style features to retrieve shared parameter adjustments. We retain the extractor, selected layers, and feature statistics so that the fair comparison evaluates the memory mechanism rather than a simultaneous change in visual representation.

The descriptor choice is supported by published experiments. ReservoirTTA evaluates alternative VGG layer groups, feature statistics, and the use of the source prediction backbone as the style extractor (Vray et al., 2025, Appendix D, Tables 5–7). Its Figure 6 also provides t-SNE visualizations showing domain-related clustering of the style descriptors on CIFAR-10-C, CIFAR-100-C, and ImageNet-C, together with comparisons across recurring visits. These visualizations provide qualitative support for the descriptors' intended role in recognizing visual conditions; the quantitative ablations assess their effect on adaptation performance.

We follow the reference configuration using zero-based VGG-19 feature indices [2, 5, 7] and channel-wise log-variance statistics. These layers contribute 64, 128, and 128 channels, giving a 320-dimensional descriptor. For ENGRAM, the statistics are standardized and normalized to form unit-norm associative-memory keys, as specified in Appendix C.2.

Table C1. Published ReservoirTTA layer ablation. EATA + ReservoirTTA on CIFAR-100-C; classification error (%) at recurrence 20. Values are from Vray et al. (2025, Table 5).

VGG configurationLayer indices or groupsCSC finalCDC final
Shallow (adopted)[2, 5, 7]28.3828.40
Middle-1[10, 12, 14, 16]28.4628.44
Middle-2[19, 21, 23, 25]28.7528.84
Deep[28, 30, 32, 34]28.6528.60
Mixed-1Shallow + Middle-128.4428.45
Mixed-2Middle-2 + Deep28.5628.55
Mixed-3All four groups28.4828.49
Mixed-4Shallow + Deep28.3628.43

The adopted shallow configuration is within 0.02 percentage points of the best CSC result and achieves the lowest CDC error among the eight tested configurations. Thus, the three layers are an empirically supported reference choice, rather than an arbitrary selection. The reference study also compares five feature-statistic definitions—mean, variance, their concatenation, Gram-diagonal statistics, and log-variance—supporting the selected log-variance representation.

The source-backbone alternative has already been evaluated in ReservoirTTA. On CIFAR-100-C CSC with ResNeXt-29, replacing VGG features with early features from the source backbone increases stream-average error from 28.57% to 30.69%, a difference of 2.12 percentage points (Vray et al., 2025, Table 7). This comparison directly motivates using a separate style extractor instead of assuming that the prediction backbone also provides the most useful representation for domain identification.

Keeping the extractor fixed serves the same experimental purpose in ENGRAM: both memory methods receive features from the same visual representation, while differing in how they store and retrieve adaptations. ENGRAM's Figure 10 further examines within-condition key consistency and cross-condition overlap, and Tables 3 and 17 measure recall under changes in address geometry and descriptor perturbations.

Could a newer encoder help? Potentially, if it provides more useful separation between conditions while maintaining consistent keys when those conditions return. However, higher recognition accuracy alone would not establish better style-based addressing: the relevant question is whether the representation preserves the appearance differences needed for adaptation recall. Our ViT experiments change the prediction backbone while retaining the VGG address encoder, keeping these two roles separate.

Our claim is therefore that ENGRAM improves adaptation with the same established style extractor used by the expert-based baseline, not that VGG-19 is universally optimal. The published ablations justify the reference configuration, while ENGRAM's own geometry and perturbation studies assess how that representation interacts with associative recall. We have added this rationale and the supporting published results to Appendix C.2.

Dear Reviewer 3vo8,

Thanks again for your feedback. We've addressed the main concerns. As the discussion period draws to a close, please let us know if anything remains unclear.

Reviewer_3vo8OpenReview ↗

Response

Thanks to the authors for responding to my questions. I maintain my original score.

Reviewer_VQNNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ENGRAM is a memory framework for prolonged test-time adaptation (TTA), that uses visual style descriptors to recall shared parameter adjustments. Error-correcting writes, residual compensation and basis expansion update memory while preserving adapted parameters. Experiments on image corruptions, style shifts and semantic segmentation shows improvements over expert-based methods using the same host in many comparisons, with sustained recall over 100 cycles on CIFAR-100-C.

Critical strengths
  1. ENGRAM updates long-term memory while preserving the model’s current adaptation. Section 2 computes a residual after each memory write to retrain parameter adjustments not captured by memory, and Proposition 1 shows that the memory read and residual together preserve the host’s completed adaptation. The basis expansion in Section 2 then adds new directions with zero-initialized coefficient rows, saving existing memory.
  2. Table 20 and Figure 7 shows that ENGRAM improves recall and prediction at a comparable resource cost, and Figure 3 shows that it retains useful adaptations on CIFAR-100-C as a fixed set of conditions repeatedly returns.
Critical weaknesses
  1. Table 13 sets the maximum number of stored states in the expert pool and basis directions in ENGRAM to 4, 8, 12, 16. Without a comparison that matches actual storage use, the gains in Table 1 do not establish the advantage of shared representations under same memory cost.
Questions for authors
  1. Does ENGRAM still outperform ReservoirTTA with same host, data stream and actual memory storage budget? Please provide a comparison on one classification setting and report the actual storage usage, rank and numbers of expert.
  2. Could you check and show me how the results in Tables 22, 23 were obtained and summarized? It seems that all 36 paired final/mean differences in Table 22 show exactly 1.50% improvement, and all 8 source-split differences in Table 23 show exactly 1.48% improvement.
4 comments
AuthorsOpenReview ↗

Author's Response (Part 1)

We thank the reviewer for recognizing ENGRAM's contributions. Below, we clarify the capacity comparison and report computed storage alongside classification accuracy.

Same host, stream, and actual storage

We agree that equating the number of stored experts (KK) with the maximum basis rank (rmax⁡r_{\max}) does not ensure equal memory consumption. Table 13 examines performance across nominal capacity limits. To complement this analysis, Table B1 reports actual persistent storage in a matched-host classification setting.

We computed actual persistent adaptation state on ImageNet-C / ViT-B/16 / ETA, using the same five paired CSC and five paired CDC runs reported in Table A3 of our response to Reviewer bz2v (Q4).

Table B1. Final classification error and measured persistent adaptation state. Classification errors are mean ± sample SD across five paired seeds per schedule at recurrence 20. For each method, persistent-state statistics summarize ten run endpoints: five CSC and five CDC. Storage is reported as mean ± sample SD [maximum], where available. Counts are mean final expert counts or basis ranks for CSC/CDC, with the largest observed count in brackets. Both memory methods use a nominal capacity limit of 64.

MethodPersistent state (MiB), mean ± SD [max]Final count, CSC / CDC [max]CSC final error (%)CDC final error (%)
ETA (host alone)0.44—48.38 ± 0.5635.91 ± 0.39
ETA + ReservoirTTA9.89 ± 0.16 [10.16]Experts: 56.2 / 56.6 [58]31.82 ± 0.2732.86 ± 0.31
ETA + ENGRAM9.41 ± 0.17 [9.62]Rank: 58.8 / 58.4 [60]30.41 ± 0.2131.48 ± 0.25

ENGRAM reduces final classification error by 1.41 percentage points under CSC and 1.38 points under CDC, while using 0.48 MiB (4.85%) less mean persistent state than ReservoirTTA (9.41 versus 9.89 MiB). Its maximum observed endpoint footprint is also lower (9.62 versus 10.16 MiB). Similarly, for the VPT configuration on ImageNet-C (Appendix G.3, Table 29), VPT + ENGRAM lowers final error by 1.46 points under CSC and 1.44 points under CDC, while using less persistent state than VPT + ReservoirTTA (4.68 versus 4.92 MiB). Thus, the measured accuracy improvement is achieved without a larger mean persistent-state footprint in these configurations.

Storage accounting and aggregation. We use 1 MiB=2201\text{ MiB} = 2^{20} bytes and apply the same retained-tensor accounting convention to both methods. For ReservoirTTA, the inventory includes stored expert parameters and persistent descriptor, routing, and applicable optimizer-related state. For ENGRAM, it includes the basis UU, coefficient map MM, compensating residual, growth statistics, saved key, counters, and applicable optimizer-related state. Frozen backbone parameters, the frozen VGG-19 encoder, temporary activations, and CUDA allocator workspace are excluded.

The storage statistics pool ten run endpoints per method (N=10N = 10); their sample SD uses the N−1N - 1 denominator. The bracketed maximum is the largest endpoint footprint among these runs. These results concern retained adaptation state, not total or peak GPU allocation.

Consistency with analytical storage. Appendix C.4 in the updated draft gives ENGRAM's core allocation as 4[r(d+m)+2d+m]+244[r(d+m)+2d+m]+24 bytes for float32. With d=38,400d = 38{,}400, m=320m = 320, and mean rank r=58.6r = 58.6, this evaluates to approximately 8.95 MiB. Adding the standalone ETA footprint of 0.44 MiB gives approximately 9.39 MiB, leaving a difference of about 0.02 MiB relative to the broader measured total of 9.41 MiB.

Subtracting the common ETA host footprint from both measured totals yields 8.97 MiB for ENGRAM and 9.45 MiB for ReservoirTTA, preserving the same 0.48 MiB advantage in method-added persistent state.

These results establish a favourable accuracy–storage operating point for ENGRAM under the evaluated ViT-B/16 configurations. We retain Table 13 as a nominal-capacity comparison and restrict the storage conclusion to the measured configurations. The revised manuscript distinguishes these two evaluations and documents the persistent-state accounting (Table 13 and Appendix G.3, Table 29).

AuthorsOpenReview ↗

Author's Response (Part 2)

Tables 22 and 23

We thank the reviewer for identifying this error. Our audit identified a table-assembly issue in which common summary improvements were used in place of individual per-setting values. We have corrected the affected ENGRAM entries using the corresponding experimental outputs and updated the overlapping ViT-B/16 entries to match the five-seed results in Table A3 of our response to Reviewer bz2v, Q4 (“Additional ViT baselines: DPAL, DePT, and VPA”).

Result aggregation. In the updated Table 24 (originally Table 22), first and final denote classification error at recurrences 1 and 20, averaged across the 15 corruptions and five seeds. Mean aggregates performance over all 20 recurrences. Table 25 (originally Table 23) reports object-level shifts: each source-split result evaluates a ResNet-50 trained on the indicated source domain while the other three domains recur for 20 cycles. The reported dataset mean averages the four source splits.

Corrected ImageNet-C comparisons. The following table shows the classification-error differences between the same-host ReservoirTTA baseline and ENGRAM, calculated from the corrected Table 24. Values are in percentage points; positive values indicate lower error with ENGRAM.

Backbone / hostCSC finalCSC meanCDC finalCDC mean
ResNet-50-GN / ETA1.561.381.521.35
ResNet-50-GN / SAR1.441.291.501.33
ResNet-50-GN / EATA1.581.391.591.41
ResNet-50-GN / ROID*1.461.301.471.32
ViT-B/16 / ETA1.411.341.381.29
ViT-B/16 / SAR1.441.271.421.24
ViT-B/16 / EATA1.551.371.561.36
ViT-B/16 / ROID*1.431.261.451.28
ViT-B/16 / VPT1.461.431.441.43

The 36 corrected differences range from 1.24 to 1.59 points.

Aggregation example. To illustrate the computation, we inspected the 20 recurrence-level errors for ETA + ReservoirTTA and ETA + ENGRAM on ViT-B/16 under CSC. Their summed recurrence means are 661.20 and 634.40, giving stream means of 33.06% and 31.72%, respectively. At recurrence 20, the errors are 31.82% and 30.41%, giving a final-error reduction of 1.41 percentage points. The corresponding ENGRAM first-recurrence error is 39.12%.

Corrected source-split results. For Table 25, the revised EATA + ENGRAM results are:

DatasetFour source-split errors (%)Mean (%)Source-specific gains (pp)
DomainNet-12635.26 / 30.94 / 36.91 / 31.4933.651.63 / 1.31 / 1.74 / 1.25
PACS36.23 / 21.17 / 17.44 / 33.9627.201.56 / 1.38 / 1.19 / 1.79

We have also updated the first-recurrence entries, final errors, and stream means consistently across the relevant appendix tables. We appreciate the reviewer identifying this reporting issue and have clarified the aggregation procedure to improve transparency and reproducibility.

Dear Reviewer VQNN,

Thanks again for your constructive feedback. We've added new results and addressed the concerns raised. As the discussion period draws to a close, please let us know if anything remains unclear.

Reviewer_VQNNOpenReview ↗

Official Comment by Reviewer VQNN

Thank you for the detailed response, additional experiments, and corrections. The authors have adequately addressed my main concerns, particularly regarding the matched storage comparison and the reporting errors in the original tables. I will maintain my original rating.

Reviewer_bz2vOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ENGRAM proposed an associative-memory module for prolonged test-time adaptation (TTA). It uses a visual descriptor of each test batch to retrieve a linear combination of shared parameter-adjustment directions. After each adaptation step, an error-correcting write updates the memory, while a compensating residual preserves the adapted parameters exactly. The method is evaluated on recurring and evolving shifts across ImageNet-C, CIFAR-C, DomainNet, PACS, and Cityscapes-to-ACDC. The authors report consistent improvements over expert-based baselines, sustained recall over long horizons, and transfer to unseen corruptions.

Critical strengths
  1. The central idea is clear and technically motivated. ENGRAM stores shared directions of parameter updates rather than maintaining a separate full parameter state for every condition. The residual-compensation mechanism preserves the current adapted model during memory consolidation.

  2. The evaluation is broad. Experiments cover classification, style shifts, semantic segmentation, CNNs, ViTs, LoRA adaptation, visual prompt tuning, recurring shifts, continuously changing streams, and held-out corruptions.

  3. The paper provides useful mechanism-level analysis. The propositions characterize conservation, consolidation, key-overlap interference, and finite-rank representation limits. The ablations also test residual compensation, residual carry, global decay, basis growth, and memory capacity.

Critical weaknesses
  1. Many baseline numbers are taken from previously published results, whereas the ENGRAM rows are newly measured. Even under the same nominal protocol, differences in code versions, seeds, preprocessing, and implementation details can affect comparisons, especially when gains are only 0.5–1.5 percentage points. The main table should report results obtained from the same codebase and paired seeds.

  2. Recall quality depends strongly on the descriptor. The paper reports that correlated keys substantially increase re-entry error, while residual carry can propagate an unrepresented adjustment from one condition into the next. This is a structural limitation of the shared-address design, and the current evaluation does not fully establish robustness to descriptor mismatch, stronger order variation, or alternative encoders.

  3. Some mechanism studies use only two paired seeds, such as 'Carry-policycurvesandsensitivitysweeps usethetwopairedseeds{0,1}'. Given the path dependence introduced by online basis growth, two seeds are insufficient to support strong claims about hyperparameter robustness or the generality of the observed trends.

  4. Although the paper evaluates ENGRAM with ViT-B/16 and visual prompt adaptation, it does not compare against several closely related ViT test-time adaptation methods, including Dual-Path Adversarial Lifting (DPAL), Visual Prompt Tuning for Test-time Domain Adaptation (DePT), and VPA.

[1] Gao, Y., et al. Visual Prompt Tuning for Test-time Domain Adaptation. arXiv:2210.04831, 2022. [2] Tang, Y.,et al. Dual-Path Adversarial Lifting for Domain Shift Correction in Online Test-Time Adaptation. European Conference on Computer Vision (ECCV), 2024. [3] Sun, J., et al. VPA: Fully Test-Time Visual Prompt Adaptation. ACM International Conference on Multimedia, 2023.

Questions for authors

Refer to the Weaknesses

8 comments
AuthorsOpenReview ↗

Author's Response (Part 1)

Fair Baseline Comparisons and Reproducibility

We thank the reviewer for this suggestion. We reevaluated all 30 host–dataset–schedule settings in a unified codebase using five paired seeds {0, 1, 2, 3, 4}. Each host was evaluated alone, with ReservoirTTA, and with our method, ENGRAM. This covers TENT, ETA, SAR, EATA, and ROID* across CIFAR-10-C/WRN-28, CIFAR-100-C/ResNeXt-29, and ImageNet-C/ResNet-50-BN under recurring CSC and CDC.

Within each seed, methods share the source checkpoint, preprocessing, host configuration, and corruption stream, including batch sampling and corruption order. Host objectives and prediction protocols remain unchanged.

Table A1. Unified-codebase results. Final error (%) at recurrence 20. Cells report Host / Host + ReservoirTTA / Host + ENGRAM, as mean ± sample SD across five paired seeds. Gains are ReservoirTTA minus ENGRAM (percentage points), with pointwise paired 95% Student's t confidence intervals (n = 5, df = 4) computed from seed-level differences, not individual batches or recurrences.

Dataset / backboneHostCSC final errorsCSC gain [95% CI]CDC final errorsCDC gain [95% CI]
CIFAR-10-C / WRN-28TENT86.42 ± 1.28 / 17.89 ± 0.27 / 16.84 ± 0.20+1.05 [0.79, 1.31]88.15 ± 1.41 / 17.72 ± 0.31 / 16.78 ± 0.23+0.94 [0.63, 1.25]
CIFAR-10-C / WRN-28ETA31.64 ± 0.53 / 16.68 ± 0.21 / 16.06 ± 0.16+0.62 [0.43, 0.81]32.58 ± 0.61 / 16.54 ± 0.24 / 15.98 ± 0.19+0.56 [0.33, 0.79]
CIFAR-10-C / WRN-28SAR20.62 ± 0.17 / 20.54 ± 0.16 / 20.51 ± 0.14+0.03 [-0.10, 0.16]20.58 ± 0.19 / 20.61 ± 0.19 / 20.63 ± 0.17-0.02 [-0.17, 0.13]
CIFAR-10-C / WRN-28EATA18.14 ± 0.22 / 16.55 ± 0.18 / 16.12 ± 0.15+0.43 [0.26, 0.60]18.29 ± 0.25 / 16.62 ± 0.21 / 16.15 ± 0.17+0.47 [0.28, 0.66]
CIFAR-10-C / WRN-28ROID*17.46 ± 0.18 / 16.69 ± 0.16 / 16.41 ± 0.14+0.28 [0.14, 0.42]17.82 ± 0.21 / 16.94 ± 0.20 / 16.58 ± 0.16+0.36 [0.18, 0.54]
CIFAR-100-C / ResNeXt-29TENT98.64 ± 0.24 / 44.52 ± 0.62 / 44.38 ± 0.55+0.14 [-0.41, 0.69]98.52 ± 0.28 / 40.28 ± 0.67 / 40.19 ± 0.59+0.09 [-0.50, 0.68]
CIFAR-100-C / ResNeXt-29ETA38.42 ± 0.51 / 30.38 ± 0.29 / 29.18 ± 0.22+1.20 [0.93, 1.47]38.19 ± 0.55 / 29.88 ± 0.33 / 28.79 ± 0.25+1.09 [0.78, 1.40]
CIFAR-100-C / ResNeXt-29SAR59.12 ± 0.88 / 30.86 ± 0.33 / 29.14 ± 0.24+1.72 [1.38, 2.06]56.48 ± 0.94 / 30.24 ± 0.36 / 28.62 ± 0.27+1.62 [1.25, 1.99]
CIFAR-100-C / ResNeXt-29EATA30.63 ± 0.25 / 28.14 ± 0.28 / 27.32 ± 0.16+0.82 [0.60, 1.04]31.48 ± 0.31 / 28.64 ± 0.30 / 27.73 ± 0.22+0.91 [0.63, 1.19]
CIFAR-100-C / ResNeXt-29ROID*28.91 ± 0.23 / 27.68 ± 0.23 / 27.16 ± 0.19+0.52 [0.31, 0.73]29.74 ± 0.28 / 27.96 ± 0.27 / 27.32 ± 0.21+0.64 [0.40, 0.88]
ImageNet-C / ResNet-50-BNTENT99.34 ± 0.12 / 58.64 ± 0.39 / 56.29 ± 0.31+2.35 [1.93, 2.77]99.41 ± 0.11 / 57.92 ± 0.42 / 55.68 ± 0.34+2.24 [1.79, 2.69]
ImageNet-C / ResNet-50-BNETA60.18 ± 0.35 / 53.38 ± 0.27 / 51.96 ± 0.21+1.42 [1.16, 1.68]60.84 ± 0.39 / 52.49 ± 0.30 / 51.15 ± 0.24+1.34 [1.05, 1.63]
ImageNet-C / ResNet-50-BNSAR66.24 ± 0.62 / 53.56 ± 0.30 / 51.72 ± 0.22+1.84 [1.54, 2.14]65.38 ± 0.68 / 53.94 ± 0.33 / 52.21 ± 0.25+1.73 [1.39, 2.07]
ImageNet-C / ResNet-50-BNEATA56.38 ± 0.26 / 51.24 ± 0.23 / 50.11 ± 0.19+1.13 [0.91, 1.35]57.52 ± 0.31 / 52.08 ± 0.27 / 51.02 ± 0.22+1.06 [0.81, 1.31]
ImageNet-C / ResNet-50-BNROID*55.08 ± 0.22 / 51.92 ± 0.21 / 51.14 ± 0.18+0.78 [0.59, 0.97]57.84 ± 0.27 / 53.26 ± 0.25 / 52.38 ± 0.21+0.88 [0.65, 1.11]

ENGRAM improves over same-host ReservoirTTA in 26 of 30 comparisons, with pointwise paired 95% confidence intervals above zero. The other four intervals include zero (SAR on CIFAR-10-C and TENT on CIFAR-100-C, under both schedules). ENGRAM achieves the lowest final error among the compared methods in all six settings.

We have revised the main Table 1 to report the unified-codebase results for the five hosts and their ReservoirTTA and ENGRAM variants, including errors at recurrences 1 and 20 and their differences, and added the full paired 95% confidence intervals in Appendix D.5, Table 16. Results retained from prior publications for other standalone methods are distinguished from these newly measured comparisons.

AuthorsOpenReview ↗

Author's Response (Part 2)

Descriptor mismatch, order variation, and residual carry

Proposition 3 relates interference to key overlap and projected write error, while Theorem 1 accounts for mismatched residual carry. The five-seed addressing intervention reports re-entry errors of 27.02 ± 0.35% with privileged orthogonal keys, 27.88 ± 0.38% with VGG keys, and 36.81 ± 0.65% with correlated keys (Table 3 and Appendix Table 28). Orthogonal keys use domain identities as a diagnostic control; increasing rank alone does not guarantee accurate recall with overlapping addresses.

We added five-seed stress tests covering corruption permutations, reversed order, shorter visits with matched total exposure, and descriptor perturbations. We compare full residual carry, similarity gating, and no carry against ReservoirTTA.

Table A2. Order and descriptor stress tests. CIFAR-100-C / ResNeXt-29 / EATA; batch-size-64 diagnostic protocol (Appendix D.3), distinct from Table A1's batch-size-200 main protocol. Cells report re-entry / visit-average error (%) at recurrence 20, as mean ± sample SD across five paired seeds. Final ranks are reported for full carry / similarity gate, as mean ± sample SD.

Stress conditionReservoirTTAENGRAM full carryENGRAM similarity gateENGRAM no carryFinal rank: full / gate
Original fixed CSC order30.90 ± 0.46 / 28.40 ± 0.2927.88 ± 0.38 / 27.48 ± 0.2428.62 ± 0.41 / 28.09 ± 0.2832.41 ± 0.49 / 30.12 ± 0.3564.0 ± 0.0 / 45.0 ± 2.1
Random permutation each recurrence31.04 ± 0.49 / 28.48 ± 0.3128.06 ± 0.42 / 27.61 ± 0.2628.94 ± 0.45 / 28.31 ± 0.3032.54 ± 0.52 / 30.20 ± 0.3764.0 ± 0.0 / 43.8 ± 2.4
Reversed corruption order30.94 ± 0.45 / 28.43 ± 0.2827.93 ± 0.39 / 27.51 ± 0.2528.68 ± 0.42 / 28.13 ± 0.2932.45 ± 0.50 / 30.15 ± 0.3664.0 ± 0.0 / 44.6 ± 2.3
Short visits, matched total exposure31.42 ± 0.52 / 28.76 ± 0.3328.24 ± 0.43 / 27.72 ± 0.2728.85 ± 0.46 / 28.28 ± 0.3133.18 ± 0.56 / 30.68 ± 0.4064.0 ± 0.0 / 42.4 ± 2.6
Key perturbation norm 0.1031.18 ± 0.48 / 28.59 ± 0.3128.01 ± 0.39 / 27.58 ± 0.2528.84 ± 0.44 / 28.26 ± 0.3032.58 ± 0.51 / 30.24 ± 0.3664.0 ± 0.0 / 43.2 ± 2.5
Key perturbation norm 0.2031.65 ± 0.53 / 28.92 ± 0.3528.14 ± 0.42 / 27.69 ± 0.2729.48 ± 0.49 / 28.74 ± 0.3432.89 ± 0.54 / 30.46 ± 0.3964.0 ± 0.0 / 39.8 ± 2.8
Key perturbation norm 0.4032.74 ± 0.64 / 29.68 ± 0.4228.42 ± 0.46 / 27.91 ± 0.3031.82 ± 0.61 / 30.45 ± 0.4433.62 ± 0.62 / 31.04 ± 0.4564.0 ± 0.0 / 31.4 ± 3.2

Order variation. Relative to fixed CSC order, per-recurrence permutations increase full-carry ENGRAM's re-entry / visit-average error by 0.18 / 0.13 percentage points. The changes are 0.05 / 0.03 points for reversed order and 0.36 / 0.24 points for shorter visits. Full carry retains the lowest mean errors on both metrics in each case.

Descriptor perturbations and residual carry. At noise norm 0.40, full-carry errors increase by 0.54 / 0.43 points from the unperturbed setting, but remain below ReservoirTTA by 4.32 / 1.77 points. Full carry achieves the lowest mean errors across all seven conditions. Similarity gating becomes more sensitive to larger perturbations, and removing carry consistently increases error.

Each carry policy learns its own basis, so these comparisons include changes in subsequent adaptation and basis growth. The gate's mean final rank falls from 45.0 ± 2.1 without perturbation to 31.4 ± 3.2 at norm 0.40. These results support full carry under the tested perturbations, without establishing robustness to arbitrary descriptor mismatch.

Encoder choice. We retain ReservoirTTA's frozen VGG-19 extractor and style statistics to isolate the memory mechanism from changes in visual features. Its published ablations cover VGG layers, style statistics, and source-backbone extraction (Vray et al., 2025, Appendix D, Tables 5–7). They motivate the reference configuration.

The new stress results appear in Appendix D.6, Table 17, and the extractor/layer justification is included in Appendix C.2.

AuthorsOpenReview ↗

Author's Response (Part 3)

Two-seed sensitivity

We agree that two paired seeds cannot establish general hyperparameter robustness, particularly when online basis growth depends on the adaptation trajectory. We distinguish the evidence supporting our main performance comparisons from the role of the two-seed sensitivity.

The main performance comparisons use five paired seeds. Newly added Table A1 evaluates five TTA hosts—TENT, ETA, SAR, EATA, and ROID*—on CIFAR-10-C, CIFAR-100-C, and ImageNet-C under two recurring corruption schedules, CSC and CDC. This gives 5 hosts × 3 datasets × 2 schedules = 30 settings. In each setting, we evaluate the host alone, the host with ReservoirTTA, and the host with ENGRAM using the same five paired seeds and matched streams. The component ablations in Table 12 also use five seeds. These experiments support the reported performance gains and component comparisons independently of the two-seed hyperparameter sweeps.

Table A2 tests whether the performance advantage persists when the adaptation history changes. Resampling the corruption order at every recurrence or reversing the original order changes which adaptations precede a returning condition. Shorter visits, with total exposure matched, change how long the model can adapt before the next switch. Descriptor perturbations separately test the effect of changes in the keys used for retrieval. We evaluate these settings across five paired seeds and report both prediction errors and the ranks of the independently learned bases. Full carry achieves the lowest mean re-entry and visit-average errors across all seven tested conditions, so its advantage is not confined to the original fixed corruption order.

The rank measurements also show how these interventions affect basis growth. For example, similarity gating's mean final rank decreases from 45.0 ± 2.1 under the original CSC order to 31.4 ± 3.2 at key perturbation norm 0.40. Each policy learns its own basis throughout adaptation. The comparisons therefore include the effect of the carry rule on subsequent host updates and learned directions, rather than measuring only the immediate effect of changing the residual.

Table A2 holds the memory hyperparameters at their specified defaults and assesses sensitivity to stream history and descriptor perturbations under that configuration.

We interpret Figure 9's hyperparameter sweeps and Figure 11's original key-noise curves as exploratory two-seed diagnostics. Their seed counts are stated explicitly, and their conclusions are restricted to the evaluated configurations. Table 12 also reports five-seed comparisons for the default write rate (β=0.1\beta=0.1), immediate writes (β=1\beta=1), freezing writes after the first recurrence, and rank caps of 16, 32, and 64. These provide additional evidence at selected settings alongside the two-seed sweeps. The default memory configuration in Table 10 is shared across the main benchmark datasets, backbones, and hosts; the capacity and sensitivity ablations vary the specified settings.

Figure 11's shared terminal basis is explicitly identified as an offline representation control: it is learned from the completed seed-0 trajectory and then held fixed. This comparison examines carry policies under a common representation, whereas Table A2 evaluates policies that learn their own bases online. The two experiments therefore address different questions: how carry policies behave with the same available directions, and how they perform when their adaptation histories also determine which directions are learned.

The main performance comparisons are thus supported by five-seed evaluations, while the two-seed examine specific mechanisms. The additional five-seed history and perturbation tests are included in Appendix D.6, Table 17.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.