- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
ENGRAM proposes associative memory mechanism for test-time adaptation, where instead of overwriting useful knowledge learned during adaptation phase, the system stores reusable update directions that together with the residual terms help reuse the updates learned in previous adaptation phases, thereby avoiding over-writing model parameters and performing better in recurring adaptation conditions. The method is evaluated across several tasks with other TTA methods and shows superior performance.
Critical strengths
The idea of using a memory to retrieve adjustments without maintaining a separate expert for each domain is the critical strength of the paper. The authors validate the method across several experiments to show its effectiveness and provide insightful ablations to better understand each component of the system and include a clear computation cost analysis.
Critical weaknesses
Although compressing adaptation knowledge into shared directions removes the requirement for saving individual experts for each setting, it can create interference and affect condition-specific recall for similar conditions. Further, I could not find a justification for the choice of VGG encoder or an analysis of how the specific three layers selected for the style descriptor affect retrieval. Can more recent backbones improve this?
Questions for authors
See above.
4 comments
Author's Response (Part 2)
Why VGG-19 and layers [2, 5, 7]? Could a newer encoder help?
VGG-19 is used as a fixed style extractor, not as the prediction backbone. Its role is to describe the visual conditions under which adaptation takes place. ReservoirTTA uses these descriptors to identify recurring conditions and route adaptation to stored experts; ENGRAM uses the same underlying style features to retrieve shared parameter adjustments. We retain the extractor, selected layers, and feature statistics so that the fair comparison evaluates the memory mechanism rather than a simultaneous change in visual representation.
The descriptor choice is supported by published experiments. ReservoirTTA evaluates alternative VGG layer groups, feature statistics, and the use of the source prediction backbone as the style extractor (Vray et al., 2025, Appendix D, Tables 5–7). Its Figure 6 also provides t-SNE visualizations showing domain-related clustering of the style descriptors on CIFAR-10-C, CIFAR-100-C, and ImageNet-C, together with comparisons across recurring visits. These visualizations provide qualitative support for the descriptors' intended role in recognizing visual conditions; the quantitative ablations assess their effect on adaptation performance.
We follow the reference configuration using zero-based VGG-19 feature indices [2, 5, 7] and channel-wise log-variance statistics. These layers contribute 64, 128, and 128 channels, giving a 320-dimensional descriptor. For ENGRAM, the statistics are standardized and normalized to form unit-norm associative-memory keys, as specified in Appendix C.2.
Table C1. Published ReservoirTTA layer ablation. EATA + ReservoirTTA on CIFAR-100-C; classification error (%) at recurrence 20. Values are from Vray et al. (2025, Table 5).
| VGG configuration | Layer indices or groups | CSC final | CDC final |
|---|---|---|---|
| Shallow (adopted) | [2, 5, 7] | 28.38 | 28.40 |
| Middle-1 | [10, 12, 14, 16] | 28.46 | 28.44 |
| Middle-2 | [19, 21, 23, 25] | 28.75 | 28.84 |
| Deep | [28, 30, 32, 34] | 28.65 | 28.60 |
| Mixed-1 | Shallow + Middle-1 | 28.44 | 28.45 |
| Mixed-2 | Middle-2 + Deep | 28.56 | 28.55 |
| Mixed-3 | All four groups | 28.48 | 28.49 |
| Mixed-4 | Shallow + Deep | 28.36 | 28.43 |
The adopted shallow configuration is within 0.02 percentage points of the best CSC result and achieves the lowest CDC error among the eight tested configurations. Thus, the three layers are an empirically supported reference choice, rather than an arbitrary selection. The reference study also compares five feature-statistic definitions—mean, variance, their concatenation, Gram-diagonal statistics, and log-variance—supporting the selected log-variance representation.
The source-backbone alternative has already been evaluated in ReservoirTTA. On CIFAR-100-C CSC with ResNeXt-29, replacing VGG features with early features from the source backbone increases stream-average error from 28.57% to 30.69%, a difference of 2.12 percentage points (Vray et al., 2025, Table 7). This comparison directly motivates using a separate style extractor instead of assuming that the prediction backbone also provides the most useful representation for domain identification.
Keeping the extractor fixed serves the same experimental purpose in ENGRAM: both memory methods receive features from the same visual representation, while differing in how they store and retrieve adaptations. ENGRAM's Figure 10 further examines within-condition key consistency and cross-condition overlap, and Tables 3 and 17 measure recall under changes in address geometry and descriptor perturbations.
Could a newer encoder help? Potentially, if it provides more useful separation between conditions while maintaining consistent keys when those conditions return. However, higher recognition accuracy alone would not establish better style-based addressing: the relevant question is whether the representation preserves the appearance differences needed for adaptation recall. Our ViT experiments change the prediction backbone while retaining the VGG address encoder, keeping these two roles separate.
Our claim is therefore that ENGRAM improves adaptation with the same established style extractor used by the expert-based baseline, not that VGG-19 is universally optimal. The published ablations justify the reference configuration, while ENGRAM's own geometry and perturbation studies assess how that representation interacts with associative recall. We have added this rationale and the supporting published results to Appendix C.2.
Dear Reviewer 3vo8,
Thanks again for your feedback. We've addressed the main concerns. As the discussion period draws to a close, please let us know if anything remains unclear.
Response
Thanks to the authors for responding to my questions. I maintain my original score.
Author's Response (Part 1)
We thank the reviewer for highlighting the interference trade-off and the importance of descriptor design. We have clarified how descriptor quality affects recall and added the rationale for the VGG-19 features.
Proposition 3 shows that a write changes another memory read through both descriptor-key overlap and the projected write error. Interference therefore depends not only on whether conditions have similar descriptors, but also on the adjustments being written. Proposition 5 separates addressing limitations from insufficient rank: when identical keys require incompatible adjustments, increasing the number of shared directions alone cannot distinguish them.
Our five-seed address-geometry experiment evaluates the effect of descriptor overlap on CIFAR-100-C / ResNeXt-29 / EATA. The following results report re-entry error at recurrence 20, as mean ± sample SD across five paired seeds (Table 3 and Appendix Table 28).
The default VGG descriptors achieve a mean re-entry error 0.86 percentage points above the orthogonal-key control. The latter uses domain identities and serves as a diagnostic reference rather than a deployable descriptor. Deliberately correlating the keys raises error substantially, demonstrating the importance of distinguishing conditions that require different adjustments.
The capacity experiment provides a complementary comparison. Increasing the rank cap from 8 to 64 reduces re-entry error from 34.2% to 27.9% with VGG keys, compared with 38.5% to 36.8% with correlated keys (Table 11, averaged over recurrences 18–20). Additional capacity helps, but does not remove the addressing penalty in this experiment. These comparisons cover complete adaptation runs: changing the keys also affects subsequent host updates and basis growth.
We also added five-seed tests of order variation and descriptor perturbations under the batch-size-64 diagnostic protocol. These appear in Appendix D.6, Table 17, and in Table A2 of our response to Reviewer bz2v (Q2).
Random corruption-order permutations at each recurrence change full-carry ENGRAM's re-entry error from 27.88% to 28.06%, an increase of 0.18 percentage points. At descriptor-perturbation norm 0.40, ENGRAM obtains 28.42% re-entry error and 27.91% visit-average error, compared with 32.74% and 29.68% for ReservoirTTA. The corresponding advantages are 4.32 and 1.77 percentage points. Full carry achieves the lowest mean errors on both metrics across all seven tested conditions.
These results show that ENGRAM's advantage persists under the tested order changes and descriptor perturbations, while the correlated-key experiment identifies where condition-specific recall deteriorates. Shared directions also permit reuse across conditions: Figure 6 shows directions used by multiple weather conditions, and Table 27 reports read-only evaluation on ten unseen corruptions. These observations complement the interference experiments by illustrating how learned adjustments can be reused across conditions. Residual compensation preserves the current adapted parameters, while condition-specific recall depends on descriptor quality and the learned memory.