- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper argues that visual representation learning should rely not only on learned transformations but also on persistent visual patterns that can be selectively. To address this, this paper proposes ReMeViT, a novel memory mechanism for visual backbones that stores and reuses prior visual experience, such as textures, edge configurations, and part-like structures, across network stages and training phases. ReMeViT consists of Local Query Encoding, Visual Pattern Recall, and Memory Recontextualization, forming a comprehensive and coherent working-memory mechanism for retaining, retrieving, and recontextualizing visual patterns. Extensive experiments across image classification, semantic segmentation, object detection, and other settings, together with in-depth analyses of network depth and individual components, demonstrate the effectiveness of ReMeViT and its design choices.
Critical strengths
-
The proposed visual pattern memory mechanism provides a principled way to reuse visual information across network stages, going beyond conventional feature transformation and aggregation, and represents the paper's primary conceptual contribution.
-
The method is evaluated across diverse vision tasks, including image classification, semantic segmentation, and object detection. The reported improvements, together with ablation studies and analyses across network depths, provide evidence for both the effectiveness of ReMeViT and the contributions of its individual components. This breadth of validation strengthens confidence in the practical value of the proposed approach.
Critical weaknesses
-
Although ReMeViT combines sparse top-k memory retrieval, context-conditioned gating, and cross-depth memory sharing in a coherent framework, these components are related to existing key-value memory mechanisms[1, 2]. This paper could more convincingly isolate the fundamental novelty of this combination and establish its advantages over existing memory designs. This limits the strength of its conceptual contribution.
-
ReMeViT is evaluated primarily on DeiT and Swin, leaving its compatibility with a broader range of visual architectures insufficiently explored. Evaluations on additional backbones, such as CNNs and SSMs, would help establish the generality of the proposed memory mechanism and clarify whether its benefits extend beyond the specific architectures studied. This limits confidence in the broader applicability of the approach.
[1] Lample G, et al. Large memory layers with product keys [2] Miller A, et al. Key-value memory networks for directly reading documents
Based on the above, the author should clearly explain the difference between ReMeViT and existing key-value memory methods[1, 2], and validate the effectiveness of ReMeViT on additional backbones like CNNs and SSMs.
Questions for authors
-
Could the authors provide a detailed complexity analysis of ReMeViT, including the computational and memory costs of local query encoding, top-k memory retrieval, and memory recontextualization, to substantiate its efficiency and scalability claims?
-
How do the authors ensure that jointly learning the memory bank and retrieval mechanism does not introduce unstable or noisy memory responses during early training? Is there empirical evidence regarding memory behavior and training stability at different stages?