acceptodds
Withdrawn from ICLR 2027

Learning to See through Mnemonic Recall

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Modern visual backbones largely follow an operator-centric view of representation learning, where prior experience shapes the transformations applied to inputs. Yet experience can also contribute as persistent visual content that is selectively recalled to inform current perception. Inspired by memory-based reinstatement in human perception, we formulate visual representation learning as the joint contribution of dynamic computation and conditional recall. This work introduces ReMeViT, a family of visual Transformer backbones equipped with visual pattern memory that retains recurring local patterns as persistent and addressable states. Current visual cues can selectively evoke relevant patterns, while context-conditioned gating adapts their contribution to the current observation. The separation of stored content from contextual use also allows a shared source of visual experience across different backbone architectures. We instantiate ReMeViT on both DeiT and Swin, which yield consistent performance gains across classification and dense prediction tasks. Further results support visual pattern memory as a source of reusable perceptual priors that benefit new tasks, strengthen recognition under occlusion, and enhance representations learned without supervision.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
visual representation learning, visual backbone, visual memory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VqdyOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper argues that visual representation learning should rely not only on learned transformations but also on persistent visual patterns that can be selectively. To address this, this paper proposes ReMeViT, a novel memory mechanism for visual backbones that stores and reuses prior visual experience, such as textures, edge configurations, and part-like structures, across network stages and training phases. ReMeViT consists of Local Query Encoding, Visual Pattern Recall, and Memory Recontextualization, forming a comprehensive and coherent working-memory mechanism for retaining, retrieving, and recontextualizing visual patterns. Extensive experiments across image classification, semantic segmentation, object detection, and other settings, together with in-depth analyses of network depth and individual components, demonstrate the effectiveness of ReMeViT and its design choices.

Critical strengths
  1. The proposed visual pattern memory mechanism provides a principled way to reuse visual information across network stages, going beyond conventional feature transformation and aggregation, and represents the paper's primary conceptual contribution.

  2. The method is evaluated across diverse vision tasks, including image classification, semantic segmentation, and object detection. The reported improvements, together with ablation studies and analyses across network depths, provide evidence for both the effectiveness of ReMeViT and the contributions of its individual components. This breadth of validation strengthens confidence in the practical value of the proposed approach.

Critical weaknesses
  1. Although ReMeViT combines sparse top-k memory retrieval, context-conditioned gating, and cross-depth memory sharing in a coherent framework, these components are related to existing key-value memory mechanisms[1, 2]. This paper could more convincingly isolate the fundamental novelty of this combination and establish its advantages over existing memory designs. This limits the strength of its conceptual contribution.

  2. ReMeViT is evaluated primarily on DeiT and Swin, leaving its compatibility with a broader range of visual architectures insufficiently explored. Evaluations on additional backbones, such as CNNs and SSMs, would help establish the generality of the proposed memory mechanism and clarify whether its benefits extend beyond the specific architectures studied. This limits confidence in the broader applicability of the approach.

[1] Lample G, et al. Large memory layers with product keys [2] Miller A, et al. Key-value memory networks for directly reading documents

Based on the above, the author should clearly explain the difference between ReMeViT and existing key-value memory methods[1, 2], and validate the effectiveness of ReMeViT on additional backbones like CNNs and SSMs.

Questions for authors
  1. Could the authors provide a detailed complexity analysis of ReMeViT, including the computational and memory costs of local query encoding, top-k memory retrieval, and memory recontextualization, to substantiate its efficiency and scalability claims?

  2. How do the authors ensure that jointly learning the memory bank and retrieval mechanism does not introduce unstable or noisy memory responses during early training? Is there empirical evidence regarding memory behavior and training stability at different stages?

Reviewer_FFqbOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes new architecture for visual representation learning. The method uses external memory bank where input query can leverage stored knowledge of visual patterns. The memory block is attached after transformer blocks and the memory is shared across layers with layer-specific query encoders. The method is evaluated on image classification, semantic segmentation, and object detection tasks.

Critical strengths
  • Clean structure. The paper is generally clearly written and well organized. It was easy to follow and capture the contribution of this paper.

  • Sufficient experiments. The evaluation covers three key vision tasks, and covered diverse experiments that show the method improves the performance.

Critical weaknesses
  • Lack of justification for method. I don't fully understand how can external memory be helpful compared to standard attention and feedforward network. What is stored in the memory and how is it beneficial compared to parameterizing it into transformer blocks? Section 4.4 is the closest result from the paper, but it is quite short and it was unclear for me how to read the Figure 4. Could the authors elaborate more on this?

  • Missing explanations on some results. In Table 4, random memory memory shows better than no memory. If the model is leveraging something from memory, and if the memory is useless, it can be expected to perform poor than without memory. But the result is showing that there is still a performance gain. This makes me wonder how significant the performance gain is in other tables as well. In Table 11, ImageNet result gets better with larger #M, but line 471 says it reaches the best at M=1024.

Questions for authors
  • What is the motivation of using convolution network for query encoding and recontextualization, instead of projection matrix like in attention block?
  • The architecture comes at a cost of external memory. Considering the performance gain and resource tradeoff, in which use cases this method can be useful in the real world?
  • Typo in line 199: "In contrast, Our" -> "In contrast, our"
Reviewer_MTRGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ReMeViT, a novel ViT archtecture aiming to leverage learned visual patterns. The core argument is that ViT should not only remember learned transformations but also learned visual patterns. The authors propose to use local patch features to retrieve the top-k relevant entries from a shared learnable memory bank, and then through residual connections, applies context-conditioned gating to integrate the retrieved visual patterns into the ViT. Experiments on various datasets highlight the performance advantage of this proposed archtecture.

Critical strengths
  1. The architecture is coherent and reasonably straightforward, the motivation makes sense in my opinion
  2. The authors cover a wide range of evaluations and performance improvement is clearly seen on almost all of them.
Critical weaknesses
  1. This work's core contribution and idea looks and reads quite similar to Engram [1], both using persistent memory retrieval followed by contextual gating. It seems like an adaption to the ViT domain thus limiting novelty.
  1. The experiments do not isolate whether separate memory descriptors and cross-layer memory sharing actually outperform simpler gated key–value memory alternatives. I recommedn the authors to consider two more ablations 1) replace the separate memory descriptors with retrieval keys (D=R), keeping the same gate and 2) compare shared vs. layer-specific memory banks using identical gating and matched total parameters.

[1] Cheng et. al., Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models

Questions for authors

Please see weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.