acceptodds
Withdrawn from ICLR 2027

Persistent Memory for Visual Recognition under Corruption and Missing Evidence

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

14 views

Abstract

Can persistent memory support visual recognition when the input is corrupted or incomplete, particularly when labeled training data are limited? We investigate this question through the Vision Hopfield Memory Network (V-HMN), a model that integrates associative retrieval with iterative feature refinement across layers. Separate local and global memory banks store training-derived representations of patch neighborhoods and image-level context. During inference, each branch retrieves a weighted combination of stored patterns and uses the discrepancy between the retrieved and current representations to guide feature updates. This design makes information from training examples available through both learned parameters and persistent representations, while keeping the memory fixed at evaluation. The retrieved entries and their weights also make the memory component of inference inspectable. We study this mechanism through common image corruptions, controlled occlusion and masking, and training with limited labels, restricting memory construction to the declared training budget. Comparisons with the same architectural scaffold trained without retrieval, alongside interventions that disable reads or alter memory contents, distinguish the effects of learning with memory from those of accessing it during inference. Our analyses examine whether memory contributes beyond clean recognition performance, how its contribution changes with label availability, and when retrieval corrects predictions or introduces errors. By connecting these outcomes to local and global retrieval, memory coverage, and the entries used during computation, we investigate both the potential and the limitations of persistent memory for recognition under degraded evidence.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Associative memory, Modern Hopfield networks, Brain-inspired machine learning, Memory-augmented neural networks

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_uYXwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors introduce a new architecture which stores local and global representations inferred from training data. These stored representations serve as anchoring devices for predicting new data. This appears to help in the task of learning in low-data regimes.

Critical strengths

The results on table 3 suggest the authors have found a promising research direction.

Critical weaknesses

The presentation quality is not of a publishable standard. The paper lacks a coherent narrative structure. There should be a clear path from the problem introduction and motivation, current state of the problem, the paper’s contribution and claims, and evidence for those claims. Instead, the paper reads as an endless list of disjoint experiments with no clear takeaway for the reader. The writing style at the fine-grained level is extremely dense, difficult to parse, and often incoherent.

The writing quality suggests blatant mis-use of AI. I am not confident that the authors read their own manuscript before submission.

The results are underwhelming. Assuming the purpose is to show that the method (which is never explicitly stated as the contribution) does better on classification, the results in table (1) show that this method is beaten by ResNet-18. The results in table 3 are promising. But, why is ReseNet-18 not used there?

Dubious design decisions are not discussed My understanding is that the slots are filled with a FIFO queue of images. Does that mean during inference, the slots are taken up by whatever the last k images were seen during training? What are the consequences of just using the last k images you saw? Why not learn those memories in Hopfield fashion?

The Hopfield framing is misleading. The FIFO queue of slots does not reflect any sort of associative memory. The way in which the slots are used does not indicate an energy function, which the authors confer.

Questions for authors

I would not improve my score without a complete rewrite of the paper. I have expanded on my concerns with the writing below so that the authors may use this feedback in their next draft.

The poor structure of the paper is reflected in the abstract of the paper. The abstract is an endless list of passive research-coded phrases: “[We] experiment x”, “[we] investigate x”, “[we] examine x”, “the analysis …”. None of these statements constitute a contribution, claim, or a result. A contribution will begin with a phrase like "we introduce x". A claim has active phrasing like “x improves y”. If a number is included, that is a result which can be used to support the claim.

The method is explained in an flood of math without context or motivation. There is far too much shorthand and terms are used before they are defined.

Experimental setups and results are reported robotically while their motivation and consequences are under-discussed. Every question posed should have a clear answer. Instead of a conclusion, for some reason, every experimental subsection ends with disclaimers. Untethered disclaimers do not count as discussion.

The discussion and conclusion section contains neither a discussion nor a conclusion.

The authors use multi-syllabic phrases which are inappropriately stitched together into unwieldy sentences. I would advise against the authors using semi-colons in their next draft. Too much jargon is freely used without definition. Paragraphs contain unrelated sentences stacked on top of each other, rather than being complete modular blocks with singular arguments.

I would encourage the authors to write the first draft without any AI assistance.

Reviewer_7KGiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents an interesting alternative architecture in which stored representations of previous training examples are retrieved to modify ongoing visual processing. Each layer has access to a local memory bank containing image-patch features and a global memory bank containing whole-image features. These memories are used to refine current features, potentially improving recognition under image corruption or missing visual information. While the model does not consistently outperform a matched network without memory, this does not necessarily diminish its value as an alternative computational account of visual recognition. The use of persistent associative memory and iterative feature refinement is potentially relevant to biological perception, where recognition may involve interactions between current sensory input and stored experiences.

Critical strengths
  1. An interesting architectural idea, although previous work has explored combining Hopfield networks or associative prototype memories with CNNs for object recognition.
  2. Fairly comprehensive and careful evaluations across different datasets and corruption conditions, including comparisons with networks trained without memory and ablations of the retrieval process.
  3. The analyses of local and global memory contributions, and of selective versus nonselective retrieval, provide some insight into how memory influences recognition.
Critical weaknesses
  1. As a method for improving robustness, the evidence is relatively weak. As an investigation of memory-based visual computation, the paper is more interesting and potentially worthwhile. However, it remains unclear whether memory retrieval supports a qualitatively different computational mechanism, such as reconstructing missing information, or primarily serves as an alternative way of transforming features. A stronger demonstration of the distinctive functional role of memory would substantially increase the significance of the work.
  2. The results do not consistently support the claim that persistent memory improves robustness. On CIFAR-100 and Tiny ImageNet, the matched scaffold trained without memory actually performs slightly better. The ImageNet result is encouraging, but V-HMN is not compared against a matched ImageNet-scale scaffold using the same training recipe. Thus, it is difficult to determine how much of the improvement comes from the memory mechanism itself.
  3. V-HMN performs best among seven architectures in the fractional-data comparisons, but its advantage over the matched scaffold is small and tends to disappear at larger training budgets or with longer training. Given the additional memory storage and retrieval computations, the cost-benefit trade-off is not well established.
Questions for authors
  1. The model is conceptually reminiscent of several biological mechanisms, including associative memory, recurrent processing, and interactions between sensory inputs and stored experiences. Have the authors considered how V-HMN might relate to biological visual processing, and whether it offers any insights into the computational role of memory in perception?
  2. Have the authors explored potential benefits of V-HMN beyond classification accuracy? For example, explicit memory could potentially support rapid incorporation of new experiences or provide insight into which previous experiences influence recognition. The paper already includes some retrieval analyses, but it would be interesting to know whether these capabilities offer practical or computational advantages over conventional networks.
Reviewer_NMXnOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper use two separate local and global path and add memory banks to each of the path in a vision model. The local path handles patches (a small part of the image), while the global path handle the entire image itself. When a query comes in, the memory bank in each path responses by providing relevant representations and use them to refine itermediate features.

Critical strengths
  1. The separation of local and global memory makes sense. In most associative memory works (especially those focus in visions), it is usually that case that the whole memory pattern itself is been processed. So, most associative memory models are weak in scenarios where local feature is more distinguishable than global features.
  2. The comparison to various controls are meaningful. Including ablations/variants such as "Trained scaffold", "Uniformed trained", "Mean read" provides further insights of how memory banks participate in inference.
Critical weaknesses
  1. The main controlled comparisons do not favor adding the memory bank. As in Table 1, "Trained scaffold" (same structure, without memory bank, in my understanding) looks a bit better than "V-HMN" itself, and the "Uniformed trained" control is also competitive, so it seems like adding memory banks is not helping.
  2. Winning against "V-HMN bypass at inference" is not persuasive. In my understanding, bypass variant means that it loads a normal V-HMN checkpoint and disable the memory bank during inference. So, the model is trained to depend on the memory bank during training, and disabling the memory bank during inference would harm the performance for sure. So, a fairer comparison is to compare normal V-HMN with the "Trained scaffold", where inference are not taught to depend on the will-be-disappearing memory bank.
  3. In Table 4, V-HMN use 310 epoch steps, and RandAugment, MixUp, CutMix, etc. (a lot of optimizations), while ResNet-50 V1 only uses 90 epoch steps, and less optimization tricks. So, it is not sure if the comparison against ResNet-50 V1 is fair enough or not. Thus, the gain cannot be attribute to adding the memory banks alone.
  4. The motivation of adding memory banks are missing. Although the idea of local / global path is meaningful to some extent, but the reasoning why the structure is designed in this way is missing in the manuscript.
  5. In Related Works, existing work [1][2][3] that also take advantage of local and global memory should also be included and discussed. Also, remember to cite the Hopfield network paper [4] as well.
Questions for authors

See the weaknesses.

References

[1] Krotov, D., 2021. Hierarchical associative memory. arXiv preprint arXiv:2107.06446.

[2] Alonso, N. and Krichmar, J.L., 2024. A sparse quantized hopfield network for online-continual memory. Nature Communications, 15(1), p.3722.

[3] Wang, S., Pan, Y., Shen, Z., Zhang, M., Wang, H. and Li, G., 2026, April. Adaptive hopfield network: Rethinking similarities in associative memory. In International Conference on Learning Representations (Vol. 2026, pp. 116942-116974).

[4] Hopfield, J.J., 1982. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(8), pp.2554-2558.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.