acceptodds
Under review as a conference paper at ICLR 2027

ENCORE: A Biologically Plausible Encoding Optimization for Online Associative Memory

Abstract

Episodic memory requires rapid, one-shot encoding of experiences as they arrive, yet associative memory models such as Hopfield network and their modern variants (Modern Hopfield Network) are typically formulated and tested in the batch setting, where all patterns are stored simultaneously. We study the online associative memory problem, in which patterns arrive sequentially and must be encoded via one-shot exposure. Prior works have improved capacity along two axes: retrieval dynamics (e.g., Modern Hopfield Network) and learning rules (e.g., Storkey rule). Inspired by the hippocampus architecture and the neuronal memory allocation mechanism in the brain, we explore a third, orthogonal axis: optimizing the encoded patterns themselves. We propose an associative memory system in which the encoding dynamics are explicitly designed to minimized interference with previously stored encoded patterns. On online associative memory benchmarks, our system outperforms both Modern Hopfield networks and Hopfield network with Storkey learning rule by more than 5x in MNIST and 2x in ImageNet 128x128. These results suggest that pattern encoding, long overlooked relative to retrieval dynamics and learning rules, is a significant lever for designing associative memory in realistic online settings.

Primary area:
applications to neuroscience & cognitive science
Keywords:
online associative memory, neuronal memory allocation, encoding optimization, hippocampus pathway

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BSM5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ENCORE, an online associative memory system that optimizes each incoming pattern’s internal encoding before storage. Inspired by hippocampal memory allocation, it combines dimensionality expansion and sparse coding with iterative encoding optimization that balances input fidelity, neuron usage, and interference with existing memories. The selected codes are stored using local learning rules. Experiments on synthetic patterns, MNIST, and ImageNet latents report increased online retrieval capacity, while ablations indicate complementary benefits from usage balancing and interference reduction.

Critical strengths
  1. The paper explicitly incorporates memory history into the selection of new codes, balancing input fidelity, neuron usage, and interference. This provides a useful perspective complementary to improvements in retrieval dynamics and storage rules.
  2. The design connects dimensionality expansion, sparse coding, adaptive neuron allocation, and recurrent pattern completion. The encoding objective has interpretable computational components, while Hebbian and delta-rule updates establish associations between inputs and selected code.
  3. The expansion-only control helps isolate the benefits of encoding optimization. Additional analyses of neuron usage and code overlap provide evidence for the distinct, complementary roles of allocation balancing and interference reduction.
Critical weaknesses
  1. The MHN comparison does not fully establish the scope of the claimed advantage. The baseline updates its memory matrix through backpropagation using only the current sample, making it susceptible to forgetting. This evaluates a particular online learning strategy rather than MHN retrieval more broadly. Stronger online MHN baselines under an explicit storage budget, or clearer justification of the chosen protocol, are needed to support the comparison.
  2. The computational and storage costs of the capacity gains are insufficiently characterized. ENCORE introduces iterative encoding search and additional encoding and decoding matrices. Although the expansion-only ablation supports the value of encoding optimization, the paper does not report runtime, memory usage, or sensitivity to the iteration budget. It therefore remains unclear whether the advantage persists under comparable resource budgets.
Questions for authors
  1. The encoding ablations provide promising evidence. Could the authors justify the current-sample-only MHN training protocol and test whether the gains persist against stronger online MHN baselines under a matched storage budget?
  2. The capacity gains are substantial. What additional encoding time and storage do they require, and how does the capacity–cost tradeoff change with the encoding iteration budget?
1 comment
AuthorsOpenReview ↗

Justification for the MHN protocol, added stronger baselines, iteration budget sweep and matched storage capacity

Thank you for the detailed comments. We have added stronger online MHN baselines with explicit storage budgets, a cost analysis and a sweep of the iteration budget.

Q1) Justify the current-sample-only MHN protocol

Please see Section 2.1 of the Author Response.

Q2) Test stronger online MHN baselines under a matched storage budget

Please see Section 2.2 of the Author Response.

Q3) What additional encoding time and storage do the gains require, and how does the capacity–cost trade-off change with the iteration budget?

  1. Storage and time (Sec. 4.4; App. G, Tables 3–5). ENCORE stores 45.0M learned and 40.3M fixed weights (341 MB), against 4.0M for a Hopfield network on the input patterns and 10.0M for the bank. A write takes 2.7 ms at Tγ=1T_\gamma = 1, against 0.09 ms (Hebbian), 0.33 ms (Storkey), 0.03 ms (bank) and 100–180 ms (gradient write); retrieval takes 1.05 ms against 0.64–0.85 ms. Most of this cost belongs to the expand-and-sparsify architecture. The encoding optimization step (Algorithm 1) adds no weights (it reuses W and keeps one usage value per CA3 unit), 0.5 ms per iteration to a write, and nothing to retrieval.
  2. Iteration budget (App. H, Table 9). Sweep on the three synthetic families, 5 seeds:
Tγ0125 (paper)1050
m*, pattern retrieval (mean of families)267380038024793979547866
m*, code retrieval (random)261494488991885988108784
Write time (ms)2.202.703.204.677.1026.4
Share of the total decrease of the objective0%96.2%98.8%99.6%99.7%99.7%

One iteration is sufficient. Pattern-retrieval capacity varies by at most 5% for Tγ≥1T_\gamma \geq 1, and code-retrieval capacity is highest at Tγ=1T_\gamma = 1. A larger budget adds time and no capacity, so the best trade-off is Tγ=1T_\gamma = 1:. The same holds on MNIST and ImageNet latents (Table 6). All initial experiments used Tγ=5T_\gamma = 5, the paper now mention this in Sec. 2.4.

Q4) Does the advantage persist under comparable resource budgets?

Please see Section 3.4 of the Author Response.

Thank you for your constructive feedback and informative review! Please let us know if there's anything that you'd like to discuss!

Reviewer_MfpoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ENCORE, a biologically inspired online associative memory framework that optimizes memory representations during encoding. It combines dimensionality expansion, sparse neuronal allocation, and interference-aware optimization to improve memory capacity. Experiments on synthetic patterns, MNIST, and ImageNet latents show substantial capacity improvements.

Critical strengths

S1. Optimizing memory representations at encoding time, rather than focusing exclusively on retrieval dynamics or learning rules, is a meaningful direction for online associative memory.

S2. The reported capacity improvements are substantial. Importantly, the authors include a same-expansion Hopfield baseline, providing evidence that encoding optimization contributes beyond dimensionality expansion alone.

Critical weaknesses

W1. Baseline fairness and resource accounting. The Modern Hopfield baseline is trained only on the current example through backpropagation, resulting in severe forgetting. This implementation may not adequately represent stronger online associative memory alternatives. Furthermore, the expanded encoding and recurrent matrices introduce substantial storage overhead, but capacity comparisons are not normalized by total memory usage.

W2. The component ablations do not cleanly isolate the proposed mechanisms. Substituting Eq. (4) into Eq. (5) shows that the effective coefficient of neuronal usage is α−β. Therefore, setting α=0 for the "interference-only" ablation actually reverses the usage preference, rather than simply removing allocation optimization. This confounds the interpretation of Figures 5 and 6.

Questions for authors

How does ENCORE compare with stronger online memory baselines under matched total storage and computational budgets?

Can the authors repeat the ablations by independently controlling the effective usage coefficient (α−β) and the interference coefficient β?

1 comment
AuthorsOpenReview ↗

Added stronger Hopfield baselines, matched storage capacity comparison, decoupled ablation analysis.

Thank you for the careful reading, in particular of the ablation study. We have added stronger online baselines, comparisons at identical and at matched storage, and an ablation with independent coefficients.

Q1) The backpropagation-trained MHN may not represent stronger online associative memories.

Please see Author Response, Section 1 for the Scope Clarification and Section 2 for Experiments on Additional Baselines.

Q2) Capacity is not normalized by total memory usage. How does ENCORE compare under matched storage and computational budgets?

Please see Author Response, Section 3 for general storage and computational cost analysis and Section 3.4 for the matched storage comparison.

Q3) Substituting Eq. (4) into Eq. (5) gives an effective usage coefficient α − β, so α = 0 reverses the usage preference. Can the ablation be repeated with independent coefficients?

You are right about the equations as written, and we have reformulate the equation such that the impact of each component can be fully isolated and re-run the ablation on fully isolated component.

  1. The initial equation does not include the normalization terms. Our implementation rescales each history-dependent term to the norm of the input drive, which Eq. (6) did not state. With this rescaling the effective usage coefficient of the full model was α−βρ\alpha - \beta\rho, where ρ=∥u∥/∥Wη−u∥\rho = \lVert u\rVert / \lVert W\eta - u\rVert falls from 0.27 to 0.03 as patterns accumulate: 0.96–0.996, not 0.85 (App. I). In the interference-only arm the usage was also removed from the recurrent term, so its usage coefficient was exactly 0, not reversed.
  2. Decoupled formulation. The objective and the drive now have independent coefficients λa\lambda_a (allocation) and λi\lambda_i (interference), with the normalization written out (Eqs. 5–6). We re-ran the full ablation with this formulation (Sec. 4.3, Figs. 5–6).
(λa,λi)RandomCorrelatedLow-rank
No encoding optimization (0, 0)4931 ± 514386 ± 343659 ± 41
Allocation only (1, 0)6670 ± 796335 ± 885869 ± 80
Interference only (0, 0.15)7041 ± 596417 ± 165476 ± 46
ENCORE (1, 0.15)8173 ± 628078 ± 697457 ± 91
  1. The initial conclusions from the ablation study still hold. The full model exceeds both single-term models in every pattern family and every seed, and each single term adds 35–60% over no optimization. The diagnostics of Fig. 6 also hold: the allocation term keeps per-neuron usage balanced (s.d. 0.39 against 24.1 without it, at 5,000 patterns), and the interference term lowers the overlap of a new code with its nearest stored code (0.087 against 0.094). The decoupled model agrees with the earlier parameterization, which Figs. 2–4 use, within about two standard errors (Table 10).
  2. Correction of the (0, 0) control. In the initial submission the "Expansion-only" model held about 630 patterns because of a tie-breaking bug. With the corrected value above, the full optimization gains 1.7–2.1× on synthetic patterns.

Thank you for your careful reading and informative review! Please let us know if there's anything that you'd like to discuss!

Reviewer_rCQPOpenReview ↗
Rating
3(+1)
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This work improve the memory mechanism in online streaming setting by using an improved, better internal coding and brain-inspired memory processing work flow. The proposed framework ENCORE uses a structure that is inspired by the hippocampus so that it expands input into an optimized high-dimensional sparse representation, and apply local learning rule to better bridge the input and the internal representation.

Critical strengths
  1. The idea is interesting, internal representation is an area that is long omitted by the field of associative memory. Formerly, associative memory studies focus on how to improve the separation and the similarity function while letting the internal representation being the input itself. I believe that using a more regulated and controlled internal representation would gain extra expressiveness compared to using the input directly.
  2. The experimental results in this work is clear. There is clear advantage compared to the baselines, and the ablation studies well supports the claim of the study.
Critical weaknesses
  1. The connection with the biological hippocampus is weak. This is hard to tell how the current framework is similar to how hippocampus actually is. The author should consider how strong the connection between current architecture and the hippocampus is, and reconsider to what extent should the analog be claimed.
  2. The Hopfield baselines is insufficient in the experiments. Modern Hopfield networks operate over an pattern "bank" (similar to the tokens/KV cache of the softmax attention), so they do not necessitates the "overwriting" as they can simply append new things to the back. The current baseline are too weak, the Hopfield networks should include a more modern version [1][2]; Also, a online memory should also be included as a stronger baseline [3].
  3. The representation expansion is helpful, but the analysis of time/space complexity is also needed. Detailed efficiency measurements such as parameter counts, memory storage bytes, write time, and retrieval time are needed for a convincing comparison.
Questions for authors

See the weakness.

References:

[1] Hu, J.Y.C., Yang, D., Wu, D., Xu, C., Chen, B.Y. and Liu, H., 2023. On sparse modern hopfield model. Advances in neural information processing systems, 36, pp.27594-27608.

[2] Wang, S., Pan, Y., Shen, Z., Zhang, M., Wang, H. and Li, G., 2026, April. Adaptive hopfield network: Rethinking similarities in associative memory. In International Conference on Learning Representations (Vol. 2026, pp. 116942-116974).

[3] Alonso, N. and Krichmar, J.L., 2024. A sparse quantized hopfield network for online-continual memory. Nature Communications, 15(1), p.3722.

4 comments
AuthorsOpenReview ↗

Narrow biological claims, added stronger Hopfield baselines, and storage and computational cost analysis

Thank you for the constructive comments and the relevant references! Based on your suggestion, we have narrowed the biological claims, added a memory-bank MHN and additional baselines, and ran a full cost analysis on storage and computation.

Q1) The connection with the hippocampus is weak. The authors need to reconsider to what extent the analog should be claimed.

We agree that we need to be more specific about the level of biological analog to the hippocampus in the claim. The revision has separate three levels of analog (architecture, functionality, physiology):

  1. Architecture analog from the hippocampus anatomy. Our architecture is based on the hippocampus anatomy, including the sparsity level, the 10x expansion from EC to DG, the sparse DG → CA3 mossy fiber projection, and the recurrent CA3 network (Andersen 2006).
  2. The terms in the encoding objective is functional analogy, not biological mechanisms. The terms of the encoding objective (usage and homeostatic excitability, interference and feedback inhibition) is motivated by actual biological process observed in the brain (see Table 1 in Appendix D for references). However, we do not claim that our memory encoding procedure is actually performed by the hippocampus in the biological memory encoding process.
  3. No physiology modeling. We do not claim ENCORE models hippocampal physiology.

The contribution is now stated as "hippocampus-inspired architecture with local one-shot learning" (Sec. 1), and "biologically plausible" refers only to the first level.

Q2) The Hopfield baselines are insufficient: MHNs operate over a pattern bank and need not overwrite; include more modern versions [1][2] and an online memory [3].

Please see Author Response, Section 1 for the Scope Clarification and Section 2 for Experiments on Additional Baselines.

Q3) An analysis of time/space complexity is needed: parameter counts, storage bytes, write time and retrieval time.

Please see Author Response, Section 3 for Analysis on storage and computational cost.

Thank you for your informative review and constructive feedback! Please let us know if there's anything that you'd like to discuss!

References

  • Andersen, P., Morris, R., Amaral, D., Bliss, T. & O'Keefe, J. (eds.) (2006). The Hippocampus Book. Oxford University Press.

I appreciate the effort made by the authors, and I will raise my score. My concerns has been addressed mostly, and I agree with the author that we cannot compare slot/distributed memory directly as they have different time and space complexity, and I appreciate that the author have evaluated MHNs with the same settings.

One more thing I am curious about is how do ENCORE compare with a new distributed memory that improve the classic Hopfield network: [4] https://www.nature.com/articles/s41467-026-78777-2. The outcome doesn't matter, as this is a concurrent work, and I am just curious about the result.

Best regards,

References

[4] Blümel, M., Schneider, A. C., Neuhaus, V., Ehrlich, D. A., Graetz, M., Wibral, M., ... & Priesemann, V. (2026). Redundancy maximization as a principle of associative memory learning in Hopfield networks. Nature Communications, 17(1), 11499.

Thank you for your response! We appreciate your time reading the rebuttal and glad that it has addressed most of your concerns! We thank you for pointing us to the recent related work [4] (Redundancy maximization as a principle of associative memory learning in Hopfield networks, RedMax). We have reproduced the RedMax results and tested it in our setting! Below we outline the comparison between RedMax and ENCORE, and showed that ENCORE outperformed RedMax in the online setting with synthetic data.

RedMax Summary. The paper analyze the mutual information between the input and output variables in classical Hopfield network using Partial Information Decomposition (PID) to decompose them into redundancy (shared information between recurrent and target signal), unique, and synergetic component, then observed that in the regime below the capacity threshold, redundancy components dominate and collapse in the regime above capacity. Motivated by this observation, the paper develop a training framework to maximize this redundancy component, and report capacity increase to α=1.59\alpha=1.59 (from α=0.14\alpha=0.14 in the case of classical Hopfield network) in random dense synthetic patterns setting.

How RedMax relate to ENCORE. RedMax works in the same distributed memory representation as ours. While RedMax increase the memory capacity via improve learning rule via the proposed objective of maximizing redundancy, ENCORE increase the memory capacity by optimizing which encoded pattern to store. Another important distinction is that RedMax works on the batch setting, the model weights are trained with all patterns available for 5000 epochs (Supplementary Table 1: The model parameters). ENCORE works on the online setting, in which each pattern is only seen once. Below list the comparison for ENCORE and RedMax.

RedMaxENCORE
Improvement AxisLearning rule: propose a new learning objectiveEncoding: which code is stored.
Memory typeDistributed, N² weightsDistributed, expanded CA3 plus maps
Batch vs OnlineBatch, about 5,000 passes over all patternsOnline, one exposure
Patterns testedDense random ±1, N = 100.Sparse binary, correlated, low-rank, MNIST images, ImageNet latents
Capacity testTest for fixed point: Initialize as stored pattern, test whether the pattern stay at fixed point (p2, Section “Redundancy as a computational goal”)Test for pattern completion: Initialize from 25%-corrupted cue (partial cue for synthetic patterns; pixel drop, noise, right mask for image patterns).

Experiments. We use the authors' released code and reproduced the results at the paper’s setting (100 neurons, dense random patterns gives a capacity of 1.55–1.60N (reported: 1.59N)). We then ran it in our synthetic setting (2,000 units, 5% activity, 4.0M learned weights as for the Hebbian and Storkey networks), with retrieval from 75% partial cues as in our paper, in two forms:

  1. Online. Patterns arrive in chunks of 100 and 200 (since the redundancy component cannot be computed with a single pattern). Each chunk is trained for 50 steps and is never revisited.
  2. Batch, as published (reference). At every load a new network is trained on all patterns stored so far, for up to 2,000 epochs.
Model (4.0M learned weights unless noted)RandomCorrelatedLow-rank
Online write
RedMax, chunks of 100891424254
RedMax, chunks of 200812588804
Hopfield, Storkey172113731209
ENCORE at 4.0M learned weights940929914
ENCORE (45.0M learned weights)819480127490
Batch training (reference)
RedMax39824983≥10000

One modification was needed. With the published settings (±1 states, learning rate 0.05) the rule did not train on 5%-sparse patterns at 2,000 units, because the recurrent inputs diverge. We therefore present the presynaptic states as 0/1 instead of ±1 and tune the learning rate of the online form

In the online setting, RedMax holds 254–804 patterns, against 7,490–8,194 for full ENCORE and 914-940 on matched weight ENCORE. Even when all patterns can be revisited, RedMax still underperformed ENCORE in the online setting in the random and correlated patterns, but outperformed in the low-rank patterns. Because RedMax changes how the weights are learned and ENCORE changes which codes are stored, the two can be combined, for example by training ENCORE's recurrent weights with the rule of RedMax during offline consolidation. We will discuss RedMax in the related work and add this comparison to the appendix.

Reference

  • [4] Blümel, M., Schneider, A. C., Neuhaus, V., Ehrlich, D. A., Graetz, M., Wibral, M., Makkeh, A. & Priesemann, V. (2026). Redundancy maximization as a principle of associative memory learning in Hopfield networks. Nature Communications, 17(1), 11499.

I thank the author for comparing ENCORE with RedMax and fulfilled my curiosity. The performance of ENCORE looks really promising, and I would increase my confidence.

Comments on the paper

5 comments
AuthorsOpenReview ↗

Author Response (1/5)

We thank the reviewers for their detailed comments and constructive feedback. We're glad that the all three reviewers agree that our work on optimizing memory representations at encoding time (ENCORE) is an interesting idea and a meaningful direction for online associative memory ("The idea is interesting, I believe that using a more regulated and controlled internal representation would gain extra expressiveness" (rCQP), "meaningful direction for online associative memory." (Mfpo), "provides a useful perspective complementary to improvements in retrieval dynamics and storage rules" (BSM5)). The reviewers also think that the experiment results is clear, show clear advantage to the baseline, and the ablation studies support the claim for distinct role of each optimizing components ("The experimental results in this work is clear." (rCQP), "Additional analyses of neuron usage and code overlap provide evidence for the distinct, complementary roles of allocation balancing and interference reduction" (BSM5)).

However, the reviewers were concerned about the Modern Hopfield Network (MHN) baselines and the storage and computational cost of each components in our model. Specifically, reviewers suggested that the current MHN baseline, which is trained via back-propagation on the current sample, is too weak, and suggest including the following baselines: (1) Family of memory-bank MHN, which store memories in a pattern bank, with one row per memory (rCQP). This family includes standard MHN using softmax function for retrieval, sparse MHN using sparsemax function for retrieval [1], and adaptive MHN using an learned similarity function [2], (2) Family of online Hopfield networks, including Sparse Quantized Hopfield Network (SQHN) [3], which assign a new unit for new pattern if the best similarity score is below a time-dependent threshold. Furthermore, reviewers also have concerns about the storage and computational cost for the reported capacity gain (BSM5), and request measurements on retrieval capacity on matched total storage (Mfpo) and efficiency measurements such as parameter counts, memory storage bytes, write time, and retrieval time (rCQP).

We believe that the clarifications and additional experiments below, together with our per-reviewer replies, address these concerns. Section, figure and table numbers refer to the revised PDF, where changes are marked in blue.

AuthorsOpenReview ↗

Author Response (2/5)

1. Scope Clarification: Online Associative Memory with Distributed Memory Representation.

1.1. Distributed memory vs slot memory representation: Reviewers rCQP, Mfpo and BSM5 point out that a Modern Hopfield network can append each new pattern to a memory bank instead of overwriting earlier ones, and suggest such models as baselines [1, 2, 3]. We would like to clarify that these models and ENCORE address two different settings (slot memory vs distributed memory), a distinction that is well-characterized in the MHN literature (Krotov & Hopfield, 2021; Hoover et al., 2024).

  • Slot Memory Representation: Each memory is assigned to a single unit (or neuron). The pattern-bank MHN [1, 2] appends one row per pattern, and SQHN [3] recruits a new neuron for each novel pattern. Since memories do not share unit, they do not interfere, and the write operation does not need to decide which unit to write to reduce memory interference. In return, these models "require storing each pattern in a separate set of synaptic weights, which leads to the increase of the number of synaptic weights when new patterns are introduced" (Hoover et al., 2024), and with a fixed number of hidden units their capacity is bounded by that number, Nmem≤NhN_{mem} \le N_h (Krotov & Hopfield, 2021).
  • Distributed Memory Representation: Each memory is stored in many units, and each unit takes part in many memories. Hoover et al. (2024) characterizes distributed representation as "additional memories are stored in the already existing set of weights by redistributing the information about new memories across the already existing network parameters". The classical Hopfield network, the hippocampal CA3 circuit and ENCORE are in this category. Memories are distributed, share weights and therefore interfere, so capacity depends on which units in the existing neuron pool each new memory recruits. Therefore, ENCORE addresses the distributed setting. Our main question, how to choose the distributed code of a new memory given what is already stored, arises only in this setting.

1.2. Why distributed memory representation? Slot memory vs distributed memory is the classical debate in memory representation research. Compared to slot memory representation, distributed memory representations offer the following advantages:

  1. Fault tolerance and robustness: Since memory is stored in multiple units, the memory can still be recovered in the case of a single unit failure.
  2. Fixed-size storage and retrieval: In slot memory, the memory bank size and retrieval time grows linearly with the number of pattern stored. For distributed memory, the memory weights and retrieval stay fixed. Distributed Representation Dense Associative Memory, DrDAM (Hoover et al., 2024) was motivated for this reason.
  3. Potential for generalization and for storage beyond individual episodes: Kafraj et al. (2026) show that when hidden neurons encode "basic components shared across many memories", capacity can become "exponential in the number of hidden units", and the hidden code "preserves class-discriminative structure".
  4. The brain stores memory in distributed representations: In the brain, particularly the hippocampus, whose circuit ENCORE follows, a memory is held by a sparse ensemble of neurons (an engram) and not by a single cell (Josselyn & Tonegawa, 2020).
AuthorsOpenReview ↗

Author Response (3/5)

2. Experiments on additional baselines.

2.1. Justification for the back-propagation MHN baseline. Given that the setting for our main question is the distributed memory representation setting, the MHN with fixed-size weight matrix trained via backpropagation address the question that we’d like to investigate: how to learn the fixed-size distributed memory representation of associative memory models in an online setting? We also note that both the original papers for Dense Associative Memory (Krotov and Hopfield 2016) and Modern Hopfield Network (Ramsauer 2021) use backpropagation to train the memory weight (Krotov and Hopfield 2016, section 4, use BP to train the memory matrix K=2000 with MNIST train dataset; Ramsauer et al. (2021) HopfieldLayer is a learned memory matrix via BP). Multiple following works (Kafraj, Krotov & Latham (2026), Alonso & Krichmar (2024)) also use BP as the baseline approach to train fix-sized weight matrix in MHN.

2.2. Additional baselines for the distributed memory representation setting.

To address the reviewer’s concerns about backpropagation-trained MHN being a weak baseline, we’ve added two more baselines in the distributed memory representation setting, including (1) Distributed Representation Dense Associative Memory (DrDAM, Hoover 2024), which use random feature projection to form shared weight matrix and (2) a distributed variant of Sparsed Quantized Hopfield Network (SQHN), which uses the same number of distributed units as ENCORE. We report the retrieval capacity of these additional baselines in Fig. 2-4, for the synthetic, MNIST, and ImageNet data. In all 9 conditions (3 dataset and 3 corruption types), in the distributed memory setting, ENCORE outperform other tested methods: It holds 4.8–6.8× more patterns than the Hopfield network with the Storkey rule on synthetic patterns and MNIST, and 23x on ImageNet, and across the three datasets 1.4–58× more than the distributed SQHN and 8–300× more than DrDAM. The encoding optimization accounts for a large part of this: with identical weights and storage, it raises capacity by 1.7–2.1× on synthetic patterns, 9.1–11.1× on MNIST and 1.3–1.9× on ImageNet latents.

ModelSynthetic (10,000 patterns)MNIST (5,000 digits)ImageNet latents (1,000 latents)
Distributed memories: one fixed set of shared weights
MHN, gradient write326
DrDAM634–8912–16
SQHN, distributed variant139–5819770–77
Hopfield, Hebbian256–16989–122
Hopfield, Storkey1209–1721221–2564
ENCORE without encoding optimization3659–4931107–18450–232
ENCORE7490–81941186–173995–296
Slot memories (reference): one unit per pattern, 5,000 units (2,560 for ImageNet)
MHN bank, softmax9002–9399≥5000≥1000
MHN bank, sparsemax [1]9138–9378≥5000≥1000
SQHN [3]9184–9231≥5000≥1000

While the slot memory representation does not apply to our setting, we also include the retrieval capacity of the memory-bank MHN with softmax and sparsemax similarity functions as a reference (Figs. 2–4, Table 11). As expected for a memory that reserves one unit per pattern, the bank holds as many patterns as it has units: with 5,000 units its accuracy stays above 0.95 up to about 5,200–5,300 synthetic patterns, and it retrieves all MNIST digits and ImageNet latents, whose streams (5,000 and 1,000 patterns) are not longer than its number of units (5,000 and 2,560). ENCORE, in which all patterns share the same weights, reaches 90–96% of that capacity on synthetic patterns (Table 11).

We have revised the paper accordingly. The abstract, the introduction and a new paragraph in Sec. 2.1 state that ENCORE addresses fixed-size distributed memories.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.