acceptodds
Under review as a conference paper at ICLR 2027

The Art of Remembering: Scaling Laws for Transformer Associative Memories

Abstract

Memory is a fundamental component of intelligent systems. Classical neural memories using Hopfield networks showed that retrieval is best done by energy minimization, where a partial or noisy query is iteratively refined until it settles on a stored memory. However, modern architectures largely ignore energy minimization and instead retrieve stored memories primarily through feed-forward propagation, raising the question of whether energy-based retrieval still matters once neural memories are scaled. To answer this, we compare four retrieval processes: a single forward pass in Feed-Forward Transformers, a diffusion process in Diffusion Transformers, iterative refinement in Recurrent Transformers, and energy minimization in Energy-Based Transformers (EBTs). For each architecture, we train Transformer associative memories and characterize scaling laws and trends across three fundamental properties of neural memories: capacity, generalization to corrupted queries, and continual learning. Across all three properties, we find that EBTs consistently scale best, and that their benefits grow with scale. For example, EBTs have a capacity scaling exponent over 40% higher than other architectures, and store 3.5x more memories than the next-best architecture at scale. Beyond capacity, EBTs generalize better than models 8x larger, and continually learn new memories over 5x faster than other Transformer models. Together, these results show that the classical advantages of energy-based retrieval hold for modern Transformers at scale, offering guidance for designing memory systems in language models, world models, and other architectures.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Associative memory, scaling laws, transformers, energy-based models, memory capacity, generalization, continual learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_SVpJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This manuscript compares four Transformer-based associative memory retrieval mechanisms, namely feed-forward (FFT), diffusion (DiT), recurrent (RT), and energy-based (EBT). The central claim is that explicit energy minimization scales best among all mechanisms, and that this advantage grows with model size. Experiments are conducted on an autoassociative image reconstruction task using COCO image latents.

Critical strengths
  1. The research question is well-defined and the experimental design is clean. Isolating the retrieval mechanism while keeping the Transformer backbone consistent is the right approach to answering this question.
  2. The scaling law framework makes a substantive methodological contribution. Fitting power-law exponents rather than reporting single-point comparisons is a meaningful choice that yields more informative conclusions.
  3. Experimental details are thorough. Appendix B provides sufficient information for reproducibility, including model configurations, training objectives, computational cost accounting, and the fitting procedure.
Critical weaknesses
  1. Generalizability of the findings is unverified. All experiments are based on autoassociative image reconstruction with a fixed masking corruption scheme. Whether the scaling advantage of EBT extends to heteroassociative settings remains untested. The manuscript acknowledges this in the limitations section, but the claims made in the abstract and introduction go beyond what the experiments actually support.
  2. EBT hyperparameters are not tuned, while the baselines are. The manuscript notes this asymmetry, but it prevents the comparison from establishing a fair upper bound for the competing methods. At minimum, a sensitivity analysis over EBT step size and number of descent steps should be provided.
  3. The anomalous generalization performance of DiT lacks explanation. DiT consistently stores fewer memories than FFT and RT (Table B.4), yet generalizes better to corrupted queries (Figure 4, Table B.6). The manuscript acknowledges this dissociation but offers no mechanistic explanation for it.
  4. Single training runs limit the reliability of the scaling law fits. Appendix B.2 states that each experimental configuration is trained only once. Scaling law fits are sensitive to noise at individual data points, particularly at the smallest and largest model scales.
Questions for authors
  1. EBT hyperparameters were not tuned while the baselines were. The authors are encouraged to provide a sensitivity analysis over step size and number of descent steps.
  2. DiT stores fewer memories than other mechanisms yet generalizes better to corrupted queries. The authors are encouraged to provide a mechanistic explanation for this dissociation.
  3. Each configuration is trained only once. The authors are encouraged to demonstrate the reliability of the fitted scaling exponents through repeated runs or confidence intervals.
1 comment

We thank the reviewer for the positive assessment, and for noting the well-defined question, clean experimental design, scaling-law framework, and thorough experimental details.

W1: “Generalizability of the findings is unverified. All experiments are based on autoassociative image reconstruction with a fixed masking corruption scheme. Whether the scaling advantage of EBT extends to heteroassociative settings remains untested. [...] the claims made in the abstract and introduction go beyond what the experiments actually support.”

Response: We thank the reviewer and agree that our claims should match the scope of our experiments. We have scoped the paper to autoassociative retrieval, and now present implications for language and world models as motivation rather than conclusions. We also checked that the results are not tied to the input format, and found that with 16 tokens per image instead of 4, EBT's relative advantage over FFT is preserved (e.g., 8k vs. 3k memories at XS), suggesting that our findings are robust to the input format and size.

W2 + Q1: “EBT hyperparameters are not tuned, while the baselines are. The manuscript notes this asymmetry, but it prevents the comparison from establishing a fair upper bound for the competing methods. At minimum, a sensitivity analysis over EBT step size and number of descent steps should be provided.”

Response: To clarify, all models, including EBT, had their learning rates tuned, and the baselines additionally had their architecture-specific settings tuned (RT passes and DiT integration steps, Appendix B.2). Only EBT's energy-specific hyperparameters were left at default settings. We add a sensitivity analysis over the number of descent steps: more steps improve EBT, with 8 steps memorizing 2,000 memories at 3XS in roughly half the training steps of our default 4, while 2 steps is too few to train within our budget. Our reported results therefore use a non-optimal setting and are likely conservative. We now note this in the Limitations section. We also find that removing spectral normalization does not change performance.

W3 + Q2: “The anomalous generalization performance of DiT lacks explanation. DiT consistently stores fewer memories than FFT and RT (Table B.4), yet generalizes better to corrupted queries (Figure 4, Table B.6). The manuscript acknowledges this dissociation but offers no mechanistic explanation for it.”

Response: We thank the reviewer for this insightful question. Our hypothesis is that DiT is trained on all interpolated states between the query and the memory, so its velocity field is fit over a neighborhood around each memory rather than a single input. This widens the region from which retrieval succeeds (robustness), but requires fitting many more inputs per memory, lowering capacity. EBT avoids this tradeoff, as a single learned minimum provides both precise storage and a basin of attraction. We have added this discussion to Sec. 4.2.

W4 + Q3: “Single training runs limit the reliability of the scaling law fits. Appendix B.2 states that each experimental configuration is trained only once. Scaling law fits are sensitive to noise at individual data points, particularly at the smallest and largest model scales.”

Response: We trained 3 seeds at two model sizes for all four architectures. Measured capacities were identical across seeds (capacity is measured at 1,000-memory granularity), so the fitted exponents and the ranking of architectures are unchanged. Doubling the training budget to 400k steps also leaves the relative capacities unchanged.

We hope these additions address your concerns, and would be grateful if you would consider raising your score.

Reviewer_oiR8OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper conducts an empirical comparison of Feedforward, Recurrent, Diffusion, and Energy-Based Transformers on an autoassociative memory task. The authors demonstrate the EBT having a higher capacity, better generalization to more-corrupted queries, and faster learning of additional items when re-training on an expanded dataset.

Critical strengths
  1. The authors empirically show that, under the tested conditions, EBT has a substantial performance advantage over three related transformer-based models.
  2. The authors make a thoughtful series of comparisons in an effort to investigate whether explicit energy-gradient updates account for the EBT's performance advantage -- the FFT to evaluate the benefits of any iterative computation at all, and DiT and RT to evaluate two types (time-dependent or autonomous, resp.) of iterative reconstruction in contrast to EBT's specialized energy-descent dynamics.
Critical weaknesses
  1. The paper does not fully rule out differences in evaluation/training accounting for performance, and is limited in evaluation to one dataset/masking regime without control over its input statistics. For instance, RT only takes two recurrent steps compared to EBT's four; loss is computed only on the RT's final step compared to all intermediate steps of EBT (the authors test up to four steps on the RT but without intermediate supervision); EBT has additional regularization. In the continual learning task, the number of old memories is held constant across architectures, which leaves different fractions of unused capacity for the networks to insert new memories -- this alone may be sufficient to explain the differences in CL performance.

  2. The mechanistic explanation "a memory needs to only be a minimum ... a corrupted query needs to only start within that minimum’s basin ... writing a new memory needs to only carve a new minimum" is not directly tested.

  • The EBT takes four steps of energy descent without demonstrating convergence, nor that minima correspond to stored memories.
  • The basins of attraction are not investigated (and, indeed, the RT also can have similar basins, potentially with longer rollouts)
  • The minima structure is not evaluated before/after training (again, with the RT potentially also carving out new attractors for new memories).

A less critical problem is the framing and motivation are not developed with precision, the introduction conflating/blurring distinct concepts, e.g., keys being "stored" (they are generated together with the query/value) and queries returning the "closest match" (not generally the case for implicit memories nor softmax attention), corrupted queries addressing explicit memory banks being conflated with reconstruction of corrupted contents (e.g. in NTM, intact contents can be retrieved directly; energy minimization need not return the nearest memory). As a side note, the reference formatting is incorrect (maybe using \cite{...} instead of \citep{...})

Questions for authors
  • Does RT improve with intermediate supervision?
  • What happens to the old memories during the continual learning experiment? e.g., are they forgotten and re-learned, or does their recall stay high during the second training phase?
  • Does EBT's continual learning performance remain high with comparable initial available capacity?
4 comments

(Part 1 of 2)

We thank the reviewer for the detailed review.

W1 + Q1: “The paper does not fully rule out differences in evaluation/training accounting for performance [...] For instance, RT only takes two recurrent steps compared to EBT's four; loss is computed only on the RT's final step compared to all intermediate steps of EBT (the authors test up to four steps on the RT but without intermediate supervision); EBT has additional regularization.”

Response: We thank the reviewer for highlighting these controls, which we agree are important. To determine the effect of these controls, we trained RT with loss on every intermediate step, matching EBT, at both 2 and 4 passes. Deep supervision improves RT (4k → 9k memories at Small), but additional passes still hurt (2 passes outperform 4), and RT with its best settings still scales worse than EBT (15k vs. 9k at Small, with the gap growing with scale). We also ablate EBT, where removing spectral normalization does not change performance, and final-step-only supervision lowers it. Since EBT retains a clear advantage, the gains come from energy-guided retrieval rather than these training choices, which we further support by showing that EBT's descent converges to low-energy minima at stored memories (see our response to W4). We have added deep-supervised RT to the revision as an improved baseline, and are extending its scaling to more sizes.

W2: “[...] and is limited in evaluation to one dataset/masking regime without control over its input statistics.”

Response: We thank the reviewer for this suggestion. While our main experiments use a single dataset, Sec. 4.2 does evaluate retrieval across 32 masking levels beyond the training regime. To further check that our findings are not tied to a particular input format, we also trained models with 16 tokens per image instead of 4. Although absolute capacities change, EBT's advantage over FFT is preserved (e.g., 8k vs. 3k memories at XS), suggesting that the relative trends are robust to the input representation and size.

W3 + Q3: “In the continual learning task, the number of old memories is held constant across architectures, which leaves different fractions of unused capacity for the networks to insert new memories -- this alone may be sufficient to explain the differences in CL performance.”

Response: We thank the reviewer for this insightful observation. We rerun continual learning at Small with the old set filling the same fraction (~7%) of each model's measured capacity. EBT (capacity 15k) starts from 1,000 old memories and learns 1,000 new ones in ~9,200 steps, while FFT (capacity 5k) starts from 333 and learns 333 new ones in ~32,500 steps. EBT therefore learns new memories ~3.5× faster than FFT even while writing 3× as many, demonstrating that the continual learning benefits of EBTs hold even when spare capacity is matched.

(Part 2 of 2)

W4: “The mechanistic explanation [...] is not directly tested. The EBT takes four steps of energy descent without demonstrating convergence, nor that minima correspond to stored memories. The basins of attraction are not investigated (and, indeed, the RT also can have similar basins, potentially with longer rollouts). The minima structure is not evaluated before/after training [...].”

Response: We appreciate this careful critique, and now test the mechanism explicitly with an energy-landscape analysis of an EBT trained on 2,000 memories at 87.5% masking. (i) Convergence. When descent runs for up to 64 steps (vs. 4 in training), energy and gradient norm plateau within about 10 steps and recall stays at 100%. (ii) Minima. Stored memories have clearly lower energy than held-out images, with no overlap, and roughly 10× smaller gradient norm, consistent with stored memories being near-stationary low-energy points. (iii) Writing. After the EBT learns 100 new memories, their energy drops well below that of held-out images (whose energy rises slightly), while the energy of old memories rises only slightly and stays low. The reviewer is also right that RT can form stable attractors, as RT rolled out to 64 passes (vs. 2 in training) also keeps 100% recall. We therefore do not claim RT lacks attractors, but rather that EBT makes them explicit through a learned energy, which we find scales better and writes new memories faster. We agree the mechanism is best framed as a hypothesis that warrants further evidence, and have revised the paper (Sec. 3 and the Conclusion) accordingly.

Q2: “What happens to the old memories during the continual learning experiment? e.g., are they forgotten and re-learned, or does their recall stay high during the second training phase?”

Response: We add old-memory recall curves for all architectures (with 100 and 1,000 new memories). All models show a brief drop in old-memory recall at the start of the second phase, when new memories are introduced. EBT and DiT recover quickly, FFT stays near 90% for thousands of steps, and RT nearly forgets the old set and relearns it. Once the new memories are learned, old memories are retained by all models. For EBT, new-memory and joint convergence nearly coincide (9,235 vs. 9,268 steps with 1,000 new memories).

Minor (W5): “A less critical problem is the framing and motivation are not developed with precision [...] As a side note, the reference formatting is incorrect (maybe using \cite{...} instead of \citep{...})”

Response: We thank the reviewer for these precise corrections. We have revised the introduction and Figure 1 accordingly (stored keys, "closest match," and corrupted keys vs. corrupted contents) and fixed the citation formatting.

We hope these experiments and revisions address your concerns, and would be grateful if you would consider raising your score.

Thanks to the authors for their additional work and improvements to the paper. I've raised my score 2 --> 3.

Thank you for engaging with our rebuttal and for raising your score. Your feedback has improved the paper.

Reviewer_LfcSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper compares feed-forward, recurrent, diffusion, and energy-based Transformers as learned associative memories. On a shared latent-image task, it evaluates capacity scaling, recall under stronger masking, and replay-based memory updates. EBTs achieve the strongest capacity and retrieval results and require fewer optimization steps for updates.

Critical strengths

S1. Systematic comparison. The common task and evaluation protocol provide a useful comparison of four retrieval mechanisms across model sizes.

S2. Substantial empirical gains. At the largest matched size, EBT stores 28,000 images versus 8,000 for FFT. The capacity comparison also considers estimated training FLOPs.

S3. Informative evaluation. Comparing capacity, corruption robustness, and memory updates reveals meaningful differences between these abilities, especially in DiT's ranking.

Critical weaknesses

These concerns are not critical, but addressing them could further strengthen the paper.

W1. Broader significance. The four-token image task gives limited evidence about advantages over simpler learned memories or benefits within larger systems. Full replay and update counts also limit the practical interpretation of the continual-learning results.

W2. Mechanistic attribution. It remains difficult to isolate the contribution of energy minimization from other training choices. For example, EBT is supervised at every refinement step, whereas RT is supervised only at the final step.

W3. Robustness of scaling. Each capacity configuration uses a single run and a fixed training budget. The sensitivity of the fitted exponents to these choices and the fitting range remains unclear.

Questions for authors

Q1. What broader problem in memory design do these results inform, and what motivates the Transformer backbone over simpler learned memories?

Q2. Do existing ablations distinguish the effect of energy-guided retrieval from differences in supervision and normalization?

Q3. How sensitive are the exponents to the fitting range and training budget? Any evidence from repeated runs would help. Log-log plots would also make the scaling fits easier to assess.

3 comments

We thank the reviewer for the positive assessment, including for noting the systematic comparison, the substantial memory-capacity gains of EBTs, and the informative evaluation.

W1: “Broader significance. The four-token image task gives limited evidence about advantages over simpler learned memories or benefits within larger systems. Full replay and update counts also limit the practical interpretation of the continual-learning results.”

Response: We chose the four-token latent task so that sweeps from 5M to 1.6B parameters, with many memory-set sizes per model, were tractable while holding everything except the retrieval mechanism fixed. We agree that this is a controlled first step rather than evidence within larger systems, and have clarified this scope in the revision. To check that the results do not depend on the four-token setup, we rerun capacity with a smaller patch size (16 tokens per image) at 3XS, 2XS, and XS: absolute capacities change, but EBT's advantage over FFT is preserved (2k/4k/8k vs. 1k/2k/3k memories). For continual learning, we use full replay to isolate write speed. We also tested partial replay (10% of old memories): every architecture retained the replayed and new memories but forgot the non-replayed old memories. Together with our main results, this shows that while some architectures (notably EBT) write new memories much faster than others, all architectures still struggle with continual learning without full replay, which is why we use full replay to compare write speed. We thank the reviewer again for this insightful suggestion, and will include a more detailed analysis of continual learning beyond full replay in the final version.

W2 + Q2: “Mechanistic attribution. It remains difficult to isolate the contribution of energy minimization from other training choices. For example, EBT is supervised at every refinement step, whereas RT is supervised only at the final step.”

Response: We thank the reviewer for raising this important point. We now train RT with per-step (deep) supervision, matching EBT. This improves RT, but EBT still stores more memories, and the gap grows with scale (EBT vs. deep-supervised RT: 2k vs. 2k at 3XS, 5k vs. 4k at 2XS, 10k vs. 7k at XS, 15k vs. 9k at Small). Even with deep supervision, more passes hurt RT (2 passes outperform 4). For EBT, removing spectral normalization does not change performance, and final-step-only supervision lowers it, consistent with per-step supervision helping both architectures. Since EBT retains a clear advantage when both use per-step supervision, this indicates that the advantages are primarily attributable to energy-guided retrieval rather than supervision or normalization. Consistent with this, a new analysis of EBT's learned energy landscape shows that descent converges to low-energy minima at stored memories, and that writing new memories lowers their energy (see General Response).

W3 + Q3: “Robustness of scaling. Each capacity configuration uses a single run and a fixed training budget. The sensitivity of the fitted exponents to these choices and the fitting range remains unclear.”

Response: We trained 3 seeds at two model sizes for all four architectures. Measured capacities were identical across seeds (capacity is measured at 1,000-memory granularity), so the fitted exponents are unchanged. We also doubled the training budget to 400k steps, which slightly increased the absolute capacity of every model, but the relative capacities and scaling trends were unchanged.

Q1: “What broader problem in memory design do these results inform, and what motivates the Transformer backbone over simpler learned memories?”

Response: An increasing number of systems now include explicit neural memories, such as memory modules and memory layers for LLMs (MemoryLLM, Wang et al., 2024; Larimar, Das et al., 2024; Memory Layers, Berges et al., 2025; Titans, Behrouz et al., 2024) and memory for long-horizon, consistent world models (WorldMem, Xiao et al., 2026; Wu et al., 2025). Because these memories must store many items, they increasingly use deep, expressive networks (e.g., Titans finds deep memory modules more effective than shallow ones). Transformer layers themselves also act as associative memories (Geva et al., 2021; Ramsauer et al., 2021; Bietti et al., 2023). Our results inform how such memories should retrieve once they are scaled, and a Transformer backbone lets us test this directly, since it matches the architectures used in these systems and lets us vary only the retrieval mechanism across four very different procedures. Simpler learned memories have mostly been studied at small scale (Sec. 2), and our GPCN baseline (Appendix C) does not reach the capacity of even our smallest Transformer.

We hope these additions address your concerns, and would be grateful if you would consider raising your score.

Thank you for the additional experiments and careful clarifications. The stronger RT baseline, the supervision and normalization ablations, and the seed and training-budget checks strengthen the empirical findings. I also appreciate the energy-landscape analysis and the clearer distinction between the observed results and the proposed mechanistic explanation.

The broader applicability and the practical interpretation of the continual-learning results remain limited, but I find the controlled comparison informative and valuable. I maintain my rating.

Thank you for engaging with our rebuttal and for the helpful assessment. We agree that broader applicability and more realistic continual learning are important next steps, so we note both in the Limitations.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Reviewer Response

We thank all reviewers for their thoughtful feedback. We appreciate all reviewers for recognizing our research question being well-defined and the design clean and systematic (LfcS, SVpJ), the comparisons thoughtful (oiR8), EBT's capacity gains substantial (LfcS, oiR8), and the scaling-law framework a meaningful contribution (SVpJ). The shared concerns were (1) robustness of the scaling results, (2) whether the advantage of Energy-Based Transformers (EBTs) over Feed-Forward (FFT), Diffusion (DiT), and Recurrent (RT) Transformers could come from training differences rather than energy minimization, and (3) the mechanism behind EBT's advantage. We address these with new experiments, all of which have been added to the revised paper (changes highlighted in blue):

  • Robustness of scaling: Across 3 seeds at two model sizes, measured capacities (the largest memory set recalled at ≥99%) were identical for all architectures (capacity is measured at 1,000-memory granularity). Doubling the training budget to 400k steps slightly increases every model's capacity, but leaves the relative capacities unchanged, with EBT still performing best (Appendix B.5).
  • Stronger RT baseline: Adding per-step (deep) supervision improves RT (4k → 9k memories at Small), but EBT still stores more memories, and the gap between EBT and RT still grows with scale (2k vs. 2k at 3XS/16M parameters, 15k vs. 9k at Small/129M parameters). Even with deep supervision, more passes still hurt RT (2 passes outperform 4), whereas EBT benefits from more descent steps. These results show RT can be improved, but it still trails EBT, increasingly so at larger scales (Sec. 4.1, Appendix B.8).
  • EBT Key ablations: To test whether EBT's advantage comes from its extra regularization or per-step supervision rather than energy minimization, we ablate both. Removing spectral normalization does not change EBT's performance, and final-step-only supervision lowers it, mirroring RT, so per-step supervision helps both architectures and does not explain EBT's lead. More descent steps also improve EBT (8 steps outperform our default of 4), so our reported EBT results are likely conservative, and we leave scaling EBTs with tuned hyperparameters to future work (Appendix B.9).
  • Energy landscape: We find that gradient descent on EBT’s energy landscape converges to energy minima (energy and gradient norm plateau, and recall stays at 100% for up to 64 steps despite training with 4), where stored memories sit at clearly lower energy than held-out images, and writing new memories lowers their energy while held-out images' energy rises slightly. This supports our hypothesis that stored memories become energy minima that descent can retrieve (Sec. 3, Appendix B.10).
  • Continual learning with matched spare capacity: In the original continual learning setup, every model started from the same 1,000 old memories, which filled different fractions of each model's capacity. We rerun continual learning with the old set filling the same fraction (~7%) of each model's measured capacity: EBT (capacity 15k) starts from 1,000 old memories and learns 1,000 new ones in ~9,200 steps, while FFT (capacity 5k) starts from 333 and learns 333 new ones in ~32,500 steps. EBT therefore learns new memories ~3.5× faster than FFT even while writing 3× as many, showing that its faster continual learning is not explained by having more unused capacity (Sec. 4.3, Appendix B.7).
  • Input format and size: With 16 tokens per image instead of 4, absolute capacities change, but EBT's advantage over FFT is preserved (8k vs. 3k memories at XS), suggesting that EBT's relative advantage is robust to the input format and size (Appendix B.5).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.