acceptodds
Under review as a conference paper at ICLR 2027

Competitor Interactions and Recall Lifetimes in Spherical Log-Sum-Exp Associative Memory

Abstract

How long does a retrieved memory persist under noisy updates? We study this question in spherical log-sum-exp associative memory, defining recall loss as the target losing to competitors for a fixed, uninterrupted interval. We test two ways to simplify the dynamics: representing a bank by its target–competitor similarities, and tracking only the overlaps with the target and leading rival. Both omit information relevant to recall. In a controlled three-key example, banks with identical target similarities, initial logits, margin and entropy have mean recall times differing by a factor of approximately 1.74. A numerical solution of the joint dynamics closely reproduces this ratio without fitting a time scale. A separate control shows that the effect persists with distinct keys. In two pairs of 128-key banks, changing the collective drift changes the lifetime ratio even though the exit boundaries remain fixed. Within a bank, replicated short-time tests find different future gaps from states with the same target and leading-rival overlaps and leader. Conditional equilibrium averaging improves the held-out aggregate lifetime ratio over isolated-pair addition, although individual mean times remain systematically biased. These finite-dimensional examples show how interactions among competitors and unresolved state coordinates affect predictions of noisy recall lifetimes.

Primary area:
learning theory
Keywords:
dense associative memory, modern Hopfield networks, recall lifetimes, first-passage times, competitor interactions, coarse-graining

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6efMOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2(+1)
Soundness
3
Read review
Paper summary

The paper studies the problem of incorrect recall of a specific class of retrieval dynamics (spherical log-sum-exp associative memory) in the presence of noise. The paper is able to resolve some nuances in the estimation of these errors by taking into account additional structure in relation to the relationship between the other memory items, which for this particular retrieval task serve as distractors. They also illustrate some counterexamples where the simpler estimates (like considering only the closest distractor option) would lead to errors by ignoring these interaction terms. Overall, the problem is very narrow in scope and the presentation particularly opaque, making the submission seem of limited relevance to the ICLR community.

Critical strengths

Not much positive to say here, beyond perhaps illustrating failure modes of previous approaches

Critical weaknesses
  • no rationale provided for why the problem matters
  • significance of the results is unclear
  • there is insufficient explanations provided on all fronts: problem formulation, definition of variables, logic progression between steps in the proofs.
  • please explain in more detail the numerical setup : why those specific parameters? what were you hoping to show?
Questions for authors

Please clarify: the nature of the dynamics, the assumed noise structure, the memory item distribution Is there any way of providing some geometric intuition for the steps of the counterexample construction or the proof steps? What does the difference in estimated numbers mean in practical terms?

2 comments

Thank you for taking the time to review our paper. Your question about what the lifetime numbers mean gets to the motivation for the work. Suppose two memories are equally far from their nearest distractor and both are retrieved correctly. If the updates continue in the presence of noise, can one memory lose the lead much sooner than the other? This is a question about retention after successful recall. The revised Introduction starts from this question.

What evolves, and what counts as a loss? The stored items are fixed unit vectors. The state starts at the target. At each step it moves toward the softmax-weighted average of the stored vectors, receives a fresh Gaussian perturbation tangent to the sphere, and is normalized to unit length. The noise has the same variance in every tangent direction. It represents disturbances during repeated updates. Calibrating its strength to a particular system would require measurements. Figure R1 now plots the cosine similarities along one trajectory. The target briefly loses the lead and recovers. Loss is confirmed later, after 40 consecutive losing observations at step h=0.0025h=0.0025 (nominal window Δ=0.1\Delta=0.1). The lifetime we report is the mean time from initialization to this confirmation.

The geometry of the counterexample. Put the target at the top of a sphere and place two rivals at the same latitude, each with cosine similarity 0.7 to the target. Now move one rival along that latitude. None of the initial scores changes. What changes is how the two rivals respond to the same displacement of the state. If they coincide, a step toward one favors both, and their pulls add. If they lie on opposite sides of the target, the same step favors one and disfavors the other. Their sideways pulls at the target cancel. In the covariance calculation in Section 3, we project one noise increment onto the two target–rival difference vectors. The revised derivation shows where the angle between the rivals enters. This establishes a difference in the local dynamics. Section 4 then tests whether it makes a measurable difference to retention.

At N=18N=18 and Δ=0.1\Delta=0.1 (Figure 1b), the mean time to confirmed loss was 24.74 model time units for coincident rivals and 43.14 for separated rivals. Thus, with the same initial scores, the target was retained about 74% longer on average in the second bank. The planar calculation predicted a ratio of 1.735 without fitting a time scale. Simulation gave 1.744. Table R1 uses new trajectories to vary the confirmation window. The ratio ranges from 1.37 to 2.17, with the separated bank retaining the target longer at all five windows. The rotation control with distinct rivals shows an appreciable drop in lifetime at the strongly aligned settings (Table 1, Appendix B).

Why these numerical settings? We fixed the softmax sharpness at β=1\beta=1 and the noise strength at T=0.2T=0.2 so that the comparisons isolate the change in geometry. We chose the planar construction because it permits an independent calculation of the lifetime. The 128-key experiment asks how well we can estimate retention when many rivals contribute. In N=24N=24 dimensions, we prescribe each rival’s similarity to the target and independently draw the direction of its perpendicular component uniformly on the unit sphere in the space orthogonal to the target. We then hold the keys fixed during simulation. In A, four rivals have target cosine 0.7 and the other 123 have cosine 0. In B, one rival has cosine 0.7 and the other 126 have cosine 0.52717. These choices match the initial nearest-rival margin and softmax entropy. The two-coordinate model predicted a B/A lifetime ratio of 1.29 and underestimated the mean in B on each of the four held-out banks. The approximate isolated-pair calculation predicted that B would retain the target more than twice as long as A (ratio 2.11). In simulation, the geometric mean of the four B/A ratios was 1.35, an increase in retention of about 35%.

Reviewer_6efMOpenReview ↗

rebuttal received

I thank the authors for the clarifications, and spelling out the motivation and the problem formulation in more understandable terms. The additional clarification on the numerics gave me a clearer idea of what was implemented and why. That said, the scope of the work still seems incremental and of limited relevance to the ICLR. The poor novelty and significance scores remain an issue. The clarity of the text has improved somewhat so i'll increase that score accordingly.

Reviewer_4HeAOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies spherical log-sum-exp memories. Specifically, it attempts to characterize the lifetime of a target memory through a noisy inference process. The authors provide a theoretical analysis and examples that show that by ignoring interactions between interfering memories, the lifetime estimation of the target memory becomes distorted. The authors then propose a different modeling approach which achieves a better approximation, yet still does not close the gap with the measured lifetimes.

Critical strengths
  1. The paper studies associative memories, which are a core component in sequence models. The authors analyze several lifetime estimation cases and shed light on the importance of accounting for interactions between interfering memories.
  2. The theoretical analysis is supported by synthetic experiments.
Critical weaknesses
  1. The paper is very hard to follow. It would be useful to add a clear narrative connecting all sections and their experiments. At the moment, this is mostly left implicit and must be inferred by the reader.
  2. The paper does not test its theoretical ideas on real data. Therefore, it is not clear whether the theory explains real-world systems.
Questions for authors

See weaknesses

2 comments

Thank you for taking the time to review our paper. We appreciate your point about the connection between the experiments. We have revised the abstract and introduction to make the question behind each experiment explicit. We also agree that the evidence from the constructed banks leaves open how large these effects are in a learned memory.

1. How the sections fit together. We start with the target already retrieved and ask how long it remains the winner under noisy updates. A brief loss of the lead can be followed by recovery. Figure R1 in Sec. 2 now shows such a trajectory and the later interval that counts as confirmed loss.

The first question is whether knowing the similarity of every rival to the target is enough to predict this lifetime. Sec. 3 shows where the angle between two rivals enters the dynamics. A change in the local dynamics alone does not tell us how much the lifetime changes, so Sec. 4 tests this directly. We keep both target–rival similarities fixed and change the angle between the rivals. At N=18N=18, with a confirmation window of 0.1, the mean lifetime is 1.744 times longer in the separated geometry than in the aligned one. Solving the joint dynamics gives 1.735 without fitting a time scale.

The second question is what information we need to keep when there are many rivals. In Sec. 5 we use 128-key banks and change which keys contribute to the update, while all 128 still compete to be the winner. This changes the lifetime ratio. Sec. 6 then asks a more specific question within one bank. Even when the target overlap, the leading rival’s overlap and its identity are all fixed, the other keys still carry information about the target’s lead a short time later.

This leads to the approximation in Sec. 7. We keep the two overlaps and average over the other coordinates using their conditional equilibrium distribution. On four held-out banks, the predicted aggregate B/A lifetime ratio was 1.289, compared with 1.349 in simulation and 2.11 from treating target–rival pairs separately. As the reviewer observed, the improvement in the ratio does not settle the accuracy of the individual lifetimes. The approximation underestimated the mean in condition B on all four held-out banks.

2. Relevance to real data. All banks in the paper are synthetic. We chose them because they allow comparisons that would be difficult to interpret in a learned bank. For example, in Sec. 4 we can rotate one rival while preserving both rivals’ similarities to the target. The revised text also reports where the effect appears: the mean lifetimes were close at rival–rival similarities of −0.02-0.02 and 0.490.49 (43.60 and 43.02), then fell to 35.52 at 0.7450.745. How often learned keys have such strongly correlated rivals is an empirical question.

The connection to sequence models is through the softmax update with tied keys and values. A direct next test would be to use normalized keys from a trained model, measure their rival–rival similarities, and compare predicted and simulated retention times under the same update rule. Our present results establish the effect for the specified memory dynamics. Whether it explains behavior in a trained sequence model remains open.

Reviewer_4HeAOpenReview ↗

Response to rebuttal

Thank you for your rebuttal. While the clarifications make the ideas easier to follow, the relevance of the work remains limited due to the lack of experiments on real data (Weakness 2). I will keep my score.

Reviewer_8eq6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies the noisy behavior of spherical log-sum-exp associative memory. Spherical memory is quite similar to some work on exponential Hopfield networks and exponential BAMs from the 1990s. In this type of model, the patterns are stored directly. Personally, I think this type of model loses the concept of distributed encoding. Once the system is hacked, the stored patterns can be directly identified.

For this type of exponential memory, the noise resistance is very strong. Empirically, convergence is also very fast.

Furthermore, if the patterns are stored directly, there may be no need for iterative recall. By computing the dot products using a simple algorithm, we can identify the stored item.

The paper also does not explain where the noise comes from.

Critical strengths

If the authors can justify the source of the noise, the paper is interesting from a purely theoretical point of view.

Critical weaknesses

The paper does not explain where the noise comes from. Does it represent hardware noise, thermal noise, or another type of noise? Is the noise used in the simulations reasonable?

The theoretical development is somewhat brief. Detailed derivations, such as those of the covariance expressions, are not provided. Hence, I cannot verify the results. When noise is introduced during the recall process, I would consider the model to be a type of exponential Boltzmann machine.

The presentation is unclear. Why is there no figure illustrating the dynamics, trajectories, and cosine similarities?

Please refer to a classical paper by Shun-Ichi Amari and Kenjiro Maginu published in 1988. This paper showed that the cosine similarity to the target does not fully determine future behavior. Future similarities also depend on variables associated with the previous trajectory.

The current paper presents similar observations. I strongly encourage the authors to read the paper by Amari and Maginu. Afterwards, the authors may have better ideas for constructing the mean dynamics. in a much more easy way. Sometimes, reading some old papers is very useful.

Questions for authors
  1. Show a figure illustrating the dynamics and trajectories.
  2. Discuss whether the current model can be regarded as a type of exponential Boltzmann machine.
  3. Why do the patterns need to be stored directly?
  4. If the approach of Amari and Maginu were applied, would similar results be obtained?
  5. Are the dynamics mean dynamics? Can you define a confident dynamics.
1 comment

Thank you for taking the time to review our paper and for your thoughtful questions. Your suggestion to connect our results to Amari and Maginu helped us explain more clearly what the two overlaps tell us about recall. We have added this comparison to Sec. 1 and a trajectory to Sec. 2 so that the reader can see how recall is lost and recovered along a single path.

Amari and Maginu (Question 4). We agree with the reviewer that the target overlap alone can leave out information needed to predict subsequent recall. Amari and Maginu (1988) showed how correlations in the interference from other patterns build up over successive updates in a binary associative memory. We now cite their work and explain the connection to our test. In Sec. 6, we fix the overlaps with both the target and its strongest rival, as well as the rival’s identity. Even then, states with more softmax weight on the remaining keys had a smaller mean target lead 0.1 time units later. The difference was −0.00167-0.00167 and, in an independent repeat, −0.00216-0.00216. Their approach suggests a way to model this dependence. Whether it would also predict our recall lifetimes is open. That calculation would have to account for the possibility that the target recovers before a losing run is complete.

Trajectories (Question 1). Figure R1 shows the cosine similarities to the target and both rivals along one simulated trajectory. The target first loses the lead at t=24.295t=24.295 and regains it at 24.33024.330, so the clock resets. The shaded interval shows the later run of 40 consecutive losing observations, with loss confirmed at 24.562524.5625. Table R1 in Sec. 4 shows how the mean lifetimes change with the length of the confirmation window.

Source of the noise. We model disturbances that continue after retrieval. At each step in Eq. (3), we draw a fresh Gaussian vector, project it onto the tangent plane of the sphere and scale it by 2Th/N\sqrt{2Th/N}, where hh is the time step. In the continuous Langevin model this is isotropic thermal noise. We chose T=0.2T=0.2 to compare banks at a common noise level. Relating this value to a particular device or neural system would require an independent calibration. Sec. 2 now explains this choice.

We also added the intermediate steps for the covariance in Eq. (5). If vi=a0−aiv_i=a_0-a_i, the target’s lead over rival ii is δi=vi⊤y\delta_i=v_i^\top y. Two gaps receive projections of the same Brownian increment. With Py=I−yy⊤P_y=I-yy^\top, their covariance per unit time is (2T/N)vi⊤Pyvj(2T/N)v_i^\top P_yv_j. Expanding gives (2T/N)(1−ϕi−ϕj+Gij−δiδj)(2T/N)(1-\phi_i-\phi_j+G_{ij}-\delta_i\delta_j). This shows where the similarity GijG_{ij} between the two rivals enters, even when their similarities to the target are fixed.

Relation to a Boltzmann machine (Question 2). There is a Boltzmann interpretation of the equilibrium law. The log-sum-exp energy gives a Gibbs density on the sphere. Here β\beta sets the sharpness of the weights over keys, and TT sets the noise strength. The evolving state is one continuous unit vector following the Langevin dynamics in Eq. (2). A classical Boltzmann machine updates stochastic binary units. The update rule matters for our question because processes with the same equilibrium law can have different recall lifetimes.

Explicit storage and iterative recall (Question 3). We agree that one round of dot products identifies the nearest key for a fixed query. Our experiment starts at the target, so its identity is already known. We continue the updates to measure how long it takes for the target to stay behind its rivals for a prescribed interval. Explicit storage is an assumption of the model, and anyone with access to the bank can indeed read the keys. It also lets us vary the angle between two rivals while keeping their similarities to the target fixed. At N=18N=18 and Δ=0.1\Delta=0.1, the separated/aligned lifetime ratio in the planar example is 1.744 in simulation. The joint dynamics predict 1.735 without fitting a time scale.

Mean dynamics and confidence (Question 5). Eq. (3) describes an individual random trajectory. Eq. (9) averages over unresolved coordinates at fixed overlaps and gives a reduced stochastic process. The mean we report is the expected time from the target to confirmed loss. In condition B, the reduced model underestimated this mean on all four development banks and all four held-out banks. The additional checks on 4B leave open how much of this discrepancy comes from the omitted coordinates (Appendix D.1).

If by “confidence” the reviewer means the probability that recall survives to time tt, a natural quantity is S(t)=Pr⁡(τc>t)S(t)=\Pr(\tau_c>t). It can be estimated by the fraction of trajectories with no confirmed loss by tt. The confidence intervals in the paper quantify sampling uncertainty in the reported means and contrasts.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.