- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2(+1)
- Soundness
- 3
Read review
Paper summary
The paper studies the problem of incorrect recall of a specific class of retrieval dynamics (spherical log-sum-exp associative memory) in the presence of noise. The paper is able to resolve some nuances in the estimation of these errors by taking into account additional structure in relation to the relationship between the other memory items, which for this particular retrieval task serve as distractors. They also illustrate some counterexamples where the simpler estimates (like considering only the closest distractor option) would lead to errors by ignoring these interaction terms. Overall, the problem is very narrow in scope and the presentation particularly opaque, making the submission seem of limited relevance to the ICLR community.
Critical strengths
Not much positive to say here, beyond perhaps illustrating failure modes of previous approaches
Critical weaknesses
- no rationale provided for why the problem matters
- significance of the results is unclear
- there is insufficient explanations provided on all fronts: problem formulation, definition of variables, logic progression between steps in the proofs.
- please explain in more detail the numerical setup : why those specific parameters? what were you hoping to show?
Questions for authors
Please clarify: the nature of the dynamics, the assumed noise structure, the memory item distribution Is there any way of providing some geometric intuition for the steps of the counterexample construction or the proof steps? What does the difference in estimated numbers mean in practical terms?
2 comments
rebuttal received
I thank the authors for the clarifications, and spelling out the motivation and the problem formulation in more understandable terms. The additional clarification on the numerics gave me a clearer idea of what was implemented and why. That said, the scope of the work still seems incremental and of limited relevance to the ICLR. The poor novelty and significance scores remain an issue. The clarity of the text has improved somewhat so i'll increase that score accordingly.
Thank you for taking the time to review our paper. Your question about what the lifetime numbers mean gets to the motivation for the work. Suppose two memories are equally far from their nearest distractor and both are retrieved correctly. If the updates continue in the presence of noise, can one memory lose the lead much sooner than the other? This is a question about retention after successful recall. The revised Introduction starts from this question.
What evolves, and what counts as a loss? The stored items are fixed unit vectors. The state starts at the target. At each step it moves toward the softmax-weighted average of the stored vectors, receives a fresh Gaussian perturbation tangent to the sphere, and is normalized to unit length. The noise has the same variance in every tangent direction. It represents disturbances during repeated updates. Calibrating its strength to a particular system would require measurements. Figure R1 now plots the cosine similarities along one trajectory. The target briefly loses the lead and recovers. Loss is confirmed later, after 40 consecutive losing observations at step h=0.0025 (nominal window Δ=0.1). The lifetime we report is the mean time from initialization to this confirmation.
The geometry of the counterexample. Put the target at the top of a sphere and place two rivals at the same latitude, each with cosine similarity 0.7 to the target. Now move one rival along that latitude. None of the initial scores changes. What changes is how the two rivals respond to the same displacement of the state. If they coincide, a step toward one favors both, and their pulls add. If they lie on opposite sides of the target, the same step favors one and disfavors the other. Their sideways pulls at the target cancel. In the covariance calculation in Section 3, we project one noise increment onto the two target–rival difference vectors. The revised derivation shows where the angle between the rivals enters. This establishes a difference in the local dynamics. Section 4 then tests whether it makes a measurable difference to retention.
At N=18 and Δ=0.1 (Figure 1b), the mean time to confirmed loss was 24.74 model time units for coincident rivals and 43.14 for separated rivals. Thus, with the same initial scores, the target was retained about 74% longer on average in the second bank. The planar calculation predicted a ratio of 1.735 without fitting a time scale. Simulation gave 1.744. Table R1 uses new trajectories to vary the confirmation window. The ratio ranges from 1.37 to 2.17, with the separated bank retaining the target longer at all five windows. The rotation control with distinct rivals shows an appreciable drop in lifetime at the strongly aligned settings (Table 1, Appendix B).
Why these numerical settings? We fixed the softmax sharpness at β=1 and the noise strength at T=0.2 so that the comparisons isolate the change in geometry. We chose the planar construction because it permits an independent calculation of the lifetime. The 128-key experiment asks how well we can estimate retention when many rivals contribute. In N=24 dimensions, we prescribe each rival’s similarity to the target and independently draw the direction of its perpendicular component uniformly on the unit sphere in the space orthogonal to the target. We then hold the keys fixed during simulation. In A, four rivals have target cosine 0.7 and the other 123 have cosine 0. In B, one rival has cosine 0.7 and the other 126 have cosine 0.52717. These choices match the initial nearest-rival margin and softmax entropy. The two-coordinate model predicted a B/A lifetime ratio of 1.29 and underestimated the mean in B on each of the four held-out banks. The approximate isolated-pair calculation predicted that B would retain the target more than twice as long as A (ratio 2.11). In simulation, the geometric mean of the four B/A ratios was 1.35, an increase in retention of about 35%.