- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors propose a theoretical framework for a unified stability analysis of a particular class of associative memory models. The models in question are formulated in the style of Universal Hopfield Networks https://arxiv.org/abs/2202.04557. These are maps consisting of a scoring rule, a separation rule, and a readout. For such models, the authors offer a way to analyse stability. The construction is applied to many published models of associative memory.
Critical strengths
The most appealing part of the analysis is a strive for generality. It would be great to have stability results for a large class of memory models, and certainty about their stability properties without needing to repeat a tedious stability analysis for each of them separately. The authors partially succeed in this by defining -admissible models and studying when the model is -admissible.
Another strong side is rigour: most of the statements made in the article are in the form of lemmas and theorems. Such a structure makes it easier to understand the content of the paper and formally check the conclusions of the authors. I read the proofs selectively and found them convincing.
Critical weaknesses
There are a few critical problems with the results of the paper.
The article is poorly written and organised. Many verbal explanations are unnatural; it is hard to decipher them and hard to understand their meaning. Certain concepts and setups are not explained or explained indirectly, which also makes the paper harder to navigate.
The authors' goal was, presumably, to analyse a large class of memory models and draw some useful conclusions based on this analysis. The intended generality is reflected in many places in the article: (i) the title is a general statement about Hopfield networks; (ii) the authors analyse 32 published designs of Hopfield networks; (iii) the theory itself is written for networks with three general components. While the generality of analysis is a noble goal, in my view, the analysis of the authors is not general and does not allow drawing broad conclusions on the stability of memory models. This is my main technical objection, so I explain it in detail below.
Questions for authors
Technical contribution and implication for stability
On a technical level, the contribution offers a stability analysis. The approach is based on the well-known Lyapunov theory: in place of studying trajectories of the dynamical system, try to find a scalar function (energy) that is non-increasing (better, decreasing) on trajectories; if such a function exists under certain mild conditions, nothing but fixed points can be found in the domain when the conditions on the energy function hold.
The authors define their energy function in Definition 1, and call the map "-admissible" if, under their definition, a particular choice of energy parameters ensures that the energy is non-decreasing on the trajectory.
Crucially, if the conditions stated in Definition 1 fail, it does not mean that there is no Lyapunov energy function for a given dynamics. Yet, the authors make it sound as if being "non-admissible" is something bad. Note that "G" is often dropped, e.g., in the introduction: "We apply these results to 32 published retrieval maps. Eight of them are shown not to be admissible." Relatedly, in Table 1, the authors state "no constant metric and no convex potential make the map admissible".
Given this assessment and context, here are my questions:
a. What is the reason the authors decide to call this particular property "G-admissible" and why do they find it is fine to switch to simply "admissible" and "not admissible", which is a much stronger property, normally meaning "can not be allowed"?
b. Suppose my map is "inadmissible" according to the authors' criteria. So what? It doesn't mean the dynamics is bad, no fixed points exist, no other Lyapunov function exists, etc. It only means the particular construction of the authors does not work. Then why should the readers care that the map is not "admissible"?
c. In my view, the title of the paper, being too general, is misleading. Can the authors explain how their results support the assertion made in the title?
Setups and assumptions
I want to clarify several assumptions and preferences the authors made without a proper discussion:
- Why is stability important at all? Is it assumed that the map is iterated many times until convergence? Note that there are some contributions where it is suggested that the map is applied once and appropriate results are shown, e.g., in https://arxiv.org/abs/2008.02217.
- After Definition 1, the authors write: "The definition guarantees only descent, with no lower bound on and no memories at the fixed points." Is it assumed that should directly correspond to memories? If so, why? Is it appropriate in the authors' view that there are fixed points that are distinct from but are defined by them?
- Often, an additional requirement for Hopfield networks is biological plausibility, e.g., https://arxiv.org/abs/2008.06996. Do the authors believe it is an important property, or are they ready to abandon it? Why not focus on simple systems such as gradient flows, e.g., https://arxiv.org/abs/1910.02720?
Organisation of the paper
I personally find many phrases to be unnatural; below is an incomplete list. I explain why a few of them sound weird, and the rest are listed with no explanation. In general, it is often possible to infer what the meaning of a particular construction is, but it is often unnecessarily hard.
-
line 260 "A map without an energy need not cycle."
There are limit cycles in dynamical systems, so the trajectory (states) forms a cycle under a given map. So while it is possible to infer what "map cycles" means, the phrasing is unnatural.
Another problem is that "without an energy" is undefined. As I explained above, the results of the authors do not imply the absence of energy, at best an absence of a particular energy.
My translation of this phrase is: "Even when the conditions listed in Definition 1 do not hold, or, more generally, when a discrete dynamical system admits no global Lyapunov function, its trajectory does not need to contain limit cycles."
-
line 325 "An adaptive score that keeps its energy"
What is the meaning of this phrase precisely? What does it mean to "keep someone's energy"?
My translation is: "In this section, we show how to construct an update rule based on a particular score function such that it admits a global energy function from Definition 1."
-
line 164 "What the pattern readout admits"
-
line 263 "To see which maps can cycle"
-
line 324 "A monotone gate gives the gradient readout an exact energy"
-
line 368 "The energy holds throughout training"
-
line 400 "Where the construction stops."
-
line 475 "Our map pays for its energy in iterations."
-
line 478 "We expect slow contraction to cause both the gap and the cost."
In general, I would insist on improving the organisation of the paper.
For example, a few phrases at the beginning of the introduction explain the setup with a weird mixture of equations and words. It is unclear why patterns and are specified explicitly and never used in this section, while simultaneously scores, the separation function, and queries are explained only in words.
After such an introduction, Section 2, where the technical setup is really covered, does not go into much detail. Why not write here that we denote memory patterns by , that we assume a particular structure of the map, that we iterate the resulting map, etc.? Why not define the stability problem and back up the claim from the introduction that "in both frameworks the energy is proved for a different map from the one that the model iterates"? Sure enough, all this information is available or inferable from the context, but it is scattered in various places in the article, so it becomes harder for the reader to understand what is going on, unless they already have all the necessary context.
5 comments
Response to Reviewer tyjU (2/2): setup, assumptions and writing
Setup 1: "Why is stability important at all? Is it assumed that the map T(x) is iterated many times until convergence?"
No. Definition 1 bounds every single step, , so a map with a constant-metric energy never raises , even when it is applied once, as in the one-step retrieval of Ramsauer et al. (2021). The revised Section 4 adds that our verdicts say nothing about whether one step retrieves the right pattern (line 312). Stability matters when the map is iterated. The adaptive network's paper states an energy that decreases monotonically and converges (its Theorem 2, quoted in Appendix E.4), although its released code applies the map once (line 152). When we iterate it as a multiple-instance pooling layer trained through five unrolled steps at learning rate 0.1, its period-two cycles change the predicted label between steps 999 and 1000 in 9 to 26% of held-out bags on squared distances, and in 9 to 27% with both bases (Table 2).
Setup 2: "Is it assumed that ξ^μ should directly correspond to memories? […] Is it appropriate in the authors' view that there are fixed points that are distinct from ξ^μ but are defined by them?"
No result in the paper assumes that the stored patterns are fixed points. Definition 1 guarantees descent only, and the revision says after it that the definition "neither bounds E below nor ensures that the fixed points are stored patterns" (line 166). We think such fixed points are appropriate. The modern Hopfield network has them too: Ramsauer et al. describe metastable states that average over subsets of patterns. Appendix G.3 shows iterates of our map that stop away from every stored pattern.
Setup 3: "Do the authors believe it is an important property, or are they ready to abandon it? Why not focus on simple systems such as gradient flows"
Biological plausibility is outside our scope, and Section 2 now says so (line 133). Our gradient readout is itself a unit gradient step, , on in the metric . Section 2 now cites Bartunov et al. (2020), whose memory model retrieves by gradient descent on a learned energy with learned step sizes. We study the discrete map because the published models apply or iterate one, and a descent proof for continuous-time dynamics does not cover it. The universal Hopfield paper, for example, proves descent for continuous-time dynamics, while its code runs a discrete update that has no constant-metric energy at a tested configuration (Section 2, Table 1).
Writing: "I personally find many phrases to be unnatural […] In general, I would insist on improving the organisation of the paper."
We agree with the reviewer's translations, and we rewrote each phrase:
| Submission | Revision |
|---|---|
| A map without an energy need not cycle. | A map without a constant-metric energy need not have cycles. (line 314) |
| An adaptive score that keeps its energy | An exact energy for the order-statistic score (Section 5) |
| What the pattern readout admits | When the pattern readout has a constant-metric energy (Section 3.1) |
| To see which maps can cycle | To see when a pattern readout's iterates enter a cycle (line 318) |
| A monotone gate gives the gradient readout an exact energy | With a monotone gate, the gradient-readout map has an exact energy (Theorem 2) |
| The energy holds throughout training | With a softmax-parameterised gate, the map has an exact energy at every parameter value (Corollary 1) |
| Where the construction stops. | Limits of the construction. (line 405) |
| Our map pays for its energy in iterations. | The guarantee of Theorem 2 comes at a cost in iterations. (line 515) |
| We expect slow contraction to cause both the gap and the cost. | Our map contracts slowly, which likely causes both the accuracy gap and the cost. (line 520) |
We also gathered the setup in one place. The first paragraph of the introduction now defines the score, the separation and the inverse temperature in symbols (lines 38 to 46). Section 2 opens with the memory patterns , the map and its iteration, points to a notation table (Table 4) and says which results assume which readout. Its new paragraph "The stability problem" (line 145) states our question and backs the claim of the introduction with the universal Hopfield case above and the adaptive network, whose proof of its Theorem 2 uses a score without the sort (Appendix E.4).
We thank the reviewer again for these detailed comments, which made the paper clearer. We would be happy to answer any further questions.
Follow-up before the discussion closes
Dear Reviewer tyjU,
Thank you again for your detailed comments. Since the discussion period ends on November 18, we would like to check whether the revised PDF addresses your main points. In particular, the main text no longer writes "admissible" without a metric. After Definition 1, it now states that a map without a constant-metric energy may still converge and have a Lyapunov function of another form. We remain available for any further questions.
Best regards, The authors
Response to Reviewer tyjU (1/2): the name, what a cross means, and the title (a, b, c)
We thank the reviewer for the careful reading, and for the kind words about the generality and the rigour of the analysis. Line numbers refer to the revised PDF, where added and reworded passages are in blue.
We agree. Dropping the G was a mistake, and "not admissible" reads as a verdict that our results do not support. We chose the name because the condition admits a map into the energy argument of the modern Hopfield network. Each step becomes a concave-convex step (Yuille & Rangarajan, 2003) on E=21x⊤Gx−h, the form of the energy of Ramsauer et al. (2021) with G=I. The main text no longer writes "admissible" without a metric. It keeps "G-admissible" in Definition 1, the results and the table entries, and elsewhere says that a map has, or has no, constant-metric energy, which it defines through Definition 1 (line 86). For example:
The appendix keeps the short form and states this convention in one sentence (Appendix A).
We agree with this reading, and the revision now states it after Definition 1 (line 168): a map without a constant-metric energy "may still converge, have fixed points and have a Lyapunov function of another form, so the lack of such an energy does not mean that the map is unstable." Our 32 maps include each of these cases:
A cross matters for two reasons. First, the energy argument behind the convergence proof of Ramsauer et al. does not apply to the map as published, and none of the six papers behind the eight crosses proves an energy for these maps. Second, only a map without a constant-metric energy can have a cycle. Proposition 3 proves that once the softmax saturates, the pattern readout has an attracting cycle of period two near every pair of patterns that are each other's best match. A map with such a cycle has no function of any form that decreases at every step that moves.
We agree that the title is broader than the results. "The readout" is supported. Under the pattern readout and for patterns in general position, generic splitting scores have such an energy in the Euclidean metric essentially only for the dot product (Theorem 1), and under the gradient readout every semi-convex score has one (Proposition 1). "Decides" overstates. Necessity under the pattern readout is proved only for splitting scores, and three of the eight crosses come from the score or the separation. "Hopfield networks" is also too broad, since Definition 1 covers only discrete-time maps in a constant metric. The revised abstract and introduction therefore say "restricts" in place of "decides". To avoid any confusion during the review process, we have kept the current title. However, if the paper is accepted, we will revise it accordingly, for example to "The Readout Restricts Which Scores Give a Modern Hopfield Update a Constant-Metric Energy", if the program chairs permit the change.